Skip to content

fix: preserve RemoteTemporaryFile type when working with remote storage #4290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: 3.1
Choose a base branch
from

Conversation

hschimpf
Copy link

@hschimpf hschimpf commented Apr 3, 2025

1️⃣ Why should it be added? What are the benefits of this change?

This PR fixes a critical bug in distributed environments where Laravel Excel exports are queued but processed across multiple servers. When using a remote disk for temporary files, if a different server processes the initial export job than the one that received the request, the export fails.

The issue occurs because the current code flow replaces the RemoteTemporaryFile instance with a new LocalTemporaryFile when trying to ensure the local file exists 1. This type change breaks the subsequent operations that expect a RemoteTemporaryFile and the remote file is never updated 2. This results in an empty file being loaded when the next job in the chain tries to process it.

The fix preserves the RemoteTemporaryFile type while ensuring the local directory structure exists for writing, making queued exports work reliably in distributed environments.

2️⃣ Does it contain multiple, unrelated changes? Please separate the PRs out.

No, this PR contains a single focused fix for the RemoteTemporaryFile handling issue.

3️⃣ Does it include tests, if possible?

No, as this is specifically related to multi-server distributed environments which would be challenging to simulate in automated tests.

4️⃣ Any drawbacks? Possible breaking changes?

No breaking changes. The modification to RemoteTemporaryFile::sync() adds an optional parameter with a default value that preserves the original behavior.

5️⃣ Mark the following tasks as done:

  • Checked the codebase to ensure that your feature doesn't already exist.
  • Take note of the contributing guidelines.
  • Checked the pull requests to ensure that another person hasn't already submitted a fix.
  • [ ] Added tests to ensure against regression. (not applicable in this case)

6️⃣ Thanks for contributing! 🙌


Additional Information

Problem Details:
In a distributed environment with multiple servers (A and B), when using a remote disk for temporary files:

  1. Server A handles the initial request from a controller that queues the Excel export through Exportable::queue()
    • A LocalTemporaryFile instance is created 3 (RemoteTemporaryFile if a temporalDisk is specified 4).
    • LocalTemporaryFile touches the file on the local disk 5.
    • RemoteTemporaryFile touches the file on the remote server 6.
  2. The export jobs are queued in a chain 7 (QueueExportAppendQueryToSheet|AppendDataToSheet|... → CloseSheetStoreQueuedExport).
  3. If Server B picks up the QueueExport job, it detects that the local file doesn't exist 1.
  4. In the original version, it created a new LocalTemporaryFile instance, changing the type from RemoteTemporaryFile to LocalTemporaryFile.
  5. When saving the Excel file, the remote file was never updated because the condition instanceof RemoteTemporaryFile 2 failed.
  6. Subsequent jobs failed because when they load the temporary file, they load an empty file from the remote location (since it was never updated).

The fix preserves the RemoteTemporaryFile type by modifying RemoteTemporaryFile::sync() to accept an optional parameter that allows creating the directory structure without copying content (which would be redundant since it's about to be overwritten anyway) 2.


Footnotes

  1. https://github.com/SpartnerNL/Laravel-Excel/blob/e25d44a2d91da9179cd2d7fec952313548597a79/src/Writer.php#L179-L182 2

  2. https://github.com/SpartnerNL/Laravel-Excel/blob/e25d44a2d91da9179cd2d7fec952313548597a79/src/Writer.php#L188-L191 2 3

  3. https://github.com/SpartnerNL/Laravel-Excel/blob/e25d44a2d91da9179cd2d7fec952313548597a79/src/QueuedWriter.php#L64

  4. https://github.com/SpartnerNL/Laravel-Excel/blob/e25d44a2d91da9179cd2d7fec952313548597a79/src/Files/TemporaryFileFactory.php#L35-L37

  5. https://github.com/SpartnerNL/Laravel-Excel/blob/e25d44a2d91da9179cd2d7fec952313548597a79/src/Files/LocalTemporaryFile.php#L17

  6. https://github.com/SpartnerNL/Laravel-Excel/blob/e25d44a2d91da9179cd2d7fec952313548597a79/src/Files/RemoteTemporaryFile.php#L40

  7. https://github.com/SpartnerNL/Laravel-Excel/blob/e25d44a2d91da9179cd2d7fec952313548597a79/src/QueuedWriter.php#L76

…rage

- Added optional parameter to `RemoteTemporaryFile::sync()` allowing directory creation without content copying
- Modified `Writer` to use `RemoteTemporaryFile::sync(false)` instead of creating a new `LocalTemporaryFile`
- Prevents type change from `RemoteTemporaryFile` to `LocalTemporaryFile` while ensuring local directory structure exists for writing
@hschimpf
Copy link
Author

hschimpf commented Apr 3, 2025

This PR may be related and solve the following issues: #2796, #3583, and #3863

@hschimpf hschimpf force-pushed the remote-temporary-file branch from db131e9 to 228e4af Compare April 3, 2025 15:48
@leonardyrj
Copy link

Any prediction of releasing this PR? I'm having the same problem.

@patrickbrouwers
Copy link
Member

If you can use his fork in your project to test if it fixes the problem, I ll see if I can release it. I don't have a reproduction situation I can test this one, so no time to look into it right now

@leonardyrj
Copy link

I followed your tip in encapsulating everything in a single job. That way it worked!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants