GH-46222: [Python] Allow to specify footer when opening IPC file for writing #46354
+64
−10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
chaubold
changed the title
chaubold
force-pushed
the
GH-46222-allow-set-ipc-footer-pyarrow
branch
2 times, most recently
from
17c2586 to
af34416
Compare
|
Hey @chaubold ! Thanks for doing this, PR looks pretty straightforward, I can take a look at it tomorrow. |
github-actions
bot
added
awaiting changes
chaubold
force-pushed
the
GH-46222-allow-set-ipc-footer-pyarrow
branch
from
af34416 to
e139f66
Compare
github-actions
bot
added
awaiting change review
chaubold
force-pushed
the
GH-46222-allow-set-ipc-footer-pyarrow
branch
2 times, most recently
from
b23205b to
990ab60
Compare
chaubold
force-pushed
the
GH-46222-allow-set-ipc-footer-pyarrow
branch
from
990ab60 to
0a62797
Compare
|
Sorry for all the iterations here. But now the docstring complains are resolved and only the |
github-actions
bot
added
awaiting changes
chaubold
force-pushed
the
GH-46222-allow-set-ipc-footer-pyarrow
branch
from
56c834c to
bd7c065
Compare
raulcd
reviewed
May 9, 2025
| buffer = sink.getvalue() | ||
| with pa.ipc.open_file(buffer) as r: | ||
| assert r.metadata == meta | ||
|
|
There was a problem hiding this comment.
Could you add two empty lines to fix the linter failure?
Suggested change
github-actions
bot
added
awaiting changes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The Arrow APIs for other languages offer to set the footer metadata of an IPC file, but
pyarrowwas lacking this. I opened #46222 because we need this feature at KNIME, and took a shot at suggesting an implementation. Happy to get your feedback on the change!What changes are included in this PR?
A new keyword argument
metadatainRecordBatchFileWriter.__init__as well as inipc.new_file. The value isNoneby default to not break backwards compatibility.Also added a
metadataproperty to theRecordBatchFileReaderto be able to extract the metadata easily.Are these changes tested?
Yes, by a unit test.
Are there any user-facing changes?
Yes, see above :)