[BugFix] SpillableHashJoinProbeOperator memory consumption for building partition hash table #56813
+36
−4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
Fixes #56491
SpillableHashJoinProbeOperator
will take much memory to restore/rebuild the partition hash table when DoP is elevated. Although there is a Memory Resource Manager that could help to control the memory usage inSpillableHashJoinProbeOperators
( by limiting the size of selected partition to smaller than 2 * spill_mem_table_size ), Memory Resource Manager hasn't considered multi-operators.Considering Q97 in TPC-DS, it will take about 3-4 GB to rebuild the partition hash table at a time
Memory is calculated as below:
This PR will introduce a config called
spillHashJoinProbeOpMaxBytes
to help reduce memory consumption when building the partition hash table by limiting memory usage across all probe operators.What I'm doing:
Fixes #56491
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: