[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache #56871

satanson · 2025-03-13T08:11:29Z

Why I'm doing:

When query cache is enabled, some query report errors as follows:

ava.lang.ClassCastException: class com.starrocks.analysis.StringLiteral cannot be cast to class com.starrocks.analysis.DateLiteral (com.starrocks.analysis.StringLiteral and com.starrocks.analysis.DateLiteral are in unnamed module of loader 'app')
        at com.starrocks.catalog.PartitionKey.successor(PartitionKey.java:363) ~[starrocks-fe.jar:?]
        at com.starrocks.planner.FragmentNormalizer.toClosedOpenRange(FragmentNormalizer.java:124) ~[starrocks-fe.jar:?]
        at com.starrocks.planner.FragmentNormalizer.getPartitionRangePredicates(FragmentNormalizer.java:576) ~[starrocks-fe.jar:?]
        at com.starrocks.planner.OlapScanNode.decomposeRangePredicates(OlapScanNode.java:1238) ~[starrocks-fe.jar:?]
        at com.starrocks.planner.OlapScanNode.normalizeConjuncts(OlapScanNode.java:1266) ~[starrocks-fe.jar:?]
        at com.starrocks.planner.OlapScanNode.toNormalForm(OlapScanNode.java:1360) ~[starrocks-fe.jar:?]
        at com.starrocks.planner.PlanNode.normalize(PlanNode.java:998) ~[starrocks-fe.jar:?]
        at com.starrocks.planner.FragmentNormalizer.normalizeSubTree(FragmentNormalizer.java:383) ~[starrocks-fe.jar:?]
        at com.starrocks.planner.FragmentNormalizer.normalizeSubTree(FragmentNormalizer.java:372) ~[starrocks-fe.jar:?]
        at com.starrocks.planner.FragmentNormalizer.normalizeSubTree(FragmentNormalizer.java:372) ~[starrocks-fe.jar:?]
        at com.starrocks.planner.FragmentNormalizer.normalize(FragmentNormalizer.java:811) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.plan.PlanFragmentBuilder.finalizeFragments(PlanFragmentBuilder.java:398) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.plan.PlanFragmentBuilder.createPhysicalPlan(PlanFragmentBuilder.java:243) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.StatementPlanner.createQueryPlanWithReTry(StatementPlanner.java:336) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:133) ~[starrocks-fe.jar:?]
        at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:92) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:557) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:356) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:551) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:885) ~[starrocks-fe.jar:?]
        at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]

we can reproduce this error as follow steps:

create a hive or iceberg table

CREATE TABLE `store_sales2_flat` (
  `ss_sold_date_sk` bigint(20) DEFAULT NULL,
  `ss_sold_time_sk` bigint(20) DEFAULT NULL,
  `ss_item_sk` bigint(20) DEFAULT NULL,
  `ss_customer_sk` bigint(20) DEFAULT NULL,
  `ss_sold_date` varchar(1073741824) DEFAULT NULL
)
PARTITION BY (ss_sold_date);

ingest some rows which contains ss_sold_date = '20001231', notice that ss_sold_date's format is yyyyMMdd.
create a mv on it, use str2date(ss_sold_date, '%Y%m%d') as partittion by expr.

CREATE MATERIALIZED VIEW `mv2` (`ss_sold_date`, `ss_customer_sk`, `_ca0002`)
COMMENT "MV recommended by AutoMV"
PARTITION BY (str2date(`ss_sold_date`, '%Y%m%d'))
DISTRIBUTED BY HASH(`ss_sold_date`) BUCKETS 64 
ORDER BY (ss_sold_date)
REFRESH ASYNC START("2023-12-01 10:00:00") EVERY(INTERVAL 1 DAY)
PROPERTIES (
"replicated_storage" = "true",
"replication_num" = "3",
"force_external_table_query_rewrite" = "CHECKED",
"session.enable_spill" = "true",
"storage_medium" = "HDD"
)
AS SELECT `store_sales2_flat`.`ss_sold_date`, `store_sales2_flat`.`ss_customer_sk`, count(1) AS `_ca0002`
FROM `flat_tpcds_db`.`store_sales2_flat`
GROUP BY `store_sales2_flat`.`ss_sold_date`, `store_sales2_flat`.`ss_customer_sk`;

issue the query and get the error

mysql> explain costs select count(*) from emr_iceberg_test.flat_tpcds_db.store_sales2_flat where ss_sold_date = '20001231';
ERROR 1064 (HY000): class com.starrocks.analysis.StringLiteral cannot be cast to class com.starrocks.analysis.DateLiteral (com.starrocks.analysis.StringLiteral and com.starrocks.analysis.DateLiteral are in unnamed module of loader 'app')

The reason is that when query cache is enabled, we try to decompose the the filter ss_sold_date='20001231', at first is converted to a range ['20001231', '20001231'], it happens overlap MV's partition ['2000-12-31', ''2001-01-01'), then the range tries to be converted a closedOpen range and it fails, since '20001231' can not be recognized as a legal date string(legcal date string must be in format %Y-%m-%d).

What I'm doing:

MV's partition expr as follows can be decomposed

slotRef
date_trunc
str2date(dt, '%Y-%m-d');

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

…query cache Signed-off-by: satanson <[email protected]>

stdpain · 2025-03-13T08:15:07Z

fe/fe-core/src/main/java/com/starrocks/planner/FragmentNormalizer.java

            Pair<Long, Range<PartitionKey>> partitionKeyRange = rangeMap.get(i);
            // when the range is to total cover this partition, we also cache it
-            if (!range.isEmpty()) {
-                selectedRangeMap.put(partitionKeyRange.first, range.toString());
+            if (optRange.isPresent() && !optRange.get().isEmpty()) {


Is it possiable two different partition has the same selectedRange?

Impossible, at present, only single column range partition is take into consideration, so partitions are not intersected with each other. A selected range is just a sub-range of a certain partition.

fe/fe-core/src/main/java/com/starrocks/planner/OlapScanNode.java

sonarqubecloud · 2025-03-13T09:08:40Z

Quality Gate passed

Issues
17 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2025-03-13T10:43:20Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2025-03-13T10:43:56Z

[FE Incremental Coverage Report]

❌ fail : 29 / 38 (76.32%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/catalog/ExpressionRangePartitionInfoV2.java	0	1	00.00%	[247]
🔵	com/starrocks/planner/OlapScanNode.java	23	30	76.67%	[1341, 1344, 1354, 1383, 1384, 1385, 1387]
🔵	com/starrocks/planner/FragmentNormalizer.java	5	6	83.33%	[585]
🔵	com/starrocks/catalog/ExpressionRangePartitionInfo.java	1	1	100.00%	[]

github-actions · 2025-03-13T10:44:21Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2025-03-14T02:17:31Z

@Mergifyio backport branch-3.2

github-actions · 2025-03-14T02:17:31Z

@Mergifyio backport branch-3.3

github-actions · 2025-03-14T02:17:32Z

@Mergifyio backport branch-3.4

mergify · 2025-03-14T02:17:37Z

backport branch-3.2

✅ Backports have been created

#56907 [BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache (backport #56871) has been created for branch branch-3.2 but encountered conflicts

mergify · 2025-03-14T02:17:38Z

backport branch-3.3

✅ Backports have been created

#56908 [BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache (backport #56871) has been created for branch branch-3.3 but encountered conflicts

mergify · 2025-03-14T02:17:39Z

backport branch-3.4

✅ Backports have been created

#56909 [BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache (backport #56871) has been created for branch branch-3.4

…query cache (#56871) Signed-off-by: satanson <[email protected]> (cherry picked from commit 7b82d5f) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/catalog/ExpressionRangePartitionInfo.java # fe/fe-core/src/main/java/com/starrocks/catalog/ExpressionRangePartitionInfoV2.java # fe/fe-core/src/main/java/com/starrocks/planner/FragmentNormalizer.java # fe/fe-core/src/main/java/com/starrocks/planner/OlapScanNode.java

…query cache (#56871) Signed-off-by: satanson <[email protected]> (cherry picked from commit 7b82d5f) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/planner/FragmentNormalizer.java

…query cache (#56871) Signed-off-by: satanson <[email protected]> (cherry picked from commit 7b82d5f)

…query cache (backport #56871) (#56909) Co-authored-by: satanson <[email protected]>

…query cache (#56871) Signed-off-by: satanson <[email protected]> (cherry picked from commit 7b82d5f) Signed-off-by: satanson <[email protected]>

…query cache (backport #56871) Signed-off-by: satanson <[email protected]> (cherry picked from commit 7b82d5f) Signed-off-by: satanson <[email protected]>

…query cache (backport #56871) (#56935) Signed-off-by: satanson <[email protected]>

…query cache (backport #56871) (#56933) Signed-off-by: satanson <[email protected]>

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in …

e1f0a6a

…query cache Signed-off-by: satanson <[email protected]>

satanson requested a review from a team as a code owner March 13, 2025 08:11

github-actions bot added 3.3 3.4 3.2 labels Mar 13, 2025

mergify bot assigned satanson Mar 13, 2025

stdpain reviewed Mar 13, 2025

View reviewed changes

starrocks-cr bot reviewed Mar 13, 2025

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/planner/OlapScanNode.java Show resolved Hide resolved

LiShuMing approved these changes Mar 14, 2025

View reviewed changes

kangkaisen approved these changes Mar 14, 2025

View reviewed changes

ZiheLiu approved these changes Mar 14, 2025

View reviewed changes

satanson enabled auto-merge (squash) March 14, 2025 02:10

kangkaisen disabled auto-merge March 14, 2025 02:17

kangkaisen merged commit 7b82d5f into main Mar 14, 2025
69 of 70 checks passed

kangkaisen deleted the query_cache_mv_part_by_non_slot_ref branch March 14, 2025 02:17

github-actions bot removed 3.2 3.3 3.4 labels Mar 14, 2025

mergify bot mentioned this pull request Mar 14, 2025

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache (backport #56871) #56907

Closed

18 tasks

mergify bot pushed a commit that referenced this pull request Mar 14, 2025

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in …

e0e5c44

…query cache (#56871) Signed-off-by: satanson <[email protected]> (cherry picked from commit 7b82d5f)

This was referenced Mar 14, 2025

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache (backport #56871) #56908

Closed

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache (backport #56871) #56909

Merged

wanpengfei-git pushed a commit that referenced this pull request Mar 14, 2025

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in …

ed7e3cc

…query cache (backport #56871) (#56909) Co-authored-by: satanson <[email protected]>

github-actions bot added the 3.4-merged label Mar 14, 2025

satanson mentioned this pull request Mar 14, 2025

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache (backport #56871) #56933

Merged

24 tasks

satanson mentioned this pull request Mar 14, 2025

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache (backport #56871) #56935

Merged

24 tasks

kangkaisen pushed a commit that referenced this pull request Mar 14, 2025

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in …

2689c2e

…query cache (backport #56871) (#56935) Signed-off-by: satanson <[email protected]>

github-actions bot added the 3.2-merged label Mar 14, 2025

kangkaisen pushed a commit that referenced this pull request Mar 14, 2025

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in …

eac7d72

…query cache (backport #56871) (#56933) Signed-off-by: satanson <[email protected]>

github-actions bot added the 3.3-merged label Mar 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache #56871

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache #56871

satanson commented Mar 13, 2025

stdpain Mar 13, 2025

satanson Mar 13, 2025

sonarqubecloud bot commented Mar 13, 2025

github-actions bot commented Mar 13, 2025

github-actions bot commented Mar 13, 2025

github-actions bot commented Mar 13, 2025

github-actions bot commented Mar 14, 2025

github-actions bot commented Mar 14, 2025

github-actions bot commented Mar 14, 2025

mergify bot commented Mar 14, 2025 •

edited

Loading

mergify bot commented Mar 14, 2025 •

edited

Loading

mergify bot commented Mar 14, 2025 •

edited

Loading

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache #56871

[BugFix] MV partitioned by non-SlotRef Expr can not be decomposed in query cache #56871

Conversation

satanson commented Mar 13, 2025

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

stdpain Mar 13, 2025

Choose a reason for hiding this comment

satanson Mar 13, 2025

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 13, 2025

Quality Gate passed

github-actions bot commented Mar 13, 2025

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Mar 13, 2025

[FE Incremental Coverage Report]

file detail

github-actions bot commented Mar 13, 2025

[BE Incremental Coverage Report]

github-actions bot commented Mar 14, 2025

github-actions bot commented Mar 14, 2025

github-actions bot commented Mar 14, 2025

mergify bot commented Mar 14, 2025 • edited Loading

✅ Backports have been created

mergify bot commented Mar 14, 2025 • edited Loading

✅ Backports have been created

mergify bot commented Mar 14, 2025 • edited Loading

✅ Backports have been created

mergify bot commented Mar 14, 2025 •

edited

Loading

mergify bot commented Mar 14, 2025 •

edited

Loading

mergify bot commented Mar 14, 2025 •

edited

Loading