Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids) fix str_to_date date value out of range #49033

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[fix](Nereids) fix str_to_date date value out of range #49033

wants to merge 1 commit into from
+31 −15

Conversation

LiBinfeng-01
Copy link
Collaborator

@LiBinfeng-01 LiBinfeng-01 commented Mar 13, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

SELECT STR_TO_DATE('2025-04-31', '%Y-%m-%d') AS result; would be changed to result 2025-04-30 in smart mode

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32372 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39bc87e91805fbdd224363d3a737f52de217a248, data reload: false

------ Round 1 ----------------------------------
q1	17692	5241	5105	5105
q2	2041	276	161	161
q3	10431	1313	688	688
q4	10207	1037	531	531
q5	7523	2397	2317	2317
q6	189	162	132	132
q7	896	741	593	593
q8	9295	1271	1016	1016
q9	5013	4891	4726	4726
q10	6825	2324	1885	1885
q11	486	273	252	252
q12	345	349	219	219
q13	17766	3697	3100	3100
q14	229	222	216	216
q15	519	479	487	479
q16	627	610	589	589
q17	566	880	338	338
q18	6807	6485	6357	6357
q19	1216	953	551	551
q20	319	327	195	195
q21	2781	2122	1970	1970
q22	1062	1001	952	952
Total cold run time: 102835 ms
Total hot run time: 32372 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5229	5219	5218	5218
q2	242	331	235	235
q3	2189	2810	2312	2312
q4	1435	1818	1405	1405
q5	4211	4107	4156	4107
q6	206	164	127	127
q7	1864	1923	1782	1782
q8	2625	2681	2549	2549
q9	7324	7126	7166	7126
q10	2991	3219	2756	2756
q11	585	531	503	503
q12	709	813	634	634
q13	3502	3908	3261	3261
q14	275	300	269	269
q15	514	473	476	473
q16	657	694	654	654
q17	1141	1671	1309	1309
q18	7874	7538	7449	7449
q19	802	797	893	797
q20	1993	2079	1864	1864
q21	5507	5020	4685	4685
q22	1090	1056	995	995
Total cold run time: 52965 ms
Total hot run time: 50510 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186662 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39bc87e91805fbdd224363d3a737f52de217a248, data reload: false

query1	1008	402	374	374
query2	6542	1925	1917	1917
query3	6788	227	226	226
query4	26390	23639	23060	23060
query5	4335	668	510	510
query6	296	199	194	194
query7	4607	485	293	293
query8	300	246	244	244
query9	8597	2628	2617	2617
query10	489	312	267	267
query11	15963	15248	15048	15048
query12	169	112	108	108
query13	1656	526	412	412
query14	10060	6987	6719	6719
query15	203	193	167	167
query16	7479	638	486	486
query17	1208	729	571	571
query18	1969	410	310	310
query19	191	191	167	167
query20	123	120	114	114
query21	212	122	106	106
query22	4131	4312	4279	4279
query23	33920	33079	33198	33079
query24	7678	2393	2433	2393
query25	521	453	383	383
query26	1209	273	155	155
query27	2115	479	336	336
query28	3881	2427	2399	2399
query29	718	553	445	445
query30	281	216	207	207
query31	947	888	782	782
query32	73	67	67	67
query33	566	377	302	302
query34	787	852	501	501
query35	804	819	756	756
query36	959	1003	886	886
query37	114	95	74	74
query38	4321	4100	4135	4100
query39	1443	1412	1405	1405
query40	204	119	111	111
query41	56	53	54	53
query42	117	105	103	103
query43	510	528	503	503
query44	1319	784	782	782
query45	176	169	174	169
query46	824	1019	624	624
query47	1762	1786	1714	1714
query48	373	426	295	295
query49	774	502	441	441
query50	666	721	436	436
query51	4138	4218	4243	4218
query52	106	106	99	99
query53	232	258	188	188
query54	518	496	439	439
query55	85	81	90	81
query56	276	267	258	258
query57	1114	1138	1047	1047
query58	250	243	251	243
query59	2457	2656	2687	2656
query60	301	270	270	270
query61	122	120	117	117
query62	811	729	665	665
query63	230	189	194	189
query64	4190	989	692	692
query65	4492	4348	4338	4338
query66	1113	404	297	297
query67	15634	15560	15461	15461
query68	7931	859	501	501
query69	460	310	314	310
query70	1197	1108	1080	1080
query71	443	293	280	280
query72	5526	3738	3736	3736
query73	812	741	353	353
query74	9012	8857	8772	8772
query75	3717	3236	2703	2703
query76	3597	1173	771	771
query77	797	386	283	283
query78	10088	10005	9247	9247
query79	2581	808	594	594
query80	604	549	448	448
query81	499	262	224	224
query82	703	126	98	98
query83	174	169	153	153
query84	249	88	76	76
query85	845	353	306	306
query86	381	305	277	277
query87	4359	4459	4350	4350
query88	3711	2275	2264	2264
query89	387	318	282	282
query90	1892	221	223	221
query91	144	134	113	113
query92	76	66	57	57
query93	1740	1065	576	576
query94	647	398	311	311
query95	367	277	266	266
query96	483	561	274	274
query97	3296	3402	3298	3298
query98	232	205	202	202
query99	1350	1389	1288	1288
Total cold run time: 274203 ms
Total hot run time: 186662 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39bc87e91805fbdd224363d3a737f52de217a248, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.23	0.08	0.07
query4	1.61	0.10	0.11
query5	0.58	0.56	0.55
query6	1.19	0.71	0.72
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.59	0.52	0.51
query10	0.59	0.62	0.58
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	2.81	2.83	2.68
query15	0.92	0.85	0.86
query16	0.38	0.37	0.39
query17	1.07	1.01	1.01
query18	0.21	0.20	0.19
query19	1.89	1.78	2.02
query20	0.01	0.01	0.01
query21	15.35	0.92	0.55
query22	0.75	1.23	0.62
query23	14.98	1.38	0.65
query24	6.97	2.15	0.79
query25	0.50	0.23	0.08
query26	0.55	0.16	0.12
query27	0.06	0.05	0.06
query28	9.06	0.80	0.44
query29	12.56	3.99	3.32
query30	0.26	0.09	0.06
query31	2.82	0.59	0.39
query32	3.23	0.55	0.46
query33	2.96	2.98	3.04
query34	15.73	5.19	4.53
query35	4.54	4.57	4.54
query36	0.69	0.50	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 104.67 s
Total hot run time: 30.74 s

morrySnow
morrySnow reviewed Mar 14, 2025
View reviewed changes
...apache/doris/nereids/trees/expressions/functions/executable/DateTimeExtractAndTransform.java Outdated
@@ -631,19 +636,19 @@ public static Expression strToDate(StringLikeLiteral str, StringLikeLiteral form
boolean hasMicroPart = org.apache.doris.analysis.DateLiteral
.hasMicroSecondPart(format.getStringValue());
return DateTimeV2Literal.fromJavaDateType(DateUtils.getTime(DateUtils.formatBuilder(format.getValue())
.toFormatter(), str.getValue()), hasMicroPart ? 6 : 0);
.toFormatter().withResolverStyle(ResolverStyle.STRICT), str.getValue()), hasMicroPart ? 6 : 0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we refactor formatBuilder lightly to avoid call .toFormatter().withResolverStyle(ResolverStyle.STRICT) too many times in different place

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

toFormatter is one of DateTimeFormatterBuilder.java functions, which is not easy to refactor. And we use it a lot, it may cause a wide range of affection

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32276 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7198191dcf0b8b70c8202fd90a1f36a342fe2cf5, data reload: false

------ Round 1 ----------------------------------
q1	17577	5227	5117	5117
q2	2039	295	162	162
q3	10448	1210	722	722
q4	10203	999	512	512
q5	7522	2359	2315	2315
q6	185	161	132	132
q7	925	764	594	594
q8	9287	1204	1079	1079
q9	4938	4679	4853	4679
q10	6845	2278	1861	1861
q11	491	268	251	251
q12	345	344	227	227
q13	17762	3672	3057	3057
q14	232	232	210	210
q15	571	487	491	487
q16	638	618	594	594
q17	556	869	349	349
q18	6906	6368	6317	6317
q19	1408	950	561	561
q20	316	325	193	193
q21	2722	2220	1892	1892
q22	1019	1022	965	965
Total cold run time: 102935 ms
Total hot run time: 32276 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5212	5194	5202	5194
q2	244	332	234	234
q3	2156	2749	2301	2301
q4	1455	1850	1431	1431
q5	4328	4244	4274	4244
q6	211	176	128	128
q7	1917	1913	1772	1772
q8	2667	2656	2608	2608
q9	7374	7323	7308	7308
q10	3020	3257	2726	2726
q11	571	503	479	479
q12	705	748	571	571
q13	3453	3961	3327	3327
q14	291	288	284	284
q15	523	475	467	467
q16	638	676	654	654
q17	1136	1623	1331	1331
q18	7649	7543	7410	7410
q19	824	779	821	779
q20	1989	2025	1875	1875
q21	5401	4961	4698	4698
q22	1073	1033	1026	1026
Total cold run time: 52837 ms
Total hot run time: 50847 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191584 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7198191dcf0b8b70c8202fd90a1f36a342fe2cf5, data reload: false

query1	1426	987	994	987
query2	6400	1986	1935	1935
query3	11006	4475	4587	4475
query4	25857	23699	23399	23399
query5	4708	640	478	478
query6	313	198	221	198
query7	3996	512	289	289
query8	300	254	236	236
query9	8539	2615	2607	2607
query10	496	307	255	255
query11	15438	15452	14977	14977
query12	178	111	104	104
query13	1687	533	382	382
query14	10396	6268	6203	6203
query15	204	181	169	169
query16	7517	636	518	518
query17	1209	712	560	560
query18	1913	425	349	349
query19	197	186	168	168
query20	131	126	121	121
query21	203	133	106	106
query22	4597	4530	4288	4288
query23	34318	33324	33412	33324
query24	7370	2440	2436	2436
query25	509	469	413	413
query26	1209	271	157	157
query27	2706	467	319	319
query28	5173	2436	2410	2410
query29	700	574	424	424
query30	273	226	190	190
query31	921	865	769	769
query32	74	66	104	66
query33	545	376	297	297
query34	808	865	535	535
query35	818	857	771	771
query36	980	1012	899	899
query37	120	108	78	78
query38	4160	4379	4159	4159
query39	1494	1444	1449	1444
query40	207	114	102	102
query41	52	49	49	49
query42	118	107	116	107
query43	513	533	497	497
query44	1320	807	796	796
query45	187	175	168	168
query46	862	1023	646	646
query47	1845	1884	1854	1854
query48	389	413	310	310
query49	794	502	434	434
query50	700	758	419	419
query51	4352	4330	4270	4270
query52	111	109	96	96
query53	239	280	191	191
query54	494	501	413	413
query55	84	87	92	87
query56	274	261	261	261
query57	1161	1174	1088	1088
query58	263	251	259	251
query59	2767	2873	2546	2546
query60	295	326	267	267
query61	121	129	120	120
query62	801	721	672	672
query63	227	209	191	191
query64	4063	1019	689	689
query65	4504	4455	4492	4455
query66	961	403	294	294
query67	16186	15518	15139	15139
query68	8152	870	502	502
query69	477	345	269	269
query70	1180	1098	1058	1058
query71	428	300	265	265
query72	5618	3567	3785	3567
query73	733	735	345	345
query74	8978	8964	8673	8673
query75	3409	3164	2695	2695
query76	3406	1203	735	735
query77	768	370	272	272
query78	9984	10127	9395	9395
query79	1853	841	595	595
query80	618	524	510	510
query81	485	260	221	221
query82	231	120	97	97
query83	179	165	146	146
query84	240	94	72	72
query85	760	353	313	313
query86	376	318	281	281
query87	4403	4584	4453	4453
query88	3190	2317	2299	2299
query89	387	321	283	283
query90	1937	221	225	221
query91	146	156	114	114
query92	70	62	59	59
query93	1945	1047	575	575
query94	646	402	292	292
query95	359	276	263	263
query96	488	567	280	280
query97	3281	3374	3322	3322
query98	233	202	193	193
query99	1334	1413	1312	1312
Total cold run time: 279073 ms
Total hot run time: 191584 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7198191dcf0b8b70c8202fd90a1f36a342fe2cf5, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.57	0.55	0.57
query6	1.16	0.72	0.72
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.60	0.52	0.53
query10	0.56	0.60	0.58
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.63	0.61	0.60
query14	2.82	2.82	2.74
query15	0.92	0.86	0.85
query16	0.40	0.37	0.39
query17	1.03	1.02	1.02
query18	0.22	0.19	0.19
query19	1.93	1.79	2.01
query20	0.01	0.01	0.01
query21	15.38	0.91	0.54
query22	0.76	1.38	0.96
query23	14.76	1.36	0.59
query24	6.93	1.47	1.32
query25	0.51	0.25	0.09
query26	0.53	0.16	0.13
query27	0.06	0.06	0.05
query28	9.94	0.82	0.44
query29	12.52	3.91	3.28
query30	0.25	0.09	0.06
query31	2.83	0.59	0.39
query32	3.22	0.54	0.46
query33	3.06	3.01	3.12
query34	15.78	5.05	4.51
query35	4.52	4.58	4.50
query36	0.66	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.4 s
Total hot run time: 31.54 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
dev/2.1.x dev/3.0.x p0_w
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@LiBinfeng-01 @hello-stephen @doris-robot @morrySnow