Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

branch-3.0: [fix](Export) Clone SessionVariable for each ExportTaskExecutor #48239 #48254

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from
Open

branch-3.0: [fix](Export) Clone SessionVariable for each ExportTaskExecutor #48239 #48254

wants to merge 1 commit into from
+2 −1

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #48239

@Thearas
Copy link
Contributor

Thearas commented Feb 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 24, 2025
@Thearas
Copy link
Contributor

Thearas commented Feb 24, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40568 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b5e410e4f59f093a4a32f39a239545621fecce7, data reload: false

------ Round 1 ----------------------------------
q1	17584	7376	7219	7219
q2	2056	183	170	170
q3	10787	1084	1144	1084
q4	10574	785	728	728
q5	7735	2797	2842	2797
q6	231	146	142	142
q7	971	610	596	596
q8	9347	1955	2012	1955
q9	6635	6328	6366	6328
q10	7056	2284	2324	2284
q11	465	259	263	259
q12	406	212	212	212
q13	17831	2999	3014	2999
q14	250	226	213	213
q15	576	527	526	526
q16	659	592	588	588
q17	971	595	597	595
q18	7280	6541	6732	6541
q19	1394	1108	997	997
q20	477	208	198	198
q21	3979	3320	3139	3139
q22	1111	1003	998	998
Total cold run time: 108375 ms
Total hot run time: 40568 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7301	7230	7249	7230
q2	333	236	229	229
q3	2902	2957	2903	2903
q4	2031	1796	1815	1796
q5	5662	5693	5710	5693
q6	225	139	140	139
q7	2213	1779	1848	1779
q8	3357	3515	3507	3507
q9	8718	8897	8853	8853
q10	3656	3612	3588	3588
q11	614	510	514	510
q12	775	616	624	616
q13	11369	3183	3134	3134
q14	318	294	277	277
q15	584	520	527	520
q16	687	655	636	636
q17	1848	1613	1630	1613
q18	8210	7646	7491	7491
q19	1637	1654	1528	1528
q20	2124	1840	1850	1840
q21	5583	5403	5265	5265
q22	1206	1049	1040	1040
Total cold run time: 71353 ms
Total hot run time: 60187 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198251 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b5e410e4f59f093a4a32f39a239545621fecce7, data reload: false

query1	1297	933	936	933
query2	6234	2137	2136	2136
query3	10952	4472	4461	4461
query4	66887	28868	23641	23641
query5	5018	452	448	448
query6	403	174	180	174
query7	5646	308	308	308
query8	305	229	224	224
query9	9085	2667	2662	2662
query10	469	276	260	260
query11	17641	15300	15902	15300
query12	163	109	102	102
query13	1567	462	421	421
query14	10843	7726	6870	6870
query15	221	185	185	185
query16	7076	495	534	495
query17	1067	576	612	576
query18	1983	335	315	315
query19	228	163	165	163
query20	121	115	115	115
query21	207	105	103	103
query22	4687	4661	4374	4374
query23	35295	34371	34411	34371
query24	6103	2957	2928	2928
query25	539	446	414	414
query26	667	175	167	167
query27	1899	357	360	357
query28	4480	2450	2414	2414
query29	682	470	440	440
query30	244	167	163	163
query31	1021	879	846	846
query32	68	59	57	57
query33	458	286	301	286
query34	909	508	507	507
query35	870	725	736	725
query36	1053	980	987	980
query37	122	67	70	67
query38	4113	4112	3980	3980
query39	1540	1493	1471	1471
query40	202	100	99	99
query41	49	48	50	48
query42	115	100	96	96
query43	533	522	514	514
query44	1177	843	854	843
query45	186	169	169	169
query46	1138	740	713	713
query47	2019	1872	1957	1872
query48	480	381	383	381
query49	731	394	392	392
query50	855	437	439	437
query51	7393	7260	7113	7113
query52	104	86	85	85
query53	260	181	180	180
query54	549	457	454	454
query55	79	77	74	74
query56	258	241	245	241
query57	1237	1122	1126	1122
query58	214	209	213	209
query59	3240	3004	2892	2892
query60	281	246	270	246
query61	112	104	107	104
query62	827	754	730	730
query63	213	188	191	188
query64	1353	658	664	658
query65	3294	3198	3220	3198
query66	704	302	315	302
query67	15730	15625	15617	15617
query68	4058	596	571	571
query69	425	257	261	257
query70	1182	1133	1141	1133
query71	351	260	250	250
query72	6068	3980	3950	3950
query73	745	349	345	345
query74	10240	9130	9114	9114
query75	3347	2660	2625	2625
query76	1833	1154	1081	1081
query77	545	298	283	283
query78	10737	9723	9616	9616
query79	1274	600	585	585
query80	821	444	433	433
query81	509	252	237	237
query82	1337	96	85	85
query83	248	138	139	138
query84	286	84	84	84
query85	865	305	288	288
query86	334	306	295	295
query87	4407	4291	4235	4235
query88	3350	2407	2377	2377
query89	417	296	297	296
query90	2022	187	187	187
query91	184	150	149	149
query92	67	49	48	48
query93	1258	542	554	542
query94	778	300	295	295
query95	348	257	253	253
query96	602	281	277	277
query97	3343	3270	3179	3179
query98	214	215	198	198
query99	1755	1429	1461	1429
Total cold run time: 319446 ms
Total hot run time: 198251 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.77 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b5e410e4f59f093a4a32f39a239545621fecce7, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.23	0.06	0.07
query4	1.63	0.11	0.10
query5	0.54	0.50	0.50
query6	1.15	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.56	0.49	0.49
query10	0.55	0.54	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.61
query14	2.82	2.73	2.76
query15	0.90	0.84	0.83
query16	0.39	0.39	0.41
query17	1.06	1.07	1.07
query18	0.23	0.21	0.22
query19	1.99	1.81	2.03
query20	0.02	0.01	0.00
query21	15.36	0.61	0.57
query22	2.51	2.13	1.79
query23	16.91	0.93	0.97
query24	3.26	0.90	1.90
query25	0.25	0.18	0.06
query26	0.50	0.13	0.13
query27	0.03	0.05	0.04
query28	9.86	1.09	1.06
query29	12.58	3.17	3.16
query30	0.25	0.06	0.06
query31	2.88	0.39	0.38
query32	3.24	0.46	0.45
query33	2.97	3.00	3.01
query34	17.30	4.53	4.56
query35	4.62	4.57	4.56
query36	0.68	0.49	0.51
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.81 s
Total hot run time: 32.77 s

@BePPPower @morningman
[fix](Export) Clone SessionVariable for each ExportTaskExecutor (#4…
225af7b 
…8239)

Problem Summary:

Because each `StmtExecutor` may modify the `SessionVariable`, we should
ensure each ExportTaskExecutor has its own cloned SessionVariable.
@morningman morningman force-pushed the auto-pick-48239-branch-3.0 branch from 8b5e410 to 225af7b Compare February 24, 2025 13:56
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40745 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 225af7b54fc529089e55bc883981ff16e15a0401, data reload: false

------ Round 1 ----------------------------------
q1	17600	7377	7223	7223
q2	2052	181	181	181
q3	10622	1100	1124	1100
q4	10545	706	701	701
q5	7753	2790	2842	2790
q6	243	147	146	146
q7	988	604	639	604
q8	9349	1952	2008	1952
q9	6560	6335	6379	6335
q10	6994	2292	2311	2292
q11	482	266	262	262
q12	399	218	213	213
q13	17790	2988	2993	2988
q14	238	206	210	206
q15	567	527	523	523
q16	666	588	586	586
q17	967	565	559	559
q18	7150	6696	6717	6696
q19	1398	1130	1093	1093
q20	483	210	202	202
q21	4206	3185	3140	3140
q22	1059	983	953	953
Total cold run time: 108111 ms
Total hot run time: 40745 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7261	7196	7217	7196
q2	332	238	231	231
q3	2889	2928	2958	2928
q4	2010	1775	1873	1775
q5	5677	5705	5712	5705
q6	220	143	145	143
q7	2226	1802	1805	1802
q8	3330	3518	3551	3518
q9	8737	8973	8882	8882
q10	3587	3568	3534	3534
q11	608	507	520	507
q12	806	591	600	591
q13	8143	3224	3176	3176
q14	304	268	269	268
q15	568	511	524	511
q16	681	657	660	657
q17	1838	1621	1600	1600
q18	8322	7659	7643	7643
q19	1654	1556	1510	1510
q20	2084	1862	1898	1862
q21	5634	5500	5457	5457
q22	1105	1059	1006	1006
Total cold run time: 68016 ms
Total hot run time: 60502 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197976 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 225af7b54fc529089e55bc883981ff16e15a0401, data reload: false

query1	1334	952	935	935
query2	6235	2023	2024	2023
query3	10961	4214	4456	4214
query4	65966	28993	23948	23948
query5	5051	450	436	436
query6	409	177	177	177
query7	5653	325	309	309
query8	306	225	222	222
query9	9427	2679	2679	2679
query10	486	270	249	249
query11	17686	15017	15903	15017
query12	155	101	104	101
query13	1584	442	441	441
query14	9730	6823	6876	6823
query15	211	184	178	178
query16	7105	513	492	492
query17	1087	591	623	591
query18	1781	330	322	322
query19	231	165	170	165
query20	122	116	113	113
query21	229	103	103	103
query22	4843	4579	4494	4494
query23	34448	34137	34330	34137
query24	6742	2897	2899	2897
query25	538	423	427	423
query26	668	174	173	173
query27	2111	351	365	351
query28	4209	2480	2426	2426
query29	752	459	420	420
query30	241	166	159	159
query31	990	807	846	807
query32	74	53	58	53
query33	446	292	278	278
query34	918	501	518	501
query35	840	735	742	735
query36	1104	946	936	936
query37	115	63	68	63
query38	4063	4159	4074	4074
query39	1544	1667	1475	1475
query40	209	94	100	94
query41	52	48	47	47
query42	120	98	96	96
query43	528	492	487	487
query44	1171	822	816	816
query45	176	173	181	173
query46	1171	726	725	725
query47	2003	1912	1916	1912
query48	477	389	404	389
query49	745	393	388	388
query50	849	421	413	413
query51	7279	7111	7116	7111
query52	99	88	89	88
query53	269	174	183	174
query54	549	441	445	441
query55	78	75	79	75
query56	257	239	246	239
query57	1244	1121	1086	1086
query58	210	205	209	205
query59	3134	3108	3123	3108
query60	290	248	236	236
query61	113	122	117	117
query62	822	715	755	715
query63	215	187	182	182
query64	1385	686	634	634
query65	3301	3218	3191	3191
query66	709	307	316	307
query67	15865	15542	15446	15446
query68	3993	580	581	580
query69	422	275	257	257
query70	1170	1135	1135	1135
query71	358	252	249	249
query72	6443	4041	4198	4041
query73	741	343	344	343
query74	10300	9368	9394	9368
query75	3329	2658	2657	2657
query76	2000	1040	1073	1040
query77	484	271	267	267
query78	10629	9546	9524	9524
query79	1107	592	612	592
query80	737	414	419	414
query81	504	242	232	232
query82	498	84	87	84
query83	159	143	141	141
query84	276	78	84	78
query85	810	290	294	290
query86	320	306	289	289
query87	4420	4368	4404	4368
query88	3661	2411	2347	2347
query89	414	294	289	289
query90	2028	183	183	183
query91	181	150	146	146
query92	60	50	51	50
query93	1088	550	545	545
query94	761	284	293	284
query95	358	255	252	252
query96	602	289	284	284
query97	3297	3197	3185	3185
query98	220	202	200	200
query99	1665	1439	1427	1427
Total cold run time: 316750 ms
Total hot run time: 197976 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 225af7b54fc529089e55bc883981ff16e15a0401, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.22	0.06	0.07
query4	1.63	0.11	0.10
query5	0.52	0.53	0.53
query6	1.13	0.73	0.73
query7	0.02	0.02	0.03
query8	0.05	0.03	0.03
query9	0.57	0.51	0.50
query10	0.54	0.56	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.10
query13	0.61	0.59	0.61
query14	2.77	2.82	2.85
query15	0.89	0.83	0.82
query16	0.38	0.38	0.38
query17	1.04	1.04	1.06
query18	0.23	0.23	0.22
query19	1.89	1.80	2.02
query20	0.01	0.01	0.01
query21	15.37	0.58	0.57
query22	3.03	1.77	2.02
query23	17.19	0.94	0.87
query24	2.74	1.47	1.74
query25	0.26	0.12	0.12
query26	0.38	0.14	0.13
query27	0.04	0.05	0.04
query28	9.83	1.13	1.08
query29	12.59	3.23	3.20
query30	0.25	0.06	0.06
query31	2.86	0.40	0.37
query32	3.25	0.46	0.46
query33	3.01	3.00	3.07
query34	16.91	4.53	4.43
query35	4.59	4.52	4.45
query36	0.66	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.35 s
Total hot run time: 33.27 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Thearas @doris-robot @morningman @dataroaring @BePPPower