Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[optimize](short circuit) avoid set cacheId when non prepared execute #40525

Merged
merged 1 commit into from
Sep 12, 2024
Merged

[optimize](short circuit) avoid set cacheId when non prepared execute #40525

merged 1 commit into from
Sep 12, 2024

Conversation

eldenmoon
Copy link
Member

Only set cacheID for prepared statement excute phase, otherwise leading to many redundant cost in BE side

@eldenmoon
[optimize](short circuit) avoid set cacheId when non prepared execute
332273f 
Only set cacheID for prepared statement excute phase,
otherwise leading to many redundant cost in BE side
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38019 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 332273fbb0fc4ef557b73d24828545d17c810c20, data reload: false

------ Round 1 ----------------------------------
q1	17620	4368	4296	4296
q2	2019	195	183	183
q3	11990	936	1135	936
q4	10510	808	691	691
q5	7760	2878	2832	2832
q6	230	138	137	137
q7	957	621	611	611
q8	9317	2090	2083	2083
q9	7233	6536	6553	6536
q10	7006	2235	2226	2226
q11	458	250	246	246
q12	407	222	229	222
q13	18881	3084	3078	3078
q14	290	239	251	239
q15	534	490	488	488
q16	520	443	414	414
q17	975	746	653	653
q18	7369	6864	6841	6841
q19	1393	1026	1065	1026
q20	677	342	334	334
q21	3954	2935	3015	2935
q22	1135	1012	1025	1012
Total cold run time: 111235 ms
Total hot run time: 38019 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4309	4280	4232	4232
q2	384	279	274	274
q3	2920	2673	2669	2669
q4	1975	1608	1627	1608
q5	5698	5705	5708	5705
q6	228	136	137	136
q7	2257	1872	1809	1809
q8	3295	3416	3460	3416
q9	8923	8794	8802	8794
q10	3528	3442	3409	3409
q11	607	512	509	509
q12	835	671	647	647
q13	14539	3175	3287	3175
q14	332	289	296	289
q15	546	493	491	491
q16	527	492	509	492
q17	1817	1552	1542	1542
q18	8155	7899	7782	7782
q19	1767	1663	1752	1663
q20	2175	1905	1917	1905
q21	5832	5710	5604	5604
q22	1133	1026	1050	1026
Total cold run time: 71782 ms
Total hot run time: 57177 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192430 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 332273fbb0fc4ef557b73d24828545d17c810c20, data reload: false

query1	1272	903	864	864
query2	6358	1955	1895	1895
query3	10732	4371	4098	4098
query4	59588	24465	23262	23262
query5	5430	497	510	497
query6	419	158	153	153
query7	5758	295	292	292
query8	331	220	224	220
query9	8944	2520	2496	2496
query10	493	287	258	258
query11	18018	15027	15488	15027
query12	164	101	108	101
query13	1576	403	408	403
query14	11205	7013	7444	7013
query15	255	169	180	169
query16	7587	496	519	496
query17	1171	575	574	574
query18	1938	287	294	287
query19	300	148	146	146
query20	132	110	114	110
query21	212	106	104	104
query22	4612	4445	4706	4445
query23	34353	33463	33365	33365
query24	5966	2850	2774	2774
query25	528	376	387	376
query26	681	156	156	156
query27	1793	272	275	272
query28	3850	2023	2011	2011
query29	691	397	416	397
query30	234	155	151	151
query31	933	776	807	776
query32	88	51	53	51
query33	445	290	274	274
query34	862	480	468	468
query35	811	721	724	721
query36	1070	930	947	930
query37	147	81	85	81
query38	4043	3844	3872	3844
query39	1441	1418	1386	1386
query40	196	114	107	107
query41	47	43	45	43
query42	113	92	93	92
query43	531	473	477	473
query44	1060	761	731	731
query45	196	164	165	164
query46	1087	716	720	716
query47	1868	1784	1785	1784
query48	359	284	285	284
query49	764	432	442	432
query50	817	405	417	405
query51	7046	6864	6842	6842
query52	101	86	85	85
query53	247	175	177	175
query54	561	454	489	454
query55	73	73	74	73
query56	274	260	248	248
query57	1165	1088	1094	1088
query58	216	229	227	227
query59	3098	2888	2907	2888
query60	298	270	256	256
query61	101	99	105	99
query62	750	637	671	637
query63	216	182	183	182
query64	2839	736	673	673
query65	3207	3116	3193	3116
query66	683	343	333	333
query67	15408	15278	15309	15278
query68	2971	583	561	561
query69	389	274	270	270
query70	1207	1139	1142	1139
query71	345	273	275	273
query72	4975	4050	3995	3995
query73	743	320	322	320
query74	9265	8910	8790	8790
query75	3356	2715	2696	2696
query76	1462	932	967	932
query77	525	320	318	318
query78	9647	9126	9020	9020
query79	1005	532	523	523
query80	677	543	501	501
query81	464	228	230	228
query82	232	140	140	140
query83	186	150	152	150
query84	257	80	84	80
query85	696	292	287	287
query86	292	287	286	286
query87	4362	4328	4304	4304
query88	2920	2292	2285	2285
query89	385	292	286	286
query90	1903	189	190	189
query91	127	97	100	97
query92	58	47	49	47
query93	1020	530	525	525
query94	702	307	313	307
query95	311	248	256	248
query96	582	263	265	263
query97	3203	3068	3078	3068
query98	272	197	200	197
query99	1501	1277	1284	1277
Total cold run time: 303792 ms
Total hot run time: 192430 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 332273fbb0fc4ef557b73d24828545d17c810c20, data reload: false

query1	0.04	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.67	0.08	0.07
query5	0.50	0.49	0.49
query6	1.14	0.74	0.73
query7	0.01	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.50	0.50
query10	0.54	0.58	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.58	0.59
query14	1.41	1.45	1.43
query15	0.84	0.82	0.82
query16	0.37	0.37	0.38
query17	1.07	1.03	1.04
query18	0.21	0.21	0.20
query19	1.84	1.70	1.81
query20	0.01	0.02	0.01
query21	15.38	0.67	0.67
query22	4.12	6.45	1.74
query23	18.32	1.38	1.27
query24	2.14	0.22	0.22
query25	0.14	0.08	0.07
query26	0.26	0.18	0.19
query27	0.08	0.08	0.07
query28	13.18	1.02	0.99
query29	12.58	3.32	3.26
query30	0.24	0.05	0.05
query31	2.88	0.40	0.38
query32	3.26	0.48	0.48
query33	3.00	2.97	3.03
query34	17.08	4.44	4.42
query35	4.54	4.44	4.49
query36	0.66	0.49	0.46
query37	0.19	0.16	0.15
query38	0.16	0.14	0.15
query39	0.06	0.04	0.04
query40	0.15	0.13	0.14
query41	0.09	0.06	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.07 s
Total hot run time: 31.14 s

airborne12
airborne12 approved these changes Sep 12, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

lxr599
lxr599 approved these changes Sep 12, 2024
View reviewed changes
Copy link
Contributor

@lxr599 lxr599 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit 7fdd0be into apache:master Sep 12, 2024
26 of 29 checks passed
@eldenmoon eldenmoon deleted the opt-pq-1 branch September 12, 2024 07:05
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Sep 18, 2024
@eldenmoon
[optimize](short circuit) avoid set cacheId when non prepared execute (…
7d1b99e 
…apache#40525)

Only set cacheID for prepared statement excute phase, otherwise leading
to many redundant cost in BE side
@eldenmoon eldenmoon mentioned this pull request Sep 18, 2024
Merged
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Sep 18, 2024
@eldenmoon
[optimize](short circuit) avoid set cacheId when non prepared execute (…
3dc79a5 
…apache#40525)

Only set cacheID for prepared statement excute phase, otherwise leading
to many redundant cost in BE side
@eldenmoon eldenmoon mentioned this pull request Sep 18, 2024
Merged
eldenmoon added a commit that referenced this pull request Sep 18, 2024
@eldenmoon
[pick](short circuit) avoid set cacheId when non prepared execute (#4…
956b56c 
…0909)

#40525
eldenmoon added a commit that referenced this pull request Sep 20, 2024
@eldenmoon
[pick](short circuit) avoid set cacheId when non prepared execute (#4…
54dd4a3 
…0910)

#40525
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@airborne12 airborne12 airborne12 approved these changes

@lxr599 lxr599 lxr599 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@eldenmoon @doris-robot @airborne12 @lxr599 @wm1581066