Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data #40562

Merged
merged 10 commits into from
Sep 19, 2024
Merged

[fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data #40562

merged 10 commits into from
Sep 19, 2024

Conversation

hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Sep 9, 2024

Add retry when calculating delete bitmap timeout on broker load , like stream load doing.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Sep 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38123 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 86efeaf20e73bb10d90f9e57b39c71eb6616cc83, data reload: false

------ Round 1 ----------------------------------
q1	17615	4411	4310	4310
q2	2016	195	187	187
q3	11727	1008	1115	1008
q4	10510	698	721	698
q5	7718	2830	2816	2816
q6	225	140	137	137
q7	945	622	604	604
q8	9338	2065	2056	2056
q9	7125	6492	6499	6492
q10	6987	2200	2237	2200
q11	467	248	252	248
q12	391	232	234	232
q13	17766	3071	3083	3071
q14	286	241	247	241
q15	550	489	505	489
q16	538	447	431	431
q17	987	666	722	666
q18	7259	6827	6905	6827
q19	1397	1142	1106	1106
q20	677	331	333	331
q21	4192	3121	2972	2972
q22	1110	1001	1013	1001
Total cold run time: 109826 ms
Total hot run time: 38123 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4585	4302	4335	4302
q2	386	279	275	275
q3	2852	2637	2656	2637
q4	1978	1784	1705	1705
q5	5660	5688	5748	5688
q6	223	142	142	142
q7	2228	1861	1840	1840
q8	3302	3414	3463	3414
q9	8856	8892	8823	8823
q10	3572	3382	3357	3357
q11	625	505	534	505
q12	840	656	674	656
q13	13846	3272	3304	3272
q14	321	279	293	279
q15	557	506	496	496
q16	562	495	508	495
q17	1836	1522	1549	1522
q18	8125	7875	7901	7875
q19	1761	1644	1610	1610
q20	2153	1894	1906	1894
q21	5790	5529	5564	5529
q22	1129	1050	1054	1050
Total cold run time: 71187 ms
Total hot run time: 57366 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192142 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86efeaf20e73bb10d90f9e57b39c71eb6616cc83, data reload: false

query1	1255	880	876	876
query2	6317	1947	1898	1898
query3	10600	3982	3806	3806
query4	59274	26722	23192	23192
query5	5356	531	540	531
query6	412	166	164	164
query7	5821	289	294	289
query8	324	226	226	226
query9	8878	2498	2503	2498
query10	491	295	274	274
query11	17999	15045	15548	15045
query12	158	109	113	109
query13	1570	427	400	400
query14	11376	6846	7248	6846
query15	248	179	173	173
query16	7590	452	459	452
query17	1130	574	581	574
query18	2078	317	310	310
query19	298	153	149	149
query20	117	123	120	120
query21	208	109	104	104
query22	4570	4378	4464	4378
query23	34513	33558	33533	33533
query24	5956	2836	2840	2836
query25	523	389	397	389
query26	685	154	155	154
query27	1785	274	276	274
query28	3738	2024	1999	1999
query29	661	413	405	405
query30	269	150	152	150
query31	914	759	788	759
query32	81	56	55	55
query33	500	300	292	292
query34	890	496	466	466
query35	853	720	717	717
query36	1076	935	950	935
query37	144	79	88	79
query38	3982	3790	3914	3790
query39	1440	1420	1375	1375
query40	198	118	114	114
query41	48	44	45	44
query42	119	100	96	96
query43	530	486	484	484
query44	1122	736	743	736
query45	198	174	171	171
query46	1087	752	765	752
query47	1875	1756	1809	1756
query48	380	296	310	296
query49	785	474	445	445
query50	827	413	436	413
query51	6963	6966	6885	6885
query52	99	87	85	85
query53	257	185	183	183
query54	563	447	458	447
query55	76	73	72	72
query56	268	253	266	253
query57	1193	1029	1087	1029
query58	216	224	264	224
query59	3234	3120	2899	2899
query60	290	270	276	270
query61	99	97	104	97
query62	784	665	646	646
query63	228	192	189	189
query64	2802	680	688	680
query65	3216	3125	3149	3125
query66	676	334	330	330
query67	15451	15274	15159	15159
query68	2960	572	587	572
query69	385	287	284	284
query70	1202	1131	1118	1118
query71	357	295	281	281
query72	5280	3976	4052	3976
query73	773	318	318	318
query74	9115	8872	8782	8782
query75	3333	2723	2691	2691
query76	1492	1004	978	978
query77	557	343	334	334
query78	9812	9058	9058	9058
query79	1041	536	546	536
query80	693	515	510	510
query81	474	232	232	232
query82	240	143	139	139
query83	171	152	150	150
query84	247	76	76	76
query85	681	299	290	290
query86	313	313	291	291
query87	4428	4370	4241	4241
query88	2923	2279	2274	2274
query89	379	293	290	290
query90	1980	195	190	190
query91	125	100	160	100
query92	62	48	49	48
query93	1071	538	537	537
query94	738	292	297	292
query95	322	251	257	251
query96	585	261	262	261
query97	3184	3054	3095	3054
query98	223	210	206	206
query99	1511	1311	1280	1280
Total cold run time: 304321 ms
Total hot run time: 192142 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86efeaf20e73bb10d90f9e57b39c71eb6616cc83, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.09	0.08
query5	0.49	0.50	0.48
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.05	0.05
query9	0.53	0.47	0.48
query10	0.53	0.56	0.53
query11	0.15	0.12	0.12
query12	0.16	0.12	0.11
query13	0.60	0.59	0.59
query14	1.39	1.40	1.41
query15	0.83	0.83	0.83
query16	0.39	0.37	0.36
query17	1.06	1.02	1.01
query18	0.22	0.20	0.21
query19	1.82	1.75	1.82
query20	0.02	0.01	0.01
query21	15.41	0.67	0.67
query22	4.51	7.07	1.79
query23	18.25	1.48	1.26
query24	2.05	0.23	0.22
query25	0.15	0.08	0.08
query26	0.28	0.19	0.17
query27	0.07	0.08	0.07
query28	13.28	1.02	1.00
query29	12.64	3.37	3.30
query30	0.24	0.05	0.05
query31	2.86	0.42	0.40
query32	3.24	0.48	0.49
query33	2.98	3.01	2.99
query34	17.08	4.40	4.40
query35	4.40	4.46	4.42
query36	0.66	0.48	0.48
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.15	0.13	0.13
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.23 s
Total hot run time: 31.24 s

@hust-hhb hust-hhb changed the title [fix](cloud-mow) Add retry when calculating delete bitmap timeout on broker load [fix](cloud-mow) Add retry when calculating delete bitmap timeout when loading data Sep 11, 2024
@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38292 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2fa6753d8d043c75d7366fcd4b4a31f16edb700, data reload: false

------ Round 1 ----------------------------------
q1	17602	4396	4373	4373
q2	2017	185	188	185
q3	11699	977	1142	977
q4	10512	746	811	746
q5	7719	2868	2859	2859
q6	229	139	138	138
q7	949	643	606	606
q8	9297	2074	2102	2074
q9	7270	6581	6541	6541
q10	6989	2245	2228	2228
q11	468	248	250	248
q12	400	229	233	229
q13	17756	3130	3058	3058
q14	275	240	237	237
q15	527	481	485	481
q16	523	430	428	428
q17	981	664	689	664
q18	7356	6975	6821	6821
q19	1383	1060	1094	1060
q20	710	330	340	330
q21	4045	3067	2970	2970
q22	1134	1048	1039	1039
Total cold run time: 109841 ms
Total hot run time: 38292 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4418	4317	4300	4300
q2	390	283	280	280
q3	2878	2677	2654	2654
q4	1967	1613	1631	1613
q5	5668	5700	5742	5700
q6	232	143	140	140
q7	2260	1882	1792	1792
q8	3320	3430	3458	3430
q9	8901	8927	8890	8890
q10	3609	3431	3397	3397
q11	602	533	512	512
q12	827	661	648	648
q13	13944	3223	3323	3223
q14	325	286	287	286
q15	548	486	481	481
q16	536	487	504	487
q17	1847	1573	1579	1573
q18	8077	7742	7778	7742
q19	1749	1600	1508	1508
q20	2166	1901	1957	1901
q21	5735	5514	5526	5514
q22	1165	1068	1034	1034
Total cold run time: 71164 ms
Total hot run time: 57105 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197246 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2fa6753d8d043c75d7366fcd4b4a31f16edb700, data reload: false

query1	1267	884	928	884
query2	6396	1957	1891	1891
query3	10604	4179	4011	4011
query4	60095	25065	23209	23209
query5	5085	516	524	516
query6	424	172	166	166
query7	5617	306	297	297
query8	308	220	220	220
query9	7751	2535	2525	2525
query10	437	311	263	263
query11	17302	15308	15442	15308
query12	159	107	102	102
query13	1456	409	391	391
query14	10794	7645	7370	7370
query15	213	185	184	184
query16	6698	483	484	483
query17	1088	583	563	563
query18	1448	295	291	291
query19	204	145	150	145
query20	120	112	107	107
query21	220	107	107	107
query22	4742	4361	4414	4361
query23	34439	33839	33518	33518
query24	5995	2928	2996	2928
query25	511	412	422	412
query26	601	163	156	156
query27	1567	280	285	280
query28	3745	2053	2035	2035
query29	672	440	434	434
query30	239	160	154	154
query31	976	757	740	740
query32	74	53	53	53
query33	414	291	288	288
query34	869	490	474	474
query35	835	725	702	702
query36	1070	945	943	943
query37	146	87	86	86
query38	4051	3923	3921	3921
query39	1422	1390	1404	1390
query40	191	114	116	114
query41	48	47	45	45
query42	120	96	93	93
query43	517	461	462	461
query44	1111	764	747	747
query45	195	165	166	165
query46	1087	784	771	771
query47	1906	1796	1795	1795
query48	373	306	299	299
query49	771	436	467	436
query50	835	430	431	430
query51	6910	6967	6864	6864
query52	99	88	88	88
query53	260	178	181	178
query54	571	471	468	468
query55	76	76	75	75
query56	271	263	260	260
query57	1205	1082	1044	1044
query58	220	270	232	232
query59	3030	2748	2723	2723
query60	333	274	280	274
query61	103	97	102	97
query62	767	662	653	653
query63	218	189	187	187
query64	1350	687	691	687
query65	3213	3131	3214	3131
query66	654	337	341	337
query67	15661	15403	15300	15300
query68	1686	848	842	842
query69	419	330	383	330
query70	1223	1100	1173	1100
query71	350	356	344	344
query72	4597	3596	2468	2468
query73	598	587	585	585
query74	9018	8997	8954	8954
query75	3080	2998	3019	2998
query76	952	885	867	867
query77	449	425	417	417
query78	9302	9389	9217	9217
query79	904	886	868	868
query80	822	845	822	822
query81	455	258	262	258
query82	264	266	266	266
query83	196	192	193	192
query84	201	108	109	108
query85	587	410	397	397
query86	304	315	303	303
query87	4293	4314	4273	4273
query88	4333	4133	4144	4133
query89	374	369	361	361
query90	796	321	317	317
query91	127	127	132	127
query92	79	79	82	79
query93	924	919	916	916
query94	412	362	380	362
query95	438	427	426	426
query96	474	474	474	474
query97	3102	3080	3104	3080
query98	246	237	233	233
query99	1316	1273	1267	1267
Total cold run time: 294470 ms
Total hot run time: 197246 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2fa6753d8d043c75d7366fcd4b4a31f16edb700, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.67	0.10	0.09
query5	0.50	0.48	0.49
query6	1.14	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.50
query10	0.54	0.57	0.53
query11	0.17	0.12	0.12
query12	0.16	0.13	0.12
query13	0.61	0.60	0.59
query14	1.41	1.41	1.40
query15	0.84	0.82	0.82
query16	0.35	0.38	0.37
query17	1.04	1.06	1.00
query18	0.21	0.20	0.20
query19	1.94	1.78	1.85
query20	0.01	0.01	0.02
query21	15.66	0.67	0.66
query22	4.55	6.53	2.24
query23	18.24	1.32	1.41
query24	2.07	0.23	0.23
query25	0.14	0.09	0.08
query26	0.26	0.17	0.17
query27	0.07	0.07	0.08
query28	13.25	1.02	1.00
query29	12.65	3.38	3.32
query30	0.25	0.06	0.07
query31	2.84	0.42	0.40
query32	3.24	0.47	0.50
query33	2.97	2.98	3.08
query34	17.12	4.39	4.40
query35	4.51	4.49	4.46
query36	0.65	0.50	0.46
query37	0.19	0.17	0.16
query38	0.17	0.14	0.14
query39	0.05	0.04	0.04
query40	0.16	0.13	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.8 s
Total hot run time: 31.82 s

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38366 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3ccbe272044546a4d5d606a72ca07f4196c3003, data reload: false

------ Round 1 ----------------------------------
q1	17667	4462	4381	4381
q2	2012	190	199	190
q3	10441	1154	1190	1154
q4	10128	796	645	645
q5	7746	2888	2815	2815
q6	231	140	140	140
q7	972	618	614	614
q8	9322	2087	2084	2084
q9	7259	6567	6556	6556
q10	7060	2206	2123	2123
q11	473	240	243	240
q12	397	223	230	223
q13	18826	3111	3140	3111
q14	286	234	242	234
q15	539	487	480	480
q16	540	434	436	434
q17	991	729	733	729
q18	7476	6874	6990	6874
q19	1412	1133	1070	1070
q20	701	335	333	333
q21	4575	3008	2935	2935
q22	1101	1001	1056	1001
Total cold run time: 110155 ms
Total hot run time: 38366 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4487	4333	4349	4333
q2	420	286	279	279
q3	2868	2676	2666	2666
q4	1910	1671	1698	1671
q5	5467	5391	5463	5391
q6	223	133	132	132
q7	2091	1719	1729	1719
q8	3219	3374	3390	3374
q9	8464	8475	8462	8462
q10	3475	3204	3232	3204
q11	593	527	508	508
q12	782	616	606	606
q13	15890	3159	3109	3109
q14	300	272	272	272
q15	525	478	493	478
q16	516	487	476	476
q17	1814	1512	1474	1474
q18	7707	7490	7568	7490
q19	1693	1515	1546	1515
q20	2088	1816	1829	1816
q21	5605	5406	5308	5308
q22	1119	1046	1043	1043
Total cold run time: 71256 ms
Total hot run time: 55326 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192739 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3ccbe272044546a4d5d606a72ca07f4196c3003, data reload: false

query1	914	379	366	366
query2	6453	1959	1920	1920
query3	6642	207	214	207
query4	30391	23110	23463	23110
query5	4188	524	510	510
query6	252	161	157	157
query7	4570	302	290	290
query8	270	221	220	220
query9	8660	2507	2512	2507
query10	456	299	287	287
query11	17945	15254	15231	15231
query12	164	97	96	96
query13	1630	380	356	356
query14	9976	7007	6976	6976
query15	293	168	174	168
query16	8027	438	437	437
query17	1565	561	554	554
query18	2123	283	281	281
query19	230	150	138	138
query20	121	111	110	110
query21	210	103	100	100
query22	4586	4065	4047	4047
query23	34347	33612	33846	33612
query24	10904	2940	2876	2876
query25	655	376	379	376
query26	1164	153	159	153
query27	2394	270	275	270
query28	7340	2039	2014	2014
query29	808	404	405	404
query30	308	162	151	151
query31	1025	777	797	777
query32	95	59	57	57
query33	749	302	290	290
query34	953	462	488	462
query35	885	737	722	722
query36	1135	919	963	919
query37	167	90	87	87
query38	4082	3869	3820	3820
query39	1444	1448	1399	1399
query40	203	120	113	113
query41	48	50	45	45
query42	114	97	96	96
query43	533	475	481	475
query44	1314	800	741	741
query45	196	164	167	164
query46	1108	730	752	730
query47	1877	1790	1825	1790
query48	365	293	292	292
query49	1077	434	481	434
query50	811	410	421	410
query51	7005	6910	6892	6892
query52	99	88	84	84
query53	253	181	182	181
query54	830	444	455	444
query55	75	76	78	76
query56	282	260	252	252
query57	1228	1096	1064	1064
query58	257	237	235	235
query59	3032	2971	2857	2857
query60	300	272	266	266
query61	104	100	103	100
query62	834	662	647	647
query63	234	201	197	197
query64	4184	664	637	637
query65	3281	3157	3164	3157
query66	1088	347	341	341
query67	16101	15307	15521	15307
query68	3124	845	844	844
query69	432	325	325	325
query70	1199	1181	1175	1175
query71	351	340	347	340
query72	6172	3493	3463	3463
query73	584	595	589	589
query74	9006	8921	8877	8877
query75	3151	3328	2953	2953
query76	1812	871	878	871
query77	504	424	411	411
query78	9436	9254	9638	9254
query79	907	895	865	865
query80	899	820	797	797
query81	468	264	261	261
query82	269	268	262	262
query83	195	194	194	194
query84	239	109	107	107
query85	637	403	380	380
query86	348	322	315	315
query87	4333	4355	4314	4314
query88	4261	4027	4010	4010
query89	384	375	378	375
query90	1310	317	319	317
query91	126	124	127	124
query92	77	75	77	75
query93	960	924	900	900
query94	458	399	387	387
query95	425	410	412	410
query96	467	464	464	464
query97	3098	3108	3124	3108
query98	229	228	220	220
query99	1415	1273	1268	1268
Total cold run time: 282230 ms
Total hot run time: 192739 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.88% (9452/25626)
Line Coverage: 28.24% (77734/275287)
Region Coverage: 27.63% (40119/145186)
Branch Coverage: 24.25% (20386/84076)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d3ccbe272044546a4d5d606a72ca07f4196c3003_d3ccbe272044546a4d5d606a72ca07f4196c3003/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 31.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d3ccbe272044546a4d5d606a72ca07f4196c3003, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.09	0.08
query5	0.50	0.49	0.49
query6	1.12	0.75	0.73
query7	0.02	0.02	0.02
query8	0.05	0.05	0.04
query9	0.54	0.49	0.48
query10	0.56	0.57	0.54
query11	0.15	0.12	0.12
query12	0.15	0.13	0.12
query13	0.60	0.59	0.60
query14	1.38	1.41	1.39
query15	0.85	0.82	0.82
query16	0.38	0.36	0.38
query17	1.04	0.98	1.03
query18	0.17	0.17	0.19
query19	1.86	1.91	1.74
query20	0.02	0.01	0.01
query21	15.39	0.66	0.65
query22	4.33	6.66	1.87
query23	18.28	1.45	1.34
query24	2.12	0.24	0.22
query25	0.16	0.08	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.21	1.03	1.01
query29	12.58	3.40	3.43
query30	0.24	0.05	0.05
query31	2.88	0.41	0.38
query32	3.27	0.49	0.48
query33	2.98	3.01	3.03
query34	16.88	4.42	4.37
query35	4.46	4.45	4.41
query36	0.65	0.47	0.51
query37	0.18	0.15	0.16
query38	0.15	0.16	0.14
query39	0.05	0.04	0.03
query40	0.16	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.94 s
Total hot run time: 31.38 s

dataroaring
dataroaring reviewed Sep 14, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/BrokerLoadJob.java Outdated
dbId, tableList, transactionId, commitInfos, getLoadJobFinalOperation());
afterLoadingTaskCommitTransaction(tableList);
afterCommit();
finishJob = true;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just break instead of using finishJob.

Suggested change
finishJob = true;
finishJob = true;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use return instead of break, unlock table will do in finally code before return

@hust-hhb
Copy link
Contributor Author

run buildall

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 43493 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2261e978c0024b2d9de5e720d0847a2d39e10556, data reload: false

------ Round 1 ----------------------------------
q1	17581	7388	7237	7237
q2	2050	198	187	187
q3	10521	1418	1430	1418
q4	10317	993	1071	993
q5	7744	3237	3184	3184
q6	245	155	161	155
q7	1056	645	636	636
q8	9475	2046	2086	2046
q9	6825	6365	6369	6365
q10	7043	2596	2564	2564
q11	439	256	263	256
q12	405	234	236	234
q13	17780	3029	3027	3027
q14	292	250	264	250
q15	593	545	524	524
q16	526	436	429	429
q17	1023	991	975	975
q18	7446	6830	7049	6830
q19	1381	1275	1258	1258
q20	606	350	332	332
q21	3961	3638	3598	3598
q22	1105	1040	995	995
Total cold run time: 108414 ms
Total hot run time: 43493 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7231	7190	7214	7190
q2	356	243	230	230
q3	2905	2954	2970	2954
q4	1967	1990	1993	1990
q5	5493	5593	5454	5454
q6	238	148	146	146
q7	2047	1717	1656	1656
q8	3272	3359	3353	3353
q9	8513	8447	8548	8447
q10	3437	3503	3492	3492
q11	581	459	468	459
q12	798	602	586	586
q13	8635	3077	3075	3075
q14	294	268	286	268
q15	586	522	524	522
q16	484	458	460	458
q17	1771	1737	1748	1737
q18	7996	7616	7504	7504
q19	1741	1738	1725	1725
q20	2054	1807	1810	1807
q21	5821	5441	5355	5355
q22	1111	1003	973	973
Total cold run time: 67331 ms
Total hot run time: 59381 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195605 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2261e978c0024b2d9de5e720d0847a2d39e10556, data reload: false

query1	933	384	394	384
query2	6465	1799	1765	1765
query3	6659	214	221	214
query4	26100	24172	24164	24164
query5	5560	560	534	534
query6	269	170	163	163
query7	4582	315	324	315
query8	286	224	238	224
query9	8484	2610	2617	2610
query10	488	287	274	274
query11	15936	15432	15523	15432
query12	156	102	107	102
query13	1702	408	388	388
query14	11410	6833	7187	6833
query15	214	178	176	176
query16	7617	509	527	509
query17	1545	603	575	575
query18	2071	300	296	296
query19	195	149	152	149
query20	125	114	117	114
query21	217	109	108	108
query22	4550	4336	4295	4295
query23	34608	34013	33695	33695
query24	9585	3094	3144	3094
query25	678	418	418	418
query26	1171	163	163	163
query27	2355	302	292	292
query28	6557	2108	2082	2082
query29	952	436	434	434
query30	293	164	159	159
query31	995	787	806	787
query32	107	59	65	59
query33	749	309	307	307
query34	923	483	486	483
query35	904	756	749	749
query36	1082	910	944	910
query37	169	86	83	83
query38	4103	3968	3906	3906
query39	1453	1420	1431	1420
query40	219	123	122	122
query41	51	51	50	50
query42	125	97	99	97
query43	497	451	441	441
query44	1243	811	781	781
query45	202	178	174	174
query46	1115	847	809	809
query47	1911	1775	1838	1775
query48	384	301	297	297
query49	1109	487	460	460
query50	911	453	442	442
query51	6955	6912	6916	6912
query52	109	94	92	92
query53	264	187	190	187
query54	835	477	484	477
query55	84	88	81	81
query56	298	288	292	288
query57	1229	1117	1079	1079
query58	271	245	255	245
query59	2967	2659	2472	2472
query60	313	285	296	285
query61	125	238	103	103
query62	910	679	672	672
query63	224	187	211	187
query64	4600	707	683	683
query65	3298	3160	3147	3147
query66	1443	304	306	304
query67	16182	15637	15427	15427
query68	3333	873	864	864
query69	439	327	328	327
query70	1166	1144	1164	1144
query71	347	344	341	341
query72	6139	3440	3365	3365
query73	589	584	580	580
query74	9429	9179	9207	9179
query75	3168	3065	2985	2985
query76	1926	843	845	843
query77	431	405	397	397
query78	9439	9331	9291	9291
query79	917	890	871	871
query80	914	830	810	810
query81	456	269	274	269
query82	267	264	260	260
query83	193	189	190	189
query84	243	109	105	105
query85	649	423	404	404
query86	320	311	330	311
query87	4378	4515	4385	4385
query88	4185	4096	4100	4096
query89	385	377	368	368
query90	1262	323	317	317
query91	125	127	125	125
query92	79	76	75	75
query93	1053	1079	1047	1047
query94	610	403	362	362
query95	443	423	422	422
query96	475	474	476	474
query97	3137	3131	3159	3131
query98	230	230	225	225
query99	1553	1294	1289	1289
Total cold run time: 278402 ms
Total hot run time: 195605 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2261e978c0024b2d9de5e720d0847a2d39e10556, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.08
query5	0.52	0.51	0.49
query6	1.13	0.72	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.56	0.54	0.52
query10	0.56	0.58	0.58
query11	0.16	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.61	0.61
query14	1.46	1.47	1.48
query15	0.90	0.88	0.88
query16	0.36	0.35	0.37
query17	1.06	1.03	1.02
query18	0.16	0.16	0.17
query19	1.94	1.77	1.81
query20	0.01	0.01	0.01
query21	15.39	0.66	0.65
query22	4.30	8.23	1.48
query23	18.22	1.46	1.49
query24	2.26	0.23	0.22
query25	0.19	0.09	0.08
query26	0.30	0.18	0.19
query27	0.09	0.08	0.08
query28	13.18	1.14	1.10
query29	12.63	3.35	3.35
query30	0.24	0.06	0.06
query31	2.86	0.42	0.41
query32	3.23	0.50	0.49
query33	3.08	3.06	3.07
query34	15.45	4.34	4.36
query35	4.42	4.34	4.34
query36	0.68	0.49	0.49
query37	0.21	0.18	0.16
query38	0.17	0.15	0.15
query39	0.05	0.04	0.04
query40	0.18	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.05	0.05
Total cold run time: 109.05 s
Total hot run time: 31.53 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.91% (9473/25663)
Line Coverage: 28.28% (77907/275491)
Region Coverage: 27.68% (40231/145323)
Branch Coverage: 24.29% (20442/84154)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2261e978c0024b2d9de5e720d0847a2d39e10556_2261e978c0024b2d9de5e720d0847a2d39e10556/report/index.html

dataroaring
dataroaring previously approved these changes Sep 14, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 14, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 19, 2024
@hust-hhb
Copy link
Contributor Author

run buildall

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41265 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0b1426545ef083873de900caf90b98e2ee785ac3, data reload: false

------ Round 1 ----------------------------------
q1	17577	8116	7298	7298
q2	2052	162	160	160
q3	10583	1055	1154	1055
q4	10538	737	743	737
q5	7760	3069	3089	3069
q6	237	155	152	152
q7	1015	618	607	607
q8	9451	2053	2053	2053
q9	6816	6402	6449	6402
q10	7032	2282	2308	2282
q11	445	263	254	254
q12	405	215	225	215
q13	17784	3000	2982	2982
q14	242	218	226	218
q15	575	537	527	527
q16	702	615	618	615
q17	983	772	789	772
q18	7230	6583	6624	6583
q19	1398	929	1064	929
q20	569	296	287	287
q21	4079	3275	3069	3069
q22	1111	1009	999	999
Total cold run time: 108584 ms
Total hot run time: 41265 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7276	7298	7312	7298
q2	329	236	230	230
q3	3020	2954	2966	2954
q4	2032	1849	1843	1843
q5	5569	5565	5622	5565
q6	244	155	151	151
q7	2234	1834	1819	1819
q8	3328	3448	3445	3445
q9	8735	9020	8778	8778
q10	3504	3457	3507	3457
q11	591	477	490	477
q12	816	639	629	629
q13	9876	3182	3149	3149
q14	293	279	284	279
q15	559	522	552	522
q16	739	683	671	671
q17	1819	1632	1573	1573
q18	8198	7798	7809	7798
q19	1729	1542	1584	1542
q20	2184	1887	1885	1885
q21	5726	5321	5466	5321
q22	1145	1088	1074	1074
Total cold run time: 69946 ms
Total hot run time: 60460 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.33% (9585/25676)
Line Coverage: 28.71% (79212/275885)
Region Coverage: 28.19% (41029/145522)
Branch Coverage: 24.82% (20902/84222)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0b1426545ef083873de900caf90b98e2ee785ac3_0b1426545ef083873de900caf90b98e2ee785ac3/report/index.html

zhannngchen
zhannngchen approved these changes Sep 19, 2024
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024
dataroaring
dataroaring approved these changes Sep 19, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 53fdd9e into apache:master Sep 19, 2024
24 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Sep 19, 2024
@hust-hhb @dataroaring
[fix](cloud-mow) Add retry when calculating delete bitmap timeout whe…
b2c742f 
…n loading data (#40562)

Add retry when calculating delete bitmap timeout on broker load , like
stream load doing.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhannngchen zhannngchen zhannngchen approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hust-hhb @doris-robot @zhannngchen @dataroaring