[enhance](paimon) opt count pushdown for paimon and refactor be logic #46911

suxiaogang223 · 2025-01-13T10:53:42Z

What problem does this PR solve?

Related PR: #44038

Obtain directly from statistics for select count(*) from paimon_table
Refactor TableFormatReader, move same logic of IcebergTableReader, PaimonTableReader and TransactionalHiveReader to TableFormatReader

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-01-13T10:53:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 · 2025-02-21T07:17:34Z

run buildall

doris-robot · 2025-02-21T07:31:27Z

TeamCity cloud ut coverage result:
Function Coverage: 82.25% (1061/1290)
Line Coverage: 65.79% (17595/26746)
Region Coverage: 65.33% (8670/13271)
Branch Coverage: 55.24% (4674/8462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ef33aa70f28e0294d2512e30c7eac3eabec8e8f0_ef33aa70f28e0294d2512e30c7eac3eabec8e8f0_cloud/report/index.html

doris-robot · 2025-02-21T08:44:30Z

TPC-H: Total hot run time: 31662 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef33aa70f28e0294d2512e30c7eac3eabec8e8f0, data reload: false

------ Round 1 ----------------------------------
q1	17624	5347	5167	5167
q2	2053	300	189	189
q3	10387	1312	734	734
q4	10219	1018	537	537
q5	7552	2963	2325	2325
q6	206	170	136	136
q7	947	772	642	642
q8	9321	1363	1208	1208
q9	4947	4529	4534	4529
q10	6852	2332	1873	1873
q11	487	281	253	253
q12	349	360	221	221
q13	17776	3699	3183	3183
q14	246	226	215	215
q15	534	479	458	458
q16	623	627	580	580
q17	638	912	344	344
q18	7280	6274	6029	6029
q19	1543	972	583	583
q20	321	325	194	194
q21	2850	2118	1947	1947
q22	361	335	315	315
Total cold run time: 103116 ms
Total hot run time: 31662 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5236	5203	5231	5203
q2	244	336	247	247
q3	2205	2686	2341	2341
q4	1410	1821	1334	1334
q5	4262	4157	4181	4157
q6	217	167	127	127
q7	1896	1866	1791	1791
q8	2648	2601	2555	2555
q9	7208	7087	7091	7087
q10	3024	3194	2755	2755
q11	581	515	507	507
q12	665	778	602	602
q13	3514	3886	3316	3316
q14	276	295	284	284
q15	496	461	472	461
q16	636	666	622	622
q17	1148	1620	1348	1348
q18	7525	7324	7372	7324
q19	797	839	1038	839
q20	1966	2002	1898	1898
q21	5545	5027	4870	4870
q22	602	592	550	550
Total cold run time: 52101 ms
Total hot run time: 50218 ms

doris-robot · 2025-02-21T08:55:57Z

TPC-DS: Total hot run time: 184542 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef33aa70f28e0294d2512e30c7eac3eabec8e8f0, data reload: false

query1	1031	381	366	366
query2	6517	1922	1881	1881
query3	6794	211	209	209
query4	26313	23800	23671	23671
query5	4326	683	531	531
query6	313	197	187	187
query7	4608	504	304	304
query8	304	245	237	237
query9	8620	2591	2592	2591
query10	486	304	251	251
query11	15653	15127	15218	15127
query12	156	115	110	110
query13	1664	545	390	390
query14	9741	6217	6624	6217
query15	212	207	179	179
query16	7673	646	459	459
query17	1201	736	588	588
query18	1992	413	310	310
query19	197	220	162	162
query20	123	122	117	117
query21	209	131	111	111
query22	4323	4374	4463	4374
query23	34725	33228	32929	32929
query24	7729	2405	2370	2370
query25	532	443	380	380
query26	801	282	157	157
query27	2123	511	321	321
query28	3870	2415	2409	2409
query29	667	536	425	425
query30	233	190	161	161
query31	912	828	764	764
query32	74	64	61	61
query33	549	394	301	301
query34	768	868	515	515
query35	787	824	743	743
query36	963	989	875	875
query37	119	99	100	99
query38	4353	4147	4164	4147
query39	1438	1571	1388	1388
query40	214	118	110	110
query41	57	51	49	49
query42	122	101	104	101
query43	521	513	472	472
query44	1342	800	774	774
query45	178	173	159	159
query46	879	1051	650	650
query47	1770	1771	1723	1723
query48	394	429	304	304
query49	739	487	413	413
query50	694	748	431	431
query51	4203	4196	4099	4099
query52	107	111	93	93
query53	231	260	198	198
query54	493	499	411	411
query55	86	84	78	78
query56	258	264	248	248
query57	1110	1161	1049	1049
query58	256	238	244	238
query59	2621	2946	2572	2572
query60	288	272	254	254
query61	121	119	119	119
query62	812	735	669	669
query63	241	191	197	191
query64	3554	990	652	652
query65	3264	3149	3153	3149
query66	916	409	308	308
query67	15789	15527	15253	15253
query68	8117	792	515	515
query69	485	295	281	281
query70	1232	1132	1092	1092
query71	464	305	281	281
query72	5765	3593	3683	3593
query73	736	766	356	356
query74	9282	9156	9015	9015
query75	3339	3173	2768	2768
query76	3214	1187	763	763
query77	708	414	296	296
query78	9955	10147	9262	9262
query79	2078	832	634	634
query80	675	545	478	478
query81	493	282	236	236
query82	652	127	96	96
query83	192	167	154	154
query84	288	92	77	77
query85	771	352	312	312
query86	348	321	284	284
query87	4407	4438	4332	4332
query88	2905	2261	2239	2239
query89	413	322	294	294
query90	1910	202	196	196
query91	138	138	109	109
query92	75	61	57	57
query93	1153	1002	590	590
query94	699	410	303	303
query95	354	273	256	256
query96	510	546	281	281
query97	2802	2840	2743	2743
query98	227	208	199	199
query99	1422	1394	1266	1266
Total cold run time: 270168 ms
Total hot run time: 184542 ms

doris-robot · 2025-02-21T09:01:18Z

ClickBench: Total hot run time: 30.93 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef33aa70f28e0294d2512e30c7eac3eabec8e8f0, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.43	0.41	0.40
query6	1.17	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.61	0.52	0.51
query10	0.57	0.59	0.57
query11	0.16	0.11	0.10
query12	0.14	0.11	0.12
query13	0.62	0.60	0.61
query14	2.67	2.69	2.69
query15	0.91	0.86	0.87
query16	0.38	0.38	0.38
query17	1.00	1.00	1.03
query18	0.22	0.19	0.20
query19	1.90	1.96	1.75
query20	0.01	0.01	0.02
query21	15.36	0.90	0.55
query22	0.77	1.12	0.89
query23	14.72	1.38	0.62
query24	7.65	0.96	1.06
query25	0.50	0.32	0.18
query26	0.60	0.16	0.16
query27	0.06	0.05	0.05
query28	9.46	0.86	0.44
query29	12.55	3.89	3.25
query30	0.25	0.09	0.07
query31	2.81	0.58	0.38
query32	3.22	0.54	0.46
query33	2.98	3.01	3.07
query34	15.68	5.15	4.48
query35	4.50	4.52	4.56
query36	0.66	0.53	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.1 s
Total hot run time: 30.93 s

doris-robot · 2025-02-21T09:36:46Z

TeamCity be ut coverage result:
Function Coverage: 44.34% (11701/26392)
Line Coverage: 33.92% (98005/288946)
Region Coverage: 33.00% (50183/152081)
Branch Coverage: 28.65% (25225/88042)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ef33aa70f28e0294d2512e30c7eac3eabec8e8f0_ef33aa70f28e0294d2512e30c7eac3eabec8e8f0/report/index.html

suxiaogang223 · 2025-02-21T14:30:02Z

run buildall

doris-robot · 2025-02-21T14:51:20Z

TeamCity cloud ut coverage result:
Function Coverage: 82.25% (1061/1290)
Line Coverage: 65.79% (17596/26746)
Region Coverage: 65.34% (8671/13271)
Branch Coverage: 55.24% (4674/8462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0fcde840e4f6632e1428501f1f8fbd37fe6878ba_0fcde840e4f6632e1428501f1f8fbd37fe6878ba_cloud/report/index.html

doris-robot · 2025-02-21T15:46:25Z

TPC-H: Total hot run time: 31565 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0fcde840e4f6632e1428501f1f8fbd37fe6878ba, data reload: false

------ Round 1 ----------------------------------
q1	17577	5403	5085	5085
q2	2043	294	167	167
q3	10421	1233	747	747
q4	10201	1006	516	516
q5	7499	2442	2299	2299
q6	190	169	131	131
q7	894	723	602	602
q8	9306	1333	1179	1179
q9	4853	4630	4679	4630
q10	6833	2300	1868	1868
q11	457	285	257	257
q12	347	356	218	218
q13	17765	3687	3173	3173
q14	245	228	210	210
q15	518	467	471	467
q16	626	618	590	590
q17	550	841	324	324
q18	6822	6137	6364	6137
q19	1215	952	534	534
q20	314	325	182	182
q21	2782	2145	1940	1940
q22	365	331	309	309
Total cold run time: 101823 ms
Total hot run time: 31565 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5103	5137	5129	5129
q2	233	332	240	240
q3	2149	2705	2262	2262
q4	1432	1825	1368	1368
q5	4256	4158	4151	4151
q6	208	164	126	126
q7	1871	1826	1653	1653
q8	2592	2630	2568	2568
q9	7198	7189	7132	7132
q10	3033	3176	2794	2794
q11	580	533	494	494
q12	681	763	606	606
q13	3535	3845	3333	3333
q14	279	301	274	274
q15	508	463	472	463
q16	645	690	629	629
q17	1137	1572	1336	1336
q18	7616	7361	7270	7270
q19	786	828	877	828
q20	1959	2018	1895	1895
q21	5407	5028	4836	4836
q22	607	584	548	548
Total cold run time: 51815 ms
Total hot run time: 49935 ms

doris-robot · 2025-02-21T15:58:27Z

TPC-DS: Total hot run time: 190729 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0fcde840e4f6632e1428501f1f8fbd37fe6878ba, data reload: false

query1	1304	958	920	920
query2	6243	1932	1867	1867
query3	10950	4392	4479	4392
query4	53918	24591	23370	23370
query5	5168	551	490	490
query6	367	183	181	181
query7	5123	494	288	288
query8	318	240	236	236
query9	6413	2579	2583	2579
query10	426	300	270	270
query11	15245	15185	14972	14972
query12	173	114	110	110
query13	1145	534	389	389
query14	10641	6697	6659	6659
query15	203	199	188	188
query16	7110	662	461	461
query17	1079	738	608	608
query18	1530	433	322	322
query19	225	204	185	185
query20	131	129	133	129
query21	213	122	111	111
query22	4573	4283	4410	4283
query23	33945	33352	33257	33257
query24	5757	2426	2455	2426
query25	502	464	391	391
query26	677	276	162	162
query27	1922	487	339	339
query28	2841	2498	2460	2460
query29	565	552	426	426
query30	206	195	156	156
query31	876	891	792	792
query32	76	61	62	61
query33	466	363	294	294
query34	737	858	514	514
query35	824	858	760	760
query36	959	1001	902	902
query37	125	99	74	74
query38	4357	4351	4316	4316
query39	1488	1483	1464	1464
query40	203	112	100	100
query41	50	48	56	48
query42	125	111	102	102
query43	507	523	497	497
query44	1352	800	798	798
query45	181	174	163	163
query46	891	1065	662	662
query47	1836	1894	1796	1796
query48	388	432	322	322
query49	689	505	428	428
query50	714	755	446	446
query51	4242	4310	4261	4261
query52	108	99	95	95
query53	238	257	184	184
query54	479	505	422	422
query55	88	81	83	81
query56	285	282	272	272
query57	1170	1205	1119	1119
query58	242	247	274	247
query59	2792	2810	2789	2789
query60	299	286	264	264
query61	127	119	111	111
query62	725	746	727	727
query63	238	192	196	192
query64	1487	1044	680	680
query65	3257	3158	3135	3135
query66	713	395	319	319
query67	15900	15735	15596	15596
query68	5243	766	517	517
query69	509	297	258	258
query70	1183	1077	1129	1077
query71	422	297	262	262
query72	6022	3616	3703	3616
query73	1040	744	352	352
query74	9062	9149	8938	8938
query75	3202	3238	2695	2695
query76	3753	1175	739	739
query77	563	355	273	273
query78	10243	10187	9270	9270
query79	1951	850	605	605
query80	820	537	454	454
query81	534	285	234	234
query82	398	126	91	91
query83	188	167	158	158
query84	289	97	73	73
query85	811	342	301	301
query86	366	294	281	281
query87	4590	4500	4434	4434
query88	2973	2222	2204	2204
query89	390	316	292	292
query90	1679	191	194	191
query91	139	138	108	108
query92	64	58	55	55
query93	2160	1018	575	575
query94	652	383	302	302
query95	344	268	255	255
query96	492	537	267	267
query97	2776	2845	2729	2729
query98	216	198	209	198
query99	1307	1401	1265	1265
Total cold run time: 292528 ms
Total hot run time: 190729 ms

doris-robot · 2025-02-21T16:03:48Z

ClickBench: Total hot run time: 31.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0fcde840e4f6632e1428501f1f8fbd37fe6878ba, data reload: false

query1	0.04	0.04	0.03
query2	0.11	0.05	0.05
query3	0.28	0.05	0.05
query4	1.61	0.07	0.08
query5	0.40	0.50	0.40
query6	1.16	0.65	0.65
query7	0.02	0.02	0.02
query8	0.05	0.05	0.05
query9	0.63	0.52	0.52
query10	0.58	0.58	0.57
query11	0.25	0.13	0.13
query12	0.25	0.13	0.13
query13	0.62	0.61	0.62
query14	2.67	2.71	2.75
query15	0.98	0.90	0.89
query16	0.38	0.36	0.36
query17	1.06	1.04	1.03
query18	0.18	0.20	0.19
query19	1.98	1.85	2.02
query20	0.01	0.02	0.01
query21	15.37	0.97	0.66
query22	0.92	1.05	0.76
query23	14.76	1.58	0.75
query24	5.60	0.55	0.28
query25	0.17	0.10	0.08
query26	0.56	0.22	0.18
query27	0.08	0.09	0.08
query28	11.00	1.13	0.55
query29	12.54	4.13	3.45
query30	0.27	0.08	0.06
query31	2.81	0.60	0.42
query32	3.24	0.59	0.48
query33	3.06	3.07	3.16
query34	16.46	5.06	4.37
query35	4.46	4.45	4.45
query36	0.64	0.50	0.50
query37	0.21	0.18	0.16
query38	0.17	0.15	0.16
query39	0.05	0.04	0.04
query40	0.19	0.15	0.16
query41	0.10	0.06	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 106.03 s
Total hot run time: 31.11 s

fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java

regression-test/suites/external_table_p0/paimon/test_paimon_deletion_vector.groovy

suxiaogang223 · 2025-02-24T02:08:52Z

run buildall

doris-robot · 2025-02-24T02:32:38Z

TeamCity cloud ut coverage result:
Function Coverage: 82.25% (1061/1290)
Line Coverage: 65.76% (17589/26746)
Region Coverage: 65.30% (8666/13271)
Branch Coverage: 55.24% (4674/8462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/84706137a7cd1853c0d089241974979489653bc6_84706137a7cd1853c0d089241974979489653bc6_cloud/report/index.html

doris-robot · 2025-02-24T03:43:20Z

TPC-H: Total hot run time: 31514 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 84706137a7cd1853c0d089241974979489653bc6, data reload: false

------ Round 1 ----------------------------------
q1	17613	5220	5061	5061
q2	2046	299	167	167
q3	10407	1305	733	733
q4	10203	1025	533	533
q5	7538	2508	2331	2331
q6	190	167	132	132
q7	903	780	600	600
q8	9304	1337	1188	1188
q9	4931	4679	4678	4678
q10	6831	2326	1872	1872
q11	470	278	256	256
q12	347	352	228	228
q13	17776	3688	3039	3039
q14	239	240	214	214
q15	518	469	452	452
q16	622	605	597	597
q17	580	885	348	348
q18	6764	6185	6088	6088
q19	1219	972	542	542
q20	330	325	196	196
q21	2837	2154	1954	1954
q22	369	328	305	305
Total cold run time: 102037 ms
Total hot run time: 31514 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5111	5084	5082	5082
q2	237	340	240	240
q3	2202	2691	2322	2322
q4	1414	1880	1367	1367
q5	4227	4123	4184	4123
q6	210	163	122	122
q7	1861	1848	1666	1666
q8	2635	2597	2523	2523
q9	7386	7299	7210	7210
q10	3026	3156	2759	2759
q11	586	502	490	490
q12	681	788	624	624
q13	3553	3921	3252	3252
q14	288	303	291	291
q15	501	469	471	469
q16	631	670	636	636
q17	1118	1560	1374	1374
q18	7671	7240	7185	7185
q19	769	874	1016	874
q20	1951	2024	1897	1897
q21	5317	5036	4831	4831
q22	590	573	552	552
Total cold run time: 51965 ms
Total hot run time: 49889 ms

doris-robot · 2025-02-24T03:54:50Z

TPC-DS: Total hot run time: 183617 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 84706137a7cd1853c0d089241974979489653bc6, data reload: false

query1	993	413	371	371
query2	6541	1896	1839	1839
query3	6801	211	209	209
query4	26090	23703	23413	23413
query5	4336	649	490	490
query6	283	198	189	189
query7	4607	507	304	304
query8	290	240	241	240
query9	8600	2568	2570	2568
query10	480	309	263	263
query11	17270	15049	14836	14836
query12	161	109	108	108
query13	1663	542	397	397
query14	9778	6486	6087	6087
query15	221	199	179	179
query16	7482	616	470	470
query17	1203	709	566	566
query18	1981	409	298	298
query19	204	186	157	157
query20	123	117	114	114
query21	213	129	103	103
query22	4332	4765	4556	4556
query23	34885	33622	32699	32699
query24	7994	2388	2410	2388
query25	551	490	379	379
query26	1227	272	148	148
query27	2570	493	327	327
query28	4273	2417	2390	2390
query29	718	537	418	418
query30	229	184	159	159
query31	937	856	788	788
query32	71	64	62	62
query33	549	357	316	316
query34	755	849	511	511
query35	805	835	757	757
query36	947	982	890	890
query37	119	95	76	76
query38	4061	4269	4171	4171
query39	1448	1412	1392	1392
query40	206	114	102	102
query41	54	52	49	49
query42	127	107	101	101
query43	494	494	489	489
query44	1308	775	777	775
query45	180	172	160	160
query46	866	1041	648	648
query47	1785	1784	1725	1725
query48	388	432	306	306
query49	783	490	412	412
query50	678	740	418	418
query51	4419	4118	4149	4118
query52	110	101	97	97
query53	238	262	183	183
query54	486	483	420	420
query55	78	87	79	79
query56	253	261	253	253
query57	1147	1121	1074	1074
query58	249	232	242	232
query59	2690	2581	2747	2581
query60	285	281	266	266
query61	120	121	120	120
query62	820	723	664	664
query63	237	188	200	188
query64	4271	985	650	650
query65	3246	3151	3143	3143
query66	1149	399	303	303
query67	15709	15576	15507	15507
query68	7296	786	511	511
query69	438	293	277	277
query70	1212	1127	1133	1127
query71	402	286	268	268
query72	5728	3597	3692	3597
query73	633	736	346	346
query74	9221	9148	8638	8638
query75	3251	3156	2665	2665
query76	2940	1178	756	756
query77	755	399	299	299
query78	10038	10078	9372	9372
query79	2309	824	596	596
query80	602	556	460	460
query81	572	275	235	235
query82	511	129	96	96
query83	176	186	151	151
query84	243	96	73	73
query85	795	368	304	304
query86	394	298	299	298
query87	4554	4573	4343	4343
query88	4386	2228	2255	2228
query89	390	319	283	283
query90	1836	199	196	196
query91	147	146	106	106
query92	68	60	59	59
query93	1890	1009	565	565
query94	660	414	286	286
query95	346	273	259	259
query96	497	549	288	288
query97	2823	2827	2736	2736
query98	223	203	202	202
query99	1328	1388	1306	1306
Total cold run time: 274831 ms
Total hot run time: 183617 ms

doris-robot · 2025-02-24T04:00:11Z

ClickBench: Total hot run time: 30.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 84706137a7cd1853c0d089241974979489653bc6, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.10	0.11
query5	0.41	0.41	0.41
query6	1.17	0.64	0.66
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.62	0.51	0.53
query10	0.56	0.57	0.56
query11	0.16	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.61
query14	2.70	2.77	2.75
query15	0.94	0.86	0.86
query16	0.37	0.37	0.37
query17	0.98	1.02	1.00
query18	0.22	0.20	0.20
query19	1.95	1.75	2.00
query20	0.01	0.01	0.01
query21	15.35	0.89	0.55
query22	0.76	1.18	0.62
query23	14.98	1.38	0.65
query24	7.02	0.88	1.82
query25	0.48	0.40	0.08
query26	0.56	0.16	0.14
query27	0.05	0.06	0.04
query28	9.90	0.91	0.43
query29	12.60	3.97	3.26
query30	0.25	0.08	0.06
query31	2.88	0.56	0.39
query32	3.21	0.55	0.46
query33	3.01	3.03	3.03
query34	15.70	5.06	4.51
query35	4.56	4.53	4.53
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.09	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.02
Total cold run time: 105.36 s
Total hot run time: 30.55 s

doris-robot · 2025-02-24T04:34:47Z

TeamCity be ut coverage result:
Function Coverage: 44.51% (11753/26408)
Line Coverage: 34.01% (98270/288986)
Region Coverage: 33.08% (50325/152130)
Branch Coverage: 28.72% (25296/88068)
Coverage Report: http://coverage.selectdb-in.cc/coverage/84706137a7cd1853c0d089241974979489653bc6_84706137a7cd1853c0d089241974979489653bc6/report/index.html

…in expain string

suxiaogang223 · 2025-02-24T07:30:01Z

run buildall

doris-robot · 2025-02-24T07:49:36Z

TeamCity cloud ut coverage result:
Function Coverage: 82.25% (1061/1290)
Line Coverage: 65.73% (17580/26746)
Region Coverage: 65.28% (8663/13271)
Branch Coverage: 55.15% (4667/8462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/26ea3839748cef73ba4797eb851701eca35812b6_26ea3839748cef73ba4797eb851701eca35812b6_cloud/report/index.html

doris-robot · 2025-02-24T08:25:56Z

TPC-H: Total hot run time: 31678 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26ea3839748cef73ba4797eb851701eca35812b6, data reload: false

------ Round 1 ----------------------------------
q1	17629	5393	5164	5164
q2	2061	297	162	162
q3	10519	1294	750	750
q4	10266	1042	550	550
q5	8008	2483	2291	2291
q6	199	172	134	134
q7	946	759	592	592
q8	9293	1312	1180	1180
q9	4916	4661	4648	4648
q10	6836	2311	1912	1912
q11	479	284	267	267
q12	361	361	231	231
q13	17773	3697	3060	3060
q14	227	237	208	208
q15	521	470	479	470
q16	628	616	586	586
q17	579	879	336	336
q18	6585	6290	6127	6127
q19	1516	953	564	564
q20	321	327	192	192
q21	2832	2250	1954	1954
q22	373	339	300	300
Total cold run time: 102868 ms
Total hot run time: 31678 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5336	5191	5106	5106
q2	250	343	233	233
q3	2214	2693	2343	2343
q4	1466	1831	1364	1364
q5	4276	4158	4191	4158
q6	220	167	126	126
q7	1870	1855	1769	1769
q8	2634	2594	2642	2594
q9	7292	7180	7144	7144
q10	3039	3257	2818	2818
q11	573	514	497	497
q12	766	825	627	627
q13	3415	3965	3337	3337
q14	293	297	274	274
q15	495	482	470	470
q16	646	696	641	641
q17	1147	1605	1318	1318
q18	7753	7324	7351	7324
q19	892	935	987	935
q20	2046	1989	1874	1874
q21	5478	5067	4737	4737
q22	610	559	528	528
Total cold run time: 52711 ms
Total hot run time: 50217 ms

suxiaogang223 · 2025-02-24T08:31:11Z

run buildall

doris-robot · 2025-02-24T08:48:15Z

TeamCity cloud ut coverage result:
Function Coverage: 82.25% (1061/1290)
Line Coverage: 65.74% (17583/26746)
Region Coverage: 65.28% (8663/13271)
Branch Coverage: 55.22% (4673/8462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/afc4b053ee98cb0403d9c4fe842628e5664e269b_afc4b053ee98cb0403d9c4fe842628e5664e269b_cloud/report/index.html

doris-robot · 2025-02-24T09:33:44Z

TeamCity be ut coverage result:
Function Coverage: 44.50% (11752/26407)
Line Coverage: 33.99% (98231/288973)
Region Coverage: 33.08% (50316/152110)
Branch Coverage: 28.72% (25290/88052)
Coverage Report: http://coverage.selectdb-in.cc/coverage/afc4b053ee98cb0403d9c4fe842628e5664e269b_afc4b053ee98cb0403d9c4fe842628e5664e269b/report/index.html

suxiaogang223 · 2025-02-24T11:08:11Z

run buildall

doris-robot · 2025-02-24T11:24:04Z

TeamCity cloud ut coverage result:
Function Coverage: 82.25% (1061/1290)
Line Coverage: 65.76% (17587/26746)
Region Coverage: 65.29% (8665/13271)
Branch Coverage: 55.22% (4673/8462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/04fef8071f0fd7c41a3ee2e3d7016f62449cb25e_04fef8071f0fd7c41a3ee2e3d7016f62449cb25e_cloud/report/index.html

doris-robot · 2025-02-24T12:02:35Z

TPC-H: Total hot run time: 31966 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04fef8071f0fd7c41a3ee2e3d7016f62449cb25e, data reload: false

------ Round 1 ----------------------------------
q1	17691	5474	5233	5233
q2	2053	311	174	174
q3	10838	1276	784	784
q4	10524	1069	541	541
q5	8940	2485	2407	2407
q6	222	182	134	134
q7	924	770	606	606
q8	9328	1489	1263	1263
q9	4886	4634	4775	4634
q10	6864	2315	1879	1879
q11	513	284	270	270
q12	363	375	217	217
q13	17779	3741	3024	3024
q14	230	224	201	201
q15	519	475	459	459
q16	654	618	574	574
q17	599	885	347	347
q18	7019	6205	6192	6192
q19	1568	1035	569	569
q20	321	329	184	184
q21	2956	2297	1977	1977
q22	376	326	297	297
Total cold run time: 105167 ms
Total hot run time: 31966 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5401	5335	5450	5335
q2	246	337	231	231
q3	2223	2691	2338	2338
q4	1435	1858	1326	1326
q5	4219	4102	4142	4102
q6	224	168	125	125
q7	1974	1963	1757	1757
q8	2690	2671	2652	2652
q9	7215	7075	7098	7075
q10	3091	3238	2788	2788
q11	592	510	482	482
q12	698	719	578	578
q13	3486	3872	3293	3293
q14	288	290	294	290
q15	511	464	452	452
q16	657	664	661	661
q17	1177	1651	1352	1352
q18	7700	7251	7414	7251
q19	856	887	1119	887
q20	1969	2047	1862	1862
q21	5567	4970	4673	4673
q22	607	575	543	543
Total cold run time: 52826 ms
Total hot run time: 50053 ms

doris-robot · 2025-02-24T12:09:15Z

TeamCity be ut coverage result:
Function Coverage: 44.50% (11752/26407)
Line Coverage: 34.00% (98263/288973)
Region Coverage: 33.09% (50329/152110)
Branch Coverage: 28.73% (25294/88052)
Coverage Report: http://coverage.selectdb-in.cc/coverage/04fef8071f0fd7c41a3ee2e3d7016f62449cb25e_04fef8071f0fd7c41a3ee2e3d7016f62449cb25e/report/index.html

doris-robot · 2025-02-24T12:14:44Z

TPC-DS: Total hot run time: 191597 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 04fef8071f0fd7c41a3ee2e3d7016f62449cb25e, data reload: false

query1	1329	948	951	948
query2	6216	1928	1911	1911
query3	10973	4400	4481	4400
query4	55965	25428	23772	23772
query5	5166	523	494	494
query6	396	197	199	197
query7	5265	520	319	319
query8	330	248	245	245
query9	6803	2784	2793	2784
query10	433	352	255	255
query11	15179	14996	14808	14808
query12	166	118	105	105
query13	1213	549	407	407
query14	10155	6842	6511	6511
query15	207	201	174	174
query16	7100	636	455	455
query17	1059	717	600	600
query18	1545	419	316	316
query19	201	194	165	165
query20	126	122	121	121
query21	207	123	102	102
query22	4374	4436	4480	4436
query23	34244	33612	33290	33290
query24	5788	2452	2412	2412
query25	468	472	394	394
query26	713	280	155	155
query27	2007	507	358	358
query28	2910	2618	2582	2582
query29	636	601	468	468
query30	224	195	157	157
query31	882	892	817	817
query32	77	100	61	61
query33	442	355	301	301
query34	777	893	528	528
query35	807	833	756	756
query36	976	1025	914	914
query37	124	111	76	76
query38	4130	4356	4300	4300
query39	1492	1450	1484	1450
query40	201	117	104	104
query41	51	52	49	49
query42	132	108	115	108
query43	532	532	495	495
query44	1449	860	849	849
query45	177	173	171	171
query46	934	1072	693	693
query47	1845	1871	1802	1802
query48	442	451	343	343
query49	713	514	443	443
query50	778	784	445	445
query51	4259	4313	4287	4287
query52	118	107	101	101
query53	243	282	193	193
query54	497	512	426	426
query55	89	83	87	83
query56	297	275	270	270
query57	1133	1157	1111	1111
query58	247	243	266	243
query59	2847	2852	2748	2748
query60	316	315	316	315
query61	155	138	138	138
query62	751	759	661	661
query63	243	199	209	199
query64	1585	1135	710	710
query65	3296	3141	3147	3141
query66	760	385	289	289
query67	15886	15638	15287	15287
query68	5389	799	532	532
query69	548	308	283	283
query70	1223	1157	1123	1123
query71	461	307	287	287
query72	5938	3593	3680	3593
query73	1007	771	374	374
query74	9140	9179	8834	8834
query75	3254	3149	2707	2707
query76	3871	1176	741	741
query77	536	391	286	286
query78	9916	10067	9203	9203
query79	2131	896	640	640
query80	600	540	454	454
query81	477	280	240	240
query82	447	130	106	106
query83	181	238	160	160
query84	295	93	87	87
query85	787	335	310	310
query86	415	313	269	269
query87	4643	4487	4544	4487
query88	3995	2381	2362	2362
query89	422	324	289	289
query90	1843	193	196	193
query91	137	141	111	111
query92	71	57	52	52
query93	1959	1026	576	576
query94	662	414	300	300
query95	346	263	256	256
query96	599	580	299	299
query97	2772	2883	2721	2721
query98	228	217	212	212
query99	1332	1446	1262	1262
Total cold run time: 296706 ms
Total hot run time: 191597 ms

doris-robot · 2025-02-24T12:20:05Z

ClickBench: Total hot run time: 30.99 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04fef8071f0fd7c41a3ee2e3d7016f62449cb25e, data reload: false

query1	0.03	0.03	0.03
query2	0.10	0.04	0.04
query3	0.27	0.05	0.05
query4	1.60	0.07	0.08
query5	0.42	0.42	0.39
query6	1.17	0.66	0.65
query7	0.02	0.02	0.01
query8	0.06	0.05	0.05
query9	0.61	0.51	0.52
query10	0.56	0.56	0.56
query11	0.24	0.13	0.12
query12	0.24	0.13	0.13
query13	0.63	0.61	0.63
query14	2.71	2.72	2.69
query15	0.99	0.88	0.88
query16	0.37	0.37	0.37
query17	1.05	1.03	1.05
query18	0.17	0.18	0.18
query19	1.97	1.79	2.07
query20	0.01	0.02	0.01
query21	15.35	0.98	0.65
query22	0.92	1.00	0.81
query23	14.70	1.56	0.74
query24	5.48	0.59	0.28
query25	0.16	0.10	0.09
query26	0.56	0.23	0.18
query27	0.09	0.09	0.09
query28	11.01	1.16	0.54
query29	12.54	4.14	3.38
query30	0.28	0.08	0.05
query31	2.81	0.60	0.42
query32	3.22	0.59	0.48
query33	3.04	3.04	3.13
query34	16.55	5.14	4.43
query35	4.55	4.48	4.49
query36	0.63	0.51	0.51
query37	0.20	0.16	0.16
query38	0.17	0.16	0.15
query39	0.05	0.04	0.04
query40	0.18	0.16	0.15
query41	0.11	0.06	0.05
query42	0.07	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 105.94 s
Total hot run time: 30.99 s

suxiaogang223 marked this pull request as draft January 13, 2025 10:53

suxiaogang223 force-pushed the table_count_opt branch from f2b8b1a to 00d24fb Compare January 13, 2025 10:55

suxiaogang223 force-pushed the table_count_opt branch from 1c66856 to 4d1346a Compare January 26, 2025 09:59

suxiaogang223 force-pushed the table_count_opt branch from 4d1346a to 7ea07ce Compare February 11, 2025 09:09

suxiaogang223 added 5 commits February 20, 2025 11:37

opt count for paimon

f95d2c1

impl count opt for paimon jni

1e55325

fix

7f1913d

support raw file with deletion vector

bafbe10

fix paimon_reader

80e7eee

suxiaogang223 force-pushed the table_count_opt branch from 7ea07ce to 80e7eee Compare February 20, 2025 03:37

suxiaogang223 added 2 commits February 20, 2025 16:53

use cardinality to optimize paimon dv table count(*)

58095b0

can't find symbol isEmpty

11f6662

suxiaogang223 marked this pull request as ready for review February 20, 2025 11:15

suxiaogang223 added 4 commits February 20, 2025 19:33

fix

a709849

fix bug

1884797

fix

243579a

add cases

ef33aa7

fix case

0fcde84

morningman added dev/3.0.x-experimental dev/2.1.x-experimental and removed dev/2.1.x labels Feb 23, 2025

morningman reviewed Feb 23, 2025

View reviewed changes

suxiaogang223 added 2 commits February 24, 2025 00:38

refactor optimize count logic and clean TableFormatReader code

417b538

fix fe

8470613

suxiaogang223 changed the title ~~[enhance](paimon) opt count pushdown for paimon~~ [enhance](paimon) opt count pushdown for paimon and refactor be logic Feb 24, 2025

fix

b57c4fc

suxiaogang223 added 2 commits February 24, 2025 14:15

add setPushDownCount in FileScanNode to unify count optimize display …

2a0e00c

…in expain string

add more case by explain

26ea383

add fe ut

afc4b05

add lisence

04fef80

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[enhance](paimon) opt count pushdown for paimon and refactor be logic #46911

[enhance](paimon) opt count pushdown for paimon and refactor be logic #46911

suxiaogang223 commented Jan 13, 2025 •

edited

Loading

hello-stephen commented Jan 13, 2025

suxiaogang223 commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

suxiaogang223 commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

suxiaogang223 commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

suxiaogang223 commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

suxiaogang223 commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

suxiaogang223 commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

[enhance](paimon) opt count pushdown for paimon and refactor be logic #46911

Are you sure you want to change the base?

[enhance](paimon) opt count pushdown for paimon and refactor be logic #46911

Conversation

suxiaogang223 commented Jan 13, 2025 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

hello-stephen commented Jan 13, 2025

suxiaogang223 commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

suxiaogang223 commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

doris-robot commented Feb 21, 2025

suxiaogang223 commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

suxiaogang223 commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

suxiaogang223 commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

suxiaogang223 commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

doris-robot commented Feb 24, 2025

suxiaogang223 commented Jan 13, 2025 •

edited

Loading