Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement](external catalog)Optimize the process of refreshing catalog #39205

Merged
merged 4 commits into from
Aug 14, 2024

Conversation

wuwenchi
Copy link
Contributor

Proposed changes

When the catalog attributes have not changed, refreshing the catalog only requires processing the cache, without rebuilding the entire catalog.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the doing label Aug 12, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 39319 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bdf8e87804d2ab227d53ab15c39f8d81f88b380f, data reload: false

------ Round 1 ----------------------------------
q1	17637	4396	4281	4281
q2	2021	202	198	198
q3	11685	1024	1103	1024
q4	10522	730	707	707
q5	7514	2525	2513	2513
q6	221	137	137	137
q7	969	591	584	584
q8	9409	1903	1902	1902
q9	9278	6540	6520	6520
q10	7016	2196	2207	2196
q11	442	239	250	239
q12	395	227	228	227
q13	18922	2997	2938	2938
q14	290	236	236	236
q15	516	505	482	482
q16	500	396	385	385
q17	976	659	674	659
q18	7948	7490	7386	7386
q19	6908	977	997	977
q20	694	327	343	327
q21	5347	4415	4585	4415
q22	1113	996	986	986
Total cold run time: 120323 ms
Total hot run time: 39319 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4471	4231	4253	4231
q2	384	272	272	272
q3	2871	2793	2884	2793
q4	1947	1747	1741	1741
q5	5522	5422	5495	5422
q6	233	139	141	139
q7	2173	1787	1771	1771
q8	3274	3451	3442	3442
q9	8721	8853	8739	8739
q10	3432	3214	3318	3214
q11	617	495	511	495
q12	836	679	609	609
q13	17010	3142	3128	3128
q14	330	293	302	293
q15	541	494	497	494
q16	506	446	440	440
q17	1815	1515	1492	1492
q18	8219	7954	7979	7954
q19	1790	1619	1515	1515
q20	3002	1913	1870	1870
q21	5554	5058	5321	5058
q22	1138	1031	1049	1031
Total cold run time: 74386 ms
Total hot run time: 56143 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203019 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bdf8e87804d2ab227d53ab15c39f8d81f88b380f, data reload: false

query1	1610	1457	1468	1457
query2	6339	1956	1916	1916
query3	13677	8477	302	302
query4	30581	23218	23323	23218
query5	3851	515	531	515
query6	273	181	162	162
query7	3937	305	294	294
query8	246	220	194	194
query9	8417	2500	2490	2490
query10	531	486	466	466
query11	16892	15068	15090	15068
query12	156	96	96	96
query13	1539	372	387	372
query14	9776	6856	7727	6856
query15	264	230	247	230
query16	7276	523	483	483
query17	1243	609	578	578
query18	1860	320	310	310
query19	213	162	171	162
query20	122	109	124	109
query21	214	119	108	108
query22	4543	4498	4259	4259
query23	34464	33713	33593	33593
query24	10950	2653	2630	2630
query25	575	374	402	374
query26	702	154	156	154
query27	2136	278	282	278
query28	5915	2043	2046	2043
query29	754	419	415	415
query30	256	155	155	155
query31	966	773	770	770
query32	87	55	56	55
query33	617	297	298	297
query34	919	467	507	467
query35	954	836	792	792
query36	1107	946	931	931
query37	139	79	76	76
query38	4247	4158	4082	4082
query39	1427	1397	1406	1397
query40	193	121	120	120
query41	46	44	45	44
query42	121	96	94	94
query43	497	489	475	475
query44	1155	727	721	721
query45	227	200	212	200
query46	1181	794	819	794
query47	1902	1791	1785	1785
query48	370	299	304	299
query49	841	433	442	433
query50	859	420	414	414
query51	6775	6758	6720	6720
query52	104	93	92	92
query53	267	181	181	181
query54	903	449	450	449
query55	78	73	77	73
query56	307	257	254	254
query57	1188	1067	1091	1067
query58	251	228	242	228
query59	2993	2808	2840	2808
query60	300	268	263	263
query61	98	94	91	91
query62	818	641	669	641
query63	217	184	182	182
query64	9267	2301	1817	1817
query65	3233	3154	3197	3154
query66	750	337	329	329
query67	15398	14839	14715	14715
query68	4528	557	549	549
query69	415	378	396	378
query70	1207	1137	1060	1060
query71	469	285	279	279
query72	19011	16554	16414	16414
query73	810	330	329	329
query74	9154	8765	8779	8765
query75	3606	2743	2741	2741
query76	3053	1042	1012	1012
query77	669	329	318	318
query78	10238	9035	8945	8945
query79	2151	551	536	536
query80	1823	519	580	519
query81	607	228	228	228
query82	764	138	142	138
query83	311	147	149	147
query84	280	77	80	77
query85	1756	283	283	283
query86	450	314	320	314
query87	4696	4408	4531	4408
query88	3933	2480	2473	2473
query89	425	302	297	297
query90	1773	195	201	195
query91	122	96	99	96
query92	60	49	50	49
query93	3079	540	531	531
query94	815	355	298	298
query95	365	264	269	264
query96	660	274	273	273
query97	3215	3036	3108	3036
query98	225	199	206	199
query99	1629	1317	1271	1271
Total cold run time: 312413 ms
Total hot run time: 203019 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.68 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bdf8e87804d2ab227d53ab15c39f8d81f88b380f, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.50	0.49
query6	1.12	0.72	0.73
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.53	0.50	0.48
query10	0.55	0.55	0.55
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.58
query14	0.79	0.79	0.80
query15	0.85	0.81	0.82
query16	0.36	0.34	0.36
query17	0.99	1.00	0.98
query18	0.22	0.21	0.22
query19	1.80	1.74	1.76
query20	0.01	0.02	0.01
query21	15.40	0.74	0.65
query22	4.45	6.74	2.01
query23	18.24	1.32	1.37
query24	2.16	0.21	0.21
query25	0.15	0.07	0.08
query26	0.31	0.21	0.21
query27	0.46	0.23	0.23
query28	13.29	1.02	0.99
query29	12.60	3.25	3.24
query30	0.24	0.05	0.05
query31	2.87	0.40	0.39
query32	3.30	0.47	0.47
query33	2.87	2.90	2.89
query34	17.05	4.42	4.33
query35	4.44	4.40	4.42
query36	0.65	0.49	0.49
query37	0.19	0.15	0.16
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.15	0.12	0.11
query41	0.09	0.04	0.04
query42	0.06	0.04	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.9 s
Total hot run time: 30.68 s

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40302 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 852bf934f0f068ec2a382ac31c80fb6b68eedd1e, data reload: false

------ Round 1 ----------------------------------
q1	17899	4527	4427	4427
q2	2957	200	189	189
q3	11299	1186	1023	1023
q4	10463	751	701	701
q5	7777	2843	2862	2843
q6	230	142	154	142
q7	991	616	615	615
q8	9498	2045	2093	2045
q9	8564	6592	6581	6581
q10	7040	2239	2228	2228
q11	448	253	245	245
q12	394	221	219	219
q13	18811	2985	3009	2985
q14	288	245	243	243
q15	529	476	491	476
q16	505	384	391	384
q17	972	782	727	727
q18	8110	7395	7316	7316
q19	3847	1059	997	997
q20	689	323	349	323
q21	5829	4588	4577	4577
q22	1133	1017	1016	1016
Total cold run time: 118273 ms
Total hot run time: 40302 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4425	4333	4277	4277
q2	379	285	267	267
q3	2867	2652	2634	2634
q4	1924	1657	1639	1639
q5	5398	5381	5341	5341
q6	224	129	131	129
q7	2050	1679	1680	1679
q8	3145	3335	3367	3335
q9	8373	8389	8407	8389
q10	3364	3149	3177	3149
q11	595	496	505	496
q12	795	626	597	597
q13	17241	2993	3034	2993
q14	293	273	261	261
q15	528	478	474	474
q16	480	428	404	404
q17	1783	1495	1511	1495
q18	7632	7540	7451	7451
q19	1682	1536	1614	1536
q20	2008	1811	1800	1800
q21	5123	4984	5163	4984
q22	1089	1022	1022	1022
Total cold run time: 71398 ms
Total hot run time: 54352 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185256 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 852bf934f0f068ec2a382ac31c80fb6b68eedd1e, data reload: false

query1	913	381	374	374
query2	6461	1894	1862	1862
query3	6657	209	216	209
query4	34357	23285	23070	23070
query5	4223	494	517	494
query6	303	172	168	168
query7	4584	300	292	292
query8	263	202	211	202
query9	8505	2453	2433	2433
query10	436	269	276	269
query11	17809	15005	15089	15005
query12	152	104	98	98
query13	1634	382	358	358
query14	10399	7374	7593	7374
query15	241	162	160	160
query16	7761	523	448	448
query17	1552	546	535	535
query18	1770	277	279	277
query19	185	144	135	135
query20	113	105	107	105
query21	206	120	100	100
query22	4288	4153	4112	4112
query23	33866	33134	33320	33134
query24	11510	2828	2784	2784
query25	651	379	384	379
query26	1554	155	158	155
query27	2845	275	278	275
query28	7320	2048	2037	2037
query29	983	419	395	395
query30	289	147	145	145
query31	962	745	734	734
query32	99	51	53	51
query33	743	283	279	279
query34	965	463	471	463
query35	844	711	721	711
query36	1098	934	921	921
query37	151	80	81	80
query38	3952	3801	3839	3801
query39	1420	1371	1380	1371
query40	268	113	111	111
query41	45	45	45	45
query42	118	101	99	99
query43	491	469	457	457
query44	1155	733	740	733
query45	188	163	166	163
query46	1116	723	744	723
query47	1851	1781	1788	1781
query48	365	288	302	288
query49	1161	415	420	415
query50	801	404	404	404
query51	6790	6737	6640	6640
query52	101	93	95	93
query53	250	193	185	185
query54	1049	494	443	443
query55	80	72	79	72
query56	268	247	258	247
query57	1157	1071	1067	1067
query58	238	220	224	220
query59	2999	2701	2618	2618
query60	292	251	259	251
query61	95	99	95	95
query62	821	638	648	638
query63	223	194	186	186
query64	10681	2283	1714	1714
query65	3213	3402	3164	3164
query66	1321	357	330	330
query67	15336	14980	14886	14886
query68	4601	550	542	542
query69	398	270	268	268
query70	1171	1113	1115	1113
query71	416	273	261	261
query72	7197	2232	1989	1989
query73	760	323	324	323
query74	9225	8816	8758	8758
query75	3501	2719	2680	2680
query76	2798	938	975	938
query77	494	314	318	314
query78	9572	9091	9838	9091
query79	2562	589	529	529
query80	1138	489	476	476
query81	587	224	225	224
query82	961	139	133	133
query83	211	148	147	147
query84	238	82	76	76
query85	1546	271	261	261
query86	520	262	301	262
query87	4359	4246	4127	4127
query88	3742	2416	2421	2416
query89	383	289	284	284
query90	1839	201	195	195
query91	170	97	95	95
query92	69	51	51	51
query93	1839	554	546	546
query94	852	294	281	281
query95	349	259	269	259
query96	594	285	276	276
query97	3187	3113	3029	3029
query98	219	203	204	203
query99	1500	1269	1246	1246
Total cold run time: 301535 ms
Total hot run time: 185256 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.69 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 852bf934f0f068ec2a382ac31c80fb6b68eedd1e, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.08
query5	0.48	0.50	0.48
query6	1.12	0.73	0.74
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.55	0.49	0.48
query10	0.54	0.53	0.54
query11	0.15	0.11	0.12
query12	0.15	0.11	0.12
query13	0.60	0.60	0.58
query14	0.74	0.78	0.78
query15	0.88	0.81	0.82
query16	0.38	0.38	0.37
query17	1.01	1.01	1.01
query18	0.23	0.22	0.23
query19	1.82	1.71	1.77
query20	0.01	0.01	0.01
query21	15.42	0.75	0.67
query22	4.25	7.42	1.99
query23	18.29	1.32	1.17
query24	2.21	0.22	0.22
query25	0.15	0.09	0.09
query26	0.29	0.21	0.21
query27	0.46	0.23	0.22
query28	13.23	1.02	1.00
query29	12.61	3.33	3.29
query30	0.24	0.05	0.05
query31	2.89	0.39	0.40
query32	3.26	0.47	0.48
query33	2.92	2.94	2.97
query34	16.76	4.40	4.37
query35	4.41	4.46	4.38
query36	0.66	0.47	0.46
query37	0.18	0.15	0.15
query38	0.15	0.16	0.15
query39	0.06	0.04	0.04
query40	0.14	0.13	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.05
Total cold run time: 109.56 s
Total hot run time: 30.69 s

@wuwenchi
Copy link
Contributor Author

run feut

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 14, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit e70e91b into apache:master Aug 14, 2024
29 of 30 checks passed
yiguolei pushed a commit that referenced this pull request Aug 17, 2024
…alog for 2.1 (#39205) (#39186)

## Proposed changes

bp: #39205

When the catalog attributes have not changed, refreshing the catalog
only requires processing the cache, without rebuilding the entire
catalog.
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
…alog (#39205)

## Proposed changes

When the catalog attributes have not changed, refreshing the catalog
only requires processing the cache, without rebuilding the entire
catalog.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged doing reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants