Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](cloud-mow) make the waiting time on getDeleteBitmapUpdateLock more reasonable #40341

Merged
merged 1 commit into from
Sep 18, 2024
Merged

[improve](cloud-mow) make the waiting time on getDeleteBitmapUpdateLock more reasonable #40341

merged 1 commit into from
Sep 18, 2024

Conversation

hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Sep 3, 2024

now the expired time of mow delete bitmap lock on ms is 60s, so fe should try to wait at most 60s to getDeleteBitmapUpdateLock.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Sep 3, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38004 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7db9f14607e45f0bb42d8ebbfd45bfd44fcd515f, data reload: false

------ Round 1 ----------------------------------
q1	17704	4368	4331	4331
q2	2032	176	180	176
q3	11970	978	1179	978
q4	10512	737	668	668
q5	7744	2853	2832	2832
q6	226	141	138	138
q7	962	613	598	598
q8	9322	2075	2089	2075
q9	7138	6524	6530	6524
q10	6990	2231	2230	2230
q11	465	245	244	244
q12	396	223	228	223
q13	17766	3095	3048	3048
q14	274	234	243	234
q15	533	494	497	494
q16	585	521	514	514
q17	983	735	743	735
q18	7291	6800	6819	6800
q19	1405	1088	1087	1087
q20	668	333	330	330
q21	4296	2961	2734	2734
q22	1102	1061	1011	1011
Total cold run time: 110364 ms
Total hot run time: 38004 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4375	4373	4288	4288
q2	381	273	273	273
q3	2828	2629	2681	2629
q4	1894	1676	1612	1612
q5	5591	5660	5724	5660
q6	254	147	138	138
q7	2214	1845	1801	1801
q8	3334	3410	3449	3410
q9	8795	8883	8786	8786
q10	3584	3363	3421	3363
q11	612	529	540	529
q12	856	630	671	630
q13	14856	3279	3286	3279
q14	316	290	299	290
q15	562	494	490	490
q16	620	567	578	567
q17	1851	1552	1571	1552
q18	8188	7774	7815	7774
q19	1751	1554	1524	1524
q20	2154	1930	1882	1882
q21	5816	5499	5393	5393
q22	1129	1074	1071	1071
Total cold run time: 71961 ms
Total hot run time: 56941 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193599 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7db9f14607e45f0bb42d8ebbfd45bfd44fcd515f, data reload: false

query1	1251	894	882	882
query2	6353	2039	2016	2016
query3	10769	4117	4179	4117
query4	60456	25200	23185	23185
query5	5392	499	488	488
query6	396	162	166	162
query7	5787	292	288	288
query8	296	209	210	209
query9	8878	2482	2465	2465
query10	487	268	257	257
query11	18314	15088	15356	15088
query12	151	103	101	101
query13	1569	405	386	386
query14	11081	7487	7402	7402
query15	223	182	182	182
query16	7375	486	510	486
query17	1200	581	576	576
query18	1841	289	305	289
query19	296	145	150	145
query20	131	109	107	107
query21	202	105	107	105
query22	4777	4615	4808	4615
query23	34048	33279	33286	33279
query24	5967	2945	2915	2915
query25	508	380	379	379
query26	650	149	155	149
query27	1735	279	275	275
query28	3824	2053	2038	2038
query29	651	413	424	413
query30	240	154	162	154
query31	903	751	748	748
query32	89	54	55	54
query33	480	303	293	293
query34	857	494	478	478
query35	837	729	698	698
query36	1064	931	969	931
query37	147	89	92	89
query38	4030	3950	3925	3925
query39	1445	1384	1403	1384
query40	196	112	118	112
query41	47	44	43	43
query42	111	94	92	92
query43	512	493	481	481
query44	1091	728	726	726
query45	197	165	160	160
query46	1092	730	756	730
query47	1905	1785	1818	1785
query48	376	288	297	288
query49	759	408	430	408
query50	821	409	417	409
query51	7036	6936	6926	6926
query52	97	87	86	86
query53	251	180	182	180
query54	601	446	456	446
query55	78	75	72	72
query56	274	259	264	259
query57	1175	1076	1078	1076
query58	225	219	221	219
query59	3114	3037	2904	2904
query60	302	293	283	283
query61	104	98	100	98
query62	738	654	645	645
query63	212	179	189	179
query64	2319	676	676	676
query65	3226	3140	3122	3122
query66	683	344	338	338
query67	15607	15333	15281	15281
query68	3019	577	578	577
query69	392	294	279	279
query70	1210	1147	1134	1134
query71	363	278	286	278
query72	5217	3993	4065	3993
query73	759	333	331	331
query74	9188	8814	8830	8814
query75	3381	2700	2715	2700
query76	1351	1012	990	990
query77	537	316	319	316
query78	9697	9118	9451	9118
query79	1071	545	539	539
query80	699	520	509	509
query81	467	240	232	232
query82	250	140	139	139
query83	174	147	153	147
query84	257	78	76	76
query85	692	282	287	282
query86	307	281	305	281
query87	4452	4377	4293	4293
query88	2999	2417	2332	2332
query89	393	287	278	278
query90	1948	189	192	189
query91	124	97	109	97
query92	58	50	51	50
query93	1070	532	532	532
query94	704	298	304	298
query95	319	246	249	246
query96	589	269	262	262
query97	3186	3086	3056	3056
query98	227	207	199	199
query99	1540	1284	1255	1255
Total cold run time: 304489 ms
Total hot run time: 193599 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7db9f14607e45f0bb42d8ebbfd45bfd44fcd515f, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.07
query5	0.50	0.50	0.50
query6	1.12	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.54	0.55	0.53
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.58
query14	2.06	2.05	2.05
query15	0.88	0.82	0.81
query16	0.35	0.36	0.36
query17	1.03	1.00	1.00
query18	0.22	0.19	0.19
query19	1.96	1.73	1.77
query20	0.02	0.01	0.01
query21	15.38	0.66	0.65
query22	4.94	7.67	1.32
query23	18.21	1.37	1.30
query24	2.16	0.23	0.21
query25	0.16	0.08	0.07
query26	0.28	0.17	0.18
query27	0.08	0.07	0.08
query28	13.24	1.02	1.00
query29	12.62	3.37	3.33
query30	0.24	0.05	0.06
query31	2.88	0.39	0.38
query32	3.26	0.47	0.48
query33	2.95	3.03	3.04
query34	16.94	4.40	4.38
query35	4.50	4.48	4.54
query36	0.65	0.46	0.46
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.12
query41	0.09	0.06	0.04
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 111.46 s
Total hot run time: 31.41 s

dataroaring
dataroaring approved these changes Sep 14, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 14, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

zhannngchen
zhannngchen approved these changes Sep 18, 2024
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhannngchen zhannngchen merged commit ef519c7 into apache:master Sep 18, 2024
27 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Sep 20, 2024
@hust-hhb @dataroaring
[improve](cloud-mow) make the waiting time on getDeleteBitmapUpdateLo…
3c6e35f 
…ck more reasonable (#40341)

now the expired time of mow delete bitmap lock on ms is 60s, so fe
should try to wait at most 60s to getDeleteBitmapUpdateLock.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@zhannngchen zhannngchen zhannngchen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@hust-hhb @doris-robot @zhannngchen @dataroaring @gavinchou