Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chore](io) Add debug log for critical file operations #46770

Merged
merged 1 commit into from
Jan 12, 2025
Merged

[chore](io) Add debug log for critical file operations #46770

merged 1 commit into from
Jan 12, 2025
+9 −0

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Jan 10, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

dataroaring
dataroaring approved these changes Jan 10, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 10, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 32681 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4beca42c7e8259722aadcd0f2c13b2b4e2d7c86c, data reload: false

------ Round 1 ----------------------------------
q1	17578	7110	6026	6026
q2	2045	302	177	177
q3	10414	1252	713	713
q4	10207	877	451	451
q5	7532	2187	1972	1972
q6	200	176	151	151
q7	901	729	610	610
q8	9221	1314	1152	1152
q9	5199	4889	4879	4879
q10	6752	2294	1862	1862
q11	496	280	252	252
q12	348	356	222	222
q13	17758	3693	3109	3109
q14	234	236	208	208
q15	565	496	497	496
q16	623	621	600	600
q17	580	844	315	315
q18	7051	6440	6507	6440
q19	1212	950	526	526
q20	313	327	201	201
q21	2816	2280	1997	1997
q22	368	343	322	322
Total cold run time: 102413 ms
Total hot run time: 32681 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6162	6233	6185	6185
q2	237	331	236	236
q3	2259	2649	2300	2300
q4	1394	1759	1385	1385
q5	4381	4721	4833	4721
q6	183	176	149	149
q7	2074	1972	1836	1836
q8	2591	2723	2637	2637
q9	7188	7144	7171	7144
q10	3018	3300	2855	2855
q11	594	529	498	498
q12	694	759	611	611
q13	3440	3821	3260	3260
q14	287	306	285	285
q15	565	527	498	498
q16	645	695	640	640
q17	1182	1731	1251	1251
q18	7700	7544	7070	7070
q19	762	774	1070	774
q20	1894	1967	1809	1809
q21	5446	5030	4959	4959
q22	638	594	538	538
Total cold run time: 53334 ms
Total hot run time: 51641 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188660 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4beca42c7e8259722aadcd0f2c13b2b4e2d7c86c, data reload: false

query1	961	380	365	365
query2	6534	2413	2315	2315
query3	6722	216	218	216
query4	33604	23509	23250	23250
query5	4429	635	470	470
query6	290	201	204	201
query7	4617	480	305	305
query8	306	242	249	242
query9	9549	2662	2653	2653
query10	467	307	254	254
query11	17869	15683	15070	15070
query12	153	107	109	107
query13	1647	499	383	383
query14	10976	6901	7074	6901
query15	229	192	175	175
query16	7300	549	456	456
query17	1553	718	544	544
query18	1944	398	285	285
query19	216	176	150	150
query20	114	109	110	109
query21	210	119	103	103
query22	4371	4378	4262	4262
query23	34035	32933	33018	32933
query24	6404	2213	2278	2213
query25	537	449	382	382
query26	1201	263	161	161
query27	2015	461	327	327
query28	5242	2422	2404	2404
query29	649	537	418	418
query30	227	188	155	155
query31	904	878	764	764
query32	124	58	56	56
query33	499	342	291	291
query34	745	868	500	500
query35	785	780	712	712
query36	992	1035	938	938
query37	125	103	74	74
query38	4273	4146	4055	4055
query39	1480	1521	1403	1403
query40	208	116	104	104
query41	55	51	53	51
query42	123	107	103	103
query43	556	537	492	492
query44	1280	803	792	792
query45	176	172	158	158
query46	857	1017	637	637
query47	1844	1817	1783	1783
query48	379	399	297	297
query49	780	476	390	390
query50	615	639	377	377
query51	6933	6786	6951	6786
query52	100	111	92	92
query53	218	259	180	180
query54	480	484	413	413
query55	82	81	86	81
query56	270	271	292	271
query57	1172	1139	1078	1078
query58	236	230	231	230
query59	3183	3148	2978	2978
query60	279	248	255	248
query61	114	115	114	114
query62	858	764	720	720
query63	227	193	195	193
query64	4506	1001	674	674
query65	3253	3153	3203	3153
query66	1068	448	330	330
query67	15723	15590	15464	15464
query68	7652	720	534	534
query69	459	294	267	267
query70	1185	1124	1134	1124
query71	407	356	257	257
query72	6080	3795	3809	3795
query73	631	741	364	364
query74	10342	9057	8826	8826
query75	3285	3170	2649	2649
query76	3161	1182	776	776
query77	502	417	267	267
query78	10123	9980	9855	9855
query79	4068	764	580	580
query80	780	518	428	428
query81	505	282	235	235
query82	513	143	122	122
query83	203	171	152	152
query84	285	93	73	73
query85	784	390	292	292
query86	397	302	299	299
query87	4526	4390	4404	4390
query88	4664	2173	2148	2148
query89	402	326	287	287
query90	1846	194	186	186
query91	133	138	108	108
query92	71	53	54	53
query93	2011	862	538	538
query94	695	389	295	295
query95	339	271	256	256
query96	486	607	289	289
query97	2844	2974	2843	2843
query98	231	207	195	195
query99	1609	1525	1386	1386
Total cold run time: 291092 ms
Total hot run time: 188660 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4beca42c7e8259722aadcd0f2c13b2b4e2d7c86c, data reload: false

query1	0.03	0.03	0.02
query2	0.08	0.02	0.04
query3	0.23	0.08	0.07
query4	1.60	0.11	0.11
query5	0.42	0.40	0.41
query6	1.15	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.02
query9	0.58	0.52	0.50
query10	0.54	0.56	0.54
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.60
query14	2.83	2.73	2.88
query15	0.89	0.82	0.82
query16	0.39	0.39	0.37
query17	1.08	1.07	1.06
query18	0.25	0.21	0.21
query19	1.94	1.83	1.95
query20	0.01	0.01	0.01
query21	15.36	0.93	0.57
query22	0.75	0.95	0.74
query23	15.02	1.44	0.58
query24	3.02	0.89	0.92
query25	0.26	0.24	0.07
query26	0.21	0.15	0.14
query27	0.05	0.06	0.06
query28	13.85	1.55	1.05
query29	12.62	3.96	3.26
query30	0.24	0.10	0.06
query31	2.83	0.58	0.39
query32	3.22	0.57	0.48
query33	3.14	3.28	3.04
query34	16.65	5.08	4.49
query35	4.53	4.48	4.47
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.83 s
Total hot run time: 31.27 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 39.35% (10255/26059)
Line Coverage: 30.52% (87423/286411)
Region Coverage: 29.56% (44546/150719)
Branch Coverage: 26.11% (22799/87324)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4beca42c7e8259722aadcd0f2c13b2b4e2d7c86c_4beca42c7e8259722aadcd0f2c13b2b4e2d7c86c/report/index.html

@dataroaring dataroaring merged commit 5a34929 into apache:master Jan 12, 2025
26 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 12, 2025
@w41ter
[chore](io) Add debug log for critical file operations (#46770)
c4aa22f
@github-actions github-actions bot mentioned this pull request Jan 12, 2025
Merged
@w41ter w41ter deleted the add_debug_log branch January 13, 2025 02:03
w41ter added a commit that referenced this pull request Jan 13, 2025
@github-actions @w41ter
branch-3.0: [chore](io) Add debug log for critical file operations #4…
2648152 
…6770 (#46837)

Cherry-picked from #46770

Co-authored-by: walter <maochuan@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/2.1.x-conflict dev/3.0.x reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@w41ter @Thearas @dataroaring @doris-robot @Yukang-Lian