Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improvement](stream load) display the time of receiving network data #40735

Merged
merged 1 commit into from
Sep 14, 2024
Merged

[improvement](stream load) display the time of receiving network data #40735

merged 1 commit into from
Sep 14, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Sep 12, 2024

Display the time of receiving network data.

For example:

Stream load result: {
    "TxnId": 19,
    "Label": "c10ee032-c19c-484f-80b2-16f6c1d0c209",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 6,
    "NumberLoadedRows": 6,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 60,
    "LoadTimeMs": 32,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 4,
    "ReceiveDataTimeMs": 6,
    "CommitAndPublishTimeMs": 22
}

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Sep 12, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39470 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4c9f032858992ac1ef6c72e1f2110644437b9f93, data reload: false

------ Round 1 ----------------------------------
q1	17612	4149	4153	4149
q2	2016	186	190	186
q3	10455	1264	1388	1264
q4	10464	968	1068	968
q5	7734	3168	3179	3168
q6	223	136	134	134
q7	1032	663	637	637
q8	9464	1999	2020	1999
q9	6745	6285	6328	6285
q10	7007	2511	2449	2449
q11	444	256	245	245
q12	424	233	235	233
q13	17757	3047	3024	3024
q14	275	244	256	244
q15	576	493	489	489
q16	534	421	430	421
q17	994	940	926	926
q18	7326	6523	6709	6523
q19	1388	1234	1234	1234
q20	610	334	327	327
q21	3907	3564	3557	3557
q22	1099	1020	1008	1008
Total cold run time: 108086 ms
Total hot run time: 39470 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4222	4199	4198	4198
q2	341	242	231	231
q3	2908	2891	2921	2891
q4	1951	1972	1939	1939
q5	5792	5713	5830	5713
q6	228	129	132	129
q7	2122	1829	1762	1762
q8	3342	3404	3452	3404
q9	8784	8821	8750	8750
q10	3526	3653	3589	3589
q11	589	475	465	465
q12	835	617	662	617
q13	9527	3291	3152	3152
q14	313	267	283	267
q15	562	507	512	507
q16	549	463	461	461
q17	1813	1742	1716	1716
q18	8545	7886	8104	7886
q19	1776	1752	1753	1752
q20	2153	1893	1874	1874
q21	5898	5888	5789	5789
q22	1155	1013	1019	1013
Total cold run time: 66931 ms
Total hot run time: 58105 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199206 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4c9f032858992ac1ef6c72e1f2110644437b9f93, data reload: false

query1	1277	872	916	872
query2	6304	1729	1751	1729
query3	10652	3967	3772	3772
query4	58168	25985	23737	23737
query5	5067	522	524	522
query6	304	167	165	165
query7	5546	308	297	297
query8	284	232	228	228
query9	6006	2465	2460	2460
query10	429	287	278	278
query11	15932	15584	15492	15492
query12	154	101	98	98
query13	1407	395	399	395
query14	10565	7376	6737	6737
query15	214	187	179	179
query16	7122	526	504	504
query17	1163	607	600	600
query18	1940	320	313	313
query19	207	155	156	155
query20	122	115	113	113
query21	215	113	109	109
query22	4585	4408	4497	4408
query23	34334	33961	33554	33554
query24	6199	3127	3041	3041
query25	535	408	419	408
query26	611	157	159	157
query27	1573	292	294	292
query28	2777	2054	2034	2034
query29	694	429	434	429
query30	221	162	160	160
query31	976	784	832	784
query32	77	59	60	59
query33	500	305	309	305
query34	903	486	472	472
query35	856	745	750	745
query36	1037	912	916	912
query37	139	90	82	82
query38	4023	3998	3976	3976
query39	1430	1415	1407	1407
query40	208	114	115	114
query41	49	46	47	46
query42	120	102	105	102
query43	459	465	449	449
query44	1182	763	764	763
query45	203	170	168	168
query46	1102	834	822	822
query47	1890	1771	1816	1771
query48	367	293	293	293
query49	743	474	536	474
query50	929	441	440	440
query51	7074	6881	6857	6857
query52	102	88	88	88
query53	258	185	182	182
query54	575	456	469	456
query55	77	77	71	71
query56	295	268	255	255
query57	1206	1094	1096	1094
query58	226	233	245	233
query59	2697	2623	2525	2525
query60	303	288	279	279
query61	102	120	96	96
query62	822	675	692	675
query63	222	187	186	186
query64	1623	695	703	695
query65	3219	3171	3172	3171
query66	676	335	355	335
query67	16215	15534	15708	15534
query68	2020	872	885	872
query69	444	326	343	326
query70	1160	1124	1167	1124
query71	360	347	351	347
query72	6031	3375	3438	3375
query73	604	577	582	577
query74	9320	9221	9258	9221
query75	3155	2993	3030	2993
query76	990	880	866	866
query77	523	430	403	403
query78	9501	9295	9295	9295
query79	950	908	875	875
query80	873	861	868	861
query81	457	263	261	261
query82	267	264	262	262
query83	192	189	188	188
query84	229	106	107	106
query85	653	405	392	392
query86	349	321	324	321
query87	4480	4528	4425	4425
query88	4144	4082	4089	4082
query89	378	377	378	377
query90	1661	315	303	303
query91	122	120	122	120
query92	78	77	75	75
query93	1071	1065	1072	1065
query94	623	389	368	368
query95	456	425	422	422
query96	470	471	472	471
query97	3120	3158	3149	3149
query98	229	272	222	222
query99	1724	1308	1316	1308
Total cold run time: 294126 ms
Total hot run time: 199206 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4c9f032858992ac1ef6c72e1f2110644437b9f93, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.03
query3	0.22	0.05	0.05
query4	1.67	0.10	0.08
query5	0.51	0.50	0.51
query6	1.13	0.73	0.74
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.59	0.53	0.53
query10	0.56	0.58	0.60
query11	0.16	0.12	0.12
query12	0.15	0.13	0.12
query13	0.62	0.62	0.60
query14	1.47	1.48	1.46
query15	0.89	0.89	0.89
query16	0.36	0.37	0.37
query17	1.04	1.04	1.03
query18	0.16	0.16	0.17
query19	1.89	1.78	1.81
query20	0.02	0.01	0.01
query21	15.40	0.68	0.67
query22	4.04	8.36	1.14
query23	18.07	1.28	1.27
query24	2.30	0.22	0.23
query25	0.19	0.08	0.08
query26	0.29	0.18	0.18
query27	0.09	0.08	0.08
query28	13.16	1.01	0.98
query29	12.60	3.37	3.38
query30	0.25	0.05	0.06
query31	2.87	0.42	0.42
query32	3.23	0.50	0.49
query33	3.09	3.07	3.09
query34	15.44	4.35	4.34
query35	4.35	4.37	4.32
query36	0.68	0.49	0.50
query37	0.19	0.17	0.16
query38	0.17	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 108.45 s
Total hot run time: 30.9 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.88% (9454/25632)
Line Coverage: 28.25% (77757/275258)
Region Coverage: 27.65% (40145/145170)
Branch Coverage: 24.27% (20400/84060)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4c9f032858992ac1ef6c72e1f2110644437b9f93_4c9f032858992ac1ef6c72e1f2110644437b9f93/report/index.html

liaoxin01
liaoxin01 approved these changes Sep 12, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@liaoxin01 liaoxin01 changed the title [feature](stream load) display the time of receiving network data [improvement](stream load) display the time of receiving network data Sep 12, 2024
dataroaring
dataroaring approved these changes Sep 14, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 03ff755 into apache:master Sep 14, 2024
26 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Sep 19, 2024
@sollhui @dataroaring
[improvement](stream load) display the time of receiving network data (…
7fc850f 
…#40735)

Display the time of receiving network data.

For example:
```
Stream load result: {
    "TxnId": 19,
    "Label": "c10ee032-c19c-484f-80b2-16f6c1d0c209",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 6,
    "NumberLoadedRows": 6,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 60,
    "LoadTimeMs": 32,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 4,
    "ReceiveDataTimeMs": 6,
    "CommitAndPublishTimeMs": 22
}
```
This was referenced Sep 23, 2024
Merged
Merged
yiguolei pushed a commit that referenced this pull request Sep 24, 2024
@sollhui
[improvement](stream load) display the time of receiving network data (…
602275d 
…#40735) (#41139)

pick (#40735)

Display the time of receiving network data.

For example:
```
Stream load result: {
    "TxnId": 19,
    "Label": "c10ee032-c19c-484f-80b2-16f6c1d0c209",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 6,
    "NumberLoadedRows": 6,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 60,
    "LoadTimeMs": 32,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 4,
    "ReceiveDataTimeMs": 6,
    "CommitAndPublishTimeMs": 22
}
```
dataroaring pushed a commit that referenced this pull request Sep 26, 2024
@sollhui
[improve](stream-load) add observability on receiving HTTP request (#…
9b14720 
…41136)

pick #30432 and
#40735

---------

Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com>
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.16-merged dev/2.1.7-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@sollhui @doris-robot @liaoxin01 @dataroaring @yiguolei