Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](stream-load) add observability on receiving HTTP request #30432

Merged
merged 1 commit into from
Feb 1, 2024
Merged

[improve](stream-load) add observability on receiving HTTP request #30432

merged 1 commit into from
Feb 1, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Jan 26, 2024 •
edited
Loading

Proposed changes

We meet occasional slow stream-load, ReadDataTimeMs is very fast but WriteDataTimeMs is too long, according to some logs, it can be inferred that data from receive to vtablet_writer is slow, but can not judge the root cause is the scanner slow or is it slow to receive data. Finally, finding the network is too slow, but it costs lots of time. Therefore, recording the time on receiving HTTP request is helpful to users to know why the load is slow.

Observability effect

bvar
dIK2Q8phqj
BkrtkIfR93

logs

finished to execute stream load. label=5fb08e7b-bfa4-440a-881f-81097ab986c5, txn_id=23043, query_id=f54a858b9118ef65-9f9f6e362c689ea4, receive_data_co
st_ms=216, read_data_cost_ms=7803, write_data_cost_ms=9437

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@sollhui sollhui force-pushed the stream_load_observability branch from dff7b50 to 882d93d Compare January 26, 2024 09:19
@sollhui
Copy link
Contributor Author

sollhui commented Jan 26, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39270 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 882d93d0d56d3dcc3ebe3d3e9452ac2f20f1a5ca, data reload: false

------ Round 1 ----------------------------------
q1	17661	5803	5292	5292
q2	2054	146	132	132
q3	10600	978	947	947
q4	4642	770	826	770
q5	7689	2928	3009	2928
q6	186	121	124	121
q7	1089	739	716	716
q8	9262	2025	2028	2025
q9	7263	6418	6392	6392
q10	8103	2460	2428	2428
q11	421	223	201	201
q12	744	298	287	287
q13	18006	3336	3367	3336
q14	295	264	253	253
q15	549	515	497	497
q16	471	408	430	408
q17	938	557	557	557
q18	7564	6978	6864	6864
q19	1552	1297	1464	1297
q20	613	349	357	349
q21	6980	3162	3261	3162
q22	808	314	308	308
Total cold run time: 107490 ms
Total hot run time: 39270 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5352	5358	5187	5187
q2	324	223	208	208
q3	3324	3207	3195	3195
q4	2115	2019	2057	2019
q5	5868	5882	5882	5882
q6	199	119	120	119
q7	2293	1902	1850	1850
q8	3198	3382	3391	3382
q9	8933	8899	8726	8726
q10	5290	3929	3824	3824
q11	563	453	439	439
q12	814	653	611	611
q13	16897	3160	3135	3135
q14	294	271	265	265
q15	553	498	496	496
q16	531	460	472	460
q17	1854	1818	1795	1795
q18	9532	19508	9636	9636
q19	24118	1580	1528	1528
q20	4621	1983	1911	1911
q21	14490	5416	5299	5299
q22	969	521	553	521
Total cold run time: 112132 ms
Total hot run time: 60488 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176242 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 882d93d0d56d3dcc3ebe3d3e9452ac2f20f1a5ca, data reload: false

query1	932	335	323	323
query2	6576	1937	1924	1924
query3	6698	201	199	199
query4	31458	22402	22183	22183
query5	4453	439	442	439
query6	253	155	153	153
query7	4606	269	259	259
query8	222	176	177	176
query9	8392	2538	2520	2520
query10	415	242	232	232
query11	16994	15689	15559	15559
query12	121	69	65	65
query13	1682	382	381	381
query14	9794	6899	6750	6750
query15	211	177	190	177
query16	5699	253	250	250
query17	958	475	478	475
query18	1802	260	249	249
query19	176	134	131	131
query20	73	71	67	67
query21	188	133	130	130
query22	4912	4473	4743	4473
query23	31478	31083	30803	30803
query24	11592	2858	2773	2773
query25	589	311	311	311
query26	1763	150	138	138
query27	3172	287	287	287
query28	7510	1821	1805	1805
query29	1805	639	609	609
query30	273	134	137	134
query31	952	724	752	724
query32	66	47	50	47
query33	691	210	213	210
query34	1134	456	465	456
query35	883	764	791	764
query36	1273	1223	1223	1223
query37	93	59	61	59
query38	3356	3269	3259	3259
query39	1305	1266	1275	1266
query40	343	84	85	84
query41	36	35	34	34
query42	90	83	78	78
query43	513	465	445	445
query44	1053	690	693	690
query45	196	174	176	174
query46	1049	665	648	648
query47	1588	1525	1549	1525
query48	407	323	310	310
query49	1247	293	287	287
query50	696	312	310	310
query51	5274	5252	5213	5213
query52	86	75	74	74
query53	329	260	261	260
query54	252	185	184	184
query55	81	73	79	73
query56	178	168	162	162
query57	973	922	951	922
query58	176	147	151	147
query59	2978	2788	2470	2470
query60	208	179	180	179
query61	83	91	85	85
query62	636	360	350	350
query63	292	260	252	252
query64	6011	1805	1777	1777
query65	3332	3260	3251	3251
query66	1298	317	315	315
query67	15619	15282	15259	15259
query68	10916	535	546	535
query69	597	293	299	293
query70	1683	1459	1542	1459
query71	536	212	227	212
query72	4846	2848	2866	2848
query73	2127	325	312	312
query74	6982	6385	6316	6316
query75	4690	2352	2323	2323
query76	6737	1032	1036	1032
query77	662	231	236	231
query78	9017	9017	8549	8549
query79	1009	509	488	488
query80	591	315	315	315
query81	444	196	197	196
query82	203	80	74	74
query83	136	118	117	117
query84	273	69	67	67
query85	1122	325	333	325
query86	381	381	403	381
query87	3459	3370	3316	3316
query88	3298	2224	2231	2224
query89	437	357	355	355
query90	2019	187	184	184
query91	143	116	128	116
query92	52	43	43	43
query93	3212	462	415	415
query94	1263	158	162	158
query95	496	451	460	451
query96	638	318	316	316
query97	4268	4151	4179	4151
query98	203	195	191	191
query99	1029	708	718	708
Total cold run time: 293043 ms
Total hot run time: 176242 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 882d93d0d56d3dcc3ebe3d3e9452ac2f20f1a5ca, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.03	0.03
query3	0.23	0.05	0.05
query4	1.70	0.07	0.06
query5	0.53	0.52	0.52
query6	1.27	0.64	0.63
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.49	0.53
query10	0.57	0.56	0.56
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.61	0.61	0.60
query14	0.78	0.79	0.82
query15	0.81	0.77	0.78
query16	0.37	0.38	0.40
query17	1.01	1.04	1.00
query18	0.24	0.27	0.22
query19	1.85	1.83	1.74
query20	0.02	0.01	0.01
query21	15.41	0.58	0.54
query22	2.26	2.50	2.15
query23	17.33	0.87	0.72
query24	2.32	0.72	0.92
query25	0.41	0.31	0.11
query26	0.40	0.15	0.15
query27	0.06	0.06	0.06
query28	12.03	0.78	0.77
query29	12.49	3.29	3.18
query30	0.52	0.49	0.50
query31	2.78	0.34	0.35
query32	3.37	0.47	0.48
query33	3.26	3.24	3.25
query34	15.79	4.35	4.31
query35	4.33	4.32	4.38
query36	1.13	1.10	1.10
query37	0.06	0.05	0.05
query38	0.03	0.04	0.02
query39	0.02	0.02	0.01
query40	0.19	0.13	0.14
query41	0.08	0.02	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.24 s
Total hot run time: 31.01 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 882d93d0d56d3dcc3ebe3d3e9452ac2f20f1a5ca with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.2 seconds inserted 10000000 Rows, about 757K ops/s

@sollhui sollhui force-pushed the stream_load_observability branch from 882d93d to c81ad8d Compare January 26, 2024 11:37
@sollhui
Copy link
Contributor Author

sollhui commented Jan 26, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38966 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c81ad8d52239a21969bd93a50037a32b916e40a8, data reload: false

------ Round 1 ----------------------------------
q1	17679	5113	5425	5113
q2	2037	139	134	134
q3	10603	950	961	950
q4	4641	778	800	778
q5	7657	2944	2926	2926
q6	180	123	120	120
q7	1084	726	721	721
q8	9236	2020	2036	2020
q9	7311	6427	6395	6395
q10	8127	2448	2480	2448
q11	411	212	205	205
q12	761	283	282	282
q13	17979	3348	3340	3340
q14	286	261	254	254
q15	558	507	501	501
q16	472	399	404	399
q17	929	619	520	520
q18	7575	6731	6686	6686
q19	1589	1366	1444	1366
q20	563	324	335	324
q21	6964	3173	3230	3173
q22	808	313	311	311
Total cold run time: 107450 ms
Total hot run time: 38966 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5497	5266	5125	5125
q2	335	212	212	212
q3	3321	3241	3144	3144
q4	2157	2097	2044	2044
q5	5973	5920	5961	5920
q6	198	120	115	115
q7	2283	1891	1880	1880
q8	3225	3365	3386	3365
q9	8961	8779	9006	8779
q10	5490	3750	3832	3750
q11	555	442	438	438
q12	803	616	630	616
q13	16899	3157	3116	3116
q14	267	254	270	254
q15	555	510	500	500
q16	519	483	458	458
q17	1862	1847	1872	1847
q18	9505	19832	9643	9643
q19	24515	1598	1518	1518
q20	4671	1927	1906	1906
q21	14632	5647	5643	5643
q22	1006	539	565	539
Total cold run time: 113229 ms
Total hot run time: 60812 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 175860 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c81ad8d52239a21969bd93a50037a32b916e40a8, data reload: false

query1	931	333	322	322
query2	6568	2010	1806	1806
query3	6700	196	195	195
query4	30795	22343	22135	22135
query5	4452	430	367	367
query6	248	151	145	145
query7	4602	260	260	260
query8	222	169	181	169
query9	8440	2517	2509	2509
query10	407	217	200	200
query11	16817	15421	15449	15421
query12	120	67	68	67
query13	1681	368	384	368
query14	9731	6656	6672	6656
query15	209	181	187	181
query16	5695	253	241	241
query17	956	481	460	460
query18	1783	244	250	244
query19	181	138	124	124
query20	70	64	67	64
query21	195	132	127	127
query22	5193	5031	5049	5031
query23	31319	30813	30556	30556
query24	11627	2759	2762	2759
query25	584	308	308	308
query26	1728	135	136	135
query27	3199	282	280	280
query28	7640	1838	1821	1821
query29	1680	622	610	610
query30	273	136	136	136
query31	974	727	749	727
query32	70	53	45	45
query33	698	211	204	204
query34	1138	466	465	465
query35	871	776	748	748
query36	1370	1224	1243	1224
query37	92	59	63	59
query38	3330	3246	3225	3225
query39	1300	1247	1235	1235
query40	201	85	86	85
query41	38	39	38	38
query42	88	77	82	77
query43	546	484	505	484
query44	1095	686	685	685
query45	188	182	174	174
query46	1064	658	664	658
query47	1677	1575	1552	1552
query48	393	318	304	304
query49	1138	279	275	275
query50	666	313	300	300
query51	5236	5233	5189	5189
query52	90	78	78	78
query53	321	251	255	251
query54	235	186	175	175
query55	75	79	82	79
query56	167	165	147	147
query57	1013	893	926	893
query58	181	156	149	149
query59	2988	2801	2770	2770
query60	201	174	173	173
query61	87	81	82	81
query62	614	345	365	345
query63	287	260	271	260
query64	4971	1757	1735	1735
query65	3294	3260	3251	3251
query66	1326	319	307	307
query67	15652	15112	15068	15068
query68	16459	501	483	483
query69	600	279	286	279
query70	2291	1500	1503	1500
query71	508	215	201	201
query72	5018	2801	2812	2801
query73	4483	318	303	303
query74	7003	6395	6401	6395
query75	5020	2336	2292	2292
query76	6270	1020	1018	1018
query77	760	247	229	229
query78	9772	8821	8713	8713
query79	2432	489	498	489
query80	638	315	310	310
query81	483	211	204	204
query82	201	83	77	77
query83	170	113	109	109
query84	278	65	67	65
query85	1106	341	322	322
query86	380	397	378	378
query87	3500	3321	3306	3306
query88	2799	2198	2188	2188
query89	421	352	338	338
query90	2012	183	187	183
query91	151	114	126	114
query92	51	43	47	43
query93	1287	437	408	408
query94	1360	154	160	154
query95	501	447	436	436
query96	605	322	318	318
query97	4261	4123	4163	4123
query98	193	187	182	182
query99	1063	698	687	687
Total cold run time: 299797 ms
Total hot run time: 175860 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c81ad8d52239a21969bd93a50037a32b916e40a8, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.02	0.03
query3	0.23	0.05	0.04
query4	1.68	0.06	0.06
query5	0.54	0.53	0.52
query6	1.24	0.65	0.62
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.54	0.50	0.48
query10	0.55	0.54	0.55
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.61	0.60	0.61
query14	0.77	0.82	0.78
query15	0.79	0.77	0.77
query16	0.37	0.39	0.38
query17	1.03	0.97	1.03
query18	0.25	0.23	0.23
query19	1.83	1.81	1.76
query20	0.01	0.01	0.01
query21	15.42	0.58	0.58
query22	2.34	2.17	2.14
query23	17.19	0.76	0.78
query24	2.91	0.70	1.37
query25	0.40	0.26	0.10
query26	0.47	0.13	0.14
query27	0.06	0.04	0.06
query28	11.35	0.76	0.75
query29	12.52	3.08	3.16
query30	0.54	0.50	0.52
query31	2.78	0.34	0.34
query32	3.37	0.49	0.49
query33	3.25	3.16	3.21
query34	15.84	4.33	4.31
query35	4.38	4.32	4.23
query36	1.10	1.07	1.08
query37	0.07	0.05	0.05
query38	0.03	0.03	0.02
query39	0.02	0.02	0.01
query40	0.15	0.15	0.13
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.11 s
Total hot run time: 30.64 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c81ad8d52239a21969bd93a50037a32b916e40a8 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

@sollhui sollhui force-pushed the stream_load_observability branch from c81ad8d to cbcb699 Compare January 27, 2024 02:53
@sollhui
Copy link
Contributor Author

sollhui commented Jan 27, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 37404 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cbcb6998621b798b06ec737f0a296f186841b677, data reload: false

------ Round 1 ----------------------------------
q1	17659	5067	4871	4871
q2	2041	148	138	138
q3	10686	933	923	923
q4	4711	770	730	730
q5	7714	2876	2740	2740
q6	187	118	119	118
q7	1121	727	709	709
q8	9362	2036	2006	2006
q9	7160	6295	6317	6295
q10	8087	2420	2421	2420
q11	414	200	205	200
q12	785	287	282	282
q13	18018	3242	3275	3242
q14	273	246	243	243
q15	536	518	489	489
q16	456	408	403	403
q17	931	569	516	516
q18	6786	6285	6025	6025
q19	1565	1458	1377	1377
q20	626	339	365	339
q21	7130	3039	3087	3039
q22	811	299	300	299
Total cold run time: 107059 ms
Total hot run time: 37404 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5191	5008	4985	4985
q2	332	242	236	236
q3	3025	2753	2709	2709
q4	1781	1504	1514	1504
q5	5215	5230	5395	5230
q6	198	114	115	114
q7	2201	1837	1804	1804
q8	3097	3204	3204	3204
q9	8703	8689	11929	8689
q10	5599	3411	3478	3411
q11	556	462	477	462
q12	712	610	561	561
q13	16958	3052	3102	3052
q14	283	253	262	253
q15	530	488	486	486
q16	524	470	453	453
q17	1863	1765	1681	1681
q18	8951	10923	8629	8629
q19	23991	1572	1559	1559
q20	4591	1922	1904	1904
q21	15278	4816	4730	4730
q22	1124	459	472	459
Total cold run time: 110703 ms
Total hot run time: 56115 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173064 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cbcb6998621b798b06ec737f0a296f186841b677, data reload: false

query1	949	333	321	321
query2	6570	1882	1900	1882
query3	6718	201	197	197
query4	32217	22108	21952	21952
query5	4448	418	353	353
query6	264	150	148	148
query7	4609	258	258	258
query8	238	178	173	173
query9	8563	2295	2292	2292
query10	415	204	198	198
query11	18057	15374	15300	15300
query12	130	69	65	65
query13	1711	367	372	367
query14	9111	6506	6506	6506
query15	203	178	174	174
query16	5699	255	250	250
query17	954	470	471	470
query18	1816	260	247	247
query19	175	136	133	133
query20	76	76	69	69
query21	201	138	128	128
query22	4611	4695	4681	4681
query23	31126	30438	30321	30321
query24	11903	2793	2771	2771
query25	590	302	305	302
query26	1801	139	134	134
query27	3194	285	283	283
query28	7766	1835	1820	1820
query29	1832	622	599	599
query30	276	137	137	137
query31	922	706	735	706
query32	76	49	51	49
query33	692	209	209	209
query34	1138	449	454	449
query35	885	780	771	771
query36	1307	1206	1146	1146
query37	95	61	59	59
query38	3275	3195	3154	3154
query39	1309	1264	1247	1247
query40	342	86	85	85
query41	38	36	35	35
query42	91	79	83	79
query43	514	502	478	478
query44	1093	686	686	686
query45	188	180	174	174
query46	1046	658	649	649
query47	1603	1562	1483	1483
query48	401	310	301	301
query49	1194	285	285	285
query50	696	315	304	304
query51	5305	5196	5174	5174
query52	84	85	73	73
query53	311	256	252	252
query54	249	176	195	176
query55	79	70	70	70
query56	166	157	157	157
query57	1010	915	921	915
query58	183	151	153	151
query59	2659	2513	2405	2405
query60	197	175	174	174
query61	82	83	81	81
query62	634	357	350	350
query63	265	270	268	268
query64	6032	1744	1759	1744
query65	3285	3226	3235	3226
query66	1244	319	314	314
query67	14349	14198	14287	14198
query68	3988	498	493	493
query69	428	297	293	293
query70	1447	1514	1523	1514
query71	257	211	205	205
query72	3775	2856	2799	2799
query73	697	310	305	305
query74	6755	6367	6346	6346
query75	2798	2329	2292	2292
query76	2611	1035	979	979
query77	347	233	226	226
query78	9059	8859	8605	8605
query79	2632	491	481	481
query80	1967	325	324	324
query81	515	199	195	195
query82	691	83	85	83
query83	238	117	116	116
query84	280	73	77	73
query85	2288	350	337	337
query86	519	404	367	367
query87	3541	3280	3281	3280
query88	4109	2154	2140	2140
query89	430	348	372	348
query90	1928	186	179	179
query91	165	115	117	115
query92	52	45	52	45
query93	4484	428	450	428
query94	1326	161	158	158
query95	499	455	443	443
query96	634	306	317	306
query97	4236	4122	4113	4113
query98	198	198	190	190
query99	1091	678	719	678
Total cold run time: 283247 ms
Total hot run time: 173064 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cbcb6998621b798b06ec737f0a296f186841b677, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.68	0.09	0.10
query5	0.52	0.51	0.52
query6	1.19	0.62	0.62
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.54	0.50	0.50
query10	0.55	0.57	0.54
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.60	0.60	0.60
query14	0.77	0.80	0.81
query15	0.79	0.77	0.80
query16	0.37	0.39	0.38
query17	1.01	0.99	1.03
query18	0.21	0.25	0.25
query19	1.79	1.80	1.78
query20	0.01	0.01	0.01
query21	15.41	0.58	0.59
query22	2.51	2.86	2.08
query23	17.36	0.83	0.84
query24	2.36	0.70	1.29
query25	0.26	0.31	0.17
query26	0.47	0.13	0.13
query27	0.05	0.04	0.05
query28	11.66	0.83	0.87
query29	12.56	3.17	3.24
query30	0.66	0.54	0.55
query31	2.78	0.36	0.35
query32	3.36	0.47	0.48
query33	3.22	3.18	3.20
query34	15.97	4.41	4.37
query35	4.43	4.31	4.24
query36	1.11	1.06	1.06
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.22 s
Total hot run time: 31.15 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit cbcb6998621b798b06ec737f0a296f186841b677 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.2 seconds inserted 10000000 Rows, about 704K ops/s

@sollhui sollhui force-pushed the stream_load_observability branch 2 times, most recently from 97f596a to caabe77 Compare January 27, 2024 09:23
@sollhui
Copy link
Contributor Author

sollhui commented Jan 27, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 37733 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b4e7848cc055858ccb3f0249195afba7a880276, data reload: false

------ Round 1 ----------------------------------
q1	17593	4802	5161	4802
q2	2060	160	141	141
q3	10565	945	949	945
q4	6413	787	813	787
q5	7759	2988	2988	2988
q6	186	122	122	122
q7	1135	750	743	743
q8	9285	2063	2075	2063
q9	7678	6519	6317	6317
q10	8114	2452	2403	2403
q11	416	198	216	198
q12	773	281	266	266
q13	17991	3288	3348	3288
q14	272	259	246	246
q15	519	491	498	491
q16	469	395	415	395
q17	941	509	498	498
q18	6963	6115	5971	5971
q19	1571	1439	1368	1368
q20	593	345	343	343
q21	7139	3054	3137	3054
q22	819	310	304	304
Total cold run time: 109254 ms
Total hot run time: 37733 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5010	4926	4812	4812
q2	345	231	226	226
q3	3027	2773	2657	2657
q4	1835	1648	1624	1624
q5	5408	5290	5338	5290
q6	192	116	115	115
q7	2116	1794	1849	1794
q8	3126	3215	3268	3215
q9	8601	8842	8508	8508
q10	8150	3463	3514	3463
q11	572	480	480	480
q12	757	664	573	573
q13	16974	3090	3070	3070
q14	280	239	279	239
q15	527	490	496	490
q16	528	457	463	457
q17	1904	1784	1661	1661
q18	8747	8519	10477	8519
q19	15949	1560	1564	1560
q20	4607	1918	1904	1904
q21	14504	4839	4703	4703
q22	995	507	482	482
Total cold run time: 104154 ms
Total hot run time: 55842 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8621/23823)
Line Coverage: 28.26% (70530/249532)
Region Coverage: 27.26% (36387/133495)
Branch Coverage: 24.06% (18651/77516)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5b4e7848cc055858ccb3f0249195afba7a880276_5b4e7848cc055858ccb3f0249195afba7a880276/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 172596 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b4e7848cc055858ccb3f0249195afba7a880276, data reload: false

query1	950	332	326	326
query2	6567	1858	1937	1858
query3	6696	198	192	192
query4	31159	22136	22098	22098
query5	4471	429	363	363
query6	253	152	149	149
query7	4614	259	251	251
query8	235	175	167	167
query9	8553	2272	2253	2253
query10	425	198	209	198
query11	17882	15513	15246	15246
query12	124	69	65	65
query13	1683	377	361	361
query14	9130	6402	6416	6402
query15	205	179	179	179
query16	5694	257	247	247
query17	953	474	459	459
query18	1804	260	253	253
query19	172	131	126	126
query20	74	72	70	70
query21	200	129	137	129
query22	4918	4707	4640	4640
query23	31205	30361	30313	30313
query24	12435	2728	2758	2728
query25	593	312	314	312
query26	1940	141	132	132
query27	3199	276	279	276
query28	7771	1831	1819	1819
query29	2010	624	606	606
query30	281	134	135	134
query31	901	722	748	722
query32	78	52	56	52
query33	699	205	203	203
query34	1177	456	464	456
query35	863	750	758	750
query36	1381	1213	1183	1183
query37	93	60	62	60
query38	3313	3157	3168	3157
query39	1289	1244	1233	1233
query40	345	86	80	80
query41	38	38	35	35
query42	87	83	81	81
query43	505	457	489	457
query44	1123	679	678	678
query45	192	174	172	172
query46	1046	642	655	642
query47	1585	1524	1492	1492
query48	379	319	304	304
query49	1210	282	275	275
query50	678	296	305	296
query51	5309	5133	5168	5133
query52	83	81	71	71
query53	316	251	261	251
query54	239	179	185	179
query55	79	82	76	76
query56	166	161	162	161
query57	1008	884	890	884
query58	174	153	153	153
query59	2734	2457	2451	2451
query60	201	170	170	170
query61	89	83	82	82
query62	641	336	372	336
query63	275	257	262	257
query64	5998	1775	1757	1757
query65	3294	3224	3225	3224
query66	1348	319	317	317
query67	14368	13997	14223	13997
query68	3907	479	489	479
query69	434	300	312	300
query70	1523	1576	1521	1521
query71	284	210	210	210
query72	3749	2850	2839	2839
query73	683	310	307	307
query74	6653	6229	6256	6229
query75	2771	2336	2291	2291
query76	2616	1026	1054	1026
query77	349	229	236	229
query78	9128	8683	8632	8632
query79	2085	501	487	487
query80	1326	316	310	310
query81	517	194	202	194
query82	679	82	82	82
query83	219	115	113	113
query84	279	72	73	72
query85	2175	341	316	316
query86	513	405	416	405
query87	3425	3333	3248	3248
query88	3926	2142	2152	2142
query89	400	355	339	339
query90	1924	189	182	182
query91	146	114	118	114
query92	55	42	41	41
query93	2193	416	443	416
query94	1342	165	165	165
query95	504	453	447	447
query96	625	306	311	306
query97	4268	4089	4161	4089
query98	209	197	190	190
query99	1003	696	732	696
Total cold run time: 279410 ms
Total hot run time: 172596 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b4e7848cc055858ccb3f0249195afba7a880276, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.02	0.02
query3	0.22	0.06	0.05
query4	1.67	0.10	0.09
query5	0.54	0.51	0.51
query6	1.19	0.64	0.65
query7	0.01	0.01	0.02
query8	0.04	0.02	0.02
query9	0.54	0.51	0.49
query10	0.55	0.55	0.54
query11	0.11	0.09	0.08
query12	0.11	0.09	0.09
query13	0.60	0.60	0.62
query14	0.81	0.79	0.78
query15	0.79	0.78	0.79
query16	0.38	0.39	0.39
query17	1.03	1.04	1.01
query18	0.22	0.26	0.26
query19	1.80	1.69	1.76
query20	0.02	0.01	0.01
query21	15.41	0.58	0.58
query22	2.45	2.19	1.58
query23	17.46	0.83	0.82
query24	2.44	0.78	1.29
query25	0.27	0.29	0.10
query26	0.56	0.14	0.14
query27	0.05	0.05	0.06
query28	11.13	0.85	0.84
query29	12.58	3.30	3.08
query30	0.68	0.55	0.55
query31	2.79	0.34	0.36
query32	3.36	0.48	0.49
query33	3.20	3.18	3.25
query34	15.76	4.30	4.31
query35	4.34	4.40	4.44
query36	1.08	1.06	1.07
query37	0.06	0.05	0.05
query38	0.03	0.02	0.03
query39	0.02	0.01	0.02
query40	0.16	0.12	0.15
query41	0.07	0.02	0.02
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.67 s
Total hot run time: 30.63 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5b4e7848cc055858ccb3f0249195afba7a880276 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

liaoxin01
liaoxin01 previously approved these changes Jan 30, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sollhui sollhui force-pushed the stream_load_observability branch from 5b4e784 to 424a208 Compare January 30, 2024 11:27
@sollhui sollhui force-pushed the stream_load_observability branch from 424a208 to 3c2af3a Compare January 30, 2024 11:30
@sollhui
Copy link
Contributor Author

sollhui commented Jan 30, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 37390 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd, data reload: false

------ Round 1 ----------------------------------
q1	17915	4604	4549	4549
q2	2551	142	143	142
q3	11211	947	955	947
q4	4947	855	747	747
q5	7917	2979	2904	2904
q6	192	122	125	122
q7	1180	725	714	714
q8	9294	2005	2177	2005
q9	7306	6333	6292	6292
q10	8093	2447	2444	2444
q11	419	212	205	205
q12	807	298	290	290
q13	17999	3321	3307	3307
q14	268	255	267	255
q15	517	482	486	482
q16	462	395	403	395
q17	940	544	523	523
q18	6850	6091	6026	6026
q19	1565	1361	1340	1340
q20	567	337	338	337
q21	6681	3071	3087	3071
q22	809	293	304	293
Total cold run time: 108490 ms
Total hot run time: 37390 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4493	4414	4425	4414
q2	318	229	220	220
q3	2984	2872	2844	2844
q4	1922	1708	1657	1657
q5	5185	5265	5289	5265
q6	194	114	113	113
q7	2168	1815	1781	1781
q8	3134	3233	3270	3233
q9	8394	8301	8299	8299
q10	5811	3535	3564	3535
q11	550	459	459	459
q12	757	563	569	563
q13	13882	3120	3085	3085
q14	289	248	260	248
q15	530	491	488	488
q16	499	468	457	457
q17	1855	1684	1692	1684
q18	7905	7794	7699	7699
q19	1593	1512	1555	1512
q20	2112	1947	1907	1907
q21	4857	4638	4505	4505
q22	577	486	469	469
Total cold run time: 70009 ms
Total hot run time: 54437 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.18% (8624/23835)
Line Coverage: 28.24% (70506/249695)
Region Coverage: 27.24% (36383/133580)
Branch Coverage: 24.02% (18631/77562)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd_3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 172986 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd, data reload: false

query1	928	338	319	319
query2	6566	2094	1927	1927
query3	6697	208	208	208
query4	28155	22107	22025	22025
query5	4452	355	359	355
query6	245	162	167	162
query7	4603	270	261	261
query8	245	186	195	186
query9	9128	2241	2231	2231
query10	404	199	203	199
query11	16701	15294	15293	15293
query12	122	67	60	60
query13	1668	370	370	370
query14	9351	6550	6427	6427
query15	229	178	186	178
query16	6525	254	253	253
query17	1903	483	462	462
query18	1935	248	248	248
query19	213	131	135	131
query20	72	71	69	69
query21	203	131	124	124
query22	5095	4949	4917	4917
query23	31279	30219	30412	30219
query24	12354	2719	2778	2719
query25	583	310	303	303
query26	1991	138	133	133
query27	2900	288	284	284
query28	7627	1837	1802	1802
query29	2169	626	604	604
query30	285	137	138	137
query31	890	707	712	707
query32	90	55	51	51
query33	707	205	207	205
query34	1067	449	464	449
query35	855	774	758	758
query36	1326	1167	1245	1167
query37	176	56	56	56
query38	3252	3161	3120	3120
query39	1306	1251	1252	1251
query40	325	79	81	79
query41	37	36	33	33
query42	86	84	83	83
query43	546	507	515	507
query44	1100	682	673	673
query45	188	179	173	173
query46	1039	631	625	625
query47	1547	1546	1498	1498
query48	386	304	303	303
query49	1269	281	297	281
query50	678	313	311	311
query51	5280	5122	5163	5122
query52	90	79	82	79
query53	332	251	269	251
query54	253	188	187	187
query55	81	76	74	74
query56	178	168	156	156
query57	1011	898	906	898
query58	178	152	153	152
query59	2356	2399	2402	2399
query60	196	179	178	178
query61	86	83	82	82
query62	629	362	343	343
query63	277	263	258	258
query64	5567	1760	1726	1726
query65	3282	3216	3189	3189
query66	1491	322	310	310
query67	14907	14347	14418	14347
query68	3951	497	492	492
query69	414	305	314	305
query70	1559	1565	1492	1492
query71	303	221	230	221
query72	3697	2836	2854	2836
query73	695	321	317	317
query74	6548	6152	6146	6146
query75	2819	2291	2290	2290
query76	2899	933	926	926
query77	322	232	228	228
query78	9059	8840	8593	8593
query79	2131	483	481	481
query80	1335	333	324	324
query81	532	199	195	195
query82	674	77	76	76
query83	235	121	111	111
query84	234	72	72	72
query85	1988	326	334	326
query86	494	386	358	358
query87	3388	3293	3243	3243
query88	3806	2265	2231	2231
query89	432	350	343	343
query90	1936	186	189	186
query91	152	117	116	116
query92	52	45	41	41
query93	1945	437	434	434
query94	1364	157	156	156
query95	490	466	452	452
query96	623	330	321	321
query97	4261	4127	4143	4127
query98	203	198	186	186
query99	948	674	677	674
Total cold run time: 277181 ms
Total hot run time: 172986 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.11
query5	0.54	0.51	0.52
query6	1.24	0.65	0.64
query7	0.02	0.01	0.02
query8	0.03	0.03	0.02
query9	0.55	0.51	0.50
query10	0.54	0.54	0.56
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.60	0.62	0.60
query14	0.79	0.80	0.79
query15	0.80	0.78	0.76
query16	0.39	0.37	0.39
query17	1.00	1.05	1.06
query18	0.20	0.28	0.23
query19	1.89	1.80	1.77
query20	0.01	0.01	0.01
query21	15.41	0.60	0.57
query22	2.43	2.10	1.55
query23	17.38	0.88	0.91
query24	2.23	1.08	0.46
query25	0.36	0.18	0.15
query26	0.39	0.13	0.14
query27	0.05	0.05	0.05
query28	12.16	0.84	0.82
query29	12.56	3.15	3.11
query30	0.62	0.55	0.53
query31	2.78	0.34	0.35
query32	3.37	0.48	0.48
query33	3.22	3.20	3.21
query34	15.74	4.29	4.38
query35	4.29	4.27	4.33
query36	1.10	1.05	1.05
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.16	0.12	0.13
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.3 s
Total hot run time: 30.31 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@sollhui sollhui changed the title [improve](stream-load) record time on receiving HTTP request [improve](stream-load) add observability on receiving HTTP request Feb 1, 2024
dataroaring
dataroaring approved these changes Feb 1, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 1, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 1, 2024

PR approved by at least one committer and no changes requested.

@yiguolei yiguolei merged commit d65d0c4 into apache:master Feb 1, 2024
30 of 32 checks passed
yiguolei pushed a commit that referenced this pull request Feb 1, 2024
@sollhui @Doris-Extras
[improve](stream-load) add observability on receiving HTTP request #…
822f2b1 
…30432
@sollhui sollhui mentioned this pull request Sep 23, 2024
Merged
dataroaring pushed a commit that referenced this pull request Sep 26, 2024
@sollhui
[improve](stream-load) add observability on receiving HTTP request (#…
9b14720 
…41136)

pick #30432 and
#40735

---------

Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liaoxin01 liaoxin01 liaoxin01 left review comments

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.16-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@sollhui @doris-robot @gavinchou @liaoxin01 @dataroaring @xiaokang @yiguolei