Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](backup) Read BackupMeta with the specified meta version #38370

Merged
merged 2 commits into from
Jul 25, 2024
Merged

[fix](backup) Read BackupMeta with the specified meta version #38370

merged 2 commits into from
Jul 25, 2024
+30 −35

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Jul 25, 2024

The serialized BackupMeta has a fixed meta version, and using the corresponding version for deserialization is essential to ensure metadata compatibility. This PR fixes the issue where BackupMeta was deserialized without setting the meta version.

@w41ter
[fix](backup) Read BackupMeta with the specified meta version
3c10ed8 
The serialized BackupMeta has a fixed meta version, and using the
corresponding version for deserialization is essential to ensure
metadata compatibility. This PR fixes the issue where BackupMeta
was deserialized without setting the meta version.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Jul 25, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39360 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c10ed89acbecf0bb8a1a07775075e360d25c921, data reload: false

------ Round 1 ----------------------------------
q1	17956	5124	4257	4257
q2	2018	196	198	196
q3	11194	1178	1177	1177
q4	10404	750	743	743
q5	7610	2707	2815	2707
q6	227	143	135	135
q7	967	599	619	599
q8	9423	1916	1954	1916
q9	8952	6676	6661	6661
q10	9065	3809	3755	3755
q11	485	240	242	240
q12	392	225	219	219
q13	18445	2955	2944	2944
q14	295	232	250	232
q15	527	477	488	477
q16	497	384	380	380
q17	961	645	623	623
q18	8200	7431	7437	7431
q19	3479	1063	970	970
q20	694	326	347	326
q21	4925	3095	3162	3095
q22	348	285	277	277
Total cold run time: 117064 ms
Total hot run time: 39360 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4376	4262	4213	4213
q2	388	270	269	269
q3	3012	2882	2883	2882
q4	1988	1722	1721	1721
q5	5554	5530	5473	5473
q6	224	129	130	129
q7	2208	1809	1866	1809
q8	3316	3464	3404	3404
q9	8821	8631	8695	8631
q10	4107	3939	3788	3788
q11	606	508	498	498
q12	815	644	653	644
q13	16573	3191	3188	3188
q14	313	298	302	298
q15	533	502	493	493
q16	498	436	435	435
q17	1839	1502	1507	1502
q18	8176	7955	7760	7760
q19	1687	1507	1469	1469
q20	2036	1822	1869	1822
q21	5114	4876	4875	4875
q22	601	505	501	501
Total cold run time: 72785 ms
Total hot run time: 55804 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173462 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c10ed89acbecf0bb8a1a07775075e360d25c921, data reload: false

query1	898	366	359	359
query2	6282	1859	1821	1821
query3	6654	205	213	205
query4	28328	17527	17432	17432
query5	3788	487	483	483
query6	274	172	172	172
query7	4594	291	281	281
query8	247	208	208	208
query9	8414	2394	2409	2394
query10	447	279	269	269
query11	11453	10033	10184	10033
query12	134	87	90	87
query13	1630	374	366	366
query14	10222	7530	7394	7394
query15	216	168	171	168
query16	7751	488	433	433
query17	1597	583	542	542
query18	1995	293	295	293
query19	193	148	143	143
query20	97	87	86	86
query21	213	98	100	98
query22	4409	4231	4096	4096
query23	34075	33833	33705	33705
query24	11869	2872	2965	2872
query25	682	416	425	416
query26	1769	160	151	151
query27	2928	276	281	276
query28	7740	2078	2071	2071
query29	1156	449	444	444
query30	264	154	157	154
query31	1010	766	806	766
query32	97	59	58	58
query33	804	340	350	340
query34	957	475	495	475
query35	878	781	778	778
query36	1157	974	999	974
query37	298	89	95	89
query38	3019	2880	2814	2814
query39	997	855	828	828
query40	240	119	111	111
query41	49	43	45	43
query42	119	94	100	94
query43	506	471	473	471
query44	1211	738	726	726
query45	212	181	178	178
query46	1126	725	744	725
query47	1857	1778	1782	1778
query48	370	296	285	285
query49	935	400	407	400
query50	807	398	400	398
query51	6674	6766	6627	6627
query52	100	88	85	85
query53	251	180	184	180
query54	898	437	446	437
query55	76	76	74	74
query56	287	274	269	269
query57	1141	1046	1038	1038
query58	262	269	266	266
query59	2801	2744	2616	2616
query60	310	309	301	301
query61	118	97	94	94
query62	795	656	652	652
query63	198	178	179	178
query64	10250	2297	1729	1729
query65	3181	3113	3114	3113
query66	1255	338	361	338
query67	15701	15137	15066	15066
query68	6920	554	552	552
query69	665	401	327	327
query70	1276	1159	1084	1084
query71	535	274	271	271
query72	8984	5392	5361	5361
query73	773	326	325	325
query74	6068	5743	5661	5661
query75	5258	2660	2668	2660
query76	5025	969	946	946
query77	779	317	298	298
query78	11136	9011	9264	9011
query79	8624	522	541	522
query80	981	484	488	484
query81	586	221	217	217
query82	1233	141	133	133
query83	338	173	174	173
query84	270	77	78	77
query85	1367	361	300	300
query86	404	314	311	311
query87	3288	3161	3107	3107
query88	4849	2412	2377	2377
query89	514	291	290	290
query90	2109	190	192	190
query91	136	103	97	97
query92	59	50	52	50
query93	6690	555	537	537
query94	767	291	276	276
query95	342	251	256	251
query96	610	269	275	269
query97	3211	3039	3019	3019
query98	214	207	191	191
query99	1524	1249	1256	1249
Total cold run time: 306801 ms
Total hot run time: 173462 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3c10ed89acbecf0bb8a1a07775075e360d25c921, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.04
query4	1.68	0.07	0.07
query5	0.50	0.47	0.47
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.53	0.49	0.49
query10	0.54	0.54	0.55
query11	0.16	0.12	0.12
query12	0.15	0.13	0.13
query13	0.60	0.57	0.58
query14	0.75	0.76	0.77
query15	0.84	0.83	0.81
query16	0.36	0.37	0.35
query17	1.02	1.06	0.98
query18	0.22	0.21	0.23
query19	1.89	1.72	1.72
query20	0.01	0.01	0.01
query21	15.40	0.77	0.67
query22	4.67	6.00	2.16
query23	18.32	1.34	1.17
query24	2.23	0.22	0.22
query25	0.16	0.08	0.09
query26	0.29	0.20	0.20
query27	0.45	0.23	0.24
query28	13.21	1.02	1.00
query29	12.58	3.27	3.23
query30	0.25	0.06	0.05
query31	2.90	0.39	0.39
query32	3.23	0.47	0.47
query33	2.90	2.90	2.94
query34	17.04	4.35	4.34
query35	4.41	4.41	4.41
query36	0.65	0.46	0.49
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.03	0.04
Total cold run time: 110.21 s
Total hot run time: 30.67 s

dataroaring
dataroaring approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 25, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@w41ter
Copy link
Contributor Author

w41ter commented Jul 25, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39264 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8d204a5084d367aa4cc4078c0d810e83f8ca7052, data reload: false

------ Round 1 ----------------------------------
q1	17611	4318	4246	4246
q2	2020	200	202	200
q3	10424	1216	1117	1117
q4	10133	705	680	680
q5	7538	2704	2753	2704
q6	224	139	139	139
q7	943	594	589	589
q8	9222	1897	1904	1897
q9	8608	6529	6510	6510
q10	8834	3742	3791	3742
q11	459	249	250	249
q12	507	233	232	232
q13	18736	3019	3001	3001
q14	286	246	244	244
q15	517	475	493	475
q16	521	379	381	379
q17	979	633	618	618
q18	7976	7412	7430	7412
q19	5489	1020	1011	1011
q20	679	348	337	337
q21	5132	3198	3242	3198
q22	355	290	284	284
Total cold run time: 117193 ms
Total hot run time: 39264 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4482	4257	4280	4257
q2	380	291	281	281
q3	2969	2800	2918	2800
q4	1999	1736	1721	1721
q5	5688	5517	5517	5517
q6	224	141	132	132
q7	2169	1824	1830	1824
q8	3268	3399	3416	3399
q9	8792	8745	8852	8745
q10	4049	3880	3837	3837
q11	577	502	504	502
q12	782	638	638	638
q13	16095	3170	3142	3142
q14	325	286	300	286
q15	522	506	482	482
q16	516	450	433	433
q17	1805	1518	1533	1518
q18	8100	7829	7884	7829
q19	1718	1545	1580	1545
q20	2136	1909	1887	1887
q21	5120	5018	4844	4844
q22	573	523	525	523
Total cold run time: 72289 ms
Total hot run time: 56142 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172635 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8d204a5084d367aa4cc4078c0d810e83f8ca7052, data reload: false

query1	904	380	378	378
query2	6434	1866	1813	1813
query3	6638	208	232	208
query4	28389	17509	17152	17152
query5	3618	512	509	509
query6	280	208	161	161
query7	4585	298	293	293
query8	251	198	190	190
query9	8517	2457	2413	2413
query10	438	281	270	270
query11	11695	10053	10110	10053
query12	118	89	85	85
query13	1625	371	361	361
query14	10627	6959	7872	6959
query15	217	163	165	163
query16	7685	501	447	447
query17	1572	560	524	524
query18	1951	296	275	275
query19	184	145	140	140
query20	94	84	81	81
query21	213	97	102	97
query22	4564	4086	4082	4082
query23	34003	33611	33585	33585
query24	10941	2886	2944	2886
query25	631	401	372	372
query26	799	157	162	157
query27	2222	282	286	282
query28	7073	2089	2081	2081
query29	818	446	417	417
query30	248	154	158	154
query31	967	754	780	754
query32	100	53	55	53
query33	758	359	338	338
query34	886	504	500	500
query35	878	794	757	757
query36	1156	978	972	972
query37	142	83	85	83
query38	2956	2845	2793	2793
query39	927	856	832	832
query40	201	118	113	113
query41	45	40	42	40
query42	118	102	100	100
query43	528	470	464	464
query44	1256	728	751	728
query45	215	175	174	174
query46	1078	731	757	731
query47	1849	1769	1767	1767
query48	360	292	302	292
query49	829	441	428	428
query50	791	402	414	402
query51	6771	6704	6711	6704
query52	101	91	92	91
query53	260	181	180	180
query54	869	441	446	441
query55	75	77	80	77
query56	344	320	288	288
query57	1124	1040	1062	1040
query58	270	278	267	267
query59	2881	2661	2625	2625
query60	307	283	287	283
query61	98	94	95	94
query62	779	649	655	649
query63	215	187	178	178
query64	9187	2227	1775	1775
query65	3155	3098	3077	3077
query66	768	347	344	344
query67	15355	15030	14990	14990
query68	4549	564	563	563
query69	729	407	318	318
query70	1158	1145	1163	1145
query71	447	281	281	281
query72	7471	5343	5927	5343
query73	778	332	330	330
query74	6149	5656	5561	5561
query75	3727	2696	2664	2664
query76	2850	935	906	906
query77	675	331	303	303
query78	11354	9304	9044	9044
query79	4097	541	524	524
query80	1632	521	506	506
query81	599	215	221	215
query82	875	146	134	134
query83	344	182	182	182
query84	269	80	88	80
query85	1198	321	303	303
query86	471	327	320	320
query87	3297	3150	3077	3077
query88	4247	2391	2397	2391
query89	421	282	295	282
query90	1900	199	192	192
query91	195	103	102	102
query92	64	53	48	48
query93	4796	569	546	546
query94	941	287	297	287
query95	350	261	268	261
query96	618	274	275	274
query97	3214	3040	2994	2994
query98	222	198	200	198
query99	1594	1249	1275	1249
Total cold run time: 287164 ms
Total hot run time: 172635 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8d204a5084d367aa4cc4078c0d810e83f8ca7052, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.04
query4	1.68	0.08	0.08
query5	0.49	0.49	0.48
query6	1.12	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.48
query10	0.55	0.55	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.13
query13	0.60	0.59	0.58
query14	0.75	0.79	0.79
query15	0.87	0.82	0.82
query16	0.36	0.36	0.37
query17	1.04	1.04	1.03
query18	0.23	0.22	0.23
query19	1.85	1.78	1.67
query20	0.02	0.01	0.03
query21	15.40	0.76	0.66
query22	4.03	6.89	2.62
query23	18.33	1.39	1.28
query24	2.09	0.24	0.23
query25	0.15	0.09	0.09
query26	0.30	0.22	0.21
query27	0.46	0.23	0.23
query28	13.30	1.03	1.01
query29	12.63	3.35	3.32
query30	0.25	0.06	0.05
query31	2.86	0.41	0.38
query32	3.28	0.47	0.47
query33	2.85	2.93	2.95
query34	17.12	4.34	4.38
query35	4.41	4.43	4.53
query36	0.65	0.45	0.47
query37	0.18	0.15	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.16	0.12	0.13
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.64 s
Total hot run time: 31.44 s

@gavinchou gavinchou merged commit f898cfe into apache:master Jul 25, 2024
27 of 28 checks passed
@w41ter w41ter deleted the fix_deserialize_backup_meta_without_version branch July 26, 2024 02:18
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 26, 2024
@w41ter
[fix](backup) Read BackupMeta with the specified meta version
3b39646 
Cherry-pick apache#38370
@w41ter w41ter mentioned this pull request Jul 26, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 26, 2024
@w41ter
[fix](backup) Read BackupMeta with the specified meta version
d431b01 
Cherry-pick apache#38370
@w41ter w41ter mentioned this pull request Jul 26, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 26, 2024
@w41ter
[fix](backup) Read BackupMeta with the specified meta version
68e847e 
Cherry-pick apache#38370
w41ter added a commit that referenced this pull request Jul 26, 2024
@w41ter
[fix](backup) Read BackupMeta with the specified meta version (#38396)
df96db3 
Cherry-pick #38370
w41ter added a commit that referenced this pull request Jul 26, 2024
@w41ter
[fix](backup) Read BackupMeta with the specified meta version (#38397)
fe2fdf0 
Cherry-pick #38370
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@w41ter @dataroaring
[fix](backup) Read BackupMeta with the specified meta version (#38370)
4bb9c26 
The serialized BackupMeta has a fixed meta version, and using the
corresponding version for deserialization is essential to ensure
metadata compatibility. This PR fixes the issue where BackupMeta was
deserialized without setting the meta version.
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@w41ter @dataroaring
[fix](backup) Read BackupMeta with the specified meta version (#38370)
093f843 
The serialized BackupMeta has a fixed meta version, and using the
corresponding version for deserialization is essential to ensure
metadata compatibility. This PR fixes the issue where BackupMeta was
deserialized without setting the meta version.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@w41ter @Baymine
[fix](backup) Read BackupMeta with the specified meta version (apache…
25741ed 
…#38397)

Cherry-pick apache#38370
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@w41ter @doris-robot @gavinchou @dataroaring