You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
tip: 298
title: Reformat manifest
author: halibobo1205@gmail.com
discussions to: https://github.com/tronprotocol/TIPs/issues/298
status: Last Call
type: Standards Track
category: Core
created: 2021-07-16
Simple Summary
This TIP describes the levelDB Manifest file reformatting function.
Abstract
The c++ version of levelDB has been running for a long time, the manifest file may grow too big, resulting in high memory and long start-up time when restarting the database, for this situation, use the java version to process the levelDB manifest file for regularization before starting the database.
Motivation
Optimize c++ version of levelDB startup, reduce memory usage and time.
Specification
Check manifest size.
Bulk loading of manifest using java version.
reopen with leveldbjni
Implementation
The c++ version of levelDB is running, with the Compaction process, files are added and deleted, the levelDB sst file keeps changing, the manifest will keep growing (leveldb only rewrites this file when opening the database, it will only append at runtime), when it grows to a certain size, the database will restart, it may encounter OOM and cannot open the database.
Check manifest size.
Bulk loading of manifest using java version.
reopen with leveldbjni
1.Check manifest size
// Read "CURRENT" file, which contains a pointer to the current manifest fileFilecurrentFile = newFile(dir, Filename.currentFileName());
if (!currentFile.exists()) {
returnfalse;
}
StringcurrentName = com.google.common.io.Files.asCharSource(currentFile, UTF_8).read();
if (currentName.isEmpty() || currentName.charAt(currentName.length() - 1) != '\n') {
returnfalse;
}
currentName = currentName.substring(0, currentName.length() - 1);
Filecurrent = newFile(dir, currentName);
if (!current.isFile()) {
returnfalse;
}
longmaxSize = options.maxManifestSize();
if (maxSize < 0) {
returnfalse;
}
logger.info("CurrentName {}/{},size {} kb.", dir, currentName, current.length() / 1024);
returncurrent.length() >= maxSize * 1024 * 1024;
2.Bulk loading of manifest using java version.
// open file channeltry (FileInputStreamfis = newFileInputStream(newFile(databaseDir, currentName));
FileChannelfileChannel = fis.getChannel()) {
// read log edit logLongnextFileNumber = null;
LonglastSequence = null;
LonglogNumber = null;
LongprevLogNumber = null;
Builderbuilder = newBuilder(this, current);
LogReaderreader = newLogReader(fileChannel, throwExceptionMonitor(), true, 0);
VersionEditedit;
for (Slicerecord = reader.readRecord(); record != null; record = reader.readRecord()) {
// read version editedit = newVersionEdit(record);
// verify comparator// todo implement user comparatorStringeditComparator = edit.getComparatorName();
StringuserComparator = internalKeyComparator.name();
checkArgument(editComparator == null || editComparator.equals(userComparator),
"Expected user comparator %s to match existing database comparator ", userComparator, editComparator);
// apply editbuilder.apply(edit);
if (builder.batchSize > options.maxBatchSize()) {
Versionversion = newVersion(this);
builder.saveTo(version);
// Install recovered versionfinalizeVersion(version);
appendVersion(version);
builder = newBuilder(this, current);
}
// save edit values for verification belowlogNumber = coalesce(edit.getLogNumber(), logNumber);
prevLogNumber = coalesce(edit.getPreviousLogNumber(), prevLogNumber);
nextFileNumber = coalesce(edit.getNextFileNumber(), nextFileNumber);
lastSequence = coalesce(edit.getLastSequenceNumber(), lastSequence);
}
/** * Apply the specified edit to the current state. */publicvoidapply(VersionEditedit)
{
// Update compaction pointersfor (Entry<Integer, InternalKey> entry : edit.getCompactPointers().entrySet()) {
Integerlevel = entry.getKey();
InternalKeyinternalKey = entry.getValue();
versionSet.compactPointers.put(level, internalKey);
}
// Delete filesfor (Entry<Integer, Long> entry : edit.getDeletedFiles().entries()) {
Integerlevel = entry.getKey();
LongfileNumber = entry.getValue();
if (!versionSet.options.fast()) {
levels.get(level).deletedFiles.add(fileNumber);
batchSize++;
}
// todo missing update to addedFiles?// missing update to addedFiles for open db to release resourceif (levels.get(level).addedFilesMap.remove(fileNumber) != null) {
batchSize--;
}
}
// Add new filesfor (Entry<Integer, FileMetaData> entry : edit.getNewFiles().entries()) {
Integerlevel = entry.getKey();
FileMetaDatafileMetaData = entry.getValue();
// We arrange to automatically compact this file after// a certain number of seeks. Let's assume:// (1) One seek costs 10ms// (2) Writing or reading 1MB costs 10ms (100MB/s)// (3) A compaction of 1MB does 25MB of IO:// 1MB read from this level// 10-12MB read from next level (boundaries may be misaligned)// 10-12MB written to next level// This implies that 25 seeks cost the same as the compaction// of 1MB of data. I.e., one seek costs approximately the// same as the compaction of 40KB of data. We are a little// conservative and allow approximately one seek for every 16KB// of data before triggering a compaction.intallowedSeeks = (int) (fileMetaData.getFileSize() / 16384);
if (allowedSeeks < 100) {
allowedSeeks = 100;
}
fileMetaData.setAllowedSeeks(allowedSeeks);
if (levels.get(level).deletedFiles.remove(fileMetaData.getNumber())) {
batchSize--;
}
//levels.get(level).addedFiles.add(fileMetaData);levels.get(level).addedFilesMap.put(fileMetaData.getNumber(), fileMetaData);
batchSize++;
}
}
3.When manifest is reach certain size ,open db with Bulk.
Simple Summary
This TIP describes the levelDB Manifest file reformatting function.
Abstract
The c++ version of levelDB has been running for a long time, the manifest file may grow too big, resulting in high memory and long start-up time when restarting the database, for this situation, use the java version to process the levelDB manifest file for regularization before starting the database.
Motivation
Optimize c++ version of levelDB startup, reduce memory usage and time.
Specification
Implementation
The c++ version of levelDB is running, with the Compaction process, files are added and deleted, the levelDB sst file keeps changing, the manifest will keep growing (leveldb only rewrites this file when opening the database, it will only append at runtime), when it grows to a certain size, the database will restart, it may encounter OOM and cannot open the database.
1.Check manifest size
2.Bulk loading of manifest using java version.
3.When manifest is reach certain size ,open db with Bulk.
Rationale
Huge manifest costs a lot.
The text was updated successfully, but these errors were encountered: