Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TIP-298: Reformat manifest #298

Closed
halibobo1205 opened this issue Jul 16, 2021 · 0 comments
Closed

TIP-298: Reformat manifest #298

halibobo1205 opened this issue Jul 16, 2021 · 0 comments

Comments

@halibobo1205
Copy link
Contributor

halibobo1205 commented Jul 16, 2021

tip: 298
title: Reformat manifest	
author: halibobo1205@gmail.com
discussions to: https://github.com/tronprotocol/TIPs/issues/298
status: Last Call
type: Standards Track
category: Core
created: 2021-07-16

Simple Summary

This TIP describes the levelDB Manifest file reformatting function.

Abstract

The c++ version of levelDB has been running for a long time, the manifest file may grow too big, resulting in high memory and long start-up time when restarting the database, for this situation, use the java version to process the levelDB manifest file for regularization before starting the database.

Motivation

Optimize c++ version of levelDB startup, reduce memory usage and time.

Specification

  1. Check manifest size.
  2. Bulk loading of manifest using java version.
  3. reopen with leveldbjni

Implementation

The c++ version of levelDB is running, with the Compaction process, files are added and deleted, the levelDB sst file keeps changing, the manifest will keep growing (leveldb only rewrites this file when opening the database, it will only append at runtime), when it grows to a certain size, the database will restart, it may encounter OOM and cannot open the database.

  1. Check manifest size.
  2. Bulk loading of manifest using java version.
  3. reopen with leveldbjni

1.Check manifest size

// Read "CURRENT" file, which contains a pointer to the current manifest file
    File currentFile = new File(dir, Filename.currentFileName());
    if (!currentFile.exists()) {
      return false;
    }
    String currentName = com.google.common.io.Files.asCharSource(currentFile, UTF_8).read();
    if (currentName.isEmpty() || currentName.charAt(currentName.length() - 1) != '\n') {
      return false;
    }
    currentName = currentName.substring(0, currentName.length() - 1);
    File current = new File(dir, currentName);
    if (!current.isFile()) {
      return false;
    }
    long maxSize = options.maxManifestSize();
    if (maxSize < 0) {
      return false;
    }
    logger.info("CurrentName {}/{},size {} kb.", dir, currentName, current.length() / 1024);
    return current.length() >= maxSize * 1024 * 1024;

2.Bulk loading of manifest using java version.

        // open file channel
        try (FileInputStream fis = new FileInputStream(new File(databaseDir, currentName));
             FileChannel fileChannel = fis.getChannel()) {
            // read log edit log
            Long nextFileNumber = null;
            Long lastSequence = null;
            Long logNumber = null;
            Long prevLogNumber = null;
            Builder builder = new Builder(this, current);

            LogReader reader = new LogReader(fileChannel, throwExceptionMonitor(), true, 0);
            VersionEdit edit;
            for (Slice record = reader.readRecord(); record != null; record = reader.readRecord()) {
                // read version edit
                edit = new VersionEdit(record);

                // verify comparator
                // todo implement user comparator
                String editComparator = edit.getComparatorName();
                String userComparator = internalKeyComparator.name();
                checkArgument(editComparator == null || editComparator.equals(userComparator),
                    "Expected user comparator %s to match existing database comparator ", userComparator, editComparator);

                // apply edit
                builder.apply(edit);
                if (builder.batchSize > options.maxBatchSize()) {
                    Version version = new Version(this);
                    builder.saveTo(version);

                    // Install recovered version
                    finalizeVersion(version);

                    appendVersion(version);
                    builder = new Builder(this, current);
                }
                // save edit values for verification below
                logNumber = coalesce(edit.getLogNumber(), logNumber);
                prevLogNumber = coalesce(edit.getPreviousLogNumber(), prevLogNumber);
                nextFileNumber = coalesce(edit.getNextFileNumber(), nextFileNumber);
                lastSequence = coalesce(edit.getLastSequenceNumber(), lastSequence);
            }

/**
 * Apply the specified edit to the current state.
 */
public void apply(VersionEdit edit)
    {
    // Update compaction pointers
    for (Entry<Integer, InternalKey> entry : edit.getCompactPointers().entrySet()) {
    Integer level = entry.getKey();
    InternalKey internalKey = entry.getValue();
    versionSet.compactPointers.put(level, internalKey);
    }

    // Delete files
    for (Entry<Integer, Long> entry : edit.getDeletedFiles().entries()) {
    Integer level = entry.getKey();
    Long fileNumber = entry.getValue();
    if (!versionSet.options.fast()) {
    levels.get(level).deletedFiles.add(fileNumber);
    batchSize++;
    }

    // todo missing update to addedFiles?
    // missing update to addedFiles for open db to release resource
    if (levels.get(level).addedFilesMap.remove(fileNumber) != null) {
    batchSize--;
    }
    }

    // Add new files
    for (Entry<Integer, FileMetaData> entry : edit.getNewFiles().entries()) {
    Integer level = entry.getKey();
    FileMetaData fileMetaData = entry.getValue();

    // We arrange to automatically compact this file after
    // a certain number of seeks.  Let's assume:
    //   (1) One seek costs 10ms
    //   (2) Writing or reading 1MB costs 10ms (100MB/s)
    //   (3) A compaction of 1MB does 25MB of IO:
    //         1MB read from this level
    //         10-12MB read from next level (boundaries may be misaligned)
    //         10-12MB written to next level
    // This implies that 25 seeks cost the same as the compaction
    // of 1MB of data.  I.e., one seek costs approximately the
    // same as the compaction of 40KB of data.  We are a little
    // conservative and allow approximately one seek for every 16KB
    // of data before triggering a compaction.
    int allowedSeeks = (int) (fileMetaData.getFileSize() / 16384);
    if (allowedSeeks < 100) {
    allowedSeeks = 100;
    }
    fileMetaData.setAllowedSeeks(allowedSeeks);

    if (levels.get(level).deletedFiles.remove(fileMetaData.getNumber())) {
    batchSize--;
    }
    //levels.get(level).addedFiles.add(fileMetaData);
    levels.get(level).addedFilesMap.put(fileMetaData.getNumber(), fileMetaData);
    batchSize++;
    }
    }           

3.When manifest is reach certain size ,open db with Bulk.

  public void open() throws IOException {
    DB database = factory.open(this.srcDbPath.toFile(), this.options);
    database.close();
    }

Rationale

Huge manifest costs a lot.

@halibobo1205 halibobo1205 changed the title TIP-296: Reformat manifest TIP-298: Reformat manifest Jul 16, 2021
@lvs007 lvs007 closed this as completed Jun 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants