What type of enhancement is this?

Refactor, Tech debt reduction

What does the enhancement do?

For historical reasons, both frontend and table engines (run in datanode) are aware of multiple regions in a table. That makes implementing a distributed table with multi-regions support complicated. We need to deal table and its regions on both frontend and datanode. We also need to implement procedures for the mito engine to ensure the eventual consistency of altering a table.

We decide to turn the table engine into a region engine, which only has knowledge of regions.

• The TableEngine trait becomes RegionEngine and only deals with regions.
• The datanode acts like a RegionServer. It manages regions assigned to the node and provides region-level access.
• The frontend provides table-level access. It partitions requests and routes them to regions on datanodes.
• We can combine the StorageEngine with the mito table engine.
• Since we already store metadata of the table in metasrv, there is no need to persist them again to the table manifest. We can remove the table manifest.
• Altering multiple regions is tracked by the procedure inside metasrv. We don't need procedures for the mito engine.

Steps

We are going to create a new fork of mito crate (maybe mito2).

• A new crate mito2 chore: initialize mito2 crate #1875
• RFC docs(rfcs): Add table engine refactor RFC #1899
• Use TableInfo to replace RegionMetadata (We might still reuse the RegionSchema)
• Implement MitoRegion or MitoRegionInfo to hold metadata feat(mito): Define Version and metadata builders for mito2 #1989
• Create region feat(mito): create region in mito2 engine #1999
• Open region feat(mito): Implement open and close for mito2 regions #2052
• Close region feat(mito): Implement open and close for mito2 regions #2052
• Wal feat(mito): Implement mito2 Wal #2103
• Write region feat(mito): Write wal and memtable #2135
  • chore: write to mito2 #2250
• Delete rows feat(mito): Support deleting rows in mito2 #2275
• New memtable feat(mito2): time series memtable #2208
• Scan region
  • New Batch feat(mito): Defines the read Batch struct for mito2 #2174
  • Merge reader feat(mito): merge reader for mito2 #2210
  • Skeleton feat(mito): Skeleton for scanning a region #2230
• Flush region
  • Framework feat(mito): Flush framework for mito2 #2262
  • Write buffer manager feat: Impl write buffer manager for mito2 #2309
  • Write stall feat(mito): Stall write requests and add more flush tests #2322
• Alter region
  • Compat reader feat: Implements a reader to make schema compatible #2326
  • Skeleton feat(mito): Implement skeleton for alteration #2343
• Compact region
  • Skeleton feat(mito2): compaction #2317
  • Scheduler feat(mito): Implements compaction scheduler #2413
• region options feat(mito): make use of options in RegionCreate/OpenRequest #2436
• Combine RegionManifest and StorageManifest refactor(mito2): implement RegionManifestManager #1984
• Remove the Region trait and forward requests to the engine directly
• Refactor file table engine if necessary feat: migrate file engine from table to reigon #2365
• Remove TableEngineProcedure trait
• Remove procedure implementations in mito and file engine
• Replace mito with mito2 (rename mito2 to mito)
• Rename TableEngine to RegionEngine
• Remove the old StorageEngine and related traits

Other Enhancements

There are also some other enhancements we could achieve during refactoring mito. They may not be strictly related to this issue's topic.

• Remove dependency of log sequence from RegionMetaAction::Change
• Decouple LogStore index and engine sequence (switch to sequence per key mode)
• Remove Option from time range in FileMeta
• Implement a new RegionWriter that supports batching (group commit)
• Add column family support? We plan to remove column family