design consideration

we can divide spaces by two kinds: public vs private. Public spaces are accessed by lots of users and usually partitioned for performance or capacity. A private space is owned and accessed exclusively by a single user so does not needed to be partitioned.

Note it is a bad idea that a private space corresponds to one raft replication state machine - too much overhead (heartbeats, goroutines, et al). And there are two solution candidates:

1, multiple private spaces are assigned to one replication group in the current shared-nothing architecture
2, separation of storage/compute, no replication at the compute layer, multiple spaces hosted by a server, #749

Goal:

1. separated data space per tenant
2. scale to millions of tenants in a cluster of hundreds of servers, with tens of thousands of active tenants per server
3. inactive tenants should be offloaded to release the RAM resource and will be loaded on demand

Goal:

1. separated data space per tenant

This is a common use case for RocksDB column families. They can be created and removed on demand and they avoid adding global indexing overheads to separate column families.

2. scale to millions of tenants in a cluster of hundreds of servers, with tens of thousands of active tenants per server

3. inactive tenants should be offloaded to release the RAM resource and will be loaded on demand

RocksDB should be able to manage this transparently during regular compaction