Add Namespaces management #989

Torch3333 · 2023-08-03T01:01:20Z

Context
ScalarDB did not manage namespaces so we could not retrieve the names of existing namespaces. We could only retrieve the tables of a given namespace.

Changes
ScalarDB is now managing namespace metadata in addition to the table metadata. At the moment, the namespace metadata is only the namespace name. With the Admin API admin.getNamespaceNames(), the list of namespaces names created through ScalarDB can be retrieved.
To manage the created namespace metadata, a new table called namespace is added to the same namespace which contains the metadata table.

The namespace name containing namespace and table metadata can be set via the configuration :

Cassandra : created the new scalar.db.cassandra.metadata.keyspace configuration.
CosmosDB: we deprecated the current scalar.db.cosmos.table_metadata.database to rename it to scalar.db.cosmos.metadata.database
DynamoDB: we deprecated the existing scalar.db.dynamo.table_metadata.namespace to rename it to scalar.db.dynamo.metadata.namespace
JDBC: we deprecated the existing scalar.db.jdbc.table_metadata.schema to rename it to scalar.db.jdbc.metadata.schema

Future related changes
We will also add two related features to the Admin API and the Schema Loader tool :

upgrade allows the users to update their existing environment to be compatible with the new Scalar DB version using the namespace table
repair the namespace allows the users to repair the namespaces metadata. If for some reason, a namespace exists in the underlying storage, but its metadata is missing from the namespaces metadata managed by ScalarDB (because the namespaces table was deleted or its data is corrupted), this method will re-add the namespace to the ScalarDB namespaces metadata.

…mespaces_management # Conflicts: # core/src/integration-test/java/com/scalar/db/storage/cosmos/CosmosAdminTestUtils.java # core/src/integration-test/java/com/scalar/db/storage/dynamo/ConsensusCommitAdminIntegrationTestWithDynamo.java # core/src/integration-test/java/com/scalar/db/storage/dynamo/DynamoAdminIntegrationTest.java # core/src/integration-test/java/com/scalar/db/storage/dynamo/DynamoAdminTestUtils.java # server/src/integration-test/java/com/scalar/db/server/ServerAdminTestUtils.java

…mespaces_management

…mespaces_management # Conflicts: # core/src/main/java/com/scalar/db/storage/cosmos/CosmosConfig.java

…inator namespace name (#732)

…mespace_management_merge_master # Conflicts: # core/src/integration-test/java/com/scalar/db/storage/dynamo/ConsensusCommitAdminIntegrationTestWithDynamo.java # core/src/integration-test/java/com/scalar/db/storage/dynamo/DynamoAdminIntegrationTest.java # core/src/integration-test/java/com/scalar/db/storage/jdbc/ConsensusCommitAdminIntegrationTestWithJdbcDatabase.java # core/src/integration-test/java/com/scalar/db/storage/jdbc/JdbcAdminIntegrationTest.java # core/src/integration-test/java/com/scalar/db/storage/jdbc/JdbcAdminTestUtils.java # core/src/integration-test/java/com/scalar/db/storage/multistorage/MultiStorageAdminTestUtils.java # core/src/integration-test/java/com/scalar/db/transaction/jdbc/JdbcTransactionAdminIntegrationTest.java # core/src/main/java/com/scalar/db/api/Admin.java # core/src/main/java/com/scalar/db/storage/cosmos/CosmosAdmin.java # core/src/main/java/com/scalar/db/storage/dynamo/DynamoAdmin.java # core/src/main/java/com/scalar/db/storage/jdbc/JdbcAdmin.java # core/src/test/java/com/scalar/db/storage/cosmos/CosmosAdminTestBase.java # core/src/test/java/com/scalar/db/storage/dynamo/DynamoAdminTestBase.java # core/src/test/java/com/scalar/db/storage/dynamo/DynamoConfigTest.java # core/src/test/java/com/scalar/db/storage/jdbc/JdbcAdminTestBase.java # core/src/test/java/com/scalar/db/storage/jdbc/JdbcConfigTest.java # integration-test/src/main/java/com/scalar/db/api/DistributedStorageAdminIntegrationTestBase.java # integration-test/src/main/java/com/scalar/db/api/DistributedTransactionAdminIntegrationTestBase.java # schema-loader/src/test/java/com/scalar/db/schemaloader/SchemaOperatorTest.java # server/src/integration-test/java/com/scalar/db/server/ConsensusCommitAdminIntegrationTestWithServer.java # server/src/integration-test/java/com/scalar/db/server/DistributedStorageAdminIntegrationTestWithServer.java # server/src/integration-test/java/com/scalar/db/server/ServerAdminTestUtils.java

…mespace_management_merge_master # Conflicts: # core/src/main/java/com/scalar/db/api/Admin.java # core/src/main/java/com/scalar/db/common/CheckedDistributedStorageAdmin.java # core/src/main/java/com/scalar/db/service/AdminService.java # core/src/main/java/com/scalar/db/storage/cosmos/CosmosAdmin.java # core/src/main/java/com/scalar/db/storage/dynamo/DynamoAdmin.java # core/src/main/java/com/scalar/db/storage/jdbc/JdbcAdmin.java # core/src/main/java/com/scalar/db/storage/multistorage/MultiStorageAdmin.java # core/src/main/java/com/scalar/db/transaction/consensuscommit/ConsensusCommitAdmin.java # core/src/main/java/com/scalar/db/transaction/rpc/GrpcTransactionAdmin.java # core/src/test/java/com/scalar/db/storage/cassandra/CassandraAdminTestBase.java # core/src/test/java/com/scalar/db/storage/cosmos/CosmosAdminTestBase.java # core/src/test/java/com/scalar/db/storage/dynamo/DynamoAdminTestBase.java # core/src/test/java/com/scalar/db/storage/jdbc/JdbcAdminTestBase.java # core/src/test/java/com/scalar/db/storage/rpc/GrpcAdminTest.java # core/src/test/java/com/scalar/db/transaction/consensuscommit/ConsensusCommitAdminTestBase.java # core/src/test/java/com/scalar/db/transaction/jdbc/JdbcTransactionAdminTest.java # rpc/src/main/java/com/scalar/db/rpc/ScalarDbProto.java # schema-loader/src/main/java/com/scalar/db/schemaloader/command/SchemaLoaderCommand.java # schema-loader/src/test/java/com/scalar/db/schemaloader/SchemaLoaderTest.java # schema-loader/src/test/java/com/scalar/db/schemaloader/SchemaOperatorTest.java # schema-loader/src/test/java/com/scalar/db/schemaloader/command/SchemaLoaderCommandTest.java

…nd admin.upgrade() is executed on ScalarDB server

…mespaceNames_on_ScalarDB_server' into feat/namespaces_management

Torch3333 · 2023-08-07T01:10:10Z

core/src/main/java/com/scalar/db/api/Admin.java

+
+  /**
+   * Returns the names of the existing namespaces created through Scalar DB.
+   *
+   * @return a set of namespaces names, an empty set if no namespaces exist
+   * @throws ExecutionException if the operation fails
+   */
+  Set<String> getNamespaceNames() throws ExecutionException;


The new admin.getNamespaceNames() interface.

Torch3333 · 2023-08-07T01:25:48Z

docs/api-guide.md

+### Get the tables of a namespace
+
+You can get the tables of a namespace as follows:
+
+```java
+// Get the tables of the namespace "ns"
+Set<String> tables = admin.getNamespaceTableNames("ns");
+```


Add the doc for admin.getNamespaceTableNames("ns") since it was missing.

Torch3333 · 2023-08-07T01:26:36Z

core/src/main/java/com/scalar/db/common/CheckedDistributedStorageAdmin.java

-  /**
-   * Whether to check if the namespace exists or not. Set false when the storage does not support
-   * namespaces.
-   */
-  private final boolean checkNamespace;


All storages support namespace now so it is not needed anymore.

Torch3333 · 2023-08-07T01:28:21Z

core/src/test/java/com/scalar/db/storage/cassandra/CassandraAdminTestBase.java

+/**
+ * Abstraction that defines unit tests for the {@link CassandraAdmin}. The class purpose is to be
+ * able to run the {@link CassandraAdmin} unit tests with different values for the {@link
+ * CassandraAdmin}, notably {@link CassandraAdmin#METADATA_KEYSPACE}.
+ */
+public abstract class CassandraAdminTestBase {


Abstracted the unit test for Cassandra to be able to test with different values for the metadata keyspace.

Torch3333 · 2023-08-07T01:29:36Z

docs/api-guide.md

+### Get the namespaces
+
+You can get the existing namespaces as follows:
+
+```java
+Set<String> namespaces = admin.getNamespaceNames();
+```


Added the new API to the documentation.

Torch3333 · 2023-08-07T01:30:10Z

docs/configurations.md

@@ -49,34 +49,35 @@ The following describes the configurations available for each storage.

 - For Cassandra, the following configurations are available:



Updated the documentation for the configuration

Torch3333 · 2023-08-07T01:47:36Z

core/src/main/java/com/scalar/db/storage/cosmos/CosmosAdmin.java

  public boolean namespaceExists(String namespace) throws ExecutionException {
-    return databaseExists(namespace);
+    try {
+      getNamespacesContainer()
+          .readItem(namespace, new PartitionKey(namespace), CosmosNamespace.class);
+      return true;
+    } catch (RuntimeException e) {
+      if (e instanceof CosmosException
+          && ((CosmosException) e).getStatusCode() == CosmosErrorCode.NOT_FOUND.get()) {
+        return false;
+      }
+      throw new ExecutionException("Checking if the namespace exists failed", e);
+    }
  }


For each admin, the implementation of admin.namespaceExists() is now uniformized and checks if the given namespace is contained in the namespace metadata table.

core/src/main/java/com/scalar/db/storage/jdbc/JdbcAdmin.java

brfrn169

Left several comments. Actually, I haven't finished this review yet. I'll continue with it later. Thanks.

core/src/integration-test/java/com/scalar/db/storage/cosmos/CosmosAdminTestUtils.java

...ntegration-test/java/com/scalar/db/transaction/jdbc/JdbcTransactionAdminIntegrationTest.java

core/src/main/java/com/scalar/db/storage/cassandra/CassandraAdmin.java

core/src/main/java/com/scalar/db/storage/cosmos/CosmosAdmin.java

core/src/main/java/com/scalar/db/storage/multistorage/MultiStorageAdmin.java

brfrn169

Left some additional comments. Please take a look when you have time!

docs/schema-loader.md

schema-loader/src/test/java/com/scalar/db/schemaloader/SchemaOperatorTest.java

…ces_management # Conflicts: # core/src/main/java/com/scalar/db/storage/cassandra/CassandraAdmin.java # core/src/main/java/com/scalar/db/storage/cosmos/CosmosAdmin.java # core/src/main/java/com/scalar/db/storage/dynamo/DynamoAdmin.java # core/src/main/java/com/scalar/db/storage/jdbc/JdbcAdmin.java

brfrn169

Left a comment. But LGTM, Thank you!

brfrn169 · 2023-08-15T04:53:55Z

core/src/main/java/com/scalar/db/storage/dynamo/DynamoAdmin.java

+  }
+
+  private void dropNamespacesTableIfEmpty() throws ExecutionException {
+    String NamespaceTableFullName =


Ditto:

Suggested change

String NamespaceTableFullName =

String namespaceTableFullName =

Thank you, I revised it in 38bd49d

feeblefakie

Overall, looking good! Thank you!
Left one question.

feeblefakie · 2023-08-15T08:53:50Z

core/src/main/java/com/scalar/db/storage/multistorage/MultiStorageAdmin.java

+    // The reasoning is:
+    // - ns1 is present in the mysql storage and listed in the mapping belonging to mysql
+    //     => returned
+    // - ns2 is present in the mysql storage but listed in the mapping belonging to cassandra


Sorry, I don't fully get this.
Is this a misconfiguration by users? (ns2 is defined in MySQL but the namespace mapping says it's mapped to Cassandra?)
It would be great if you could elaborate on this.

Is this a misconfiguration by users? (ns2 is defined in MySQL but the namespace mapping says it's mapped to Cassandra?)

It can be a misconfiguration or a deliberate action to do some testing for another ScalarDB app(even though it's probably not wise to do so).

This is a corner case that is probably unlikely to happen, but I included it in this explanation to highlight only namespaces strictly defined in the configuration are returned by this method.

@Torch3333
Let me clarify.
So, there are two apps that share the same ScalarDB Cluster.
One app (app1) defines ns1 and ns2 in its config, and the other (app2) defines ns3 and ns4 in its config.
But, the actual database defines ns1, ns2, ns3, and ns4.
In this case, app1 returns ns1 and ns2, and app2 returns ns3 and ns4 by getNamespaceNames?
I'm not sure if it is an expected behavior.

In this case, app1 returns ns1 and ns2, and app2 returns ns3 and ns4 by getNamespaceNames?

Correct. Yes, that is the expected behavior.

komamitsu

LGTM! Thank you!

feeblefakie

LGTM! Thank you!

feeblefakie · 2023-08-21T11:11:50Z

@brfrn169 @Torch3333 It's OK to merge, but I left one additional comment.
#989 (comment)

brfrn169 · 2023-08-23T01:23:05Z

@feeblefakie @Torch3333 Merging this for now. If we have problems with the behavior of MultiStorageAdmin. getNamespaceNames(), let's fix it at that time. Thanks!

Torch3333 and others added 23 commits September 7, 2022 16:44

Add method to retrieve the namespace names for the Jdbc Admin (#674)

891a9db

Add method to retrieve the namespace names for Cosmos Admin (#678)

2bdb499

Add method to retrieve the namespace names for the Dynamo Admin (#679)

ba3a94d

Add method to retrieve the namespace names for Cassandra Admin (#687)

67bd54f

Update JdbcAdmin.namespaceExists() (#693)

0518eb6

Add upgrade method to jdbc admin (#696)

3e86d4e

Merge branch 'master' of github.com:scalar-labs/scalardb into feat/na…

0a6f8b6

…mespaces_management

Implement CosmosAdmin.upgrade() (#707)

a999048

Implement DynamoAdmin.upgrade() (#708)

0cdb491

Add upgrade() method to the Cassandra Admin (#709)

32bc263

Merge branch 'master' of github.com:scalar-labs/scalardb into feat/na…

f4e8bfe

…mespaces_management # Conflicts: # core/src/main/java/com/scalar/db/storage/cosmos/CosmosConfig.java

Make ConsensusCommitAdmin.getNamespaceNames() to not return the coord…

9fce068

…inator namespace name (#732)

Add upgrade command to schema loader (#736)

9e58068

Fix tests

4ebfe75

Increase wait time to fix Cassandra integration test

a1fecd1

Throws UnsupportedOperationException when admin.getNamespaceNames() a…

531a472

…nd admin.upgrade() is executed on ScalarDB server

Fix import table integration test

2197649

Merge remote-tracking branch 'origin/do_not_support_upgrade_and_getNa…

6a6d3ff

…mespaceNames_on_ScalarDB_server' into feat/namespaces_management

Support sqlite

e8a0e0f

Fix [-skip ci]

d382b29

Torch3333 force-pushed the namespaces_management branch from 703ec87 to d382b29 Compare August 7, 2023 00:28

Torch3333 added 2 commits August 7, 2023 10:08

Fix

c2c5684

Fix typo in doc [skip ci]

2468962

Torch3333 commented Aug 7, 2023

View reviewed changes

Torch3333 self-assigned this Aug 7, 2023

Torch3333 added the backward-incompatible label Aug 7, 2023

komamitsu reviewed Aug 8, 2023

View reviewed changes

core/src/main/java/com/scalar/db/storage/jdbc/JdbcAdmin.java Show resolved Hide resolved

brfrn169 reviewed Aug 8, 2023

View reviewed changes

komamitsu reviewed Aug 8, 2023

View reviewed changes

core/src/main/java/com/scalar/db/storage/multistorage/MultiStorageAdmin.java Outdated Show resolved Hide resolved

brfrn169 reviewed Aug 9, 2023

View reviewed changes

docs/schema-loader.md Outdated Show resolved Hide resolved

schema-loader/src/test/java/com/scalar/db/schemaloader/SchemaOperatorTest.java Outdated Show resolved Hide resolved

Torch3333 added 3 commits August 14, 2023 14:27

Fix komamitsu comments

fc2def1

Fix brfrn169 comments

7e670ed

Torch3333 requested review from komamitsu and brfrn169 August 15, 2023 00:25

brfrn169 approved these changes Aug 15, 2023

View reviewed changes

feeblefakie reviewed Aug 15, 2023

View reviewed changes

Torch3333 added 2 commits August 16, 2023 11:41

Fix variable name

38bd49d

Add @immutable annotation to the CosmosNamespace marshalling object

2d6ed87

komamitsu approved these changes Aug 16, 2023

View reviewed changes

Torch3333 mentioned this pull request Aug 16, 2023

Make Cosmos DTO objects immutable #1013

Merged

Torch3333 requested a review from feeblefakie August 17, 2023 03:02

feeblefakie approved these changes Aug 21, 2023

View reviewed changes

Merge branch 'master' into namespaces_management

6ca1144

brfrn169 merged commit 99010f4 into master Aug 23, 2023

brfrn169 deleted the namespaces_management branch August 23, 2023 01:53

This was referenced Sep 11, 2023

Set Cassandra default replication factor to 3 #1080

Merged

Unify styles of the exception messages of Storage Admin implementations #1083

Merged

josh-wong mentioned this pull request Sep 28, 2023

Revise the contents of the storage abstraction doc #1119

Merged

6 tasks

This was referenced Oct 19, 2023

Add Admin API to repair a namespace #1107

Merged

Schema Loader --repair-all command will also repair namespaces #1203

Merged

Torch3333 added the enhancement New feature or request label Oct 19, 2023

Torch3333 mentioned this pull request Oct 19, 2023

Add upgrade() to Admin API #1204

Merged

6 tasks

Torch3333 mentioned this pull request Jun 27, 2024

Support listing namespaces for ScalarDB v3 #2002

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Namespaces management #989

Add Namespaces management #989

Torch3333 commented Aug 3, 2023 •

edited

Loading

Torch3333 Aug 7, 2023

Torch3333 Aug 7, 2023

Torch3333 Aug 7, 2023

Torch3333 Aug 7, 2023

Torch3333 Aug 7, 2023

Torch3333 Aug 7, 2023

Torch3333 Aug 7, 2023 •

edited

Loading

brfrn169 left a comment

brfrn169 left a comment •

edited

Loading

brfrn169 left a comment

brfrn169 Aug 15, 2023

Torch3333 Aug 16, 2023

feeblefakie left a comment

feeblefakie Aug 15, 2023

Torch3333 Aug 16, 2023 •

edited

Loading

feeblefakie Aug 21, 2023 •

edited

Loading

Torch3333 Aug 22, 2023

komamitsu left a comment

feeblefakie left a comment

feeblefakie commented Aug 21, 2023

brfrn169 commented Aug 23, 2023 •

edited

Loading

		@@ -49,34 +49,35 @@ The following describes the configurations available for each storage.

		- For Cassandra, the following configurations are available:

	String NamespaceTableFullName =
	String namespaceTableFullName =

Add Namespaces management #989

Add Namespaces management #989

Conversation

Torch3333 commented Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Torch3333 Aug 7, 2023 • edited Loading

Choose a reason for hiding this comment

brfrn169 left a comment

Choose a reason for hiding this comment

brfrn169 left a comment • edited Loading

Choose a reason for hiding this comment

brfrn169 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feeblefakie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Torch3333 Aug 16, 2023 • edited Loading

Choose a reason for hiding this comment

feeblefakie Aug 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komamitsu left a comment

Choose a reason for hiding this comment

feeblefakie left a comment

Choose a reason for hiding this comment

feeblefakie commented Aug 21, 2023

brfrn169 commented Aug 23, 2023 • edited Loading

Torch3333 commented Aug 3, 2023 •

edited

Loading

Torch3333 Aug 7, 2023 •

edited

Loading

brfrn169 left a comment •

edited

Loading

Torch3333 Aug 16, 2023 •

edited

Loading

feeblefakie Aug 21, 2023 •

edited

Loading

brfrn169 commented Aug 23, 2023 •

edited

Loading