Add `upgrade()` to Admin API #1204

Torch3333 · 2023-10-18T14:51:35Z

Description

Considering we have the two following requirement:

As a ScalarDB user, I want to be able to upgrade my ScalarDB environment (schema metadata and coordinator table) to support backward incompatible changes introduced by a new version.
As a ScalarDB user, I want to be able to upgrade my ScalarDB environment to support backward incompatible changes introduced by the namespace management feature.

To address story 2 while also being able to address the more general story 1 for future work, we choose a generic API name, upgrade(). Currently, this method will only address the need to upgrade the environment because of the namespace management feature, but we can append new behaviors anytime to handle new features that break the backward compatibility of the environment.

Related issues and/or PRs

Related to Add Namespaces management #989

Changes made

Add a new void upgrade(Map<String, String> options) method to the Admin API, which will create the namespace metadata table and create metadata for the existing namespace.

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

A new --upgrade command will be added to the Schema Loader in another PR
The documentation will be updated in another PR.
upgrade() is not supported in the ScalarDB Server and JDBC Transaction Admin since these two components will be deleted before the 4.0.0 release.

Release notes

Added a new method, void upgrade(Map<String,String> options), to the Admin API. Running this method when updating an existing environment to ScalarDB 4.X is necessary to support backward incompatible changes introduced by the new "namespaces management" feature.

Torch3333 · 2023-10-19T00:24:22Z

core/src/main/java/com/scalar/db/api/Admin.java

+  /**
+   * Upgrades the Scalar DB environment to support the latest version of the Scalar DB API.
+   * Typically, you will be requested, as indicated on the release notes, to run this method after
+   * updating the Scalar DB version of your application environment.
+   *
+   * @param options options to upgrade
+   * @throws ExecutionException if the operation fails
+   */
+  void upgrade(Map<String, String> options) throws ExecutionException;


The new upgrade() Admin API.

Torch3333 · 2023-10-19T00:24:44Z

core/src/main/java/com/scalar/db/storage/cassandra/CassandraAdmin.java

@@ -406,6 +406,26 @@ public Set<String> getNamespaceNames() throws ExecutionException {
    }
  }

+  @Override
+  public void upgrade(Map<String, String> options) throws ExecutionException {


The upgrade() implementation of the CassandraAdmin

Torch3333 · 2023-10-19T00:25:07Z

core/src/main/java/com/scalar/db/storage/cosmos/CosmosAdmin.java

@@ -647,6 +647,33 @@ public Set<String> getNamespaceNames() throws ExecutionException {
    }
  }

+  @Override
+  public void upgrade(Map<String, String> options) throws ExecutionException {


The upgrade() implementation of the CosmosAdmin

Torch3333 · 2023-10-19T00:25:32Z

core/src/main/java/com/scalar/db/storage/dynamo/DynamoAdmin.java

@@ -1363,6 +1363,50 @@ public Set<String> getNamespaceNames() throws ExecutionException {
    }
  }

+  @Override
+  public void upgrade(Map<String, String> options) throws ExecutionException {


The upgrade() implementation of the DynamoAdmin

Torch3333 · 2023-10-19T00:30:11Z

core/src/main/java/com/scalar/db/storage/jdbc/JdbcAdmin.java

@@ -1008,4 +1013,39 @@ private boolean isNamespacesTableEmpty(Connection connection) throws SQLExceptio
      return !results.next();
    }
  }
+
+  @Override
+  public void upgrade(Map<String, String> options) throws ExecutionException {


The upgrade() implementation of the JdbcAdmin

brfrn169

LGTM! Thank you!

core/src/main/java/com/scalar/db/api/Admin.java

core/src/main/java/com/scalar/db/storage/cassandra/CassandraAdmin.java

core/src/main/java/com/scalar/db/storage/dynamo/DynamoAdmin.java

komamitsu · 2023-10-19T02:43:27Z

core/src/main/java/com/scalar/db/storage/cosmos/CosmosAdmin.java

+  public void upgrade(Map<String, String> options) throws ExecutionException {
+    try {
+      if (!tableMetadataContainerExists()) {
+        return;


Just quick silly questions.

It's impossible to create the table metadata container if not exists?

If not possible, don't need to throw an exception?

There may be a misunderstanding here.
This check verifies if the table metadata container does not exist, if true this means there is no table installed, so there is no namespace to import to the namespaces metadata table.
A limitation is that if there was a namespace that did not contain a ScalarDB-managed table, we wouldn't import it to the namespace table. It's because we assume such namespace are the storage system namespaces or are user-one but not used for ScalarDB.

Does this answer your question ?

Thank you! Understood 👍

komamitsu · 2023-10-19T02:43:57Z

core/src/main/java/com/scalar/db/storage/dynamo/DynamoAdmin.java

+  @Override
+  public void upgrade(Map<String, String> options) throws ExecutionException {
+    if (!metadataTableExists()) {
+      return;


Similar question to the above one

My answer is the same as #1204 (comment)

Co-authored-by: Mitsunori Komatsu <komamitsu@gmail.com>

komamitsu

LGTM! 👍

komamitsu · 2023-10-19T04:30:56Z

core/src/main/java/com/scalar/db/storage/cosmos/CosmosAdmin.java

+  public void upgrade(Map<String, String> options) throws ExecutionException {
+    try {
+      if (!tableMetadataContainerExists()) {
+        return;


Thank you! Understood 👍

Torch3333 · 2023-10-20T08:56:42Z

I pushed a fix for missing @Override annotation on some methods.
5879099

feeblefakie

LGTM! Thank you!

Torch3333 self-assigned this Oct 18, 2023

Torch3333 added the enhancement New feature or request label Oct 18, 2023

Torch3333 changed the base branch from master to repair_namespace October 18, 2023 14:52

Base automatically changed from repair_namespace to master October 19, 2023 00:00

Add upgrade() to Admin API

59c9033

Torch3333 force-pushed the add_upgrade branch from 42e8912 to 59c9033 Compare October 19, 2023 00:23

Torch3333 commented Oct 19, 2023

View reviewed changes

Torch3333 changed the title ~~Add upgrade to Admin API~~ Add upgrade method to Admin API Oct 19, 2023

Torch3333 changed the title ~~Add upgrade method to Admin API~~ Add upgrade() to Admin API Oct 19, 2023

Torch3333 marked this pull request as ready for review October 19, 2023 01:06

Torch3333 requested review from a team, komamitsu, brfrn169 and feeblefakie and removed request for a team October 19, 2023 01:06

brfrn169 approved these changes Oct 19, 2023

View reviewed changes

Fix typo in comment

0c30f4d

komamitsu reviewed Oct 19, 2023

View reviewed changes

Fix typos

05c36f8

Co-authored-by: Mitsunori Komatsu <komamitsu@gmail.com>

Torch3333 requested a review from komamitsu October 19, 2023 03:54

Apply spotless

c370cd6

komamitsu approved these changes Oct 19, 2023

View reviewed changes

Torch3333 mentioned this pull request Oct 20, 2023

Add --upgrade command to Schema Loader #1208

Merged

6 tasks

Add missing @Override annotation on getAdminUtils() methods

5879099

feeblefakie approved these changes Oct 23, 2023

View reviewed changes

feeblefakie merged commit 3e4b05b into master Oct 23, 2023
23 checks passed

feeblefakie deleted the add_upgrade branch October 23, 2023 07:52

Torch3333 mentioned this pull request Nov 22, 2023

Add documentation for the upgrade feature #1311

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `upgrade()` to Admin API #1204

Add `upgrade()` to Admin API #1204

Torch3333 commented Oct 18, 2023 •

edited

Loading

Torch3333 Oct 19, 2023 •

edited

Loading

Torch3333 Oct 19, 2023

Torch3333 Oct 19, 2023

Torch3333 Oct 19, 2023

Torch3333 Oct 19, 2023

brfrn169 left a comment

komamitsu Oct 19, 2023

Torch3333 Oct 19, 2023 •

edited

Loading

Torch3333 Oct 19, 2023

komamitsu Oct 19, 2023

komamitsu Oct 19, 2023

Torch3333 Oct 19, 2023

komamitsu left a comment

komamitsu Oct 19, 2023

Torch3333 commented Oct 20, 2023

feeblefakie left a comment

Add upgrade() to Admin API #1204

Add upgrade() to Admin API #1204

Conversation

Torch3333 commented Oct 18, 2023 • edited Loading

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Release notes

Torch3333 Oct 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brfrn169 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Torch3333 Oct 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komamitsu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Torch3333 commented Oct 20, 2023

feeblefakie left a comment

Choose a reason for hiding this comment

Add `upgrade()` to Admin API #1204

Add `upgrade()` to Admin API #1204

Torch3333 commented Oct 18, 2023 •

edited

Loading

Torch3333 Oct 19, 2023 •

edited

Loading

Torch3333 Oct 19, 2023 •

edited

Loading