Skip to content

Commit

Permalink
getting rid of Kubernetes leftovers
Browse files Browse the repository at this point in the history
  • Loading branch information
@smiklosovic
smiklosovic committed Sep 19, 2023
1 parent bcd4bd8 commit 76654bc
Show file tree
Hide file tree
Showing 17 changed files with 37 additions and 129 deletions.
12 changes: 4 additions & 8 deletions README.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,7 @@ providing they are easily implementable
any subsequent backups will upload and restores will download only the difference
* When used in connection with https://github.com/instaclustr/icarus[Instaclustr Icarus] it is possible to backup **simultaneously** so there
might be more concurrent backups which may overlap what they backup
* Used as backup/restore solution in https://github.com/instaclustr/cassandra-operator[Instaclustr Cassandra Operator] for Kubernetes
* Possible to restore whole node _from scratch_ (or whole cluster from scratch in cases where Kubernetes Operator is used)
* Possible to restore whole node / cluster _from scratch_
* In connection with Icarus, it is possible to **restore on a running cluster** so no
downtime is necessary
* It takes care of details such as initial tokens, auto bootstrapping, and so on...
Expand Down Expand Up @@ -243,8 +242,7 @@ The content of that hypothetical bucket with same data will look like this:
```
. When this tool is used in connection with Instaclustr Cassandra Sidecar, it also creates a _topology_ file. Topology file
is needed for restoration into a Kubernetes cluster.
. When this tool is used in connection with Instaclustr Cassandra Sidecar, it also creates a _topology_ file.
. Data for each node are stored under that very node, here we used UUID identifier which is host ID as Cassandra sees it, and it is unique.
Hence, it is impossible to accidentally store data for a different node as each node will have unique UUID. It may happen
that over time we will have a cluster of same name and data center of same name but the node id would be still different
Expand Down Expand Up @@ -512,7 +510,6 @@ In more detail, in-place strategy does the following:
. Checks that a remote bucket to download data from exists and errors out if it does not
. In case `--resolve-host-id-from-topology` flag is used, it will resolve a host to restore from topology file.
This is handy for cases we want to restore e.g. in the context of Kubernetes by our operator.
. Downloads a manifest—manifest contains the list of files which are logically related to a snapshot.
. Filters out the files which need to be downloaded, as some files which are present locally might be
also a part of a taken snapshot so we would download them unnecessarily.
Expand All @@ -528,7 +525,7 @@ Normally, system keyspaces are not restored and one has to set this explicitly b
In-place strategy uses also `--restore-into-new-cluster` flag. If specified, it will restore only system
keyspaces needed for successful restoring (`system_schema`) but it will not attempt to restore anything else.
In an environment like Kubernetes, we do not want to restore _everything_ because system keyspaces
We do not always want to restore _everything_ because system keyspaces
contain details like tokens, peers with ips, etc. and this information is very specific to each one so
we do not restore them. However, if we did not restore `system_schema`, the newly started node would not see
the restored data as there would not be any schema. By restoring `system_schema`, Cassandra will detect
Expand Down Expand Up @@ -1343,8 +1340,7 @@ You can find the original file in `src/main/resources/logback.xml`.
## Build and Test
There are end-to-end tests which can test all GCP, Azure, and S3 integrations as well
as integrations with Kubernetes when it comes to credential fetching.
There are end-to-end tests which can test all GCP, Azure, and S3 integrations.
Here are the test groups/profiles:
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -491,7 +491,7 @@
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven.surefire.plugin.version}</version>
<configuration>
<excludedGroups>cloudTest,azureTest,googleTest,s3Test,cephTest</excludedGroups>
<excludedGroups>cloudTest,azureTest,googleTest,s3Test</excludedGroups>
</configuration>
</plugin>
</plugins>
Expand Down
36 changes: 0 additions & 36 deletions src/main/java/com/instaclustr/esop/impl/AbstractOperationRequest.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,6 @@
import java.util.Arrays;
import java.util.Set;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
Expand All @@ -22,8 +18,6 @@

public abstract class AbstractOperationRequest extends OperationRequest {

private static final Logger logger = LoggerFactory.getLogger(AbstractOperationRequest.class);

@Option(names = {"--sl", "--storage-location"},
converter = StorageLocationTypeConverter.class,
description = "Location to which files will be backed up or restored from, in form " +
Expand All @@ -34,17 +28,6 @@ public abstract class AbstractOperationRequest extends OperationRequest {
@JsonDeserialize(using = StorageLocationDeserializer.class)
public StorageLocation storageLocation;

@Option(names = {"--k8s-namespace"},
description = "Name of Kubernetes namespace backup tool runs in, if any.",
defaultValue = "default")
@JsonProperty("k8sNamespace")
public String k8sNamespace;

@Option(names = {"--k8s-secret-name"},
description = "Name of Kubernetes secret used for credential retrieval for backup / restores when talking to a cloud storage.")
@JsonProperty("k8sSecretName")
public String k8sSecretName;

@Option(names = {"--insecure-http"},
description = "If specified, the connection to remote bucket will be insecure, instead HTTPS, HTTP will be used, currently relevant only for S3 and Azure.")
@JsonProperty("insecure")
Expand Down Expand Up @@ -80,17 +63,13 @@ public AbstractOperationRequest() {
}

public AbstractOperationRequest(final StorageLocation storageLocation,
final String k8sNamespace,
final String k8sSecretName,
final boolean insecure,
final boolean skipBucketVerification,
final ProxySettings proxySettings,
final RetrySpec retry,
final Integer concurrentConnections,
final String kmsKeyId) {
this.storageLocation = storageLocation;
this.k8sNamespace = k8sNamespace;
this.k8sSecretName = k8sSecretName;
this.insecure = insecure;
this.skipBucketVerification = skipBucketVerification;
this.proxySettings = proxySettings;
Expand All @@ -99,21 +78,6 @@ public AbstractOperationRequest(final StorageLocation storageLocation,
this.kmsKeyId = kmsKeyId;
}

@JsonIgnore
public String resolveKubernetesSecretName() {
String resolvedSecretName;

if (k8sSecretName == null) {
resolvedSecretName = String.format("cassandra-backup-restore-secret-cluster-%s", storageLocation.clusterId);
} else {
resolvedSecretName = k8sSecretName;
}

logger.info("Resolved secret name {}", resolvedSecretName);

return resolvedSecretName;
}

public void validate(final Set<String> storageProviders) {
if (storageLocation == null) {
throw new IllegalStateException("storageLocation has to be specified!");
Expand Down
6 changes: 0 additions & 6 deletions src/main/java/com/instaclustr/esop/impl/backup/BackupCommitLogsOperationRequest.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,6 @@ public BackupCommitLogsOperationRequest(@JsonProperty("storageLocation") final S
@JsonProperty("metadataDirective") final MetadataDirective metadataDirective,
@JsonProperty("cassandraCommitLogDirectory") final Path cassandraCommitLogDirectory,
@JsonProperty("commitLogArchiveOverride") final Path commitLogArchiveOverride,
@JsonProperty("k8sNamespace") final String k8sNamespace,
@JsonProperty("k8sSecretName") final String k8sSecretName,
@JsonProperty("insecure") final boolean insecure,
@JsonProperty("createMissingBucket") final boolean createMissingBucket,
@JsonProperty("skipBucketVerification") final boolean skipBucketVerification,
Expand All @@ -82,8 +80,6 @@ public BackupCommitLogsOperationRequest(@JsonProperty("storageLocation") final S
bandwidth,
concurrentConnections,
metadataDirective,
k8sNamespace,
k8sSecretName,
insecure,
createMissingBucket,
skipBucketVerification,
Expand Down Expand Up @@ -111,8 +107,6 @@ public String toString() {
.add("commitLogArchiveOverride", commitLogArchiveOverride)
.add("commitLog", commitLog)
.add("online", online)
.add("k8sNamespace", k8sNamespace)
.add("k8sSecretName", k8sSecretName)
.add("createMissingBucket", createMissingBucket)
.add("skipBucketVerification", skipBucketVerification)
.add("insecure", insecure)
Expand Down
4 changes: 0 additions & 4 deletions src/main/java/com/instaclustr/esop/impl/backup/BackupOperation.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,6 @@ private BackupOperation(@JsonProperty("type") final String type,
@JsonSerialize(using = DatabaseEntitiesSerializer.class)
@JsonDeserialize(using = DatabaseEntitiesDeserializer.class) final DatabaseEntities entities,
@JsonProperty("snapshotTag") final String snapshotTag,
@JsonProperty("k8sNamespace") final String k8sNamespace,
@JsonProperty("k8sSecretName") final String k8sBackupSecretName,
@JsonProperty("globalRequest") final boolean globalRequest,
@JsonProperty("dc") final String dc,
@JsonProperty("timeout") final Integer timeout,
Expand All @@ -102,8 +100,6 @@ private BackupOperation(@JsonProperty("type") final String type,
metadataDirective,
entities,
snapshotTag,
k8sNamespace,
k8sBackupSecretName,
globalRequest,
dc,
timeout,
Expand Down
6 changes: 0 additions & 6 deletions src/main/java/com/instaclustr/esop/impl/backup/BackupOperationRequest.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,6 @@ public BackupOperationRequest(@JsonProperty("type") final String type,
@JsonSerialize(using = DatabaseEntitiesSerializer.class)
@JsonDeserialize(using = DatabaseEntitiesDeserializer.class) final DatabaseEntities entities,
@JsonProperty("snapshotTag") final String snapshotTag,
@JsonProperty("k8sNamespace") final String k8sNamespace,
@JsonProperty("k8sSecretName") final String k8sSecretName,
@JsonProperty("globalRequest") final boolean globalRequest,
@JsonProperty("dc") final String dc,
@JsonProperty("timeout") final Integer timeout,
Expand All @@ -109,8 +107,6 @@ public BackupOperationRequest(@JsonProperty("type") final String type,
bandwidth,
concurrentConnections,
metadataDirective,
k8sNamespace,
k8sSecretName,
insecure,
createMissingBucket,
skipBucketVerification,
Expand Down Expand Up @@ -139,8 +135,6 @@ public String toString() {
.add("dataDirs", dataDirs)
.add("entities", entities)
.add("snapshotTag", snapshotTag)
.add("k8sNamespace", k8sNamespace)
.add("k8sSecretName", k8sSecretName)
.add("globalRequest", globalRequest)
.add("dc", dc)
.add("timeout", timeout)
Expand Down
4 changes: 1 addition & 3 deletions src/main/java/com/instaclustr/esop/impl/backup/BaseBackupOperationRequest.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,6 @@ public BaseBackupOperationRequest(final StorageLocation storageLocation,
final DataRate bandwidth,
final Integer concurrentConnections,
final MetadataDirective metadataDirective,
final String k8sNamespace,
final String k8sBackupSecretName,
final boolean insecure,
final boolean createMissingBucket,
final boolean skipBucketVerification,
Expand All @@ -86,7 +84,7 @@ public BaseBackupOperationRequest(final StorageLocation storageLocation,
final boolean skipRefreshing,
final List<Path> dataDirs,
final String kmsKeyId) {
super(storageLocation, k8sNamespace, k8sBackupSecretName, insecure, skipBucketVerification, proxySettings, retrySpec, concurrentConnections, kmsKeyId);
super(storageLocation, insecure, skipBucketVerification, proxySettings, retrySpec, concurrentConnections, kmsKeyId);
this.duration = duration;
this.bandwidth = bandwidth;
this.metadataDirective = metadataDirective == null ? MetadataDirective.COPY : metadataDirective;
Expand Down
4 changes: 0 additions & 4 deletions src/main/java/com/instaclustr/esop/impl/list/ListOperation.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,6 @@ private ListOperation(@JsonProperty("type") final String type,
@JsonProperty("progress") final float progress,
@JsonProperty("startTime") final Instant startTime,
@JsonProperty("storageLocation") final StorageLocation storageLocation,
@JsonProperty("k8sNamespace") final String k8sNamespace,
@JsonProperty("k8sSecretName") final String k8sSecretName,
@JsonProperty("insecure") final boolean insecure,
@JsonProperty("skipBucketVerification") final boolean skipBucketVerification,
@JsonProperty("proxySettings") final ProxySettings proxySettings,
Expand All @@ -89,8 +87,6 @@ private ListOperation(@JsonProperty("type") final String type,
@JsonProperty("concurrentConnections") final Integer concurrentConnections) {
super(type, id, creationTime, state, errors, progress, startTime, new ListOperationRequest(type,
storageLocation,
k8sNamespace,
k8sSecretName,
insecure,
skipBucketVerification,
proxySettings,
Expand Down
6 changes: 1 addition & 5 deletions src/main/java/com/instaclustr/esop/impl/list/ListOperationRequest.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,6 @@ public ListOperationRequest() {
@JsonCreator
public ListOperationRequest(@JsonProperty("type") final String type,
@JsonProperty("storageLocation") final StorageLocation storageLocation,
@JsonProperty("k8sNamespace") final String k8sNamespace,
@JsonProperty("k8sSecretName") final String k8sSecretName,
@JsonProperty("insecure") final boolean insecure,
@JsonProperty("skipBucketVerification") final boolean skipBucketVerification,
@JsonProperty("proxySettings") final ProxySettings proxySettings,
Expand All @@ -82,7 +80,7 @@ public ListOperationRequest(@JsonProperty("type") final String type,
@JsonProperty("toRequest") final boolean toRequest,
@JsonProperty("concurrentConnections") final Integer concurrentConnections,
@JsonProperty("response") final Manifest.AllManifestsReport response) {
super(storageLocation, 1, k8sNamespace, k8sSecretName, insecure, skipBucketVerification, proxySettings, retry, null);
super(storageLocation, 1, insecure, skipBucketVerification, proxySettings, retry, null);
this.json = json;
this.resolveNodes = resolveNodes;
this.humanUnits = humanUnits;
Expand Down Expand Up @@ -111,8 +109,6 @@ public static ListOperationRequest getForLocalListing(final BaseRestoreOperation
return new ListOperationRequest(
"list",
cacheLocation,
request.k8sNamespace,
request.k8sSecretName,
request.insecure,
request.skipBucketVerification,
request.proxySettings,
Expand Down
2 changes: 0 additions & 2 deletions src/main/java/com/instaclustr/esop/impl/remove/RemoveBackupOperation.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,6 @@ private RemoveBackupOperation(@JsonProperty("type") final String type,
@JsonProperty("globalRequest") final boolean globalRequest) {
super(type, id, creationTime, state, errors, progress, startTime, new RemoveBackupRequest(type,
storageLocation,
k8sNamespace,
k8sSecretName,
insecure,
skipBucketVerification,
proxySettings,
Expand Down
4 changes: 1 addition & 3 deletions src/main/java/com/instaclustr/esop/impl/remove/RemoveBackupRequest.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,6 @@ public RemoveBackupRequest() {
@JsonCreator
public RemoveBackupRequest(@JsonProperty("type") final String type,
@JsonProperty("storageLocation") final StorageLocation storageLocation,
@JsonProperty("k8sNamespace") final String k8sNamespace,
@JsonProperty("k8sSecretName") final String k8sSecretName,
@JsonProperty("insecure") final boolean insecure,
@JsonProperty("skipBucketVerification") final boolean skipBucketVerification,
@JsonProperty("proxySettings") final ProxySettings proxySettings,
Expand All @@ -80,7 +78,7 @@ public RemoveBackupRequest(@JsonProperty("type") final String type,
@JsonProperty("removeOldest") final boolean removeOldest,
@JsonProperty("concurrentConnections") final Integer concurrentConnections,
@JsonProperty("globalRequest") final boolean globalRequest) {
super(storageLocation, 1, k8sNamespace, k8sSecretName, insecure, skipBucketVerification, proxySettings, retry, null);
super(storageLocation, 1, insecure, skipBucketVerification, proxySettings, retry, null);
this.type = type;
this.backupName = backupName;
this.dry = dry;
Expand Down
Loading

0 comments on commit 76654bc

Please sign in to comment.