THREESCALE-10740 remove internal databases support #1060

MStokluska · 2025-01-20T11:23:18Z

Jira: https://issues.redhat.com/browse/THREESCALE-10740

removes reconciliation of internal databases
adds new make commands to easily spin up dev databases required by 3scale
update the 2.16 databases requirements, for postgres to 13, for redis to 7.
updates the zync postgres image to 13
adds pre-flight check on required externalDatabases fields
fix on nil pointer in watched-by logic

Verification

Local installation

run make cluster/prepare/local
run:

export NAMESPACE=3scale-test

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: s3-credentials
  namespace: $NAMESPACE
data:
  AWS_ACCESS_KEY_ID: c29tZXRoaW5nCg==
  AWS_BUCKET: c29tZXRoaW5nCg==
  AWS_REGION: dXMtd2VzdC0xCg==
  AWS_SECRET_ACCESS_KEY: c29tZXRoaW5nCg==
type: Opaque
EOF

DOMAIN=$(oc get routes console -n openshift-console -o json | jq -r '.status.ingress[0].routerCanonicalHostname' | sed 's/router-default.//')
cat << EOF | oc create -f -
kind: APIManager
apiVersion: apps.3scale.net/v1alpha1
metadata:
  name: 3scale
  namespace: $NAMESPACE
spec:
  wildcardDomain: $DOMAIN
  system:
    fileStorage:
      simpleStorageService:
        configurationSecretRef:
          name: s3-credentials
EOF

run make run
navigate to APIM and confirm that preflights failed due to missing external fields
add the external fields in apim:

...
spec:
  externalComponents:
    backend:
      redis: true
    system:
      database: true
      redis: true

confirm that the installation succeeds
uninstall and remove the namespace
repeat the steps but this time, run make cluster/prepare/local DEV_SYSTEM_DB_POSTGRES=true
the difference is that now, PostgreSQL database should be provisioned.
let the installation complete (after adding the externalComponents section to APIM)
delete namespace

OLM Upgrade with MySQL

run cluster/prepare/project
run:

cat << EOF | oc create -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: threescale-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/mstoklus/3scale-index:v0.12.2
EOF

navigate to 3scale-test ns and install 3scale 0.12.2 from threescale-dev catalog
create apim:

export NAMESPACE=3scale-test

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: s3-credentials
  namespace: $NAMESPACE
data:
  AWS_ACCESS_KEY_ID: c29tZXRoaW5nCg==
  AWS_BUCKET: c29tZXRoaW5nCg==
  AWS_REGION: dXMtd2VzdC0xCg==
  AWS_SECRET_ACCESS_KEY: c29tZXRoaW5nCg==
type: Opaque
EOF

DOMAIN=$(oc get routes console -n openshift-console -o json | jq -r '.status.ingress[0].routerCanonicalHostname' | sed 's/router-default.//')
cat << EOF | oc create -f -
kind: APIManager
apiVersion: apps.3scale.net/v1alpha1
metadata:
  name: 3scale
  namespace: $NAMESPACE
spec:
  wildcardDomain: $DOMAIN
  system:
    fileStorage:
      simpleStorageService:
        configurationSecretRef:
          name: s3-credentials
EOF

wait for installation to complete fully
at this point, you should have installation version 2.15.2 installed
update the catalog source image to: quay.io/mstoklus/3scale-index:v0.13.0
without approving the new version, confirm after few minutes that the APIManager status has changed to

Preflights failed - attempted upgrade to 2.16 have been performed but the requirements are not met, operator will keep reconciling but ensure requirements are met in order to proceed with upgrade, backend redis version mismatch - required is Redis 7.0.0; system redis version mismatch - required is Redis 7.0.0;  - re-running preflights in 10 minutes

while this message is visible, edit value of rht.comp_ver: 2.15 in apicast-production to 2.16. The change should get reverted within minute or so.
update the image of both backend-redis and system-redis to: quay.io/fedora/redis-7
Note that we are not doing an upgrade procedure here, just a hacky fix to bump redis to required versions.
wait for preflights to pass and report a message: All requirement for incoming version are met. If using automatic upgrades the upgrade will start shortly, if manual, you can proceed with approval
approve upgrade to 2.16 IF you have set manual approvals only, if not, upgrade to 2.16 will start automatically.
Confirm that once 2.16 is in "installed operators" the APIM is reporting failed preflight because databases aren't set to external and 3scale instance is still using 2.15 images (this can be verified by looking at rht.comp_ver: 2.15 label in deployments), message on APIM should be:

'Preflights failed - cannot continue with the installation. External databases are required to be set in the APIManager for system Redis, backend Redis and system database - re-running preflights in 10 minutes'

add external fields for both redis dbs in APIM:

spec:
  externalComponents:
    backend:
      redis: true
    system:
      database: true
      redis: true

Let the installation complete.
Remove the namespace, repeat the process but for PostgreSQL database as default.
Confirm that there's additional requirement on 2.15 -> 2.16 - which is PostgreSQL 13. To upgrade the database to PSQL 13 please reach out to me for a document describing the process.

…to N+1 path

… internal bump

valerymo · 2025-01-23T14:13:25Z

Makefile

@@ -107,6 +110,51 @@ $(CONTROLLER_GEN):
 .PHONY: controller-gen
 controller-gen: $(CONTROLLER_GEN)

+.PHONY: cluster/prepare/local
+cluster/prepare/local: kustomize cluster/prepare/project install cluster/create/system-redis cluster/create/backend-redis cluster/create/provision-database


@MStokluska looks very nice, but seems missing "make download" step, need add download target here also?

go mod download is not necessary. I never run it.

valerymo · 2025-01-23T16:37:52Z

Verified according to Validation notes - 2 installations , with mysql and postgres.
Working as expected

~/go/3scale-operator oc get pod NAME READY STATUS RESTARTS AGE apicast-production-5797bc85b5-6xjqh 1/1 Running 0 5m15s apicast-staging-5bb9d8c587-pd6pr 1/1 Running 0 5m15s backend-cron-d889554d4-bhzwm 1/1 Running 0 6m5s backend-listener-9448cc9c7-cckmt 1/1 Running 0 6m1s backend-redis-664479c8bb-9vv8w 1/1 Running 0 9m39s backend-worker-84798dc74b-d68vs 1/1 Running 0 6m1s system-app-68fbb9c847-6g227 3/3 Running 0 4m44s system-app-post-wv4l4 0/1 Completed 0 3m36s system-app-pre-ck5kk 0/1 Completed 0 5m17s system-memcache-b6565f76b-lnpck 1/1 Running 0 6m2s system-postgresql-f45b75766-mz4mc 1/1 Running 0 9m38s system-redis-55b7df87b-926b4 1/1 Running 0 9m41s system-searchd-669dc7599c-rfvgc 1/1 Running 0 6m2s system-searchd-manticore-reindex-cnbrk 0/1 Completed 0 5m18s system-sidekiq-55b7d64fc7-kjvkx 1/1 Running 0 5m16s zync-77555479f-744tg 1/1 Running 0 5m16s zync-database-7c446ccdf-6wn68 1/1 Running 0 5m16s zync-que-844ff6987b-7m6q8 1/1 Running 2 (5m ago) 5m16s ~/go/3scale-operator oc get deploy NAME READY UP-TO-DATE AVAILABLE AGE apicast-production 1/1 1 1 5m21s apicast-staging 1/1 1 1 5m21s backend-cron 1/1 1 1 6m11s backend-listener 1/1 1 1 6m10s backend-redis 1/1 1 1 9m45s backend-worker 1/1 1 1 6m10s system-app 1/1 1 1 4m50s system-memcache 1/1 1 1 6m8s system-postgresql 1/1 1 1 9m44s system-redis 1/1 1 1 9m47s system-searchd 1/1 1 1 6m8s system-sidekiq 1/1 1 1 5m22s zync 1/1 1 1 5m22s zync-database 1/1 1 1 5m22s zync-que 1/1 1 1 5m22s ~/go/3scale-operator

valerymo · 2025-01-23T16:38:08Z

/lgtm

austincunningham · 2025-01-24T10:50:29Z

bundle/manifests/3scale-operator.clusterserviceversion.yaml

@@ -557,15 +557,11 @@ spec:
                - name: RELATED_IMAGE_SYSTEM_MEMCACHED
                  value: mirror.gcr.io/library/memcached:1.5
                - name: RELATED_IMAGE_BACKEND_REDIS
-                  value: quay.io/fedora/redis-6:latest
+                  value: quay.io/fedora/redis-7


Why are we updating the redis image? Taught the flow was that by not updating to redis 7 we are forcing a move to external. Or is its just for development if so I don't think it should be in the CSV?

ah is it for the preflights ?

exactly, preflights, until we move to go libs we need to stick to these images :/

austincunningham · 2025-01-24T13:31:17Z

OLM Verification

make command fails if namespace exists

make cluster/prepare/project
go: downloading go1.22.10 (linux/amd64)
Error from server (AlreadyExists): project.project.openshift.io "3scale-test" already exists
make: [Makefile:156: cluster/prepare/project] Error 1 (ignored)

This is expected as all the command does is create a project.

Mysql & Redis Upgrade

install 2.15.2 via olm

apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  annotations:
    apps.3scale.net/apimanager-confirmed-requirements-version: '149630'
    apps.3scale.net/apimanager-threescale-version: 2.15.2
    apps.3scale.net/threescale-operator-version: 0.12.2
  creationTimestamp: '2025-01-24T11:51:46Z'
  generation: 2
  managedFields: ...
  name: 3scale
  namespace: 3scale-test
spec:
  apicast:
    managementAPI: status
    openSSLVerify: false
    productionSpec: {}
    registryURL: 'http://apicast-staging:8090/policies'
    responseCodes: true
    stagingSpec: {}
  appLabel: 3scale-api-management
  backend:
    cronSpec: {}
    listenerSpec: {}
    workerSpec: {}
  resourceRequirementsEnabled: true
  system:
    appSpec: {}
    fileStorage:
      simpleStorageService:
        configurationSecretRef:
          name: s3-credentials
    searchdSpec: {}
    sidekiqSpec: {}
  tenantName: 3scale
  wildcardDomain: apps.aucunnin.41x3.s1.devshift.org
  zync:
    appSpec: {}
    queSpec: {}
status:
  conditions:
    - lastTransitionTime: '2025-01-24T11:55:33Z'
      status: 'True'
      type: Available
    - message: All requirements for the current version are met
      reason: PreflightsPass
      status: 'True'
      type: Preflights
  deployments:
    ready:
      - apicast-production
      - apicast-staging
      - backend-cron
      - backend-listener
      - backend-redis
      - backend-worker
      - system-app
      - system-memcache
      - system-mysql
      - system-redis
      - system-searchd
      - system-sidekiq
      - zync
      - zync-database
      - zync-que

updated catalog source index to new image quay.io/mstoklus/3scale-index:v0.13.0

status:
  conditions:
    - lastTransitionTime: '2025-01-24T11:55:33Z'
      status: 'True'
      type: Available
    - message: 'Preflights failed - attempted upgrade to 2.16 have been performed but the requirements are not met, operator will keep reconciling but ensure requirements are met in order to proceed with upgrade, backend redis version mismatch - required is Redis 7.0.0; system redis version mismatch - required is Redis 7.0.0;  - re-running preflights in 10 minutes'
      reason: PreflightsPass
      status: 'False'
      type: Preflights

edited rht.comp_ver: 2.15 in apicast-production deployment to 2.16 and it was reverted.
updated the redis images and after some time the apimanger status updated

message: 'All requirement for incoming version are met. If using automatic upgrades the upgrade will start shortly, if manual, you can proceed with approval'

didn't have to manually approve looks like its completed, but the apimanager is still pointing at 2.15.2

metadata:
  annotations:
    apps.3scale.net/apimanager-confirmed-requirements-version: '154811'
    apps.3scale.net/apimanager-threescale-version: 2.15.2
    apps.3scale.net/threescale-operator-version: 0.12.2

It gets stuck on not being external at this stage so that's the reason the upgrade is failing as the message has update.

message: 'Preflights failed - cannot continue with the installation. External databases are required to be set in the APIManager for system Redis, backend Redis and system database - re-running preflights in 10 minutes'

updated the spec for external components and the install completed

metadata:
  annotations:
    apps.3scale.net/apimanager-confirmed-requirements-version: '154811'
    apps.3scale.net/apimanager-threescale-version: 2.16.0
    apps.3scale.net/threescale-operator-version: 0.13.0

austincunningham · 2025-01-24T13:31:24Z

/lgtm

THREESCALE-10740 remove internal databases support

f43996f

MStokluska requested a review from a team as a code owner January 20, 2025 11:23

openshift-ci bot added the do-not-merge/work-in-progress label Jan 20, 2025

MStokluska added 2 commits January 20, 2025 14:26

THREESCALE-10740 updating the requirements and fixing preflight on N …

aa48668

…to N+1 path

THREESCALE-10740 updating preflights psql image to zync image

c1da60a

MStokluska force-pushed the THREESCALE-10740 branch 3 times, most recently from 99c4929 to 0d2bae9 Compare January 21, 2025 11:05

THREESCALE-10740 attempt to fix e2e

27606ac

MStokluska force-pushed the THREESCALE-10740 branch from 0d2bae9 to 27606ac Compare January 21, 2025 13:56

MStokluska added 3 commits January 21, 2025 13:57

THREESCALE-10740 additional fixes including removal of files and zync…

c3eb7e4

… internal bump

THREESCALE-10740 move zync db to sclorg image

bcb78b1

THREESCALE-10740 refactor high availability reconciler

f43a63f

MStokluska force-pushed the THREESCALE-10740 branch from a855fd1 to f43a63f Compare January 21, 2025 18:24

MStokluska changed the title ~~[WIP] THREESCALE-10740 remove internal databases support~~ THREESCALE-10740 remove internal databases support Jan 22, 2025

openshift-ci bot removed the do-not-merge/work-in-progress label Jan 22, 2025

THREESCALE-10740 update docs

7651a80

valerymo reviewed Jan 23, 2025

View reviewed changes

austincunningham reviewed Jan 24, 2025

View reviewed changes

MStokluska merged commit 90a62af into 3scale:master Jan 24, 2025
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

THREESCALE-10740 remove internal databases support #1060

THREESCALE-10740 remove internal databases support #1060

MStokluska commented Jan 20, 2025 •

edited

Loading

valerymo Jan 23, 2025

MStokluska Jan 24, 2025

valerymo commented Jan 23, 2025

valerymo commented Jan 23, 2025

austincunningham Jan 24, 2025

austincunningham Jan 24, 2025

MStokluska Jan 24, 2025

austincunningham commented Jan 24, 2025

austincunningham commented Jan 24, 2025

THREESCALE-10740 remove internal databases support #1060

THREESCALE-10740 remove internal databases support #1060

Conversation

MStokluska commented Jan 20, 2025 • edited Loading

Verification

Local installation

OLM Upgrade with MySQL

valerymo Jan 23, 2025

Choose a reason for hiding this comment

MStokluska Jan 24, 2025

Choose a reason for hiding this comment

valerymo commented Jan 23, 2025

valerymo commented Jan 23, 2025

austincunningham Jan 24, 2025

Choose a reason for hiding this comment

austincunningham Jan 24, 2025

Choose a reason for hiding this comment

MStokluska Jan 24, 2025

Choose a reason for hiding this comment

austincunningham commented Jan 24, 2025

OLM Verification

Mysql & Redis Upgrade

austincunningham commented Jan 24, 2025

MStokluska commented Jan 20, 2025 •

edited

Loading