THREESCALE-11020 Redis TLS certs and keys for porta and backend #1035

valerymo · 2024-11-25T13:09:42Z

Jira: https://issues.redhat.com/browse/THREESCALE-11020

Add a way for the user to provide Redis TLS certs and keys for porta and backend

This PR enables Porta and Apisonator to load TLS configuration details for connecting to Redis. It introduces environment variables to specify the locations of the certificate files and indicate whether TLS mode is enabled.

For additional information, please refer to the Jira ticket and the documentation files included as part of this PR.

NOTES PR includes also integration for Watched-by feature functionality for System and Backend deployments addressed in TLS feature.

Validation

1. Install Redis Server for Test

Create testing project:

   export NAMESPACE=3scale-test
   oc new-project $NAMESPACE

Copy the entire scripts block provided below, open your terminal and paste the script into the command line. This will create the Redis server, including following resources:
- Secret: redis-tls-secret
- ConfigMap: redis-config-redis
- Deployment/pod: redis
- Service: redis


cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: redis-tls-secret
  namespace: 3scale-test
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lVYS9FaGY1UDR3WWVnYVZhaGpqT0R6ME1QVTJnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUl0eld6OHcrTDBOYXAzN0FIcXR4Rjh6OEdXRFROUnoKOUxVa0JYcXU1SkZhc1lxejYrZ2k3S3dsWUlTeEFwVzd4NXprWDZ4bjBCeGxsMXVNdzExZHVyeDJNUjhlYXFxQQpxbElFVzZwTU1kTGNndkhJUit0TGhLRFVvaVpsajdKRkk1MGYvYjhUaXRKVTByY1dhYmd4SVA4QnlINUdxSkR2CmtFZGgvNC9SbTlYeWU4SmhuMEVqRzdPUWxTM1MvOGJHSGpoV2Zjdys2QnJidDI5TE1aTzdJRmFlRVh4azRaQjAKS1VNODJ0TFhxV1VwaUdxQUhuWncrZUtBSlZDOFBzUzB5NU9aTGt4TE9GUlJnajJRYkF3TlZ1emxBN0VMQ0U4dApyVkx3OWV3aEhsVkhUcVZ0UXYwNWZVb2ZqOURkb3Yxd3F2OG1uZXJ3OHBnRVR3Yi9RemY1VGRjQ0F3RUFBYU5UCk1GRXdIUVlEVlIwT0JCWUVGRGpocmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFDdCtPNTZnanZVTUo5MFdaekFFcEFLcTNFTDY3c1I4NXlYam5EeXZML1lRRjZZSzRhUmVaZ1JICms5WCs2cDNyd2tsQmF6MzFFTFNFSGlPUkVTWDJzNW9WSVlId01SNUpsOE5VZTB6NGVBZlNNcTNtQmFuYWxneWYKQktROHZRc1UxTGVJQkpGRVJ5dG5aVE03VThFbFVsUkZSZzAzVUcyVG5ibGNxalFRa1lTMmZYT3M1czBlMHI2eQpaektXVWZhemtzTlkzSHlUcUxRTDlEZUM5STd6R0s2c2E5dHhueTFOM0I5bWJHZ1l3dlFZNDlvTm1kem9oWW16CkxmVVZJeERSWUcwNnI1UldoTTVtZDhZNmJUSGo1TExHT1A2YlEvYWJteXZxbFpuVnZjWTZaandKUzhNWVJJSk8KMmxrczl3cXRjeENla3VJZFBrOUNsNlVBZ3E5TkJoVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBakNDQWVxZ0F3SUJBZ0lVYUIyZGNLRXJuRHM0MEVqMitwT210bmpRQytVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQVAxN2tGaHBRdDhpcUlRWEJOUEozbHBrOFFGUWJnUncKc2VlQkNoa1RTYTd5S0hvQS9VTHVHY3QzZHVubWZPeWdNeEhueW9LUmR0NDVnWWlDOGNnQjE4OWRDRHo2cy84ZAp2TUdoUFUrblhkOWxyVHNtYXpLNW9McnVETmI3TXhBVDFVdjNIMXkvbFErY2tPTkttLzVhWndoNnFxL25XVFZtCkdOZUVGcDJyNUczTkVXcTJXWm1KM0RCc3pMeUdFciszdUlJbFVsaUk2bXRZUGRRd1QwR3RqTDJpVldRVUd2Q2sKYWpHclNJQUVETVFCdFNYeE1yOTlzS2VhS21iVDJ1L09BOEJjYVlqbXdZVnN4VDIzczR2UFltaUhjTFVjVFE4MApTblBLSW03NDJrTzhLZE5Id1hjN1BpejhUTVB0WTU1QkU5ZlBIaGdueVNBMVZUeWNQR1cyZEMwQ0F3RUFBYU5DCk1FQXdIUVlEVlIwT0JCWUVGT3BwbncveVd2dFo2L0llVzhUUlhXeG1uMUtXTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFCWnhOZGpZR1Y4M1RBUwp6amMyNjZnc3gwRTB0cTRVUnRGWHdFeXZTYkxvY3BmK1V2OCtmTFBYZGF0WG1lRm5MdlJZbXg0VGR1UlltMFBYCk1BRUpqcEE3UXdlM1NsSmd5R2loNk1PVVdIRFdNZnhQZWJQWUh5ZFQ3TXBKVVJNWkxjZ0tvWlVENWNKdDc1bVIKMk51Z3BvdDdxQnYybzZGdmI5cmpNYSt1WFdJUEdSekhidjJ6eVFSR1J3SUpZcitKaDNWbXcyRSt0Vit0VEUwQQovWlBxOTIxYnlWYkREZHI2aDMyUnRRN1NJUnAwLzBnWFZablY3NTB2Rm9Ub2xOK2wwLzJOeEVCWmZNSzVFcEdkCkViSFFvN2dETHg0REljZlhqZVlxUkFuYjNLbzhXK0ZrQS8wQUdQS2NSN2RIL2daVnFHMngrRk5QazQ4YzBDQkwKR0R2RjlRYncKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRRDllNUJZYVVMZklxaUUKRndUVHlkNWFaUEVCVUc0RWNMSG5nUW9aRTBtdThpaDZBUDFDN2huTGQzYnA1bnpzb0RNUjU4cUNrWGJlT1lHSQpndkhJQWRmUFhRZzgrclAvSGJ6Qm9UMVBwMTNmWmEwN0ptc3l1YUM2N2d6Vyt6TVFFOVZMOXg5Y3Y1VVBuSkRqClNwditXbWNJZXFxdjUxazFaaGpYaEJhZHErUnR6UkZxdGxtWmlkd3diTXk4aGhLL3Q3aUNKVkpZaU9wcldEM1UKTUU5QnJZeTlvbFZrRkJyd3BHb3hxMGlBQkF6RUFiVWw4VEsvZmJDbm1pcG0wOXJ2emdQQVhHbUk1c0dGYk1VOQp0N09MejJKb2gzQzFIRTBQTkVwenlpSnUrTnBEdkNuVFI4RjNPejRzL0V6RDdXT2VRUlBYeng0WUo4a2dOVlU4Cm5EeGx0blF0QWdNQkFBRUNnZ0VBRG5vaktWbUJyenJNZ3hiSmVNc2J2dS9xNzlkSElVdktiVjFxVlRwTHlBa2UKbExFL3hiWFJsVlJTWDFPQnFRWVJSS0dIYUdPa2RWYTFkalY4VjU3N1UyV04xZVcvcC85cnkyZEpHQ2FIN3YxZwpvbk0wUmlaaDdxc3Y0b3RnUkRmTnc5UHVYNTYxaGJtOGNLN1BML3k3eTdrdHpIUWJIVGlpakpTSHNpT2lIVDh1CitWc3oySzZmSnhHNUE2SE03bldvdlBuK1VydFd4OGd4OEJNQ1FOa3dUVnQvVmxzcE1wcUo1czdqeUNncGFQN2MKV0wrWnh2K2xUcFUwcnpJUTFZMWYvL0RSeUI3OVEycUl0dVhEdkJnNnI5ckExbytSV0JuNnA3WVA0T21SUE4vZQpiMVZZcG14c1BBL2ppQkhsRVFPZVJWbEVSYzhJSGVib2IxZFZ1Rm15b1FLQmdRRC9LeFlvVzNxaGNmUEYya2EvCmliZTkvKzlyQzNlTkdHUGwyaDVCeDJwK3k4dnlQb3NabmZ5STRKRzNtcXpVS1luSHFPVzhpTVpnR2ZOUlRNdVQKeVFSQitsVlFldHpOcDJUZCt4dWlCMUJmWHIyanowcVZvV3F4dFhrMklRWXFJN2NMSGZVZ2dGeUtsL2lRRGZlOQpwd0YwZ3c5Nm1vT012VWp1S1FmZjh2RTYxUUtCZ1FEK1R4SWZtc1VyY2VFdmpkeUdhNllweTBZcEx2OVhnSk0rCkxIc1ZmVTgzNXNWeVJaUDZrV1EwSGwyVCtHMDBqcStsU20vQ296cWNZeE5ObmtGMEhOdEw2SlJJWTBZZEExSFUKSmc1eTY3U2w0U0hHeWdYNTdrZEE4ajhyS1I4UXpmQ0RHT3Q5d2NwNjZ4MVlMSWEwRDYvV2h3elFOeXl3QzdnTwoxS3dvZlhyUCtRS0JnRzU4WUk2KzlYMWNVdnBUaGhpL2IvRDBGZDNhekR3cTJHNlpJRXJKSndLYUNjZnRidHQ3CnZmSWlrdFhXUW9sbkp3SnR6blB4SVR4UllEck9yc05oNGRjVHBzYy9POFpNZWU5b0lGSHJLdER3dTlwbkVsdHgKMWpuMll2S2VJQVkxQ3Jma2s5UXI0R1llWVlFMm14UGljVTNheGVRSGJYaU9LVHIrUnl1Z0RQVzFBb0dBZm5vdwoxMHNRR0tWUWkyZ1FiMElHcCs2Uy9GU0ZaYTFxalpkdHQ2aFV4OGFjR0ZNR1g2NERtZkFvTmpsdGhxQVlOeXFvCkhyTXpxU2VWS0JzM0RscHpybk1EbkdUVE1BYkFvYlF6cDNBV3JoRWp6VXdZWU03aTNTZ2R4b2R6RGRaK2NaVHAKT2VneG5hUmxPYjhiVjE0ZDQ2SFMrNU1WUkpEdmYyRENKbmtScFhFQ2dZRUFnVFB3MlVaSkJRVVdGc0NVRjczdgpGcWZSbzdZYmtweDAxcFlBcmV0ak41TTR6ZHZRUlZPQlFmVGRJTU9SbFJ4OVBnRitYL3UrT0szUzB1aS8xdWlECjhYU2QvMXFjS3ZOdUJ6ZnZNdTVGeVdKRHNCUGJaUHZuWnEzdTM5WktaYTNXNG92UE9VWlFERGlCTHNhM21CekwKcFdYNWFyVHNTazRSSDExQ28zRTN0akU9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
type: Opaque
EOF


cat << EOF | oc create -f -
apiVersion: v1
data:
  redis.conf: |+
    # redis.conf
    bind 0.0.0.0
    protected-mode no
    port 6379
    tls-port 6380
    tls-cert-file /etc/redis/certs/redis-server.crt
    tls-key-file /etc/redis/certs/redis-server.key
    tls-ca-cert-file /etc/redis/certs/ca.crt
    tls-auth-clients yes
    stop-writes-on-bgsave-error no
    save ""
kind: ConfigMap
metadata:
  name: redis-config-redis
EOF


cat << EOF | oc create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: quay.io/fedora/redis-6
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: redis-config-volume
          mountPath: /etc/redis/redis.conf
          subPath: redis.conf
        - name: redis-tls-volume
          mountPath: /etc/redis/certs
          readOnly: true
        command: ["/bin/sh", "-c", "redis-server /etc/redis/redis.conf"]
      volumes:
      - name: redis-config-volume
        configMap:
          name: redis-config-redis
      - name: redis-tls-volume
        secret:
          secretName: redis-tls-secret
EOF


cat << EOF | oc create -f -
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  ports:
    - port: 6379         # Non-TLS (unencrypted) port
      targetPort: 6379
      name: redis
    - port: 6380         # TLS port
      targetPort: 6380
      name: redis-tls
  selector:
    app: redis
  type: NodePort 
EOF

Expecting results example: redis server pod is running and service available

$ oc get svc
NAME    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
redis   NodePort   172.30.56.166   <none>        6379:31290/TCP,6380:32389/TCP   10m

$ oc get pod
NAME                     READY   STATUS    RESTARTS   AGE
redis-5dc466fc8b-764hl   1/1     Running   0          10m

2. Certificates preparing

- Create CA, Client and Server Certificates, using Server IP as Common Name (CN): - Create directory `Certs`, `cd Certs`, and run following commands to create server and client certificates, that will be used for test.

openssl genpkey -algorithm RSA -out ca.key
openssl req -x509 -key ca.key -out ca.crt -days 365 -subj "/CN=172.30.56.166"

openssl genpkey -algorithm RSA -out redis-client.key
openssl req -new -key redis-client.key -out redis-client.csr -subj "/CN=redis-client.example.com"
openssl x509 -req -in redis-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-client.crt -days 365

openssl genpkey -algorithm RSA -out redis-server.key
openssl req -new -key redis-server.key -out redis-server.csr -subj "/CN=172.30.56.166"
openssl x509 -req -in redis-server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-server.crt -days 365

Expected files to be created:

Certs $ ls
ca.crt			ca.srl			redis-client.csr	redis-server.crt	redis-server.key
ca.key			redis-client.crt	redis-client.key	redis-server.csr

3. Update Redis Server with new server certificate

Update redis-tls-secret secret, using new created:
- ca.crt
- redis-server.crt
- redis-server.key
Restart redis pod

4. Install 3scale

#### 4.1. Create Redis secrets for 3scale

- Below is a script to create the system-redis and backend-redis secrets with dummy client certificates. In the next step, we will replace these dummy certificates with valid client certificates, using the UI for convenience. The valid certificates will be sourced from the files created in the previous section. Additionally, we will update the Redis server URL to reflect the service IP of our Redis server.

You may prepare the secrets in whichever way is most convenient for you.

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: system-redis
  namespace: 3scale-test
  labels:
    apimanager.apps.3scale.net/watched-by: system
    app: 3scale-api-management
    threescale_component: system
data:
  SENTINEL_HOSTS: ''
  SENTINEL_ROLE: ''
  REDIS_SSL_CA: cmVwbGFjZW1lCg==
  REDIS_SSL_CERT: cmVwbGFjZW1lCg==
  REDIS_SSL_KEY: cmVwbGFjZW1lCg==
  URL: cmVkaXM6Ly8xNzIuMzAuMjA3LjE5MTo2MzgwLzE=
type: Opaque
EOF

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: backend-redis
  namespace: 3scale-test
  labels:
    apimanager.apps.3scale.net/watched-by: backend
    app: 3scale-api-management
    threescale_component: backend
data:
  REDIS_STORAGE_URL: cmVkaXM6Ly8xNzIuMzAuMjA3LjE5MTo2MzgwLzA=
  REDIS_QUEUES_SENTINEL_HOSTS: ''
  REDIS_STORAGE_SENTINEL_ROLE: ''
  REDIS_SSL_CA: cmVwbGFjZW1lCg==
  REDIS_SSL_CERT: cmVwbGFjZW1lCg==
  REDIS_SSL_KEY: cmVwbGFjZW1lCg==
  REDIS_SSL_QUEUES_CA: cmVwbGFjZW1lCg==
  REDIS_SSL_QUEUES_CERT: cmVwbGFjZW1lCg==
  REDIS_SSL_QUEUES_KEY: cmVwbGFjZW1lCg==
  REDIS_QUEUES_URL: cmVkaXM6Ly8xNzIuMzAuMjA3LjE5MTo2MzgwLzE=
  REDIS_QUEUES_SENTINEL_ROLE: ''
  REDIS_STORAGE_SENTINEL_HOSTS: ''
type: Opaque
EOF

Update Client Certificates in system-redis and backedn-redis secrets via UI. The following tables are for matching data field names with the certificate files created before:
Secret: system-redis

Data field	Certificate file name
REDIS_SSL_CA	ca.crt
REDIS_SSL_CERT	redis-client.crt
REDIS_SSL_KEY	redis-client.key

Secret: backend-redis

Data field	Certificate file name
REDIS_SSL_CA	ca.crt
REDIS_SSL_CERT	redis-client.crt
REDIS_SSL_KEY	redis-client.key
REDIS_SSL_QUEUES_CA	ca.crt
REDIS_SSL_QUEUES_CERT	redis-client.crt
REDIS_SSL_QUEUES_KEY	redis-client.key

Please note:

We are using a common CA for both the Redis server and client certificates.
We are using the same client certificates for both the Redis system and the backend, including QUEUES.
Please don't forget to update the Redis IP in the URLs to match the service IP and CN.
Use secure Redis URLs. They should look like rediss://:6380/0, where rediss indicates a secure connection and port 6380 is used for secure Redis connections. For example:
- REDIS_QUEUES_URL: rediss://172.30.56.166:6380/1
- REDIS_STORAGE_URL: rediss://172.30.56.166:6380/0

4.2. Create s3-credentials secret

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata: 
  name: s3-credentials
  namespace: 3scale-test
data: 
  AWS_ACCESS_KEY_ID: QUtJQVY2SVpYMk9ZQ09OWERFSlkK
  AWS_SECRET_ACCESS_KEY: aU5VbWdZY3hjSDF3azBlUlB0SytmTERHVVMvU0hxM1pKNVBlQy9xYQo=
  AWS_BUCKET: dm1vY2NzZjZxOGxyZWRoYXRyaG9hbW9wZXJhdG9ydGhyZWUtYW9mZwo=
  AWS_REGION: ZXUtd2VzdC0xCg==
type: Opaque
EOF

4.3. Run Install and Downloads commands

To allow using External Redis:

export PREFLIGHT_CHECKS_BYPASS=true

Run Install and Downloads commands
Please make sure you are in 3scale-operator directory, and run following commands:

make install
make download

4.4. Create APIManager CR and Run Operator

Please set wildcardDomain before creation APIManager:

cat << EOF | oc create -f -
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: example-apimanager
spec:
  redisTLSEnabled: true
  system: 
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  wildcardDomain: <apps.xxx.xxx.xx.devshift.org>
  externalComponents:
    backend:
      redis: true
    system:
      redis: true 
EOF

run Operator to install 3scale:
```
make run
```

5 Check results and Troubleshooting

5.1 Check Environment Variables and Certificates in Pods

- Check that env vars are defined in pods, check cert files:

System pods: system-sidekiq and system-app

Run the following commands to verify that the certificate environment variables are defined and the certificate files are populated for the System:

env |grep -E "REDIS_CLIENT_CERT|BACKEND_REDIS_CLIENT_CERT"
env |grep -E "REDIS_CA_FILE|BACKEND_REDIS_CA_FILE"
env |grep -E "REDIS_PRIVATE_KEY|BACKEND_REDIS_PRIVATE_KEY"
env |grep -E "REDIS_SSL|BACKEND_REDIS_SSL"

cat /tls/system-redis/system-redis-ca.crt
cat /tls/system-redis/system-redis-client.crt
cat /tls/system-redis/system-redis-private.key
cat /tls/backend-redis-ca.crt
cat /tls/backend-redis-client.crt
cat /tls/backend-redis-private.key

Example of system environment variable checking and the expected output:

sh-5.1$ env |grep -E "REDIS_CLIENT_CERT|BACKEND_REDIS_CLIENT_CERT"
env |grep -E "REDIS_CA_FILE|BACKEND_REDIS_CA_FILE"
env |grep -E "REDIS_PRIVATE_KEY|BACKEND_REDIS_PRIVATE_KEY"
env |grep -E "REDIS_SSL|BACKEND_REDIS_SSL"
REDIS_CLIENT_CERT=/tls/system-redis/system-redis-client.crt
BACKEND_REDIS_CLIENT_CERT=/tls/backend-redis-client.crt
REDIS_CA_FILE=/tls/system-redis/system-redis-ca.crt
BACKEND_REDIS_CA_FILE=/tls/backend-redis-ca.crt
REDIS_PRIVATE_KEY=/tls/system-redis/system-redis-private.key
BACKEND_REDIS_PRIVATE_KEY=/tls/backend-redis-private.key
REDIS_SSL=1
BACKEND_REDIS_SSL=1
sh-5.1$

Backend pods:backend-cron, backend-listener, backend-worker
Run the following commands to verify that the certificate environment variables are defined and the certificate files are populated for the Backend:

env |grep -E "CONFIG_REDIS_CA_FILE|CONFIG_QUEUES_CA_FILE"  
env |grep -E "CONFIG_REDIS_CERT|CONFIG_QUEUES_CERT"
env |grep -E "CONFIG_REDIS_PRIVATE_KEY|CONFIG_QUEUES_PRIVATE_KEY"
env |grep -E "CONFIG_REDIS_SSL|CONFIG_QUEUES_SSL"

cat /tls/config-queues-ca.crt
cat /tls/config-queues-client.crt
cat /tls/config-queues-private.key
cat /tls/backend-redis-ca.crt
cat /tls/backend-redis-client.crt
cat /tls/backend-redis-private.key

Example of backend environment variable checking and the expected output:

sh-4.4$ env |grep -E "CONFIG_REDIS_CA_FILE|CONFIG_QUEUES_CA_FILE"       
CONFIG_QUEUES_CA_FILE=/tls/config-queues-ca.crt
CONFIG_REDIS_CA_FILE=/tls/backend-redis-ca.crt
sh-4.4$                                                          
sh-4.4$ env |grep -E "CONFIG_REDIS_CERT|CONFIG_QUEUES_CERT"      
CONFIG_QUEUES_CERT=/tls/config-queues-client.crt
CONFIG_REDIS_CERT=/tls/backend-redis-client.crt
sh-4.4$ 
sh-4.4$ env |grep -E "CONFIG_REDIS_PRIVATE_KEY|CONFIG_QUEUES_PRIVATE_KEY"
CONFIG_REDIS_PRIVATE_KEY=/tls/backend-redis-private.key
CONFIG_QUEUES_PRIVATE_KEY=/tls/config-queues-private.key
sh-4.4$ 
sh-4.4$ env |grep -E "CONFIG_REDIS_SSL|CONFIG_QUEUES_SSL"
CONFIG_QUEUES_SSL=1
CONFIG_REDIS_SSL=1
sh-4.4$

5.2. Troubleshooting

If you encounter issues with the Redis server, one approach is to test the connection using the throwaway-redis pod, which includes redis-cli . This simplifies the process of checking TLS connections to Redis.

5.2.1 Using the throwaway-redis Pod

In this example, we assume that the throwaway-redis pod is not yet present in the 3scale-test project. Since we used the export PREFLIGHT_CHECKS_BYPASS=true environment variable earlier, the pod needs to be created; follow the steps below:

Create the throwaway-redis pod:
- Stop the Operator: Press Ctrl+C.
- Unset the preflight environment variable:
```
unset PREFLIGHT_CHECKS_BYPASS
```
- Restart the Operator:
```
make run
```
- In a separate terminal, check that the throwaway-redis pod is created.
- Stop the Operator again once the pod is created.
- Set the preflight bypass variable:
```
export PREFLIGHT_CHECKS_BYPASS=true
```
- Restart the Operator:
```
make run
```
Copy certificates to the throwaway-redis pod:
- Navigate to the directory where you saved the certificate files, then run the following commands to copy them to the pod:
```
oc cp ca.crt throwaway-redis:/tmp
oc cp redis-client.crt throwaway-redis:/tmp
oc cp redis-client.key throwaway-redis:/tmp        
```

Login to the throwaway-redis pod and test the Redis connection using redis-cli:

Use the following command to access the throwaway-redis pod:
```
oc rsh throwaway-redis
```

Once inside the pod, verify that the certificates are correctly copied:

sh-5.1$ ls -l /tmp
total 12
-rw-r--r--. 1 redis root 1123 Dec 2 14:40 ca.crt
-rw-r--r--. 1 redis root 1115 Dec 2 14:40 redis-client.crt
-rw-------. 1 redis root 1704 Dec 2 14:40 redis-client.key

- Now, attempt to connect to the Redis server using the redis-cli command with TLS enabled:

  ```
  sh-5.1$ redis-cli -h 172.30.56.166 --tls -p 6380 --cert redis-client.crt --key redis-client.key --cacert ca.crt ping   
  ```
  If successful, you should see the response:
  ```
  PONG
  ``` 
- Possible Error:
If you encounter the following error:
  ```
  Could not negotiate a TLS connection: Invalid CA Certificate File/Directory    
  ```
    - Resolution:  
Ensure that the certificates on both the server side and in the throwaway-redis pod are up-to-date and correctly populated.

Check non-TLS communication from the throwaway-redis pod:
You can also check the Redis connection without TLS to verify basic connectivity. Run the following:
```
oc rsh throwaway-redis
cd /tmp
sh-5.1$ redis-cli -h 172.30.56.166 -p 6379 ping 
```

If the connection is successful, you should see:
PONG
If the non-TLS connection works, but the TLS connection does not, consider updating the server certificates. To do this:
- Update the Redis testing server's secret: redis-tls-secret.
- Recreate the Redis testing server pod:
redis-5dc466fc8b-vxpbh 1/1 Running 0 4s

Recheck the TLS connection from throwaway-redis pod:
After updating the server certificates and recreating the Redis pod, attempt the TLS connection again from the throwaway-redis pod. This time it looks good:
sh-5.1$ redis-cli -h 172.30.56.166 --tls -p 6380 --cert redis-client.crt --key redis-client.key --cacert ca.crt ping
The expected response should now be:

PONG

5.2.2 More things to check and useful commands

check Logs in Init container of system-sidekiq pod:

oc logs system-sidekiq-7796796778-sw7p4 -c check-svc
Connected to rediss://172.30.56.166:6380/1
Connected to rediss://172.30.56.166:6380/1
Connected to rediss://172.30.56.166:6380/1
Connected to rediss://172.30.56.166:6380/1

Connect to Init container of system-sidekiq pod:

kubectl exec -it  system-sidekiq-7796796778-sw7p4 -c check-svc  -- /bin/sh
sh-4.4$ 

sh-4.4$ env |grep tls   
BACKEND_REDIS_PRIVATE_KEY=/tls/backend-redis-private.key
REDIS_CLIENT_CERT=/tls/system-redis/system-redis-client.crt
REDIS_CA_FILE=/tls/system-redis/system-redis-ca.crt
BACKEND_REDIS_CA_FILE=/tls/backend-redis-ca.crt
REDIS_PRIVATE_KEY=/tls/system-redis/system-redis-private.key
BACKEND_REDIS_CLIENT_CERT=/tls/backend-redis-client.crt

sh-4.4$ env |grep REDIS_SSL
REDIS_SSL=1
BACKEND_REDIS_SSL=1

6 TLS - Test Cases, Documentation

6.1 Test Cases

In the previous sections, we provided details for testing scenarios where the APIManager CR options are as follows: redisTLSEnabled: true and both Redis external components are enabled (set to true), as shown in the following CR example:

 apiVersion: apps.3scale.net/v1alpha1
 kind: APIManager
 metadata:
   name: example-apimanager
 spec:
   redisTLSEnabled: true
   externalComponents:
     backend:
       redis: true
     system:
       redis: true

However, we conducted additional tests, and below is a table listing all test cases and scenarios that were tested for the current PR

N	Test case / Scenario for redis system/backend	All Running	admin portal opened	Comment
1	TLS flag + External: system = true, backend = true	+	+
2	TLS flag + External: system = true, backend= false	+	+	*
3	TLS flag + External: system = false, backend = true	+	+
4	TLS flag + Internal: system = false, backend = false	+	+	not relevant now
5	No TLS flag + External: system = true,backend = true	+	+
6	No TLS flag + No externalComponents	+	+	not relevant now

*Test 2. System-app, system-sidekiq are running in TLS mode even if Backend certificates are dummy ("replaceme")

6.2 TLS - Documentation

Please see documentation updated in this PR - operator-user-guide - Setting Redis TLS Environment variables

7. Check Watched-by Integration for Backend and System

- Labels should be set on Secrets:

~ oc describe secret system-redis |grep watched
Labels:       apimanager.apps.3scale.net/watched-by=system
~ oc describe secret backend-redis |grep watched
Labels:       apimanager.apps.3scale.net/watched-by=backend

Check pods:

oc describe deploy backend-cron |grep Anno |grep secret-resource
oc describe deploy backend-listener |grep Anno |grep secret-resource
oc describe deploy backend-worker |grep Anno |grep secret-resource
oc describe deploy backend-cron |grep Anno |grep secret-resource
oc describe deploy system-app |grep Anno |grep secret-resource
oc describe deploy system-sidekiq |grep Anno |grep secret-resource

Expected result - watched-by annotation found in deployments:

 Annotations:      apimanager.apps.3scale.net/backend-redis-secret-resource-version: 1606453
  Annotations:      apimanager.apps.3scale.net/backend-redis-secret-resource-version: 1606453
  Annotations:      apimanager.apps.3scale.net/backend-redis-secret-resource-version: 1606453
  Annotations:      apimanager.apps.3scale.net/backend-redis-secret-resource-version: 1606453
  Annotations:      apimanager.apps.3scale.net/system-redis-secret-resource-version: 1606382
  Annotations:      apimanager.apps.3scale.net/system-redis-secret-resource-version: 1606382

austincunningham · 2024-12-05T08:56:27Z

pkg/3scale/amp/component/system.go

+
+	SystemSecretSystemRedisSslCa   = "SSL_CA"
+	SystemSecretSystemRedisSslCert = "SSL_CERT"
+	SystemSecretSystemRedisSslKey  = "SSL_KEY"


@valerymo think we need to agree on a naming scheme here as it may be possible that a customer has different certs for Redis and the System-db. Propose that we prefix with CACHE_ as may not be using redis going forward

Suggested change

SystemSecretSystemRedisSslKey = "SSL_KEY"

SystemSecretSystemRedisSslCa = "CACHE_SSL_CA"

SystemSecretSystemRedisSslCert = "CACHE_SSL_CERT"

SystemSecretSystemRedisSslKey = "CACHE_SSL_KEY"

and I use the prefix SYSTEMDB_

Suggested change

SystemSecretSystemRedisSslKey = "SSL_KEY"

SystemSecretSslCa = "SYSTEMDB_SSL_CA"

SystemSecretSslCert = "SYSTEMDB_SSL_CERT"

SystemSecretSslKey = "SYSTEMDB_SSL_KEY"

What do you think ?

@austincunningham thanks for comment.
It's Done,

"REDIS_" prefix was added to "SSL" names that appear in backend-redis and system-redis secrtes

done in separate commit

Tested locally

Validation notes updated
Thanks

MStokluska · 2025-01-10T09:38:45Z

apis/apps/v1alpha1/apimanager_types.go

@@ -1535,3 +1537,26 @@ type APIManagerList struct {
 func init() {
 	SchemeBuilder.Register(&APIManager{}, &APIManagerList{})
 }
+
+func (apimanager *APIManager) IsSystemRedisTLSEnabled() bool {


If the RedisTLSEnabled is a global flag, why do we make a differentiation between system and backend redis here?

Can you have system redis TLS enabled but not worker TLS enabled? Doesn't system app connect to system redis and backend redis - which would to me mean, you can't have TLS on system and not on worker? Not sure about the other way around

Thank you for comment @MStokluska . We can't have TLS on System if no TLS on Backend. But it can be vice-a-versa - TLS on Backend but not on System. This is what I see from Jira - System is depended to Backend (system has env vars from backend-redis secret), but Backend is not related to system-redis secret. Hope it's ok. But please tell me if I need check it from Joan. Thanks

So based on what you are saying:
Backend TLS - ON and System TLS - OFF is supported.
Backend TLS - OFF and System TLS - ON is not supported.

Based on the Jira description, it looks to me like they actually can be enabled independent of each other since backend is not reliant on system (meaning from backend pov, system SSL can be on or off). System is reliant on backend redis, but there seems to be individual flags for system/backend in system envs:
BACKEND_REDIS_SSL - REDIS_SSL.
Do you agree? Is this how it is currently working?

Yes, this is how it's currently working.

ok thanks, do we have any indication (apim error or so) to let users know that tls on backend must be enabled if system tls is enabled?

MStokluska · 2025-01-10T09:45:57Z

pkg/3scale/amp/component/backend.go

@@ -398,6 +437,9 @@ func (backend *Backend) buildBackendWorkerEnv() []v1.EnvVar {
 		)
 	}

+	if backend.Options.RedisTLSEnabled {


we already have these in buildBackendCommonEnv - common in my understanding applies to both, worker and listener - is this really required?

Fixed. Thank you

MStokluska · 2025-01-10T09:46:03Z

pkg/3scale/amp/component/backend.go

@@ -422,6 +464,9 @@ func (backend *Backend) buildBackendListenerEnv() []v1.EnvVar {
 			v1.EnvVar{Name: "CONFIG_LISTENER_PROMETHEUS_METRICS_ENABLED", Value: "true"},
 		)
 	}
+	if backend.Options.RedisTLSEnabled {


we already have these in buildBackendCommonEnv - common in my understanding applies to both, worker and listener - is this really required?

Fixed. Thank you

MStokluska · 2025-01-10T09:49:30Z

pkg/3scale/amp/component/backend.go

+func (backend *Backend) backendVolumes() []v1.Volume {
+	res := []v1.Volume{}
+
+	if backend.Options.RedisTLSEnabled {


just side note - so are we saying here that Redis TLS can be enabled on worker or listener IF system fails to have TLS enabled?

System TLS certs are coming from both system-redis and backend-redis secrets. System TLS envs BACKEND_REDIS_CA_FILE, BACKEND_REDIS_PRIVATE_KEY - they are populated from Backend-redis secret. So System is depends on Backend TLS definitions.
But Backend TLS is not related to system - all certs are coming from backend-redis.
Hope I understand the quesion. Thank you Michal

Yes. I wrote before, as I understand - it can be TLS on backend, but not on system. Backend TLS is not dependent to system, but system TLS is dependent to backend. System is populating BACKEND_REDIS_CLIENT_CERT and BACKEND_REDIS_PRIVATE_KEY from backend-redis secret.

pkg/3scale/amp/component/redis.go

pkg/3scale/amp/component/redis_options.go

pkg/3scale/amp/component/system.go

MStokluska · 2025-01-10T10:17:25Z

pkg/3scale/amp/operator/redis_options_provider.go

@@ -69,12 +70,14 @@ func (r *RedisOptionsProvider) GetRedisOptions() (*component.RedisOptions, error
 	r.setPersistentVolumeClaimOptions()
 	r.setPriorityClassNames()
 	r.setTopologySpreadConstraints()
+	//r.setTLSEnabled()


is this required?

cleaned. Thank you

pkg/3scale/amp/operator/redis_options_provider.go

austincunningham · 2025-01-10T10:25:46Z

doc/apimanager-reference.md

+| REDIS_SSL_KEY | The private key for the Redis client certificate | Required to set TLS Redis connection. Only for TLS                                                                |
+| REDIS_SSL_QUEUES_CA | Redis Queues Certificate Authority (CA) certificate | Required to set TLS Redis connection. Only for TLS                                                                |
+| REDIS_SSL_QUEUES_CERT | Redis Queues client certificate | Required to set TLS Redis connection. Only for TLS                                                                |
+| REDIS_SSL_QUEUES_KEY | The private key for the Redis Queues client certificate | Required to set TLS Redis connection. Only for TLS                                                                |


Are you missing SSL_MODE ?

hey @austincunningham ,
I'm not using SSL_MODE but using REDIS_SSL and BACKEND_REDIS_SSL (for system), CONFIG_REDIS_SSL and CONFIG_QUEUES_SSL (for backend),
as it defined in Task.
These env vars are set in Pods, but not in secret.
They set to true if Any of related env vars defined in secret. This is how it's defined in Task/Requirements,
I rechecked it with Joan - thread https://app.slack.com/client/E030G10V24F/search).
These are also links to porta/backend:
System/porta:
- BACKEND_REDIS_SSL: https://github.com/3scale/porta/blob/master/config/examples/backend_redis.yml#L14
- REDIS_SS: https://github.com/3scale/porta/blob/master/config/examples/redis.yml#L13
Backend/apisonator:
- CONFIG_REDIS_SSL: https://github.com/3scale/apisonator/blob/master/openshift/3scale_backend.conf#L44
- CONFIG_QUEUES_SSL: https://github.com/3scale/apisonator/blob/master/openshift/3scale_backend.conf#L27

Not sure that is the same as ssl mode.

Although not sure if the same tls values are available for REDIS as are available for MySQL and Postgres

🤔

DATABASE_SSL_MODE. Values: Mysql ref., Postgres ref.

Don't see similar documented in redis docs so guess its ok

@austincunningham , I'm not using SSL_MODE but using REDIS_SSL and BACKEND_REDIS_SSL (for system), CONFIG_REDIS_SSL and CONFIG_QUEUES_SSL (for backend),
as it defined in Task.
These env vars are set in pods, but not in secret. And these env vars are used in Porta and apisonator (links below)
These env vars are set to true if Any of related env vars defined in secret.
I rechecked it with Joan - thread https://app.slack.com/client/E030G10V24F/search).
These are also links to porta/backend:
System/porta:
- BACKEND_REDIS_SSL: https://github.com/3scale/porta/blob/master/config/examples/backend_redis.yml#L14
- REDIS_SS: https://github.com/3scale/porta/blob/master/config/examples/redis.yml#L13
Backend/apisonator:
- CONFIG_REDIS_SSL: https://github.com/3scale/apisonator/blob/master/openshift/3scale_backend.conf#L44
- CONFIG_QUEUES_SSL: https://github.com/3scale/apisonator/blob/master/openshift/3scale_backend.conf#L27
Thank you

austincunningham · 2025-01-14T10:20:06Z

doc/apimanager-reference.md

-| AppLabel | `appLabel` | string | No | `3scale-api-management` | The value of the `app` label that will be applied to the API management solution
-| TenantName | `tenantName` | string | No | `3scale` | Tenant name under the root that Admin UI will be available with -admin suffix.
+| **Field** | **json/yaml field**| **Type** | **Required** | **Default value** | **Description**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| --- | --- | --- | --- | --- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|


goland does this sort of auto edit, I usually edit md files in another ide to stop this.

@austincunningham , Thanks a lot! I updated sections related to Redis TLS . I used now VSCodium.
Looks like there are more places where extra spaces added, but didn't touch them. Table of contents updated, as VSCodium required this, I checked, seems it's working fine.

austincunningham · 2025-01-14T10:33:40Z

doc/operator-user-guide.md

+  - Client Private Key
+- TLS certificate files are populated from the **backend-redis** and **system-redis** secrets.
+
+- The tables below show the mapping between TLS certificate environment variables in the pods, their corresponding definitions in the related Redis backend and system secrets.


Think we should only be documenting the ENV VAR the user has to set and not the ones set my the operator to avoid confusion

@austincunningham thank you for comment. It's User Guide. I thought it could help to User to understand how it's working, and investigate/debug if required. If you strongly disagree I will remove this, but seems to me it can be useful. Thanks

MStokluska · 2025-01-23T12:36:40Z

There is a new functionality added to resync-routes when zync is enabled.
This causes a job to be created, resync-route.
The job keeps failing due to:

Failed to load CA Certificate or CA Path
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:37:in `initialize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:24:in `new'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:24:in `ssl_context'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client/config.rb:125:in `ssl_context'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:134:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:50:in `initialize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:746:in `new'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:746:in `block in connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client/middlewares.rb:12:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:745:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:732:in `raw_connection'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:697:in `ensure_connected'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:292:in `call_v'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis/client.rb:90:in `call_v'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:152:in `block in send_command'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:151:in `synchronize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:151:in `send_command'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis/commands/connection.rb:21:in `ping'
/opt/system/lib/tasks/boot.rake:26:in `block (3 levels) in <top (required)>'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:95:in `with'
/opt/system/app/lib/system/redis_pool.rb:26:in `public_send'
/opt/system/app/lib/system/redis_pool.rb:26:in `block in method_missing'

MStokluska · 2025-01-23T12:43:35Z

The secrets when TLS is enabled are annotated with:
apimanager.apps.3scale.net/watched-by: system and apimanager.apps.3scale.net/watched-by: backend
This doesn't work.

To enable watched-by you need apimanager.apps.3scale.net/watched-by: apimanager

MStokluska · 2025-01-23T12:55:53Z

In backend-redis, we have redis queues and redis storage entries - think we could maintain that?
Or does it have to be now, redis queues and redis?

valerymo · 2025-01-23T13:06:21Z

In backend-redis, we have redis queues and redis storage entries - think we could maintain that? Or does it have to be now, redis queues and redis?

It's in requirements, as in Jira, if I understand the question :
Backend:
CONFIG_REDIS_CA_FILE and CONFIG_QUEUES_CA_FILE for the CA certificate
CONFIG_REDIS_CERT and CONFIG_QUEUES_CERT for the client certificate
CONFIG_REDIS_PRIVATE_KEY and CONFIG_QUEUES_PRIVATE_KEY for the client key
CONFIG_REDIS_SSL and CONFIG_QUEUES_SSL must be set to true when any of the above is present

MStokluska · 2025-01-23T13:10:14Z

In backend-redis, we have redis queues and redis storage entries - think we could maintain that? Or does it have to be now, redis queues and redis?

It's in requirements, as in Jira, if I understand the question : Backend: CONFIG_REDIS_CA_FILE and CONFIG_QUEUES_CA_FILE for the CA certificate CONFIG_REDIS_CERT and CONFIG_QUEUES_CERT for the client certificate CONFIG_REDIS_PRIVATE_KEY and CONFIG_QUEUES_PRIVATE_KEY for the client key CONFIG_REDIS_SSL and CONFIG_QUEUES_SSL must be set to true when any of the above is present

I think I would change that to be clear and follow what we have, alternatively, bring it up in jira please

MStokluska · 2025-01-23T13:14:02Z

When enabling TLS with wrong certs (I've tried with storage) what happens is that although the authentication in backend listener fails, the products can be created and promoted to stage/prod.
Auth fails on api requests.
Then, when fixing up the certs, the requests are still failing. I believe this requires a redis re-sync?

I guess, porta does some validation right, so maybe an idea here would be to restart system-app pod on every update of the TLS secrets to perform the validation (and hopefully catch the incorrect certs and block api access?)

WDYT?

valerymo · 2025-01-23T13:16:23Z

The secrets when TLS is enabled are annotated with: apimanager.apps.3scale.net/watched-by: system and apimanager.apps.3scale.net/watched-by: backend This doesn't work.

To enable watched-by you need apimanager.apps.3scale.net/watched-by: apimanager

@MStokluska It's working, I tested watched-by in my PR. The reason why it's working is here - checked only key of lable, not value. But Secret itself it's created by end-user, so actually nothing change in code, as I think.

valerymo · 2025-01-23T13:22:31Z

When enabling TLS with wrong certs (I've tried with storage) what happens is that although the authentication in backend listener fails, the products can be created and promoted to stage/prod. Auth fails on api requests. Then, when fixing up the certs, the requests are still failing. I believe this requires a redis re-sync?

I guess, porta does some validation right, so maybe an idea here would be to restart system-app pod on every update of the TLS secrets to perform the validation (and hopefully catch the incorrect certs and block api access?)

WDYT?

@MStokluska , all backend pods (listener, worker, cron) and system (app and sidekiq) are restarting if any change in secrets system-redis and backend-redis (that hold certificates), as whatched-by is available on backend-redis and system-redis

valerymo · 2025-01-23T13:40:47Z

In backend-redis, we have redis queues and redis storage entries - think we could maintain that? Or does it have to be now, redis queues and redis?

It's in requirements, as in Jira, if I understand the question : Backend: CONFIG_REDIS_CA_FILE and CONFIG_QUEUES_CA_FILE for the CA certificate CONFIG_REDIS_CERT and CONFIG_QUEUES_CERT for the client certificate CONFIG_REDIS_PRIVATE_KEY and CONFIG_QUEUES_PRIVATE_KEY for the client key CONFIG_REDIS_SSL and CONFIG_QUEUES_SSL must be set to true when any of the above is present

I think I would change that to be clear and follow what we have, alternatively, bring it up in jira please

@MStokluska , will discuss it. Thank you for comments

MStokluska · 2025-01-23T14:03:52Z

When enabling TLS with wrong certs (I've tried with storage) what happens is that although the authentication in backend listener fails, the products can be created and promoted to stage/prod. Auth fails on api requests. Then, when fixing up the certs, the requests are still failing. I believe this requires a redis re-sync?
I guess, porta does some validation right, so maybe an idea here would be to restart system-app pod on every update of the TLS secrets to perform the validation (and hopefully catch the incorrect certs and block api access?)
WDYT?

@MStokluska , all backend pods (listener, worker, cron) and system (app and sidekiq) are restarting if any change in secrets system-redis and backend-redis (that hold certificates), as whatched-by is available on backend-redis and system-redis

In your current implementation, watched-by doesn't work correctly. When I've fixed it, watched-by on backend-redis doesn't restart system pod.
Another point is, is systems init container validation enough to block api requests?

valerymo · 2025-01-23T14:47:25Z

There is a new functionality added to resync-routes when zync is enabled. This causes a job to be created, resync-route. The job keeps failing due to:

Failed to load CA Certificate or CA Path
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:37:in `initialize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:24:in `new'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:24:in `ssl_context'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client/config.rb:125:in `ssl_context'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:134:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:50:in `initialize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:746:in `new'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:746:in `block in connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client/middlewares.rb:12:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:745:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:732:in `raw_connection'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:697:in `ensure_connected'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:292:in `call_v'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis/client.rb:90:in `call_v'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:152:in `block in send_command'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:151:in `synchronize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:151:in `send_command'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis/commands/connection.rb:21:in `ping'
/opt/system/lib/tasks/boot.rake:26:in `block (3 levels) in <top (required)>'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:95:in `with'
/opt/system/app/lib/system/redis_pool.rb:26:in `public_send'
/opt/system/app/lib/system/redis_pool.rb:26:in `block in method_missing'

@MStokluska are you suggest to open a Jir

When enabling TLS with wrong certs (I've tried with storage) what happens is that although the authentication in backend listener fails, the products can be created and promoted to stage/prod. Auth fails on api requests. Then, when fixing up the certs, the requests are still failing. I believe this requires a redis re-sync?
I guess, porta does some validation right, so maybe an idea here would be to restart system-app pod on every update of the TLS secrets to perform the validation (and hopefully catch the incorrect certs and block api access?)
WDYT?

@MStokluska , all backend pods (listener, worker, cron) and system (app and sidekiq) are restarting if any change in secrets system-redis and backend-redis (that hold certificates), as whatched-by is available on backend-redis and system-redis

In your current implementation, watched-by doesn't work correctly. When I've fixed it, watched-by on backend-redis doesn't restart system pod. Another point is, is systems init container validation enough to block api requests?

@MStokluska , Current PR - Completed, tested, and confirmed to work exactly as specified in Jira/Requirements, as well as in alignment with our team prior meeting discussions and my prior conversation with System/Juan (see team chat for details).
Regarding your suggestion to add additional Redis TLS validation to the Operator, this represents a change in the requirements. To address it, I've opened a follow-up task: THREESCALE-11453. Thank you for your suggestion. cc: @briangallagher .

valerymo requested a review from a team as a code owner November 25, 2024 13:09

openshift-ci bot added the do-not-merge/work-in-progress label Nov 25, 2024

valerymo mentioned this pull request Nov 25, 2024

[WIP]THREESCALE-11020 Redis TLS certs and keys for porta and backend #1025

Closed

valerymo changed the title ~~[WIP] New - THREESCALE-11020 Redis TLS certs and keys for porta and backend~~ [WIP] THREESCALE-11020 Redis TLS certs and keys for porta and backend Nov 25, 2024

valerymo force-pushed the THREESCALE-11020-2 branch 4 times, most recently from 5414a05 to 2eea781 Compare December 2, 2024 13:34

valerymo changed the title ~~[WIP] THREESCALE-11020 Redis TLS certs and keys for porta and backend~~ THREESCALE-11020 Redis TLS certs and keys for porta and backend Dec 4, 2024

openshift-ci bot removed the do-not-merge/work-in-progress label Dec 4, 2024

valerymo force-pushed the THREESCALE-11020-2 branch from 1347947 to d661921 Compare December 4, 2024 13:42

austincunningham reviewed Dec 5, 2024

View reviewed changes

valerymo force-pushed the THREESCALE-11020-2 branch from 921879a to 0ea718f Compare December 12, 2024 12:15