@rimusz Thanks for opening this issue!

I've had this on my list for a while to look at, although I haven't gotten to a fully blown setup yet. Could you describe your use case in more detail?

Happy to update this issue once I've done some more testing :-)

@lfittl I'm just looking an easy way to bootstrap citus cluster on kubernetes for now

@lfittl I would love to have an easy way to bootstrap citus cluster on Kubernetes as e.g. youtube vitess does for mysql http://vitess.io/getting-started/

@rimusz 💯, I'm a fan of the work YouTube did with Vitess.

I haven't been able to find time for this just yet (we've been busy with the launch of Citus Cloud), but its on my list still - although someone else from the team might find time sooner.

Out of curiosity, are you running on Google Compute Engine, or is this for another K8S-based environment?

@lfittl I'm a big fan of GKE, no needs to setup/maintain CoreOS based Kube clusters anymore :)
This is why I tweeted back about Citus Cloud on Google Cloud as well.

This would indeed be awesome. And I think with the release of Kubernetes 1.3, and the inclusion of "Pet Sets", this would fit perfectly with the goal of Citus.

+1

Big +1 here as well. Has anybody started working on this yet? If so I'd like to help out!

UPDATE: After looking around a bit I found this: https://github.com/jberkus/atomicdb/tree/master/citus_petset - haven't tested it myself yet.

@xarg -- As you noted, @jberkus has been looking at Kubernetes integration with Citus.

Josh had a talk at PGConf SV where he demoed a Citus cluster deployment with Kubernetes: http://www.pgconfsv.com/ccp-containerized-clustered-postgres-0

Once the talk is online, I'll also post the link to it here.

Also happy to send an introduction to @jberkus and also be a part of this conversation. We're paying close attention to Kubernetes.

First cut at this is up on my project: https://github.com/jberkus/atomicdb/tree/master/citus_petset

Not really production-quality yet, working on it. Contributions welcome.

@jberkus Glad for your great work! Question: "Due to an outstanding bug with PetSet and PVs on AWS;" Could you link to that issue so I can follow it?

that actually has to do with KubeAdm more than anything.

I'll be building a Helm chart this week.

Hello Team,

Do you have any plans to have citus Integrated with Kubernetes?.

I heard, that citus has plans to integrate with K8s in their pipeline.

Thanks and Regards,
Arjun.M

Any updates?

+1

👍 Any updates on this?

Check this:

https://github.com/jinhong-/citus-k8s

+1 came up in a customer conversation as an important item.
cc @lfittl

Looking to run this in Kube as well. I would love to help out with an OSS solution if I can. Writing an helm chart is probably the easiest way, but an operator is probably the best. @lfittl

Official helm integreted in documentation while be real plus....

Working example for "local-ssd-storage" aka affinity on special node node01.vm.domain.com where available.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: database
spec:
  capacity:
    storage: 100Gi
  # volumeMode field requires BlockVolume Alpha feature gate to be enabled.
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-ssd-storage
  local:
    path: /mnt/disks/ssd1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node01.vm.domain.com

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: citus-master-pvc
  labels:
    app: citus-master
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: local-ssd-storage
  local:
    path: /mnt/local-storage/ssd/disk1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node01.vm.domain.com

apiVersion: v1
kind: Service
metadata:
  name: citus-master
  labels:
    app: citus-master
spec:
  selector:
    app: citus-master
  ports:
  - port: 5432

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: citus-master
spec:
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node01.vm.domain.com
  selector:
    matchLabels:
      app: citus-master
  replicas: 1
  template:
    metadata:
      labels:
        app: citus-master
    spec:
      containers:
      - name: citus
        image: citusdata/citus:8.0.0
        ports:
        - containerPort: 5432
        env:
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: citus-secrets
              key: password
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: citus-secrets
              key: password
        volumeMounts:
        - name: storage
          mountPath: /var/lib/postgresql/data
        livenessProbe:
          exec:
            command:
            - ./pg_healthcheck
          initialDelaySeconds: 60
      volumes:
        - name: storage
          persistentVolumeClaim:
            claimName: citus-master-pvc`

apiVersion: v1
kind: Service
metadata:
  name: citus-workers
  labels:
    app: citus-workers
spec:
  selector:
    app: citus-workers
  clusterIP: None
  ports:
  - port: 5432

apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: citus-worker
spec:
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node01.vm.domain.com
  selector:
    matchLabels:
      app: citus-workers
  serviceName: citus-workers
  replicas: 3
  template:
    metadata:
      labels:
        app: citus-workers
    spec:
      containers:
      - name: citus-worker
        image: citusdata/citus:8.0.0
        lifecycle:
          postStart:
            exec:
              command: 
              - /bin/sh
              - -c
              - if [ ${POD_IP} ]; then psql --host=citus-master --username=postgres --command="SELECT * from master_add_node('${HOSTNAME}.citus-workers', 5432);" ; fi
        ports:
        - containerPort: 5432
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: citus-secrets
              key: password
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: citus-secrets
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: wstorage
          mountPath: /var/lib/postgresql/data
        livenessProbe:
          exec:
            command:
            - ./pg_healthcheck
          initialDelaySeconds: 60
  volumeClaimTemplates:
  - metadata:
      name: wstorage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 50Gi
      storageClassName: local-ssd-storage    
      local:
        path: /mnt/local-storage/ssd/disk1
      nodeAffinity:
        required:
          nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - node01.vm.domain.com

apiVersion: v1
kind: Secret
metadata:
  name: citus-secrets
type: Opaque
data:
  password: XXXXXXXXXXXXXXXXXXXXXXXXXXX

Thanks for those kubernetes files of a citus deployment! I was wondering if anybody got it to work to run multiple independent citus instances in the same k8s cluster. I tried changing ports but that didn't got well. I miss something to assign workers to a specific master, don't know where to look

@jenswachtel You might also want to give Azure Arc enabled PostgreSQL Hyperscale a try:

https://docs.microsoft.com/en-us/azure/azure-arc/data/create-postgresql-hyperscale-server-group

This is the officially supported version of Citus on Kubernetes (with a slightly more complex product name 😄), thats currently in Public Preview. Supports Citus running on Kubernetes anywhere (i.e. not just on Azure).

Running multiple Citus instances is a built-in capability that you can easily use from either the "azdata" CLI, or directly through Kubernetes resource definitions.

@lfittl wow, thank you so much! It works perfectly!

I can highly recommend Azure ARC with Citus. I tested it on AKS, but looks like you can do this with GCP etc.

Pay attention to the smaller details e.g. don't provision cluster with different availability zones, it works pretty neatly also with existing clusters!

Step 1)
Activate AKS ARC on Cluster & Connect
Described here
https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/connect-cluster#install-cli-extensions

Step 2)
Install ARC Data Services on Cluster
I can recommend going the way via Azure Portal
Add > "Azure Arc data Controller"
Fill out blanks and download juniper notebook which you then will have to run inside
Azure Data Studio
(minor addition, add this line to Step 5 in the scripts
os.environ["ACCEPT_EULA"] = "yes"

Step 3)
Create Citus Group Cluster as mentioned by @lfittl
https://docs.microsoft.com/en-us/azure/azure-arc/data/create-postgresql-hyperscale-server-group

I am so amazed, I can finally provision citus clusters in a matter of minutes!

helm chart for citus data can someone help me with that

@jenswachtel does it require having microsoft accounts and or install stuff on azure? or is it entirely controlled by say GCP?

Please be honest and close this issue as this won't be done officially. MS won't provide us with a helm chart to install this on kubernetes. They try to nudge customers to Azure.
Azure arc is a paid service.

passing by, just to say I got a working POC on kubernetes https://github.com/docteurklein/citus-test

@docteurklein Yes, thanks for the start. The docker images are over a year old. MS really tries to raise the bar for entry. At least you can build your own fairly current docker images from this branch

@GeroL The hub images look pretty recent to me https://hub.docker.com/r/citusdata/citus/tags

We where seriously looking at switching from CockroachDB to Citus, but it seems it won't be officially supported to run it on Kubernetes and we'd be funneled to the paid Azure Arc service.

Can we can an official statement on this please and stop jumping around the subject.

A working deploy https://github.com/wenerme/kube-stub-cluster/tree/main/citus , based on docteurklein/citus-test

I created a helm chart based on @wenerme version, but without parameters, anyone can fork and open a pull request for this repo to customize the parameters in values.yaml. https://github.com/sergioprates/citus-chart. I think that together we will make this real haha. This example was tested in kind and aks.

This may be useful if you want to run PostgreSQL on Kubernetes: https://stackgres.io/ - 100% open source
It supports installing the citus extension too.

@jachinte doesn't look like stackgres does anything else than loading the extension as shared preload.
What we really need is an operator, an actor observing and triggering known workflows for day-1/day-2 operations, by diffing a declarative CRD.

Like:

• if I change the cluster definition from 3 nodes to 2, I need to:
  • first drain the citus node (move shards to remaining nodes)
  • mark the node as inactive (unregister).
• adding a new node
  • spawn a pod (or a cnpg.io Cluster CRD)
  • register this as a new citus node
  • call "rebalance shards" method
• major upgrade
  • roll-out upgrade steps node-by-node

For now there is nothing out there able to do that (except maybe the azure arc).

Maybe yugabyte-db could be an alternative for these requirements https://github.com/yugabyte/yugabyte-db . It supports PostgreSQL syntax

something plan?

@xiaoweihong, yes, please stay tuned.

https://github.com/sergioprates/prates-charts

Based in @wenerme.

@CyberDem0n details please :) are you working on an operator or do you plan yet another static chart?

@docteurklein, @steven-sheehy, @ccpu, @fkpwolf, @baikjy0215: https://github.com/zalando/patroni/blob/master/kubernetes/README.md#citus-on-k8s - Citus on K8s wih Patroni for HA

Now someone could wrap it into Helm chart (but not me :)) or maintainers of Postgres Operators (most of operators are based on Patroni) could take it over and integrate.

That's awesome @CyberDem0n, nice work!

It looks like the Stackgres operator team is interested in adding Citus support via your Patroni integration. 🚀

Stackgres has initial support for Citus in its 1.5.0-beta1 release. Try it out and let them know of any feedback.

is it me or is stackgres mis-interpreting the "shard" terminology with "worker node"?

The description I read here https://stackgres.io/doc/1.5/reference/crd/sgshardedcluster/#body-inner

The shards are a group of StackGres clusters where the partitioned data chunks are stored.
When referring to the cluster in the descriptions above it apply to any cluster of the shards

doesn't seem to match the one here : https://docs.citusdata.com/en/latest/get_started/concepts.html?highlight=shard#shards

a shard as containing a subset of the rows of a distributed table in a smaller table within a worker node

In the meantime, I'm trying to bootstrap a kube-native (no patroni) operator using kube-rs (https://github.com/docteurklein/citus-operator), although it doesn't have any citus-specific logic right now. My next step is to automate shard rebalancing when adding and removing citus-worker-nodes.

Those nodes are https://cnpg.io clusters BTW, I'm glad I can compose with the existing ecosystem.

Any news on it))