Linux and macOS script to create a k3d (k3s in docker) cluster for development including:

• Cert Manager provision and manage TLS certificates in Kubernetes
• Cilium eBPF-based networking, security, and observability
• Grafana visualize metrics, logs, and traces
• Ingress-NGINX ingress controller for Kubernetes using NGINX
• Kubernetes Dashboard general-purpose web UI for Kubernetes
• Loki log aggregation system
• MetalLB network load-balancer implementation
• OpenFaaS serverless functions made simple
• Prometheus monitoring system and time series database

A working docker installation is required. Additional tooling will be downloaded automatically if they are not available: helm, k3d and kubectl.

Docker Desktop for Mac does not support routing to containers by IP address meaning that cluster nodes and load balancer addresses cannot be accessed directly. This functionality is supported natively by Linux and requires additional tooling on macOS. One such utility is docker-mac-net-connect which can be installed via homebrew:

brew install chipmk/tap/docker-mac-net-connect
brew services start chipmk/tap/docker-mac-net-connect

Use ./create.sh to create the cluster. Once started the following ports will be accessible via localhost:

• 8080 Ingress Controller HTTP port
• 8443 Ingress Controller HTTPS port
• 6443 Kubernetes API server

To configure the cluster use the command-line options:

Usage: 
  ./create.sh [options]

Options:
  -n, --cluster-name <>        cluster name (default: "default")
  -i, --cluster-id <>          cilium cluster id: 1..255 (default: 0)
  -c, --cni <>                 cni plugin: "cilium" | "calico" | "flannel" (default: "cilium")
  -l, --load-balancer <>       load balancer implementation: "metallb" | "servicelb" (default: "metallb")
  -a, --api-port <>            server api port (default: 6443)
  -P, --proxy-protocol         enable proxy protocol for ingress communication
  -p, --proxy-http-port <>     ingress http port to expose on host (default: 8080)
  -t, --proxy-tls-port <>      ingress tls port to expose on host (default: 8443)
  -L, --proxy-no-labels        do not add traefik labels to the proxy container
  -d, --proxy-host <>          traefik router host (e.g. k3s.localhost)
  -k, --proxy-certresolver <>  traefik certResolver (default: "letsencrypt")
  -e, --proxy-entrypoint <>    traefik entryPoint (default: "websecure")
  -s, --proxy-service <>       traefik service name (default: "k3s-${cluster_name}")
  -h, --help                   display help

To delete the cluster, run ./destroy.sh.

The Cilium CNI plugin (--cni="cilium") supports creating a cluster mesh:

Cluster mesh extends the networking datapath across multiple clusters. It allows endpoints in all connected clusters to communicate while providing full policy enforcement. Load-balancing is available via Kubernetes annotations.

To create a cluster capable of joining a mesh, pass --cluster-id <> as an argument to ./create.sh. Each cluster must have a unique ID between 1 and 255. cilium-cli is required to connect the clusters together and additional network configuration/bridging must be performed to allow traffic between the otherwise isolated docker networks. See ./clustermesh.sh for an example of creating a 2 cluster mesh.

The token for the dashboard can be created by running kubectl -n kubernetes-dashboard create token admin-user.

To access the dashboard visit https://console.k3s.localhost:8443/#login.

Alternatively, use kubectl proxy and visit http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#login.

Use kubectl port-forward, e.g. kubectl -n kube-system port-forward service/hubble-ui 8081:80 would make the service available at http://localhost:8081.

The password can be retrieved by running kubectl -n monitoring get secret prometheus-operator-grafana -o jsonpath="{.data.admin-password}" | base64 --decode.

To access the ui visit https://grafana.k3s.localhost:8443/login.

Alternatively, use kubectl port-forward, e.g. kubectl -n monitoring port-forward service/prometheus-operator-grafana 3000:80 would make the service available at http://localhost:3000/login.

Use kubectl port-forward, e.g. kubectl -n monitoring port-forward service/prometheus-operator-prometheus 9090:9090 would make the service available at http://localhost:9090/graph.

Use kubectl port-forward, e.g. kubectl -n monitoring port-forward service/prometheus-operator-alertmanager 9093:9093 would make the service available at http://localhost:9093/#/alerts.

The password can be retrieved by running kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode.

To access the ui visit https://fns.k3s.localhost:8443/ui/.

Alternatively, use kubectl port-forward, e.g. kubectl -n openfaas port-forward service/gateway 5000:8080 would make the service available at http://localhost:5000/ui/.