Skip to content

Lazztech/Lazztech.Lab

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Lazztech.Lab

k3s based infrastructure for homelab, smarthome, productivity, collaboration, education, or more. k3s was selected for its ease of deployment and repeatability of services deployed to it.

k3OS can be used for a dedicated machine/VM or k3d/k3s for local testing.

Security Basics

Create a secret, based on the values below, that holds the default admin username & password that will be injected into various services.

$ kubectl create secret generic admin --from-literal username=USERNAME --from-literal password="PASSWORD"

Setup network policies.

# apply internal namespace network policy
$ kubectl apply -f k8s/network-policies/network-policy.yaml

# configure ingress to pass through client ip addresses for whitelist support
$ kubectl patch svc traefik -n kube-system -p '{"spec":{"externalTrafficPolicy":"Local"}}'

Setup SSO provider.

# prepare environment variable secrets for authentik
$ kubectl create secret generic authentik --from-literal AUTHENTIK_SECRET_KEY="SECRET_KEY" --from-literal AUTHENTIK_POSTGRESQL__PASSWORD="PASSWORD"
# add helm repo for authintik
$ helm repo add authentik https://charts.goauthentik.io
# update
$ helm repo update
# install authentik or apply any updates to the authentik-config.yaml
$ helm upgrade authentik authentik/authentik -f helm/authentik-config.yaml

# go to https://<your server>/if/flow/initial-setup/ for initial setup

# uninstall command if needed
$ helm uninstall authentik

Database

One primary Postgres instance shall be configured and used across most the services. This is done with the kubegres operator for ease of managment.

# install kubegres operator
$ kubectl apply -f https://raw.githubusercontent.com/reactive-tech/kubegres/v1.15/kubegres.yaml

# watch status of kubegres
$ watch kubectl get all -n kubegres-system

# deploy postgres database
$ kubectl apply -f k8s/postgres.yaml

# check status of backup cron job
$ kubectl get CronJob

# deploy pgAdmin client to manage database
# connect to postgres.default.svc.cluster.local
# port is 5432
# user is "postgres"
# password is admin password secret
$ kubectl apply -f k8s/pgAdmin.yaml

# uninstall command if needed
$ kubectl delete -f https://raw.githubusercontent.com/reactive-tech/kubegres/v1.15/kubegres.yaml

SSL

https://cert-manager.io/docs/installation/kubernetes/

# install cert-manager
$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml

# uninstall command if needed
$ kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml

Monitoring & Logging

Kube prometheus stack via helm installs prometheus, alertmanager & grafana.

# add helm repo
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# install kube-prometheus-stack
# this command can also be used to update the
$ helm upgrade prometheus prometheus-community/kube-prometheus-stack --values helm/kube-prometheus-stack-config.yaml

# install loki
$ helm upgrade --install loki grafana/loki -f helm/loki-config.yaml
# uninstall commands if needed
$ helm uninstall prometheus
$ helm uninstall loki

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

Grafana:

Loki can now be added as a datasource in Grafana. See http://docs.grafana.org/features/datasources/loki/ for more detail.

Add loki as a data source in grafana with http://loki:3100 as the url.

Metrics can be made to work with the monitoring stack from above via opening up the settings for your context:

  • Go to prometheus section
  • Set to prometheus-operator
  • Set prometheus service address to default/prometheus-kube-prometheus-prometheus:9090

Networked Storage

# install support for samba shares
$ helm repo add csi-driver-smb https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/charts
$ helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --version v1.4.0

# uninstall if needed
$ helm uninstall csi-driver-smb -n kube-system

Resource Recommender

Using a combination of the kubernetes vpa (vertical pod autoscaler) & a project called goldilocks by fairwinds you can make educated decisions on how you set your resource limits & requests based on how your services actually get used.

# add helm repo
$ helm repo add fairwinds-stable https://charts.fairwinds.com/stable

# install fairwinds paired down vpa deployment
$ helm install vpa fairwinds-stable/vpa --namespace vpa --create-namespace

# install fairwinds goldilocks to the default namespace
$ helm install goldilocks --namespace default fairwinds-stable/goldilocks

# activate goldilocks on the default namespace
$ helm install goldilocks --namespace default fairwinds-stable/goldilocks

# apply load to the service you're focused on...
# then open the goldilocks dashboard to get recommendations on resources
$ kubectl -n default port-forward svc/goldilocks-dashboard 8080:80

Kubernetes Dashboard

Lens is recommended:

# install dashboard
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

k3OS tips and tricks

# resize root partition to 100%
$ sudo parted /dev/sda resizepart 1
# then enter yes, and 100% after that

# print root filesystem size
$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       127G   82G   39G  68% /

# resize root filesystem
$ sudo resize2fs /dev/sda1
resize2fs 1.45.6 (20-Mar-2020)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old_desc_blocks = 16, new_desc_blocks = 128
The filesystem on /dev/sda1 is now 268435200 (4k) blocks long.

# confirm that it is now at the expected size
$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1      1011G   82G  887G   9% /
# editing k3OS config https://github.com/rancher/k3os#configuration-reference
$ sudo vi /var/lib/rancher/k3os/config.yaml
$ sudo reboot

Services

  • ✅ : runs stably
  • 🚧 : needs work though runs
  • 🛑 : work in progress & may not deploy
  • ❌ : no longer used
Status Service Name Purpose Comments
Ackee Analytics Pretty stable
Adminer DB Admin Pretty stable
Calibre-web Ebooks Pretty stable
Cluster Issuer Issues SSL Needs cert-manager helm
Internal Cert Internal SSL Uses DNS01 via cert-manager helm
Internal DDNS *.internal subdomain to LAN IP Works great
Lazztech Cert External SSL Needs cert-manager helm
Lazztech DDNS *.lazz.tech domains to WAN IP Needs cert-manager helm
Code-Server VSCode Web Server Needs more recourses
🚧 Deepstack AI web interface Various uses
Docker-Registry Registry & UI works
🚧 Double-take Facial Recognition WIP
Drone CICD WIP
Jenkins CICD Simple & well documented with Blue Ocean for containers
Freeipa AD Alternative Deploys though not yet documented
🚧 Frigate Object detection NVR Uses Google Coral USB TPU
🚧 Geoip geoip For analytics
Ghost Wordpress alternative Works great behind CDN
Gitea Git server Works well for mirrors
Home Assistant Home Automation Assumes usb zigbee
Homer Start page Works great
🚧 Jellyfin Media server WIP
🚧 Keycloak SSO Deploys though not yet documented
🚧 Matrix Chat Needs work though seems good
Minio Object Storage Works nicely
MongoDB NoSQL Document db
Mosquitto MQTT Document db
🚧 Nextcloud GSuite Alternative WIP
🚧 Node-red Low code automation WIP
QuakeJS WASM Quake3 Free for all!
Redis Key value & cache Handy
Scrutiny Hard drive monitoring Handy
Snapdrop Airdrop alternative Dissatisfied with usability/reliability
Uptime-Kuma Status Page Stable
Wg-access-server Wireguard & UI Needs work or replacement
Wikijs Wiki Switching from Dokuwiki

Troubleshooting

Certs on k3os seem to automatically change every year or so, so you'll neect to update your kubeconfig to regain access. https://serverfault.com/questions/1032367/kubectl-get-nodes-error-you-must-be-logged-in-to-the-server-unauthorized-ho

Releases

No releases published

Packages

No packages published