This repository contains the infrastructure code needed to run a proof of concept "Storage Forest" with Rook/Ceph.

A "Storage Forest" is a storage network spanning many different devices and strategies for storing data, mimicking the diversity of a physical forest.

Different storage mechanisms are good for different cost profiles, but there's been less excitement around heterogeneous SANs. In 2022 (when this project was completed) Ceph is capable of combining consumer and enterprise grade storage into a heterogeneous SAN quite easily, and with the addition of Rook and Kubernetes, making that storage easier to access than ever before.

Rook and Ceph combine to provide:

• Flexible storage provisioning and orchestration with Rook
• RBD mirroring for moving data between storage classes
• Image Live Migration
• Object Storage (S3 & Swift compatibility)
• Shared Filesystem (similar to NFS)
• network block device (a la nbd/ISCSI)

With some automation, workloads can be moved between tiers of service and automatic mirroring and backup archival system can be built, and that's just the beginning!

This particular Storage Forest has the components listed below.

• 3x ODROID M1

For more complete directions on how to prep the compute nodes, see docs/compute-setup.md.

Optionally, you can also use the ODROID HC4 for HDD storage as well, though kernel rebuilding and some other hacks are required.

• 2x eMMC (for OS
• 2x NVMe
• 2x SSD
• 2x HDD

As you might imagine, combining the disparate components of the storage forest is a complicated task.

The forest is managed with:

• Ansible for bare metal terraforming
• Kubernetes (deployed with k0s) for workload orchestration
• Ceph (via Rook for storage management) for storage orchestration

While other systems may have done the job, this stack is wonderfully robust and current.

You can replicate this setup yourself with the exact same or comparable hardware.

Update ansible/inventory.yml with the correct IPs for machines you want to configure

See docs/compute-setup.md for full instructions on how to set up HardKernel ODROID hardware.

You can kick off the automation that will provision and terraform the machines by running make:

make

make will:

• Run ansible
• Run k0sctl to provision a kubernetes cluster
• Install rook
• Set up expected StorageClasses and other requirements for using the cluster

After the cluster has been set up, cluster configurations will be available locally to use to run workloads.

See the following files for example workloads with fio (Flexible I/O tester):

• k8s/base/fio-hdd.pvc.yaml
• k8s/base/fio.job.yaml

Along with simply creating a storage network spanning different kinds of devices, the Stoarge Forest