Castellum is a vertical autoscaling service for OpenStack. It can perform autonomous resize operations, upscaling assets with a high usage and downscaling assets with a low usage.

In this document:

• Terminology
• Building and running
  • Oslo policy
  • Prometheus metrics

In other documents:

• Supported asset types
• API specification
• Notes for developers/contributors

• An asset is a thing that Castellum can resize. Assets have a size and usage, such that 0 <= usage <= size.
  • example: "NFS share 2180c598-58f3-4d1d-b03e-303db22de1be"
• A resource is the sum of all assets in a certain authentication scope. Castellum's behavior is configured at this level, e.g. thresholds and resizing steps. See API specification for details.
  • example: "NFS shares in project 5ceb23209bef4292b9ec97eb3e664f74"
• An operation is a single resize performed by Castellum.

Operations move through the following states:

• Created: The asset's usage has crossed one of the thresholds configured on the resource.
• Confirmed: The asset's usage has stayed at problematic levels for the configured delay. (For the critical threshold, there is no delay, so operations move from "created" to "confirmed" automatically.)
• Greenlit: The operation has been approved by a user. (If no approval requirement is configured, operations move from "confirmed" into "greenlit" automatically.)
• Cancelled: While an operation was not yet greenlit, the asset's usage moved back to normal levels.
• Succeeded: The resize operation was completed successfully.
• Failed/Errored: The resize operation was attempted, but failed or errored.

Problems with resizing fall into two categories: Failures need to be addressed by the project/domain administrators (e.g. upsize failed because of insufficient quota), while errors are unexpected backend errors that the OpenStack administrator needs to take care of (e.g. outage of an API used by Castellum).

Build with make && make install, or with docker build if that's to your taste. Castellum has three different components that you all need to run for a complete installation:

• castellum api <config-file> provides an OpenStack-style HTTP-based REST API. To add TLS, put this behind a reverse proxy.
• castellum observer <config-file> discovers assets and (based on their status) creates, confirms and cancels resize operations.
• castellum worker <config-file> performs the actual resizing.

The API and worker components can be scaled horizontally at will. The observer cannot be scaled. Do not run more than one instance of it at a time.

The API component has audit trail support and can be configured to send audit events to a RabbitMQ server.

All components receive configuration via environment variables. The following variables are recognized:

All components also expect a positional argument containing the path of a YAML configuration file. Below is a working example for a configuration file:

max_asset_sizes:
  - asset_type: 'nfs-shares(-group:.+)?'
    value: 16384

project_seeds:
  - project_name: myproject
    domain_name: mydomain
    resources:
      nfs-shares:
        critical_threshold: { usage_percent: 95 }
        size_steps: { percent: 10 }
        size_constraints: { max_size: 8192 }
    disabled_resources:
      - 'project-quota:.*'

The following fields are allowed:

All regexes are matched against the entire asset type string, i.e. a leading ^ and trailing $ are always added implicitly.

When applying project seeds, projects that do not exist in Keystone will be skipped without logging an error.

Castellum understands access rules in the oslo.policy JSON format. An example can be seen at docs/example-policy.json. The following rules are expected:

• project:access gates access to all endpoints relating to a project, even if more specific rules are checked later on.
• project:show:<asset_type_shortened> gates access to all endpoints relating to a project resource.
• project:edit:<asset_type_shortened> gates access to the PUT and DELETE endpoints relating to a project resource.

All project-level policy rules can use the following object attributes:

%(project_id)s           <- deprecated, use the next one instead
%(target.project.id)s
%(target.project.name)s
%(target.project.domain.id)s
%(target.project.domain.name)s

When policy rule names reference the asset type, only the part of the asset type up until the first colon is used. For example, access to project resources with asset type project-quota:compute:instances would be gated by the rules project:show:project-quota and project:edit:project-quota.

See also: List of available API attributes

Each component (API, observer and worker) exposes Prometheus metrics via HTTP, on the /metrics endpoint. The following metrics are exposed:

Note that castellum_asset_resizes{task_outcome="success"} is incremented whenever a PendingOperation is consumed and converted into a FinishedOperation, even if that operation moved into state "failed" or "errored". The counter castellum_asset_resizes{task_outcome="failure"} is only incremented when a greenlit operation cannot be moved out of the "greenlit" state at all. Resize operations that move into state "failed" or "errored" are counted by castellum_operation_state_transitions{to_state=~"failed|errored"}.