Chaos-Engineering-Style pipelines to make sure Weaviate behaves correctly, even when we try to sabotage it a bit.
Currently there are two independent pipelines:
This pipelines is a Chaos-Monkey-style pipeline where we aim to import a large dataset, but an external source constantly introduces crashes. The goal is to make sure that every crash is recoverable and after the importer is through all objects were indeed imported.
This pipelines imports a specific dataset (which will be generated as part of the CI) and meassures the recall on it. Then Weaviate is stopped and restarted with all state coming from disk and the recall is measured again. This pipeline makes sure that all async operations we perform on commit logs (combining, condensing, etc.) do not degrade the quality of the vector index.
- Bash
- Docker / Docker Compose
jq
Everything is entirely containerized and all pipelines can be started with
simple bash scripts. You can find the scripts in the root folder, such as
./import_while_crashing.sh and ./compare_recall_after_restart.sh. Or simply
check the Github actions YAML files for examples.