This repository contains the modified Numenta Anomaly Benchmark (NAB) v1.1 code with our dataset and labels used to evaluate our model, VMFT-LAD along with 4 other models implemented in the original NAB repo.
The NAB scores for all evaluated models on our VM failure log dataset, under 2 different evaluation criterion,
- The model is expected to detect failure indicators after the fault injection point and even within the post-failure region.
| Detector | Standard Profile | Reward Low FP | Reward Low FN |
|---|---|---|---|
| VMFT-LAD | 98.16 | 97.77 | 98.44 |
| Numenta HTM | 66.63 | 61.04 | 71.64 |
| KNN CAD | 32.13 | -22.05 | 42.32 |
| EXPoSE | 42.32 | 37.07 | 67.33 |
| ARTime | 73.98 | 56.92 | 77.89 |
| DeepLog | 71.82 | 43.75 | 76.06 |
- Requires the model to detect failures before the failure point (Early failure detection).
| Detector | Standard Profile | Reward Low FP | Reward Low FN |
|---|---|---|---|
| VMFT-LAD | 90.74 | 90.36 | 89.67 |
| Numenta HTM | 21.52 | 16.22 | 3.13 |
| KNN CAD | 1.47 | -54.94 | 0.21 |
| EXPoSE | 52.50 | 18.42 | 56.17 |
| ARTime | 48.53 | 26.99 | 46.55 |
| DeepLog | 71.50 | 43.30 | 75.79 |
The evaluation dataset contains a corpus of 695 timeseries data files simulating different VM failure scenarios,
| Dataset name | Description | File count |
|---|---|---|
| Benign | Normal state dataset that does not contain any VM failure | 152 |
| HDD | Simulated short-term HDD failures using SCSI_debug module | 134 |
| Buffer I/O | Simulated buffer I/O error | 144 |
| CPU | CPU over-allocation failure dataset | 128 |
| OOM | Out of Memory failure dataset | 137 |
- The master branch contains the data labels for criteria-1: relaxed.
- For the data and labels of criteria-2: strict, switch to eval-failure branch.
You need to manually install the following:
Use the Github links provided in the right sidebar.
From inside the checkout directory:
pip install -r requirements.txt
pip install . --user
If you want to manage dependency versions yourself, you can skip dependencies with:
pip install . --user --no-deps
If you are actively working on the code and are familiar with manual PYTHONPATH setup:
pip install -e . --install-option="--prefix=/some/other/path/"
conda env create
cd /path/to/nab
python run.py -d expose --detect --windowsFile labels/combined_windows_new.json
This will run the EXPoSE detector (for KNN CAD replace 'expose' with 'knncad')
Instructions on how to run HTM can be found in the nab/detectors/numenta directory.
cd /path/to/nab
cd docker
docker-compose up
cd /path/to/nab
python run.py -d vmft-lad,knncad,expose,numenta,ARTime --optimize --score --normalize --windowsFile labels/combined_windows_new.json --skipConfirmation