sample end-to-end benchmarking flow + cleanup (#117)

* cleanup * removing in favour of byte size experiment * provide a sample experiment * sample benchmarking flow update * update root of directory
dsg-uwaterloo · Dec 29, 2023 · 52ab2a3 · 52ab2a3
1 parent b90ee0b
commit 52ab2a3
Show file tree

Hide file tree

Showing 25 changed files with 11,043 additions and 360 deletions.
diff --git a/ortoa-tee/.gitignore b/ortoa-tee/.gitignore
@@ -11,7 +11,6 @@ install/
 
 # e2e benchmarking flow
 out/
-*.csv
 
 # Miscellaneous
 .vscode/

diff --git a/ortoa-tee/data/README.md b/ortoa-tee/data/README.md
@@ -5,8 +5,7 @@
 ```txt
 .
 ├── experiments                # Benchmarking experiments
-├── sample_seed                # Sample seed and operations files
-├── tests                      # Tests used while developing ORTOA
+├── sample                     # Sample end-to-end benchmarking flow
 └── README.md
 ```
 
@@ -21,22 +20,12 @@ We define experiments in `.yaml` format used to microbenchmark the ORTOA-TEE pro
 │   ├── byte_size              # Varying the size of the values (in bytes)
 │   ├── client_concurrency     # Varying the number of concurrent clients
 │   ├── db_size                # Varying the number of keys in the DB
-│   └── percent_write          # Varying the % of read vs. write operations
+│   ├── percent_write          # Varying the % of read vs. write operations
+│   ├── real_world_data        # Benchmarking ORTOA-TEE on real-world data
+│   └── server_location        # Benchmarking ORTOA-TEE as network latency increases
 └── ...
 ```
 
-## `seed`
+## `sample`
 
-Sample seed and operations `.csv` files that are consumed by the `ortoa-client` program to perform database accesses with the ORTOA protocol.
-
-## `tests`
-
-Tests defined in the same `.yaml` format as `experiments`. Consumed by the benchmarking tool while developing ORTOA-TEE to test the success/failure of the program.
-
-```txt
-.
-├── ...
-├── tests
-│   └── byte_size_tests        # How big can we push the size of the values?
-└── ...
-```
+A sample end-to-end benchmarking flow complete with a seed file, an operations file, a benchmarking `.yaml` config and a `README` with the relevant commands.
diff --git a/ortoa-tee/data/sample/README.md b/ortoa-tee/data/sample/README.md
@@ -0,0 +1,21 @@
+# Benchmarking Sample
+
+## Files provided:
+
+- `data/sample/sample_seed.csv`: Sample DB seed file
+- `data/sample/sample_operations.csv`: Sample operations file
+
+## Running the benchmark
+
+```bash
+# Create a venv and install the SDK if not done
+ORTOA/ortoa-tee/ $ python3 -m venv .venv  # If you don't have a venv ready
+ORTOA/ortoa-tee/ $ . .venv/bin/activate
+ORTOA/ortoa-tee/ $ pip install -e extras/
+
+# Source script library
+ORTOA/ortoa-tee/ $ source scripts/ortoa-lib.sh
+
+# Run the benchmark
+ORTOA/ortoa-tee/ $ ortoa-benchmark -e data/sample/sample_experiment.yaml
+```
diff --git a/ortoa-tee/data/sample/sample_experiment.yaml b/ortoa-tee/data/sample/sample_experiment.yaml
@@ -0,0 +1,16 @@
+name: Sample Experiment
+
+output_directory: sample_experiment/
+
+metadata:
+  description: A sample experiment to help demonstrate the benchmarking flow.
+
+client_config:
+  data:
+    data_type: seed
+    seed: ortoa-tee/data/sample/sample_seed.csv
+    operations: ortoa-tee/data/sample/sample_operations.csv
+
+  flags:
+    - name: nthreads
+      value: 32
-Original file line number
+Diff line change
@@ Expand Up / @@ -11,7 +11,6 @@ install/ @@
     # e2e benchmarking flow
     out/
-    *.csv
     # Miscellaneous
     .vscode/
@@ Expand Down @@