16 changes: 8 additions & 8 deletions 16 README.md

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,12 +1,12 @@
		# Oblishard [](https://github.com/dsg-uwaterloo/oblishard/actions/workflows/go.yml)
		Welcome to **Oblishard**, a fault-tolerant and highly scalable Oblivious RAM (ORAM) data store designed to provide strong security guarantees by hiding access patterns from the storage server and randomizing client access.
		# Treebeard [](https://github.com/dsg-uwaterloo/treebeard/actions/workflows/go.yml)
		Welcome to **Treebeard**, a fault-tolerant and highly scalable Oblivious RAM (ORAM) data store designed to provide strong security guarantees by hiding access patterns from the storage server and randomizing client access.
		## Key Features:
		* **Scalability**: Oblishard is designed with horizontal scalability in mind, enabling scaling the system without leaking security information.
		* **Fault Tolerance**: Using the Raft consensus algorithm, Oblishard ensures the resilience of your data store even in the face of node crashes or failures, enhancing the overall reliability of your system.
		* **High Throughput**: Oblishard achieves high-throughput performance, ensuring that your applications can handle large volumes of data seamlessly and efficiently.
		* **Configurable**: Oblishard recognizes the diverse needs of different projects and offers a high level of configurability to adapt to your specific requirements.
		* **Easy to Deploy**: We've designed the setup of Oblishard to be straightforward, ensuring that you can integrate our secure and high-performance data store effortlessly into your projects.
		* **Easy to Extend**: Oblishard's modular architecture makes it easy to extend functionality. You can add new storage layers to Oblishard to support different use cases.
		* **Scalability**: Treebeard is designed with horizontal scalability in mind, enabling scaling the system without leaking security information.
		* **Fault Tolerance**: Using the Raft consensus algorithm, Treebeard ensures the resilience of your data store even in the face of node crashes or failures, enhancing the overall reliability of your system.
		* **High Throughput**: Treebeard achieves high-throughput performance, ensuring that your applications can handle large volumes of data seamlessly and efficiently.
		* **Configurable**: Treebeard recognizes the diverse needs of different projects and offers a high level of configurability to adapt to your specific requirements.
		* **Easy to Deploy**: We've designed the setup of Treebeard to be straightforward, ensuring that you can integrate our secure and high-performance data store effortlessly into your projects.
		* **Easy to Extend**: Treebeard's modular architecture makes it easy to extend functionality. You can add new storage layers to Treebeard to support different use cases.
		## Installation
		### Protocol Buffers
Expand Down

38 changes: 19 additions & 19 deletions 38 ansible/deploy.yaml

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -33,16 +33,16 @@
		- python3-docker
		- socat
		- name: Create oblishard directory
		- name: Create treebeard directory
		ansible.builtin.file:
		path: /root/oblishard
		path: /root/treebeard
		state: directory
		mode: '0755'
		- name: Remove previous log files
		ansible.builtin.shell:
		cmd: find . -name "*.log" -delete
		chdir: /root/oblishard
		chdir: /root/treebeard
		- name: Include endpoints as vars
		ansible.builtin.include_vars:
Expand All		@@ -61,11 +61,11 @@
		- name: Move Jaeger file
		template:
		src: ./templates/docker-compose.yaml.j2
		dest: /root/oblishard/jaeger.yaml
		dest: /root/treebeard/jaeger.yaml
		- name: Run Jaeger
		shell:
		cmd: "docker compose -f jaeger.yaml up -d"
		chdir: /root/oblishard
		chdir: /root/treebeard
		- name: Create Socat reverse proxy service
		template:
		src: templates/socat.service.j2
Expand Down Expand Up		@@ -125,7 +125,7 @@
		- name: Create new directories
		ansible.builtin.file:
		path: "{{ '/root/oblishard/' + item[0] + '/' + item[1].id | string + '/' + item[1].replicaid | default('') | string }}"
		path: "{{ '/root/treebeard/' + item[0] + '/' + item[1].id | string + '/' + item[1].replicaid | default('') | string }}"
		state: directory
		mode: '0755'
		delegate_to: "{{ item[1].deploy_host }}"
Expand All		@@ -139,7 +139,7 @@
		- name: Copy executables
		ansible.builtin.copy:
		src: "../cmd/{{ item[0] }}/{{ item[0] }}"
		dest: "{{ '/root/oblishard/' + item[0] + '/' + item[0] }}"
		dest: "{{ '/root/treebeard/' + item[0] + '/' + item[0] }}"
		mode: '0777'
		delegate_to: "{{ item[1].deploy_host }}"
		loop: >-
Expand All		@@ -152,31 +152,31 @@
		- name: Copy client
		ansible.builtin.copy:
		src: "../cmd/client/client"
		dest: "/root/oblishard/"
		dest: "/root/treebeard/"
		mode: '0777'
		- name: Copy configs
		ansible.builtin.copy:
		src: "{{ experiment_path }}/{{ item[0] }}"
		dest: "/root/oblishard/{{ item[0] }}"
		dest: "/root/treebeard/{{ item[0] }}"
		mode: '0666'
		delegate_to: "{{ item[1] }}"
		loop: "{{ ['router_endpoints.yaml', 'shardnode_endpoints.yaml', 'oramnode_endpoints.yaml', 'redis_endpoints.yaml', 'parameters.yaml', 'trace.txt'] | product(groups['all']) | list }}"
		- name: Create oramnode systemd services
		template:
		src: templates/oblishard-oramnode.service.j2
		dest: "/lib/systemd/system/oblishard-oramnode-{{ item.id }}-{{ item.replicaid }}.service"
		src: templates/treebeard-oramnode.service.j2
		dest: "/lib/systemd/system/treebeard-oramnode-{{ item.id }}-{{ item.replicaid }}.service"
		delegate_to: "{{ item.deploy_host }}"
		with_items: "{{ oramnode_endpoints.endpoints }}"
		- name: Stop previous services
		ansible.builtin.shell:
		cmd: systemctl stop oblishard-*
		cmd: systemctl stop treebeard-*
		- name: Start oramnode services
		ansible.builtin.systemd:
		name: "oblishard-oramnode-{{ item.id }}-{{ item.replicaid }}"
		name: "treebeard-oramnode-{{ item.id }}-{{ item.replicaid }}"
		state: restarted
		daemon_reload: true
		delegate_to: "{{ item.deploy_host }}"
Expand All		@@ -186,14 +186,14 @@
		- name: Create shardnode systemd services
		template:
		src: templates/oblishard-shardnode.service.j2
		dest: "/lib/systemd/system/oblishard-shardnode-{{ item.id }}-{{ item.replicaid }}.service"
		src: templates/treebeard-shardnode.service.j2
		dest: "/lib/systemd/system/treebeard-shardnode-{{ item.id }}-{{ item.replicaid }}.service"
		delegate_to: "{{ item.deploy_host }}"
		with_items: "{{ shardnode_endpoints.endpoints }}"
		- name: Start shardnode services
		ansible.builtin.systemd:
		name: "oblishard-shardnode-{{ item.id }}-{{ item.replicaid }}"
		name: "treebeard-shardnode-{{ item.id }}-{{ item.replicaid }}"
		state: restarted
		daemon_reload: true
		delegate_to: "{{ item.deploy_host }}"
Expand All		@@ -203,14 +203,14 @@
		- name: Create router systemd services
		template:
		src: templates/oblishard-router.service.j2
		dest: "/lib/systemd/system/oblishard-router-{{ item.id }}.service"
		src: templates/treebeard-router.service.j2
		dest: "/lib/systemd/system/treebeard-router-{{ item.id }}.service"
		delegate_to: "{{ item.deploy_host }}"
		with_items: "{{ router_endpoints.endpoints }}"
		- name: Start router services
		ansible.builtin.systemd:
		name: "oblishard-router-{{ item.id }}"
		name: "treebeard-router-{{ item.id }}"
		state: restarted
		daemon_reload: true
		delegate_to: "{{ item.deploy_host }}"
Expand Down

4 changes: 2 additions & 2 deletions 4 ansible/experiment.yaml

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -6,9 +6,9 @@
		tasks:
		- name: Run the Client
		ansible.builtin.shell:
		cmd: "/root/oblishard/client -duration 20 -conf /root/oblishard -output /root/oblishard/output.txt -logpath /root/oblishard/client.log"
		cmd: "/root/treebeard/client -duration 20 -conf /root/treebeard -output /root/treebeard/output.txt -logpath /root/treebeard/client.log"
		- name: Get the output
		ansible.builtin.fetch:
		src: "/root/oblishard/output.txt"
		src: "/root/treebeard/output.txt"
		dest: "{{ experiment_output_path }}"
		flat: yes

12 changes: 0 additions & 12 deletions 12 ansible/templates/oblishard-router.service.j2

Show comments

View file Edit file Delete file

Load diff

This file was deleted.

8 changes: 4 additions & 4 deletions 8 ...e/templates/oblishard-oramnode.service.j2 → ...e/templates/treebeard-oramnode.service.j2

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,11 +1,11 @@
		[Unit]
		Description=Oblishard Oramnode
		Description=Treebeard Oramnode
		Requires=network.target
		After=syslog.target network.target
		[Service]
		Type=simple
		ExecStart=/root/oblishard/oramnode/oramnode \
		ExecStart=/root/treebeard/oramnode/oramnode \
		-oramnodeid {{ item.id }} \
		-bindip {{ item.local_bind_ip }} \
		-advip {{ item.exposed_ip }} \
Expand All		@@ -16,8 +16,8 @@ ExecStart=/root/oblishard/oramnode/oramnode \
		# this assumes there are 3 replicas (Dangerous assmumption)
		{{'-joinaddr=' + oramnode_endpoints.endpoints[item.id * 3].exposed_ip + ':' + oramnode_endpoints.endpoints[item.id * 3].port | string }} \
		{% endif %}
		-conf /root/oblishard/ \
		-logpath {{ '/root/oblishard/oramnode/' + item.id | string + '/' + item.replicaid | string + '/all.log' }}
		-conf /root/treebeard/ \
		-logpath {{ '/root/treebeard/oramnode/' + item.id | string + '/' + item.replicaid | string + '/all.log' }}
		Restart=on-failure
		[Install]
Expand Down

12 changes: 12 additions & 0 deletions 12 ansible/templates/treebeard-router.service.j2

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,12 @@
		[Unit]
		Description=Treebeard Router
		Requires=network.target
		After=syslog.target network.target
		[Service]
		Type=simple
		ExecStart=/root/treebeard/router/router -routerid {{ item.id }} -ip {{ item.local_bind_ip }} -port {{ item.port }} -conf /root/treebeard/ -logpath /root/treebeard/router/{{ item.id }}/all.log
		Restart=on-failure
		[Install]
		WantedBy=multi-user.target

12 changes: 6 additions & 6 deletions 12 .../templates/oblishard-shardnode.service.j2 → .../templates/treebeard-shardnode.service.j2

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,13 +1,13 @@
		[Unit]
		Description=Oblishard Shardnode
		Description=Treebeard Shardnode
		Requires=network.target
		After=syslog.target network.target
		[Service]
		Type=simple
		StandardOutput=file:/root/oblishard/shardnode/{{ 'shardnode-' + item.id | string + '-' + item.replicaid | string + '.stdout.log' }}
		StandardError=file:/root/oblishard/shardnode/{{ 'shardnode-' + item.id | string + '-' + item.replicaid | string + '.stderr.log' }}
		ExecStart=/root/oblishard/shardnode/shardnode \
		StandardOutput=file:/root/treebeard/shardnode/{{ 'shardnode-' + item.id | string + '-' + item.replicaid | string + '.stdout.log' }}
		StandardError=file:/root/treebeard/shardnode/{{ 'shardnode-' + item.id | string + '-' + item.replicaid | string + '.stderr.log' }}
		ExecStart=/root/treebeard/shardnode/shardnode \
		-shardnodeid {{ item.id }} \
		-bindip {{ item.local_bind_ip }} \
		-advip {{ item.exposed_ip }} \
Expand All		@@ -18,8 +18,8 @@ ExecStart=/root/oblishard/shardnode/shardnode \
		# this assumes there are 3 replicas (Dangerous assmumption)
		{{'-joinaddr=' + shardnode_endpoints.endpoints[item.id * 3].exposed_ip + ':' + shardnode_endpoints.endpoints[item.id * 3].port | string }} \
		{% endif %}
		-conf /root/oblishard/ \
		-logpath {{ '/root/oblishard/shardnode/' + item.id | string + '/' + item.replicaid | string + '/all.log' }}
		-conf /root/treebeard/ \
		-logpath {{ '/root/treebeard/shardnode/' + item.id | string + '/' + item.replicaid | string + '/all.log' }}
		Restart=on-failure
		[Install]
Expand Down

2 changes: 1 addition & 1 deletion 2 api/oramnode.proto

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,6 +1,6 @@
		syntax = "proto3";
		option go_package = "github.com/dsg-uwaterloo/oblishard/api/oramnode";
		option go_package = "github.com/dsg-uwaterloo/treebeard/api/oramnode";
		package oramnode;
Expand Down

2 changes: 1 addition & 1 deletion 2 api/router.proto

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,6 +1,6 @@
		syntax = "proto3";
		option go_package = "github.com/dsg-uwaterloo/oblishard/api/router";
		option go_package = "github.com/dsg-uwaterloo/treebeard/api/router";
		package router;
Expand Down

2 changes: 1 addition & 1 deletion 2 api/shardnode.proto

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,6 +1,6 @@
		syntax = "proto3";
		option go_package = "github.com/dsg-uwaterloo/oblishard/api/shardnode";
		option go_package = "github.com/dsg-uwaterloo/treebeard/api/shardnode";
		package shardnode;
Expand Down

8 changes: 4 additions & 4 deletions 8 cmd/client/main.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -8,10 +8,10 @@ import (
		"path"
		"time"
		"github.com/dsg-uwaterloo/oblishard/pkg/client"
		"github.com/dsg-uwaterloo/oblishard/pkg/config"
		"github.com/dsg-uwaterloo/oblishard/pkg/tracing"
		"github.com/dsg-uwaterloo/oblishard/pkg/utils"
		"github.com/dsg-uwaterloo/treebeard/pkg/client"
		"github.com/dsg-uwaterloo/treebeard/pkg/config"
		"github.com/dsg-uwaterloo/treebeard/pkg/tracing"
		"github.com/dsg-uwaterloo/treebeard/pkg/utils"
		"github.com/rs/zerolog/log"
		"go.opentelemetry.io/otel"
		)
Expand Down

10 changes: 5 additions & 5 deletions 10 cmd/oramnode/main.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -7,11 +7,11 @@ import (
		"os"
		"path"
		"github.com/dsg-uwaterloo/oblishard/pkg/config"
		oramnode "github.com/dsg-uwaterloo/oblishard/pkg/oramnode"
		"github.com/dsg-uwaterloo/oblishard/pkg/profile"
		"github.com/dsg-uwaterloo/oblishard/pkg/tracing"
		"github.com/dsg-uwaterloo/oblishard/pkg/utils"
		"github.com/dsg-uwaterloo/treebeard/pkg/config"
		oramnode "github.com/dsg-uwaterloo/treebeard/pkg/oramnode"
		"github.com/dsg-uwaterloo/treebeard/pkg/profile"
		"github.com/dsg-uwaterloo/treebeard/pkg/tracing"
		"github.com/dsg-uwaterloo/treebeard/pkg/utils"
		"github.com/rs/zerolog/log"
		)
Expand Down

10 changes: 5 additions & 5 deletions 10 cmd/router/main.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -7,11 +7,11 @@ import (
		"os"
		"path"
		"github.com/dsg-uwaterloo/oblishard/pkg/config"
		"github.com/dsg-uwaterloo/oblishard/pkg/profile"
		router "github.com/dsg-uwaterloo/oblishard/pkg/router"
		"github.com/dsg-uwaterloo/oblishard/pkg/tracing"
		"github.com/dsg-uwaterloo/oblishard/pkg/utils"
		"github.com/dsg-uwaterloo/treebeard/pkg/config"
		"github.com/dsg-uwaterloo/treebeard/pkg/profile"
		router "github.com/dsg-uwaterloo/treebeard/pkg/router"
		"github.com/dsg-uwaterloo/treebeard/pkg/tracing"
		"github.com/dsg-uwaterloo/treebeard/pkg/utils"
		"github.com/rs/zerolog/log"
		)
Expand Down

10 changes: 5 additions & 5 deletions 10 cmd/shardnode/main.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -7,11 +7,11 @@ import (
		"os"
		"path"
		"github.com/dsg-uwaterloo/oblishard/pkg/config"
		"github.com/dsg-uwaterloo/oblishard/pkg/profile"
		shardnode "github.com/dsg-uwaterloo/oblishard/pkg/shardnode"
		"github.com/dsg-uwaterloo/oblishard/pkg/tracing"
		"github.com/dsg-uwaterloo/oblishard/pkg/utils"
		"github.com/dsg-uwaterloo/treebeard/pkg/config"
		"github.com/dsg-uwaterloo/treebeard/pkg/profile"
		shardnode "github.com/dsg-uwaterloo/treebeard/pkg/shardnode"
		"github.com/dsg-uwaterloo/treebeard/pkg/tracing"
		"github.com/dsg-uwaterloo/treebeard/pkg/utils"
		"github.com/rs/zerolog/log"
		)
Expand Down

2 changes: 1 addition & 1 deletion 2 go.mod

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,4 +1,4 @@
		module github.com/dsg-uwaterloo/oblishard
		module github.com/dsg-uwaterloo/treebeard
		go 1.20
Expand Down

6 changes: 3 additions & 3 deletions 6 pkg/client/client.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -7,9 +7,9 @@ import (
		"math/rand"
		"time"
		routerpb "github.com/dsg-uwaterloo/oblishard/api/router"
		"github.com/dsg-uwaterloo/oblishard/pkg/config"
		"github.com/dsg-uwaterloo/oblishard/pkg/rpc"
		routerpb "github.com/dsg-uwaterloo/treebeard/api/router"
		"github.com/dsg-uwaterloo/treebeard/pkg/config"
		"github.com/dsg-uwaterloo/treebeard/pkg/rpc"
		"github.com/redis/go-redis/v9"
		"github.com/rs/zerolog/log"
		"go.opentelemetry.io/otel/trace"
Expand Down

10 changes: 5 additions & 5 deletions 10 pkg/e2e/e2e_test.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -5,11 +5,11 @@ import (
		"testing"
		"time"
		"github.com/dsg-uwaterloo/oblishard/pkg/client"
		"github.com/dsg-uwaterloo/oblishard/pkg/config"
		"github.com/dsg-uwaterloo/oblishard/pkg/oramnode"
		"github.com/dsg-uwaterloo/oblishard/pkg/router"
		"github.com/dsg-uwaterloo/oblishard/pkg/shardnode"
		"github.com/dsg-uwaterloo/treebeard/pkg/client"
		"github.com/dsg-uwaterloo/treebeard/pkg/config"
		"github.com/dsg-uwaterloo/treebeard/pkg/oramnode"
		"github.com/dsg-uwaterloo/treebeard/pkg/router"
		"github.com/dsg-uwaterloo/treebeard/pkg/shardnode"
		"github.com/rs/zerolog/log"
		)
Expand Down

6 changes: 3 additions & 3 deletions 6 pkg/oramnode/client.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -6,9 +6,9 @@ import (
		"math"
		"math/rand"
		shardnodepb "github.com/dsg-uwaterloo/oblishard/api/shardnode"
		"github.com/dsg-uwaterloo/oblishard/pkg/config"
		"github.com/dsg-uwaterloo/oblishard/pkg/rpc"
		shardnodepb "github.com/dsg-uwaterloo/treebeard/api/shardnode"
		"github.com/dsg-uwaterloo/treebeard/pkg/config"
		"github.com/dsg-uwaterloo/treebeard/pkg/rpc"
		"github.com/rs/zerolog/log"
		"google.golang.org/grpc"
		"google.golang.org/grpc/credentials/insecure"
Expand Down

10 changes: 5 additions & 5 deletions 10 pkg/oramnode/server.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -8,11 +8,11 @@ import (
		"sync/atomic"
		"time"
		pb "github.com/dsg-uwaterloo/oblishard/api/oramnode"
		"github.com/dsg-uwaterloo/oblishard/pkg/commonerrs"
		"github.com/dsg-uwaterloo/oblishard/pkg/config"
		"github.com/dsg-uwaterloo/oblishard/pkg/rpc"
		strg "github.com/dsg-uwaterloo/oblishard/pkg/storage"
		pb "github.com/dsg-uwaterloo/treebeard/api/oramnode"
		"github.com/dsg-uwaterloo/treebeard/pkg/commonerrs"
		"github.com/dsg-uwaterloo/treebeard/pkg/config"
		"github.com/dsg-uwaterloo/treebeard/pkg/rpc"
		strg "github.com/dsg-uwaterloo/treebeard/pkg/storage"
		"github.com/hashicorp/raft"
		"github.com/rs/zerolog/log"
		"go.opentelemetry.io/otel"
Expand Down

Loading