A very simple and fast key-value pure-go store but persisting data to disk, with a "localStorage-like" API.
This is the pure-golang version of the original scdb
scdb may not be production-ready yet. It works, quite well but it requires more vigorous testing.
Coming from front-end web
development, localStorage was always
a convenient way of quickly persisting data to be used later by a given application even after a restart.
Its API was extremely simple i.e. localStorage.getItem(), localStorage.setItem(), localStorage.removeItem()
, localStorage.clear().
Coming to the backend (or even desktop) development, such an embedded persistent data store with a simple API was hard to come by.
scdb is meant to be like the 'localStorage' of backend and desktop (and possibly mobile) systems. Of course to make it a little more appealing, it has some extra features like:
- Time-to-live (TTL) where a key-value pair expires after a given time
- Non-blocking reads from separate processes, and threads.
- Fast Sequential writes to the store, queueing any writes from multiple processes and threads.
- golang +v1.18
-
Ensure you have golang +v1.18 installed. You can check the official instructions for how to do that.
-
Initialize a new go modules project
mkdir example-go-scdb
cd example-go-scdb
go mod init github.com/<your-username>/example-go-scdb- Install the package
go get github.com/sopherapps/go-scdb/scdb- Create a main.go file
touch main.go- Add the following code to the main.go file
package main
import (
"fmt"
"github.com/sopherapps/go-scdb/scdb"
"log"
)
func main() {
records := map[string][]byte{
"hey": []byte("English"),
"hi": []byte("English"),
"salut": []byte("French"),
"bonjour": []byte("French"),
"hola": []byte("Spanish"),
"oi": []byte("Portuguese"),
"mulimuta": []byte("Runyoro"),
}
var maxKeys uint64 = 1_000_000
var redundantBlocks uint16 = 1
var poolCapacity uint64 = 10
var compactionInterval uint32 = 1_800
store, err := scdb.New(
"db",
&maxKeys,
&redundantBlocks,
&poolCapacity,
&compactionInterval)
if err != nil {
log.Fatalf("error opening store: %s", err)
}
defer func() {
_ = store.Close()
}()
// inserting without ttl
for k, v := range records {
err := store.Set([]byte(k), v, nil)
if err != nil {
log.Fatalf("error inserting without ttl: %s", err)
}
}
// inserting with ttl of 5 seconds
var ttl uint64 = 5
for k, v := range records {
err := store.Set([]byte(k), v, &ttl)
if err != nil {
log.Fatalf("error inserting with ttl: %s", err)
}
}
// updating - just set them again
updates := map[string][]byte{
"hey": []byte("Jane"),
"hi": []byte("John"),
"hola": []byte("Santos"),
"oi": []byte("Ronaldo"),
"mulimuta": []byte("Aliguma"),
}
for k, v := range updates {
err := store.Set([]byte(k), v, nil)
if err != nil {
log.Fatalf("error updating: %s", err)
}
}
// getting
for k := range records {
value, err := store.Get([]byte(k))
if err != nil {
log.Fatalf("error getting: %s", err)
}
fmt.Printf("Key: %s, Value: %s", k, value)
}
// deleting
for k := range records {
err := store.Delete([]byte(k))
if err != nil {
log.Fatalf("error deleting: %s", err)
}
}
// clearing
err = store.Clear()
if err != nil {
log.Fatalf("error clearing: %s", err)
}
// compacting (Use sparingly, say if database file is too big)
err = store.Compact()
if err != nil {
log.Fatalf("error compacting: %s", err)
}
}- Run the module
go run main.go Contributions are welcome. The docs have to maintained, the code has to be made cleaner, more idiomatic and faster, and there might be need for someone else to take over this repo in case I move on to other things. It happens!
Please look at the CONTRIBUTIONS GUIDELINES
You can also look in the ./docs folder of the rust scdb to get up to speed with the internals of scdb e.g.
scdb is meant to be used in multiple languages of choice. However, the bindings for most of them are yet to be developed.
For other programming languages, see the main README
- Ensure you have golang +v1.18 installed. You can check the official instructions for how to do that.
- Clone this repo and enter its root folder
git clone https://github.com/sopherapps/go-scdb.git && cd go-scdb- Install dependencies
go mod tidy- Run the tests command
go test ./... -timeout 30s -race- Run benchmarks
go test -bench=. ./scdb -run=^#On a average PC
cpu: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
BenchmarkStore_Clear/Clear-8 40608 30342 ns/op
BenchmarkStore_Clear/Clear_with_ttl:_3600-8 40650 28866 ns/op
BenchmarkStore_Compact/Compact-8 1 2053004395 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_hey-8 296575 4236 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_hi-8 175813 6824 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_salut-8 182779 6713 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_bonjour-8 174636 6808 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_hola-8 185263 6905 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_oi-8 175563 6755 ns/op
BenchmarkStore_DeleteWithoutTtl/Delete_key_mulimuta-8 179035 6890 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_hey-8 273160 4323 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_hi-8 174811 6674 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_salut-8 174747 6700 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_bonjour-8 184989 6719 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_hola-8 177486 6865 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_oi-8 169952 6816 ns/op
BenchmarkStore_DeleteWithTtl/Delete_key_mulimuta-8 186657 6691 ns/op
BenchmarkStore_GetWithoutTtl/Get_hey-8 1304970 918.2 ns/op
BenchmarkStore_GetWithoutTtl/Get_hi-8 1316133 915.6 ns/op
BenchmarkStore_GetWithoutTtl/Get_salut-8 1299086 906.9 ns/op
BenchmarkStore_GetWithoutTtl/Get_bonjour-8 1311649 921.9 ns/op
BenchmarkStore_GetWithoutTtl/Get_hola-8 1313056 912.8 ns/op
BenchmarkStore_GetWithoutTtl/Get_oi-8 1302331 908.8 ns/op
BenchmarkStore_GetWithoutTtl/Get_mulimuta-8 1317252 918.0 ns/op
BenchmarkStore_GetWithTtl/Get_hey-8 1000000 1062 ns/op
BenchmarkStore_GetWithTtl/Get_hi-8 1000000 1083 ns/op
BenchmarkStore_GetWithTtl/Get_salut-8 996957 1079 ns/op
BenchmarkStore_GetWithTtl/Get_bonjour-8 1000000 1066 ns/op
BenchmarkStore_GetWithTtl/Get_hola-8 1000000 1071 ns/op
BenchmarkStore_GetWithTtl/Get_oi-8 1000000 1062 ns/op
BenchmarkStore_GetWithTtl/Get_mulimuta-8 1000000 1059 ns/op
BenchmarkStore_SetWithoutTtl/Set_hey_English-8 159267 7296 ns/op
BenchmarkStore_SetWithoutTtl/Set_hi_English-8 127662 9400 ns/op
BenchmarkStore_SetWithoutTtl/Set_salut_French-8 122229 9521 ns/op
BenchmarkStore_SetWithoutTtl/Set_bonjour_French-8 129010 9431 ns/op
BenchmarkStore_SetWithoutTtl/Set_hola_Spanish-8 129002 9469 ns/op
BenchmarkStore_SetWithoutTtl/Set_oi_Portuguese-8 124683 9400 ns/op
BenchmarkStore_SetWithoutTtl/Set_mulimuta_Runyoro-8 120543 9663 ns/op
BenchmarkStore_SetWithTtl/Set_hey_English-8 171721 8766 ns/op
BenchmarkStore_SetWithTtl/Set_hi_English-8 122996 9660 ns/op
BenchmarkStore_SetWithTtl/Set_salut_French-8 120986 9733 ns/op
BenchmarkStore_SetWithTtl/Set_bonjour_French-8 123841 10434 ns/op
BenchmarkStore_SetWithTtl/Set_hola_Spanish-8 120808 9698 ns/op
BenchmarkStore_SetWithTtl/Set_oi_Portuguese-8 123492 9797 ns/op
BenchmarkStore_SetWithTtl/Set_mulimuta_Runyoro-8 121556 10063 ns/op
PASS
ok github.com/sopherapps/go-scdb/scdb 66.489s
- Optimize
Getoperation - Optimize the
Compactoperation
- The GopherAcademy Article on avoiding GC overhead with large heaps was helpful in the validation of the memory representation of buffers as byte arrays.
Licensed under both the MIT License
Copyright (c) 2022 Martin Ahindura
"This is real love - not that we loved God, but that He loved us and sent His Son as a sacrifice to take away our sins."
-- 1 John 4: 10
All glory be to God.