Pseudo random (deterministic) data generator as a (micro) service. It is a wrapper for the FastFaker Go package.
The need to generate random (user or other type) data, to mockup, test or stress your own systems.
I've built this project to be used as an internal service, to generate large amounts of fake data.
To run your own benchmarks see call_benchmark.sh. For the best performance I recommend the following:
- Count (batch) value: 100
- concurrent requests: CPUCores * 1.5
Tests done on Pseudoservice 2.0.0 using ApacheBench 2.3 ab -n 8000 -c 6 -kdq -m GET http://localhost:8080/users/100?token=SECRET42. The tests ignore any network latency, does not unmarshall the response, reuses HTTP connections (keep-alive) and were made from the local machine (where pseudoservice was running), meaning you will have at least a 20% lower performance in a real-world scenario.
- t2.micro (~1 CPU), AWS Linux
Time taken for tests: 7.649 seconds
Complete requests: 8000
Requests per second: 1045.86 [#/sec] (mean)
Time per request: 5.737 [ms] (mean)
Time per request: 0.956 [ms] (mean, across all concurrent requests)
Transfer rate: 45236.55 [Kbytes/sec] received
Translation: 45Mb/s fake data = 104500 fake users/s- t2.xlarge (~4 CPU), AWS Linux
Time taken for tests: 2.825 seconds
Requests per second: 2832.02 [#/sec] (mean)
Time per request: 2.119 [ms] (mean)
Time per request: 0.353 [ms] (mean, across all concurrent requests)
Transfer rate: 122314.75 [Kbytes/sec] received
Translation: 122Mb/s fake data = 283200 users/s- c5.2xlarge (~8 CPU), AWS Linux
Time taken for tests: 2.035 seconds
Total transferred: 353697150 bytes
HTML transferred: 352137150 bytes
Requests per second: 3931.71 [#/sec] (mean)
Time per request: 1.526 [ms] (mean)
Time per request: 0.254 [ms] (mean, across all concurrent requests)
Transfer rate: 169755.12 [Kbytes/sec] received
Translation: 170Mb/s fake data = 393100 users/sYou run it as a simple HTTP server, in a private network, and the clients can make requests to get random data, based on their custom templates/needs.
Global optional parameters:
?seed=42- if given, the result will be deterministic, as in for all calls the same data will be returned.token=SECRET42- the APIKEY{count}- how many results should be generated, integer [1,500]
Contains the OpenID/swagger documentation for this API.
Returns 200 if the service is available.
Generate random data based on a given template. Supports any string, including JSON schema.
The template can contains variables like ~name~ or ~email~. For a full list see the FastFaker variables, but instead of {} you must use the ~ delimiter (it is more URL friendly).
#the template is URL encoded: Hello ~name~!
curl -X GET "http://localhost:8080/custom/3?token=SECRET42&seed=42&template=Hello%20~name~%21"
{"results":[
"Hello Jeromy Schmeler!",
"Hello Kim Steuber!",
"Hello Jacky Borer!"
],"seed":42}
#template: {name:"~name~",age:~digit~~digit~}
curl -X GET "http://localhost:8080/custom/3?token=SECRET42&seed=42&template=%22%7Bname%3A%22~name~%22%2Cage%3A~digit~%7D%22"
{"results":[
"\"{name:\"Jeromy Schmeler\",age:53}\"",
"\"{name:\"Dustin Jones\",age:62}\"",
"\"{name:\"Keely Hartmann\",age:12}\""
],"seed":42}
#template: ~country~
curl -X GET "http://localhost:8080/custom/6?token=SECRET42&seed=42&template=~country~"
{"results":[
"Tajikistan",
"Cameroon",
"Cote Divoire",
"Turkmenistan",
"Ethiopia",
"Afghanistan"
],"seed":42}For more examples see the FastFaker examples and GoDoc.
Generate random users based on a preconfigured set of properties, with a friend relationship between them.
This endpoint was created to simulate a Social network and test a Graph database: davidescus/10minFor300Gb.
#fast and random user
curl "http://localhost:8080/api/v1/users/1?token=SECRET42"
{"seed":6124420007038740542,"nextseed":0,"users":[
{
"id":"a4419f4f-f7f9-4c99-a746-5799b281a5d6","
age":209,
"name":"Jonas Borer",
"company":"core frictionless Inc",
"position":"Senior Paradigm Associate",
"email":"JonasBorer@leadsolutions.org",
"country":"Yemen"
}
]}
#slower but deterministic call (seed=42)
curl "http://localhost:8080/api/v1/users/1?token=SECRET42&seed=42"
{"seed":42,"nextseed":43,"users":[
{"
id":"538c7f96-b164-4f1b-97bb-9f4bb472e89f",
"age":62,
"name":"Jacky Borer",
"company":"Ameliorated mindshare Inc",
"position":"International Assurance Orchestrator",
"email":"JackyBorer@leadsolutions.org",
"country":"Greece"
}
]}a). Download the binaries from a Release
wget https://github.com/bgadrian/pseudoservice/releases/download/v2.0.0/pseudoservice.tar.gz
tar -xzf pseudoservice.tar.gz
./pseudoservice/linux/pseudoservice --api-key=SECRET42 --read-timeout=1s --write-timeout=1s --keep-alive=15s --listen-limit=1024 --max-header-size=3KiB --host=0.0.0.0 --port=8080OR b). Get the docker image from bgadrian/pseudoservice/
docker run --name pseudoservice -p 8080:8080 -d -e APIKEY=MYSECRET bgadrian/pseudoservice
curl "http://localhost:8080/api/v1/users/1?token=MYSECRET&seed=42"
OR c). Hard way, requires Go 1.11, make and git
git clone git@github.com:bgadrian/pseudoservice.git
cd pseudoservice
make run
#OR
make build
env PORT=8080 ./build/pseudoserviceThe binary has the following env variables:
PORT- http listening port (8080)APIKEY- secret string to be served at?token=SECRET42- for more see
pseudoservice --help
The server accepts gzip and it was generated using swagger (open API), to see a full documentation of the service access the 127.0.0.1:8080/docs endpoint.
All the issues
Bledea Georgescu Adrian https://coder.today