K8PetStore: A scalable kubernetes application with automated load gen…

…eration (kubernetes#3137).
duda · Mar 13, 2015 · c552e91 · c552e91
1 parent 13253d0
commit c552e91
Show file tree

Hide file tree

Showing 30 changed files with 3,506 additions and 0 deletions.
diff --git a/examples/k8petstore/README.md b/examples/k8petstore/README.md
@@ -0,0 +1,123 @@
+## Welcome to k8PetStore
+
+This is a follow up to the Guestbook example, which implements a slightly more real world demonstration using 
+
+the same application architecture.
+
+- It leverages the same components (redis, Go REST API) as the guestbook application
+- It comes with visualizations for graphing whats happening in Redis transactions, along with commandline printouts of transaction throughput
+- It is hackable : you can build all images from the files is in this repository (With the exception of the data generator, which is apache bigtop).
+- It generates massive load using a semantically rich, realistic transaction simulator for petstores
+
+This application will run a web server which returns REDIS records for a petstore application.
+It is meant to simulate and test high load on kubernetes or any other docker based system.
+
+If you are new to kubernetes, and you haven't run guestbook yet, 
+
+you might want to stop here and go back and run guestbook app first.  
+
+The guestbook tutorial will teach you alot about the basics of kubernetes, and we've tried not to be redundant here.
+
+## Architecture of this SOA
+
+A diagram of the overall architecture of this application can be seen in arch.dot (you can paste the contents in any graphviz viewer, including online ones such as http://sandbox.kidstrythisathome.com/erdos/.
+
+## Docker image dependencies
+
+Reading this section is optional, only if you want to rebuild everything from scratch.
+
+This project depends on three docker images which you can build for yourself and save
+in your dockerhub "dockerhub-name".
+
+Since these images are already published under other parties like redis, jayunit100, and so on,
+so you don't need to build the images to run the app. 
+
+If you do want to build the images, you will need to build and push these 3 docker images.
+
+- dockerhub-name/bigpetstore-load-generator, which generates transactions for the database.
+- dockerhub-name/redis, which is a simple curated redis image.
+- dockerhub-name/k8petstore, which is the web app image.
+
+## Get started with the WEBAPP 
+
+The web app is written in Go, and borrowed from the original Guestbook example by brendan burns.
+
+We have extended it to do some error reporting, persisting of JSON petstore transactions (not much different then guestbook entries),
+
+and supporting of additional REST calls, like LLEN, which returns the total # of transactions in the database.
+
+To run it locally, you simply need to run basic Go commands.  Assuming you have Go set up, do something like: 
+
+```
+#Assuming your gopath is in / (i.e. this is the case, for example, in our Dockerfile).
+go get main
+go build main
+export STATIC_FILES=/tmp/static
+/gopath/bin/main
+```
+
+## Set up the data generator
+
+The web front end provides users an interface for watching pet store transactions in real time as they occur.
+
+To generate those transactions, you can use the bigpetstore data generator.  Alternatively, you could just write a 
+
+shell script which calls "curl localhost:3000/k8petstore/rpush/blahblahblah" over and over again :).  But thats not nearly
+
+as fun, and its not a good test of a real world scenario where payloads scale and have lots of information content. 
+
+Similarly, you can locally run and test the data generator code, which is Java based, you can pull it down directly from 
+
+apache bigtop.
+
+Directions for that are here : https://github.com/apache/bigtop/tree/master/bigtop-bigpetstore/bigpetstore-transaction-queue
+
+You will likely want to checkout the branch 2b2392bf135e9f1256bd0b930f05ae5aef8bbdcb, which is the exact commit which the current k8petstore was tested on.
+
+## Set up REDIS
+
+Install and run redis locally.  This can be done very easily on any Unix system, and redis starts in an insecure mode so its easy 
+
+to develop against.
+
+Install the bigpetstore-transaction-queue generator app locally (optional), but for realistic testing.
+Then, run the go app directly.  You will have to get dependencies using go the first time (will add directions later for that, its easy).
+
+## Now what? 
+
+Once you have done the above 3 steps, you have a working, from source, locally runnable version of the k8petstore app, now, we can try to run it in kubernetes.
+
+## Hacking, testing, benchmarking
+
+Once the app is running, you can go to the location of publicIP:3000 (the first parameter in the script).  In your browser, you should see a chart 
+
+and the k8petstore title page, as well as an indicator of transaction throughput, and so on.  You should be able to modify  
+
+You can modify the HTML pages, add new REST paths to the Go app, and so on.
+
+## Running in kubernetes
+
+Now that you are done hacking around on the app, you can run it in kubernetes.  To do this, you will want to rebuild the docker images (most likely, for the Go web-server app), but less likely for the other images which you are less likely to need to change. Then you will push those images to dockerhub.
+
+Now, how to run the entire application in kubernetes? 
+
+To simplify running this application, we have a single file, k8petstore.sh, which writes out json files on to disk.  This allows us to have dynamic parameters, without needing to worry about managing multiplejson files.
+
+You might want to change it to point to your customized Go image, if you chose to modify things.  
+
+like the number of data generators (more generators will create more load on the redis master).
+
+So, to run this app in kubernetes, simply run `k8petstore.sh`.
+
+## Future
+
+In the future, we plan to add cassandra support.  Redis is a fabulous in memory data store, but it is not meant for truly available and resilient storage.  
+
+Thus we plan to add another tier of queueing, which empties the REDIS transactions into a cassandra store which persists.   
+
+## Questions
+
+For questions on running this app, you can ask on the google containers group.
+
+For questions about bigpetstore, and how the data is generated, ask on the apache bigtop mailing list.
+
diff --git a/examples/k8petstore/Vagrantfile b/examples/k8petstore/Vagrantfile
@@ -0,0 +1,40 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+require 'fileutils'
+
+#$fes = 1
+#$rslavess = 1 
+
+Vagrant.configure("2") do |config|
+
+    config.vm.define "rmaster" do |rm|     
+        rm.vm.provider "docker" do |d|
+            d.vagrant_vagrantfile = "./docker-host/Vagrantfile"  
+            d.build_dir = "redis-master"
+            d.name = "rmaster"
+            d.create_args = ["--privileged=true", "-m", "1g"]
+            #d.ports = [ "6379:6379" ]
+            d.remains_running = true
+        end
+    end
+
+    config.vm.define "frontend" do |fe|
+        fe.vm.provider "docker" do |d|
+            d.vagrant_vagrantfile = "./docker-host/Vagrantfile"
+            d.build_dir = "web-server"
+            d.name = "web-server"
+            d.create_args = ["--privileged=true"]
+            d.remains_running = true
+            d.create_args = d.create_args << "--link" << "rmaster:rmaster"
+            d.ports = ["3000:3000"]
+            d.env = {"REDISMASTER_SERVICE_HOST"=>"rmaster","REDISMASTER_SERVICE_PORT"=>"6379"} 
+        end
+    end
+
+
+
+    ### Todo , add data generator.
+
+
+end
diff --git a/examples/k8petstore/bps-data-generator/README.md b/examples/k8petstore/bps-data-generator/README.md
@@ -0,0 +1,15 @@
+# How to generate the bps-data-generator container #
+
+This container is maintained as part of the apache bigtop project.
+
+To create it, simply 
+
+`git clone https://github.com/apache/bigtop`
+
+and checkout the last exact version (will be updated periodically).
+
+`git checkout -b aNewBranch 2b2392bf135e9f1256bd0b930f05ae5aef8bbdcb`
+
+then, cd to bigtop-bigpetstore/bigpetstore-transaction-queue, and run the docker file, i.e. 
+
+`Docker build -t -i jayunit100/bps-transaction-queue`.
diff --git a/examples/k8petstore/build-push-containers.sh b/examples/k8petstore/build-push-containers.sh
@@ -0,0 +1,13 @@
+#K8PetStore version is tied to the redis version.  We will add more info to version tag later.
+#Change the 'jayunit100' string below to you're own dockerhub name and run this script.
+#It will build all the containers for this application and publish them to your dockerhub account
+version="r.2.8.19"
+docker build -t jayunit100/k8-petstore-redis:$version ./redis/
+docker build -t jayunit100/k8-petstore-redis-master:$version ./redis-master
+docker build -t jayunit100/k8-petstore-redis-slave:$version ./redis-slave
+docker build -t jayunit100/k8-petstore-web-server:$version ./web-server
+
+docker push jayunit100/k8-petstore-redis:$version
+docker push jayunit100/k8-petstore-redis-master:$version
+docker push jayunit100/k8-petstore-redis-slave:$version
+docker push jayunit100/k8-petstore-web-server:$version
diff --git a/examples/k8petstore/dev/README b/examples/k8petstore/dev/README
@@ -0,0 +1,35 @@
+### Local development 
+
+1) Install Go
+
+2) Install Redis
+
+Now start a local redis instance 
+
+```
+redis-server
+```
+
+And run the app 
+
+```
+export GOPATH=~/Development/k8hacking/k8petstore/web-server/
+cd $GOPATH/src/main/
+## Now, you're in the local dir to run the app.  Go get its depenedencies.
+go get
+go run PetStoreBook.go
+```
+
+Once the app works the way you want it to, test it in the vagrant recipe below.  This will gaurantee that you're local environment isn't doing something that breaks the containers at the  versioning level.
+
+### Testing 
+
+This folder can be used by anyone interested in building and developing the k8petstore application.
+
+This is for dev and test.
+
+`vagrant up` gets you a cluster with the app's core components running.
+
+You can rename Vagrantfile_atomic to Vagrantfile if you want to try to test in atomic instead.
+
+** Now you can run the code on the kubernetes cluster with reasonable assurance that any problems you run into are not bugs in the code itself :) *
diff --git a/examples/k8petstore/dev/Vagrantfile b/examples/k8petstore/dev/Vagrantfile
@@ -0,0 +1,44 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+require 'fileutils'
+
+#$fes = 1
+#$rslavess = 1 
+
+Vagrant.configure("2") do |config|
+
+    config.vm.define "rmaster" do |rm|     
+        rm.vm.provider "docker" do |d|
+            d.vagrant_vagrantfile = "./hosts/Vagrantfile"  
+            d.build_dir = "../redis-master"
+            d.name = "rmaster"
+            d.create_args = ["--privileged=true"]
+            #d.ports = [ "6379:6379" ]
+            d.remains_running = true
+        end
+    end
+
+   puts "sleep 20 to make sure container is up..."
+   sleep(20)    
+   puts "resume"
+
+    config.vm.define "frontend" do |fe|
+        fe.vm.provider "docker" do |d|
+            d.vagrant_vagrantfile = "./hosts/Vagrantfile"
+            d.build_dir = "../web-server"
+            d.name = "web-server"
+            d.create_args = ["--privileged=true"]
+            d.remains_running = true
+            d.create_args = d.create_args << "--link" << "rmaster:rmaster"
+            d.ports = ["3000:3000"]
+            d.env = {"REDISMASTER_SERVICE_HOST"=>"rmaster","REDISMASTER_SERVICE_PORT"=>"6379"} 
+        end
+    end
+
+
+
+    ### Todo , add data generator.
+
+
+end
diff --git a/examples/k8petstore/dev/hosts/Vagrantfile b/examples/k8petstore/dev/hosts/Vagrantfile
@@ -0,0 +1,11 @@
+VAGRANTFILE_API_VERSION = "2"
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  config.vm.box = "jayunit100/centos7"
+  config.vm.provision "docker"
+  config.vm.provision "shell", inline: "ps aux | grep 'sshd:' | awk '{print $2}' | xargs kill"
+  config.vm.provision "shell", inline: "yum install -y git && service firewalld stop && service docker restart"  
+  config.vm.provision "shell", inline: "docker ps -a | awk '{print $1}' | xargs --no-run-if-empty docker rm -f || ls"
+  config.vm.network :forwarded_port, guest: 3000, host: 3000
+
+end
diff --git a/examples/k8petstore/dev/test.sh b/examples/k8petstore/dev/test.sh
@@ -0,0 +1,31 @@
+## First set up the host VM.  That ensures
+## we avoid vagrant race conditions.
+set -x 
+
+cd hosts/ 
+echo "note: the VM must be running before you try this"
+echo "if not already running, cd to hosts and run vagrant up"
+vagrant provision
+#echo "removing containers"
+#vagrant ssh -c "sudo docker rm -f $(docker ps -a -q)"
+cd ..
+
+## Now spin up the docker containers
+## these will run in the ^ host vm above.
+
+vagrant up
+
+## Finally, curl the length, it should be 3 .
+
+x=`curl localhost:3000/llen`
+
+for i in `seq 1 100` do
+    if [ x$x == "x3" ]; then 
+       echo " passed $3 "
+       exit 0
+    else
+       echo " FAIL" 
+    fi
+done
+
+exit 1 # if we get here the test obviously failed.
diff --git a/examples/k8petstore/k8petstore.dot b/examples/k8petstore/k8petstore.dot
@@ -0,0 +1,9 @@
+ digraph k8petstore {
+
+    USERS -> publicIP_proxy -> web_server;
+    bps_data_generator -> web_server [arrowhead = crow, label = "http://$FRONTEND_SERVICE_HOST:3000/rpush/k8petstore/{name..address..,product=..."];
+    external -> web_server [arrowhead = crow, label="  http://$FRONTEND_SERVICE_HOST/k8petstore/llen:3000"];
+    web_server -> redis_master [label=" RESP : k8petstore, llen"];
+    redis_master -> redis_slave [arrowhead = crow] [label="replication (one-way)"];
+}
+