Launch Elasticsearch and Kibana automatically #3292

satnam6502 · 2015-01-07T23:08:15Z

This PR changes our setup to automatically launch an Elasticsearch pod for ingesting logs and a Kibana pod for viewing logs when logging with Fluentd to Elasticsearch is selected. The URLs for Elasticsearch and the Kibana dashboard are reported at cluster creation time:

.. setting up Fluentd to Elasticsearch logging with Kibana viewer
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-elk/global/firewalls/fluentd-elasticsearch-logging].
NAME                          NETWORK SRC_RANGES RULES                      SRC_TAGS TARGET_TAGS
fluentd-elasticsearch-logging default 0.0.0.0/0  tcp:5601,tcp:9200,tcp:9300          kubernetes-minion

Cluster logs are ingested into Elasticsearch running at http://130.211.126.68:9200
Kibana logging dashboard will be available at http://23.236.59.213:5601

I have also changed the names of the Elasticsearch and Kibana pods and services (adding -logging) to avoid any name clashes with other instances of Elasticsearch in the cluster.

thockin · 2015-01-08T00:10:01Z

cluster/addons/fluentd-elasticsearch/es-pod.yml

@@ -0,0 +1,25 @@
+apiVersion: v1beta1
+kind: Pod


These need to run under replication controllers, not bare pods. Monitoring is wrong here, too @vishh #2408

Yes, I had intended to change this to have a replication controller (still with one controlled pod). I'll make the change.

@thockin: I am working on replication controllers. I will add them along with a few other enhancements to monitoring.

satnam6502 · 2015-01-08T19:09:07Z

@thockin : changed to using replication controllers, all yml files changed to yaml and updated documentation. PTAL.

satnam6502 · 2015-01-08T19:57:17Z

I just did one more tweak to add tag version numbers to the two Dockerfiles that we create (both at tag 1.0 here) and then I mention this tag in the pod specs. This will allow us to update the logging work without breaking existing clusters.

thockin · 2015-01-08T22:14:13Z

cluster/aws/config-default.sh

@@ -38,6 +38,7 @@ PORTAL_NET="10.0.0.0/16"

 # Optional: Install node monitoring.
 ENABLE_NODE_MONITORING=true
+ELASTICSEARCH_LOGGING_REPLICAS=1


Should probably be lumped with other logging flags, rather than with node-monitoring

Oops, my mistake, fixed.

thockin · 2015-01-08T22:16:54Z

If we use this at cluster setup, it needs to move to cluster/addons

zmerlynn · 2015-01-09T00:23:46Z

This PR is going to cause a headache for GKE. Please make sure to be delicate around the GKE provider on the OSS side or you'll break GKE Jenkins. Check it as disable initially there, create a bug, assign it to me. I'll consider how to generically launch other pods when I'm looking at #3305.

(Looking further, looks like you default to off. Great!)

satnam6502 · 2015-01-09T00:38:39Z

For GKE, can we just set ENABLE_NODE_LOGGING=false ? Is that not good enough?

zmerlynn · 2015-01-09T00:59:59Z

Yeah, you can just set it to false for now. It looks like it's default off if you don't plumb the variable through? (Or did I misread the PR?)

You can just "export KUBERNETES_PROVIDER=gke" and then "go run hack/e2e.go -up" etc. to get gke.

satnam6502 · 2015-01-09T02:32:56Z

@thockin : I think I have done your bidding. PTAL.
@zmerlynn : I will check the GKE provider to make sure I have not broken anything.
All other provider maintainers please note changes to config-default.sh
I also added a few missing --project flags which are needed when the project is set in config-default.sh @cjcullen @jlowdermilk @mbforbes

satnam6502 · 2015-01-09T04:01:30Z

@zmerlynn : running the e2e tests with the GKE provider did not give an encouraging result. Running with just -up works.

$ hack/e2e-test.sh
....
2015/01/08 19:55:48 Passed tests: certs.sh[1/1] goe2e.sh[1/1] monitoring.sh[1/1]
2015/01/08 19:55:48 Flaky tests:
2015/01/08 19:55:48 Failed tests: basic.sh[0/1] guestbook.sh[0/1] liveness.sh[0/1] pd.sh[0/1] private.sh[0/1] services.sh[0/1] update.sh[0/1]
2015/01/08 19:55:48 7 test(s) failed.
2015/01/08 19:55:48 Running: teardown
...

thockin · 2015-01-09T04:21:49Z

cluster/aws/config-default.sh

+ENABLE_CLUSTER_LOGGING=false
+if [[ "${ENABLE_NODE_LOGGING-}" == "true" ]]; then
+  if [[ "${LOGGING_DESTINATION-}" == "elasticsearch" ]]; then
+    if [[ "${ENABLE_CLUSTER_LOGGING-}" == "true" ]]; then


Do you really need all the logic, or can you just set the variable, or even just set it and comment the line out? Same for others...

I think I added that logic because you wanted the ELASTICSEARCH_LOGGING_REPLICAS definition guarded? Initially, I did just set the variable.

Hmm, I don't recall saying that. Maybe something I said was too cryptic? Either way, I think this should be a config file, not a program.

I am sure I just mis-understood you. Shall I remove two of the three tests, or just set the variable at the top-level like I did originally?

I think simpler is better in the config files.

On Thu, Jan 8, 2015 at 8:40 PM, Satnam Singh notifications@github.com
wrote:

In cluster/aws/config-default.sh
#3292 (diff)
:

LOGGING_DESTINATION=elasticsearch # options: elasticsearch, gcp
+
+# Optional: When set to true, Elasticsearch and Kibana will be setup as part of the cluster bring up.
+ENABLE_CLUSTER_LOGGING=false
+if [[ "${ENABLE_NODE_LOGGING-}" == "true" ]]; then

if [[ "${LOGGING_DESTINATION-}" == "elasticsearch" ]]; then

if [[ "${ENABLE_CLUSTER_LOGGING-}" == "true" ]]; then

I am sure I just mis-understood you. Shall I remove two of the three
tests, or just set the variable at the top-level like I did originally?

Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/3292/files#r22701289
.

satnam6502 · 2015-01-09T06:38:28Z

@thockin : moved predicate to inside the bring up and tear down functions for cluster logging; removed conditional guards for setting ELASTICSEARCH_LOGGING_REPLICAS. Tested. PTAL.

zmerlynn · 2015-01-09T13:00:51Z

@satnam: Sorry, to test on GKE you'll also need --check_version_skew=false.

On Thu, Jan 8, 2015 at 8:01 PM, Satnam Singh notifications@github.com
wrote:

@zmerlynn https://github.com/zmerlynn : running the e2e tests with the
GKE provider did not give an encouraging result. Running with just -up
works.

$ hack/e2e-test.sh
....
2015/01/08 19:55:48 Passed tests: certs.sh[1/1] goe2e.sh[1/1] monitoring.sh[1/1]
2015/01/08 19:55:48 Flaky tests:
2015/01/08 19:55:48 Failed tests: basic.sh[0/1] guestbook.sh[0/1] liveness.sh[0/1] pd.sh[0/1] private.sh[0/1] services.sh[0/1] update.sh[0/1]
2015/01/08 19:55:48 7 test(s) failed.
2015/01/08 19:55:48 Running: teardown
...

—
Reply to this email directly or view it on GitHub
#3292 (comment)
.

satnam6502 · 2015-01-09T15:18:41Z

Yes, I finally worked that out after trying various configurations and good to know about --check_version_skew=false for reducing the output verbiage. Thanks.

thockin · 2015-01-09T20:18:03Z

cluster/kube-up.sh

@@ -54,4 +54,6 @@ if [[ "${ENABLE_CLUSTER_DNS}" == "true" ]]; then
      | "${KUBE_ROOT}/cluster/kubectl.sh" create -f -
 fi

+setup-fluentd-elasticsearch-logging


last nit: rename to setup-logging

satnam6502 · 2015-01-09T22:09:46Z

Changed the bring up and tear down names to setup-logging and teardown-loging.
Merging with the approval of @thockin

Launch Elasticsearch and Kibana automatically

thockin reviewed Jan 8, 2015
View reviewed changes

satnam6502 force-pushed the es-kibana branch 6 times, most recently from dc655af to 1b2f57d Compare January 8, 2015 19:08

satnam6502 force-pushed the es-kibana branch 2 times, most recently from 11e55e5 to 73a298b Compare January 8, 2015 19:48

thockin reviewed Jan 8, 2015
View reviewed changes

satnam6502 force-pushed the es-kibana branch from 73a298b to 958893e Compare January 9, 2015 02:29

thockin reviewed Jan 9, 2015
View reviewed changes

satnam6502 force-pushed the es-kibana branch from 958893e to 7fb8aa9 Compare January 9, 2015 06:36

satnam6502 force-pushed the es-kibana branch 2 times, most recently from f9ae7bb to 5090cff Compare January 9, 2015 18:14

thockin reviewed Jan 9, 2015
View reviewed changes

Launch Elasticsearch and Kibana automatically

295bd37

satnam6502 force-pushed the es-kibana branch from 5090cff to 295bd37 Compare January 9, 2015 21:42

satnam6502 added a commit that referenced this pull request Jan 9, 2015

Merge pull request #3292 from satnam6502/es-kibana

d5d8049

Launch Elasticsearch and Kibana automatically

satnam6502 merged commit d5d8049 into kubernetes:master Jan 9, 2015

This was referenced Jan 11, 2015

Create test for default Elasticsearch and Kibana #3388

Closed

Proposal for Deferred Creation of Addon Objects #3579

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Launch Elasticsearch and Kibana automatically #3292

Launch Elasticsearch and Kibana automatically #3292

satnam6502 commented Jan 7, 2015

thockin Jan 8, 2015

satnam6502 Jan 8, 2015

vishh Jan 8, 2015

satnam6502 commented Jan 8, 2015

satnam6502 commented Jan 8, 2015

thockin Jan 8, 2015

satnam6502 Jan 8, 2015

thockin commented Jan 8, 2015

zmerlynn commented Jan 9, 2015

satnam6502 commented Jan 9, 2015

zmerlynn commented Jan 9, 2015

satnam6502 commented Jan 9, 2015

satnam6502 commented Jan 9, 2015

thockin Jan 9, 2015

satnam6502 Jan 9, 2015

thockin Jan 9, 2015

satnam6502 Jan 9, 2015

thockin Jan 9, 2015

satnam6502 commented Jan 9, 2015

zmerlynn commented Jan 9, 2015

satnam6502 commented Jan 9, 2015

thockin Jan 9, 2015

satnam6502 commented Jan 9, 2015

Launch Elasticsearch and Kibana automatically #3292

Launch Elasticsearch and Kibana automatically #3292

Conversation

satnam6502 commented Jan 7, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

satnam6502 commented Jan 8, 2015

satnam6502 commented Jan 8, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Jan 8, 2015

zmerlynn commented Jan 9, 2015

satnam6502 commented Jan 9, 2015

zmerlynn commented Jan 9, 2015

satnam6502 commented Jan 9, 2015

satnam6502 commented Jan 9, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

satnam6502 commented Jan 9, 2015

zmerlynn commented Jan 9, 2015

satnam6502 commented Jan 9, 2015

Choose a reason for hiding this comment

satnam6502 commented Jan 9, 2015