Reduce size of images fluentd-gcp and fluentd-elasticsearch #29883

aledbf · 2016-08-01T22:26:01Z

replaces #26652

aledbf/fluentd-elasticsearch                   1.19 769ece5c8ba8 About an hour ago 269.9 MB
gcr.io/google_containers/fluentd-elasticsearch 1.18 0a8cbfbea7f7 5 weeks ago       530.3 MB

aledbf/fluentd-gcp                             1.22 ef979b82a767 About an hour ago 307.9 MB
gcr.io/google_containers/fluentd-gcp           1.21 0ef09b1bcfd7 2 weeks ago       498.5 MB

closes #29782

This change is

aledbf · 2016-08-01T22:26:10Z

ping @a-robinson

k8s-bot · 2016-08-01T22:26:22Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot · 2016-08-01T22:26:41Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot · 2016-08-01T22:26:59Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot · 2016-08-01T22:27:19Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot · 2016-08-01T22:27:39Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

bprashanth · 2016-08-02T16:36:06Z

ok to test

a-robinson · 2016-08-02T18:09:42Z

@piosz, mind reviewing this?

aledbf · 2016-08-11T03:21:20Z

@a-robinson @piosz Is there anything I can do to help in the review of this PR?

piosz · 2016-08-11T07:54:19Z

cluster/saltbase/salt/fluentd-es/fluentd-es.yaml

@@ -8,7 +8,7 @@ metadata:
 spec:
  containers:
  - name: fluentd-elasticsearch
-    image: gcr.io/google_containers/fluentd-elasticsearch:1.17
+    image: aledbf/fluentd-elasticsearch:1.19


We want to use official images. I can make a build once the PR is merged.

Right. I'm just following the procedure defined in the Makefile

piosz · 2016-08-11T07:54:50Z

cc @jimmidyson

piosz · 2016-08-11T08:03:05Z

Overall looks good. What's the difference between this and #26652?

piosz · 2016-08-11T08:03:37Z

cc @igorpeshansky

jimmidyson · 2016-08-11T09:04:53Z

cluster/addons/fluentd-elasticsearch/fluentd-es-image/build.sh

+sed -i -e "s/USER=td-agent/USER=root/" -e "s/GROUP=td-agent/GROUP=root/" /etc/init.d/td-agent
+
+# Install the Elasticsearch Fluentd plug-in.
+td-agent-gem install fluent-plugin-kubernetes_metadata_filter fluent-plugin-elasticsearch


Should the versions be specified for compatibility? Subsequent builds of this image could install different versions if more recent versions have been released?

Also does td-agent-gem support --no-doc flag like standard gem to ensure no docs get installed?

Dependencies pinned ~~following http://docs.fluentd.org/articles/plugin-management~~

Also does td-agent-gem support --no-doc flag like standard gem to ensure no docs get installed?

done

aledbf · 2016-08-11T11:47:25Z

Overall looks good. What's the difference between this and #26652?

Now Ubuntu Xenial is supported for the logging agent
#26652 (comment)

igorpeshansky · 2016-08-11T12:39:59Z

Now Ubuntu Xenial is supported for the logging agent

FWIW, as I recall, that was just a documentation change. There logging agent was system-independent, being in Ruby, and didn't need any changes to support Ubuntu Xenial.

bgrant0607 · 2016-08-11T17:01:35Z

ref #29782

aledbf · 2016-08-16T16:48:35Z

We want to use official images. I can make a build once the PR is merged.

@piosz I put the official repo but this requires the publication of the images before the merge of the PR (it will not pass the tests)

aledbf · 2016-08-22T23:44:05Z

@piosz ping

piosz · 2016-08-23T06:20:45Z

cluster/saltbase/salt/fluentd-es/fluentd-es.yaml

@@ -8,7 +8,7 @@ metadata:
 spec:
  containers:
  - name: fluentd-elasticsearch
-    image: gcr.io/google_containers/fluentd-elasticsearch:1.17


You should do this change in a follow up PR, once the image is pushed

piosz · 2016-08-23T06:23:11Z

@aledbf sorry for the delay. The PR look goods. The plan is:

merge this PR without changing the image version (as commented inline)
I will build the new images
you will sent the follow up PR which bumps the image version

piosz · 2016-08-23T07:43:17Z

cluster/addons/fluentd-gcp/fluentd-gcp-image/Makefile

@@ -28,7 +28,7 @@

 .PHONY:	kbuild kpush

-TAG = 1.21
+TAG = 1.25


Why are you bumping this to 1.25?

Why are you bumping this to 1.25?

because the latest version I can pull from gcr is 1.24

That's weird but it's true.

k8s-bot · 2016-08-23T12:03:06Z

GCE e2e build/test passed for commit e2c5015.

piosz · 2016-08-23T12:38:12Z

LGTM, thanks for the change!

k8s-github-robot · 2016-08-23T12:51:43Z

Automatic merge from submit-queue

piosz · 2016-08-23T13:06:06Z

I've build the the following images:
gcr.io/google_containers/fluentd-elasticsearch:1.19
gcr.io/google_containers/fluentd-gcp:1.25

Could you please bump the versions as I wrote here #29883 (comment)?

aledbf · 2016-08-23T13:12:18Z

@piosz done

Automatic merge from submit-queue Update fluent images continues #29883 fix #29782 ```release-note Reduced size of fluentd images. ```

aledbf · 2016-10-20T13:41:06Z

@mindjiver please check the comments in #33584

…k8s_metadata Automatic merge from submit-queue Enable kubernetes_metadata by default for ELK stack Looks like it was accidentally removed and was not restored back in this PR #29883 Because actually this plugin still exists in the image, but new ELK deployment don't allow you to index namespaces, pod names, etc.

googlebot added the cla: yes label Aug 1, 2016

k8s-github-robot assigned zmerlynn Aug 1, 2016

k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Aug 1, 2016

aledbf force-pushed the fluent-image-size branch 2 times, most recently from 0a5add9 to a2e9b11 Compare August 2, 2016 17:43

a-robinson assigned piosz and unassigned zmerlynn Aug 2, 2016

piosz reviewed Aug 11, 2016
View reviewed changes

jimmidyson reviewed Aug 11, 2016
View reviewed changes

aledbf force-pushed the fluent-image-size branch from a2e9b11 to 080a770 Compare August 11, 2016 12:13

aledbf force-pushed the fluent-image-size branch 3 times, most recently from d5cb3c2 to 9c605f3 Compare August 11, 2016 15:31

aledbf force-pushed the fluent-image-size branch from 9c605f3 to ebc5ec6 Compare August 16, 2016 16:45

piosz added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note-label-needed labels Aug 23, 2016

piosz reviewed Aug 23, 2016
View reviewed changes

piosz added this to the v1.4 milestone Aug 23, 2016

piosz reviewed Aug 23, 2016
View reviewed changes

aledbf added 2 commits August 23, 2016 08:21

Reduce the size of fluentd images

5753a31

Remove kubernetes metadata plugin

e2c5015

aledbf force-pushed the fluent-image-size branch from ebc5ec6 to e2c5015 Compare August 23, 2016 11:21

piosz added lgtm "Looks good to me", indicates that a PR is ready to be merged. retest-not-required labels Aug 23, 2016

k8s-github-robot merged commit e5fbea6 into kubernetes:master Aug 23, 2016

aledbf mentioned this pull request Aug 23, 2016

Update fluent images #31239

Merged

aledbf deleted the fluent-image-size branch August 23, 2016 13:12

k8s-github-robot pushed a commit that referenced this pull request Aug 25, 2016

Merge pull request #31239 from aledbf/update-fluent-images

74b0104

Automatic merge from submit-queue Update fluent images continues #29883 fix #29782 ```release-note Reduced size of fluentd images. ```

kayrus mentioned this pull request Sep 27, 2016

Enable kubernetes_metadata by default for ELK stack #33584

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce size of images fluentd-gcp and fluentd-elasticsearch #29883

Reduce size of images fluentd-gcp and fluentd-elasticsearch #29883

aledbf commented Aug 1, 2016 •

edited by bgrant0607

Loading

aledbf commented Aug 1, 2016

k8s-bot commented Aug 1, 2016

k8s-bot commented Aug 1, 2016

k8s-bot commented Aug 1, 2016

k8s-bot commented Aug 1, 2016

k8s-bot commented Aug 1, 2016

bprashanth commented Aug 2, 2016

a-robinson commented Aug 2, 2016

aledbf commented Aug 11, 2016

piosz Aug 11, 2016

aledbf Aug 11, 2016

piosz commented Aug 11, 2016

piosz commented Aug 11, 2016

piosz commented Aug 11, 2016

jimmidyson Aug 11, 2016

jimmidyson Aug 11, 2016

aledbf Aug 11, 2016 •

edited

Loading

aledbf commented Aug 11, 2016

igorpeshansky commented Aug 11, 2016 •

edited

Loading

bgrant0607 commented Aug 11, 2016

aledbf commented Aug 16, 2016

aledbf commented Aug 22, 2016

piosz Aug 23, 2016

aledbf Aug 23, 2016

piosz commented Aug 23, 2016

piosz Aug 23, 2016

aledbf Aug 23, 2016

piosz Aug 23, 2016

k8s-bot commented Aug 23, 2016

piosz commented Aug 23, 2016

k8s-github-robot commented Aug 23, 2016

piosz commented Aug 23, 2016 •

edited

Loading

aledbf commented Aug 23, 2016

aledbf commented Oct 20, 2016

Reduce size of images fluentd-gcp and fluentd-elasticsearch #29883

Reduce size of images fluentd-gcp and fluentd-elasticsearch #29883

Conversation

aledbf commented Aug 1, 2016 • edited by bgrant0607 Loading

aledbf commented Aug 1, 2016

k8s-bot commented Aug 1, 2016

k8s-bot commented Aug 1, 2016

k8s-bot commented Aug 1, 2016

k8s-bot commented Aug 1, 2016

k8s-bot commented Aug 1, 2016

bprashanth commented Aug 2, 2016

a-robinson commented Aug 2, 2016

aledbf commented Aug 11, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piosz commented Aug 11, 2016

piosz commented Aug 11, 2016

piosz commented Aug 11, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aledbf Aug 11, 2016 • edited Loading

Choose a reason for hiding this comment

aledbf commented Aug 11, 2016

igorpeshansky commented Aug 11, 2016 • edited Loading

bgrant0607 commented Aug 11, 2016

aledbf commented Aug 16, 2016

aledbf commented Aug 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piosz commented Aug 23, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-bot commented Aug 23, 2016

piosz commented Aug 23, 2016

k8s-github-robot commented Aug 23, 2016

piosz commented Aug 23, 2016 • edited Loading

aledbf commented Aug 23, 2016

aledbf commented Oct 20, 2016

aledbf commented Aug 1, 2016 •

edited by bgrant0607

Loading

aledbf Aug 11, 2016 •

edited

Loading

igorpeshansky commented Aug 11, 2016 •

edited

Loading

piosz commented Aug 23, 2016 •

edited

Loading