Convert node level logging agents to v1beta3 #7274

satnam6502 · 2015-04-24T00:37:42Z

@lavalamp @bgrant0607 I can create these pods manually but they are not being picked up by the manifest process -- can you see what I might be doing wrong? Thanks.

satnam6502 · 2015-04-24T01:05:51Z

@zmerlynn

bgrant0607 · 2015-04-24T01:37:21Z

cluster/saltbase/salt/fluentd-es/fluentd-es.manifest

@@ -11,7 +14,7 @@ containers:
        mountPath: /var/lib/docker/containers
      - name: varlog
        mountPath: /varlog
-volumes:
+  volumes:


Did you try kubectl create --validate?

Please read v1beta3 conversion tips:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/api.md#v1beta3-conversion-tips

volumes could be failing validation. source should be inlined and hostDir has been replaced by hostPath:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/api/v1beta3/types.go#L203

Also, please rename the file to .yaml.

bgrant0607 · 2015-04-24T01:44:07Z

Ref #5475

satnam6502 · 2015-04-25T00:50:06Z

I can create the pods for node level logging manually:

$ kubectl create --validate -f fluentd-es.yaml 
pods/fluentd-elasticsearch
$ cd ../fluentd-gcp/
$ kubectl create --validate -f fluentd-gcp.yaml 
pods/fluentd-cloud-logging

and they go into the Running state. However they are not picked up during the manifest process. @zmerlynn -- any pointers on why this might be happening or what I can do to debug this? Thanks.

bgrant0607 · 2015-04-25T17:29:16Z

@satnam6502 Did you fix the volume source for v1beta3?

Enable -v 3 or higher on kubelet and look for the log messages from the reader:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubelet/config/file.go

satnam6502 · 2015-04-25T17:43:54Z

No. I will dig onto the log later perhaps. But for now I am trying to understand how to use --validate. I was in hysterisis about hosthPath vs. hostDir but --validate did not seem to say anything and both create commands created a pod. I need to understand how to use --validate.

$ cat fluentd-es.yaml 
apiVersion: v1beta3
kind: Pod
metadata:
  name: fluentd-elasticsearch
spec:
  containers:
  - name: fluentd-elasticsearch
    image: gcr.io/google_containers/fluentd-elasticsearch:1.4
    env:
      - name: FLUENTD_ARGS
        value: -qq
    volumeMounts:
      - name: varlog
        mountPath: /varlog
  volumes:
  - name: varlog
    source:
      hostPath:
        path: /var/log
$ kubectl create --validate -f fluentd-es.yaml
pods/fluentd-elasticsearch
$ kubectl delete -f fluentd-es.yaml 
$ vi fluentd-es.yaml  (change hostPath back to hostDir)
$ cat fluentd-es.yaml 
apiVersion: v1beta3
kind: Pod
metadata:
  name: fluentd-elasticsearch
spec:
  containers:
  - name: fluentd-elasticsearch
    image: gcr.io/google_containers/fluentd-elasticsearch:1.4
    env:
      - name: FLUENTD_ARGS
        value: -qq
    volumeMounts:
      - name: varlog
        mountPath: /varlog
  volumes:
  - name: varlog
    source:
      hostDir:
        path: /var/log
$ kubectl create --validate -f fluentd-es.yaml 
pods/fluentd-elasticsearch

satnam6502 · 2015-04-25T17:56:13Z

From http://kubernetes.io/third_party/swagger-ui/#!/v1beta3/replacePod I see it should be hostPath but perhaps --validate does not help since it is an optional field?

satnam6502 · 2015-04-25T18:28:18Z

Looking at the kubelet logs on one of the nodes it seems that it still wants the old version of the manifest file?

I0425 18:20:25.363209    4748 file.go:123] Can't process config file "/etc/kubernetes/manifests/fluentd-es.yaml": /
etc/kubernetes/manifests/fluentd-es.yaml: read 'apiVersion: v1beta3
kind: Pod
metadata:
  name: fluentd-elasticsearch
spec:
  containers:
  - name: fluentd-elasticsearch
    image: gcr.io/google_containers/fluentd-elasticsearch:1.4
    env:
      - name: FLUENTD_ARGS
        value: -qq
    volumeMounts:
      - name: varlog
        mountPath: /varlog
  volumes:
  - name: varlog
    hostPath:
      path: /var/log
', but couldn't parse as neither manifest (invalid manifest: [version: required value containers: required value]) 
nor pod (couldn't get version/kind: invalid character 'a' looking for beginning of value).

satnam6502 · 2015-04-25T18:55:01Z

So I suspect I need to change code in the kubelet to make it accept a regular pod specification.

satnam6502 · 2015-04-25T19:40:07Z

How about I do the following as a stepping stone towards depreciating the manifest mechanism. First, change the kubelet to just create anything it finds in the manifest directories. Change the v1beta3 pod spec for the node level log collectors into a replication controller spec.

Then (a bit later) implement a one-per-node controller -- perhaps as a plug-in? -- @thockin -- and then just use this controller (from the config scripts rather than the kubelet) to place the v1beta3 replication controller for node level logging on each node.

How does that sound @bgrant0607 ? Adding @dchen1107 for good measure.

bgrant0607 · 2015-04-25T20:48:44Z

@satnam6502 There is supposed to be code in Kubelet to read pods, but it must be broken. We do need to fix it.

It appears that file_test.go generates manifests and then reads them in. We need comparable code for writing pods using the specified API version (v1beta3, in particular).
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubelet/config/file_test.go

cc @wojtek-t @nikhiljindal

wojtek-t · 2015-04-27T05:12:19Z

It appears that file_test.go generates manifests and then reads them in. We need comparable code for writing pods using the specified API version (v1beta3, in particular).
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubelet/config/file_test.go

@bgrant0607 - this code already exists - see TestReadPodsFromFile case.

IIUC, the problem is that we don't support yaml representation - @satnam6502 changing it to json should fix the problem

satnam6502 · 2015-04-27T05:43:16Z

I'd rather add YAML support because I would like to keep these pod spec as easy to read as possible for humans.

wojtek-t · 2015-04-27T07:41:52Z

This was a more general approach. Basically, yaml support has been removed in #5533.
@smarterclayton is probably a good person to talk about the reason of it.

Basically the reason why it doesn't work for yaml is that pkg/conversion/decode.go doesn't support yaml (this support was explicitly removed in the PR I mentioned above).

smarterclayton · 2015-04-27T11:10:04Z

We do support yaml from the clients. It's converted to json on read. We used to support json in the kubelet, and I don't see it being difficult to support at kubelet the same way.

smarterclayton · 2015-04-27T11:10:49Z

It's only the api which doesn't support yaml directly.

satnam6502 · 2015-04-27T20:58:52Z

I've done as @wojtek-t suggests -- the v1beta3 pod specifications are in JSON now which seems to work.

$ kubectl get pods
POD                                            IP           CONTAINER(S)              IMAGE(S)                                                                            HOST                                    LABELS                                                                      STATUS    CREATED
elasticsearch-logging-wzmxj                    10.244.2.2   elasticsearch-logging     gcr.io/google_containers/elasticsearch:1.0                                          kubernetes-minion-igkh/104.197.26.147   kubernetes.io/cluster-service=true,name=elasticsearch-logging               Running   About an hour
etcd-server-kubernetes-master                               etcd-container            gcr.io/google_containers/etcd:2.0.9                                                 kubernetes-master/                      <none>                                                                      Running   About an hour
fluentd-elasticsearch-kubernetes-minion-794j   10.244.0.3   fluentd-elasticsearch     gcr.io/google_containers/fluentd-elasticsearch:1.4                                  kubernetes-minion-794j/104.197.49.90    <none>                                                                      Running   About an hour
fluentd-elasticsearch-kubernetes-minion-frky   10.244.1.4   fluentd-elasticsearch     gcr.io/google_containers/fluentd-elasticsearch:1.4                                  kubernetes-minion-frky/104.197.12.87    <none>                                                                      Running   About an hour
fluentd-elasticsearch-kubernetes-minion-igkh   10.244.2.3   fluentd-elasticsearch     gcr.io/google_containers/fluentd-elasticsearch:1.4                                  kubernetes-minion-igkh/104.197.26.147   <none>                                                                      Running   About an hour
fluentd-elasticsearch-kubernetes-minion-x3w8   10.244.3.2   fluentd-elasticsearch     gcr.io/google_containers/fluentd-elasticsearch:1.4                                  kubernetes-minion-x3w8/104.197.16.133   <none>                                                                      Running   About an hour
kibana-logging-1hlds                           10.244.1.2   kibana-logging            gcr.io/google_containers/kibana:1.2                                                 kubernetes-minion-frky/104.197.12.87    kubernetes.io/cluster-service=true,name=kibana-logging                      Running   About an hour
kube-apiserver-kubernetes-master                            kube-apiserver            gcr.io/google_containers/kube-apiserver:6e041e030b9fe74963fbcd46a1452794            kubernetes-master/                      <none>                                                                      Running   About an hour
kube-controller-manager-kubernetes-master                   kube-controller-manager   gcr.io/google_containers/kube-controller-manager:d1b165268f6fb527750deb8423e18d31   kubernetes-master/                      <none>                                                                      Running   About an hour
kube-dns-ehbk3                                 10.244.1.3   etcd                      quay.io/coreos/etcd:v2.0.3                                                          kubernetes-minion-frky/104.197.12.87    k8s-app=kube-dns,kubernetes.io/cluster-service=true,name=kube-dns           Running   About an hour
                                                            kube2sky                  gcr.io/google_containers/kube2sky:1.2                                                                                                                                                                             
                                                            skydns                    gcr.io/google_containers/skydns:2015-03-11-001                                                                                                                                                                    
kube-scheduler-kubernetes-master                            kube-scheduler            gcr.io/google_containers/kube-scheduler:c963235209411565086c7d7157355732            kubernetes-master/                      <none>                                                                      Running   About an hour
monitoring-heapster-controller-2tj34           10.244.0.2   heapster                  gcr.io/google_containers/heapster:v0.10.0                                           kubernetes-minion-794j/104.197.49.90    kubernetes.io/cluster-service=true,name=heapster,uses=monitoring-influxdb   Running   About an hour
monitoring-influx-grafana-controller-50mvz     10.244.3.3   influxdb                  gcr.io/google_containers/heapster_influxdb:v0.3                                     kubernetes-minion-x3w8/104.197.16.133   kubernetes.io/cluster-service=true,name=influxGrafana                       Running   About an hour
                                                            grafana                   gcr.io/google_containers/heapster_grafana:v0.6

satnam6502 · 2015-04-27T21:05:03Z

I've checked that logs can be viewed with Kibana (by adding a certificate to Chrome) and this indeed works. Now checking Cloud Logging.

bgrant0607 · 2015-04-27T23:26:46Z

Please file another issue for kubelet yaml support.

bgrant0607 · 2015-04-27T23:27:56Z

Note, however:

.../pkg/kubelet$ grep -r -i yaml .
./config/common.go:     "github.com/ghodss/yaml"
./config/common.go:     if err = yaml.Unmarshal(data, &manifest); err != nil {
./config/common.go:     if err = yaml.Unmarshal(data, &manifests); err != nil {

bgrant0607 · 2015-04-27T23:29:43Z

Those references were from manifest decoding. We could do the same for pod decoding.

satnam6502 · 2015-04-28T00:20:29Z

I've verified that the JSON versions of these pod specifications work by observing log lines being ingested using Kibana (for Elasticsearch) and the Cloud Logging Console for cloud logging.

I propose that we review/merge this change and if I get time I could adjust the kubelet to also accept YAML in which case I could revert these JSON files to their YAML equivalents.

bgrant0607 · 2015-04-28T00:24:13Z

LGTM

wojtek-t · 2015-04-28T06:15:32Z

I propose that we review/merge this change and if I get time I could adjust the kubelet to also accept YAML in which case I could revert these JSON files to their YAML equivalents.

+1
Can you please file another issue for it?

The change overall LGTM - I will merge it.

Convert node level logging agents to v1beta3

googlebot added the cla: yes label Apr 24, 2015

bgrant0607 reviewed Apr 24, 2015
View reviewed changes

bgrant0607 self-assigned this Apr 24, 2015

satnam6502 force-pushed the v1beta3 branch 2 times, most recently from c6846f3 to a0601c0 Compare April 25, 2015 00:47

satnam6502 force-pushed the v1beta3 branch from a0601c0 to 7be87fb Compare April 25, 2015 18:07

satnam6502 force-pushed the v1beta3 branch from 7be87fb to f1f4351 Compare April 27, 2015 19:38

satnam6502 force-pushed the v1beta3 branch from f1f4351 to fb3c0af Compare April 27, 2015 21:46

satnam6502 force-pushed the v1beta3 branch from fb3c0af to b302ed3 Compare April 27, 2015 23:48

satnam6502 mentioned this pull request Apr 27, 2015

The Kubelet should accept YAML pod specifications #7406

Closed

Convert node level logging agents to v1beta3

4a0cc79

satnam6502 force-pushed the v1beta3 branch from b302ed3 to 4a0cc79 Compare April 28, 2015 00:22

bgrant0607 added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 28, 2015

wojtek-t added a commit that referenced this pull request Apr 28, 2015

Merge pull request #7274 from satnam6502/v1beta3

92a0b38

Convert node level logging agents to v1beta3

wojtek-t merged commit 92a0b38 into kubernetes:master Apr 28, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert node level logging agents to v1beta3 #7274

Convert node level logging agents to v1beta3 #7274

satnam6502 commented Apr 24, 2015

satnam6502 commented Apr 24, 2015

bgrant0607 Apr 24, 2015

bgrant0607 commented Apr 24, 2015

satnam6502 commented Apr 25, 2015

bgrant0607 commented Apr 25, 2015

satnam6502 commented Apr 25, 2015

satnam6502 commented Apr 25, 2015

satnam6502 commented Apr 25, 2015

satnam6502 commented Apr 25, 2015

satnam6502 commented Apr 25, 2015

bgrant0607 commented Apr 25, 2015

wojtek-t commented Apr 27, 2015

satnam6502 commented Apr 27, 2015

wojtek-t commented Apr 27, 2015

smarterclayton commented Apr 27, 2015

smarterclayton commented Apr 27, 2015

satnam6502 commented Apr 27, 2015

satnam6502 commented Apr 27, 2015

bgrant0607 commented Apr 27, 2015

bgrant0607 commented Apr 27, 2015

bgrant0607 commented Apr 27, 2015

satnam6502 commented Apr 28, 2015

bgrant0607 commented Apr 28, 2015

wojtek-t commented Apr 28, 2015

Convert node level logging agents to v1beta3 #7274

Convert node level logging agents to v1beta3 #7274

Conversation

satnam6502 commented Apr 24, 2015

satnam6502 commented Apr 24, 2015

bgrant0607 Apr 24, 2015

Choose a reason for hiding this comment

bgrant0607 commented Apr 24, 2015

satnam6502 commented Apr 25, 2015

bgrant0607 commented Apr 25, 2015

satnam6502 commented Apr 25, 2015

satnam6502 commented Apr 25, 2015

satnam6502 commented Apr 25, 2015

satnam6502 commented Apr 25, 2015

satnam6502 commented Apr 25, 2015

bgrant0607 commented Apr 25, 2015

wojtek-t commented Apr 27, 2015

satnam6502 commented Apr 27, 2015

wojtek-t commented Apr 27, 2015

smarterclayton commented Apr 27, 2015

smarterclayton commented Apr 27, 2015

satnam6502 commented Apr 27, 2015

satnam6502 commented Apr 27, 2015

bgrant0607 commented Apr 27, 2015

bgrant0607 commented Apr 27, 2015

bgrant0607 commented Apr 27, 2015

satnam6502 commented Apr 28, 2015

bgrant0607 commented Apr 28, 2015

wojtek-t commented Apr 28, 2015