couldn't wait to clean up our current name mess, but we might still need to hash those labels.

@dchen1107 is the plan to use labels and drop the prefix naming weirdness?

also it just got merged.

we might keep some naming stuff, just so 'docker ps' is useful, but we can
stop jamming 6 or 7 fields into the name.

On Tue, Mar 17, 2015 at 10:10 AM, Timothy St. Clair <
notifications@github.com> wrote:

@dchen1107 https://github.com/dchen1107 is the plan to use labels and
drop the prefix naming weirdness?

—
Reply to this email directly or view it on GitHub
#3764 (comment)
.

Cool, the label PR was merged to docker 1.6.

Yes, we did plan to cleanup naming mess by pushing container hash and uid to metadata, only leaving container name, pod name and namespace. But before that we need to make sure query based on labels does work as what we expected and there is no significant performance downgrade.

This would be great to see. Has there been any movement on this?

@jimmidyson What would you like to see?

We support Docker versions that don't have labels support, so we can't depend on it yet.

I'd like to see kubernetes metadata in labels, e.g. namespace, pod name, container name, even pod labels too. My use case is to enrich runtime data (logs, metrics. etc) without having to make API server requests, removing the need to distribute secrets & set up auth to the API server to do this. While I can parse the container name to get a lot of this info, this is fragile, dependent on an implementation detail. If Docker labels aren't present (e.g. because running on old version of Docker) then I could fall back to container name parsing, but labels would be much easier.

Understood. Yes, we want to enable that. I wouldn't use Docker labels, though. See also #560.

Unless I've misunderstood, #560 seems to talk more about pods knowing about their own kubernetes context, rather than being able to see a container's kubernetes context from the outside. Is there another way to get this? Out of interest, why not Docker labels?

@jimmidyson From what/where do you want to access the Docker labels, and how would it discover them?

BTW, do you know whether Docker labels can be updated? I don't remember what the final state was in that area.

So right now thinking of enriching logging data (Docker logs) with Kubernetes metadata. This is currently being done by a pod (fluentd) running on each node, mounting the Docker log dir to watch log files & parsing the container name to extract Kubernetes metadata. Want to extend this to add in labels, etc which are currently only available via a call to the API server. Whether it's Docker labels or not it seems more efficient & flexible to have this data available without requiring a call to the API server.

Not sure whether Docker labels can be updated. If not, that would definitely point to a different solution.

@vmarmol @vishh @satnam6502 How do we plan to deliver labels to fluentd, etc. for logging and monitoring?

@bgrant0607 we're also encoding some Kubernetes-metadata into the container labels (like what pod the belong to) so they can either (1) query Docker directly and make the mapping themselves, or (2) ask the Kubelet through the undocumented API. cAdvisor is the first consumer AFAIK and it is doing (1). Heapster is using this data through cAdvisor.

I imagine we do not want to surface Docker labels as Kubernetes.

@vmarmol with 1 are you just mapping from docker container name to kubernetes metadata or is there further info available by querying docker? How do you get a pod's labels by querying docker? I can't see that info available through docker.

@jimmidyson today we just export a label that specifies the pod's full name. We don't push any other metadata to Docker (including labels). I'd be curious what usecases you have in mind. For the most part, we're encouraging contacting the Kubelet to get this type of information instead as the details of how the containers are un are up to the Kubelet. For example, if we use rkt, the Docker labels won't even exist.

@vmarmol yeah that's what I thought. Agree that docker labels are too low level implementation dependent to use now.

Pod's full name isn't too useful for searching/aggregations. When I'm building dashboards I want to be able to visualise data in different ways, based on different factors: e.g. pod name or labels (show me all the logs matching these pod name or labels), docker image (compare historical performance/utilisation between different versions of same docker image), etc.

Basically the more info available the easier the more flexible visualisation can be created.

I have some fluentd plugins I've written to enrich with a lot of this data already, but was looking for something that didn't require calls to api server or kubelet. Are you interested in adding this level of metadata to the logging addon & heapster?

Labels are intended to flow from configuration, through the API and control plane, to the node, to the containers, and to logging and monitoring systems. We definitely want to support that.

Exposing labels is the purview of what we call the "downward API" -- #386. I don't want to encourage calls to Kubelet, other than by maintained control-plane components, such as cAdvisor. It has too wide of an API.

It's true that we don't want to build in application/user dependencies on the Docker API because it compromises isolation, security, and privacy, and because we plan to support multiple runtimes.

@jimmidyson: Tools like fluentd should have native support for kubernetes, similar to how they support docker. Hacking using labels might be helpful in the short term, but does not seem like a good long term solution. We haven't tackled native kube support for logging yet. It is post v1.0 goal.

@bgrant0607 suggested using annotations for getting application specific metadata plumbed through to logging and monitoring components like fluentd, heapster, etc.

@dchen1107: Does kubelet fallback to overloading container names in the case of rocket?

Annotations for communicating necessary info between components.

Labels for conveying user-applied identifying metadata.

@vishh Fluentd doesn't have native support for docker AFAIK. It simply tails docker log files & adds container name. Please correct me if I'm wrong though. I have a couple of fluentd plugins that enrich with docker & kubernetes metadata which you might find useful.

Although it may be a post v1.0 goal, I'd like to contribute to it now because I personally see it as very important. I'm not sure what you mean by hacking using labels. If you're taking of docker labels then yes we all agree that is a bad solution. But having logs/metrics tagged with kubernetes labels is essential.

@vishh rkt has it's own store of data. Most of the pod's data is offloaded to it.

But having logs/metrics tagged with kubernetes labels is essential.

Agree.

@jimmidyson: Agreed. Labels are necessary. We don't have a way to tag logs with labels yet. Can you provide pointers to your custom fluentd plugins?

@vishh - see https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter & https://github.com/fabric8io/fluent-plugin-docker_metadata_filter, including config examples & filtered output.

cc @satnam6502 @a-robinson

@jimmidyson : how do you set up the auth information for speaking to the api server? I guess I'd like to know how to specify the config params -- is this done when the gem file is installed through flags?
This looks great -- I hope I can use it for Kubernetes logging to Elasticsearch and Cloud Logging.

@mr-salty : the Fluentd plugin looks like what we wanted to write.

@satnam6502 - currently requires cert/key files so using secrets. This is working nicely but we can tweak as required, service accounts when they land.

Configuration options in the fluent.conf are detailed in the readme. How about I put in a pr to update the existing fluentd image in kubernetes repo?

Well there are two images to update: one for Fluentd to Elasticsearch and the other for Fluentd to Cloud Logging. I did read the README.md but I did not see any mention of a flentd.conf file?
I'm keen to do this myself (and learn) unless you have a burning desire to do it (I am sure you know better than me about how to make this change) in which case -- go for it!

@jimmidyson : why not use a bearer token to authenticate to the apiserver?

@jimmidyson : Regarding the cache, what if a label gets removed/updated from a container, but a log line is emitted after that point -- but the label is in the cache -- so then the log line gets associated with the wrong label and/or value?

Can we open an issue or add to existing to track the logging as we've kind of hijacked this one!

The configuration options in the readme are options to add in to the fluent.conf filters. How would you feel about building a base image with required gems & creating the es/gcl images on top of that, only differing in configuration files?

@jimmidyson let's continue the discussion on #8001

@satnam6502 - I have the same concern about the cache but it's required to reduce unnecessary calls to api server. Options are putting a TTL on cached info (easy but how long TTL?) or using watches on pod info to update cache (slightly trickier but should be doable).

Bearer tokens would also have to handle refreshes, certs seem easier for now, but we can change if needed.

#5093 is an in-progress solution that pulls labels from the apiserver.

Is there any progress regarding this?

@aanm Some related things are working in progress here #15089. :)

@Random-Liu I'll just link this one as well moby/moby#15496 ;-)

It's very well feasible, that for example a logging tool (like Sematext Logsene or Fluentd) wants to collect the logs from all containers, and would then do some sort of routing (or tagging) based on the source container. This could be perfectly solved in a dynamic way with labels, without having to touch logging configuration for each new pod/container. However such a "low-level" logging tool shouldn't necessarily care about the orchestration layer (Kubernetes) because in many cases that might be completely irrelevant. That's why it would be very good, if pod/custom labels could be propagated to the Docker container.

@szakasz: We need to be able to update labels. Docker doesn't support that yet. Is it possible to build a simple bridge from k8s to the tools you mentioned, to make them k8s compatible?

@szakasz We added some docker container labels in K8s 1.2 #15089 (See here)

Hi,
It 's possible build Dockerfile using Kubernetes??