Would it be better to make this explicit in the manifest (VAR_A=$K8_POD_ID) rather than implicit? I ask because in Openshift we found this led to lazy coupling, and in scenarios where you have code that wants a unique id, you want to be able to alias names instead. At least that couples k8s to an image via a specific resource provided by k8s, vs making the image dependent on k8s.

@smarterclayton Hmm. I see the case for doing the substitution in k8s rather than in the container, though I do have some concerns about introducing yet another substitution mechanism.

I was thinking that, at least for now, any additional information the container wanted about itself could be retrieved by querying the apiserver, which would obviously also couple the querier to k8s.

I think we are going to need to standardize on a "downward-facing API", including this type of contextual information (and also IP address, DNS name, auth info, etc.), but also a variety of notifications and/or hooks (#140). If we had hooks, we could make the context info available to the hooks, which could then normalize the info for the application. I see hooks as being necessarily specific to the hosting framework and other glue services.

That's true, although one strong goal in docker has been to try and offer via libchan/libswarm a "Docker" abstraction where on multiple platforms images can invoke consistent actions that affect the platform. That applies to more generic operations, whereas k8s specific actions will always require some downwards API. The abstraction is a long term play that requires active participation by a number of stakeholders and also implemented, demonstrated use cases, so it's not an argument for avoiding trying.

I don't think the downward API is bad, but some of the use cases (unique id for an instance, what ip do I have) might benefit from looser coupling or by trying to find the most lowest common denominator expression. In the end image authors can always ignore those settings, but you do want image users to have the tools to bend a recaltricant image to their will.

As much as I want to keep this simple, Clayton's argument of using a non
k8s native image in k8s by application of clever env vars is compelling.

But what scope? Env vars only? Env vars and command lines?
On Jul 9, 2014 5:08 PM, "Clayton Coleman" notifications@github.com wrote:

That's true, although one strong goal in docker has been to try and offer
via libchan/libswarm a "Docker" abstraction where on multiple platforms
images can invoke consistent actions that affect the platform. That applies
to more generic operations, whereas k8s specific actions will always
require some downwards API. The abstraction is a long term play that
requires active participation by a number of stakeholders and also
implemented, demonstrated use cases, so it's not an argument for avoiding
trying.

I don't think the downward API is bad, but some of the use cases (unique
id for an instance, what ip do I have) might benefit from looser coupling
or by trying to find the most lowest common denominator expression. In the
end image authors can always ignore those settings, but you do want image
users to have the tools to bend a recaltricant image to their will.

Reply to this email directly or view it on GitHub
#386 (comment)
.

I find it compelling, too. Internally, we have thousands of uses of the downward API, and I agree it introduces unnecessary coupling. OTOH, I'd like to think more about the configuration and parameterization story before introducing substitution.

As opposed to the k8s-specific mechanism I proposed, perhaps we could start by thinking about what a typical application would need to know about itself and how that information could be provided in the least entangled manner.

I propose a general approach: We should use/extend standard networking and filesystem-based mechanisms for the downward and upward container APIs as much as possible. We should not use custom environment variables (which can't be dynamically updated) nor REST APIs, and the mechanisms should not be intrinsically specific to Kubernetes (so no kubernetes in paths/names).

Downward:

• Service discovery
  • localhost for containers within the same pod
  • IP-per-service (DESIGN: Services v2 #1107) + DNS for services (DNS #146, Proposal: Internal Dynamic DNS service #1261) + SRV for services, if needed
  • Pod IP address for custom discovery/LB self-registration mechanisms: SIOCGIFCONF
  • /etc/hosts for special-case cluster and local services, such as apiserver and cadvisor
  • user-defined environment variables to specify (predictable) hostnames and/or ports, as necessary
• IDs
  • hostname set to pod name
  • /etc/machine-id set to pod UID
• Standard environment variables
  • Hopefully mostly covered by Docker, and/or build-file ENV lines, and/or manifest env clauses
  • May want to adopt one of the distribution file-based mechanisms, such as /etc/environment (ubuntu), to facilitate dynamic updates and propagation to child processes
• Resource discovery
  • /proc: requires Linux fixes (http://fabiokung.com/2014/03/13/memory-inside-linux-containers/)
  • /dev: any special devices mounted
  • volumes: no special mechanism, since the user can specify where they are mounted
• OS discovery: /proc/version and /etc/os-version
• ssh keys: whatever file location(s) and/or standard env. vars. are appropriate (e.g., SSH_AUTH_SOCK)
• Context propagation, such as for logging, monitoring, and configuration: Virtual machine environments, such as VMWare, EC2, and GCE, provide mechanisms which are tightly coupled to those environments. We could use a mechanism that could at least potentially be supported more widely.
  • Labels (Pass pod labels to containers and to lifecycle hooks #560): /etc/labels in key=value format (as with /etc/machine-info)
  • Annotations (Document annotations and guidelines regarding when to use annotations vs labels #1201): /etc/annotations.d/
• Events: lifecycle hooks (PreStart and PostStop event hooks #140)

TODO: Master election, shard assignment.

Upward:

• Container output/messages
  • /dev/console or /dev/something (Capture application termination messages/output #139)

References:

• FSH
• LSB
• Container RFC

Also, while we could make the paths configurable for some of these files, I don't think we need to. Users could set their own environment variables and evaluate them on the command line (as described in https://github.com/GoogleCloudPlatform/kubernetes/wiki/User-FAQ), or could extract the information in a PreStart hook (#140, #1445).

Missing here - resource limits and usage?

@thockin That's "resource discovery".

We need to think hard about what cgroups interfaces we expose. On the one
hand, it's nice to be able to run sub-containers yourself. On the other
hand, the kernel world does NOT want to support that operating mode and it
is very fragile for all but the most sophisticated users.

On Mon, Oct 6, 2014 at 9:13 PM, bgrant0607 notifications@github.com wrote:

@thockin https://github.com/thockin That's "resource discovery".

Reply to this email directly or view it on GitHub
#386 (comment)
.

@thockin User subcontainers is #1568. Perhaps the solution to that overlaps with resource discovery, or perhaps not. Rather than directly reading cgroup files, I'd be fine if they were wrapped with a system utility and/or library, but it shouldn't require a daemon and definitely shouldn't require a HTTP API. Ideally, Linux apps would not need to be container-aware in order to work.

We don't need to design it here, but we should definitely make sure Vish,
Victor, and Rohit are game to support whatever we decide. I'm not sure
daemon-free can work in this new world, which makes me very sad.

On Mon, Oct 6, 2014 at 9:58 PM, bgrant0607 notifications@github.com wrote:

@thockin https://github.com/thockin User subcontainers is #1568
#1568. Perhaps
the solution to that overlaps with resource discovery, or perhaps not.
Rather than directly reading cgroup files, I'd be fine if they were wrapped
with a system utility and/or library, but it shouldn't require a daemon and
definitely shouldn't require a HTTP API. Ideally, Linux apps would not need
to be container-aware in order to work.

Reply to this email directly or view it on GitHub
#386 (comment)
.

Being discussed in Docker land: moby/moby#8427

We did some discussion on this topic starting around here: #2316 (comment)

Rancher's metadata service: http://rancher.com/introducing-rancher-metadata-service-for-docker/

I propose we close this umbrella bug and open new issues for specific points remaining.

As much as I want to keep this simple, Clayton's argument of using a non
k8s native image in k8s by application of clever env vars is compelling.

But what scope? Env vars only? Env vars and command lines?
On Jul 9, 2014 5:08 PM, "Clayton Coleman" notifications@github.com wrote:

That's true, although one strong goal in docker has been to try and offer
via libchan/libswarm a "Docker" abstraction where on multiple platforms
images can invoke consistent actions that affect the platform. That applies
to more generic operations, whereas k8s specific actions will always
require some downwards API. The abstraction is a long term play that
requires active participation by a number of stakeholders and also
implemented, demonstrated use cases, so it's not an argument for avoiding
trying.
I don't think the downward API is bad, but some of the use cases (unique
id for an instance, what ip do I have) might benefit from looser coupling
or by trying to find the most lowest common denominator expression. In the
end image authors can always ignore those settings, but you do want image
users to have the tools to bend a recaltricant image to their will.

Reply to this email directly or view it on GitHub
#386 (comment)
.

Create a deployment as follows
 Name: nginx-dns
 Exposed via a service: nginx-dns
 Ensure that the service & pod are accessible via their respective DNS records
 The container(s) within any Pod(s) running as a part of this deployment should use the nginx
image
Next, use the utility nslookup to look up the DNS records of the service & pod and write the
output to /opt/..../service.dns and /opt/..../pod.dns respectively. Ensure you use the
busybox:1.28 image (or earlier) for any testing, an the latest release has an unpstream bug which
impacts the use of nslookup.

😊
kubectl run nginx-dns --image=nginx
kubectl expose deployment/nginx-dns --port=80 --name=nginx-dns
kubectl run busybox -it --rm --image=busybox:1.28 sh
nslookup nginx-dns #svc 记录
nslookup pod-ip-addr #pod 记录
Nslookup 的结果不能直接输出到文件，我是粘贴到notepad 上再用