-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pods need to pre-declare service links iff they want the environment variables created #1768
Comments
How will services v2 interact with namespaces and authorization? If namespaces are used to separate different companies or different organizations within a large company, then probably most of the time namespace owners will:
Even a cluster's users are confined to a few cooperative organizational units, they might want:
On the other hand, having to declare all dependencies seems like a bad user experience. There is probably a reason why we have wished for this but not implemented it for so many years. One compromise might be to default to fully connected within a namespace but require explicit connection across namespaces. |
Generally predeclaration seems valuable for making arbitrary software to work, and automatic injection seems useful for Kube designed software. Predeclaration works well for controlling dependencies, and automatic injection works well for forcing tolerance of missing components (degrading components?).
|
|
What would an explicit dependency from a pod to a service look like:
|
+100 |
@bgrant0607 a client of mine wants to run mini user applications using untrusted user images as linked services .. so security is 1. 2, have different services with the same name so say users A and B both have a REDIS and a WEB service. |
@kelonye Namespaces were intended to address the multi-user issue. As for trusted vs. not, that's the firewall issue (#2880) -- which clients are permitted to see a service. If we continue to support the environment variables, we are going to need to make them on request only. Creating so many variables for all services in a cluster, even just within a namespace, is a scalability problem. That has nothing to do with accessibility, though. It would just save the user from having to write a pre-start hook to resolve the service's DNS name and dump it into a file to be read by the container. A separate issue is start-up dependencies. Though they're fragile, a number of applications do make (bad) assumptions about startup order, so we'll need to support them in some form in our deployment workflow mechanism(s) (#1704). |
The following use-case requires predeclaration:
Predeclaration would also play a role in the cross-namespace use-case:
I'll also note that there is a use-case where you want to opt-out of predeclaration and get info / firewall rules for every service in a pod's namespace. This adds a wrinkle because it divides pods into two types, those that follow the normal rules and those that don't, which would definitely impact scheduling in order to keep an opted-out pod from wrecking port allocation on a host where all other pods predeclare. I think @erictune's suggestion of defaulting to fully connected within a namespace and requiring predeclaration of cross namespace dependencies is a good middle ground. @bgrant0607 @erictune @smarterclayton As a next step, how about a PR to explore the predeclaration mechanism in a vacuum. I would suggest that
The first iteration could change the |
Moving BoundPodFactory stuff to kubelet is good and necessary, but @erictune may be working on that already. If we switch to DNS and deprecate env vars completely, does that eliminate the need for predeclaration? That sounds much easier... |
@lavalamp I don't think switching to DNS and deprecating env vars fully eliminates the need for predeclaration. There there's still the problem of iptables rules on the node. |
You still have to know what DNS you're looking forward to, and software that runs on different namespaces or clusters won't have the same DNS. You need something injected into the container that lets legacy software react to the cluster topology. Predeclaration for adaption is a key thing, there's lots of software out there that doesn't know anything about X_SERVICE_HOST at all. Also, that same software has to work outside of a cluster as well - on a local dev box how would you point your app to your db (except by using env or mutating a file on disk)?
|
|
@smarterclayton We can roll adaptation into the POC, I will think through a design and propose a model here |
@pmorie can you give a more concrete example of a system with legacy software that needs service links? |
@erictune As a more detailed example, say I have an image that depends on a specially formatted environment variable with a URL for a service. As an example format, take:
This use case is to be able to adapt to these special requirements an image may have without changing the image. It's definitely the case that there's a pretty sizable subset of these cases that can be addressed by performing the translation via the shell and either setting containers' commands to set the variables or wrapping an image with another image containing a script. It's arguable that for those cases, the adaptation mechanism isn't necessary. However, it is necessary for images that do not contain a shell, or that use the |
Or even the mysql client (http://dev.mysql.com/doc/refman/5.0/en/environment-variables.html) MYSQL_HOST Neither of those match our existing env. ----- Original Message -----
|
@smarterclayton and I discussed this offline and think the adaptation use-case doesn't depend on predeclaration of services nor does it need to be coupled to services at all. Here's another use-case that might require pre-declaration:
Consider the following to address that use-case:
Any thoughts? @smarterclayton @erictune @bgrant0607 @thockin @lavalamp |
Also, in the context of the above, it would be good to produce an event after some time if a pod which is scheduled cannot start, but that's perhaps another issue. |
Previous discussion of this last type of dependency was in #2385. It has been my hope that DNS will eliminate the creation order problem between services and their clients. Creation-order dependencies are problematic since containers can go down at any time, and they have unclear meaning when updating or replacing objects (such as with rolling updates). That said, sometimes there are unavoidable turn-up dependencies, such as to initialize stateful services, such as databases or message brokers. I envision handling such dependencies in deployment automation: #1704. |
I just experienced this, as additional confirmation: The new directory-reading feature of kubectl is not so useful for containers using service environment variables. Someone tried to do:
|
@smarterclayton I notice a huge delay when spawning bash via "docker exec" if there are thousands of services (I tested with 3000) in the namespace. It would be great to have options to disable service env when creating docker process. The test I run is as following: (kubelet 1.6) |
cc @kubernetes/sig-network-feature-requests @kubernetes/sig-node-feature-requests |
Issues go stale after 90d of inactivity. Prevent issues from auto-closing with an If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or |
/remove-lifecycle stale |
Now that service links are optional (dealing with some of the performance and collision issues that they used to cause), we're mostly waiting on a way to declare a subset of them for an application here, and it seems no headway has been made on that recently, I'm going to bump this down to important-longterm. /priority important-longterm |
/remove-priority important-soon |
/remove-kind design
|
Realistically, no. Service links are not something we're going to do more to support. |
Forked from #1107 and #386.
Now seems like a good time to decide whether we want to require/encourage/allow pods to declare services they depend upon. Internally, we've often wished we had such a mechanism.
Not only would pre-declaration reduce accidental/lazy coupling, but it would also improve scalability by reducing the number of iptables rules that must be created. Pre-declaration would also be compatible with Docker's approach to links. If we supported service aliasing in these declarations, that would facilitate dependency injection for tests and a wide variety of deployment adaptation scenarios, which seems like a compelling alternative to custom environment variables, command-line flags, dynamic configuration services, and so on.
However, it at least needs possible to opt out of static declaration and/or enforcement of service dependencies, such as in the case that dependent services are registered dynamically -- think of a proxy, load balancer, web browser, monitoring service, or naming/discovery service running in a container.
We'd also definitely need to support more flavors of services (cardinal services, headless services, master election, sharding, ...) in order for most clients to be able to utilize the pre-declaration mechanism.
Something else to consider is how to address dependencies pulled in by client libraries, though perhaps it's not unreasonable to require client libraries to be transparent regarding which services they access.
/cc @thockin @smarterclayton
The text was updated successfully, but these errors were encountered: