Shared Pod Storage (e.g. Config Storage) #6923
Comments
|
@smarterclayton can you comment on this? |
|
Will do
|
|
We've talked about exposing "Etcd-as-a-service" - allowing clients to request an endpoint with which they can interact using etcd client tools directly, but secured and provisioned dynamically. A subset of the keyspace would be carved up for each use case and offered on a distinct endpoint. One advantage would be the ability to scale that out - for simple systems it could reuse the apiserver, for larger systems it could be sharded and decoupled. Once volume as a service lands (persistent volumes) it should be progressively easier to do things like run gluster and use shared, secured mounts (service accounts and security contexts will let you allocate unique Unix uids for each namespace as needed, for sharing remote storage). Single server etcd should become much easier when pds are in place - you should be easily able to run small etcd servers that are resilient to failure (perhaps not clustered) for low cost (40-50mb per?) Trying to think of other ideas we've covered... |
|
The nascent config API object? On Thu, Apr 16, 2015 at 7:36 PM, Clayton Coleman notifications@github.com
|
|
I thinking would be an exact match for the etcd API, and be targeted at etcd. I feel something is wrong about mutating the etcd API to be even more generic, mostly because then people need custom clients in every language.
|
|
Yeah, if we expose etcd it should be as a service, not because it happens On Thu, Apr 16, 2015 at 9:27 PM, Brian Grant notifications@github.com
|
|
Yeah, I'm thinking an optional API endpoint either provisioned automatically for clients or available via requests. In the long term, this is effectively the service broker pattern - request an instance of X (where X can be anything) provisioned and attached to your namespace (external service created, environment set on an RC, secrets set in the ns that grant access). You can then talk to your local service to access a remote resource without having to know anything about the implementation of that resource. Common in paas environments.
|
|
PaaSes give me the impression that there's a hard distinction between Apps and Services -- the former are run on the PaaS while the latter are run using an underlying IaaS orchestration layer. I think it's great if we can support that, but even better if applications that would be run as Services in a PaaS environment could be run on Kubernetes. In this case, we definitely want to be able to run etcd clusters on Kubernetes, using features such as nominal services (#260). That said, I'd like extensions/plugins to be on equal footing with core objects. I don't see a compelling reason to not allow them to use the same etcd instance (via the apiserver) so long as we apply proper per-user (not per-plugin) storage quotas and request rate limits. In my experience, users can't really do more damage than they'd do with just pods alone: out-of-control replication, crash loops, enormous (multi-megabyte) command line args or env var lists, ... As for watch scaling, I fully expect that eventually virtually every container running in the cluster will have multiple watches active simultaneously. We're going to need a replicated fanout layer to handle that. |
|
----- Original Message -----
Agree - it should be easy to bridge the distinction and we should have the concepts that blur the line between
The tools and interaction for something running in the cluster to all of the above should be identical - only provisioning changes.
|
thockin
added
kind/support
bgrant0607
added
area/app-lifecycle
area/extensibility
sig/api-machinery
ghost
removed
the
|
Closing for lack of activity and because we have config map now. |
I am working on a Kubernetes extension that would run in its own pod but would like to store some information (e.g. configuration) that has similar requirements as the data stored in API Server. In particular it would be nice if the storage had the following properties:
Possibilities:
Use Volumes
Applications can always mount a volume and write their own storage to files. This is not ideal for a few reasons:
Have applications run their own storage as a pod
Running your own etcd cluster is an obvious choice. However, running storage as a pod is actually quite tricky. Durability and consistency are difficult to provide and are often better maintained by a dedicated team that knows about backup and recovery. Durability itself requires some off pod storage which brings you back to volumes. Storage pods that are restarted need to rediscover and reattach the persistent storage used by a previous instance even though their identities are not related in any way (see Nominal Services).
API Server's Etcd Cluster
The etcd server used by the API Server provides almost all of these features with the exception of isolation. One possibility would be to expose a thin shim service at the API Server that wraps its underlying etcd cluster and re-exposes scoped portions of its namespace to pods. The shim would enforce scoping and authorization. The shim could also provide some adaption of the etcd watch interface to be more consistent with the semantics exposed by the API Server itself.
Pros: This possibility unifies the problems of durability, consistency, addressability with the API Servers almost identical requirements. Since any deployment must already solve these issues for API Server it doesn't create any additional burden. In hosted environments (e.g. GKE) the host provides Etcd and its durable storage in a way that is independent of the Kubernetes model allowing for backup, restore, and survivability to be implemented in ways that would be difficult if run directly as a pod. API Server already implements a sophisticated RESTful web service endpoint with authentication, authorization, timestamping, and watch support. This logic could be shared by the shim without addition complexity or duplication.
Cons: Opening up the API Server's Etcd server (even if the shim correctly implements isolation) to third-party applications will create additional load on both API Server and Etcd. This would affect the responsiveness of API Server and scalability in ways that may be difficult to predict. Bugs in the shim or Etcd might expose the data from either the master or other applications to corruption or deletion.
The text was updated successfully, but these errors were encountered: