Can we start with clear statements of requirement? What we have with local
volumes is already pretty durable, as long as the pod stays put. That may
be a side effect of the implementation, but maybe we should keep it.

Alternatively, maybe the answer is "don't use local storage". Just like
GCE has a PD associated with a VM, we could have something similar with
pods.

But that is getting ahead - I don't feel like I really understand the
required aspects of this.

On Thu, Jul 24, 2014 at 5:52 PM, bgrant0607 notifications@github.com
wrote:

I renamed this issue to narrow it to the specific use case.

Support for durable local storage is an issue that has been raised by
several partners in discussions, and is evident in every example
application we've looked at (Guestbook, Acme Air, Drupal). This is a
requirement for running a database, other storage system (e.g., HDFS,
Zookeeper, etcd), SSD-based cache, etc.

Support for more types of volumes ( #97
#97 ) is maybe
necessary but definitely not sufficient. We also need to represent the
storage devices as allocatable resources ( #168
#168 ).

As I mentioned in #146
#146 , pods are
currently relatively disposable and don't have durable identities. So, I
think the main design question is this: do we conflate the identity of the
storage with the identity of the pod and try to increase the durability of
the pod, or do we represent durable local volumes as objects with an
identity and lifetime independent of the pods? The latter would
permit/require creation of a new pod that could be attached to pre-existing
storage.

The latter is somewhat attractive, but would obstruct local restarts,
which is desirable for high availability and bootstrapping, and wouldn't
interact well with replicationController, due to the need to create/manage
an additional object and also to match individual pods and volumes, which
would reduce the fungibility of the pods.

So, I'm going to suggest we go with pod durability.

Rather than a single pin/persistence bit, I suggest we go with
forgiveness: a list of (event type, optional duration, optional rate)
of disruption events (e.g., host unreachability) the pod will tolerate. We
could support an any event type and infinite duration for pods that
want to be pinned regardless of what happens.

This approach would generalize nicely for cases where, for example,
applications wanted to endure reboots but give up in the case of extended
outages or in the case that the disk goes bad. We're also going to want to
use a similar spec for availability requirements / failure tolerances of
sets of pods.

Ideally, the pod could be restarted/recreated by Kubelet directly. This
would likely require checkpointing #489
#489 , but
initially we'd at least need to be able to:

not transition the pod to a stopped state and/or delete it, or at
least be able to recreate it with the same identity

recover pre-existing storage from a well known place

Regarding the former, we probably need to introduce some indication of
outages into the pod status -- probably not the primary state enum, but in
a separate readiness field.

Regarding the latter, there are cases where it is convenient to place the
storage in the host in a user-specified location, to facilitate debugging,
data recovery, etc. without needing to look up long host-specific
system-generated identifiers, though that's probably not a requirement for
v0.

It might be nice to have a way for a durable pod to have a way to request
to delete itself without making an API call. Some people have suggested
that run-until-success (i.e., exit 0) is not a sufficiently reliable way to
convey this. Perhaps we could use an empty volume on exit as the signal.
Certainly that would mean there wasn't any valuable data to worry about,
and it would be easy for an application to drop an empty file there if it
just wanted to stay put.

Support for raw SSD should be filed as a separate issue, if desired.

@thockin https://github.com/thockin @johnwilkes
https://github.com/johnwilkes

Reply to this email directly or view it on GitHub
#598 (comment)
.

Durable local storage #598

Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions