Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PetSets PVCs are not spread across Zones #27256

Closed
justinsb opened this issue Jun 12, 2016 · 8 comments
Closed

PetSets PVCs are not spread across Zones #27256

justinsb opened this issue Jun 12, 2016 · 8 comments
Labels
area/stateful-apps priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@justinsb
Copy link
Member

PVC volumes are currently created in a single zone, or at least not reliably spread across zones. Presumably someone running a zookeeper PetSet on HA k8s wants zone-failure tolerance, so we should spread the created volumes across zones.

I'm going to work up a strawman approach to this, but wanted to record the issue separately.

cc @bprashanth
@bprashanth
Copy link
Contributor

we should be able to express zone spreading through pod antiaffinity and the pvs should follow. Is this not the case?

@justinsb
Copy link
Member Author

@bprashanth I don't believe so. Volume creation is separate from pod creation. Currently the volumes are all created in the "master zone". It isn't clear what happens in multi-AZ scenarios, but if KCM is doing the volume creation then I expect they will still end up in one zone. I don't think we have any reasonable hope of 3 volumes landing in 3 separate zones anyway, which is what I think we want.

Going to put up a super-strawman PoC in a few minutes, hopefully should be clearer then!

@bprashanth
Copy link
Contributor

Maybe not today, but in general, why would we not want them to simply follow the pods?

@bprashanth
Copy link
Contributor

By default i mean, there are obviously scenarios where one might want to explicity specify the zone of a pv, and I still think that should be possible.

@justinsb
Copy link
Member Author

Not sure I follow the first comment. On GCE & AWS, volumes are created in a specific zone and can't be attached cross-zone and can't easily be moved. The volume's zone will actually dictate the pod's zone.

My PoC doesn't do a lot, but it does let you specify a zone explicitly! Stay tuned...

@justinsb justinsb mentioned this issue Jun 12, 2016
Closed
@justinsb
Copy link
Member Author

I'm going to tag this for the 1.3 milestone, so it stays on the radar. I think there are some options to fix this (as discussed in 27257). I think the current behaviour will be very surprising / disappointing to people trying out PetSets in 1.3.

@justinsb justinsb added this to the v1.3 milestone Jun 14, 2016
@goltermann goltermann added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jun 14, 2016
@lavalamp lavalamp removed this from the v1.3 milestone Jun 15, 2016
@lavalamp
Copy link
Member

Kicking out stuff from 1.3, it's too late.

@justinsb
Copy link
Member Author

Fix in #27553

chrislovecnm pushed a commit to chrislovecnm/kubernetes that referenced this issue Jun 21, 2016
@justinsb @chrislovecnm
GCE/AWS: Spread PetSet volume creation across zones
4f48dbc 
Long term we plan on integrating this into the scheduler, but in the
short term we use the volume name to place it onto a zone.

We hash the volume name so we don't bias to the first few zones.

If the volume name "looks like" a PetSet volume name (ending with
-<number>) then we use the number as an offset.  In that case we hash
the base name.

Fixes kubernetes#27256
@mikekap mikekap mentioned this issue Jul 22, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/stateful-apps priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@justinsb @lavalamp @davidopp @goltermann @bprashanth