Bump statefulset to apps/v1 #53415

crimsonfaith91 · 2017-10-03T23:19:20Z

What this PR does / why we need it:
This PR bumps statefulset to apps/v1.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): kubernetes/enhancements#484

Release note:

Bump StatefulSet to apps/v1.

TODO:

Investigate propagation of ScaleSpec into apps/v1

k8s-github-robot · 2017-10-03T23:19:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: crimsonfaith91
We suggest the following additional approver: wojtek-t

Assign the PR to them by writing /assign @wojtek-t in a comment when ready.

No associated issue. Update pull-request body to add a reference to an issue, or get approval with /approve no-issue

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

crimsonfaith91 · 2017-10-04T23:16:51Z

/cc @janetkuo @kow3ns
This is my first time working on bumping a controller to a new version. There is a learning curve, so it can be slow to get this done.

I am still investigating the 2 test failures: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/53415/pull-kubernetes-unit/56608/

For the first failure, it fails on this line: https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-cmd-util.sh#L1353
It fails because apps/v1beta1 statefulset cannot be found. An easy fix is to change the version from v1beta1 to v1. I added a test case for daemonset, but I was able to find extensions/v1beta1 daemonset. I have no idea why apps/v1beta1 statefulset disappears when I am adding apps/v1 codes.

For the second failure, I initially thought the failure was due to no ControllerRevisions declaration in apps/v1 types.go, but the failure still exists after adding it. From the error message, the test utilizes apps/v1beta1 statefulset (even after changing API version of YAML test files). I think this error is related to first failure (missing apps/v1beta1 statefulset).

I am not sure whether we need to add ControllerRevisions in this PR or not.

After spending a whole day debugging the failures, I think I should ask for help from someone with more experience. Not sure whether any of you have bandwidth to chat about it by person.

janetkuo · 2017-10-05T18:43:47Z

Run a cluster locally to see what resources are supported in each apps/ API versions.

ControllerRevision does not need to be moved with other resources that use it as they are independent. For example, you can have apps/v1beta1 Deployments using extensions/v1beta1 ReplicaSets.

crimsonfaith91 · 2017-10-05T23:48:24Z

Excluding apps/v1 from SUPPORTED_RESOURCES fixes the 2 failures (not recommended): https://github.com/kubernetes/kubernetes/blob/master/hack/make-rules/test-cmd.sh#L100

It looks like kubectl prioritizes apps/v1 over apps/v1beta1 (and apps/v1beta2) for statefulset tests, but not for daemonset tests. Not sure whether it is relevant to Autogen.

crimsonfaith91 · 2017-10-06T00:01:43Z

pkg/apis/apps/install/install.go

@@ -37,12 +38,13 @@ func Install(groupFactoryRegistry announced.APIGroupFactoryRegistry, registry *r
 	if err := announced.NewGroupMetaFactory(
 		&announced.GroupMetaFactoryArgs{
 			GroupName:                  apps.GroupName,
-			VersionPreferenceOrder:     []string{v1beta1.SchemeGroupVersion.Version, v1beta2.SchemeGroupVersion.Version},
+			VersionPreferenceOrder:     []string{v1beta1.SchemeGroupVersion.Version, v1beta2.SchemeGroupVersion.Version, v1.SchemeGroupVersion.Version},


@janetkuo To verify, does it mean v1beta1 or v1 is preferred? It looks to me that the order is correct.

v1beta1 is preferred

Thanks for verifying!

kow3ns · 2017-10-06T14:42:22Z

cmd/kube-apiserver/app/aggregator.go

@@ -205,6 +205,7 @@ var apiVersionPriorities = map[schema.GroupVersion]priority{
 	// to my knowledge, nothing below here collides
 	{Group: "apps", Version: "v1beta1"}:                          {group: 17800, version: 1},
 	{Group: "apps", Version: "v1beta2"}:                          {group: 17800, version: 1},


should these have equal priority?

It may be due to different API groups have different priorities. extensions/v1beta1 daemonset works fine and passes the test, but not apps/v1beta1 statefulset. Instead, apps/v1 statefulset is preferred by default, which fails the 2 kubectl tests.

To answer your question, I am not sure about it - didn't work on apiserver/aggregator codes before. I lean towards thinking them should have different priorities though.

I tried to ask how priority should be set but didn't get a response yet #49391 (comment)

crimsonfaith91 · 2017-10-06T19:03:40Z

@janetkuo I may have found the real reason behind why apps/v1 is preferred over apps/v1beta1 and apps/v1beta2. The discovery client reads info from a file named serverresources.json in the order of extensions/v1beta1, apps/v1, apps/v1beta1, and apps/v1beta2. This explains why extensions/v1beta1 daemonset works, but not apps/v1beta1 statefulset.

I1006 11:56:19.920211  107472 cached_discovery.go:72] returning cached discovery info from /tmp/tmp.RIjhHQtS8f/.kube/cache/discovery/127.0.0.1_8080/extensions/v1beta1/serverresources.json
I1006 11:56:19.920248  107472 cached_discovery.go:72] returning cached discovery info from /tmp/tmp.RIjhHQtS8f/.kube/cache/discovery/127.0.0.1_8080/apps/v1/serverresources.json
I1006 11:56:19.920288  107472 cached_discovery.go:72] returning cached discovery info from /tmp/tmp.RIjhHQtS8f/.kube/cache/discovery/127.0.0.1_8080/apps/v1beta1/serverresources.json
I1006 11:56:19.920342  107472 cached_discovery.go:72] returning cached discovery info from /tmp/tmp.RIjhHQtS8f/.kube/cache/discovery/127.0.0.1_8080/apps/v1beta2/serverresources.json

janetkuo · 2017-10-06T20:01:54Z

@liggitt we recently added apps/v1 in 1.9. We set the preferred version of apps as follows (expect app/v1 to come last):

kubernetes/pkg/apis/apps/install/install.go

Line 41 in 9813a52

    
           VersionPreferenceOrder:     []string{v1beta1.SchemeGroupVersion.Version, v1beta2.SchemeGroupVersion.Version, v1.SchemeGroupVersion.Version},

kubectl get sts --v=6 returns apps/v1beta1 StatefulSet correctly.

However, when this PR bumps StatefulSet to apps/v1 (StatefulSet is already in apps/v1beta1 and apps/v1beta2), kubectl tries to get apps/v1 StatefulSet first instead (kubectl get sts --v=6 returns apps/v1 StatefulSet instead of apps/v1beta1) and causes some kubectl tests to fail. Do you know what'd cause this? Would it be related to #53040 (comment)?

liggitt · 2017-10-06T21:18:04Z

Do you know what'd cause this? Would it be related to #53040 (comment)?

no. what order does it appear in discovery? did you register the "apps/v1" in cmd/kube-apiserver/app/aggregator.go, and if so, with what priority?

crimsonfaith91 · 2017-10-06T21:31:25Z

@liggitt

what order does it appear in discovery?

The order is extensions/v1beta1, apps/v1, apps/v1beta1, and finally apps/v1beta2. The failing kubectl tests depend on apps/v1beta1, but was given apps/v1 by default, resulting in the failures.

I1006 11:56:19.920211  107472 cached_discovery.go:72] returning cached discovery info from /tmp/tmp.RIjhHQtS8f/.kube/cache/discovery/127.0.0.1_8080/extensions/v1beta1/serverresources.json
I1006 11:56:19.920248  107472 cached_discovery.go:72] returning cached discovery info from /tmp/tmp.RIjhHQtS8f/.kube/cache/discovery/127.0.0.1_8080/apps/v1/serverresources.json
I1006 11:56:19.920288  107472 cached_discovery.go:72] returning cached discovery info from /tmp/tmp.RIjhHQtS8f/.kube/cache/discovery/127.0.0.1_8080/apps/v1beta1/serverresources.json
I1006 11:56:19.920342  107472 cached_discovery.go:72] returning cached discovery info from /tmp/tmp.RIjhHQtS8f/.kube/cache/discovery/127.0.0.1_8080/apps/v1beta2/serverresources.json

did you register the "apps/v1" in cmd/kube-apiserver/app/aggregator.go, and if so, with what priority?

Yes: https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/aggregator.go#L208
How to specify the priority explicitly? It seems to me that priority of apps/v1beta1 should be higher (based on the order) - apps/v1beta1 is specified before apps/v1.

kow3ns · 2017-10-06T21:34:25Z

@liggitt The DaemonSet and apps/v1 PR registers v1 group. The v1beta and v1beta2 groups are at the same priority but the v1 group is at a higher priority. Does the ordering matter, and do the priorities have to be unique? If so, I'm curios as to how this works for DeamonSet. Is it because DeamonSet is in exactly one of v1beta1 and v1beta2 while StatefulSet is in all 3?

liggitt · 2017-10-06T21:36:33Z

How to specify the priority explicitly? It seems to me that priority of apps/v1beta1 should be higher based on the order - apps/v1beta1 is specified before apps/v1.

The version priority is determined by the value of the version field in the priority struct:

{Group: "apps", Version: "v1beta1"}:                          {group: 17800, version: 1},
{Group: "apps", Version: "v1beta2"}:                          {group: 17800, version: 1},
{Group: "apps", Version: "v1"}:                               {group: 17800, version: 15},

That translates to the VersionPriority on the registered APIService:

	// VersionPriority controls the ordering of this API version inside of its group.  Must be greater than zero.
	// The primary sort is based on VersionPriority, ordered highest to lowest (20 before 10).
	// The secondary sort is based on the alphabetical comparison of the name of the object.  (v1.bar before v1.foo)
	// Since it's inside of a group, the number can be small, probably in the 10s.
	VersionPriority int32

that registration placed apps/v1 at a higher priority than apps/v1beta1 and apps/v1beta2, which is why it came first in discovery

kow3ns · 2017-10-06T21:40:29Z

@LiGgit Understood. @crimsonfaith91 you should modify the v1beta2 priority to be higher than the v1beta1 priority and lower than v1.

crimsonfaith91 · 2017-10-06T21:41:58Z

@kow3ns

If so, I'm curios as to how this works for DeamonSet. Is it because DeamonSet is in exactly one of v1beta1 and v1beta2 while StatefulSet is in all 3?

DaemonSet works because extensions/v1beta1 is put before apps/v1. StatefulSet does not work because apps/v1beta1 is put after apps/v1.

crimsonfaith91 · 2017-10-06T21:43:01Z

@liggitt

The version priority is determined by the value of the version field in the priority struct:

Great!! This is the answer that I have searched for the past 2 days!

liggitt · 2017-10-06T21:51:01Z

The version priority is determined by the value of the version field in the priority struct

Note that that is the external API priority.

The internal storage priority is separately controllable, which lets us advertise one order to API clients, while still storing a backlevel version in etcd for compatibility with HA API upgrades. That is what the etcd_storage_versions test checks, which version is persisted in etcd (which should always store a version readable by n-1 API servers for resources which existed in a prior release)

kow3ns · 2017-10-06T22:51:12Z

@liggitt we are not switching the storage versions. We still have to verify storage version upgrades.

crimsonfaith91 · 2017-10-07T00:54:50Z

@kow3ns Based on our discussion offline, I started looking on REST client codes to fix the kubectl statefulset history test failure. I was able to resolve the incompatibility between apps/v1 and apps/v1beta1 by setting group version for the REST config based on API version of the client, but encountered following error:

error: unable to find history controlled by StatefulSet nginx: failed to retrieve StatefulSet nginx: Get http://localhost/apis/apps/v1beta1/namespaces/namespace1507336185582694575/statefulsets/nginx: dial tcp [::1]:80: getsockopt: connection refused

This error makes a lot of sense. The kubectl test was "given" apps/v1 statefulset (as apps/v1 has higher priority), but had to fetch apps/v1beta1 statefulset, which was unavailable.

I plan to convert the statefulset from apps/v1 version to apps/v1beta1 version before fetching it. With the conversion, the fetch should be successful and the test should pass. WDYT?

liggitt · 2017-10-07T01:00:52Z

error: unable to find history controlled by StatefulSet nginx: failed to retrieve StatefulSet nginx: Get http://localhost/apis/apps/v1beta1/namespaces/namespace1507336185582694575/statefulsets/nginx: dial tcp [::1]:80: getsockopt: connection refused

This error makes a lot of sense. The kubectl test was "given" apps/v1 statefulset (as apps/v1 has higher priority), but had to fetch apps/v1beta1 statefulset, which was unavailable.

That does not make sense to me... if you create an apps/v1 statefulset, it should be accessible as an apps/v1beta1 stateful set as well

liggitt · 2017-10-07T01:01:49Z

also, that error appears to be a failure to reach the API server, not a 404 on a particular API group/version/resource

liggitt · 2017-10-07T01:03:56Z

staging/src/k8s.io/api/apps/v1/types.go

 	DeprecatedTemplateGeneration   = "deprecated.daemonset.template.generation"
 )

+// ScaleSpec describes the attributes of a scale subresource
+type ScaleSpec struct {


@DirectXMan12 @deads2k for propagation of scale into apps/v1... I thought we really didn't want to do this

#49504 indicates that this is still not clear. Currently we have an individual scale in each group version. If there is a plan to change this in 1.9 and migrate to a different strategy it is not concrete afaik

the current approach has known problems (HPA must hardcode custom clients for everything it is asked to scale) and scale was added to the apps group without considering how it would actually interact with autoscaling. we should complete the design that accounts for that before propagating those problems into a stable API group.

Agreed that this warrants more discussion. Added a TODO on this PR.

We did consider the auto-scaling interaction when adding Scale to the apps group, as well as the interaction with eviction controller and pod disruption budgets. In #42752 we decided to add a scale sub resource for all objects that made sense. If we are going to pivot on the implementation, that is fine, and importing the scale subresource from another location will be a relatively minor change. If the duck typing approach form #49504 is taken, I still think we should support the scale subresource in the apps group because we have it in extensions/v1beta1 and users of Deployment may have built expectations about it into their code.

We did consider the auto-scaling interaction when adding Scale to the apps group

An approach that required hard-coding HPA to yet another API group's Scale kind seems like a mistake.

In #42752 we decided to add a scale sub resource for all object that made sense.

A scale subresource makes sense. Representing it with a different group/version/kind in every API group that must still have the same schema does not. We can add scale subresources without redefining Scale kinds in every API group.

If the duck typing approach form #49504 is taken...

I strongly hope it is not. We don't use duck-typing for other parts of our API, I don't think we should start promoting it now.

I still think we should support the scale subresource in the apps group because we have it in extensions/v1beta1

The scale subresource is a good idea, but I have yet to see a compelling reason to define different Scale kinds in multiple API groups that are required to have identical schemas for duck typing. That seems like the worst of both worlds... you can't have a generic scale client that can use the same Scale kind against all /scale subresources, yet you don't gain any flexibility in having per-group Scale kinds because of duck typing constraints.

Before merging yet another Scale type (in a long-term supported version, no less), is there a reason not to settle on a common Scale kind?

An approach that required hard-coding HPA to yet another API group's Scale kind seems like a mistake.

Perhaps, but if we decide to replace it with a kind from a different group version rather than using one resource per group the PR to implement this, imo, can be handled with a follow up.

Before merging yet another Scale type (in a long-term supported version, no less), is there a reason not to settle on a common Scale kind?

We want to issue and merge the PRs to promote the workloads API. There will be follow up PRs to add Deployment, to add ReplicaSet, to add Condition types #51594, and to deal with any other outstanding issues. If we choose a different, and definitive, home for the scale sub-resource, I would prefer to deal with it in a follow up PR (which should be trivial) rather than blocking all other development pending resolution of the conversation in #49504.

If we choose a different, and definitive, home for the scale sub-resource I would prefer to deal with it in a follow up PR

We shouldn't merge apps/v1 Scale until this question is settled. It can be pulled out of this PR to unblock it.

kow3ns · 2017-10-07T01:09:28Z

@liggitt the kubectl code converts an unversioned client into a version client by passing the internal rest client directly to the constructor. This seems to cause the versioned client to call against the v1 endpoint and attempt conversion into an v1beta1 StatefulSet.

crimsonfaith91 · 2017-10-07T01:16:10Z

@liggitt

if you create an apps/v1 statefulset, it should be accessible as an apps/v1beta1 stateful set as well

Agreed, but the problem that i want to solve requires a different use case. We intended following priority order: apps/v1 has higher priority than apps/v1beta2; apps/v1beta2 has higher priority than apps/v1beta1. However, kubectl statefulset history test requires apps/v1beta1 statefulset, but it will always "given" apps/v1 statefulset. This is because we use versioned clients when developing kubectl rollout command for statefulset: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/rollback.go#L293

that error appears to be a failure to reach the API server, not a 404 on a particular API group/version/resource

I can verify the error is not about failure to reach the API server. I tested it with apps/v1beta1 statefulset by excluding apps/v1 API version during kubectl test setup, and the test passed.

liggitt · 2017-10-07T01:41:03Z

the kubectl code converts an unversioned client into a version client by passing the internal rest client directly to the constructor. This seems to cause the versioned client to call against the v1 endpoint and attempt conversion into an v1beta1 StatefulSet.

This is because we use versioned clients when developing kubectl rollout command for statefulset: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/rollback.go#L293

Ah. That is not the correct way to construct a versioned client, and is a bug that needs to be fixed.

func (f *ring1Factory) Rollbacker is taking whatever version object is passed to it, constructing a rest client for that particular version, then discarding the version info when calling RollbackerFor(). RollbackerFor() then hardcodes a particular version for that kind. You can't mix those two approaches to construct a client. Options:

pass the full mapping.GroupVersionKind to RollbackerFor and deal with possible versions
pass a complete external clientset to RollbackerFor (f.KubernetesClientSet()) and just get the versioned client you need from that (lets you get rid of versionedExtensionsClientV1beta1 and versionedAppsClientV1beta1)

kow3ns · 2017-10-07T01:41:18Z

@crimsonfaith91 the error you showed me earlier was a casting error inside of the client's conversion, and is likely due to the config problem we discussed. As @liggitt points out, the error you've posted above is a transport layer connection error. This is a different problem.

kow3ns · 2017-10-07T02:05:18Z

@liggitt that is precisely the conversation @crimsonfaith91 and I had offline :)

crimsonfaith91 · 2017-10-10T01:19:20Z

@liggitt

pass the full mapping.GroupVersionKind to RollbackerFor and deal with possible versions

pass a complete external clientset to RollbackerFor (f.KubernetesClientSet()) and just get the versioned client you need from that (lets you get rid of versionedExtensionsClientV1beta1 and versionedAppsClientV1beta1)

Thank you very much for detailed explanation about rollbacker! I will have a discussion tomorrow with the person implementing the codes that I referred when developing rollout command for statefulset. For now, I lean towards method 2.

crimsonfaith91 · 2017-10-10T01:20:39Z

@kow3ns Finally fix the failed tests in newest run!

crimsonfaith91 · 2017-10-11T18:19:24Z

closing in dup of: #53679

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 3, 2017

k8s-github-robot assigned m1093782566 and wojtek-t Oct 3, 2017

k8s-github-robot added the kind/new-api label Oct 3, 2017

k8s-ci-robot assigned crimsonfaith91 and unassigned m1093782566 and wojtek-t Oct 3, 2017

crimsonfaith91 force-pushed the sts-gvk branch from 8fcba9a to bb61b1f Compare October 4, 2017 01:16

k8s-ci-robot requested review from janetkuo and kow3ns October 4, 2017 23:16

crimsonfaith91 force-pushed the sts-gvk branch from fde34bf to bb61b1f Compare October 4, 2017 23:45

janetkuo self-assigned this Oct 5, 2017

crimsonfaith91 force-pushed the sts-gvk branch 2 times, most recently from 2534a5b to c2e9544 Compare October 5, 2017 23:37

crimsonfaith91 commented Oct 6, 2017

View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 6, 2017

kow3ns reviewed Oct 6, 2017

View reviewed changes

crimsonfaith91 force-pushed the sts-gvk branch from e7b912e to fb26e6a Compare October 6, 2017 18:00

k8s-github-robot removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. kind/new-api labels Oct 6, 2017

crimsonfaith91 added 2 commits October 6, 2017 12:49

bump statefulset to apps/v1

dbf5c4a

Autogen

35a684f

crimsonfaith91 force-pushed the sts-gvk branch from fb26e6a to 35a684f Compare October 6, 2017 19:50

liggitt reviewed Oct 7, 2017

View reviewed changes

fix API version order

cfe6685

crimsonfaith91 force-pushed the sts-gvk branch from 539118f to cfe6685 Compare October 10, 2017 17:50

crimsonfaith91 closed this Oct 11, 2017

Bump statefulset to apps/v1 #53415

Bump statefulset to apps/v1 #53415

Conversation

crimsonfaith91 commented Oct 3, 2017 • edited Loading

k8s-github-robot commented Oct 3, 2017

crimsonfaith91 commented Oct 4, 2017 • edited Loading

janetkuo commented Oct 5, 2017 • edited Loading

crimsonfaith91 commented Oct 5, 2017 • edited Loading

crimsonfaith91 Oct 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crimsonfaith91 Oct 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crimsonfaith91 Oct 6, 2017 • edited Loading

Choose a reason for hiding this comment

crimsonfaith91 Oct 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crimsonfaith91 commented Oct 6, 2017 • edited Loading

janetkuo commented Oct 6, 2017

liggitt commented Oct 6, 2017

crimsonfaith91 commented Oct 6, 2017 • edited Loading

kow3ns commented Oct 6, 2017 • edited Loading

liggitt commented Oct 6, 2017 • edited Loading

kow3ns commented Oct 6, 2017

crimsonfaith91 commented Oct 6, 2017

crimsonfaith91 commented Oct 6, 2017 • edited Loading

liggitt commented Oct 6, 2017

kow3ns commented Oct 6, 2017

crimsonfaith91 commented Oct 7, 2017 • edited Loading

liggitt commented Oct 7, 2017

liggitt commented Oct 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kow3ns Oct 7, 2017 • edited Loading

Choose a reason for hiding this comment

liggitt Oct 7, 2017 • edited Loading

Choose a reason for hiding this comment

kow3ns Oct 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kow3ns commented Oct 7, 2017 • edited Loading

crimsonfaith91 commented Oct 7, 2017 • edited Loading

liggitt commented Oct 7, 2017 • edited Loading

kow3ns commented Oct 7, 2017

kow3ns commented Oct 7, 2017

crimsonfaith91 commented Oct 10, 2017

crimsonfaith91 commented Oct 10, 2017 • edited Loading

crimsonfaith91 commented Oct 11, 2017

crimsonfaith91 commented Oct 3, 2017 •

edited

Loading

crimsonfaith91 commented Oct 4, 2017 •

edited

Loading

janetkuo commented Oct 5, 2017 •

edited

Loading

crimsonfaith91 commented Oct 5, 2017 •

edited

Loading

crimsonfaith91 Oct 6, 2017 •

edited

Loading

crimsonfaith91 Oct 6, 2017 •

edited

Loading

crimsonfaith91 Oct 6, 2017 •

edited

Loading

crimsonfaith91 Oct 6, 2017 •

edited

Loading

crimsonfaith91 commented Oct 6, 2017 •

edited

Loading

crimsonfaith91 commented Oct 6, 2017 •

edited

Loading

kow3ns commented Oct 6, 2017 •

edited

Loading

liggitt commented Oct 6, 2017 •

edited

Loading

crimsonfaith91 commented Oct 6, 2017 •

edited

Loading

crimsonfaith91 commented Oct 7, 2017 •

edited

Loading

kow3ns Oct 7, 2017 •

edited

Loading

liggitt Oct 7, 2017 •

edited

Loading

kow3ns Oct 7, 2017 •

edited

Loading

kow3ns commented Oct 7, 2017 •

edited

Loading

crimsonfaith91 commented Oct 7, 2017 •

edited

Loading

liggitt commented Oct 7, 2017 •

edited

Loading

crimsonfaith91 commented Oct 10, 2017 •

edited

Loading