Surface cloud-provider-specific information in Services of type LoadBalancer #52670

yujuhong · 2017-09-18T18:35:58Z

Service of type LoadBalancer is backed by a cloud-provider-specific load balancer. Due to the inherent differences in networking, each cloud provider supports a very different sets of features and/or configurations exposed via annotations. For example, the Service annotations supported on AWS:

kubernetes/pkg/cloudprovider/providers/aws/aws.go

Lines 83 to 145 in 8c025bc

    
           const ServiceAnnotationLoadBalancerProxyProtocol = "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol" 
        
           // ServiceAnnotationLoadBalancerAccessLogEmitInterval is the annotation used to 
        
           // specify access log emit interval. 
        
           const ServiceAnnotationLoadBalancerAccessLogEmitInterval = "service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval" 
        
           // ServiceAnnotationLoadBalancerAccessLogEnabled is the annotation used on the 
        
           // service to enable or disable access logs. 
        
           const ServiceAnnotationLoadBalancerAccessLogEnabled = "service.beta.kubernetes.io/aws-load-balancer-access-log-enabled" 
        
           // ServiceAnnotationLoadBalancerAccessLogS3BucketName is the annotation used to 
        
           // specify access log s3 bucket name. 
        
           const ServiceAnnotationLoadBalancerAccessLogS3BucketName = "service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name" 
        
           // ServiceAnnotationLoadBalancerAccessLogS3BucketPrefix is the annotation used 
        
           // to specify access log s3 bucket prefix. 
        
           const ServiceAnnotationLoadBalancerAccessLogS3BucketPrefix = "service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix" 
        
           // ServiceAnnotationLoadBalancerConnectionDrainingEnabled is the annnotation 
        
           // used on the service to enable or disable connection draining. 
        
           const ServiceAnnotationLoadBalancerConnectionDrainingEnabled = "service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled" 
        
           // ServiceAnnotationLoadBalancerConnectionDrainingTimeout is the annotation 
        
           // used on the service to specify a connection draining timeout. 
        
           const ServiceAnnotationLoadBalancerConnectionDrainingTimeout = "service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout" 
        
           // ServiceAnnotationLoadBalancerConnectionIdleTimeout is the annotation used 
        
           // on the service to specify the idle connection timeout. 
        
           const ServiceAnnotationLoadBalancerConnectionIdleTimeout = "service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout" 
        
           // ServiceAnnotationLoadBalancerCrossZoneLoadBalancingEnabled is the annotation 
        
           // used on the service to enable or disable cross-zone load balancing. 
        
           const ServiceAnnotationLoadBalancerCrossZoneLoadBalancingEnabled = "service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled" 
        
           // ServiceAnnotationLoadBalancerExtraSecurityGroups is the annotation used 
        
           // one the service to specify additional security groups to be added to ELB created 
        
           const ServiceAnnotationLoadBalancerExtraSecurityGroups = "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups" 
        
           // ServiceAnnotationLoadBalancerCertificate is the annotation used on the 
        
           // service to request a secure listener. Value is a valid certificate ARN. 
        
           // For more, see http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-listener-config.html 
        
           // CertARN is an IAM or CM certificate ARN, e.g. arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012 
        
           const ServiceAnnotationLoadBalancerCertificate = "service.beta.kubernetes.io/aws-load-balancer-ssl-cert" 
        
           // ServiceAnnotationLoadBalancerSSLPorts is the annotation used on the service 
        
           // to specify a comma-separated list of ports that will use SSL/HTTPS 
        
           // listeners. Defaults to '*' (all). 
        
           const ServiceAnnotationLoadBalancerSSLPorts = "service.beta.kubernetes.io/aws-load-balancer-ssl-ports" 
        
           // ServiceAnnotationLoadBalancerBEProtocol is the annotation used on the service 
        
           // to specify the protocol spoken by the backend (pod) behind a listener. 
        
           // If `http` (default) or `https`, an HTTPS listener that terminates the 
        
           //  connection and parses headers is created. 
        
           // If set to `ssl` or `tcp`, a "raw" SSL listener is used. 
        
           // If set to `http` and `aws-load-balancer-ssl-cert` is not used then 
        
           // a HTTP listener is used. 
        
           const ServiceAnnotationLoadBalancerBEProtocol = "service.beta.kubernetes.io/aws-load-balancer-backend-protocol" 
        
           // ServiceAnnotationLoadBalancerAdditionalTags is the annotation used on the service 
        
           // to specify a comma-separated list of key-value pairs which will be recorded as 
        
           // additional tags in the ELB. 
        
           // For example: "Key1=Val1,Key2=Val2,KeyNoVal1=,KeyNoVal2" 
        
           const ServiceAnnotationLoadBalancerAdditionalTags = "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags"

There are many known downsides of using annotations (e.g., lack of validation, defaulting, etc), but that's not the focus of this issue.

One problem with hiding all these vastly different implementations behind a generic Service object is that it's hard for users to introspect the status using the kubernetes API.
In fact, the LoadBalancerStatus contains only a single field, LoadBalancerIngress, which is the lowest common denominator of all possible load balancers.

// LoadBalancerStatus represents the status of a load-balancer.
type LoadBalancerStatus struct {
	// Ingress is a list containing ingress points for the load-balancer.
	// Traffic intended for the service should be sent to these ingress points.
	// +optional
	Ingress []LoadBalancerIngress `json:"ingress,omitempty" protobuf:"bytes,1,rep,name=ingress"`
}

Users would not be able to verify whether a LB is configured as expected, or whether a change to an annotation has taken effect or not (and all annotations are mutable). To actually verify the correctness of a load balancer, a user would need in-depth understanding of the both cloud networking and the implementation of cloud-provider code in the controller while interacting directly with the cloud platform UI. That doesn't lead to great user experience.

Some potential solutions that have been brought up in previous discussions:

Add an opaque key-value map as part of LoadBalancerStatus.
Include cloud-provide-specific information in Events. (caveat: events are ephemeral and can be spammy when used this way).

Thoughts?

@kubernetes/sig-network-feature-requests @thockin @bowei

The text was updated successfully, but these errors were encountered:

thockin · 2017-09-18T22:01:32Z

I like the idea of status for this. We've tried to stay away from string-string maps in general, but it seems appropriate here. Something like:

status:
  loadBalancer:
    ingress:
    - 1.2.3.4
    properties:
        cloud.google.com/fowarding-rule: a1234567890abcdef
        cloud.google.com/somethign: SomeValue

"properties" is a terrible name, but you get the idea. Better than storing annotations perhaps, though it is sort of a slippery distinction.

@smarterclayton @bgrant0607

bowei · 2017-09-18T23:28:01Z

Should there be something like a HasError: bool field at the top level? Seems useful to make that piece generic

yujuhong · 2017-09-19T00:47:18Z

Should there be something like a HasError: bool field at the top level? Seems useful to make that piece generic

I agree that some generic fields will be useful (in addition to the opaque field), and I think it can be more than a boolean field. There is a long discussion thread on state/phase/condition/outlook in #7856. I am not sure whether any conclusion has been drawn yet.

smarterclayton · 2017-09-19T01:29:30Z

Another option (for consistency) - have the controller write to a new typed object that is mostly status (AWSLoadBalancerService) and have the service point to it via Ref. Do it as a CRD to start. On Sep 18, 2017, at 8:47 PM, Yu-Ju Hong <notifications@github.com> wrote: Should there be something like a HasError: bool field at the top level? Seems useful to make that piece generic I agree that some generic fields will be useful (in addition to the opaque field), and I think it can be more than a boolean field. There is a long discussion thread on state/phase/condition/outlook in #7856 <#7856>. I am not sure whether any conclusion has been drawn yet. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#52670 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p9KQepg-oMRoILW6fjgaxc1jPaQJks5sjw8pgaJpZM4PbVzz> .

thockin · 2017-09-19T07:04:11Z

That level of reconciliation feels heavy for something we don't really know we'll use beyond map[string]string... On Mon, Sep 18, 2017 at 6:29 PM, Clayton Coleman <notifications@github.com> wrote:

…

Another option (for consistency) - have the controller write to a new typed object that is mostly status (AWSLoadBalancerService) and have the service point to it via Ref. Do it as a CRD to start. On Sep 18, 2017, at 8:47 PM, Yu-Ju Hong ***@***.***> wrote: Should there be something like a HasError: bool field at the top level? Seems useful to make that piece generic I agree that some generic fields will be useful (in addition to the opaque field), and I think it can be more than a boolean field. There is a long discussion thread on state/phase/condition/outlook in #7856 <#7856>. I am not sure whether any conclusion has been drawn yet. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/kubernetes/kubernetes/issues/ 52670#issuecomment-330395718>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p9KQepg- oMRoILW6fjgaxc1jPaQJks5sjw8pgaJpZM4PbVzz> . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#52670 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVHNz7TKZa-Tvh0CHxsMp3KPGwAqwks5sjxkSgaJpZM4PbVzz> .

bgrant0607 · 2017-09-23T00:17:39Z

As for general properties, such as cases where the LB is not functioning, I agree those should just be fields, as proposed in #7856. Further motivation is discussed in #34363 and #1899.

As for cloud-specific properties or other types of extensions (e.g., with kube-proxy replaced by another solution), I think we have a number of cases where we need that across the system. Some are discussed in #31571 and in the architectural roadmap.

I can see why @smarterclayton suggested a CRD in this case.

Rather than annotations, I would like to be able to express a schema for extension parameters, sort of like StorageClass, but also with parameter descriptions and types. That starts to look like what was proposed for CRD validation. Not only could we then validate the parameters, but users could also see what options were available.

Similarly for status, we'd want to at least document what information the user could expect to be provided. And the cloud provider controller would need a mechanism to report it. A CRD with a status subresource would provide such a mechanism.

That seems much cleaner than annotations and a map of strings.

Whether we choose a status map or a CRD, I think we're going to need significant changes to Service next year, before moving it to a new API group, so we will have an opportunity to change this in the future.

OOC, is the Service type NodePort for AWS rather than type LoadBalancer? Do we need to support the additional status properties for multiple types of services?

cc @kubernetes/sig-architecture-proposals

thockin · 2017-10-09T05:49:52Z

As for general properties, such as cases where the LB is not functioning, I agree those should just be fields, as proposed in #7856. Further motivation is discussed in #34363 and #1899. As for cloud-specific properties or other types of extensions (e.g., with kube-proxy replaced by another solution), I think we have a number of cases where we need that across the system. Some are discussed in #31571 and in the architectural roadmap. I can see why @smarterclayton suggested a CRD in this case.

I can see it too, but it feels like a heavy answer. It is more elegant, especially for pure status. How would life-cycle work? Google's CloudProvider registers CRD for `GoogleCloudLoadBalancerStatus`. User creates Service `type: LoadBalancer` Google's service LB controller actuates against GCP Google's CloudProvider instantiates `GoogleCloudLoadBalancerStatus` and links it from Service.status.LoadBalancer User deletes Service Now what? We don't want to leave that status laying around. A finalizer, I guess?

Rather than annotations, I would like to be able to express a schema for extension parameters, sort of like StorageClass, but also with parameter descriptions and types. That starts to look like what was proposed for CRD validation. Not only could we then validate the parameters, but users could also see what options were available.

Yeah, if this is to be schema'd, I don't want to do that more than one place.

Similarly for status, we'd want to at least document what information the user could expect to be provided. And the cloud provider controller would need a mechanism to report it. A CRD with a status subresource would provide such a mechanism.

"With a status subresource" or "as a status expander"? I don't know that this would have any non-status fields, but I guess consistency?

That seems much cleaner than annotations and a map of strings. Whether we choose a status map or a CRD, I think we're going to need significant changes to Service next year, before moving it to a new API group, so we will have an opportunity to change this in the future.

Agree. I want to be expedient, and I think a string-map is honestly good enough here, but I see the allure of a CRD for consistency across many situations.

OOC, is the Service type NodePort for AWS rather than type LoadBalancer? Do we need to support the additional status properties for multiple types of services?

AWS uses LoadBalancer and NodePort types the same as GCP. AWS LoadBalancer is more like GCP's ingress - a full proxy.

smarterclayton · 2017-10-09T15:50:44Z

On Mon, Oct 9, 2017 at 1:50 AM, Tim Hockin ***@***.***> wrote: > As for general properties, such as cases where the LB is not functioning, I agree those should just be fields, as proposed in #7856. Further motivation is discussed in #34363 and #1899. > > As for cloud-specific properties or other types of extensions (e.g., with kube-proxy replaced by another solution), I think we have a number of cases where we need that across the system. Some are discussed in #31571 and in the architectural roadmap. > > I can see why @smarterclayton suggested a CRD in this case. I can see it too, but it feels like a heavy answer. It is more elegant, especially for pure status. How would life-cycle work? Google's CloudProvider registers CRD for `GoogleCloudLoadBalancerStatus`. User creates Service `type: LoadBalancer` Google's service LB controller actuates against GCP Google's CloudProvider instantiates `GoogleCloudLoadBalancerStatus` and links it from Service.status.LoadBalancer User deletes Service Now what? We don't want to leave that status laying around. A finalizer, I guess?

Garbage collection

> Rather than annotations, I would like to be able to express a schema for extension parameters, sort of like StorageClass, but also with parameter descriptions and types. That starts to look like what was proposed for CRD validation. Not only could we then validate the parameters, but users could also see what options were available. Yeah, if this is to be schema'd, I don't want to do that more than one place.

A CRD is supposed to be as easy as a map[string]string but typed.

> Similarly for status, we'd want to at least document what information the user could expect to be provided. And the cloud provider controller would need a mechanism to report it. A CRD with a status subresource would provide such a mechanism. "With a status subresource" or "as a status expander"? I don't know that this would have any non-status fields, but I guess consistency?

And until we have a status subresource, you can still fetch the CRD (you just have to remember that it's not "trusted" nor is any RBAC enforced).

> That seems much cleaner than annotations and a map of strings. > > Whether we choose a status map or a CRD, I think we're going to need significant changes to Service next year, before moving it to a new API group, so we will have an opportunity to change this in the future. Agree. I want to be expedient, and I think a string-map is honestly good enough here, but I see the allure of a CRD for consistency across many situations.

As above, if CRD isn't as easy as map[string]string and better in every way (get, explain, validation support) then CRD has failed.

…

> OOC, is the Service type NodePort for AWS rather than type LoadBalancer? Do we need to support the additional status properties for multiple types of services? AWS uses LoadBalancer and NodePort types the same as GCP. AWS LoadBalancer is more like GCP's ingress - a full proxy. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#52670 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p8vs1yJmjLi7dx5YPjDUUiDhjzwKks5sqbQWgaJpZM4PbVzz> .

thockin · 2017-10-09T16:16:35Z

On Mon, Oct 9, 2017 at 8:51 AM, Clayton Coleman <notifications@github.com> wrote:

On Mon, Oct 9, 2017 at 1:50 AM, Tim Hockin ***@***.***> wrote: > > As for general properties, such as cases where the LB is not > functioning, I agree those should just be fields, as proposed in #7856. > Further motivation is discussed in #34363 and #1899. > > > > As for cloud-specific properties or other types of extensions (e.g., > with kube-proxy replaced by another solution), I think we have a number of > cases where we need that across the system. Some are discussed in #31571 > and in the architectural roadmap. > > > > I can see why @smarterclayton suggested a CRD in this case. > > I can see it too, but it feels like a heavy answer. It is more > elegant, especially for pure status. How would life-cycle work? > > Google's CloudProvider registers CRD for `GoogleCloudLoadBalancerStatus`. > User creates Service `type: LoadBalancer` > Google's service LB controller actuates against GCP > Google's CloudProvider instantiates `GoogleCloudLoadBalancerStatus` > and links it from Service.status.LoadBalancer > User deletes Service > > Now what? We don't want to leave that status laying around. A > finalizer, I guess? Garbage collection > > > > Rather than annotations, I would like to be able to express a schema for > extension parameters, sort of like StorageClass, but also with parameter > descriptions and types. That starts to look like what was proposed for CRD > validation. Not only could we then validate the parameters, but users could > also see what options were available. > > Yeah, if this is to be schema'd, I don't want to do that more than one > place. A CRD is supposed to be as easy as a map[string]string but typed. > > > > Similarly for status, we'd want to at least document what information > the user could expect to be provided. And the cloud provider controller > would need a mechanism to report it. A CRD with a status subresource would > provide such a mechanism. > > "With a status subresource" or "as a status expander"? I don't know > that this would have any non-status fields, but I guess consistency? And until we have a status subresource, you can still fetch the CRD (you just have to remember that it's not "trusted" nor is any RBAC enforced). > > > > That seems much cleaner than annotations and a map of strings. > > > > Whether we choose a status map or a CRD, I think we're going to need > significant changes to Service next year, before moving it to a new API > group, so we will have an opportunity to change this in the future. > > Agree. I want to be expedient, and I think a string-map is honestly > good enough here, but I see the allure of a CRD for consistency across > many situations. As above, if CRD isn't as easy as map[string]string and better in every way (get, explain, validation support) then CRD has failed.

Some on. Adding a string map is 45 seconds of work. Adding a CRD, and code to install it, is somewhat more. Maybe not prohibitively so, but it's not the same, unless I am doing CRD totally wrong.

…

> > OOC, is the Service type NodePort for AWS rather than type LoadBalancer? > Do we need to support the additional status properties for multiple types > of services? > > AWS uses LoadBalancer and NodePort types the same as GCP. AWS > LoadBalancer is more like GCP's ingress - a full proxy. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#52670 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ABG_p8vs1yJmjLi7dx5YPjDUUiDhjzwKks5sqbQWgaJpZM4PbVzz> > . > — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

bgrant0607 · 2017-10-25T23:24:41Z

Ref #18451, #9106, #8992

bgrant0607 · 2017-10-25T23:28:55Z

Probably nothing prevents the service controller from writing annotations via the service /status endpoint. Is there a reason to not do that for now?

thockin · 2017-10-25T23:36:48Z

No, annotations work, for now. Just looking for patterns.

…

On Wed, Oct 25, 2017 at 4:29 PM, Brian Grant ***@***.***> wrote: Probably nothing prevents the service controller from writing annotations via the service /status endpoint. Is there a reason to not do that for now? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#52670 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVAdaU_cU2kQnvCQCDAiB9RfK7IEdks5sv8RMgaJpZM4PbVzz> .

bgrant0607 · 2017-10-26T21:38:15Z

In terms of patterns, I would like to be able to express application "topology" in a fully portable manner, but then also automatically discover provider-specific spec and status extensions. I think that implies we will want to embed the extensions in the portable resources, but that we do need a way to specify the schema for discovery purposes. I had originally wanted to specify such extension schemas in FooClass-style resources, but with something more like JSON schema than just a map of strings. We could potentially even automatically inject the schemas into the appropriate places in our OpenAPI spec at runtime, once we have the ability to express unions.

That may seem heavyweight, but we need more formal specs in order to build the kinds of docs and tools we need. The mass of annotations and maps of arbitrary strings we have is undiscoverable and incomprehensible to almost everyone but core project contributors.

Ref #31571.

thockin · 2017-10-27T05:41:14Z

Why embed rather than link-to-CRD or similar, as we discussed previously?

…

On Thu, Oct 26, 2017 at 2:38 PM, Brian Grant ***@***.***> wrote: In terms of patterns, I would like to be able to express application "topology" in a fully portable manner, but then also automatically discover provider-specific spec and status extensions. I think that implies we will want to embed the extensions in the portable resources, but that we do need a way to specify the schema for discovery purposes. I had originally wanted to specify such extension schemas in FooClass-style resources, but with something more like JSON schema than just a map of strings. We could potentially even automatically inject the schemas into the appropriate places in our OpenAPI spec at runtime, once we have the ability to express unions. That may seem heavyweight, but we need more formal specs in order to build the kinds of docs and tools we need. The mass of annotations and maps of arbitrary strings we have is undiscoverable and incomprehensible to almost everyone but core project contributors. Ref #31571 <#31571>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#52670 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVAJa_2qqwW_ib7rWnxSh4cepExLeks5swPvmgaJpZM4PbVzz> .

thockin · 2019-05-02T22:14:32Z

Moving discussion to #70159

sftim · 2022-12-21T15:55:58Z

For tidiness
/close not-planned

yujuhong added kind/feature Categorizes issue or PR as related to a new feature. sig/network Categorizes an issue or PR as relevant to SIG Network. labels Sep 18, 2017

thockin self-assigned this Sep 19, 2017

k8s-ci-robot added sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. kind/design Categorizes issue or PR as related to design. labels Sep 23, 2017

This was referenced Dec 27, 2017

Expose load balancer status in service.status.loadBalancer #8992

Closed

Service status should include info about cloud load-balancer #18451

Closed

thockin added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label Dec 27, 2017

bgrant0607 mentioned this issue Jan 23, 2018

Allow service controller to indicate that it has cleared load balancer resources #32157

Closed

cmluciano mentioned this issue Jan 23, 2018

Creating External Loadbalancer should present some "progress status" #9106

Closed

thockin added the triage/unresolved Indicates an issue that can not or will not be resolved. label Mar 8, 2019

thockin removed the triage/unresolved Indicates an issue that can not or will not be resolved. label May 2, 2019

thockin closed this as completed May 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Surface cloud-provider-specific information in Services of type LoadBalancer #52670

Surface cloud-provider-specific information in Services of type LoadBalancer #52670

yujuhong commented Sep 18, 2017 •

edited

Loading

thockin commented Sep 18, 2017

bowei commented Sep 18, 2017

yujuhong commented Sep 19, 2017

smarterclayton commented Sep 19, 2017 via email

thockin commented Sep 19, 2017 via email

bgrant0607 commented Sep 23, 2017

thockin commented Oct 9, 2017 via email

smarterclayton commented Oct 9, 2017 via email

thockin commented Oct 9, 2017 via email

bgrant0607 commented Oct 25, 2017 •

edited

Loading

bgrant0607 commented Oct 25, 2017

thockin commented Oct 25, 2017 via email

bgrant0607 commented Oct 26, 2017

thockin commented Oct 27, 2017 via email

thockin commented May 2, 2019

sftim commented Dec 21, 2022

Surface cloud-provider-specific information in Services of type LoadBalancer #52670

Surface cloud-provider-specific information in Services of type LoadBalancer #52670

Comments

yujuhong commented Sep 18, 2017 • edited Loading

thockin commented Sep 18, 2017

bowei commented Sep 18, 2017

yujuhong commented Sep 19, 2017

smarterclayton commented Sep 19, 2017 via email

thockin commented Sep 19, 2017 via email

bgrant0607 commented Sep 23, 2017

thockin commented Oct 9, 2017 via email

smarterclayton commented Oct 9, 2017 via email

thockin commented Oct 9, 2017 via email

bgrant0607 commented Oct 25, 2017 • edited Loading

bgrant0607 commented Oct 25, 2017

thockin commented Oct 25, 2017 via email

bgrant0607 commented Oct 26, 2017

thockin commented Oct 27, 2017 via email

thockin commented May 2, 2019

sftim commented Dec 21, 2022

yujuhong commented Sep 18, 2017 •

edited

Loading

bgrant0607 commented Oct 25, 2017 •

edited

Loading