GCE: Add Alpha feature "Network Tiers" for external L4 load balancers #51301

yujuhong · 2017-08-25T00:48:07Z

Special notes for your reviewer:
The PR has been manually tested in a GCE e2e cluster for the following conditions:

When network-tier is not enabled in gce.conf, network tier annotations are completely ignored by the controller.
When network-tier is enabled in gce.conf:
- Service w/ Standard tier: create a standard-tier LB.
- Update Service to use a different tier: tear down the existing forwarding rule and release the IP before creating a new LB.
- Service w/ an invalid tier value: ensureExternalLoadBalancer() returns an error, and controller emits an event.
- Service w/ a user-owned static IP: check if the tier matches, if not, returns an error and emits an event.

I uploaded an e2e test #51483. You're welcome to review that one too.

Release note:

GCE: Service object now supports "Network Tiers" as an Alpha feature via annotations.

yujuhong · 2017-08-25T00:56:17Z

@nicksardo @bowei I split the changes into multiple commits so that it'd be easier to review. If you prefer multiple PRs, let me know.

nicksardo · 2017-08-29T00:29:07Z

pkg/cloudprovider/providers/gce/gce_addresses_fakes.go

+	if _, exists := cas.addrsByRegionAndName[region][name]; !exists {
+		return makeGoogleAPINotFoundError("")
+	}
+	delete(cas.addrsByRegionAndName[region], name)


delete(cas.reservedAddrs, addr.Address)

Fixed in the new commit.

nicksardo · 2017-08-29T01:00:08Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+	if isNotFound(err) {
+		return nil
+	}
+	if err != nil {


nit: else if

nicksardo · 2017-08-29T01:29:38Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+
+// reconcileFwdRuleNetworkTiers check the network tiers of existing forwarding
+// rule and delete the rule if the tier does not matched the desired tier.
+func reconcileFwdRuleNetworkTiers(s CloudForwardingRuleService, region, name, lbRef string, desiredNetTier NetworkTier) error {


Using the word reconcile seems strange to me here because you're only deleting the forwarding rule/address if the tier needs changed.

I suggest wripping the two reconcile funcs apart and introduce a higher level func which checks tier equality and deletes the resources if necessary.

func deleteWrongNetworkTieredResources(...){ fwdRuleOk, err := fwdRuleNetworkTierEquals(gce, region, name, lbRef, desiredNetTier) if err!= nil { ... } if !fwdRuleOk { err = s.DeleteRegionForwardingRule(name, region) } addrOk, err := addressNetworkTierEquals(gce, region, name, lbRef, desiredNetTier) if err!= nil { ... } if !addrOk { err = s.DeleteRegionAddress(name, region) } }

Agree that the names are confusing. The function was initially intended to do more, but that didn't pan out.
Renamed the functions. I still keep the address and forwarding rule verification in two functions to make testing easier.

yujuhong

Addressed comments. Thanks!

yujuhong · 2017-08-29T17:15:31Z

pkg/cloudprovider/providers/gce/gce_addresses_fakes.go

+	if _, exists := cas.addrsByRegionAndName[region][name]; !exists {
+		return makeGoogleAPINotFoundError("")
+	}
+	delete(cas.addrsByRegionAndName[region], name)


Fixed in the new commit.

yujuhong · 2017-08-29T18:33:40Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+	if isNotFound(err) {
+		return nil
+	}
+	if err != nil {


yujuhong · 2017-08-29T19:32:56Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+
+// reconcileFwdRuleNetworkTiers check the network tiers of existing forwarding
+// rule and delete the rule if the tier does not matched the desired tier.
+func reconcileFwdRuleNetworkTiers(s CloudForwardingRuleService, region, name, lbRef string, desiredNetTier NetworkTier) error {


Agree that the names are confusing. The function was initially intended to do more, but that didn't pan out.
Renamed the functions. I still keep the address and forwarding rule verification in two functions to make testing easier.

nicksardo · 2017-08-29T21:06:54Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+			addressObj.Address = existingIP
+		}
+		if err = s.ReserveRegionAddress(addressObj, region); err != nil {
+			if !isHTTPErrorCode(err, http.StatusConflict) {


Or BadRequest

nicksardo · 2017-08-29T21:07:00Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+			addressObj.Address = existingIP
+		}
+		if err = s.ReserveAlphaRegionAddress(addressObj, region); err != nil {
+			if !isHTTPErrorCode(err, http.StatusConflict) {


Or BadRequest

nicksardo · 2017-08-29T21:20:58Z

I don't see anything wrong with it, though we should get another pair of eyes. @bowei @MrHohn ?

yujuhong · 2017-08-29T23:37:46Z

Flakes caused by #51379

/retest

yujuhong · 2017-08-30T00:53:59Z

/test pull-kubernetes-e2e-gce-bazel

MrHohn

Looks good and logic makes sense to me.

Couple minor comments.

MrHohn · 2017-08-29T22:48:10Z

pkg/cloudprovider/providers/gce/gce_annotations.go

+	// NetworkTierAnnotationKey is annotated on a Service object to indicate which
+	// network tier a GCP LB should use. The valid values are "Standard" and
+	// "Premium" (default).
+	NetworkTierAnnotationKey      = "cloud.google.com/network-tier"


Do we need to specify alpha in the key? I forgot the reason... @nicksardo

I can't find it but I think it's recommended not to use alpha in the annotation anymore.

Good question. I suggest double checking with thockin.

/cc @thockin

Spoke to @thockin offline. This is ok.

MrHohn · 2017-08-29T23:21:58Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

-		glog.Infof("EnsureLoadBalancer(%v(%v)): creating forwarding rule, IP %s", loadBalancerName, serviceName, ipAddressToUse)
-		if err := gce.createForwardingRule(loadBalancerName, serviceName.String(), gce.region, ipAddressToUse, ports); err != nil {
+		glog.Infof("EnsureLoadBalancer(%v(%v)): creating forwarding rule, IP %s (tier: %s)", loadBalancerName, serviceName, ipAddressToUse, netTier)
+		if err := createForwardingRule(gce, loadBalancerName, serviceName.String(), gce.region, ipAddressToUse, gce.targetPoolURL(loadBalancerName), ports, netTier); err != nil {


Could we still call targetPoolURL() in createForwardingRule()? Seems a bit unclean in this way...

Humm, I see why it is hard, targetPoolURL desires some info from the gce struct but there is no obvious way for CloudForwardingRuleService to expose those. And we need CloudForwardingRuleService for mocking and testing.

Never mind :/

Yep. I could fix the targetPool code too with more mocking, but I'd prefer leaving them alone for this change.

MrHohn · 2017-08-29T23:35:23Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_naming.go

@@ -84,6 +84,11 @@ func makeBackendServiceDescription(nm types.NamespacedName, shared bool) string

 // External Load Balancer

+// makeServiceDescription is used to generate descriptions for forwarding rules and addresses.
+func makeServiceDescription(serviceName string) string {


Maybe good to name this function in a more consistent way among this file...We already have makeBackendServiceDescription above.

But of course I couldn't think of a better name ATM :(

Can we punt this to when we have more than two functions? :-)

Can we punt this to when we have more than two functions? :-)

SGTM

MrHohn · 2017-08-29T23:55:50Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+			addressObj.Address = existingIP
+		}
+		if err = s.ReserveAlphaRegionAddress(addressObj, region); err != nil {
+			if !isHTTPErrorCode(err, http.StatusConflict) {


May be cleaner to check error after the switch? Seems like they are the exact same codes.

I like checking the err code close to the call site, but sure I'll change it.

MrHohn · 2017-08-30T00:01:17Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go

@@ -85,3 +122,54 @@ func TestVerifyRequestedIP(t *testing.T) {
 		})
 	}
 }
+
+func TestCreateForwardingRule(t *testing.T) {


TestCreateForwardingRuleWithTier?

MrHohn · 2017-08-30T00:16:32Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+		}
+		netTier := NetworkTierGCEValueToType(netTierStr)
+		if netTier != desiredNetTier {
+			glog.Errorf("verifyUserRequestedIP: requested static IP %q (name: %s) for LB %s has network tier %s, need %s.", requestedIP, existingAddress.Name, lbRef, netTier, desiredNetTier)


Tiny nit: s/need/needs

Or does it makes sense to log and return the same error content?

Error will be included in the k8s event and probably should not have too many function-specific information.

Error will be included in the k8s event and probably should not have too many function-specific information.

Good point :)

MrHohn · 2017-08-30T01:04:37Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+	// Check the current and the desired network tiers. If they do not match,
+	// tear down the existing resources -- delete the forwarding rule, and
+	// release the IP.
+	netTier, err := gce.getServiceNetworkTier(apiService)


Could we also gate network tier outside of getServiceNetworkTier()? Like reconcileXXXNetworkTiers() below, so that we don't trigger the logic at all.

I did that intentionally since getServiceNetworkTier() should always return the default tier if the feature is disabled. If I could, I'd actually prefer gating it only at the gce API calls so that we can reuse the same flow when the feature graduates to Beta. Otherwise, the beta code will be completely new code which needs to be tested all over again. I think the current implementation provides sufficient "gating" but feel free to disagree :-)

Humm...I think I some how mixed the concepts of gating alpha/beta GCE API and gating the actual feature. IMHO:

Gate alpha/beta API means no calls to them.

Gate the feature means no triggers to the logic.

Given that the GCE provider gate is about gating API but not feature, I think it makes sense to do it this way...

I think the current implementation provides sufficient "gating" but feel free to disagree :-)

But yeah technically this is sufficient :)

MrHohn · 2017-08-30T01:09:40Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+		glog.Errorf("EnsureLoadBalancer(%s): failed to get the desired network tier: %v", lbRefStr, err)
+		return nil, err
+	}
+	glog.V(2).Infof("EnsureLoadBalancer(%s): desired network tier %q ", lbRefStr, netTier)


User that doesn't enable network tier may be seeing this "network tier" log and they may think it is mistakenly enabled?

Bumped it to V(4) so that it doesn't show up in GKE clusters.

Sorry for switching side...Actually I'd prefer to keep this in V(2) for better debuggability in case things go wrong in production. (I'd like to feature gate it, or maybe it just doesn't matter)

MrHohn · 2017-08-30T01:24:21Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

@@ -991,15 +987,28 @@ func (gce *GCECloud) getServiceNetworkTier(svc *v1.Service) (NetworkTier, error)
 	return tier, nil
 }

-// reconcileFwdRuleNetworkTiers check the network tiers of existing forwarding
+func (gce *GCECloud) deleteWrongNetworkTieredResources(lbName, lbRef string, desiredNetTier NetworkTier) error {
+	if !gce.AlphaFeatureGate.Enabled(AlphaFeatureNetworkTiers) {


Humm..Seems like the feature gate check is moved inside functions. I think having it outiside may make it clear about what are involved.

Restored it.

yujuhong · 2017-08-30T02:20:04Z

@MrHohn the new commit addressed (most) of your comments.

nicksardo · 2017-08-30T05:11:23Z

Since you're making the BadRequest/StatusConflict fix in this PR, could you also change the fake service to return the both types.

func (cas *FakeCloudAddressService) ReserveAlphaRegionAddress(addr *computealpha.Address, region string) error {
	if addr.Address == "" {
		addr.Address = fmt.Sprintf("1.2.3.%d", cas.count)
		cas.count++
	}

	if addr.AddressType == "" {
		addr.AddressType = string(schemeExternal)
	}

	if cas.reservedAddrs[addr.Address] {
		msg := "IP in use"
		// Depending on the type of address, different error messages may be returned.
		switch lbScheme(addr.AddressType) {
		case schemeExternal:
			return makeGoogleAPIError(http.StatusBadRequest, msg)
		default:
			return makeGoogleAPIError(http.StatusConflict, msg)
		}
	}

	if _, exists := cas.addrsByRegionAndName[region]; !exists {
		cas.addrsByRegionAndName[region] = make(map[string]*computealpha.Address)
	}

	if _, exists := cas.addrsByRegionAndName[region][addr.Name]; exists {
		return makeGoogleAPIError(http.StatusConflict, "name in use")
	}

	cas.addrsByRegionAndName[region][addr.Name] = addr
	cas.reservedAddrs[addr.Address] = true
	return nil
}

bowei · 2017-08-30T04:52:23Z

pkg/cloudprovider/providers/gce/gce_addresses.go

@@ -132,3 +132,15 @@ func (gce *GCECloud) GetRegionAddressByIP(region, ipAddress string) (*compute.Ad
 	}
 	return nil, makeGoogleAPINotFoundError(fmt.Sprintf("Address with IP %q was not found in region %q", ipAddress, region))
 }
+
+// TODO(yujuhong): retire this function once Network Tiers becomes Beta in GCP.


can we open an issue and reference that (rather than by username). I know it's annoying, but it's better to not assign to a person.

bowei · 2017-08-30T04:56:24Z

pkg/cloudprovider/providers/gce/gce_annotations.go

+	NetworkTierAnnotationStandard = "Standard"
+	NetworkTierAnnotationPremium  = "Premium"
+
+	NetworkTierStandard NetworkTier = NetworkTierAnnotationStandard


is there a reason why can't do directly?

NetworkTierStandard NetworkTier = "Standard"

I prefer keeping the annotation key/values pure strings (not an internal NetworkTier type) since it's essentially our API.

bowei · 2017-08-30T04:59:34Z

pkg/cloudprovider/providers/gce/gce_annotations.go

+// GetServiceNetworkTier returns the network tier of GCP load balancer
+// which should be assembled, and an error if the specified tier is not
+// supported.
+func GetServiceNetworkTier(service *v1.Service) (NetworkTier, error) {


Golang usually don't add Get prefix.

Any suggestion? I am bad with naming and the Get prefix makes the intent of the function clear to me.

bowei · 2017-08-30T05:14:34Z

pkg/cloudprovider/providers/gce/gce_forwardingrule.go

@@ -133,3 +133,15 @@ func (gce *GCECloud) DeleteRegionForwardingRule(name, region string) error {

 	return gce.waitForRegionOp(op, region, mc)
 }
+
+// TODO(yujuhong): retire this function once Network Tiers becomes Beta in GCP.


link to the same issue as above

bowei · 2017-08-30T05:48:14Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

@@ -426,7 +439,7 @@ func (gce *GCECloud) DeleteExternalTargetPoolAndChecks(name, region, clusterID s
 // verifyUserRequestedIP checks the user-provided IP  to see whether it can be
 // used for the LB.  It also returns whether the IP is considered owned by the


Can you clarify this comment as the function can now return (true, err != nil), seems confusing.

Good point. I changed the function to never return (true, non-nil-err). If the err is non-nil, the boolean value should not be used anyway.

bowei · 2017-08-30T05:51:39Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+	if existingTier == desiredNetTier {
+		return nil
+	}
+	glog.V(2).Infof("%s: Network tiers do not match; existing forwarding rule: %q, desired: %q", logPrefix, existingTier, desiredNetTier)


probably just combine into one log

bowei · 2017-08-30T05:52:10Z

pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go

+	if existingTier == desiredNetTier {
+		return nil
+	}
+	glog.V(2).Infof("%s: Network tiers do not match; existing address: %q, desired: %q", logPrefix, existingTier, desiredNetTier)


combine these log lines...

bowei · 2017-08-30T05:55:37Z

pkg/cloudprovider/providers/gce/gce_util.go

+	return isHTTPErrorCode(err, http.StatusForbidden)
+}
+
+func handleNetworkTierGetError(err error) (string, error) {


is this logic not possible to just put in the GetAlphaRegionAddress, ... functions?

this is a super generic function name...

It's shared by two files -- gce_addresses.go and gce_forwardingrule.go.
I renamed it to handleAlphaNetworkTierGetError() to make it slightly more explicit, and also added a TODO with a bug number to remove this.

Automatic merge from submit-queue e2e: Add tests for network tiers in GCE This test depends on #51301, which adds the new feature. Only the `e2e: Add tests for network tiers in GCE` commit is new. #51301 should pass this new test.

k8s-github-robot · 2017-09-01T00:51:25Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bowei, yujuhong
We suggest the following additional approver: erictune

Assign the PR to them by writing /assign @erictune in a comment when ready.

Associated issue: 51483

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

pkg/apis/OWNERS
~~pkg/cloudprovider/providers/gce/OWNERS~~ [bowei]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

yujuhong · 2017-09-01T01:44:37Z

Rebased and addressed the nits. Re-applying lgtm based on #51301 (comment)
Will probably need to rebase at some point though.

This feature supports specifying what network tier (premium, standard) to use for the load balancer backing the Service (type=LoadBalancer).

yujuhong · 2017-09-01T16:27:37Z

Rebased again and verified that the rebased PR still passed the e2e test.

yujuhong · 2017-09-01T16:34:12Z

Rebased again and verified that the rebased PR still passed the e2e test.

Re-applied the lgtm.

yujuhong · 2017-09-01T18:56:52Z

/test pull-kubernetes-kubemark-e2e-gce

kubemark issue has already been fixed by a revert. Re-testing

yujuhong · 2017-09-01T21:25:13Z

/test pull-kubernetes-kubemark-e2e-gce

yujuhong · 2017-09-01T22:54:41Z

FWIW, the PR has already passed the kubemark test: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51301/pull-kubernetes-kubemark-e2e-gce/49968/

krzyzacy · 2017-09-01T23:08:38Z

@spxtr ^^ is this mean something wrong when reporting back (crier)? any idea how to recover?

spxtr · 2017-09-01T23:15:56Z

/test pull-kubernetes-kubemark-e2e-gce

krzyzacy · 2017-09-01T23:27:17Z

http://prow.k8s.io/?type=presubmit&job=pull-kubernetes-kubemark-e2e-gce

I think the job is just backlogged... the test was from a different trigger earlier and you triggered it again and now it's queueing again

spxtr · 2017-09-01T23:43:52Z

Looks like this job is regularly taking 3+ hours to complete, including queue time. Welp.

yujuhong · 2017-09-01T23:54:27Z

I think the job is just backlogged... the test was from a different trigger earlier and you triggered it again and now it's queueing again

Thanks for checking. I'll check the queue next time before triggering....

yujuhong · 2017-09-02T01:03:30Z

Failed due to some infra flake: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51301/pull-kubernetes-kubemark-e2e-gce/50032

/test pull-kubernetes-kubemark-e2e-gce

fejta-bot · 2017-09-02T09:42:52Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

cblecker · 2017-09-02T17:53:02Z

/test pull-kubernetes-kubemark-e2e-gce

k8s-github-robot · 2017-09-03T05:52:22Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-09-03T06:38:55Z

Automatic merge from submit-queue

k8s-ci-robot · 2017-09-03T07:05:21Z

@yujuhong: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-bazel	`5b129fd`	link	`/test pull-kubernetes-bazel`
pull-kubernetes-e2e-kops-aws	`76945ad`	link	`/test pull-kubernetes-e2e-kops-aws`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

yujuhong added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 25, 2017

yujuhong assigned nicksardo and bowei Aug 25, 2017

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 25, 2017

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 25, 2017

yujuhong force-pushed the net-tiers-v0.3 branch from adf763c to 5b129fd Compare August 25, 2017 15:55

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 25, 2017

yujuhong mentioned this pull request Aug 29, 2017

e2e: Add tests for network tiers in GCE #51483

Merged

nicksardo reviewed Aug 29, 2017

View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 29, 2017

yujuhong force-pushed the net-tiers-v0.3 branch from 5b129fd to 2afc5e2 Compare August 29, 2017 19:31

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 29, 2017

yujuhong commented Aug 29, 2017

View reviewed changes

nicksardo reviewed Aug 29, 2017

View reviewed changes

yujuhong added this to the v1.8 milestone Aug 29, 2017

MrHohn reviewed Aug 30, 2017

View reviewed changes

bowei reviewed Aug 30, 2017

View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 30, 2017

yujuhong force-pushed the net-tiers-v0.3 branch from bd13596 to 1a59ea2 Compare August 31, 2017 01:15

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 1, 2017

yujuhong force-pushed the net-tiers-v0.3 branch from 1a59ea2 to 447b78c Compare September 1, 2017 01:42

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 1, 2017

yujuhong force-pushed the net-tiers-v0.3 branch from 447b78c to c219d88 Compare September 1, 2017 01:44

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 1, 2017

yujuhong added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 1, 2017

GCE: Add "Network Tiers" as an Alpha feature for L4 load balancers

76945ad

This feature supports specifying what network tier (premium, standard) to use for the load balancer backing the Service (type=LoadBalancer).

yujuhong force-pushed the net-tiers-v0.3 branch from c219d88 to 76945ad Compare September 1, 2017 16:26

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 1, 2017

yujuhong added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 1, 2017

k8s-github-robot merged commit 6d32783 into kubernetes:master Sep 3, 2017

yujuhong deleted the net-tiers-v0.3 branch September 11, 2017 20:31

		@@ -426,7 +439,7 @@ func (gce *GCECloud) DeleteExternalTargetPoolAndChecks(name, region, clusterID s
		// verifyUserRequestedIP checks the user-provided IP to see whether it can be
		// used for the LB. It also returns whether the IP is considered owned by the

GCE: Add Alpha feature "Network Tiers" for external L4 load balancers #51301

GCE: Add Alpha feature "Network Tiers" for external L4 load balancers #51301

Conversation

yujuhong commented Aug 25, 2017 • edited Loading

yujuhong commented Aug 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yujuhong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicksardo commented Aug 29, 2017

yujuhong commented Aug 29, 2017

yujuhong commented Aug 30, 2017

MrHohn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yujuhong Aug 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yujuhong Aug 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yujuhong commented Aug 30, 2017

nicksardo commented Aug 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-github-robot commented Sep 1, 2017

yujuhong commented Sep 1, 2017

yujuhong commented Sep 1, 2017

yujuhong commented Sep 1, 2017

yujuhong commented Sep 1, 2017

yujuhong commented Sep 1, 2017

yujuhong commented Sep 1, 2017

krzyzacy commented Sep 1, 2017

yujuhong commented Aug 25, 2017 •

edited

Loading

yujuhong Aug 30, 2017 •

edited

Loading

yujuhong Aug 30, 2017 •

edited

Loading