Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "add gzip compression to GET and LIST requests" #47191

Merged
merged 1 commit into from
Jun 8, 2017

Conversation

liggitt
Copy link
Member

@liggitt liggitt commented Jun 8, 2017

xref #47135

This reverts commit fc650a5 added in #45666

After #45666 merged, namespace deletion started flaking significantly in e2e runs.

Disabling the gzip compression and running the problematic suite fixed the issue, though I'm still not sure why. Namespace deletion time dropped significantly with gzip disabled.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 8, 2017
@k8s-github-robot k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Jun 8, 2017
@liggitt
Copy link
Member Author

liggitt commented Jun 8, 2017

@k8s-bot pull-kubernetes-unit test this

@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 8, 2017
@liggitt
Copy link
Member Author

liggitt commented Jun 8, 2017

cc @deads2k @ilackarms

@liggitt liggitt added queue/critical-fix release-note-none Denotes a PR that doesn't merit a release note. labels Jun 8, 2017
@liggitt liggitt added this to the v1.7 milestone Jun 8, 2017
@ilackarms
Copy link
Contributor

I wonder what would cause those tests to flake. Is there a way I can try to reproduce this locally?

@liggitt
Copy link
Member Author

liggitt commented Jun 8, 2017

I wasn't able to pin it down, but some possible ideas:

  • differences in behavior with content-length vs chunked encoding when there is sufficient data in the system to exceed the 4k response buffer (the filter isn't doing anything with transfer-encoding or content-length, and is assuming the wrapped handler isn't setting those headers)
  • something with the gzip filter causing clients to wait for responses to complete, slowing them down, and stacking namespace deletion handling by the controller
  • the gzip filter possibly getting put on subresource GET operations unintentionally (hadn't verified this)

for recreation... I would create a bunch of namespaces (50?) with some number of resources in them (including running pods to exercise graceful deletion loops), then delete them all at once and observe the namespace controller's behavior

@dchen1107
Copy link
Member

/lgtm

cc/ @kubernetes/kubernetes-release-managers @foxish, the build cop

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 8, 2017
@liggitt
Copy link
Member Author

liggitt commented Jun 8, 2017

e2e: SUCCESS! -- 302 Passed | 0 Failed | 0 Pending | 344 Skipped

@liggitt
Copy link
Member Author

liggitt commented Jun 8, 2017

unit test failure in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/47191/pull-kubernetes-unit/35607/ is known flake (#42289)

recommend merging this to fix the gce-e2e job

@liggitt
Copy link
Member Author

liggitt commented Jun 8, 2017

@k8s-bot pull-kubernetes-unit test this

@mikedanese
Copy link
Member

/lgtm

@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dchen1107, liggitt, mikedanese

Associated issue: 47135

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@dchen1107
Copy link
Member

All tests are green. I am going to manually merge the pr for #47135

@dchen1107 dchen1107 merged commit acabdc4 into kubernetes:master Jun 8, 2017
@liggitt liggitt deleted the revert-gzip branch June 9, 2017 14:10
@gmarek
Copy link
Contributor

gmarek commented Jun 12, 2017

It sounds like CPU usage issue. gzip is not particularly cheap and thus we shouldn't enable it by default.

@ilackarms
Copy link
Contributor

@gmarek working on a PR with feature gating to disable gzip by default
cc @simon3z

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants