Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable --log-level=info for docker daemons for scalability jenkins jo… #22453

Merged
merged 1 commit into from
Mar 4, 2016

Conversation

dchen1107
Copy link
Member

…b and e2e test.

This should help with debugging #22166, #21866 and #22259

@k8s-github-robot
Copy link

Labelling this PR as size/S

@k8s-github-robot k8s-github-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Mar 3, 2016
@@ -69,6 +69,7 @@ ENABLE_CLUSTER_MONITORING="${KUBE_ENABLE_CLUSTER_MONITORING:-influxdb}"
# Default Log level for all components in test clusters and variables to override it in specific components.
TEST_CLUSTER_LOG_LEVEL="${TEST_CLUSTER_LOG_LEVEL:---v=4}"
KUBELET_TEST_LOG_LEVEL="${KUBELET_TEST_LOG_LEVEL:-$TEST_CLUSTER_LOG_LEVEL}"
DOCKER_TEST_LOG_LEVEL="${DOCKER_TEST_LOG_LEVEL:---log-level=info}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have it default to info instead of empty?

  1. That's redundant with the change in kubernetes-e2e.yaml;
  2. If the default drifts elsewhere, this default won't drift with it, which is probably bad.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kubernetes-e2e.yaml only affects jenkins project. I also want to enable this for dev's e2e tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally we'd prefer that test clusters look as similar to prod clusters as possible. cc @spxtr @kubernetes/goog-testing

@ikehz
Copy link
Contributor

ikehz commented Mar 3, 2016

I'm a bit skeptical of the default, otherwise LGTM.

@ikehz
Copy link
Contributor

ikehz commented Mar 3, 2016

Also I know ~zero about the salt stuff, so @mikedanese should still review that.

@k8s-bot
Copy link

k8s-bot commented Mar 3, 2016

GCE e2e build/test passed for commit 84689a7.

@mikedanese
Copy link
Member

LGTM

@mikedanese mikedanese added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 3, 2016
@@ -139,6 +139,8 @@
export REGISTER_MASTER="true"
# Reduce logs verbosity
export TEST_CLUSTER_LOG_LEVEL="--v=2"
# TODO: Remove when we figure out #22166 and other docker potential slowness
export DOCKER_TEST_LOG_LEVEL="--log-level=info"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably don't need this, since it's also specified in config-test.sh?

@dchen1107 dchen1107 added this to the v1.2 milestone Mar 3, 2016
@dchen1107 dchen1107 added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. kind/flake Categorizes issue or PR as related to a flaky test. labels Mar 3, 2016
j3ffml added a commit that referenced this pull request Mar 4, 2016
Enable --log-level=info for docker daemons for scalability jenkins jo…
@j3ffml j3ffml merged commit 7215860 into kubernetes:master Mar 4, 2016
@j3ffml
Copy link
Contributor

j3ffml commented Mar 4, 2016

This appears to have broken all gke e2e runs. I tried creating a test cluster and got the same failure. Tailing docker.log on a node I see

jeffml@gke-test1-88fedc32-node-i4ey:/var/log$ cat docker.log 
Warning: '-r' is deprecated, it will be removed soon. See usage.
time="2016-03-04T03:11:57.836369005Z" level=info msg="API listen on /var/run/docker.sock" 
time="2016-03-04T03:11:57.873855211Z" level=info msg="[graphdriver] using prior storage driver \"aufs\"" 
time="2016-03-04T03:11:58.077287383Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: WARNING: Module br_netfilter not found.\ninsmod /lib/modules/3.16.0-4-amd64/kernel/net/llc/llc.ko \ninsmod /lib/modules/3.16.0-4-amd64/kernel/net/802/stp.ko \ninsmod /lib/modules/3.16.0-4-amd64/kernel/net/bridge/bridge.ko \n, error: exit status 1" 
time="2016-03-04T03:11:58.416501132Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" 
time="2016-03-04T03:11:58.549275308Z" level=warning msg="Your kernel does not support swap memory limit." 
time="2016-03-04T03:11:58.549344506Z" level=warning msg="Your kernel does not support kernel memory limit." 
time="2016-03-04T03:11:58.549423732Z" level=warning msg="Your kernel does not support cgroup cfs period" 
time="2016-03-04T03:11:58.549442568Z" level=warning msg="Your kernel does not support cgroup cfs quotas" 
time="2016-03-04T03:11:58.550135460Z" level=info msg="Loading containers: start." 

time="2016-03-04T03:11:58.550285714Z" level=info msg="Loading containers: done." 
time="2016-03-04T03:11:58.550302981Z" level=info msg="Daemon has completed initialization" 
time="2016-03-04T03:11:58.550406336Z" level=info msg="Docker daemon" commit=a34a1d5 execdriver=native-0.2 graphdriver=aufs version=1.9.1 
time="2016-03-04T03:12:14.982675418Z" level=info msg="GET /info" 
time="2016-03-04T03:12:15.059516947Z" level=info msg="GET /version" 
time="2016-03-04T03:12:15.111925747Z" level=info msg="GET /info" 
time="2016-03-04T03:12:15.127827941Z" level=info msg="GET /version" 
time="2016-03-04T03:12:15.128955608Z" level=info msg="GET /containers/json?all=1" 
time="2016-03-04T03:12:15.134048954Z" level=info msg="GET /images/json" 
time="2016-03-04T03:12:15.149158182Z" level=info msg="GET /containers/json?all=1" 
time="2016-03-04T03:12:15.149892466Z" level=info msg="GET /version" 
time="2016-03-04T03:12:15.150325546Z" level=info msg="GET /info" 
time="2016-03-04T03:12:15.158478559Z" level=info msg="GET /info" 
time="2016-03-04T03:12:15.159848465Z" level=info msg="GET /version" 
time="2016-03-04T03:12:15.160193339Z" level=info msg="GET /images/json" 
time="2016-03-04T03:12:15.162576534Z" level=info msg="GET /containers/kubelet/json" 
time="2016-03-04T03:12:15.162741561Z" level=error msg="Handler for GET /containers/kubelet/json returned error: no such id: kubelet" 
time="2016-03-04T03:12:15.162769074Z" level=error msg="HTTP Error" err="no such id: kubelet" statusCode=404 
time="2016-03-04T03:12:15.178814379Z" level=info msg="GET /images/json" 
time="2016-03-04T03:12:15.179834266Z" level=info msg="GET /containers/json?all=1" 
time="2016-03-04T03:12:15.225028780Z" level=info msg="GET /version" 
time="2016-03-04T03:12:15.230640466Z" level=info msg="GET /containers/docker-daemon/json" 
time="2016-03-04T03:12:15.230830804Z" level=error msg="Handler for GET /containers/docker-daemon/json returned error: no such id: docker-daemon" 
time="2016-03-04T03:12:15.230857473Z" level=error msg="HTTP Error" err="no such id: docker-daemon" statusCode=404 
time="2016-03-04T03:12:20.226019136Z" level=info msg="GET /version" 
time="2016-03-04T03:12:25.227118808Z" level=info msg="GET /version" 
time="2016-03-04T03:12:30.228005010Z" level=info msg="GET /version" 
time="2016-03-04T03:12:35.229003830Z" level=info msg="GET /version" 
time="2016-03-04T03:12:40.229972371Z" level=info msg="GET /version" 
time="2016-03-04T03:12:45.231596293Z" level=info msg="GET /version" 
time="2016-03-04T03:12:50.232591092Z" level=info msg="GET /version" 
time="2016-03-04T03:12:55.234426657Z" level=info msg="GET /version" 
time="2016-03-04T03:13:00.235311225Z" level=info msg="GET /version" 
time="2016-03-04T03:13:05.236096615Z" level=info msg="GET /version" 
time="2016-03-04T03:13:10.237050235Z" level=info msg="GET /version" 
time="2016-03-04T03:13:15.167154067Z" level=info msg="GET /containers/json?all=1" 
time="2016-03-04T03:13:15.237956255Z" level=info msg="GET /version" 
time="2016-03-04T03:13:20.238841177Z" level=info msg="GET /version" 
time="2016-03-04T03:13:20.651732278Z" level=info msg="Processing signal 'terminated'" 
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
See '/usr/bin/docker daemon --help'.
flag provided but not defined: --v
...

Reverting this to unbreak e2e, we can roll forward after.

@@ -7,5 +7,9 @@
{% if pillar.get('network_provider', '').lower() == 'kubenet' %}
{% set bridge_opts = "" %}
{% endif -%}
DOCKER_OPTS="{{grains_opts}} {{e2e_opts}} {{bridge_opts}} --iptables=false --ip-masq=false --log-level=warn"
{% set log_level = pillar['log_level'] -%}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be:

{% set log_level = "--log-level=warn" -%}

or

{% set log_level = "" -%}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My fault! Copy&paste this code and forgot to remove. Sorry about this.

@gmarek
Copy link
Contributor

gmarek commented Mar 4, 2016

cc @gmarek @wojtek-t

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants