Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MESOS: forward globally declared cadvisor housekeeping flags #22974

Conversation

jdef
Copy link
Contributor

@jdef jdef commented Mar 15, 2016

@k8s-github-robot
Copy link

Labelling this PR as size/M

@k8s-github-robot k8s-github-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 15, 2016
@k8s-bot
Copy link

k8s-bot commented Mar 15, 2016

GCE e2e build/test passed for commit 622a28c.

@jdef
Copy link
Contributor Author

jdef commented Mar 15, 2016

FWIW cadvisor flags are deprecated #19432

@linsun
Copy link

linsun commented Mar 16, 2016

I think this changes work for me. Here is how I tried it:

  1. rebuild a km docker image with the change in it, plus setting these two flags when launching the scheduler: --housekeeping_interval=50s --global_housekeeping_interval=100s
  2. deploy mesos, and kubernetes as a framework on mesos using the km image. docker exec into the km container:
  • check the km scheduler process:
    00:00:00 /opt/kubernetes/km scheduler --housekeeping_interval=50s --global_housekeeping_interval=100s --address=10.143.197.73 --mesos-master=10.143.197.73:5050 --etcd-servers=http://10.143.197.73:4001 --mesos-user=root --api-servers=10.143.197.73:8888 --cluster-dns=10.10.10.10 --cluster-domain=cluster.local --v=4
  • validated in scheduler.log:

I0316 01:50:42.370195 47 service.go:503] prepared executor command "./km" with args '[./km minion --run-proxy=true --proxy-
bindall=false --proxy-logv=0 --proxy-mode=userspace --path-override= --max-log-size=10Mi --max-log-backups=5 --max-log-age=7 --c
onntrack-max=0 --conntrack-tcp-timeout-established=0 --api-servers=10.143.197.73:8888 --v=0 --allow-privileged=false --suicide-t
imeout=20m0s --mesos-launch-grace-period=5m0s --mesos-cgroup-prefix=mesos --cadvisor-port=4194 --sync-frequency=10s --contain-po
d-resources=true --enable-debugging-handlers=true --cluster-dns=10.10.10.10 --cluster-domain=cluster.local --housekeeping_interv
al=50s --global_housekeeping_interval=1m40s]'

I'm not sure how to check minion/server.go log, else, I can probably validate this line too:

log.V(2).Infof("Spawning hyperkube %v with args '%+v'", server, args)

@jdef
Copy link
Contributor Author

jdef commented Mar 16, 2016

you can look at the stderr of the mesos sandbox that's running the
executor, it shows the command lines executed for the kubelet and kube-proxy

On Tue, Mar 15, 2016 at 10:23 PM, Lin Sun notifications@github.com wrote:

I think this changes work for me. Here is how I tried it:

rebuild a km docker image with the change in it, plus setting these
two flags when launching the scheduler: --housekeeping_interval=50s
--global_housekeeping_interval=100s

2.

deploy mesos, and kubernetes as a framework on mesos using the km
image. docker exec into the km container:

check the km scheduler process:
00:00:00 /opt/kubernetes/km scheduler --housekeeping_interval=50s
--global_housekeeping_interval=100s
--address=10.143.197.73
--mesos-master=10.143.197.73:5050 --etcd-servers=
http://10.143.197.73:4001 --mesos-user=root --api-servers=
10.143.197.73:8888 --cluster-dns=10.10.10.10
--cluster-domain=cluster.local --v=4

validated in scheduler.log:

I0316 01:50:42.370195 47 service.go:503] prepared executor command "./km"
with args '[./km minion --run-proxy=true --proxy-
bindall=false --proxy-logv=0 --proxy-mode=userspace --path-override=
--max-log-size=10Mi --max-log-backups=5 --max-log-age=7 --c
onntrack-max=0 --conntrack-tcp-timeout-established=0 --api-servers=
10.143.197.73:8888 --v=0 --allow-privileged=false --suicide-t
imeout=20m0s --mesos-launch-grace-period=5m0s --mesos-cgroup-prefix=mesos
--cadvisor-port=4194 --sync-frequency=10s --contain-po
d-resources=true --enable-debugging-handlers=true
--cluster-dns=10.10.10.10 --cluster-domain=cluster.local
--housekeeping_interv al=50s --global_housekeeping_interval=1m40s]'

I'm not sure how to check minion/server.go log, else, I can probably
validate this line too:

log.V(2).Infof("Spawning hyperkube %v with args '%+v'", server, args)


You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
#22974 (comment)

@linsun
Copy link

linsun commented Mar 16, 2016

@jdef, I'm hitting an environmental issue where the executor on the host times out when downloading the km binary... seems the timeout is 1 min. trying to see if i can increase the network speed or tune the 1 min timeout value.

@jdef
Copy link
Contributor Author

jdef commented Mar 16, 2016

try the executor_registration_timeout slave option?

On Wed, Mar 16, 2016 at 11:05 AM, Lin Sun notifications@github.com wrote:

@jdef https://github.com/jdef, I'm hitting an environmental issue where
the executor on the host times out when downloading the km binary... seems
the timeout is 1 min. trying to see if i can increase the network speed or
tune the 1 min timeout value.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#22974 (comment)

@linsun
Copy link

linsun commented Mar 17, 2016

Sorry for the delay. I updated the network speed of the env, and the stderr log looks good:

I0317 03:08:12.063347 2850 task.go:220] starting task process "/opt/ibm/mesos/work/slaves/d58a68eb-5105-4be7-9ae8-577d5b24e9b2-S0/frameworks/d58a68eb-5105-4be7-9ae8-577d5b24e9b2-0001/executors/6d493c88c0ac86b3_k8sm-executor/runs/22ac059b-19ff-4ada-bc4d-65db178f3235/km" with args '[executor --api-servers=10.143.100.223:8888 --v=0 --allow-privileged=false --suicide-timeout=20m0s --mesos-launch-grace-period=5m0s --cadvisor-port=4194 --sync-frequency=10s --enable-debugging-handlers=true --cluster-dns=10.10.10.10 --cluster-domain=cluster.local --hostname-override=10.143.100.238 --kubelet-cgroups= --cgroup-root=/mesos/22ac059b-19ff-4ada-bc4d-65db178f3235 --housekeeping_interval=50s --global_housekeeping_interval=1m40s]'
I0317 03:08:12.066579 2850 task.go:257] task started executor

@jdef
Copy link
Contributor Author

jdef commented Mar 17, 2016

great. i'll mark this as mergeable. we missed the 1.2 cutoff, but this can
probably get cherrypicked for 1.2.1

On Wed, Mar 16, 2016 at 11:12 PM, Lin Sun notifications@github.com wrote:

Sorry for the delay. I updated the network speed of the env, and the
stderr log looks good:

I0317 03:08:12.063347 2850 task.go:220] starting task process
"/opt/ibm/mesos/work/slaves/d58a68eb-5105-4be7-9ae8-577d5b24e9b2-S0/frameworks/d58a68eb-5105-4be7-9ae8-577d5b24e9b2-0001/executors/6d493c88c0ac86b3_k8sm-executor/runs/22ac059b-19ff-4ada-bc4d-65db178f3235/km"
with args '[executor --api-servers=10.143.100.223:8888 --v=0
--allow-privileged=false --suicide-timeout=20m0s
--mesos-launch-grace-period=5m0s --cadvisor-port=4194 --sync-frequency=10s
--enable-debugging-handlers=true --cluster-dns=10.10.10.10
--cluster-domain=cluster.local --hostname-override=10.143.100.238
--kubelet-cgroups=
--cgroup-root=/mesos/22ac059b-19ff-4ada-bc4d-65db178f3235 --housekeeping_interval=50s
--global_housekeeping_interval=1m40s
]'
I0317 03:08:12.066579 2850 task.go:257] task started executor


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#22974 (comment)

@jdef jdef changed the title WIP/MESOS: forward globally declared cadvisor housekeeping flags MESOS: forward globally declared cadvisor housekeeping flags Mar 17, 2016
@jdef jdef added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 17, 2016
@k8s-github-robot
Copy link

Automatic merge from submit-queue

k8s-github-robot pushed a commit that referenced this pull request Mar 17, 2016
@k8s-github-robot k8s-github-robot merged commit 68892a3 into kubernetes:master Mar 17, 2016
@jdef jdef added this to the v1.2 milestone Mar 18, 2016
@jdef jdef deleted the jdef-forward-housekeeping-flags branch March 18, 2016 13:53
@bgrant0607
Copy link
Member

@jdef Why would you like this to be cherrypicked into 1.2?

@bgrant0607 bgrant0607 added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Mar 23, 2016
@jdef
Copy link
Contributor Author

jdef commented Mar 24, 2016

Some folks from IBM are doing some performance work and want the ability to
change these settings. We barely missed the cutoff for 1.2. The changeset
is relatively small and is backwards compat.

What's the criteria for getting things cherrypicked into 1.2?

On Wed, Mar 23, 2016 at 6:23 PM, Brian Grant notifications@github.com
wrote:

@jdef https://github.com/jdef Why would you like this to be
cherrypicked into 1.2?


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#22974 (comment)

eparis pushed a commit to eparis/kubernetes that referenced this pull request Mar 24, 2016
…ekeeping-flags

Auto commit by PR queue bot
(cherry picked from commit 68892a3)
@bgrant0607 bgrant0607 added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Mar 24, 2016
@k8s-cherrypick-bot
Copy link

Commit 50b41b6 found in the "release-1.2" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this s an error find help to get your PR picked.

AlainRoy pushed a commit to vmware-archive/kubernetes-archived that referenced this pull request Mar 29, 2016
…ekeeping-flags

Auto commit by PR queue bot
(cherry picked from commit 68892a3)
alena1108 pushed a commit to rancher/kubernetes that referenced this pull request May 20, 2016
…ekeeping-flags

Auto commit by PR queue bot
(cherry picked from commit 68892a3)
shyamjvs pushed a commit to shyamjvs/kubernetes that referenced this pull request Dec 1, 2016
…ekeeping-flags

Auto commit by PR queue bot
(cherry picked from commit 68892a3)
shouhong pushed a commit to shouhong/kubernetes that referenced this pull request Feb 14, 2017
…ekeeping-flags

Auto commit by PR queue bot
(cherry picked from commit 68892a3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform/mesos cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants