Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

conntrack util not installed #5671

Closed
johanneswuerbach opened this issue Aug 20, 2018 · 4 comments
Closed

conntrack util not installed #5671

johanneswuerbach opened this issue Aug 20, 2018 · 4 comments
Labels
area/nodeup sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@johanneswuerbach
Copy link
Contributor

kubernetes uses the conntrack binary to perform some UDP related cleanup operations like kubernetes/kubernetes#59286 or kubernetes/kubernetes#22573

It seems that kops is not explicitly installing the conntrack tool, causing those cleanup tasks to never be performed.

Examples of others installing it
kubernetes/kubernetes#26839, kubernetes/kubernetes#64640

1. What kops version are you running? The command kops version, will display
this information.

Version 1.10.0 (git-8b52ea6d1)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-08T16:31:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.9", GitCommit:"57729ea3d9a1b75f3fc7bbbadc597ba707d47c8a", GitTreeState:"clean", BuildDate:"2018-06-29T01:07:01Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?

AWS / CoreOS nodes

4. What commands did you run? What is the simplest way to reproduce this issue?

conntrack

5. What happened after the commands executed?

conntrack: command not found

6. What did you expect to happen?

conntrack to be installed

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-06-22T07:08:46Z
  labels:
    kops.k8s.io/cluster: xxx
  name: master-a
spec:
  image: coreos.com/CoreOS-stable-*-hvm
  machineType: t2.medium
  maxSize: 1
  minSize: 1
  nodeLabels:
    beta.kubernetes.io/fluentd-ds-ready: "true"
  role: Master
  subnets:
  - a
@geojaz
Copy link
Member

geojaz commented Aug 21, 2018 •
edited
Loading

@johanneswuerbach
Thanks for the report, sorry you're running into this problem.
Could you please clarify the exact coreos version string and AMI that you're using?

Also, it would be super helpful if you could post the logs that show where k8s tried to call out to conntrack and failed.

I'd need more information to make a definitive call, but I'm wondering if talking to coreos/redhat is the right place to push this forward? It sounds to me like a problem with the AMI, not a problem with how it was deployed. We (kops team: ie-> @justinsb ) create the debian images but I'm not sure if we're involved in the others.

Is that fair and/or useful?

@geojaz geojaz added area/nodeup sig/node Categorizes an issue or PR as relevant to SIG Node. labels Aug 21, 2018
@johanneswuerbach
Copy link
Contributor Author

We are running the latest stable CoreOS AMIs in us-east-1, currently 1800.7.0 / ami-00cc4337762ba4a52

The addition of conntrack to CoreOS itself was rejected previously, coreos/bugs#485

Going forward with this I see currently three options:

  1. Build custom AMIs, which is kind of a bummer as that will cause problems with CoreOS auto-updating
  2. Modify the launch config user-data to fetch conntrack on boot (this is already done for the kubelet binary)
  3. Create a systemd one-shot hook, which fetches conntrack

We will go with 3. now, but I would prefer to upstream this into kops and would happily work on 2., if this is accepted. Option 2 is also similar, to what is proposed for GCI in upstream k8s kubernetes/kubernetes#64640

@geojaz
Copy link
Member

geojaz commented Aug 21, 2018 •
edited
Loading

Ok that's useful context. Thanks for the additional details.

  1. I can't speak for everyone, but I would say that baking coreos images is not something that we're going to support today and probably don't want to support longer term.

  2. We could theoretically hack the launch config and grab it during node bootstrapping. I think this solution is not too terrible... but I don't love hacking the launchconfig for everyone's special case. I really wish Cores would reconsider Rename to kops #1

  3. This is probably the current path of least resistance. I was doing something similar in my apparmor hook solution here kope.io/k8s-1.9-jessie and stretch images have a broken dbus #5450. It's not ideal, but it works. I think it gets you going most quickly.

Thinking a bit deeper, option 1 is not possible this evening, but I think it's something that we may want to do as our release process automation matures.

@johanneswuerbach
Copy link
Contributor Author

johanneswuerbach commented Aug 21, 2018 •
edited
Loading

I would strongly speak against creating custom CoreOS images as this breaks tooling like https://github.com/coreos/container-linux-update-operator, which is the main reason for us to use CoreOS (no more manual security upgrades)

There are already other customisation for CoreOS (like pulling socat) and I would just see that inline with this

kops/upup/pkg/fi/cloudup/apply_cluster.go

Lines 1084 to 1085 in 56ccfac

// At this time we just copy the socat binary to all distros. Most distros will be there own
// socat binary. Container operating systems like CoreOS need to have socat added to them.

Overall the recommended approach for CoreOS is running the kubelet itself inside a container, but thats a whole different story.

@johanneswuerbach johanneswuerbach mentioned this issue Sep 3, 2018
Merged
This was referenced Mar 29, 2020
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/nodeup sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@johanneswuerbach @geojaz