Baremetal http issue on openebs-operator #1184

Doooooo0o · 2018-01-31T13:59:06Z

Bug report

Bug description

openebs-operator can't reach k8s' api server because it tries to use a https scheme instead of the http defined clusterwide.
What you expected to happen:
maya doesn't have this issue, so I expected to be the same on operator.

Bug Reproduction

Setup an up to date (1.9.1) k8s bare metal cluster. Use the firewall security model described here : https://kubernetes.io/docs/getting-started-guides/scratch/#security-models
Run the pod.

Else

Slack related archive begins here : https://openebs-community.slack.com/archives/C3NPGQ6G3/p1517394852000413 and ends here : https://openebs-community.slack.com/archives/C3NPGQ6G3/p1517405899000094

The text was updated successfully, but these errors were encountered:

kmova · 2018-02-01T18:31:46Z

The error is caused by the use of "InClusterConfig" to access the kubenetes api server which will default to use https. Similar to the DNS plugin listed above, there has to be an option to specify : insecure-skip-tls-verify that will result in setting the Config.Insecure to true.

	// Create an InClusterConfig and use it to create a client for the controller
	// to use to communicate with Kubernetes
	config, err := rest.InClusterConfig()
	if err != nil {
		glog.Errorf("Failed to create config: %v", err)
	}
	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		glog.Errorf("Failed to create client: %v", err)
	}

	// The controller needs to know what the server version is because out-of-tree
	// provisioners aren't officially supported until 1.5
	serverVersion, err := clientset.Discovery().ServerVersion()
	if err != nil {
		glog.Errorf("Error getting server version: %v", err)
	}

AmitKumarDas · 2018-02-02T04:45:01Z

Tried with various ENV options to set the scheme to http in a minikube deployment. However, it seems we need to make this work with https with Insecure option i.e. avoid checking the certs against a CA

Currently checking with sig-api-machinery if this can be done via a config push than a code change.

kmova · 2018-02-02T15:19:08Z

@theonlydoo, I think I have got the issue reproduced using minikube. I am using the following test yaml to check if the fix is going to work. Can you please apply the following yaml in your setup and share the output.

Please replace 10.128.0.12 with your kubernetes server IP address and 8080 with the insecure port.

apiVersion: v1
kind: Pod
metadata:
  name: k8s-go-client
spec:
  containers:
  - image: openebs/k8s-client:latest
    imagePullPolicy: Always
    name: k8s-go-client
    args :
    -  /go/src/github.com/openebs/test-storage/k8s-client/listpods.go
    - -master=10.128.0.12:8080

Here is the information/output of running the above test file in my minikube (using localkube) setup where apiserver is listening only on insecure port 8080:

sudo netstat -nap | grep localkube | grep LISTEN | grep -v LISTENING

Output:
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      17901/localkube 
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      17901/localkube 
tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      17901/localkube 
tcp6       0      0 :::4194                 :::*                    LISTEN      17901/localkube 
tcp6       0      0 :::10250                :::*                    LISTEN      17901/localkube 
tcp6       0      0 :::10251                :::*                    LISTEN      17901/localkube 
tcp6       0      0 :::10252                :::*                    LISTEN      17901/localkube 
tcp6       0      0 :::10255                :::*                    LISTEN      17901/localkube 
tcp6       0      0 :::30000                :::*                    LISTEN      17901/localkube 
tcp6       0      0 :::8080                 :::*                    LISTEN      17901/localkube

The API server is available at 10.128.0.12:8080

kubectl --server='10.128.0.12:8080' get nodes

Output:
NAME             STATUS    ROLES     AGE       VERSION
kmova-minikube   Ready     <none>    5h        v1.9.0

In this minikube setup, I am running the following test.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: k8s-go-client
spec:
  containers:
  - image: openebs/k8s-client:latest
    imagePullPolicy: Always
    name: k8s-go-client
    args :
    -  /go/src/github.com/openebs/test-storage/k8s-client/listpods.go
    - -master=10.128.0.12:8080

Note the way IP address and port are specified.

kubectl --server='10.128.0.12:8080' apply -f test.yaml

Output
pod "k8s-go-client" created

The above command launches a pod that compiles client-go code and fetches the list of pods and pvs. It may take 1/2 mins before the output starts to appear:

kubectl --server='10.128.0.12:8080' logs k8s-go-client

Output:
Either master or kubeconfig specified. building kube config from that..
There are 7 pods in the cluster
There are 0 PVs in the cluster
There are 7 pods in the cluster
There are 0 PVs in the cluster
There are 7 pods in the cluster
There are 0 PVs in the cluster
There are 7 pods in the cluster
.
.
.

If I don't specify the master address as below:

apiVersion: v1
kind: Pod
metadata:
  name: k8s-go-client-incluster
spec:
  containers:
  - image: openebs/k8s-client:latest
    imagePullPolicy: Always
    name: k8s-go-client
    #replace the listpods.go in the following args to your desired test file
    args :
    -  /go/src/github.com/openebs/test-storage/k8s-client/listpods.go

I see the following in the logs:

kubectl --server='10.128.0.12:8080' logs -f k8s-go-client-incluster

Ouptut:
panic: Get https://10.96.0.1:443/api/v1/persistentvolumes: dial tcp 10.96.0.1:443: getsockopt: connection refused

goroutine 1 [running]:
panic(0xe84920, 0xc42028e310)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
main.main()
        /go/src/github.com/openebs/test-storage/k8s-client/listpods.go:63 +0x3cf
exit status 2

AmitKumarDas · 2018-02-04T03:17:18Z

maya v0.5.0 has approved a checkin to fix this issue.
Same fix needs to go to maya master

Doooooo0o · 2018-02-07T14:22:56Z

openebs-archive/external-storage#31 fixes indeed

1. Why is this change necessary ? - To support volume provisioning and discovering maya-apiserver on insecure/Non SSL connections to kubernetes master. 2. How does this change address the issue ? - Add flags `--master` and `--kubeconfig`, Either of these can be used to communicate with the kubernetes-master. 3. How to verify this change ? - Set-up your kubernetes cluster to allow insecure/Non SSL connections ** This is added only for testing purpose, please use secure connections otherwise. ** - Please verify whether your setup allows the insecure connections.[See](http://yasassriratnayake.blogspot.in/2017/05/how-to-allow-insecurenon-ssl.html) 4. What side effects does this change have ? - Flags added is optional (either of `master` and `kubeconfig` can be used to connect with kubernetes master) and can be used as per requirement - These flags can be passed in yaml file with the following fields ``` image: <openebs-k8s-provisioner-image> command: ["openebs-provisioner"] args: - -master=<master's-ip>:8080 ``` 5. Other details fix: openebs/openebs#1184

1. Why is this change necessary ? - To provide support for insecure/non SSL connections on kubernetes via k8s environment variables 2. How does this change address the issue ? - Retrieve env variables directly from k8s env instead of getting as flags 3. How to verify this change ? - Test the latest image with kubernetes cluster allowing insecure/Non SSL connections. 4. What side effects does this change have ? - This commit adds a file env.go that has all the env variables defined and can be used in provisioner. 5. Other details improvement: openebs-archive#30 fix: openebs/openebs#1184

1. Why is this change necessary ? - To support volume provisioning and discovering maya-apiserver on insecure/Non SSL connections to kubernetes master. 2. How does this change address the issue ? - Add flags `--master` and `--kubeconfig`, Either of these can be used to communicate with the kubernetes-master. 3. How to verify this change ? - Set-up your kubernetes cluster to allow insecure/Non SSL connections ** This is added only for testing purpose, please use secure connections otherwise. ** - Please verify whether your setup allows the insecure connections.[See](http://yasassriratnayake.blogspot.in/2017/05/how-to-allow-insecurenon-ssl.html) 4. What side effects does this change have ? - Flags added is optional (either of `master` and `kubeconfig` can be used to connect with kubernetes master) and can be used as per requirement - These flags can be passed in yaml file with the following fields ``` image: <openebs-k8s-provisioner-image> command: ["openebs-provisioner"] args: - -master=<master's-ip>:8080 ``` 5. Other details fix: openebs/openebs#1184

1. Why is this change necessary ? - To provide support for insecure/non SSL connections on kubernetes via k8s environment variables 2. How does this change address the issue ? - Retrieve env variables directly from k8s env instead of getting as flags 3. How to verify this change ? - Test the latest image with kubernetes cluster allowing insecure/Non SSL connections. 4. What side effects does this change have ? - This commit adds a file env.go that has all the env variables defined and can be used in provisioner. 5. Other details improvement: openebs-archive/external-storage#30 fix: openebs/openebs#1184

kmova assigned AmitKumarDas and ksatchit Feb 1, 2018

kmova added the P 0 label Feb 1, 2018

kmova added repo/maya Area : Provisioning labels Feb 2, 2018

ksatchit mentioned this issue Feb 3, 2018

Include e2e tests to verify OpenEBS operator and volume pod behaviour with insecure access to kube-apiserver #1197

Closed

This was referenced Feb 3, 2018

allow maya api server to connect to K8s API using kubeconfig or K8s master address openebs-archive/maya#256

Closed

fixes 1184: allow maya api server to connect to K8s API using kubeconfig or K8s master address openebs-archive/maya#257

Merged

utkarshmani1997 mentioned this issue Feb 5, 2018

Add support for insecure/Non SSL connections on Kubernetes master openebs-archive/external-storage#29

Closed

AmitKumarDas mentioned this issue Feb 5, 2018

Allows to connect to K8s API using kubeconfig or K8s master address openebs-archive/maya#259

Merged

This was referenced Feb 5, 2018

Add support for insecure/Non SSL connections on Kubernetes master openebs-archive/external-storage#30

Merged

Add support for insecure/non SSL connection on kubernetes via env vars openebs-archive/external-storage#31

Merged

Doooooo0o closed this as completed Feb 7, 2018

kmova added this to the 0.5.2 milestone Feb 22, 2018

This was referenced Feb 22, 2018

Adds ENV variables to connect to K8s APIs #1213

Merged

Adds ENV variables to connect to K8s APIs #1218

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Baremetal http issue on openebs-operator #1184

Baremetal http issue on openebs-operator #1184

Doooooo0o commented Jan 31, 2018

kmova commented Feb 1, 2018

AmitKumarDas commented Feb 2, 2018 •

edited

Loading

kmova commented Feb 2, 2018

AmitKumarDas commented Feb 4, 2018 •

edited

Loading

Doooooo0o commented Feb 7, 2018

Baremetal http issue on openebs-operator #1184

Baremetal http issue on openebs-operator #1184

Comments

Doooooo0o commented Jan 31, 2018

Bug report

Bug description

Bug Reproduction

Else

kmova commented Feb 1, 2018

AmitKumarDas commented Feb 2, 2018 • edited Loading

kmova commented Feb 2, 2018

AmitKumarDas commented Feb 4, 2018 • edited Loading

Doooooo0o commented Feb 7, 2018

AmitKumarDas commented Feb 2, 2018 •

edited

Loading

AmitKumarDas commented Feb 4, 2018 •

edited

Loading