Skip to content

Commit

Permalink
Port Install CNI to golang (#25332)
Browse files Browse the repository at this point in the history
* Port CNI installer from shell to golang

* Capitalize acronyms in constant names

* Decouple environment variables from functions

* Make variable and function names more clear

* Convert array to set

* Fix filepath bug in tests

* Wait until main CNI config file exists to intall Istio CNI as a chained CNI plugin

* Add check install and cleanup; Keep container alive

* Cleanup on SIGINT and SIGTERM caused by killing container

* Refactor, clean up, add comments

* Sort test data JSON map keys and update cniVersions

* Remove unnecessary prefix characters for creating temp dirs

* Fix and clean up unit tests

* Extend context to createCNIConfigFile; Add unit test

* Remove relative paths and clean up CNI config e2e tests

* Fix lint errors

* Fix kubeconfig template; Add unit tests for creating kubeconfig file

* Remove install-cni.sh and dependencies; Update Dockerfile and charts

* Write kubeconfig file with default 0600 permissions

* Test script restart in CNI config integration test; cleanup

* Add unit test for checkInstall

* Add helper functions to handle json unmarshalling panic; add unit tests

* Address PR comments; cleanup

* Add test cases for standalone CNI plugin in integration tests

* Add make target for install-cni integration test

* Address PR comments; cleanup

* Decouple signal handling from install process

* Add Installer struct and refactor

* Fix lint error

* Remove absolute path to install-cni binary

Co-authored-by: Jonh Wendell <jonh.wendell@redhat.com>
  • Loading branch information
bcheung and jwendell authored Jul 20, 2020
1 parent 624e880 commit d110604
Show file tree
Hide file tree
Showing 49 changed files with 2,499 additions and 594 deletions.
13 changes: 9 additions & 4 deletions Makefile.core.mk
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,9 @@ BINARIES:=./istioctl/cmd/istioctl \
./pkg/test/echo/cmd/server \
./mixer/test/policybackend \
./operator/cmd/operator \
./cni/cmd/istio-cni ./cni/cmd/istio-cni-repair \
./cni/cmd/istio-cni \
./cni/cmd/istio-cni-repair \
./cni/cmd/install-cni \
./tools/istio-iptables

# List of binaries included in releases
Expand Down Expand Up @@ -435,22 +437,25 @@ ${ISTIO_BIN}/go-junit-report:
# This is just an alias for racetest now
test: racetest

TEST_TARGETS ?= ./pilot/... ./istioctl/... ./operator/... ./mixer/... ./galley/... ./security/... ./pkg/... ./tests/common/... ./tools/istio-iptables/... ./cni/cmd/...
TEST_TARGETS ?= ./pilot/... ./istioctl/... ./operator/... ./mixer/... ./galley/... ./security/... ./pkg/... ./tests/common/... ./tools/istio-iptables/... ./cni/cmd/... ./cni/pkg/...
# For now, keep a minimal subset. This can be expanded in the future, especially after mixer removal, which has some expensive tests that may OOM.
BENCH_TAREGTS ?= ./pilot/...
BENCH_TARGETS ?= ./pilot/...

.PHONY: racetest
racetest: $(JUNIT_REPORT) ## Runs all unit tests with race detection enabled
go test ${GOBUILDFLAGS} ${T} -race $(TEST_TARGETS) 2>&1 | tee >($(JUNIT_REPORT) > $(JUNIT_OUT))

.PHONY: benchtest
benchtest: $(JUNIT_REPORT) ## Runs all benchmarks
prow/benchtest.sh run $(BENCH_TAREGTS)
prow/benchtest.sh run $(BENCH_TARGETS)
prow/benchtest.sh compare

report-benchtest:
prow/benchtest.sh report

cni.install-test: docker.install-cni
HUB=${HUB} TAG=${TAG} go test ${GOBUILDFLAGS} -count=1 ${T} ./cni/test/...

#-----------------------------------------------------------------------------
# Target: clean
#-----------------------------------------------------------------------------
Expand Down
14 changes: 7 additions & 7 deletions cni/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,19 +226,19 @@ $ gcloud logging read "resource.type=gce_instance AND jsonPayload.SYSLOG_IDENTIF
- creates service-account `istio-cni` with `ClusterRoleBinding` to allow gets on pods' info

- `install-cni` container
- copies `istio-cni` binary and `istio-iptables.sh` to `/opt/cni/bin`
- creates kubeconfig for the service account the pod is run under
- copies `istio-cni` binary and `istio-iptables` to `/opt/cni/bin`
- creates kubeconfig for the service account the pod runs under
- injects the CNI plugin config to the config file pointed to by CNI_CONF_NAME env var
- example: `CNI_CONF_NAME: 10-calico.conflist`
- `jq` is used to insert `CNI_NETWORK_CONFIG` into the `plugins` list in `/etc/cni/net.d/${CNI_CONF_NAME}`
- the program inserts `CNI_NETWORK_CONFIG` into the `plugins` list in `/etc/cni/net.d/${CNI_CONF_NAME}`

- `istio-cni`
- CNI plugin executable copied to `/opt/cni/bin`
- currently implemented for k8s only
- on pod add, determines whether pod should have netns setup to redirect to Istio proxy
- if so, calls `istio-iptables.sh` with params to setup pod netns
- if so, calls `istio-iptables` with params to setup pod netns

- [istio-iptables.sh](tools/istio-cni-docker.mk)
- [istio-iptables](tools/istio-cni-docker.mk)
- sets up iptables to redirect a list of ports to the port envoy will listen

### Background
Expand Down Expand Up @@ -280,7 +280,7 @@ Specifically:
- [RBAC](https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/rbac.yaml)
- this creates the service account the CNI plugin is configured to use to access the kube-api-server

The installation script `install-cni.sh` injects the `istio-cni` plugin config at the end of the CNI plugin chain
The installation program `install-cni` injects the `istio-cni` plugin config at the end of the CNI plugin chain
config. It creates or modifies the file from the configmap created by the Kubernetes manifest.

#### Plugin Logic
Expand All @@ -293,7 +293,7 @@ Workflow:
1. If excluded, ignore the pod and return prevResult
1. Setup redirect rules for the pods:
1. Get the port list from pods definition
1. Setup iptables with required port list: `nsenter --net=<k8s pod netns> /opt/cni/bin/istio-iptables.sh ...`
1. Setup iptables with required port list: `nsenter --net=<k8s pod netns> /opt/cni/bin/istio-iptables ...`

Following conditions will prevent the redirect rules to be setup in the pods:

Expand Down
42 changes: 42 additions & 0 deletions cni/cmd/install-cni/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
// Copyright Istio Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
"context"
"os"
"os/signal"
"syscall"

"istio.io/istio/cni/pkg/install-cni/cmd"
"istio.io/pkg/log"
)

func main() {
// Create context that cancels on termination signal
ctx, cancel := context.WithCancel(context.Background())
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
go func(sigChan chan os.Signal, cancel context.CancelFunc) {
sig := <-sigChan
log.Infof("Exit signal received: %s", sig)
cancel()
}(sigChan, cancel)

rootCmd := cmd.GetCommand()
if err := rootCmd.ExecuteContext(ctx); err != nil {
os.Exit(1)
}
}
6 changes: 3 additions & 3 deletions cni/cmd/istio-cni/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ type Kubernetes struct {
InterceptRuleMgrType string `json:"intercept_type"`
NodeName string `json:"node_name"`
ExcludeNamespaces []string `json:"exclude_namespaces"`
CniBinDir string `json:"cni_bin_dir"`
CNIBinDir string `json:"cni_bin_dir"`
}

// PluginConf is whatever you expect your configuration json to be. This is whatever
Expand Down Expand Up @@ -146,8 +146,8 @@ func cmdAdd(args *skel.CmdArgs) error {
}
log.Infof("Getting identifiers with arguments: %s", args.Args)
log.Infof("Loaded k8s arguments: %v", k8sArgs)
if conf.Kubernetes.CniBinDir != "" {
nsSetupBinDir = conf.Kubernetes.CniBinDir
if conf.Kubernetes.CNIBinDir != "" {
nsSetupBinDir = conf.Kubernetes.CNIBinDir
}
if conf.Kubernetes.InterceptRuleMgrType != "" {
interceptRuleMgrType = conf.Kubernetes.InterceptRuleMgrType
Expand Down
5 changes: 2 additions & 3 deletions cni/deployments/kubernetes/Dockerfile.install-cni
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,12 @@ jq='1.5*' \

COPY istio-cni /opt/cni/bin/
COPY istio-iptables /opt/cni/bin/
COPY install-cni.sh /install-cni.sh
COPY filter.jq /filter.jq
COPY install-cni /usr/local/bin/
COPY istio-cni.conf.default /istio-cni.conf.tmp

# Copy over the Repair binary
COPY istio-cni-repair /opt/cni/bin/

ENV PATH=$PATH:/opt/cni/bin
WORKDIR /opt/cni/bin
CMD ["/install-cni.sh"]
CMD ["/usr/local/bin/install-cni"]
13 changes: 0 additions & 13 deletions cni/deployments/kubernetes/install/scripts/filter.jq

This file was deleted.

Loading

0 comments on commit d110604

Please sign in to comment.