Some quality of life enhancements in CNI delegation code:

- CNI config directory can be now configured - it is possible now to define which CNI config is to be used when delegating to static backends. NetworkType: name of the invoked binary, NetworkID: name of the CNI config file READM and DanmNet schema files both got a much deserved facelift
nokia · Mar 19, 2019 · e1c4f60 · e1c4f60
1 parent a518dbb
commit e1c4f60
Show file tree

Hide file tree

Showing 4 changed files with 44 additions and 32 deletions.
diff --git a/README.md b/README.md
@@ -276,7 +276,7 @@ Pay special attention to the DanmNet attribute called "NetworkType". This parame
 In case this parameter is set to "ipvlan", or is missing; then DANM's in-built IPVLAN CNI plugin creates the network (see next chapter for details).
 In case this attribute is provided and set to another value than "ipvlan", then network management is delegated to the CNI plugin with the same name.
 The binary will be searched in the configured CNI binary directory.
-Example: when a Pod is created and requests a network connection to a DanmNet with "NetworkType" set to "flannel", then DANM will delegate the creation of this network interface to the /opt/cni/bin/flannel binary.
+Example: when a Pod is created and requests a network connection to a DanmNet with "NetworkType" set to "flannel", then DANM will delegate the creation of this network interface to the <CONFIGURED_CNI_PATH_IN_KUBELET>/flannel binary.
 ##### Creating the configuration for delegated CNI operations
 We strongly believe that network management in general should be driven by one, generic API. Therefore, DANM is capable to "translate" the generic options coming from a DanmNet object into the specific "language" the delegated CNI plugin understands.
 This way users can dynamically configure various networking solutions via the same, abstract interface without caring about how a specific option is called exactly in the terminology of the delegated solution.
@@ -285,23 +285,26 @@ A generic framework supporting this method is built into DANM's code, but still
 As a result, DANM currently supports two integration levels:
 
  - **Dynamic integration level:** CNI-specific network attributes (such as IP ranges, parent host devices etc.) can be controlled on a per network level, taken directly from a DanmNet object
- - **Static integration level:** CNI-specific network attributes (such as IP ranges, parent host devices etc.) can be only configured on a per node level, via a static CNI configuration files (Note: this is the default CNI configuration method)
+ - **Static integration level:** CNI-specific network attributes (such as IP ranges, parent host devices etc.) can be only configured via static CNI configuration files (Note: this is the default CNI configuration method)
 
 Our aim is to integrate all the popular CNIs into the DANM eco-system over time, but currently the following CNI's achieved dynamic integration level:
 
  - DANM's own, in-built IPVLAN CNI plugin
 	 - Set the "NetworkType" parameter to value "ipvlan" to use this backend
-- Intel's DPDK-capable [SR-IOV CNI plugin](https://github.com/intel/sriov-cni )
+- Intel's [SR-IOV CNI plugin](https://github.com/intel/sriov-cni )
 	- Set the "NetworkType" parameter to value "sriov" to use this backend
 - Generic MACVLAN CNI from the CNI plugins example repository [MACVLAN CNI plugin](https://github.com/containernetworking/plugins/blob/master/plugins/main/macvlan/macvlan.go )
 	- Set the "NetworkType" parameter to value "macvlan" to use this backend
 
 No separate configuration needs to be provided to DANM when it connects Pods to DanmNets, if the network is backed by a CNI plugin with dynamic integration level.
 Everything happens automatically based on the DanmNet API itself!
 
-When network management is delegated to CNI plugins with static integration level; DANM will read their configuration from the configured CNI config directory.
-For example, when a Pod is connected to a DanmNet with "NetworkType" set to "flannel", DANM will pass the content of /etc/cni/net.d/flannel.conf file to the /opt/cni/bin/flannel binary by invoking a standard CNI operation.
-Generally supported DANM API-based features are configured even in this case.
+When network management is delegated to CNI plugins with static integration level; DANM reads their configuration from the configured CNI config directory.
+The directory can be configured via setting the "CNI_CONF_DIR" environment variable in DANM CNI's context (be it in the host namespace, or inside a Kubelet container). Default value is "/etc/cni/net.d".
+In case there are multiple configuration files present for the same backend, users can control which one is used in a specific network provisioning operation via the NetworkID DanmNet parameter.
+
+So, all in all: a Pod connecting to a DanmNet with "NetworkType" set to "bridge", and "NetworkID" set to "example_network" gets an interface provisioned by the <CONFIGURED_CNI_PATH_IN_KUBELET>/bridge binary based on the <CNI_CONF_DIR>/example_network.conf file!
+In addition to simply delegating the interface creation operation, generally supported DANM API-based features -such as static and dynamic IP route provisioning, flexible interface naming- are also configured by DANM.
 ##### Connecting Pods to DanmNets
 Pods can request network connections to DanmNets by defining one or more network connections in the annotation of their (template) spec field, according to the schema described in the **schema/network_attach.yaml** file.
 

diff --git a/pkg/cnidel/cniconfs.go b/pkg/cnidel/cniconfs.go
@@ -33,12 +33,12 @@ var (
 )
 
 //This function creates CNI configuration for all static-level backends
+//The CNI binary matching with NetowrkType is invoked with the CNI config file matching with NetworkID parameter
 func readCniConfigFile(netInfo *danmtypes.DanmNet) ([]byte, error) {
-  cniType := netInfo.Spec.NetworkType
-  //TODO: the path from where the config is read should not be hard-coded
-  rawConfig, err := ioutil.ReadFile("/etc/cni/net.d/" + cniType + ".conf")
+  cniConfig := netInfo.Spec.NetworkID
+  rawConfig, err := ioutil.ReadFile(cniConfigDir + "/" + cniConfig + ".conf")
   if err != nil {
-    return nil, errors.New("Could not load CNI config file for plugin:" + cniType)
+    return nil, errors.New("Could not load CNI config file: " + cniConfig +" for plugin:" + netInfo.Spec.NetworkType)
   }
   return rawConfig, nil
 }

diff --git a/pkg/cnidel/cnidel.go b/pkg/cnidel/cnidel.go
@@ -21,9 +21,7 @@ var (
   ipamType = "fakeipam"
   defaultDataDir = "/var/lib/cni/networks"
   flannelBridge = getEnv("FLANNEL_BRIDGE", "cbr0")
-  dpdkNicDriver = os.Getenv("DPDK_NIC_DRIVER")
-  dpdkDriver = os.Getenv("DPDK_DRIVER")
-  dpdkTool = os.Getenv("DPDK_TOOL")
+  cniConfigDir = getEnv("CNI_CONF_DIR", "/etc/cni/net.d")
 )
 
 // IsDelegationRequired decides if the interface creation operations should be delegated to a 3rd party CNI, or can be handled by DANM
@@ -104,7 +102,10 @@ func getCniIpamConfig(options danmtypes.DanmNetOption, ip4, ip6 string) danmtype
     subnet string
     ip string
   )
-  if options.Cidr != "" {
+  if ip4 == "" && ip6 == "" {
+    return danmtypes.IpamConfig{}
+  }
+  if ip4 != "" {
     ip = ip4
     subnet = options.Cidr
   } else {

diff --git a/schema/DanmNet.yaml b/schema/DanmNet.yaml
@@ -14,31 +14,34 @@ metadata:
 spec:
   # This parameter will be used in application description templates for referring to a desired connection point
   # Currently this parameter shall exactly match with the name of DanmNet object itself
+  # The parameter also configures which CNI configuration file is to be used if NetworkType points to a static-level CNI backend
   # MANDATORY - STRING
   NetworkID: ## NETWORK_NAME  ##
   # This parameter, if present, denotes which backend willl be used to provision the container interfaces connected to this network.
   # Currently supported values with dynamic integration level are IPVLAN (default), SRIOV, or MACVLAN.
   # - IPVLAN option results in an IPVLAN sub-interface provisioned in L2 mode, and connected to the designated host device
   # - SRIOV option pushes a pre-allocated Virtual Function of the configured host device to the container's netns
   # - MACVLAN option results in a MACVLAN sub-interface provisioned in bridge mode, and connected to the designated host device
-  # For any other CNI backend DANM will read their configuration from the configured CNI config directory.
-  # E.g. when a Pod is connected to a DanmNet with "NetworkType" set to "flannel", DANM will pass the content of /etc/cni/net.d/flannel.conf file to the /opt/cni/bin/flannel binary by invoking a standard CNI operation.
-  # The default IPVLAN backend will be used if this parameter is not specified.
-  # OPTIONAL - ONE OF {ipvlan,sriov,<NAME_OF_STATIC_LEVEL_CNI_BINARY>}
+  # For any other CNI backend DANM reads their configuration from the configured CNI config directory (value of "CNI_CONF_DIR" environment variable, default is "/etc/cni/net.d")
+  # E.g. a Pod connecting to a DanmNet with "NetworkType" set to "bridge", and "NetworkID" set to "example_network" gets an interface provisioned by the <CONFIGURED_CNI_PATH_IN_KUBELET>/bridge binary based on the <CNI_CONF_DIR>/example_network.conf file
+  # The default IPVLAN backend is used when this parameter is not specified.
+  # OPTIONAL - ONE OF {ipvlan,sriov,macvlan,<NAME_OF_ANY_STATIC_LEVEL_CNI_COMPLIANT_BINARY>}
   # DEFAULT VALUE: ipvlan
   NetworkType: ## BACKEND_TYPE ##
-  # Specific dynamic configuration options can be passed to the network provisioning backends.
-  # Dynamic configuration is supported only for IPVLAN, and SRIOV backends.
-  # Other networks are always provisioned from static configuration. Options are silently ignored if NetworkType is set to a non-dynamically integrated backend.
+  # Specific extra configuration options can be passed to the network provisioning backends.
+  # Most of the parameters are only supported for level dynamic level backends, such as IPVLAN, MACVLAN, and SRIOV.
+  # Other network interfaces are always provisioned based on static CNI configuration files.
+  # The exceptional attributes are "rt_tables", "container_prefix", and "routes". DANM generally supports the features related to these parameters for all CNI backends.
   Options:
     # Name of the parent host device (i.e. physical host NIC).
     # Sub-interfaces are connected to this NIC in case NetworkType is set to IPVLAN, or MACVLAN.
-    # A Virtual Function belonging to this Physical Function is taken-up in case NetworkType is set to SRIOV.
+    # Only has an effect with dynamically integrated backends. Ignored for other NetworkTypes.
+    # Also ignored for SR-IOV, as the pre-allocated VF belonging to the configured Kubernetes Device pool is pushed into the connecting Pod's network namespace, regardless which Physical Funtion it belongs to.
     # OPTIONAL - STRING
     host_device: ## PARENT_DEVICE_NAME ##
     # Name of a network Device Plugin resource pool
     # The device_pool parameter generally represents the base resource name of the Kubernetes Devices connected to this network.
-    # Specifically for DanmNets with "NetworkType: sriov" this option is mandatory, and it represents a pool of VFs Pods connecting to this network will get one provisioned from.
+    # Specifically for DanmNets with "NetworkType: sriov" this option is mandatory, and it represents the K8s Virtual Function Device pool connecting Pods are getting their VFs from.
     # OPTIONAL - STRING
     device_pool: ## DEVICE_PLUGIN_RESOURCE_POOL_MAME ##
     # The IPv4 CIDR notation of the subnet associated with the network.
@@ -49,26 +52,30 @@ spec:
     # IPv4 allocation will be done according to the narrowed down allocation pool parameter, if defined.
     # Allocation pool must be provided together with "cidr", and shall be included in the subnet range.
     # Only has an effect with dynamically integrated backends. Ignored for other NetworkTypes.
+    # If CIDR is provided without defining an allocation pool, it is automatically calculated for the whole netmask (minus the first, and the last IP).
+    # When the network administrator sets the allocation pool, DANM assumes the non-usable IPs (e.g. broadcast IP etc.) were already discounted.
     allocation_pool:
       start: ## FIRST_ASSIGNABLE_IP ##
       end: ## LAST_ASSIGNABLE_IP ##
     # The IPv6 CIDR notation of the subnet associated with the network.
     # Pods connecting to this network will get their IPv6s from this subnet, if defined.
     # Only has an effect with dynamically integrated backends. Ignored for other NetworkTypes.
-    # OPTIONAL - IPv6 CIDR FORMAT (e.g. "2001:db8::/45")
+    # OPTIONAL - IPv6 CIDR FORMAT (e.g. "2001:db8::/45"). Netmask of the subnet cannot be higher than /64 (i.e. /65 and upwards).
     net6: ## SUBNET_CIDR ##
-    # Interfaces connected to this network will be renamed to "container_prefix" inside the container.
-    # If not provided, DANM will dynamically allocate a name for each container interface belonging to this network in Pod instantiation time.
-    # Generally supported parameter, works with all NetworkTypes.
+    # Interfaces connected to this network are renamed to "container_prefix" inside the container.
+    # If not provided, DANM dynamically allocates a name for each container interface belonging to this network in Pod instantiation time.
+    # Generally supported parameter, works with all NetworkTypes (except where the CNI backend itself is not following the CNI standard, such is the case with Flannel).
     # Do not specify container_prefix in case multiple interfaces are provisioned from the same DanmNet.
     # OPTIONAL - STRING
     container_prefix: ## INTERNAL_IF_NAME ##
-    # Policy-based IP routes belonging to this network will be installed in this routing table, if a Pod asks for them.
+    # Policy-based IP routes belonging to this network are installed into this routing table, when a user defines them in her Pod's network allocation annotation.
     # Generally supported parameter, works with all NetworkTypes.
     # OPTIONAL - INTEGER (e.g. 201)
     rt_tables: ## HOST_UNIQUE_ROUTING_TABLE_NUMBER ##
     # IPv4 routes to be installed into the default routing table of all Pods connected to this network.
     # Generally supported parameter, works with all NetworkTypes.
+    # Note: some CNI backends, like Flannel might provision IP routes into the default routing table of a Pod on their own.
+    # Beware of clashes.
     # OPTIONAL - LIST OF DESTINATION_IPV4_CIDR:IPV4_GW ENTRIES (e.g. "10.20.0.0/24: 10.0.0.1")
     routes:
       ## IP_ROUTE_1 ##
@@ -79,16 +86,17 @@ spec:
     routes6:
       ## IP_ROUTE_1 ##
       ## IP_ROUTE_2 ##
-    # If this parameter is present then traffic going through this network will be VxLAN tagged with the provided identifier
+    # When this parameter is present, traffic flowing through the connected network interfaces is VxLAN tagged with the provided virtual ID.
     # The VxLAN tag shall be unique on the level of the underlying host.
     # Management of the VxLAN interface is handled automatically by DANM. Provisioning is generally supported for all NetworkTypes.
-    # Only IPVLAN, and MACVLAN interfaces will be automatically connected to the provisioned VxLAN.
+    # Only IPVLAN, and MACVLAN interfaces are automatically connected to the provisioned VxLAN VTEP though.
+    # VLAN and VxLAN paramaters are mutually exclusive! Defining both in the same DanmNet will result in a validation error!
     # OPTIONAL - INTEGER (e.g. 50)
     vxlan: ## VXLAN_TAG ##
-    # If this parameter is present then traffic going through this network will be VLAN tagged with the provided identifier
+    # When this parameter is present, traffic flowing through the connected network interfaces is VLAN tagged with the provided identifier.
     # The VLAN ID shall be unique on the level of the underlying host.
     # Management of the VLAN interface is handled automatically by DANM. Provisioning is generally supported for all NetworkTypes.
-    # Only dynamically supported NetworkType interfaces will be automatically connected to the provisioned VLAN.
+    # Only dynamically supported NetworkType interfaces are automatically VLAN tagged though.
     # VLAN and VxLAN paramaters are mutually exclusive! Defining both in the same DanmNet will result in a validation error!
     # OPTIONAL - INTEGER (e.g. 4000)
     vlan: ## VLAN_TAG ##