-
Notifications
You must be signed in to change notification settings - Fork 40.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #28821 from colemickens/azure-cloudprovider-pr
Automatic merge from submit-queue Add an Azure CloudProvider Implementation This PR adds `Azure` as a cloudprovider provider for Kubernetes. It specifically adds support for native pod networking (via Azure User Defined Routes) and L4 Load Balancing (via Azure Load Balancers). I did have to add `clusterName` as a parameter to the `LoadBalancers` methods. This is because Azure only allows one "LoadBalancer" object per set of backend machines. This means a single "LoadBalancer" object must be shared across the cluster. The "LoadBalancer" is named via the `cluster-name` parameter passed to `kube-controller-manager` so as to enable multiple clusters per resource group if the user desires such a configuration. There are few things that I'm a bit unsure about: 1. The implementation of the `Instances` interface. It's not extensively documented, it's not really clear what the different functions are used for, and my questions on the ML didn't get an answer. 2. Counter to the comments on the `LoadBalancers` Interface, I modify the `api.Service` object in `EnsureLoadBalancerDeleted`, but not with the intention of affecting Kube's view of the Service. I simply do it so that I can remove the `Port`s on the `Service` object and then re-use my reconciliation logic that can handle removing stale/deleted Ports. 3. The logging is a bit verbose. I'm looking for guidance on the appropriate log level to use for the chattier bits. Due to the (current) lack of Instance Metadata Service and lack of Virtual Machine Identity in Azure, the user is required to do a few things to opt-in to this provider. These things are called-out as they are in contrast to AWS/GCE: 1. The user must provision an Azure Active Directory ServicePrincipal with `Contributor` level access to the resource group that the cluster is deployed in. This creation process is documented [by Hashicorp](https://www.packer.io/docs/builders/azure-setup.html) or [on the MSDN Blog](https://blogs.msdn.microsoft.com/arsen/2016/05/11/how-to-create-and-test-azure-service-principal-using-azure-cli/). 2. The user must place a JSON file somewhere on each Node that conforms to the `AzureConfig` struct defined in `azure.go`. (This is automatically done in the Azure flavor of [Kubernetes-Anywhere](https://github.com/kubernetes/kubernetes-anywhere).) 3. The user must specify `--cloud-config=/path/to/azure.json` as an option to `kube-apiserver` and `kube-controller-manager` similarly to how the user would need to pass `--cloud-provider=azure`. I've been running approximately this code for a month and a half. I only encountered one bug which has since been fixed and covered by a unit test. I've just deployed a new cluster (and a Type=LoadBalancer nginx Service) using this code (via `kubernetes-anywhere`) and have posted [the `kube-controller-manager` logs](https://gist.github.com/colemickens/1bf6a26e7ef9484a72a30b1fcf9fc3cb) for anyone who is interested in seeing the logs of the logic. If you're interested in this PR, you can use the instructions in my [`azure-kubernetes-demo` repository](https://github.com/colemickens/azure-kubernetes-demo) to deploy a cluster with minimal effort via [`kubernetes-anywhere`](https://github.com/kubernetes/kubernetes-anywhere). (There is currently [a pending PR in `kubernetes-anywhere` that is needed](kubernetes-retired/kubernetes-anywhere#172) in conjuncture with this PR). I also have a pre-built `hyperkube` image: `docker.io/colemickens/hyperkube-amd64:v1.4.0-alpha.0-azure`, which will be kept in sync with the branch this PR stems from. I'm hoping this can land in the Kubernetes 1.4 timeframe. CC (potential code reviewers from Azure): @ahmetalpbalkan @brendandixon @paulmey CC (other interested Azure folk): @brendandburns @johngossman @anandramakrishna @jmspring @jimzim CC (others who've expressed interest): @codefx9 @edevil @thockin @rootfs
- Loading branch information
Showing
75 changed files
with
21,494 additions
and
56 deletions.
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| assignees: | ||
| - colemickens | ||
| - brendandburns |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,173 @@ | ||
| /* | ||
| Copyright 2016 The Kubernetes Authors. | ||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| */ | ||
|
|
||
| package azure | ||
|
|
||
| import ( | ||
| "io" | ||
| "io/ioutil" | ||
|
|
||
| "k8s.io/kubernetes/pkg/cloudprovider" | ||
|
|
||
| "github.com/Azure/azure-sdk-for-go/arm/compute" | ||
| "github.com/Azure/azure-sdk-for-go/arm/network" | ||
| "github.com/Azure/go-autorest/autorest/azure" | ||
| "github.com/ghodss/yaml" | ||
| ) | ||
|
|
||
| // CloudProviderName is the value used for the --cloud-provider flag | ||
| const CloudProviderName = "azure" | ||
|
|
||
| // Config holds the configuration parsed from the --cloud-config flag | ||
| type Config struct { | ||
| Cloud string `json:"cloud" yaml:"cloud"` | ||
| TenantID string `json:"tenantId" yaml:"tenantId"` | ||
| SubscriptionID string `json:"subscriptionId" yaml:"subscriptionId"` | ||
| ResourceGroup string `json:"resourceGroup" yaml:"resourceGroup"` | ||
| Location string `json:"location" yaml:"location"` | ||
| VnetName string `json:"vnetName" yaml:"vnetName"` | ||
| SubnetName string `json:"subnetName" yaml:"subnetName"` | ||
| SecurityGroupName string `json:"securityGroupName" yaml:"securityGroupName"` | ||
| RouteTableName string `json:"routeTableName" yaml:"routeTableName"` | ||
|
|
||
| AADClientID string `json:"aadClientId" yaml:"aadClientId"` | ||
| AADClientSecret string `json:"aadClientSecret" yaml:"aadClientSecret"` | ||
| AADTenantID string `json:"aadTenantId" yaml:"aadTenantId"` | ||
| } | ||
|
|
||
| // Cloud holds the config and clients | ||
| type Cloud struct { | ||
| Config | ||
| Environment azure.Environment | ||
| RoutesClient network.RoutesClient | ||
| SubnetsClient network.SubnetsClient | ||
| InterfacesClient network.InterfacesClient | ||
| RouteTablesClient network.RouteTablesClient | ||
| LoadBalancerClient network.LoadBalancersClient | ||
| PublicIPAddressesClient network.PublicIPAddressesClient | ||
| SecurityGroupsClient network.SecurityGroupsClient | ||
| VirtualMachinesClient compute.VirtualMachinesClient | ||
| } | ||
|
|
||
| func init() { | ||
| cloudprovider.RegisterCloudProvider(CloudProviderName, NewCloud) | ||
| } | ||
|
|
||
| // NewCloud returns a Cloud with initialized clients | ||
| func NewCloud(configReader io.Reader) (cloudprovider.Interface, error) { | ||
| var az Cloud | ||
|
|
||
| configContents, err := ioutil.ReadAll(configReader) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| err = yaml.Unmarshal(configContents, &az) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| if az.Cloud == "" { | ||
| az.Environment = azure.PublicCloud | ||
| } else { | ||
| az.Environment, err = azure.EnvironmentFromName(az.Cloud) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| } | ||
|
|
||
| oauthConfig, err := az.Environment.OAuthConfigForTenant(az.TenantID) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| servicePrincipalToken, err := azure.NewServicePrincipalToken( | ||
| *oauthConfig, | ||
| az.AADClientID, | ||
| az.AADClientSecret, | ||
| az.Environment.ServiceManagementEndpoint) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| az.SubnetsClient = network.NewSubnetsClient(az.SubscriptionID) | ||
| az.SubnetsClient.BaseURI = az.Environment.ResourceManagerEndpoint | ||
| az.SubnetsClient.Authorizer = servicePrincipalToken | ||
|
|
||
| az.RouteTablesClient = network.NewRouteTablesClient(az.SubscriptionID) | ||
| az.RouteTablesClient.BaseURI = az.Environment.ResourceManagerEndpoint | ||
| az.RouteTablesClient.Authorizer = servicePrincipalToken | ||
|
|
||
| az.RoutesClient = network.NewRoutesClient(az.SubscriptionID) | ||
| az.RoutesClient.BaseURI = az.Environment.ResourceManagerEndpoint | ||
| az.RoutesClient.Authorizer = servicePrincipalToken | ||
|
|
||
| az.InterfacesClient = network.NewInterfacesClient(az.SubscriptionID) | ||
| az.InterfacesClient.BaseURI = az.Environment.ResourceManagerEndpoint | ||
| az.InterfacesClient.Authorizer = servicePrincipalToken | ||
|
|
||
| az.LoadBalancerClient = network.NewLoadBalancersClient(az.SubscriptionID) | ||
| az.LoadBalancerClient.BaseURI = az.Environment.ResourceManagerEndpoint | ||
| az.LoadBalancerClient.Authorizer = servicePrincipalToken | ||
|
|
||
| az.VirtualMachinesClient = compute.NewVirtualMachinesClient(az.SubscriptionID) | ||
| az.VirtualMachinesClient.BaseURI = az.Environment.ResourceManagerEndpoint | ||
| az.VirtualMachinesClient.Authorizer = servicePrincipalToken | ||
|
|
||
| az.PublicIPAddressesClient = network.NewPublicIPAddressesClient(az.SubscriptionID) | ||
| az.PublicIPAddressesClient.BaseURI = az.Environment.ResourceManagerEndpoint | ||
| az.PublicIPAddressesClient.Authorizer = servicePrincipalToken | ||
|
|
||
| az.SecurityGroupsClient = network.NewSecurityGroupsClient(az.SubscriptionID) | ||
| az.SecurityGroupsClient.BaseURI = az.Environment.ResourceManagerEndpoint | ||
| az.SecurityGroupsClient.Authorizer = servicePrincipalToken | ||
|
|
||
| return &az, nil | ||
| } | ||
|
|
||
| // LoadBalancer returns a balancer interface. Also returns true if the interface is supported, false otherwise. | ||
| func (az *Cloud) LoadBalancer() (cloudprovider.LoadBalancer, bool) { | ||
| return az, true | ||
| } | ||
|
|
||
| // Instances returns an instances interface. Also returns true if the interface is supported, false otherwise. | ||
| func (az *Cloud) Instances() (cloudprovider.Instances, bool) { | ||
| return az, true | ||
| } | ||
|
|
||
| // Zones returns a zones interface. Also returns true if the interface is supported, false otherwise. | ||
| func (az *Cloud) Zones() (cloudprovider.Zones, bool) { | ||
| return az, true | ||
| } | ||
|
|
||
| // Clusters returns a clusters interface. Also returns true if the interface is supported, false otherwise. | ||
| func (az *Cloud) Clusters() (cloudprovider.Clusters, bool) { | ||
| return nil, false | ||
| } | ||
|
|
||
| // Routes returns a routes interface along with whether the interface is supported. | ||
| func (az *Cloud) Routes() (cloudprovider.Routes, bool) { | ||
| return az, true | ||
| } | ||
|
|
||
| // ScrubDNS provides an opportunity for cloud-provider-specific code to process DNS settings for pods. | ||
| func (az *Cloud) ScrubDNS(nameservers, searches []string) (nsOut, srchOut []string) { | ||
| return nameservers, searches | ||
| } | ||
|
|
||
| // ProviderName returns the cloud provider ID. | ||
| func (az *Cloud) ProviderName() string { | ||
| return CloudProviderName | ||
| } |
Oops, something went wrong.