Skip to content

Latest commit

 

History

History
 
 

src

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
ArmTemplates
ArmTemplates
 
 
.gitignore
.gitignore
 
 
APIM-DevOps.png
APIM-DevOps.png
 
 
FAQ.md
FAQ.md
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 

README.md

Table of Contents

  1. Running ArmTemplates Application
  2. Creator
  3. Extractor

Running ArmTemplates Application

Follow this guide for manually building and running the application

  1. git clone repository
  2. navigate to repo root
  3. build application using dotnet build src/ArmTemplates/ArmTemplates.csproj (net6.0 must be installed. Visit Download .NET 6.0 for more information)
  4. navigate to built binaries using cd .\src\ArmTemplates\bin\Debug\net6.0\
  5. run dotnet .\ArmTemplates.dll --help to view all available commands
  6. Investigate possible commands to run and choose desired

Creator

This utility creates Resource Manager templates for an API based on the OpenAPI Specification of the API. Optionally, you can provide policies you wish to apply to the API and its operations in seperate files.

Create the Config File

The utility requires one argument, --configFile, which points to a yaml file that controls the ARM templates generated by the Creator tool. The file contains a Creator Configuration object whose schema and related schemas are listed below:

Creator Configuration

Property Type Required Value
version string Yes Configuration version.
apimServiceName string Yes Name of the APIM service to deploy resources into.
policy string No Location of the global service policy XML file. Can be url or local file.
apiVersionSets Array<APIVersionSetConfiguration> No List of API Version Set configurations.
apis Array<APIConfiguration> Yes List of API configurations.
products Array<ProductConfiguration> No List of Product configurations.
namedValues Array<PropertyConfiguration> No List of Named Values
loggers Array<LoggerConfiguration> No List of Logger configurations.
authorizationServers Array<AuthorizationServerContractProperties> No List of Authorization Server configurations.
backends Array<BackendContractProperties> No List of Backend configurations.
outputLocation string Yes Local folder the utility will write templates to.
linked boolean No Determines whether the utility should create a master template that links to all generated templates.
linkedTemplatesBaseUrl string No Location that stores linked templates. Required if 'linked' is set to true.
linkedTemplatesUrlQueryString string No Query string appended to linked templates uris that enables retrieval from private storage.
tags Array<TagConfiguration> No List of Tags configurations.
baseFileName string No base file name for the templates file
serviceUrlParameters Array<ServiceUrlProperty> No List of parameterized ServiceUrl.

APIConfiguration

Property Type Required Value
name string Yes API identifier. Must be unique in the current API Management service instance.
displayName string No User-friendly name for the API.
description string No Description of the API.
serviceUrl string No Absolute URL of the backend service implementing this API.
type enum No Type of API. - http or soap
openApiSpec string Yes Location of the Open API Spec file. Can be url or local file.
openApiSpecFormat string No Format of the API definition. When the openApiSpec property refers to a local file, the program will infer the format if this property is omitted. If the openApiSpec property refers to a url, you can prevent downloading the API definition by specifying this property. Valid values are Swagger (JSON), Swagger_Json, OpenApi20 (YAML), OpenApi20_Yaml, OpenApi20_Json, OpenApi30 (YAML), OpenApi30_Yaml, or OpenApi30_Json.
policy string No Location of the API policy XML file. Can be url or local file.
suffix string Yes Relative URL uniquely identifying this API and all of its resource paths within the API Management service instance. It is appended to the API endpoint base URL specified during the service instance creation to form a public URL for this API.
subscriptionRequired boolean No Specifies whether an API or Product subscription is required for accessing the API.
subscriptionKeyParameterNames APITemplateSubscriptionKeyParameterNames No subscription key parameter name.
isCurrent boolean No Indicates if API revision is current api revision.
apiVersion string No Indicates the Version identifier of the API if the API is versioned.
apiVersionDescription string No Description of the API Version.
apiRevision string No Describes the Revision of the API. If no value is provided, default revision 1 is created.
apiRevisionDescription string No Description of the Api Revision.
apiVersionSetId string No A resource identifier for the related ApiVersionSet. Value must match the resource id on an existing version set and is irrelevant if the apiVersionSet property is supplied.
operations Dictionary<string, APIOperationPolicyConfiguration> No XML policies that will be applied to operations within the API. Keys must match the operationId property of one of the API's operations.
authenticationSettings AuthenticationSettingsContract No Collection of authentication settings included into this API.
products string No Comma separated list of existing products to associate the API with.
protocols string No Comma separated list of protocols used between client and APIM service.
diagnostic APIDiagnosticConfiguration No Diagnostic configuration.
tags string No Comma separated list of tags to associate the API with. Tags can be existing or nonexisting. For nonexisting tags, it will automatically generate new tags on the API instance

APIOperationPolicyConfiguration

Property Type Required Value
policy string Yes Location of the operation policy XML file. Can be url or local file.

APIDiagnosticConfiguration

Property Type Required Value
name enum No Name of API Diagnostic - azureEventHub or applicationInsights

Additional properties found in DiagnosticContractProperties

APIVersionSetConfiguration

Property Type Required Value
id string No ID of the API Version Set.

Additional properties found in ApiVersionSetContractProperties

ProductConfiguration

Property Type Required Value
name string No Name of the product resource. If omitted, the display name is used.
policy string No Location of the Product policy XML file. Can be url or local file.
subscriptions Array<SubscriptionConfiguration> No List of Subscriptions

Additional properties found in ProductContractProperties

SubscriptionConfiguration

Property Type Required Value
name string No Name of the subscription resource. If omitted, the display name is used.

Additional properties found in ProductContractProperties

PropertyConfiguration

Property Type Required Value
tags array No Optional tags that when provided can be used to filter the property list. - string
secret boolean No Determines whether the value is a secret and should be encrypted or not. Default value is false.
displayName string Yes Unique name of Property. It may contain only letters, digits, period, dash, and underscore characters.
value string No Value of the property. Can contain policy expressions. It can be empty or consist only of whitespace only if the keyvault parameter is set.
keyvault PropertyKeyVaultConfiguration No The keyvault settings for the property.

Additional properties found in PropertyContractProperties

PropertyKeyVaultConfiguration

Property Type Required Value
secretIdentifier string Yes KeyVault secret id which will map to the property.

LoggerConfiguration

Property Type Required Value
name string Yes Name of the Logger

Additional properties found in LoggerContractProperties

TagConfiguration

Property Type Required Value
displayName string Yes DisplayName and name of the tag

Additional properties found in TagContractProperties

APITemplateSubscriptionKeyParameterNames

Property Type Required Value
header string Yes header name of the subscription.
query string Yes query parameter name of the subscription.

Additional properties found in APITemplateSubscriptionKeyParameterNames

ServiceUrlProperty

Property Type Required Value
apiName string Yes Name of API.
serviceUrl string Yes API ServiceUrl parameter.

Sanitizing tags

While creating config file, please keep in mind that the tag resource name will be generated from sanitized tag's display name.

For example config:

tags: 
 - displayName: tag 1
 - displayName: tag 2

Will result two tag resources:

Tag 1:
  resource name = tag-1
  display name = tag 1

Tag 2:
  resource name = tag-2
  display name = tag 2

In case there are two different tags which result the same sanitized tag's name program will throw an exception with conflicting tags.

Sample Config File

The following is a full config.yml file with each property listed:

version: 0.0.1
apimServiceName: myAPIMService
policy: C:\Users\myUsername\Projects\azure-api-management-devops-example\src\ArmTemplates\Creator\ExampleFiles\XMLPolicies\globalServicePolicy.xml
apiVersionSets:
    - id: myAPIVersionSetID
      displayName: swaggerPetstoreVersionSetLinked
      description: a description
      versioningScheme: Query
      versionQueryName: versionQuery
      versionHeaderName: versionHeader
    - id: secondAPIVersionSetID
      displayName: secondSet
      description: another description
      versioningScheme: Header
      versionQueryName: versionQuery
      versionHeaderName: versionHeader
apis:
    - name: myAPI
      type: http
      displayName: My API
      description: myFirstAPI
      serviceUrl: http://myApiBackendUrl.com
      openApiSpec: C:\Users\myUsername\Projects\azure-api-management-devops-example\src\ArmTemplates\Creator\ExampleFiles\OpenApiSpecs\swaggerPetstore.json
      openApiSpecFormat: swagger
      policy: C:\Users\myUsername\Projects\azure-api-management-devops-example\src\ArmTemplates\Creator\ExampleFiles\XMLPolicies\apiPolicyHeaders.xml
      suffix: conf
      subscriptionRequired: true
      isCurrent: true
      apiVersion: v1
      apiVersionDescription: My first version
      apiVersionSetId: myAPIVersionSetID
      apiRevision: 1
      apiRevisionDescription: My first revision 
      products: myProduct   
      tags: Universe, myTag
      operations:
        addPet:
          policy: C:\Users\myUsername\Projects\azure-api-management-devops-example\src\ArmTemplates\Creator\ExampleFiles\XMLPolicies\operationRateLimit.xml
        deletePet:
          policy: C:\Users\myUsername\Projects\azure-api-management-devops-example\src\ArmTemplates\Creator\ExampleFiles\XMLPolicies\operationRateLimit.xml
      products: starter, platinum
      authenticationSettings:
        oAuth2:
            authorizationServerId: myAuthServer
            scope: myScope
      diagnostic:
        name: applicationinsights
        alwaysLog: allErrors
        loggerId: myAppInsights
        sampling:
          samplingType: fixed
          percentage: 50
        frontend: 
          request:
            headers:
            body: 
              bytes: 512
          response: 
            headers:
            body: 
              bytes: 512
        backend: 
          request:
            headers:
            body: 
              bytes: 512
          response: 
            headers:
            body: 
              bytes: 512
        enableHttpCorrelationHeaders: true
products:
    - name: platinum
      displayName: Platinum
      description: a test product
      terms: some terms
      subscriptionRequired: true
      approvalRequired: true
      subscriptionsLimit: 1
      state: notPublished
      policy: C:\Users\myUsername\Projects\azure-api-management-devops-example\src\ArmTemplates\Creator\ExampleFiles\XMLPolicies\productSetBodyBasic.xml
      subscriptions:
          - name: platinum
            primaryKey: a240691f-03fd-4557-a5cb-6e0f65cd976a
            secondaryKey: 032338aa-0076-4379-910c-32ddd42f38a1
            state: active
            allowTracing: true 
tags:
    - displayName: Universe
loggers:
    - name: myAppInsights
      loggerType: applicationInsights
      description: a test app insights
      credentials:
        instrumentationKey: 45d4v88-fdfs-4b35-9232-731d82d4d1c6
      isBuffered: true
authorizationServers:
    - displayName: myAuthServer
      description: test server
      clientRegistrationEndpoint: https://www.contoso.com/apps
      authorizationEndpoint: https://www.contoso.com/oauth2/auth
      authorizationMethods:
        - GET
      tokenEndpoint: https://www.contoso.com/oauth2/token
      supportState: true
      defaultScope: read write
      grantTypes:
        - authorizationCode
        - implicit
      bearerTokenSendingMethods:
        - authorizationHeader
      clientId: 1
      clientSecret: 2
      resourceOwnerUsername: un
      resourceOwnerPassword: pwd
backends:
    - title: myBackend
      description: description5308
      url: https://backendname2644/
      protocol: http
      credentials:
        query: 
          sv: 
            - xx
            - bb
        header: 
          x-my-1:
            - val1
            - val2
        authorization: 
          scheme: Basic
          parameter: opensesma
      proxy:
        url: http://192.168.1.1:8080
        username: Contoso\admin
        password: opensesame
      tls:
        validateCertificateChain: false
        validateCertificateName: false
outputLocation: C:\Users\myUsername\GeneratedTemplates
linked: false
linkedTemplatesBaseUrl : https://mystorageaccount.blob.core.windows.net/mycontainer
linkedTemplatesUrlQueryString : ?sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2019-12-22T23:12:53Z&st=2019-09-09T14:12:53Z&spr=https&sig=uFTldJEYPH888QVzKb7q7eLq0Xt%2Bu35UTqpFGUYo6uc%3D
baseFileName: baseName
serviceUrlParameters: 
  - apiName: myAPI
    serviceUrl: httpbin.com/myAPI

Running the Creator

Below are the steps to run the Creator from the source code:

  • Clone this repository and restore its packages using dotnet restore
  • Navigate to {repo root}/src/ARMTemplates directory
  • Run the following command: dotnet run create --configFile CONFIG_YAML_FILE_LOCATION
  • Run the following command to pass apim Name as a parameter: dotnet run create --configFile CONFIG_YAML_FILE_LOCATION --apimNameValue apimname1
  • Run the following command to pass api name to generate ARM templates only for this specified APIs semicolon separated where the api1 (api name) will be same as name used in valid.yml file for that api(here it is myBackend): dotnet run create --configFile CONFIG_YAML_FILE_LOCATION --preferredAPIsForDeployment myBackend;api2;api3;
  • Run the following command to pass BackendUrls as an json input file into the parameter(sample file available in the same path as below in this repository): dotnet run create --configFile CONFIG_YAML_FILE_LOCATION --backendurlconfigFile .\apimtemplate\Creator\ExampleFiles\BackendUrlParameter\BackendUrlParameters.json
  • Run the following command to pass AppinsightsName and Appinsights InstrumentationKey as an parameter: dotnet run create --configFile CONFIG_YAML_FILE_LOCATION --appInsightsInstrumentationKey 45d4v88-fdfs-4b35-9232-731d82d4d1c6 --appInsightsName myAppInsights
  • Run the following command to pass namedValueKeys as an parameter to provide environment specific named values with key name and value like following:
  • Add new key to namedValues section in valid yaml file then use the same key name here in this cli parameter Here namedvalue displayname can not have | or ; in their name. dotnet run create --configFile CONFIG_YAML_FILE_LOCATION --namedValues namedvalue1|namevaluevalue1;namedvalue2|namevaluevalue2 Add new key to namedValues section in valid yaml file then use the same key name here in this cli parameter Here displayName can not have | or ; in their name.
  • Run the following command dotnet run create --configFile CONFIG_YAML_FILE_LOCATION --parameterizeNamedValuesAndSecrets " " to parameterize namedValues and KeyVault Secrets and allow to use single ARM template for multiple environments

You can also run it directly from the releases.

Additionaly, the Creator can also be made available as a global dotnet CLI tool in your Azure DevOps artifacts or private NuGet repository. Build the Creator, and run the following commands to package the Creator as a dotnet tool:

dotnet pack -c Release
dotnet tool install -g --add-source .\bin\Release ArmTemplates

The Creator tool is now available anywhere on the command-line:

apim-templates create --configFile CONFIG_YAML_FILE_LOCATION

Extractor

This utility generates Resource Manager templates by extracting existing configurations of one or more APIs in an API Management instance.

Prerequisite

To be able to run the Extractor, you would first need to install the Azure CLI.

Running the Extractor

Below are the steps to run the Extractor from the source code:

  • Clone this repository and navigate to {repo root}/src/ARMTemplates
  • Restore its packages using dotnet restore
  • Make sure you have signed in using Azure CLI and have switched to the subscription containing the API Management instance from which the configurations will be extracted.
az login
az account set --subscription <subscription_id>

Extractor Arguments

You have two choices when specifying your settings:

  1. By using a json file with key-values where the keys matches the table below. Use the extractorConfig argument: extract --extractorConfig c:/temp/extractSettings.json. See more examples.
  2. Pass the arguments on the command line. For instance extract --sourceApimName my-feature-apim --destinationApimName company-stable-apim --resourceGroup my-feature-rg --fileFolder c:\temp\apim-extract --apiName MyFeatureV1Api.
Property Required Value
sourceApimName Yes Name of the source APIM instance.
destinationApimName Yes Name of the destination APIM instance.
resourceGroup Yes Name of the resource group.
fileFolder Yes Path to output folder
apiName No Name of API. If provided, Extractor executes single API extraction. Otherwise, Extractor executes full extraction. Note: This is the "Name" value as seen in the API settings, not "Display Name" and is case sensitive.
linkedTemplatesBaseUrl No Linked templates remote location. If provided, Extractor generates master template and requires linked templates pushed to remote location.
linkedTemplatesUrlQueryString No Query string appended to linked templates uris that enables retrieval from private storage.
linkedTemplatesSasToken No String appended to end of the linked templates uris that enables adding a SAS token or other query parameters.
policyXMLBaseUrl No Policy XML files remote location. If provided, Extractor generates policies folder with xml files, and requires they be pushed to remote location.
splitAPIs No If set to "true", then generate multiple api folders, each api will have a seperate folder, with a separate master template to deploy this api. If this single api has a version set, then a version set folder will generate instead, then all apis that belongs to this version set will be included in the version set folder, apis in this version set can be deployed separately using every api's master template, or they can be deployed together using the master template in "VersionSetMasterFolder" folder
apiVersionSetName No Name of the APIVersionSet. If provided, extract all apis within this apiversionset. It will generate seperate folder for each api and also a master folder to link all apis in this apiversionset
multipleAPIs No Specify multiple APIs to extract. Generate templates for each API, also generate an aggregated templates folder to deploy these APIs together at a time
includeAllRevisions No Set to "true" will extract all revisions for the single API. Will work only with "apiName" paramter, where you specify which API's revisions to extract. Generate templates for each revision, also generate an aggregated master folder to deploy these revisions together at one time. Note: there are many complicated issues with deploying revisions, make sure your deployment won't overwrite or break the existing ones
baseFileName No Specify base file name of the template files
policyXMLSasToken No Specify sasToken for fetching policy files
linkedTemplatesSasToken No Specify sasToken for fetching linkedTemplate files
paramServiceUrl No Set to "true" will parameterize all serviceUrl for each api and generate serviceUrl parameter to api template/parameter template/master template files
paramNamedValue No Set to "true" will parameterize all named values and add named values parameter to property template/parameter template/mastert template files
paramApiLoggerId No Set to "true" will parameterize all logger ids in all apis (within api templates), Also includes the "All API" monitoring configuration
paramLogResourceId No Set to "true" will parameterize all loggers' resource ids (within logger template)
serviceBaseUrl No Specify the base url where you want to run your extractor
notIncludeNamedValue No Set to "true" will not generate Named Value Templates
paramNamedValuesKeyVaultSecrets No Set to true will parameterize all named values where the value is from a key vault secret
paramBackend No Set to true will parameterize specific backend values (limited to resourceId, url and protocol)
extractGateways No Set to true will attempt to extract the Self Hosted Gateways.
overrideGroupGuids No Set to true will override the group id in output template in case it does not match with system predefined values.
overrideProductGuids No Set to true will override the product id in output template in case it does not match with system predefined values.
paramApiOauth2Scope No Set to true will parametrize the scope values for APIs in which User authorization setting set to OAuth 2.0.
apiParameters No Parameterize api parameters (Oauth2 Scope/Service Url) values for APIs in advance.
exctractSecrets No By default false. If set to "true" secrets will be extracted as well and parameters templated will be supplied with actual secret values. Currently applies to identityProvider service.
extractIdentityProviders No By default false. Set to true will attempt to extract the identity providers from service.
parametersOutputDirectoryName No If set will redefine the output folder name for parameters. By default will be equal to {service-name}-parameters
excludeBuildInGroups No By default false. Set to true will exclude builtIn groups from groupsTemplate.

Note

  • Can not use "splitAPIs" and "apiName" at the same time, since using "apiName" only extract one API
  • Can not use "apiName" and "multipleAPIs" at the same time
  • Can only "includeAllRevisions" with "apiName"

Extractor Parameter File Example

Executing a single API extraction with linked templates and policy file generation, use the following parameters:

{
    "sourceApimName": "<source-apim-name>",
    "destinationApimName": "<destination-apim-name>",
    "resourceGroup": "<resource-group>",
    "fileFolder": "<destination-file-folder>",
    "apiName": "<api_name>",
    "linkedTemplatesBaseUrl": "<linked_templates_remote_location>",
    "policyXMLBaseUrl": "<policies_remote_location>"
}

Extract all APIs with linked templates linking all apis and policy file, use the following parameters:

{
    "sourceApimName": "<source-apim-name>",
    "destinationApimName": "<destination-apim-name>",
    "resourceGroup": "<resource-group>",
    "fileFolder": "<destination-file-folder>",
    "linkedTemplatesBaseUrl": "<linked_templates_remote_location>",
    "policyXMLBaseUrl": "<policies_remote_location>"
}

Extract all APIs with seperated api folders, use the following parameters:

{
    "sourceApimName": "<source-apim-name>",
    "destinationApimName": "<destination-apim-name>",
    "resourceGroup": "<resource-group>",
    "fileFolder": "<destination-file-folder>",
    "linkedTemplatesBaseUrl": "<linked_templates_remote_location>",
    "policyXMLBaseUrl": "<policies_remote_location>",
    "splitAPIs": "true"
}

Extract all APIs within an apiversionset, use the following parameters:

{
    "sourceApimName": "<source-apim-name>",
    "destinationApimName": "<destination-apim-name>",
    "resourceGroup": "<resource-group>",
    "fileFolder": "<destination-file-folder>",
    "linkedTemplatesBaseUrl": "<linked_templates_remote_location>",
    "policyXMLBaseUrl": "<policies_remote_location>",
    "apiVersionSetName": "<api-version-set-name>"
}

Extract single API with all revisions, use the following parameters:

{
    "sourceApimName": "<source-apim-name>",
    "destinationApimName": "<destination-apim-name>",
    "resourceGroup": "<resource-group>",
    "fileFolder": "<destination-file-folder>",
    "linkedTemplatesBaseUrl": "<linked_templates_remote_location>",
    "policyXMLBaseUrl": "<policies_remote_location>",
    "apiName": "<api_name>",
    "includeAllRevisions": "true"
}

Extract multiple APIs, use the following parameters:

{
    "sourceApimName": "<source-apim-name>",
    "destinationApimName": "<destination-apim-name>",
    "resourceGroup": "<resource-group>",
    "fileFolder": "<destination-file-folder>",
    "linkedTemplatesBaseUrl": "<linked_templates_remote_location>",
    "policyXMLBaseUrl": "<policies_remote_location>",
    "multipleAPIs": "api1, api2, api3"
}

Extract single API with baseFileName, use the following parameters:

{
    "sourceApimName": "<source-apim-name>",
    "destinationApimName": "<destination-apim-name>",
    "resourceGroup": "<resource-group>",
    "fileFolder": "<destination-file-folder>",
    "linkedTemplatesBaseUrl": "<linked_templates_remote_location>",
    "policyXMLBaseUrl": "<policies_remote_location>",
    "apiName": "<api_name>",
    "baseFileName": "<base_file_name>"
}

Extract all APIs within paramServiceUrl, use the following parameters:

{
    "sourceApimName": "<source-apim-name>",
    "destinationApimName": "<destination-apim-name>",
    "resourceGroup": "<resource-group>",
    "fileFolder": "<destination-file-folder>",
    "linkedTemplatesBaseUrl": "<linked_templates_remote_location>",
    "policyXMLBaseUrl": "<policies_remote_location>",
    "paramServiceUrl": "true"
}

Extract all APIs within paramNamedValue, use the following parameters:

{
    "sourceApimName": "<source-apim-name>",
    "destinationApimName": "<destination-apim-name>",
    "resourceGroup": "<resource-group>",
    "fileFolder": "<destination-file-folder>",
    "linkedTemplatesBaseUrl": "<linked_templates_remote_location>",
    "policyXMLBaseUrl": "<policies_remote_location>",
    "paramNamedValue": "true"
}

Extract all APIs with predefined values for Api parameters, using the following Extractor configuration:

{
    "sourceApimName": "<source-apim-name>",
    "destinationApimName": "<destination-apim-name>",
    "resourceGroup": "<resource-group>",
    "fileFolder": "<destination-file-folder>",
    "linkedTemplatesBaseUrl": "<linked_templates_remote_location>",
    "apiParameters": {
      "api-name1": {
        "oauth2Scope": "scope-value-1",
        "serviceUrl": "service-url-1"
      },
      "api-name2": {
        "oauth2Scope": "scope-value-2",
        "serviceUrl": "service-url-2"
      }
    }
}

Run the extractor

dotnet run extract

You can also run it directly from the releases.

Likewise, if you package the Extractor as a dotnet CLI tool, you can run it from anywhere on the command-line:

apim-templates extract