Skip to content

Latest commit

 

History

History
 
 

docs

Pandora Overview

Prior to reading this, we recommend reading the top-level README file which contains a summary of the project.

High Level Diagram

graph LR
  subgraph Import
    APIConfig[/"config/resource-manager.hcl"/]--->Importer
    ResouceConfig[/"config/resource/some_resource_config.hcl"/]--->Importer
  end
  Importer--->DataAPI
  subgraph Generate
    DataAPI--->SDKGenerator
    DataAPI--->TerraformGenerator
  end
  SDKGenerator--->github.com/hashicorp/go-azure-sdk
  TerraformGenerator--->github.com/hashicorp/terraform-provider-azurerm
Loading

More details can be found in the main README under "how does this work".

Overview

Pandora is a suite of tools for converting the Azure API Definitions (from Azure/azure-rest-api-specs) into both a Go SDK (output as hashicorp/go-azure-sdk) and Terraform Resources within the AzureRM Provider.

Each tool is described in the main README file - however each is involved in the process and is primarily intended to be run in automation (via GitHub Actions) however can also be run locally.

Specific information on each tool can be found in the README for each tool, for example the importer-rest-api-specs tool.

Guides

Resource ID's as a primary concept

Resource ID's within Azure are unique URIs defining a resource, such as a Resource Group or Virtual Machine which have a known format.

Since Resource ID's are both integral to the Azure API and Terraform, Pandora leans on these as a primary concept - for example exposing a Struct for each Resource ID in the Go SDK (example) which is then used as an argument for each HTTP Operation (example).

When identifying candidate Terraform Resources as a part of the import process, Pandora's importer-rest-api-specs tool looks for Create (PUT), Read (GET) and Delete (DELETE) HTTP Operations against a given Resource ID (and also an Update [PATCH] if available).

As such Resource IDs are treated as a first class concept across Pandora, and (as described in the next section) are normalized where required for consistency.

How can I import and generate a Go SDK for a given Service/API Version?

See this document on how to import a new Service/API Version into Pandora's Data Format - once a PR has been sent/merged this is automatically released into hashicorp/go-azure-sdk.

How can I generate a new Resource?

First, the Service and API Version being used must be imported and the Go SDK generated into hashicorp/go-azure-sdk (see the section above).

Once that's available, this guide explains how to add a new resource and some known limitations at this point in time.

How does this differ from autorest?

Autorest is a toolchain that provides a broad spectrum of RESTful SDKs for various languages (see here for details). The Autorest process involves a relatively complicated transformation process that aims to provide SDKs that have the same "look and feel" between languages and often collects together multiple API versions into a single SDK version, meaning that resources in a single SDK version may use differing API versions in their RESTful calls. This approach has obvious upsides for developers, but also means that specific use cases of SDK usage may not be ideal.

Pandora also consumes the OpenAPI data (aka Swagger), however, it does not perform this collection of disparate versions, mandating instead to be explicit in the API version in use at all times. Pandora's first-class citizen for SDK generation is Terraform, and the only output language is Go. This difference means that the resultant generated SDKs are compatible with Terraform's requirements, and the generation process can, and will, be kept in-line with that compatibilty over time.

Data Normalization

When Pandora's rest-api-specs-importer tool runs, data from the Azure API Definitions is first parsed and then run through a normalization process, prior to being output in the data format used by the Data API Definitions.

The normalization applies both to Resource ID Segments (which allows us to workaround where a URI maybe defined twice in the Azure API Definitions with different casings) and the Fields themselves (turning these from types which are unique per Service/API to "common schema", with common behaviours).

This approach allows the generated Resources to be behaviourally consistent with existing Resources within the AzureRM Provider (for example with identity block being exposed the same way regardless of the API behaviour which differs per API - or the location field which always gets normalized).

Use of Meta Clients

The Go SDK generated by this repository outputs a Meta Client, which contains a reference to all of the SDK Clients available for a given API Version for a given Service.

Rather than defining an SDK Client for each Service, the Terraform Resources generated by Pandora's Terraform Generator output make use of these Meta Clients for each Service.

For entirely generated Services, this means that an SDK Client will be generated and configured automatically - however when adding generated Resources to an existing Service, some minor refactoring must be done to prepare for the generated SDK Client.

These changes can be found in this Pull Request - however a more detailed/step-by-step guide will follow here in the future.

Known Limitations

Some SDKs and resources cannot, unfortunately, currently be generated. Reasons include, but are not necessarily limited to:

  • Data fidelity and one-to-many APIs - Some API specs do not contain enough information to accurately determine how a resource is intended to be used. In some cases, the design of an Azure service is such a single API can be used to create multiple types of resources. For example, the same API is used to create Web Apps, Function Apps, and Logic Apps, but in each case the data required in the request payloads is markedly different and there is no way to determine the correct combinations from the data itself. Given the product familiarity required for these kinds of resources, they are unlikely to be able to be generated unless the APIs are redesigned / reimplemented to be explicit in their function.

  • Data Errors - Some API Specifications contain errors or breaches of compatibility with the ARM specification. For example, inconsistencies in Resource ID segments used throughout an API version which can result in diffs in subsequent SDK generations. These can typically be addressed by submitting fixes back to the source data at https://github.com/Azure/azure-rest-api-specs/issues. PR checks exist to attempt to detect when this happens and report it.

  • Data Sources - Generation of Terraform Data Sources is not currently supported.