Getting Module Source Code

Given the following main.tf source:

provider "aws" {

}

module "iam_account" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-account"
  version = "~> 3.0"

  account_alias = "awesome-company"

  minimum_password_length = 37
  require_numbers         = false
}

module "consul_consul-iam-policies" {
  source  = "hashicorp/consul/aws//modules/consul-iam-policies"
  version = "0.8.4"
  iam_role_id = ""
}

module "s3_bucket" {
  source = "cloudposse/s3-bucket/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  acl                      = "private"
  enabled                  = true
  user_enabled             = true
  versioning_enabled       = false
  allowed_bucket_actions   = ["s3:GetObject", "s3:ListBucket", "s3:GetBucketLocation"]
  name                     = "app"
  stage                    = "test"
  namespace                = "eg"
}

module "es" {
  source  = "git::https://github.com/terraform-community-modules/tf_aws_elasticsearch.git?ref=v1.1.0"

  domain_name                    = "my-elasticsearch-domain"
  management_public_ip_addresses = ["34.203.XXX.YYY"]
  instance_count                 = 16
  instance_type                  = "m4.2xlarge.elasticsearch"
  dedicated_master_type          = "m4.large.elasticsearch"
  es_zone_awareness              = true
  ebs_volume_size                = 100
}

A terraform init command will fetch all modules, including modules which are referenced in the modules which are defined explicitly. There's a .terraform/modules/modules.json file after init:

{
  "Modules": [
    {
      "Key": "",
      "Source": "",
      "Dir": "."
    },
    {
      "Key": "s3_bucket.s3_user",
      "Source": "cloudposse/iam-s3-user/aws",
      "Version": "0.15.1",
      "Dir": ".terraform/modules/s3_bucket.s3_user"
    },
    {
      "Key": "s3_bucket.s3_user.s3_user",
      "Source": "cloudposse/iam-system-user/aws",
      "Version": "0.20.1",
      "Dir": ".terraform/modules/s3_bucket.s3_user.s3_user"
    },
    {
      "Key": "s3_bucket.s3_user.s3_user.this",
      "Source": "cloudposse/label/null",
      "Version": "0.24.1",
      "Dir": ".terraform/modules/s3_bucket.s3_user.s3_user.this"
    },
    {
      "Key": "s3_bucket.this",
      "Source": "cloudposse/label/null",
      "Version": "0.24.1",
      "Dir": ".terraform/modules/s3_bucket.this"
    },
    {
      "Key": "consul_consul-iam-policies",
      "Source": "hashicorp/consul/aws//modules/consul-iam-policies",
      "Version": "0.8.4",
      "Dir": ".terraform/modules/consul_consul-iam-policies/modules/consul-iam-policies"
    },
    {
      "Key": "es",
      "Source": "git::https://github.com/terraform-community-modules/tf_aws_elasticsearch.git?ref=v1.1.0",
      "Dir": ".terraform/modules/es"
    },
    {
      "Key": "iam_account",
      "Source": "terraform-aws-modules/iam/aws//modules/iam-account",
      "Version": "3.9.0",
      "Dir": ".terraform/modules/iam_account/modules/iam-account"
    },
    {
      "Key": "s3_bucket",
      "Source": "cloudposse/s3-bucket/aws",
      "Version": "0.31.0",
      "Dir": ".terraform/modules/s3_bucket"
    },
    {
      "Key": "s3_bucket.s3_user.this",
      "Source": "cloudposse/label/null",
      "Version": "0.24.1",
      "Dir": ".terraform/modules/s3_bucket.s3_user.this"
    }
  ]
}

Based on this, we can process and analyse the source code of the given modules with one of the following options

hcl2json

Nice and simple, using this via wasm is likely rather straightforward.

./hcl2json ./.terraform/modules/es/variables.tf
{
    "variable": {
        "advanced_options": [
            {
                "default": {},
                "description": "Map of key-value string pairs to specify advanced configuration options. Note that the values for these configuration options must be strings (wrapped in quotes) or they may be wrong and cause a perpetual diff, causing Terraform to want to recreate your Elasticsearch domain on every apply.",
                "type": "${map(string)}"
            }
        ],

terraform config inspect

That's pretty much the output we're using right now directly from the Terraform registry. Using this via wasm isn't possible without modifications (due to disk access).

{
  "path": "/Users/sebastian/projects/cdk/tfmodules/.terraform/modules/es",
  "variables": {
    "advanced_options": {
      "name": "advanced_options",
      "type": "map(string)",
      "description": "Map of key-value string pairs to specify advanced configuration options. Note that the values for these configuration options must be strings (wrapped in quotes) or they may be wrong and cause a perpetual diff, causing Terraform to want to recreate your Elasticsearch domain on every apply.",
      "default": {},
      "required": false,
      "pos": {
        "filename": "/Users/sebastian/projects/cdk/tfmodules/.terraform/modules/es/variables.tf",
        "line": 119
      }
    }
}

@ts-terraform/hcl

This uses wasm behind the scenes to fully parse an AST of the given hcl content. It's an interesting approach, but seems to be too much at this point.

{
  "attributes": {},
  "blocks": [
    {
      "body": {
        "attributes": {
          "default": {
            "equalsRange": {
              "end": {
                "byte": 204,
                "column": 16,
                "line": 4
              },
              "filename": "foo",
              "start": {
                "byte": 203,
                "column": 15,
                "line": 4
              }
            },
            "expr": {
              "kind": "literalValueExpr",
              "srcRange": {
                "end": {
                  "byte": 209,
                  "column": 21,
                  "line": 4
                },
                "filename": "foo",
                "start": {
                  "byte": 205,
                  "column": 17,
                  "line": 4
                }
              },
              "type": "bool",
              "val": true
            },

Naming

Right now, the name of a generated module is derived from the response of the Terraform Registry. Since we're dealing with various unstructured module sources now, we'll have to come up with a naming schema and/or allow the user to override the name. That's still something to figure out.

cdktf.json

This is how the modules from above would be defined in the cdktf.json file:

{
  "language": "python",
  "app": "pipenv run python main.py",
  "terraformProviders": [
    "aws@~> 2.55"
  ],
  "terraformModules": [
    "terraform-aws-modules/iam/aws//modules/iam-account@~> 3.0",
    "hashicorp/consul/aws//modules/consul-iam-policies@0.8.4",
    "cloudposse/s3-bucket/aws",
    "git::https://github.com/terraform-community-modules/tf_aws_elasticsearch.git?ref=v1.1.0"
  ],
  "codeMakerOutput": "imports"
}

Terraform Versions

With Terraform 0.12.x it's likely to see a few error messages when running terraform init - that's an excerpt for the configuration above:

There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Warning: Provider source not supported in Terraform v0.12

  on .terraform/modules/s3_bucket.s3_user.s3_user/versions.tf line 5, in terraform:
   5:     aws = {
   6:       source  = "hashicorp/aws"
   7:       version = ">= 2.0"
   8:     }

A source was declared for provider aws. Terraform v0.12 does not support the
provider source attribute. It will be ignored.

(and 2 more similar warnings elsewhere)


Error: Custom variable validation is experimental

  on .terraform/modules/s3_bucket.s3_user.s3_user.this/variables.tf line 27, in variable "context":
  27:   validation {

This feature is currently an opt-in experiment, subject to change in future
releases based on feedback.

Activate the feature for this module by adding variable_validation to the list
of active experiments.

Terraform > 0.12 doesn't have these issues. The .terraform/modules/modules.json file seems to be consistent for all versions >= 0.12.

Getting Module Source Code

For the sake of completeness, we could in theory ship platform specific binaries with the cdktf-cli npm package or even build them on-demand via node-gyp. However, I think this should be avoided if possible.

including modules which are referenced in the modules which are defined explicitly

Given how long source generation takes, I don't think we should be generating code for the referenced modules.

Do you know if hcl2json has anything to indicate required variables? I think that is definitely useful given how many variables many modules have.

I imagine we'll end up being fairly verbose on the naming to eliminate naming collisions. Perhaps could match current names for registry sourced modules.

I think it's pretty common for modules to have <=0.12 versions and >0.12 versions. We should just make sure errors are surfaced cleanly.

including modules which are referenced in the modules which are defined explicitly

Given how long source generation takes, I don't think we should be generating code for the referenced modules.

Yes, agree 👍

Do you know if hcl2json has anything to indicate required variables? I think that is definitely useful given how many variables many modules have.

I think that could be derived from a missing default value - or are you aware of cases where that's not sufficient?

I imagine we'll end up being fairly verbose on the naming to eliminate naming collisions. Perhaps could match current names for registry sourced modules.

Sounds good.

I think it's pretty common for modules to have <=0.12 versions and >0.12 versions. We should just make sure errors are surfaced cleanly.

👍

I think that could be derived from a missing default value - or are you aware of cases where that's not sufficient?

Most likely.

WebAssembly

For reference, here a quick link collection around WebAssembly (wasm)

Supported Platforms: https://webassembly.org/roadmap/

Non-Browser Dedicated Runtimes

• https://wasmer.io/
• https://wasmtime.dev/

Options

WASI (WebAssembly System Interface)

• https://github.com/WebAssembly/WASI
• https://nodejs.org/api/wasi.html (>= v12.16)
• https://www.secondstate.io/articles/wasi-access-system-resources/
• https://radu-matei.com/blog/nodejs-wasi/
• WASI seems to be the emerging standard
• No direct build support in Go yet - see wasm: support new WASI interface golang/go#31105
• TinyGo supports it https://twitter.com/wasmerio/status/1261374695074492416

Go wasm_exec.js

• https://github.com/golang/go/blob/master/misc/wasm/wasm_exec.js
• https://www.aaron-powell.com/posts/2019-02-04-golang-wasm-1-introduction/
  • https://github.com/aaronpowell/webpack-golang-wasm-async-loader

Unrelated, but interesting

• assemblyscript.org (see status as well)
• https://webassembly.sh/ (wasmer playground)

• Make sure Module Outputs are properly typed and are using Tokens where suitable (related to Token System #525)

Gonna out-scope this, since it's significant effort to infer the types for outputs and perhaps better off as some sort of Terraform core feature anyway.

Does cdktf support specifying custom modules stored in Terraform Cloud in cdktf.json. If so how would I go about it?

Right now the only workaround seems to be pointing to the git repository.

Does cdktf support specifying custom modules stored in Terraform Cloud in cdktf.json. If so how would I go about it?

Right now the only workaround seems to be pointing to the git repository.

You should be able to specify using the same format as supported by the source parameter in hcl. See https://www.terraform.io/docs/cloud/registry/using.html#using-modules-in-configurations for more information.

Does cdktf support specifying custom modules stored in Terraform Cloud in cdktf.json. If so how would I go about it?
Right now the only workaround seems to be pointing to the git repository.

You should be able to specify using the same format as supported by the source parameter in hcl. See https://www.terraform.io/docs/cloud/registry/using.html#using-modules-in-configurations for more information.

I did as you said and ended up with the following error when running cdktf get

⠇ downloading and generating modules and providers...
There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.
╷
│ Error: Invalid module instance name
│ 
│   on main.tf.json line 1, in module:
│    1: {"terraform":{},"module":{"app.terraform.io_<REDACTED>_application-load-balancer_aws":{"source":"app.terraform.io/REDACTED/application-load-balancer/aws"}}}
│ 
│ A name must start with a letter or underscore and may contain only letters,
│ digits, underscores, and dashes.
╵
Error: non-zero exit code 1

Any help is appreciated.

Cheers.

@nandac I moved to a separate issue (#1360) for better tracking.

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.