Skip to content

Latest commit

 

History

History

msk-cluster

msk-cluster

This module creates following resources.

  • aws_msk_cluster
  • aws_msk_configuration
  • aws_msk_scram_secret_association (optional)
  • aws_security_group (optional)
  • aws_security_group_rule (optional)
  • aws_secretsmanager_secret (optional)

Requirements

Name Version
terraform >= 1.5
aws >= 4.22
random >= 3.3

Providers

Name Version
aws 5.19.0
random 3.5.1

Modules

Name Source Version
resource_group tedilabs/misc/aws//modules/resource-group ~> 0.10.0
secret tedilabs/secret/aws//modules/secrets-manager-secret ~> 0.2.0
security_group tedilabs/network/aws//modules/security-group 0.26.0

Resources

Name Type
aws_msk_cluster.this resource
aws_msk_configuration.this resource
aws_msk_scram_secret_association.this resource
random_password.this resource
aws_msk_broker_nodes.this data source
aws_subnet.this data source

Inputs

Name Description Type Default Required
broker_size (Required) The desired total number of broker nodes in the kafka cluster. It must be a multiple of the number of specified client subnets. number n/a yes
broker_subnets (Required) A list of subnet IDs to place ENIs of the MSK cluster broker nodes within. list(string) n/a yes
name (Required) Name of the MSK cluster. string n/a yes
auth_sasl_iam_enabled (Optional) Enables IAM client authentication. bool false no
auth_sasl_scram_enabled (Optional) Enables SCRAM client authentication via AWS Secrets Manager. bool false no
auth_sasl_scram_kms_key (Optional) The ARN of a KMS key to encrypt AWS SeecretsManager Secret resources for storing SASL/SCRAM authentication data. Only required when the MSK cluster has SASL/SCRAM authentication enabled. The Username/Password Authentication based on SASL/SCRAM needs to create a Secret resource in AWS SecretsManager with a custom AWS KMS Key. A secret created with the default AWS KMS key cannot be used with an Amazon MSK cluster. string null no
auth_sasl_scram_users (Optional) A list of usernames to be allowed for SASL/SCRAM authentication to the MSK cluster. The password for each username is randomly generated and stored in AWS SecretsManager secret. set(string) [] no
auth_tls_acm_ca_arns (Optional) List of ACM Certificate Authority Amazon Resource Names (ARNs). list(string) [] no
auth_tls_enabled (Optional) Enables TLS client authentication. bool false no
auth_unauthenticated_access_enabled (Optional) Enables unauthenticated access. Defaults to true. bool true no
broker_additional_security_groups (Optional) A list of security group IDs to associate with ENIs to control who can communicate with the cluster. list(string) [] no
broker_allowed_ingress_cidrs (Optional) A list of CIDR for MSK ingress access. list(string) [] no
broker_instance_type (Optional) The instance type to use for the kafka brokers. string "kafka.m5.large" no
broker_public_access_enabled (Optional) Whether to allow public access to MSK brokers. bool false no
broker_volume_provisioned_throughput (Optional) Throughput value of the EBS volumes for the data drive on each kafka broker node in MiB per second. The minimum value is 250. The maximum value varies between broker type. number null no
broker_volume_provisioned_throughput_enabled (Optional) Whether provisioned throughput is enabled or not. You can specify the provisioned throughput rate in MiB per second for clusters whose brokers are of type kafka.m5.4xlarge or larger and if the storage volume is 10 GiB or greater. Defaults to false. bool false no
broker_volume_size (Optional) The size in GiB of the EBS volume for the data drive on each broker node. Minimum value of 1 and maximum value of 16384. Defaults to 1000. number 1000 no
encryption_at_rest_kms_key (Optional) Specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If no key is specified, an AWS managed KMS ('aws/msk' managed service) key will be used for encrypting the data at rest. string "" no
encryption_in_transit_client_mode (Optional) Encryption setting for data in transit between clients and brokers. TLS, TLS_PLAINTEXT, PLAINTEXT are available. string "TLS_PLAINTEXT" no
encryption_in_transit_in_cluster_enabled (Optional) Whether data communication among broker nodes is encrypted. bool true no
kafka_server_properties (Optional) Contents of the server.properties file for configuration of Kafka. map(string) {} no
kafka_version (Optional) Kafka version to use for the MSK cluster. string "2.8.0" no
logging_cloudwatch_enabled (Optional) Indicates whether you want to enable or disable streaming broker logs to Cloudwatch Logs. bool false no
logging_cloudwatch_log_group (Optional) The name of log group on CloudWatch Logs to deliver logs to. string "" no
logging_firehose_delivery_stream (Optional) Name of the Kinesis Data Firehose delivery stream to deliver logs to. string "" no
logging_firehose_enabled (Optional) Indicates whether you want to enable or disable streaming broker logs to Kinesis Data Firehose. bool false no
logging_s3_bucket (Optional) The name of the S3 bucket to deliver logs to. string "" no
logging_s3_enabled (Optional) Indicates whether you want to enable or disable streaming broker logs to S3. bool false no
logging_s3_prefix (Optional) The prefix to append to the folder name. string "" no
module_tags_enabled (Optional) Whether to create AWS Resource Tags for the module informations. bool true no
monitoring_cloudwatch_level (Optional) The desired enhanced MSK CloudWatch monitoring level. DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, PER_TOPIC_PER_PARTITION are available. string "DEFAULT" no
monitoring_prometheus_jmx_exporter_enabled (Optional) Indicates whether you want to enable or disable the JMX Exporter. bool false no
monitoring_prometheus_node_exporter_enabled (Optional) Indicates whether you want to enable or disable the Node Exporter. bool false no
resource_group_description (Optional) The description of Resource Group. string "Managed by Terraform." no
resource_group_enabled (Optional) Whether to create Resource Group to find and group AWS resources which are created by this module. bool true no
resource_group_name (Optional) The name of Resource Group. A Resource Group name can have a maximum of 127 characters, including letters, numbers, hyphens, dots, and underscores. The name cannot start with AWS or aws. string "" no
tags (Optional) A map of tags to add to all resources. map(string) {} no
timeouts (Optional) How long to wait for the MSK cluster to be created/updated/deleted. map(string)
{
"create": "120m",
"delete": "120m",
"update": "120m"
}
no

Outputs

Name Description
arn The ARN of the MSK cluster.
auth A configuration for authentication of the Kafka cluster.
bootstrap_brokers A configuration for connecting to the Kafka cluster.
plaintext - A comma separated list of one or more hostname:port pairs of kafka brokers suitable to boostrap connectivity to the kafka cluster. Only contains value if client_encryption_in_transit_mode is set to PLAINTEXT or TLS_PLAINTEXT. AWS may not always return all endpoints so the values may not be stable across applies.
sasl_iam - A comma separated list of one or more DNS names (or IPs) and SASL IAM port pairs. Only contains value if client_encryption_in_transit_mode is set to TLS_PLAINTEXT or TLS. AWS may not always return all endpoints so the values may not be stable across applies.
sasl_scram - A comma separated list of one or more DNS names (or IPs) and SASL SCRAM port pairs. Only contains value if client_encryption_in_transit_mode is set to TLS_PLAINTEXT or TLS. AWS may not always return all endpoints so the values may not be stable across applies.
tls - A comma separated list of one or more DNS names (or IPs) and TLS port pairs kafka brokers suitable to boostrap connectivity to the kafka cluster. Only contains value if client_encryption_in_transit_mode is set to TLS_PLAINTEXT or TLS. AWS may not always return all endpoints so the values may not be stable across applies.<br> public_sasl_iam- A comma separated list of one or more DNS names (or IPs) and SASL IAM port pairs. Only contains value ifclient_encryption_in_transit_modeis set to TLS_PLAINTEXT or TLS andauth_sasl_iam_enabledistrueandbroker_public_access_enabledistrue. AWS may not always return all endpoints so the values may not be stable across applies.<br> public_sasl_scram- A comma separated list of one or more DNS names (or IPs) and SASL SCRAM port pairs. Only contains value ifclient_encryption_in_transit_modeis set to TLS_PLAINTEXT or TLS andauth_sasl_scram_enabledistrueandbroker_public_access_enabledistrue. AWS may not always return all endpoints so the values may not be stable across applies.<br> public_tls- A comma separated list of one or more DNS names (or IPs) and TLS port pairs. Only contains value ifclient_encryption_in_transit_modeis set to TLS_PLAINTEXT or TLS andbroker_public_access_enabledistrue. AWS may not always return all endpoints so the values may not be stable across applies.<br>
broker A configuration for brokers of the Kafka cluster.
size - The number of broker nodes in the kafka cluster.
instance_type - The instance type used by the kafka brokers.

public_access_enabled - Whether public access to MSK brokers is enabled.
security_groups - A list of the security groups associated with the MSK cluster.

volume - A EBS volume information for MSK brokers.
broker_nodes The information of broker nodes in the kafka cluster.
broker_security_group_id The id of security group that were created for the MSK cluster.
encryption A configuration for encryption of the Kafka cluster.
at_rest - The configuration for encryption at rest.
in_transit - The configuration for encryption in transit.
kafka_config The MSK configuration.
kafka_version The MSK cluster version.
logging A configuration for logging of the Kafka cluster.
cloudwatch - The configuration for MSK broker logs to CloudWatch Logs.
firehose - The configuration for MSK broker logs to Kinesis Firehose.
s3 - The configuration for MSK broker logs to S3 Bucket.
monitoring A configuration for monitoring of the Kafka cluster.
cloudwatch - The configuration for MSK CloudWatch Metrics.
prometheus - The configuration for Prometheus open monitoring.
name The MSK cluster name.
version Current version of the MSK Cluster used for updates.
zookeeper_connections A configuration for connecting to the Apache Zookeeper cluster.
tcp - A comma separated list of one or more IP:port pairs to use to connect to the Apache Zookeeper cluster.
tls - A comma separated list of one or more IP:port pairs to use to connect to the Apache Zookeeper cluster via TLS.