AWS CloudFormation Template for generating: VPC, Internet Gateway, Subnets, RouteTables and Security Groups, EC2 Instances, installing CorkroachDB and certificates. The cloudformation template usually takes less than 2 minutes to execute.

Once the infrastructure has been created, node and root certs will be generated and cockroachDB will be started. You can choose to have the CloudFormation Template issue the "init" command, or you can do that manually (helpful when creating a multi-region cluster).

Running this Cloudformation Template in multiple Regions will allow you to create a Mulit-Region Cockroach Cluster, but there are a few extra steps to follow such as peering the VPCs, modifying route tables and security groups (see instructions below).

When the CloudFormation stack is deleted, all resources created by the template in a region are deleted as well including the VPC, Internet Gateway, Subnets, RouteTables, Security Groups and EC2 Instances. No data is maintained or saved.

For instructions on creating a stack using this CloudFormation Template use the instructions below.


For instructions on creating a stack using this CloudFormation Template via the CLI, us the instructions below

For instructions on adding a Network Load Balancer to the infrastructure created by this CloudFormation Template, follow the instructions below

The ca.crt and ca.key created by this template are the same for all executions. Anyone with access to this CloudFormation template has the CA key for the cluster.

The keys can be roated using the instructions here.

You can also change the key by editing the CloudFormation template prior to execution.

The security groups only allow your IP to access the EC2 instances, so although the CA key is not private until changed, access to the cluster is very limited.

If you're going to execute this template in multiple regions, be sure to choose non-overlaping CIDR blocks for each region. For example:

This will allow you to easily peer the 3 VPCs.

The following values are exported by the CloudFormation Template

To create a multi-region CockroachDB Cluster:

1. When running the CloudFormation Template, choose 'NO' to init the databases (you'll do then manually when you've completed the mulit-region setup).
2. Use the CloudFormation template to create the database in the first region. Once the stack is complete, navigate to the "OUTPUTS" section and find the "JoinValue". You will use this value as a parameter in all subsequent regions.
3. Run the CloudFormation template in additional regions. Be sure to choose non-overlaping VPC CIDRs for each region. In the "ExistingJoinString" parameter use the "JoinValue" from the "OUTPUTS" section of the 1st Region.
4. Once the stacks are created in each Region, you'll need to create VPC peering between all regions (for 3 regions there will be 3 vpc peering connections, for 4 regions there would be 6 and so on.)
5. Edit the routes of public route table for each region adding the VPC CIDRs for all regions. The Destination is the VPC CIDR routes and the target is the VPC Peering connections.
6. Edit the inbound rules of the security group (instance-security-group02) in each region. The souce should be the VPC CIDRs.
7. Log on to any one of the nodes and run cockroach init. Once that is complete, the cluster should be up and running
   
   
   

When deleting stacks in a multi-region cluster, be sure to delete the VPC Peering Connections first.

Navigate to the VPC Services section of AWS and select "Your VPCs" from the side menu. Verify that the IPv4 CIDR you want to use for this Stack is not in use. For example, I may want to use 192.168.4.0/24, so I need to be sure that address space is not already in use in this region.

Without a key-pair, you won't be able to access the EC2 instances created as part of this CloudForamtion Template.

To list your existing key-pairs, navigate to the EC2 Section of the AWS Console and select "Key Pairs" from the side-bar menu which is under "Network & Security". If you need to create a new key-pair, choose the "Create key pair" button and download your new key-pair to a safe location.

To find your IP address you can use Google and search for "my ip address".

Navigate to the AWS CloudFormation Service and choose "Create stack".

In the "Prerequisite - Prepare template" section, choose "Template is ready"
In the "Specify template" section, choose "Upload a template file"
Click the "Choose file" button and select the CloudFormation yaml file (crdb_cloudformation_template.yaml).

Click here to see the parameter descriptions

On the "Configure stack options" page, you can accept everything as-is and select the "Next" button.

The final page is the "Review" page. Select the "Next" button to create the stack.

When your CloudFormation Template has completed, you will get a "CREATE_COMPLETE" Status message.

To see the resources that were created, the outputs, or to review the parameters you selected, you can choose to see the Stack Details and select one of the Tabs from the screen.

To delete the stack and all of the resources created as part of the stack, select the stack and hit the "Delete" button. Once the process is complete, the EC2 instances will be terminated, and all resources will be gone.

aws cloudformation create-stack --stack-name myteststack --template-body file://crdb_cloudformation_template.yaml --cli-input-yaml file://crdb_cloudformation_cli_parameters.yaml

In order to create the stack in the AWS CLI, you'll need to have the following two files in your directory:

• crdb_cloudformation_template.yaml
• crdb_cloudformation_cli_parameters.yaml

The crdb_cloudformation_template.yaml is the cloudformation template and the crdb_cloudformation_cli_parameters.yaml file is the parameters needed to run the cloudformation template.

To add a network load balancer to the infrastructure involves 3 steps

• creating the target group: This is the list of instances to which the load balancer will send traffic
• create the load balancer and attach it to the target group
• modify the security group to allow the load balancer to reach the target group instances

(Note that in a multi-region setup you would probably want an NLB in each region so that users get directed to most appropriate, possibly closest node.)

Assume that we have created a Cockroach Cluster in 1 region with 3 nodes:

[ec2-user@ip-192-168-3-4 ~]$ cockroach node status
  id |       address       |     sql_address     |  build  |         started_at         |         updated_at         |             locality             | is_available | is_live
-----+---------------------+---------------------+---------+----------------------------+----------------------------+----------------------------------+--------------+----------
   1 | 192.168.3.4:26257   | 192.168.3.4:26257   | v21.2.3 | 2022-01-10 20:05:52.653956 | 2022-01-20 21:17:48.212539 | region=us-west-2,zone=us-west-2a | true         | true
   2 | 192.168.3.132:26257 | 192.168.3.132:26257 | v21.2.3 | 2022-01-10 20:05:54.215074 | 2022-01-20 21:17:49.768001 | region=us-west-2,zone=us-west-2c | true         | true
   3 | 192.168.3.68:26257  | 192.168.3.68:26257  | v21.2.3 | 2022-01-10 20:05:54.519396 | 2022-01-20 21:17:50.069444 | region=us-west-2,zone=us-west-2b | true         | true

Navigate to the EC2 console and from the menu on the left, choose "Target Groups" and "Create target group".

On the "Specify group details" page:

• select the most appropriate target type, I chose "instances"
• Give the group a name (MyTargetGroup in the example below)
• select "TCP" as the protocol and 26257 as the port (26257 is the CockroachDB Port)
• and select the VPC that was created as part of the CloudFormation (you can search by name)
• enter any appropriate tags
• click "Next"

On the "Register targets" page:

• select all of the instances in the "Available instances" section
• be sure the "Ports for the selected instances" is 26257
• select the "Include as pending below" button
• select the "Create target group" button.

Navigate to the EC2 console and from the menu on the left, choose "Load Balancers".

Select "Create" Network Load Balancer from the "Select load balancer type" page.

On the "Create Network Load Balancer Page"

• Basic Configuration Section
  • Enter a load balancer name (MyLoadBalancer in the example)
  • select Internal from the Scheme radio buttons
  • select IPv4 from the IP Address type radio buttons
• Network Mapping Section
  • Enter the VPC created in the CloudFormation
  • select the private subnet for each availability zone (we're select the private subnets because we choose to create an internal load balancer. If you choose to create an internet-facing load balancer, you would need to select public subnets here.)
• Listener, Tags and Summary Sections
  • Protocol should be "TCP"
  • Port is 26257
  • Forward To is the target group created above "MyTargetGroup"
  • Add any tags that are important for you
• Select "Create load balancer"

You'll need to modify the security group (sg02) created by the CloudFormation template to allow the load balancer to reach the Cockroach Nodes.

Navigate to the EC2 console and from the menu on the left, choose "Security Groups" and find the sg02 security group.

Select the "Inbound Rules" rules tab in the bottom section of the screen and select "Edit inbound rules".

Enter a new security group rule with the following properties:

• Type is "Custom TCP"
• Port range is: 26257
• The source is the VPC CIDR range you selected during CloudFormation

Navigate back to the Target Groups Section and be sure Health Status of all the instances is "healthy" (it will take several minutes based on the health check default settings).

If you're having problems with the Health Check be sure to check your security group settings.