Developer documenation for setting up local environment (Optum#326)

* Added documentation for CLI changes. * Cleaned up references to .dce.yaml. * Reorganized terraform-based configuration to advanced section. * WIP: start of the test script. * WIP: initial version of test script through at least deploying DCE via CLI. * WIP: Shell script so far asserts a successful deployment using the DCE CLI. * Added example configuration file for tests. * Updated policies and documentation to match new support AWS Nuke services. * Added details in changelog for newly-supported services. * Regenerated UserDetailer which seems to be causaing issues in ADO build. * Changed from require to assert to avoid panic. * Reverted changes because it was a bad idea. * Removed bad comma which was probably causing JSON problems. * Updated developer documenation with setup information and hardened install_ci.sh script to be used by developers. * Updated CHANGELOG to include descriptions of changes from feature branch.
jfrconley · Mar 27, 2020 · a754c06 · a754c06
1 parent 215e0d4
commit a754c06
Show file tree

Hide file tree

Showing 5 changed files with 173 additions and 62 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -14,6 +14,9 @@
   support. See README in `tools/awsnukedocgen`.
 - Added new services supported by DCE: Kinesis Analytics, Kinesis Video, Opsworks CM, Robomaker,
   and Worklink.
+- Added new `make setup` target for installing tools on local developer machines.
+- Hardened `scripts/install_ci.sh` to be used by developers.
+- Added documenation for local developer environment setup.
 
 ## v0.28.0
 

diff --git a/Makefile b/Makefile
@@ -76,4 +76,7 @@ install:
 	go install 
 
 clean:
-	rm -rf ./bin ./vendor ./html-doc ./site
+	rm -rf ./bin ./vendor ./html-doc ./site
+
+setup:
+	./scripts/install_ci.sh
diff --git a/docs/develop.md b/docs/develop.md
@@ -2,6 +2,52 @@
 
 This page will guide you through some basic points for getting started developing against the DCE codebase.
 
+> *Note: unless otherwise noted, all commands shown here should be executed from
+> the DCE base directory*
+
+## Configuring your development environment
+
+You may find development easiest on a Mac OS or Linux-based machine. Development should be possible on Windows 10 with the [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install-win10) installed, but at the time of this writing has not been verified.
+
+You will need the following:
+
+1. [Go](https://golang.org/doc/install) (version 1.13.x)
+1. [Terraform](https://learn.hashicorp.com/terraform/getting-started/install.html) (version v0.12.x)
+1. [GNU make]() (version 3.x)
+1. [GNU bash](https://www.gnu.org/software/bash/), which is used for shell scripts
+1. An [AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/) for deploying resources
+1. The [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) _Note: if you install version 2, see "Configuring AWS CLI 2"_
+1. An [AWS IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html) with command line access.
+1. [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) (version 2.x)
+
+***Important: deploying DCE to your AWS account can incur cost.***
+
+### Configuring AWS CLI 2
+
+The AWS CLI version 2 includes a breaking change that creates problems with the automation scripts. See 
+https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html for more information.
+
+For DCE, the recommeonded solution to this problem is to add the following line 
+in your `~/aws/config` file:
+
+```ini
+[default]
+cli_pager=
+```
+
+### Getting the code locally
+
+To get the code locally fork this repo (https://github.com/Optum/dce) and then clone the repository:
+
+```
+git clone https://github.com/${mygithubid}/dce.git dce # replace with your fork's HTTPS URL
+cd dce
+make setup
+```
+
+The last command, `make setup`, will run the `scripts/install_ci.sh` script which will install
+the necessary tools for building and testing the project.
+
 ## Code Structure
 
 The DCE codebase is comprised of [Go](https://golang.org/) application code, along with [Terraform](https://terraform.io) infrastructure configuration.
@@ -13,11 +59,12 @@ The Go code is primarily located within:
 
 Each subdirectory within the [/cmd/lambda](https://github.com/Optum/dce/tree/master/cmd/lambda) directory targets an individual Lambda function of the same name.
 
+
 ## Building application code
 
 To compile the Go application code, run:
 
-```
+```bash
 make build
 ```
 
@@ -29,13 +76,28 @@ Unit tests are located within the `/cmd` and `/pkg` directories, adjacent to the
 
 Execute unit tests by running:
 
-```
+```bash
 make test
 ``` 
 
+## Code Linting
+
+When you run `make test`, the `lint` target is executed automatically. You can, however, run
+the linting by itself by using the command:
+
+```bash
+make lint
+```
+
+During `make lint`, the `scripts/lint.sh` script executes [golangci-lint](https://github.com/golangci/golangci-lint). The configuration file is `.golangci.yml`. Enabled linters and 
+rule exceptions can be found in this file.
+
+The `make lint` target also executes [tflint](https://github.com/terraform-linters/tflint)
+to lint the [terraform](https://www.terraform.io/) code found in `modules`.
+
 ## Functional Tests
 
-Functional tests are used where we want to test the integration between a number of services or verify that end-to-end behavior is working properly. For example, we rely heavily on functional tests for DynamoDB interactions, to verify that we are using the DynamoDB SDKs correctly.
+Functional tests are located in `tests` and are used to test the integration between a number of services or verify that end-to-end behavior is working properly. For example, we rely heavily on functional tests for DynamoDB interactions, to verify that we are using the DynamoDB SDKs correctly.
 
 Before running functional tests, DCE must be deployed to a test AWS account. **Functional tests truncate the database tables, so do not run them against production environments.**
 
@@ -48,6 +110,7 @@ terraform init
 terraform apply
 
 # Deploy application code to AWS
+cd ..
 make deploy 
 ``` 
 
@@ -59,4 +122,45 @@ To run functional tests:
 make test_functional
 ```
 
-Functional tests load the details of the DCE deployment from Terraform module outputs, so there is no need for additional configuration to run functional tests.
+Functional tests load the details of the DCE deployment from Terraform module outputs, so there is no need for additional configuration to run functional tests.
+
+## Before committing code
+
+The `make test` target is used by continuous integration build. A failure of the target will
+cause the build to fail, so before committing code or creating a pull request you should
+run the following commands:
+
+```bash
+make build
+make test
+```
+
+
+## Building the documentation
+
+The documentation is located in `docs` and is based on the [Sphinx project](https://docs.readthedocs.io/en/stable/intro/getting-started-with-sphinx.html) and hosted on http://readthedocs.io.
+
+If you are making changes to documenation and would like to verify the build of the 
+documentation, you will need to make sure Python 3 is installed. It is _highly recommended_
+that you use `virtualenv` and configure your workspace with the commands shown here:
+
+```bash
+virtualenv -p python3 ENV
+source ENV/bin/activate
+pip install -r docs/requirements.txt
+```
+
+With the Python requirements installed and the virtualenv sourced, use the 
+following command from the base project directory:
+
+```bash
+make documentation
+```
+
+To serve the documentation locally, run the following command:
+
+```bash
+make serve_docs
+```
+
+By default, the documenation will be served at `http://127.0.0.1:8000`.
diff --git a/scripts/generate-docs.sh b/scripts/generate-docs.sh
diff --git a/scripts/install_ci.sh b/scripts/install_ci.sh
@@ -1,19 +1,66 @@
 #!/bin/bash
+#------------------------------------------------------------------------------
+# install_ci.sh
+# The purpose of this script is install the required tools for building and
+# testing DCE
+#------------------------------------------------------------------------------
+
 set -euxo pipefail
 
-export GOBIN=$(dirname `which go`)
+# safeGoGet will get the go tool, checking to see if it exists first and doing
+# a little additional messaging
+function safeGoGet() {
+  binname=$(basename $1)
+  echo "Checking for ${binname} on the path..."
+  if [ ! -x "$(command -v ${binname})" ]; then 
+    echo -n "Getting ${binname} from $1..."
+    go get $1 >/dev/null 2>&1
+    echo "Done."
+  fi
+}
+
+if [ ! -x "$(command -v go)" ]; then
+  echo "Cannot find \'go\'. Check your $PATH to make sure it includes \'go\'." >&2
+  exit 1
+fi
 
+gopath=$(command -v go)
+export GOBIN=$(dirname ${gopath})
 export GO111MODULE=off
 
-wget -q https://github.com/wata727/tflint/releases/download/v0.13.4/tflint_linux_amd64.zip
-unzip tflint_linux_amd64.zip
-chmod +x tflint
-mv tflint $GOBIN
+# Get the proper version of tflint first and install it into the same path
+# as the other go tools.
+if [ -x "$(command -v wget)" ]; then
+  if [ ! -x "$(command -v tflint)" ]; then
+    wget -q https://github.com/wata727/tflint/releases/download/v0.13.4/tflint_linux_amd64.zip -O /tmp/tflint.zip
+    (cd /tmp && unzip tflint.zip)
+    chmod +x /tmp/tflint
+    mv /tmp/tflint $GOBIN
+  fi
+else
+  echo "Cannot find wget, which is required for installing tools."
+  exit 1
+fi
+
+#------------------------------------------------------------------------------
+# Dependencies go here.
+#------------------------------------------------------------------------------
+
+# go-junit-report is used by the test scripts to generate report output readable
+# by CI tools that can read JUnit reports
+safeGoGet github.com/jstemmer/go-junit-report
+
+# gcov is used to generate coverage information in a report that can be read
+# by CI tools
+safeGoGet github.com/axw/gocov/gocov
+safeGoGet github.com/AlekSi/gocov-xml
+safeGoGet github.com/matm/gocov-html
+
+# golangci-lint is a lint aggregator used in the lint.sh script to lint the
+# go and terraform code.
+safeGoGet github.com/golangci/golangci-lint/cmd/golangci-lint
 
-go get github.com/jstemmer/go-junit-report
-go get github.com/axw/gocov/gocov
-go get github.com/AlekSi/gocov-xml
-go get github.com/matm/gocov-html
-go get -u github.com/golangci/golangci-lint/cmd/golangci-lint
+# gosec is used for checking the code for security problems
+safeGoGet github.com/securego/gosec/cmd/gosec
 
-go get github.com/securego/gosec/cmd/gosec
+echo "Setup complete."