-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[F0otsh0T 20220720] Vault Enterprise/Splunk/FluentD/Telegraf
- Loading branch information
Showing
23 changed files
with
1,865 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| ################################################################################ | ||
| # VAULT ENTERPRISE ON DOCKER | ||
| # | ||
| # @file | ||
| # @version 0.1 | ||
| # | ||
| ########## | ||
| # PREREQUISITES | ||
| # - Docker | ||
| # - Vault CLI | ||
| # - make | ||
| # - jq | ||
| # - curl | ||
| # - PGP / pass | ||
| # - Terraform | ||
| ################################################################################ | ||
|
|
||
| #SHELL := /bin/bash | ||
|
|
||
| ################################ | ||
| # FOUNDATION | ||
| ################################ | ||
| default: help | ||
| .PHONY: clean clean-volume clean-all vault-all vault-setup vault-init vault-restart-unseal vault-restart vault-unseal | ||
|
|
||
| ################################ | ||
| # CLEAN | ||
| ################################ | ||
| clean: #target ## Clean Vault | ||
| @terraform destroy -auto-approve | ||
|
|
||
| clean-volume: #target ## Clean Vault Volume | ||
| @rm -rf /var/tmp/volume/vault-ent/* | ||
|
|
||
| clean-all: clean clean-volume #target ## Clean All | ||
|
|
||
| ################################ | ||
| # SETUP AND RUN VAULT - PERSIST DATA WITH FILE VOLUME | ||
| # MAY NEED TO ENABLE FILE SHARING TO DOCKER VIA: | ||
| # DOCKER >> PREFERENCES >> RESOURCES >> FILE SHARING | ||
| ################################ | ||
| vault-all: vault-setup vault-init #target ## All Setup Targets for Vault Services | ||
|
|
||
| vault-setup: #target ## Spin Up Vault Resources | ||
| @terraform apply -auto-approve | ||
|
|
||
| ################################ | ||
| # VAULT INIT | ||
| # INIT, RAFT JOIN, UNSEAL | ||
| ################################ | ||
| vault-init: #target ## Initialize Vault | ||
| # # Initiate Vault with 1 Unseal Key Shares & 1 Threshold to Unseal | ||
| # # Method Origin: https://github.com/jacobm3/vault-local-demo/blob/9f5cce33ba34ff2cde1bfe2183bdeca6251421f0/reinit.sh#L21 | ||
| @cd data/scripts && \ | ||
| chmod 754 *.sh && \ | ||
| ./init.sh | ||
|
|
||
| vault-restart-unseal: vault-restart vault-unseal #target ## Restart and Unseal Vault | ||
|
|
||
| ################################ | ||
| # VAULT UNSEAL | ||
| # BAU - NOT FOR INIT UNSEAL | ||
| ################################ | ||
| # # Unseal Vault ## Unseal Vault | ||
| # # Method Origin: https://github.com/jacobm3/vault-local-demo/blob/9f5cce33ba34ff2cde1bfe2183bdeca6251421f0/reinit.sh#L21 | ||
| vault-unseal: #target ## Vault Unseal (Business as Usual) | ||
| # vault operator unseal $(jq -r .unseal_keys_b64[0] < keys.json) | ||
| @cd data/scripts && \ | ||
| chmod 754 unseal-cli.sh && \ | ||
| ./unseal-cli.sh | ||
|
|
||
| ################################ | ||
| # VAULT RESTART SET API ADDR | ||
| ################################ | ||
| vault-restart: #target ## Restart Vault Containers & Reset API ADDR | ||
| @cd data/scripts && \ | ||
| chmod 754 api_addr.sh && \ | ||
| ./api_addr.sh vault_s1 vault_s2 vault_s3 | ||
|
|
||
| ################################ | ||
| # HELP | ||
| # REF GH @ jen20/hashidays-nyc/blob/master/terraform/GNUmakefile | ||
| ################################ | ||
| .PHONY: help | ||
| help: #target ## [DEFAULT] Display help for this Makefile. | ||
| @echo "Valid make targets:" | ||
| @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}' | ||
|
|
||
| check_defined = \ | ||
| $(strip $(foreach 1,$1, \ | ||
| $(call __check_defined,$1,$(strip $(value 2))))) | ||
| __check_defined = \ | ||
| $(if $(value $1),, \ | ||
| $(error Undefined $1$(if $2, ($2)))) | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,233 @@ | ||
| --- | ||
| tags: | ||
| - docker | ||
| - arm64 | ||
| - hashicorp | ||
| - hcp-vault | ||
| - vault | ||
| - hcp-terraform | ||
| - terraform | ||
| - raft | ||
| - dr | ||
| - fluentd | ||
| - splunk | ||
| - telegraf | ||
| - tech | ||
| alias: | ||
| - Vault Enterprise Raft DR Cluster | ||
| --- | ||
|
|
||
| --- | ||
| # INFRA // Vault Enterprise - Raft DR Cluster | ||
|
|
||
| Vault Enterprise in a DR Cluster with Raft Storage Backend. | ||
|
|
||
| ## Introduction | ||
|
|
||
| Credit goes to ***[nicklhw@github](https://github.com/nicklhw)*** as these Terraform templates are based off ***[nicklhw's](https://github.com/nicklhw)*** design, research, and scripts to stand up a local **VAULT** Enterprise Cluster. | ||
|
|
||
| This spins up: | ||
|
|
||
| - 3 Node (Container) Vault DR Cluster | ||
| - Telegraf | ||
| - FluentD | ||
| - Splunk (*Needs HashiCorp Vault Application) | ||
|
|
||
| ## Caveats | ||
|
|
||
| - HCLIC: You will need to provide your own Vault Enterprise License File | ||
| - Splunk Hashicorp Vault Application TGZ: You will need to provide this from your Splunk subscription | ||
| - ARM64/V8 or Mac M1: | ||
| - Splunk does not support ARM64/V8 nor Mac M1 architecture in Docker | ||
| - FluentD may have some issues running in AMD64 Emulation Mode | ||
|
|
||
| ## References | ||
|
|
||
| - https://www.terraform.io/cli/run | ||
| - https://learn.hashicorp.com/collections/terraform/docker-get-started | ||
| - https://registry.terraform.io/providers/kreuzwerker/docker | ||
| - https://hub.docker.com/_/vault | ||
|
|
||
| ## Files | ||
|
|
||
| ``` | ||
| . | ||
| ├── Makefile | ||
| ├── README.md | ||
| ├── data | ||
| │ ├── fluentd | ||
| │ │ ├── Dockerfile | ||
| │ │ └── conf | ||
| │ │ └── fluent.conf | ||
| │ ├── scripts | ||
| │ │ ├── api_addr.sh | ||
| │ │ ├── init.sh | ||
| │ │ └── unseal-cli.sh | ||
| │ ├── splunk | ||
| │ │ ├── atunnecliffe | ||
| │ │ │ ├── Dockerfile | ||
| │ │ │ ├── LICENSE | ||
| │ │ │ ├── README.md | ||
| │ │ │ ├── ca_profile.xml | ||
| │ │ │ ├── gosplunk.sh | ||
| │ │ │ └── splunkunraid.xml | ||
| │ │ ├── build | ||
| │ │ │ ├── ~~Dockerfile~~ | ||
| │ │ │ ├── files | ||
| │ │ │ │ ├── keepalive.sh | ||
| │ │ │ │ └── msg-buildtest | ||
| │ │ │ └── terraform.tfstate | ||
| │ │ ├── default.yml | ||
| │ │ └── ~~hashicorp-vault-app-for-splunk_103.tgz~~ | ||
| │ ├── telegraf | ||
| │ │ └── telegraf.conf | ||
| │ └── vault | ||
| │ ├── shared | ||
| │ │ └── ~~vault.hclic~~ | ||
| │ ├── vault_s1 | ||
| │ │ ├── addr.hcl | ||
| │ │ └── vault-server.hcl | ||
| │ ├── vault_s2 | ||
| │ │ ├── addr.hcl | ||
| │ │ └── vault-server.hcl | ||
| │ └── vault_s3 | ||
| │ ├── addr.hcl | ||
| │ └── vault-server.hcl | ||
| ├── main.tf | ||
| ├── network.tf | ||
| ├── outputs.tf | ||
| ├── provider.tf | ||
| ├── storage.tf | ||
| ├── variables.tf | ||
| ├── ~~vault.auto.tfvars~~ | ||
| └── versions.tf | ||
| ``` | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Docker | ||
| - Terraform | ||
| - jq | ||
| - GPG / PGP / Pass | ||
|
|
||
| ## Usage | ||
|
|
||
| #### Spin Up | ||
|
|
||
| ```shell | ||
| make -f Makefile vault-all | ||
| ``` | ||
|
|
||
| #### Spin Down | ||
|
|
||
| ```shell | ||
| make -f Makefile clean-all | ||
| ``` | ||
|
|
||
| ## Providers | ||
|
|
||
| | Name | Version | | ||
| |------|---------| | ||
| | <a name="local"></a> [hashicorp/local](https://registry.terraform.io/providers/hashicorp/local) | >=2.1.0 | | ||
| | <a name="docker"></a> [kreuzwerker/docker](https://registry.terraform.io/providers/kreuzwerker/docker) | >=2.16.0 | | ||
|
|
||
| ## Modules | ||
|
|
||
| No modules. | ||
|
|
||
| ## Resources | ||
|
|
||
| | Name | Type | | ||
| |------|------| | ||
| | [docker.docker_container](https://registry.terraform.io/providers/kreuzwerker/docker/latest/docs/resources/container) | resource | | ||
| | [docker.docker_image](https://registry.terraform.io/providers/kreuzwerker/docker/latest/docs/resources/image) | resource | | ||
| | [docker.docker_network](https://registry.terraform.io/providers/kreuzwerker/docker/latest/docs/resources/network) | resource | | ||
| | [local.local_file](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource | | ||
|
|
||
| ## Inputs | ||
|
|
||
| | Name | Description | Type | Default | Required | | ||
| |------|-------------|------|---------|:--------:| | ||
| | <a name="path_splunk_hashi_app"></a> [path_splunk_hashi_app](#) | Specify path on local disk to mount Splunk Container /tmp/<name of Splunk HashiCorp Vault App> | | /var/tmp/volume/vault-ent/splunk/tmp | yes | | ||
| | <a name="splunk_hashi_app"></a> [splunk_hashi_app](#) | HashiCorp Vault Splunk App Package | | hashicorp-vault-app-for-splunk_103.tgz | yes | | ||
| | <a name="path_splunk_config"></a> [path_splunk_config](#) | Specify path on local disk to mount Splunk Container /tmp/defaults | | /var/tmp/volume/vault-ent/splunk/tmp/defaults | yes | | ||
| | <a name="path_telegraf_config"></a> [path_telegraf_config](#) | Specify path on local disk to mount Telegraf Container /etc/telegraf | | /var/tmp/volume/vault-ent/telegraf/etc/telegraf | yes | | ||
| | <a name="path_fluent_config"></a> [path_fluent_config](#) | Specify path on local disk to mount FluentD Container /fluentd/etc | | /var/tmp/volume/vault-ent/fluent/etc | yes | | ||
| | <a name="path_vault_s1_config"></a> [path_vault_s1_config](#) | Specify path on local disk to mount Vault Container /vault/config | | /var/tmp/volume/vault-ent/vault_s1/config | yes | | ||
| | <a name="path_vault_s1_data"></a> [path_vault_s1_data](#) | Specify path on local disk to mount Vault Container /vault/data | | /var/tmp/volume/vault-ent/vault_s1/data | yes | | ||
| | <a name="path_vault_s2_config"></a> [path_vault_s2_config](#) | Specify path on local disk to mount Vault Container /vault/config | | /var/tmp/volume/vault-ent/vault_s1/config | yes | | ||
| | <a name="path_vault_s2_data"></a> [path_vault_s2_data](#) | Specify path on local disk to mount Vault Container /vault/data | | /var/tmp/volume/vault-ent/vault_s1/data | yes | | ||
| | <a name="path_vault_s3_config"></a> [path_vault_s3_config](#) | Specify path on local disk to mount Vault Container /vault/config | | /var/tmp/volume/vault-ent/vault_s1/config | yes | | ||
| | <a name="path_vault_s3_data"></a> [path_vault_s3_data](#) | Specify path on local disk to mount Vault Container /vault/data | | /var/tmp/volume/vault-ent/vault_s1/data | yes | | ||
| | <a name="path_vault_log"></a> [path_vault_log](#) | Specify path on local disk to mount **SHARED** Vault Container /var/log/vault | | /var/tmp/volume/vault-ent/log | yes | | ||
| | <a name="container_image_vault"></a> [container_image_vault](#) | Image Name of Vault Container | | hashicorp/vault-enterprise | yes | | ||
| | <a name="container_tag_vault"></a> [container_tag_vault](#) | Image Tag of Vault Container | | 1.11.0-ent | no | | ||
| | <a name="container_name_vault_s1"></a> [container_name_vault_s1](#) | Name of Vault Container | | vault_s1 | yes | | ||
| | <a name="container_name_vault_s2"></a> [container_name_vault_s2](#) | Name of Vault Container | | vault_s2 | yes | | ||
| | <a name="container_name_vault_s3"></a> [container_name_vault_s3](#) | Name of Vault Container | | vault_s3 | yes | | ||
| | <a name="container_image_fluentd"></a> [container_image_fluentd](#) | Image Name of FluentD Container | | fluentd | yes | | ||
| | <a name="container_tag_fluentd"></a> [container_tag_fluentd](#) | Image Tag of FluentD Container | | v1.14-1 | no | | ||
| | <a name="container_name_fluentd"></a> [container_name_fluentd](#) | Name of FluentD Container | | vault-fluentd | yes | | ||
| | <a name="container_image_splunk"></a> [container_image_splunk](#) | Image Name of Splunk Container | | splunk/splunk | yes | | ||
| | <a name="container_tag_splunk"></a> [container_tag_splunk](#) | Image Tag of Splunk Container | | 9.0.0 | no | | ||
| | <a name="container_name_splunk"></a> [container_name_splunk](#) | Name of Splunk Container | | vault-splunk | yes | | ||
| | <a name="container_image_telegraf"></a> [container_image_telegraf](#) | Image Name of Telegraf Container | | telegraf | yes | | ||
| | <a name="container_tag_telegraf"></a> [container_tag_telegraf](#) | Image Tag of Telegraf Container | | 1.23 | no | | ||
| | <a name="container_name_telegraf"></a> [container_name_telegraf](#) | Name of Telegraf Container | | vault-telegraf | yes | | ||
| | <a name="docker_port_internal_vault"></a> [docker_port_internal_vault](#) | Internal Port for Vault Container | | 8200 | no | | ||
| | <a name="docker_port_external_vault_s1"></a> [docker_port_external_vault_s1](#) | External Port for Vault S1 Container | | 18200 | yes | | ||
| | <a name="docker_port_external_vault_s2"></a> [docker_port_external_vault_s2](#) | External Port for Vault S2 Container | | 28200 | yes | | ||
| | <a name="docker_port_external_vault_s1"></a> [docker_port_external_vault_s3](#) | External Port for Vault S3 Container | | 38200 | yes | | ||
| | <a name="docker_port_internal_fluentd"></a> [docker_port_internal_fluentd](#) | Internal Port for FluentD Container | | 24224 | no | | ||
| | <a name="docker_port_external_fluentd"></a> [docker_port_external_fluentd](#) | External Port for FluentD Container | | 24224 | no | | ||
| | <a name="docker_protocol_fluentd"></a> [docker_protocol_fluentd](#) | Protocol for FluentD Container | | udp | no | | ||
| | <a name="docker_port_internal_splunk"></a> [docker_port_internal_splunk](#) | Internal Port for Splunk Container Web | | 8000 | no | | ||
| | <a name="docker_port_external_splunk"></a> [docker_port_external_splunk](#) | External Port for Splunk Container Web | | 8000 | no | | ||
| | <a name="docker_port_internal_telegraf"></a> [docker_port_internal_telegraf](#) | Internal Port for Telegraf Container | | 8125 | no | | ||
| | <a name="docker_port_external_telegraf"></a> [docker_port_external_telegraf](#) | External Port for Telegraf Container | | 8125 | no | | ||
| | <a name="docker_protocol_telegraf"></a> [docker_protocol_telegraf](#) | Protocol for Telegraf Container | | udp | no | | ||
| | <a name="docker_restart_vault"></a> [docker_restart_vault](#) | Restart Policy for Vault Container | | unless-stopped | no | | ||
| | <a name="docker_restart_splunk"></a> [docker_restart_splunk](#) | Restart Policy for Splunk Container | | always | no | | ||
| | <a name="docker_restart_telegraf"></a> [docker_restart_telegraf](#) | Restart Policy for Telegraf Container | | always | no | | ||
| | <a name="vault_root_token"></a> [vault_root_token](#) | Root Token for Vault Service | | | no | | ||
|
|
||
| ## Outputs | ||
|
|
||
| | Name | Description | | ||
| |------|-------------| | ||
| | <a name="container_name_vault_s1"></a> [container_name_vault_s1](#) | n/a | | ||
| | <a name="container_name_vault_s2"></a> [container_name_vault_s2](#) | n/a | | ||
| | <a name="container_name_vault_s3"></a> [container_name_vault_s3](#) | n/a | | ||
| | <a name="container_name_vault_fluentd"></a> [container_name_vault_fluentd](#) | n/a | | ||
| | <a name="container_name_vault_splunk"></a> [container_name_vault_splunk](#) | n/a | | ||
| | <a name="container_name_vault_telegraf"></a> [container_name_vault_telegraf](#) | n/a | | ||
| | <a name="container_ip_vault_s1"></a> [container_ip_vault_s1](#) | n/a | | ||
| | <a name="container_ip_vault_s2"></a> [container_ip_vault_s2](#) | n/a | | ||
| | <a name="container_ip_vault_s3"></a> [container_ip_vault_s3](#) | n/a | | ||
| | <a name="container_ip_vault_fluentd"></a> [container_ip_vault_fluentd](#) | n/a | | ||
| | <a name="container_ip_vault_telegraf"></a> [container_ip_vault_telegraf](#) | n/a | | ||
| | <a name="dirname_splunk_config"></a> [dirname_splunk_config](#) | n/a | | ||
| | <a name="dirname_splunk_hashi_app"></a> [dirname_splunk_hashi_app](#) | n/a | | ||
| | <a name="container_network"></a> [container_network](#) | n/a | | ||
|
|
||
|
|
||
| [[]] | ||
|
|
||
| ## Sources | ||
| - https://splunk.github.io/docker-splunk/ | ||
| - https://github.com/brianshumate/fluentd-splunk-hec | ||
| - https://docs.fluentd.org/v/0.12/input/forward | ||
| - https://docs.docker.com/config/containers/logging/fluentd/#fluentd-async | ||
| - https://www.fluentd.org/faqs | ||
| - https://github.com/fluent/fluent-plugin-splunk | ||
| - https://github.com/splunk/fluent-plugin-splunk-hec | ||
| - https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/TypesofSplunklicenses | ||
|
|
||
| ## Appendix | ||
|
|
||
| - ARM64/V8 Potential Splunk Workarounds | ||
| - https://github.com/splunk/docker-splunk/issues/516 | ||
| - https://hub.docker.com/r/odidev/docker-splunk/tags | ||
| - Splunk Container in Alpine | ||
| - https://github.com/andrew207/splunk | ||
| - |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| FROM fluent/fluentd:v1.14-1 | ||
| USER root | ||
| RUN ["fluent-gem", "install", "fluent-plugin-splunk-enterprise"] | ||
| USER fluent |
29 changes: 29 additions & 0 deletions
29
00-vault/terraform/vault-enterprise/data/fluentd/conf/fluent.conf
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| <source> | ||
| @type tail | ||
| path /var/log/vault/vault-audit.log | ||
| pos_file /var/log/vault/vault-audit-log.pos | ||
| <parse> | ||
| @type json | ||
| time_format %iso8601 | ||
| </parse> | ||
| tag vault_audit | ||
| </source> | ||
|
|
||
| <filter vault_audit> | ||
| @type record_transformer | ||
| <record> | ||
| cluster primary | ||
| </record> | ||
| </filter> | ||
|
|
||
| <filter vault_audit> | ||
| @type stdout | ||
| </filter> | ||
|
|
||
| <match vault_audit.**> | ||
| @type splunk_hec | ||
| protocol http | ||
| host vault-splunk | ||
| port 8088 | ||
| token vault-audit-hec-token | ||
| </match> |
19 changes: 19 additions & 0 deletions
19
00-vault/terraform/vault-enterprise/data/scripts/api_addr.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| #!/bin/bash | ||
|
|
||
| # This scripts detects the IP address for vault servers, writes it to a config file, then restarts Vault | ||
| for server in $@ | ||
| do | ||
| export ip=$(docker inspect ${server} -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}') | ||
| echo "IP address of ${server}: ${ip}" | ||
| echo "Writing config file: /var/tmp/volume/vault-ent/${server}/config/addr.hcl" | ||
| cat <<EOF > /var/tmp/volume/vault-ent/${server}/config/addr.hcl | ||
| api_addr = "http://${ip}:8200" | ||
| cluster_addr = "https://${ip}:8201" | ||
| EOF | ||
| cat <<EOF > ../vault/${server}/addr.hcl | ||
| api_addr = "http://${ip}:8200" | ||
| cluster_addr = "https://${ip}:8201" | ||
| EOF | ||
| echo "starting ${server}" | ||
| docker restart $server | ||
| done |
Oops, something went wrong.