KISS - Keeping Infra Simple Stupid
Create the daily used services / infrastructure for development in a single compose which are configurable.
- Regular Used Services for my works
- Fully configurable in terms of
- ENV variables
- Docker Networks
- Docker Volumes
Examples
a. Start all services
>task all
b. Start only Mongodb
>task mongodb-up
c. Watch Mongo Logs
>task mongodb-logs
d. Shutdown all services & clean up
>task shutdown
....
...
..
.
And so on !!
This way any combination of services can be Started | Stopped | Monitored | Restarting
as per the Project Dev is working on currently
Note: To leverage simple commands for start/stop/logs install Go and Taskfile
as per the instructions
https://taskfile.dev/installation/
| Service | Type | Configured |
|---|---|---|
| MongoDB | Database |
Y |
| Emqx | MQTT Broker |
Y |
| Elasticsearch | Database |
Y |
| Kibana | Analytics Dashboard |
Y |
| PostgresQl | Database |
Y |
| Redis | Cache |
Y |
| Kafka | Broker |
Y |
| Simple Kafka | Broker |
Y |
| RabbitMQ | Broker |
Y |
1. MongoDb
Copy the env.mongodb.sample file to env.mongodb & set your own credentials
cp .env.mongodb.sample .env.mongodb
a.MONGO_VERSION
b.MONGO_VOLUME_NAME
c.DOCKER_NETWORK
d.ENVIRONMENT VARIABLES = .env.mongodb
>task mongodb-up
>task mongodb-logs
>task mongodb-down2. Emqx
Copy the env.emqx.sample file to env.emqx & set your own credentials
cp .env.emqx.sample .env.emqx
a. EMQX_VERSION
b. EMQX_VOLUME_NAME
c. EMQX_CONTAINER_NAME
d. EMQX_TCP_DEFAULT
e. EMQX_DASHBOARD_DEFAULT
f. ENVIRONMENT VARIABLES = .env.emqx
>task emqx-up
>task emqx-logs
>task emqx-down3. ElasticSearch
Copy the env.elasticsearch.sample file to env.elasticsearch & set your own credentials
cp .env.elasticsearch.sample .env.elasticsearch
Step 1: >task elasticsearch-setup
**Wait for the configuration to successfully exit
for passwords & certificates to be set properly**
>task elasticsearch-up
>task elasticsearch-logs
>task elasticsearch-down
# ------------------------------ Check elastic is working ---------------------------
docker cp omfdydx-elasticsearch:/usr/share/elasticsearch/config/certs/ca/ca.crt /tmp/.
curl --cacert /tmp/ca.crt -u elastic:elasticpasswd https://localhost:3200
# -------------------------------------------------------------------------------------
a. ELASTICSEARCH_VERSION
b. ELASTICSEARCH_VOLUME_NAME
c. ELASTICSEARCH_CERTS_VOLUME_NAME
d. KIBANA_VOLUME_NAME
e. ELASTICSEARCH_TCP_DEFAULT ....et al
f. ENVIRONMENT VARIABLES = .env.elasticsearch
4. Kibana
- It is configured via the elasticsearch env. Nothing else to do
>task kibana-up
>task kibana-logs
>task kibana-down
5. Postgresql
Copy the env.postgresql.sample file to env.postgresql & set your own credentials
cp .env.postgresql.sample .env.postgresql
a. POSTGRESQL_VERSION
b. POSTGRESQL_VOLUME_NAME
c. POSTGRESQL_CONTAINER_NAME
d. POSTGRESQL_TCP_DEFAULT
e. POSTGRESQL_TCP
f. ENVIRONMENT VARIABLES = .env.postgresql
>task postgresql-up
>task postgresql-logs
>task postgresql-down6. Redis
Copy the env.redis.sample file to env.redis & set your own credentials
cp .env.redis.sample .env.redis
a. REDIS_VERSION
b. REDIS_VOLUME_NAME
c. REDIS_CONTAINER_NAME
d. REDIS_TCP_DEFAULT
e. REDIS_TCP
f. ENVIRONMENT VARIABLES = .env.redis
>task redis-up
>task redis-logs
>task redis-down
>task redis-restart7. Kafka
To set up the certificates for secure connections with clients. Kafka setup uses Kraft rather zookeeper
cd scripts && bash create-certs.sh
# Generates the secrets folder with
# - certs,
# - keystore,
# - truststore,
# - pem[Connecting to Kafka via non-Java clients]
# - clients.propertiesCopy the env.kafka.sample file to env.kafka & set your own credentials
cp .env.kafka.sample .env.kafka
a. KAFKA_VERSION
b. KAFKA_ZOOKEEPER_TLS_PASSWORD
c. NON_JAVA_CLIENT_GENERATE_PEM
d. KAFKA_CONTAINER_NAME
e. KAFKA_VOLUME_NAME
f. ENVIRONMENT VARIABLES = .env.kafka
>task kafka-up
>task kafka-logs
>task kafka-down#Checking the consumers can connect via SSL/TLS
# Get into the container 1. producer 2. consumer
docker exec -it omfdydx-kafka bash
# 1. Inside the container **9095** SSL/TLS listener
cd /opt/bitnami/kafka/bin/
./kafka-topics.sh --bootstrap-server kafka:9095 --create --topic inventory \
--partitions 3 --command-config /opt/bitnami/kafka/config/client.properties
> hello: world, good: bye
# 2. Check the consumer
cd /opt/bitnami/kafka/bin/
./kafka-console-consumer.sh --bootstrap-server kafka:9095 \
--topic inventory --consumer.config /opt/bitnami/kafka/config/client.properties --from-beginning
# 3. List the topics
./kafka-topics.sh --list --command-config /opt/bitnami/kafka/config/client.properties \
--bootstrap-server kafka:90958. Simple Kafka - No Auth & Certs - Single Node
Copy the env.safka.sample file to env.safka & set your own credentials
cp .env.safka.sample .env.safka
a. SAFKA_VOLUME_NAME
b. SAFKA_CONTAINER_NAME
c. SAFKA_PORT
d. SAFKA_PORT_DEFAULT
f. ENVIRONMENT VARIABLES = .env.safka
>task safka-up
>task safka-logs
>task safka-down8. RabbitMQ
Copy the env.rabbitmq.sample file to env.rabbitmq & set your own credentials
cp .env.rabbitmq.sample .env.rabbitmq
a. RABBITMQ_VOLUME_NAME
b. RABBITMQ_CONTAINER_NAME
c. RABBITMQ_PORT
d. RABBITMQ_PORT_DEFAULT
f. ENVIRONMENT VARIABLES = .env.rabbitmq
>task rabbitmq-up
>task rabbitmq-logs
>task rabbitmq-down
>task rabbitmq-restart