Quickly Set Up A Development Environment

Here's how to quickly set up a development environment with MySQL, PostgreSQL, MongoDB, Redis and Kafka using Docker Compose, with bitnami images, environment variables, and UI tools for each database. We'll go through the process step by step:

Step 1: Install Docker and Docker Compose

1. Install Docker: Follow the instructions for your OS from the official Docker documentation.
2. Install Docker Compose: Follow instructions from the Docker Compose installation guide

Step 2: Project Structure

Create the following project structure:

dev-environment/
├── .env
├── docker-compose.yml
├── scripts/
│ ├── mongo-init.sh
│ ├── postgres-init.sql
│ ├── create-topics.sh

Step 3: (.env) File

Create a .env file with the following content:

# MySQL Configuration
MYSQL_PORT=23306
MYSQL_USERNAME=dev-user
MYSQL_PASSWORD=dev-password
MYSQL_DATABASE=dev_database # PostgreSQL Configuration
POSTGRES_PORT=25432
POSTGRES_USERNAME=dev-user
POSTGRES_PASSWORD=dev-password
POSTGRES_DATABASE=dev_database # MongoDB Configuration
MONGO_PORT=22717
MONGO_USERNAME=dev-user
MONGO_PASSWORD=dev-password
MONGO_DATABASE=dev_database # Redis Configuration
REDIS_PORT=26379
REDIS_PASSWORD=dev-password # Kafka Configuration
KAFKA_PORT=29092
KAFKA_USERNAME=dev-user
KAFKA_PASSWORD=dev-password # UI Tools Configuration
PHPMYADMIN_PORT=280
PGADMIN_PORT=281
MONGOEXPRESS_PORT=28081
REDIS_COMMANDER_PORT=28082
KAFKA_UI_PORT=28080 # Data Directory for Volumes
DATA_DIR=./

Step 4: (docker-compose.yml) File

Create the docker-compose.yml file:

version: '3.8'
services: dev-mysql: image: bitnami/mysql:8.0.34 container_name: dev-mysql ports: - '${MYSQL_PORT}:3306' environment: - MYSQL_ROOT_PASSWORD=${MYSQL_PASSWORD} - MYSQL_USER=${MYSQL_USERNAME} - MYSQL_PASSWORD=${MYSQL_PASSWORD} - MYSQL_DATABASE=${MYSQL_DATABASE} volumes: - '${DATA_DIR}/components/mysql/data:/var/lib/mysql' dev-postgresql: image: bitnami/postgresql:15.3.0 container_name: dev-postgresql ports: - '${POSTGRES_PORT}:5432' environment: - POSTGRESQL_USERNAME=${POSTGRES_USERNAME} - POSTGRESQL_PASSWORD=${POSTGRES_PASSWORD} - POSTGRESQL_DATABASE=${POSTGRES_DATABASE} volumes: - '${DATA_DIR}/components/postgresql/data:/var/lib/postgresql/data' - ./scripts/pg-init.sh:/docker-entrypoint-initdb.d/init.sql:ro dev-mongodb: image: bitnami/mongodb:6.0.8 container_name: dev-mongodb ports: - '${MONGO_PORT}:27017' environment: - MONGO_INITDB_ROOT_USERNAME=${MONGO_USERNAME} - MONGO_INITDB_ROOT_PASSWORD=${MONGO_PASSWORD} - MONGO_INITDB_DATABASE=${MONGO_DATABASE} volumes: - '${DATA_DIR}/components/mongodb/data/db:/data/db' - '${DATA_DIR}/components/mongodb/data/logs:/data/logs' - ./scripts/mongo-init.sh:/docker-entrypoint-initdb.d/mongo-init.sh:ro dev-redis: image: bitnami/redis:7.0.12 container_name: dev-redis ports: - '${REDIS_PORT}:6379' environment: - REDIS_PASSWORD=${REDIS_PASSWORD} volumes: - '${DATA_DIR}/components/redis/data:/bitnami/redis' dev-kafka: image: bitnami/kafka:3.4.0 ports: - '${KAFKA_PORT}:9092' environment: - KAFKA_BROKER_ID=1 - KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://dev-kafka:9092 - KAFKA_DISABLE_LOG4J_CONFIG=true - KAFKA_LOG_DIRS=/bitnami/kafka/data - KAFKA_SASL_ENABLED=true - KAFKA_SASL_MECHANISM=PLAIN - KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT - KAFKA_SASL_USERS=${KAFKA_USERNAME}:${KAFKA_PASSWORD} volumes: - '${DATA_DIR}/components/kafka/data:/bitnami/kafka/data' - ./scripts/create-topics.sh:/opt/bitnami/kafka/create_topic.sh:ro command: > bash -c " /opt/bitnami/scripts/kafka/run.sh & sleep 5; /opt/bitnami/kafka/create_topic.sh; wait " phpmyadmin: image: phpmyadmin/phpmyadmin:5.2.1 container_name: dev-phpmyadmin depends_on: - dev-mysql ports: - '${PHPMYADMIN_PORT}:80' environment: - PMA_HOST=dev-mysql - PMA_PORT=3306 - PMA_USER=${MYSQL_USERNAME} - PMA_PASSWORD=${MYSQL_PASSWORD} pgadmin: image: dpage/pgadmin4:7.6 container_name: dev-pgadmin depends_on: - dev-postgresql ports: - '${PGADMIN_PORT}:80' environment: - PGADMIN_DEFAULT_EMAIL=admin@dev.com - PGADMIN_DEFAULT_PASSWORD=${POSTGRES_PASSWORD} volumes: - '${DATA_DIR}/components/pgadmin:/var/lib/pgadmin' mongo-express: image: mongo-express:1.0.0 container_name: dev-mongoexpress depends_on: - dev-mongodb ports: - '${MONGOEXPRESS_PORT}:8081' environment: - ME_CONFIG_MONGODB_ADMINUSERNAME=${MONGO_USERNAME} - ME_CONFIG_MONGODB_ADMINPASSWORD=${MONGO_PASSWORD} - ME_CONFIG_MONGODB_SERVER=dev-mongodb redis-commander: image: rediscommander/redis-commander:1.12.1 container_name: dev-redis-commander depends_on: - dev-redis ports: - '${REDIS_COMMANDER_PORT}:8081' environment: - REDIS_HOSTS=local:dev-redis:6379 - REDIS_PASSWORD=${REDIS_PASSWORD} kafka-ui: image: provectuslabs/kafka-ui:v0.7.2 container_name: dev-kafka-ui ports: - '${KAFKA_UI_PORT}:8080' environment: - KAFKA_CLUSTERS_0_NAME=local - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=dev-kafka:9092 - KAFKA_CLUSTERS_0_SASL_USER=${KAFKA_USERNAME} - KAFKA_CLUSTERS_0_SASL_PASSWORD=${KAFKA_PASSWORD} - KAFKA_CLUSTERS_0_SASL_MECHANISM=PLAIN - KAFKA_CLUSTERS_0_SECURITY_PROTOCOL=SASL_PLAINTEXT depends_on: - dev-kafka

Step 5: Scripts

Create the necessary scripts in the scripts folder.

1. mongo-init.sh:

   #!/bin/bash mongo -- "$MONGO_INITDB_DATABASE" <<EOF
   db = db.getSiblingDB('admin')
   db.auth('$MONGO_INITDB_ROOT_USERNAME', '$MONGO_INITDB_ROOT_PASSWORD')
   db = db.getSiblingDB('$MONGO_INITDB_DATABASE')
   db.createUser({ user: "$MONGO_USERNAME", pwd: "$MONGO_PASSWORD", roles: [ { role: 'root', db: '$MONGO_INITDB_DATABASE' } ]
   })
   EOF echo "MongoDB initialization script executed"
   

2. postgres-init.sql:

   CREATE TABLE IF NOT EXISTS test (id SERIAL PRIMARY KEY, name VARCHAR(50));
   

3. create-topics.sh:

   #!/bin/bash # Wait for Kafka to be ready
   until /opt/bitnami/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092; do echo "Waiting for Kafka to be ready..." sleep 2
   done # Create the topic
   /opt/bitnami/kafka/bin/kafka-topics.sh --create --topic devTopic --replication-factor 1 --partitions 8 --bootstrap-server dev-kafka:9092 echo "Topics created."
   

Step 6: Run Docker Compose

In your terminal, navigate to the dev-environment folder and run:

docker-compose up -d

This command will start all the services, each with its own container, port, and environment configuration as defined.

Step 7: Access the Databases Using UI Tools

• phpMyAdmin: Access via http://localhost:280
• Mongo Express: Access via http://localhost:28081
• pgAdmin 4: Access via http://localhost:281
• Redis Commander: Access via http://localhost:28082
• Kafka UI: Access via http://localhost:28080

Each UI tool is already configured to connect to its respective database container.

Step 8: Access via CLI

1. Access MySQL CLI

• To access the MySQL database inside the dev-mysql container:

  docker exec -it dev-mysql mysql -u${MYSQL_USERNAME} -p${MYSQL_PASSWORD} ${MYSQL_DATABASE}
  

2. Access PostgreSQL CLI

• To access the PostgreSQL database inside the dev-postgresql container:

  docker exec -it dev-postgresql psql -U ${POSTGRES_USERNAME} -d ${POSTGRES_DATABASE}
  

3. Access MongoDB CLI

• To access the MongoDB shell inside the dev-mongodb container:

  docker exec -it dev-mongodb mongo -u ${MONGO_USERNAME} -p ${MONGO_PASSWORD} --authenticationDatabase ${MONGO_DATABASE}
  

4. Access Redis CLI

• To access the Redis CLI inside the dev-redis container:

  docker exec -it dev-redis redis-cli -a ${REDIS_PASSWORD}
  

4. Access Kafka CLI

• To access the Kafka CLI inside the dev-kafka container:

   docker exec -it dev-kafka /opt/bitnami/kafka/bin/kafka-console-consumer.sh --bootstrap-server dev-kafka:9092 --topic devTopic --from-beginning
  

Summary

This setup uses Docker Compose with environment variables, bitnami images, and volume mappings to create a reproducible development environment. By using docker-compose up, you can quickly spin up or tear down the entire environment with docker-compose down, making it suitable for local development and testing.