Logging & Monitoring

Docker Logging Drivers

Docker supports multiple logging drivers that determine where container logs are sent.

Default: json-file

# Check current driver
docker info | grep "Logging Driver"
# Logging Driver: json-file

# Logs stored at
ls /var/lib/docker/containers/<container_id>/<container_id>-json.log

# View logs
docker logs web

Configure Logging Driver

# Per-container
docker run --log-driver=json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  nginx

# Global (daemon.json)
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

Available Drivers

Fluentd Driver

docker run --log-driver=fluentd \
  --log-opt fluentd-address=localhost:24224 \
  --log-opt tag="docker.{{.Name}}" \
  nginx

AWS CloudWatch

docker run --log-driver=awslogs \
  --log-opt awslogs-group=/ecs/myapp \
  --log-opt awslogs-region=us-east-1 \
  --log-opt awslogs-stream-prefix=web \
  nginx

Syslog Driver

docker run --log-driver=syslog \
  --log-opt syslog-address=tcp://192.168.1.10:514 \
  --log-opt tag="docker/web" \
  nginx

None Driver

# Disable logging entirely
docker run --log-driver=none nginx

# Useful for high-throughput containers where logging causes I/O

Log Rotation and Management

The Problem

Without log rotation, container logs grow unbounded and can fill the disk:

# Check log sizes
du -sh /var/lib/docker/containers/*/
# 2.1G  /var/lib/docker/containers/a1b2c3.../a1b2c3...-json.log

Log Rotation Configuration

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",     // Max size per log file
    "max-file": "5",       // Max number of log files
    "compress": "true"     // Compress rotated logs
  }
}

How it works:

• max-size: When a log file reaches 50MB, it's rotated
• max-file: Docker keeps up to 5 rotated files (oldest is deleted)
• compress: Rotated files are gzip compressed
• Total max per container: 50MB × 5 = 250MB

Per-Container Override

docker run --log-opt max-size=100m --log-opt max-file=10 nginx

systemd Journal Rotation

If using journald driver:

# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=1G
SystemMaxFileSize=100M
MaxRetentionSec=30day

Docker Compose Log Configuration

services:
  api:
    image: myapi
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
        compress: "true"

Manual Log Cleanup

# Truncate log file without stopping container
docker truncate $(docker inspect --format '{{.LogPath}}' web)

# Or use Docker's logrotate
docker system prune -f

# Remove all stopped container logs
docker container prune -f

Monitoring Log Growth

# Watch log size
du -sh /var/lib/docker/containers/*/

# Set up alerting
watch -n 60 'docker system df | grep Containers'

# Cron job for cleanup
0 2 * * * docker system prune -f --filter "until=24h"

Container Monitoring

Docker Stats

# Real-time resource usage
docker stats

# One-time snapshot
docker stats --no-stream

# Custom format
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}\t{{.BlockIO}}"

# Specific container
docker stats web api db

# Output:
# NAME   CPU %   MEM USAGE / LIMIT     NET I/O          BLOCK I/O
# web    0.5%    50MiB / 256MiB       1.2MB / 800kB    12MB / 0B
# api    2.1%    120MiB / 512MiB      5MB / 3MB        45MB / 12MB
# db     1.0%    256MiB / 1GiB        2MB / 1MB        120MB / 50MB

cAdvisor (Container Advisor)

# Run cAdvisor
docker run -d \
  --name cadvisor \
  --privileged \
  -p 8080:8080 \
  -v /:/rootfs:ro \
  -v /var/run:/var/run:ro \
  -v /sys:/sys:ro \
  -v /var/lib/docker/:/var/lib/docker:ro \
  gcr.io/cadvisor/cadvisor:latest

# Access at http://localhost:8080

Prometheus + Grafana

# docker-compose.yml for monitoring stack
services:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - grafana_data:/var/lib/grafana

  node-exporter:
    image: prom/node-exporter
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'

volumes:
  grafana_data:

Health Checks in Production

services:
  api:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    # Container state: healthy | unhealthy | starting

  db:
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s

Log Aggregation with ELK

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    volumes:
      - es_data:/usr/share/elasticsearch/data

  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200

volumes:
  es_data: