Centralized Logging

CloudWatch Logs is the native log management service. Every log stream belongs to a log group—by convention, use /aws/service-name/environment. For example, /aws/ecs/web-api/production groups all logs from the production web API ECS service.

Log groups control retention, permissions, and data protection. Set retention policies per group (7 days for dev, 90 days for prod, indefinite for compliance). Use resource policies to allow cross-account log access—critical in multi-account architectures where a central security account needs access to all account logs.

Subscription filters stream logs from CloudWatch Logs to other services in real time. Route logs to Lambda for processing, Kinesis Data Firehose for batch delivery to S3, or OpenSearch Service for indexing and search. A subscription filter can select logs matching a pattern—only send ERROR and FATAL logs to the alerting Lambda function, reducing cost and noise.

CloudWatch Logs Insights provides interactive querying without infrastructure. Run SQL-like queries against your logs: fields @timestamp, @message | filter @message like /Exception/ | sort @timestamp desc | limit 20 finds the most recent exceptions. Logs Insights is ideal for ad-hoc debugging but not for continuous analytics—that requires OpenSearch.

Log format standardization is critical for queryability. Use JSON log format with consistent fields: timestamp, level, service, traceId, message, and structured metadata. A well-structured log entry might be: {"timestamp":"2024-01-15T10:30:00Z","level":"ERROR","service":"payment-service","traceId":"abc-123","message":"Payment failed","amount":99.99,"currency":"USD"}.

Cost optimization: CloudWatch Logs ingestion costs $0.50/GB. Use subscription filters to route only valuable logs to long-term storage. Archive raw logs to S3 ($0.023/GB) and query with Athena ($5/TB scanned) for cost-effective historical analysis.

Amazon OpenSearch Service (formerly Elasticsearch) provides full-text search and analytics for log data. It excels where CloudWatch Logs Insights falls short: large volumes, complex aggregations, and visualization with OpenSearch Dashboards (formerly Kibana).

Architecture: Logs flow from CloudWatch Logs through a subscription filter to Kinesis Data Firehose, which batches and delivers to an S3 bucket. OpenSearch Service ingests from S3 and indexes the data. This decoupled architecture handles volume spikes—Firehose buffers data during traffic surges and OpenSearch processes it at its own pace.

Index management controls cost and performance. Use index lifecycle management (ILM) to automatically transition logs through hot, warm, cold, and delete phases. Hot indices (recent, actively queried) run on expensive SSD instances. Warm indices (older, occasionally queried) move to cheaper storage-optimized instances. Cold indices (rarely accessed) move to ultra-cheap storage. Delete after retention expires.

Dashboards in OpenSearch Dashboards provide visual log analysis. Create a dashboard with: a line chart showing error rate over time, a data table of top 10 error messages, a pie chart of errors by service, and a saved search for debugging specific trace IDs. Dashboards update in near real-time as new logs are indexed.

Use case: A platform team managing 50 microservices centralizes all application logs into OpenSearch. When an on-call engineer receives an alert, they open the dashboard, filter by service and time range, view correlated errors, drill into specific trace IDs, and identify the root cause—all within minutes instead of SSH-ing into individual instances and grepping log files.

Cost considerations: OpenSearch instances cost $0.10-$6.00/hour depending on size. A production cluster with 3 data nodes, 3 master nodes, and dedicated storage might cost $500-$2000/month. Offset this by aggressive ILM policies and only indexing fields you actually query.

For long-term log retention and compliance, S3 is the destination. CloudWatch Logs can export directly to S3, or Kinesis Data Firehose can deliver formatted log files. Store logs in a structured format—Parquet is ideal for Athena queries because it's columnar and compressed.

Partition strategy is critical for Athena performance and cost. Partition logs by date: s3://logs-bucket/service-name/year=2024/month=01/day=15/. Athena only scans partitions matching your query's WHERE clause, reducing cost from $5/TB to fractions of a cent for targeted queries.

AWS Glue crawlers automatically discover partition structure and update the Athena catalog. Schedule crawlers to run hourly or daily so new partitions are available for querying. Glue also handles schema evolution if log formats change.

Athena queries use standard SQL against S3-stored logs. Common patterns include: finding all 500 errors in the last 7 days, calculating average response time by hour, identifying the slowest API endpoints, and generating compliance reports showing all access events for a specific user.

Example query: SELECT service, COUNT(*) as error_count FROM logs WHERE year='2024' AND month='01' AND level='ERROR' AND dt BETWEEN '2024-01-15' AND '2024-01-21' GROUP BY service ORDER BY error_count DESC. This scans only the relevant partitions and returns error counts per service for a specific week.

Cost optimization: Athena costs $5 per TB scanned. Partitioning reduces scanned data dramatically. Use columnar formats (Parquet, ORC) instead of JSON—Parquet scans 10-100x less data for column-specific queries. Compress logs with Snappy or Gzip. A well-partitioned Parquet archive of 1TB of raw logs might query for $0.05 instead of $5.

Compliance: Many regulations require log retention for 1-7 years. S3 Glacier Deep Archive ($0.00099/GB/month) stores 1TB for approximately $1/month. Combine with S3 lifecycle policies to automatically transition logs from S3 Standard to Glacier after 90 days, then to Deep Archive after 1 year.