Monitoring Fundamentals

Observability is the ability to understand the internal state of a system by examining its external outputs. In cloud environments, this rests on three pillars: metrics, logs, and traces.

Metrics are numerical measurements collected over time. CPU utilization, request latency percentiles, error rates, and queue depths are all metrics. They answer questions like "Is the system healthy right now?" and "How has performance changed over the last hour?" Metrics are lightweight, easy to store, and ideal for dashboards and alerting.

Logs are discrete timestamped records of events. An application log entry might include a request ID, user identifier, status code, and processing duration. Logs provide the richest context for debugging individual failures. A web server access log showing a 503 status code tells you a request failed; the application log for that same request reveals whether the database connection timed out or a downstream service returned an error.

Traces track a single request as it flows through multiple services. In a microservices architecture, one user action might trigger calls to an API gateway, authentication service, order service, payment gateway, and notification service. A trace captures the entire journey, showing latency at each hop and where failures occurred. This is essential for diagnosing performance bottlenecks in distributed systems.

The key insight is that no single pillar is sufficient. Metrics tell you something is wrong. Traces tell you which service is slow. Logs tell you why. A mature observability strategy uses all three in concert, correlated by common identifiers like request IDs or trace IDs.

Service Level Indicators (SLIs) are the raw measurements that quantify service behavior. Common SLIs include availability (percentage of successful requests), latency (percentage of requests faster than a threshold), and throughput (requests per second). For a REST API, your SLIs might be: availability = successful responses / total requests; latency = percentage of requests completing in under 200ms.

Service Level Objectives (SLOs) are target values for your SLIs. An SLO of 99.9% availability means you aim for no more than 0.1% of requests to fail. SLOs are internal commitments that guide engineering decisions. They represent a balance between reliability and cost—achieving 99.999% availability requires significantly more infrastructure investment than 99.9%.

Service Level Agreements (SLAs) are contractual promises made to customers. An SLA might guarantee 99.5% monthly uptime and define penalties (service credits) if the target is missed. SLAs are always less stringent than SLOs—your internal target must be tighter than your external promise to provide a safety margin.

Error budgets are derived from SLOs. If your SLO is 99.9% availability, your error budget is 0.1%—roughly 43 minutes of downtime per month. Error budgets create a shared language between engineering and business teams. When the error budget is healthy, teams can ship aggressively. When it is depleted, teams must prioritize reliability over features.

Consider an e-commerce platform with an SLO of 99.95% for the checkout API. The error budget allows approximately 22 minutes of downtime per month. If a deployment causes 15 minutes of elevated error rates, the remaining budget is only 7 minutes. This data-driven approach prevents teams from either being too cautious (never deploying) or too reckless (constantly breaking things).

A monitoring strategy answers three questions: what to monitor, how to alert, and who to notify.

What to monitor follows the RED and USE methodologies. RED (Rate, Errors, Duration) focuses on request-driven services—track request rate, error rate, and latency distribution. USE (Utilization, Saturation, Errors) focuses on infrastructure—track resource utilization (CPU, memory), saturation (queue depth, connection count), and hardware errors. For databases, also monitor connection pools, replication lag, and query performance.

Alerting thresholds should be based on SLOs, not arbitrary values. Alert when the error budget is being consumed at an unsustainable rate. For latency, use percentile-based thresholds (p99 rather than average) because averages hide tail latency. A service with p50 of 50ms and p99 of 5 seconds appears healthy on average but is failing 1% of users.

Alert design principles include: every alert should require human action (page-worthy), alerts should include context (runbook links, dashboard links), and alerts should have clear severity levels. Critical alerts page immediately; warning alerts create tickets; informational alerts appear on dashboards only.

Escalation paths define what happens when an alert fires. A tier-1 on-call engineer investigates within 5 minutes. If unresolved after 15 minutes, escalate to tier-2. If customer impact is confirmed, engage the incident commander. Every escalation tier should have clear ownership and communication channels.

For a production API serving 10,000 requests per second, a practical strategy might include: RED metrics on the API gateway, USE metrics on each EC2 instance, p99 latency alert at 500ms (SLO is 99.9% under 500ms), error rate alert at 0.1%, and a synthetic canary that runs every 60 seconds from multiple regions.