Distributed Tracing

AWS X-Ray captures the path of a request through your application. A trace represents the full journey from the initial client request to the final response. Each trace contains one or more segments—one per service or resource that processes the request. Within segments, subsegments represent discrete operations: database queries, HTTP calls to external services, or queue operations.

How it works: The X-Ray SDK instruments your application code. For a Python Flask application, adding the X-Ray SDK with @xray_recorder.capture('my_function') wraps function calls in subsegments. The SDK records timing, errors, and metadata automatically. When the request hits an EC2 instance with the X-Ray daemon, the daemon buffers trace data and uploads it to X-Ray in batches.

Service maps are automatically generated visualizations of your architecture. X-Ray analyzes trace data and constructs a graph showing services, their connections, latency, error rates, and throughput. If your API calls a user service, which calls a database, the service map shows each hop with average latency and error percentage. This is invaluable for identifying bottlenecks—a service with 500ms average latency when all others are under 50ms is an obvious target for optimization.

Concrete example: A user places an order. The trace shows: API Gateway (10ms) → Order Service (200ms) → Payment Service (800ms) → Stripe API (750ms) → Order Service (10ms) → DynamoDB PutItem (15ms). The service map reveals Stripe API latency dominates. Without tracing, you might optimize DynamoDB reads when the real bottleneck is an external dependency.

X-Ray groups organize traces by attributes. Create a group for traces with fault = true to see only failed requests. Create a group for annotation.service = payment-service to focus on payment traces. Groups power the X-Ray console views and can be used with CloudWatch Synthetics for end-to-end monitoring.

Tracing every request is expensive and often unnecessary. Sampling rules control which requests are traced. X-Ray provides a default sampling rule: 1 request per second and 5% of additional requests. This captures enough data for monitoring while controlling cost.

Custom sampling rules let you define rules per service, path, or attribute. For example, trace 100% of requests to the checkout endpoint (high business value), 10% of requests to the health check (low value), and 1% of requests from internal services. Sampling rules use reservoir rates (fixed number per second) and percentage rules for fine-grained control.

Annotations are indexed, searchable key-value pairs attached to traces. Unlike metadata (which is stored but not searchable), annotations enable filtering in the X-Ray console. Tag traces with customer_id, order_id, or error_type to quickly find traces matching specific criteria. For debugging a production issue, search for annotation.error_type = payment_declined to find all traces where payments failed.

Metadata provides additional context not needed for search. Attach full request payloads, response bodies, or configuration details as metadata. Metadata is stored with the trace and visible in the trace timeline but doesn't affect search performance or cost.

Error handling: The SDK captures exceptions automatically, recording fault type (client error vs. server fault), error message, and stack trace. Combine this with annotations: catch the exception, annotate with the error code, then re-throw. This creates a searchable index of error types across all traces.

Naming conventions: Use consistent segment and subsegment names. Instead of naming subsegments by variable values (which creates unlimited unique names), use static names: dynamodb:GetItem rather than dynamodb:GetItem-user-123. Unlimited unique names explode the service map and increase costs.

OpenTelemetry is a vendor-neutral, open-source observability framework. It provides SDKs for generating traces, metrics, and logs that can be exported to any backend—X-Ray, Jaeger, Datadog, or Grafana Tempo. The key advantage is avoiding vendor lock-in: instrument once, export anywhere.

OpenTelemetry Collector acts as a proxy between your application and backends. It receives traces in OpenTelemetry format, processes them (add attributes, filter sensitive data, batch for efficiency), and exports to one or more backends. Deploy the collector as a sidecar, DaemonSet, or centralized service.

AWS Distro for OpenTelemetry (ADOT) is AWS's managed distribution. It includes the collector with AWS-specific exporters for X-Ray, CloudWatch, and Prometheus. ADOT is the recommended way to use OpenTelemetry on AWS, as it's tested and supported by AWS.

CloudWatch ServiceLens combines traces, metrics, and logs into a unified view. Enable ServiceLens by sending X-Ray traces to CloudWatch and CloudWatch Logs to CloudWatch. ServiceLens provides: a service map with latency and error data overlaid, trace analytics for finding slow traces, correlation between traces and logs (click a trace to see its logs), and anomaly detection on trace-derived metrics.

Trace analytics in ServiceLens enables powerful queries. Find all traces where the payment service took more than 1 second and returned a 500 error. Identify the slowest 1% of requests to the order service. Compare trace latency between production and staging. These queries combine trace data with CloudWatch's analytics capabilities.

Practical comparison: For an AWS-native stack with moderate complexity, X-Ray with ServiceLens provides a managed, integrated experience. For multi-cloud or complex microservices architectures, OpenTelemetry with the ADOT collector offers flexibility. Many teams start with X-Ray for simplicity and migrate to OpenTelemetry as their observability needs grow.