Skip to content
advanced Phase 83 · Observability Basics

Distributed Tracing

Distributed tracing with Jaeger, Zipkin, trace spans, and service dependency mapping

45m
0 problems
Topic Progress 0%

Jaeger Architecture

Jaeger Components

┌─────────────┐     ┌─────────────┐
│ Application │────▶│   Jaeger    │
│  (Tracer)   │     │  Collector  │
└─────────────┘     └──────┬──────┘
                           │
                    ┌──────┴──────┐
                    │   Jaeger    │
                    │    Query    │
                    └──────┬──────┘
                           │
                    ┌──────┴──────┐
                    │   Jaeger    │
                    │     UI      │
                    └─────────────┘

Magento Jaeger Config

// composer.json
require {
    "jaeger/phpencody": "^1.0"
}

// Initialize Jaeger tracer
use Jaeger\Tracer;
use Jaeger\Transport\HttpTransport;

$transport = new HttpTransport('http://jaeger:14268/api/traces');
$tracer = new Tracer('magento', $transport);

// Register as service
$this->registry->register('jaeger_tracer', $tracer);

Span Creation

// Create a span for order processing
$tracer = $this->registry->registry('jaeger_tracer');
$span = $tracer->startSpan('process_order');

$span->setTag('order.id', $orderId);
$span->setTag('customer.id', $customerId);
$span->setTag('component', 'order-service');

try {
    // Process order
    $this->paymentService->charge($order);
    $span->setTag('payment.status', 'success');
} catch (Exception $e) {
    $span->setTag('error', true);
    $span->log(['message' => $e->getMessage()]);
    throw $e;
} finally {
    $span->finish();
}

Trace Spans Structure

Span Hierarchy

Trace: POST /checkout (290ms)
├── Span: API Gateway (10ms)
├── Span: Order Service (280ms)
│   ├── Span: Validate Cart (20ms)
│   ├── Span: Create Order (50ms)
│   │   └── Span: MySQL INSERT (15ms)
│   ├── Span: Process Payment (200ms)
│   │   ├── Span: Gateway Request (180ms)
│   │   └── Span: Update Order Status (10ms)
│   └── Span: Reserve Inventory (30ms)
│       └── Span: Redis DECR (5ms)
└── Span: Send Confirmation (10ms)
    └── Span: Email Service (8ms)

Span Attributes

Attribute          | Description               | Example
───────────────────|───────────────────────────|────────────────
span.kind          | Client/Server             | server
component          | Service name              | order-service
order.id           | Business entity           | 12345
duration_ms        | Span duration             | 200
error              | Error occurred            | true
peer.service       | Called service            | payment-service

Span Events

// Log events within span
$span->log([
    'event' => 'cache_miss',
    'key' => 'product_123',
    'cache_backend' => 'redis'
]);

$span->log([
    'event' => 'sql_query',
    'query' => 'SELECT * FROM orders WHERE id = ?',
    'duration_ms' => 15
]);

Service Dependencies

Dependency Graph

┌─────────────┐     ┌─────────────┐
│   Client    │────▶│     API     │
└─────────────┘     └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              â–¼            â–¼            â–¼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │  Order   │ │  Search  │ │  Catalog │
        └────┬─────┘ └──────────┘ └──────────┘
             │
        ┌────┼────┐
        â–¼    â–¼    â–¼
    ┌──────┐ ┌──────┐ ┌──────┐
    │  DB  │ │Redis │ │  ES  │
    └──────┘ └──────┘ └──────┘

Dependency Extraction

// Extract dependencies from traces
function extractDependencies($traces) {
    $dependencies = [];

    foreach ($traces as $trace) {
        foreach ($trace['spans'] as $span) {
            $source = $span['service'];
            $target = $span['tags']['peer.service'] ?? null;

            if ($target) {
                $key = "$source->$target";
                $dependencies[$key] = ($dependencies[$key] ?? 0) + 1;
            }
        }
    }

    return $dependencies;
}

Dependency Metrics

Service → Service    | Calls/min | Avg Latency | Error Rate
─────────────────────|───────────|─────────────|─────────────
API → Order          | 500       | 280ms       | 0.5%
Order → Payment      | 500       | 200ms       | 1.2%
Order → Inventory    | 500       | 30ms        | 0.1%
API → Search         | 1000      | 50ms        | 0.2%

Trace Analysis

Performance Analysis

-- Find slow traces
SELECT trace_id, duration_ms
FROM jaeger.traces
WHERE operation = 'process_order'
  AND duration_ms > 1000
ORDER BY duration_ms DESC
LIMIT 10;

-- Analyze latency distribution
SELECT 
    duration_ms,
    COUNT(*) as count
FROM jaeger.traces
WHERE operation = 'process_order'
GROUP BY duration_ms / 100 * 100
ORDER BY duration_ms;

Error Analysis

-- Find error traces
SELECT trace_id, error_message
FROM jaeger.spans
WHERE error = true
  AND timestamp > NOW() - INTERVAL 1 HOUR;

-- Error rate by service
SELECT service, 
       COUNT(*) as total,
       SUM(CASE WHEN error THEN 1 ELSE 0 END) as errors,
       ROUND(SUM(CASE WHEN error THEN 1 ELSE 0 END) * 100.0 / COUNT(*), 2) as error_rate
FROM jaeger.spans
WHERE timestamp > NOW() - INTERVAL 1 HOUR
GROUP BY service;

Alerting on Traces

Alert Conditions:
- P99 latency > 2s: Warning
- P99 latency > 5s: Critical
- Error rate > 5%: Critical
- Missing spans in trace: Warning

Quiz

1. What is a trace span?

Question 1 options

2. What does the dependency graph show?

Question 2 options

3. How do you identify slow operations from traces?

Question 3 options

Flashcards

Question

Trace span?

Answer

Single unit of work within a distributed trace

Question

Span hierarchy shows?

Answer

Parent-child relationships between operations

Question

Service dependency graph?

Answer

Maps which services call which, with latency and error rates

Question

Jaeger components?

Answer

Agent, Collector, Query, UI

Revision Notes

Key Takeaways

  • 1. Spans represent individual operations within a trace
  • 2. Span hierarchy shows parent-child operation relationships
  • 3. Service dependencies extracted from trace data reveal architecture
  • 4. Trace analysis identifies slow operations and error patterns
  • 5. Alert on P99 latency and error rate thresholds

Interview Tips

  • Explain trace span structure and hierarchy
  • Discuss how to extract service dependencies from traces
  • Describe trace analysis for performance optimization

Cheat Sheet

Distributed Tracing:
  Jaeger: Open-source tracing system
  Zipkin: Alternative tracing system
  OpenTelemetry: Vendor-neutral standard

Spans:
  Single unit of work
  Parent-child hierarchy
  Tags: metadata attributes
  Events: point-in-time logs

Analysis:
  Slow traces: duration > threshold
  Error traces: error = true
  Dependencies: service → service calls

Alerting:
  P99 latency > 2s: Warning
  Error rate > 5%: Critical