Structured Logging

Structured vs Unstructured

Unstructured:
[2026-01-01 10:30:00] INFO: Order 12345 processed for customer 678

Structured (JSON):
{
  "timestamp": "2026-01-01T10:30:00Z",
  "level": "info",
  "message": "Order processed",
  "order_id": 12345,
  "customer_id": 678,
  "total": 99.99,
  "service": "order-service",
  "request_id": "req-abc-123"
}

Magento JSON Logging Config

// app/etc/di.xml
<type name="Magento\Framework\Logger\Monolog">
    <arguments>
        <argument name="handlers" xsi:type="array">
            <item name="json" xsi:type="object">Magento\Framework\Logger\Handler\JsonHandler</item>
        </argument>
    </arguments>
</type>

// Custom JSON formatter
class JsonFormatter extends \Monolog\Formatter\JsonFormatter {
    public function format(array $record): string {
        $record['service'] = 'magento';
        $record['environment'] = getenv('APP_ENV');
        return parent::format($record);
    }
}

Log Structure Schema

{
  "timestamp": "ISO 8601",
  "level": "info|warn|error|debug",
  "message": "Human readable message",
  "service": "service name",
  "request_id": "unique request identifier",
  "trace_id": "distributed trace ID",
  "user_id": "authenticated user",
  "context": {
    "key": "additional context"
  }
}

Environment-Based Levels

Environment  | Level   | Purpose
─────────────|─────────|────────────────────
Development  | DEBUG   | Full diagnostic info
Staging      | INFO    | Normal operations
Production   | WARNING | Only significant events
Incident     | DEBUG   | Temporary increase

Level Usage Guidelines

// DEBUG: Diagnostic details (disabled in production)
$this->logger->debug('SQL query executed', [
    'query' => $sql,
    'duration_ms' => $duration,
    'rows_affected' => $count
]);

// INFO: Normal business events
$this->logger->info('Order placed', [
    'order_id' => $order->getId(),
    'total' => $order->getTotal()
]);

// WARNING: Unexpected but handled
$this->logger->warning('Slow query detected', [
    'query' => $sql,
    'duration_ms' => $duration,
    'threshold_ms' => 1000
]);

// ERROR: Operation failed
$this->logger->error('Payment processing failed', [
    'order_id' => $order->getId(),
    'error' => $e->getMessage()
]);

// CRITICAL: System failure
$this->logger->critical('Database connection lost', [
    'host' => $host,
    'error' => $e->getMessage()
]);

Aggregation Architecture

Magento → File Logs → Filebeat → Logstash → Elasticsearch → Kibana
         (JSON)       (Ship)    (Process)    (Store)       (Visualize)

ELK Stack Config

# filebeat.yml
filebeat.inputs:
  - type: log
    paths:
      - /var/log/magento/*.log
    json.keys_under_root: true
    json.add_error_key: true

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "magento-logs-%{+yyyy.MM.dd}"

# Logstash pipeline
input {
  beats {
    port => 5044
  }
}

filter {
  json {
    source => "message"
  }
  date {
    match => ["timestamp", "ISO8601"]
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "magento-logs-%{+YYYY.MM.dd}"
  }
}

Cloud-Native Options

AWS:    CloudWatch Logs + OpenSearch
GCP:    Cloud Logging + BigQuery
Azure:  Log Analytics + Sentinel
SaaS:   Datadog, Splunk, New Relic

Common Queries

# KQL (Kibana Query Language)
level: error AND service: order-service

# Find slow queries
duration_ms > 1000 AND message: "query"

# Errors by service
level: error | stats count by service

# Error rate over time
level: error | timechart span=1h count

Log Analysis Dashboard

Key Metrics:
- Total log volume per hour
- Error rate by service
- Top error messages
- Slow request distribution
- Log source distribution

Alerts:
- Error rate > 1%: Warning
- Error rate > 5%: Critical
- Log volume drop > 50%: Warning

Log Retention

Retention Policy:
- Hot (Elasticsearch): 30 days
- Warm (S3): 90 days
- Cold (Glacier): 1 year
- Archive: 7 years (compliance)

Index Lifecycle:
- Daily indices
- ILM policy for rollover
- Delete after retention period