Messaging and Event Services

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that decouples producers and consumers. Messages are stored reliably until processed.

Standard queues offer maximum throughput with best-effort ordering and at-least-once delivery. Messages may arrive out of order and occasionally more than once. Use idempotent consumers to handle duplicates.

FIFO queues guarantee first-in-first-out ordering and exactly-once processing. They support message deduplication (by message group or content-based) and message groups for ordered processing within a group. Limited to 3,000 messages per second per queue.

# Create a FIFO queue
aws sqs create-queue \
  --queue-name orders.fifo \
  --attributes '{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "true",
    "VisibilityTimeout": "300",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:orders-dlq\",\"maxReceiveCount\":\"3\"}"
  }'

Visibility timeout prevents other consumers from processing a message while one consumer handles it. If processing takes longer than the timeout, the message becomes visible again. Set this to slightly longer than your maximum processing time.

Dead-letter queues (DLQ) capture messages that fail processing after a configurable number of receive attempts. This prevents poison messages from blocking the queue. Monitor DLQ depth and set alarms to investigate failures.

Amazon Simple Notification Service (SNS) is a pub/sub messaging service. Publishers send messages to topics, and all subscribers receive them.

Fan-out pattern: One message to an SNS topic delivers to all subscribers. Combine with SQS for durable, decoupled processing:

# Create topic
aws sns create-topic --name order-events

# Subscribe SQS queues to the topic
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:order-events \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:us-east-1:123456789012:inventory-queue

aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:order-events \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:us-east-1:123456789012:analytics-queue

Message filtering: Subscribers receive only messages matching filter policies, reducing unnecessary processing:

{
  "FilterPolicy": {
    "orderType": ["express", "priority"],
    "total": [{"numeric": [">", 100]}]
  }
}

This subscriber only receives messages where orderType is express or priority, and total exceeds 100.

Message attributes carry metadata without modifying the message body. Use them for routing decisions, trace IDs, or version information. SNS supports HTTP/S, email, SQS, Lambda, and firehose as subscription protocols.

Amazon EventBridge is a serverless event bus that routes events from AWS services, your applications, and third-party SaaS apps to targets like Lambda, Step Functions, SQS, and more.

Rules define which events to capture and where to send them. Match events using event patterns:

{
  "source": ["myapp.orders"],
  "detail-type": ["OrderCreated"],
  "detail": {
    "total": [{"numeric": [">", 100]}]
  }
}

This rule captures OrderCreated events from myapp.orders where the total exceeds 100.

Targets are the resources that receive matched events. A single rule can send to multiple targets simultaneously:

aws events put-targets \
  --rule high-value-order \
  --targets '[
    {
      "Id": "process-order",
      "Arn": "arn:aws:lambda:...:process-high-value-order"
    },
    {
      "Id": "notify-finance",
      "Arn": "arn:aws:sqs:...:finance-queue"
    }
  ]'

EventBridge Pipes connect sources to targets with optional filtering and enrichment. Pipe sources include SQS, DynamoDB Streams, Kinesis, and self-managed Kafka. Targets include Lambda, Step Functions, SQS, and more. Pipes provide exactly-once delivery and built-in error handling.

EventBridge Scheduler runs events on a schedule (cron or rate expressions). Trigger Lambda, SQS, SNS, or Step Functions at regular intervals without CloudWatch Events.

Schemas describe event structures, enabling code generation and discovery. EventBridge can discover schemas from events flowing through your bus, and you can define custom schemas in the registry.

AWS offers multiple messaging services. Choosing the right one depends on your use case.

SQS: Point-to-point message queuing. Use when you need reliable delivery, decoupling between producer and consumer, and control over processing order (FIFO). Best for task queues, work distribution, and buffering.

SNS: Pub/sub messaging. Use when one message needs to reach multiple subscribers simultaneously. Combine with SQS for fan-out with durable processing. Best for notifications, event broadcasting, and fan-out architectures.

EventBridge: Event routing with content-based filtering. Use when you need complex event matching, multiple targets from a single event, or integration with SaaS partners. Best for event-driven architectures and cross-service orchestration.

Kinesis Data Streams: Real-time streaming for high-throughput data. Use for clickstream data, IoT telemetry, or log aggregation where ordering and real-time processing matter.

MQ: Migration from on-premises message brokers (ActiveMQ, RabbitMQ). Use when your application depends on AMQP or MQTT protocols.

Decision framework:

• Need to send one message to many subscribers? Use SNS
• Need a queue with reliable processing? Use SQS
• Need complex event routing and filtering? Use EventBridge
• Need real-time stream processing? Use Kinesis
• Migrating from ActiveMQ or RabbitMQ? Use Amazon MQ

Common architecture: API Gateway publishes events to EventBridge. EventBridge routes to SQS queues for downstream consumers. Each consumer processes independently with DLQ handling failures.