SQS & SNS

Amazon SQS Fundamentals

Amazon Simple Queue Service (SQS) is a fully managed message queuing service.

Queue Types

Create Queue

# Standard queue
aws sqs create-queue --queue-name my-queue

# FIFO queue
aws sqs create-queue --queue-name my-queue.fifo --attributes '{"FifoQueue": "true"}'

# Queue with dead letter queue
aws sqs create-queue --queue-name my-queue.fifo --attributes '{
  "FifoQueue": "true",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:my-dlq.fifo\",\"maxReceiveCount\":\"3\"}",
  "MessageRetentionPeriod": "1209600",
  "VisibilityTimeout": "300"
}'

# Send message
aws sqs send-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --message-body '{"orderId": "123", "amount": 99.99}' \
  --message-attributes '{"Priority": {"DataType": "String", "StringValue": "high"}}'

# Receive message
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --max-number-of-messages 10 \
  --wait-time-seconds 20

# Delete message
aws sqs delete-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --receipt-handle <receipt-handle>

FIFO Queue Features

• Message Group ID: Messages with same group are processed in order
• Message Deduplication: Prevents duplicate messages
• Ordering: Strict ordering within a message group

Amazon SNS Fundamentals

Amazon Simple Notification Service (SNS) is a fully managed pub/sub messaging service.

Create Topic

# Create topic
aws sns create-topic --name my-topic

# Subscribe to topic
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:my-topic \
  --protocol email \
  --notification-endpoint user@example.com

# Subscribe Lambda
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:my-topic \
  --protocol lambda \
  --notification-endpoint arn:aws:lambda:us-east-1:xxx:function:my-function

# Subscribe SQS
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:my-topic \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:us-east-1:xxx:my-queue

# Publish message
aws sns publish \
  --topic-arn arn:aws:sns:us-east-1:123456789012:my-topic \
  --subject "Order Update" \
  --message '{"orderId": "123", "status": "shipped"}'

# Add filter policy
aws sns set-topic-attributes \
  --topic-arn arn:aws:sns:us-east-1:xxx:my-topic \
  --attribute-name FilterPolicy \
  --attribute-value '{"event": ["order.created", "order.shipped"]}'

SNS Protocols

SQS + SNS Integration

Fan-Out Pattern

┌─────────────────────────────────────────────────────────────┐
│                    Fan-Out Pattern                           │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Order Service ──▶ SNS Topic                               │
│                     │                                       │
│        ┌────────────┼────────────┐                         │
│        │            │            │                         │
│        ▼            ▼            ▼                         │
│     SQS Queue   SQS Queue   SQS Queue                     │
│     (Email)     (Inventory)  (Analytics)                   │
│        │            │            │                         │
│        ▼            ▼            ▼                         │
│    Email Svc    Inventory   Analytics                      │
│                  Service    Service                         │
└─────────────────────────────────────────────────────────────┘

Implementation

# Create SQS queues for each consumer
aws sqs create-queue --queue-name email-queue
aws sqs create-queue --queue-name inventory-queue
aws sqs create-queue --queue-name analytics-queue

# Subscribe queues to SNS topic
aws sns subscribe --topic-arn arn:xxx:my-topic --protocol sqs --notification-endpoint arn:aws:sqs:xxx:email-queue
aws sns subscribe --topic-arn arn:xxx:my-topic --protocol sqs --notification-endpoint arn:aws:sqs:xxx:inventory-queue
aws sns subscribe --topic-arn arn:xxx:my-topic --protocol sqs --notification-endpoint arn:aws:sqs:xxx:analytics-queue

# Grant SNS permission to send to SQS
aws sqs add-permission \
  --queue-url https://sqs.us-east-1.amazonaws.com/xxx/email-queue \
  --label sns-send \
  --aws-account-ids 123456789012 \
  --actions SendMessage

FIFO Topic + FIFO Queue

# FIFO topic for ordered delivery
aws sns create-topic --name my-topic.fifo --attributes '{"FifoTopic": "true"}'

# FIFO queue preserves order
aws sqs create-queue --queue-name my-queue.fifo --attributes '{"FifoQueue": "true"}'

# Subscribe FIFO queue to FIFO topic
aws sns subscribe --topic-arn arn:xxx:my-topic.fifo --protocol sqs --notification-endpoint arn:xxx:my-queue.fifo

Message Queue Patterns

Request/Reply Pattern

import boto3
import uuid
import json

sqs = boto3.client('sqs')
sns = boto3.client('sns')

def request_reply(topic_arn, request_queue_url, reply_queue_url, payload):
    # Send request
    correlation_id = str(uuid.uuid4())
    
    sqs.send_message(
        QueueUrl=request_queue_url,
        MessageBody=json.dumps(payload),
        MessageAttributes={
            'CorrelationId': {
                'DataType': 'String',
                'StringValue': correlation_id
            },
            'ReplyTo': {
                'DataType': 'String',
                'StringValue': reply_queue_url
            }
        }
    )
    
    # Wait for reply
    response = sqs.receive_message(
        QueueUrl=reply_queue_url,
        MessageAttributeNames=['CorrelationId'],
        MaxNumberOfMessages=1,
        WaitTimeSeconds=30
    )
    
    for message in response.get('Messages', []):
        if message['MessageAttributes']['CorrelationId']['StringValue'] == correlation_id:
            sqs.delete_message(QueueUrl=reply_queue_url, ReceiptHandle=message['ReceiptHandle'])
            return json.loads(message['Body'])

Message Grouping

# FIFO queue with message groups
aws sqs send-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/xxx/my-queue.fifo \
  --message-body '{"orderId": "123"}' \
  --message-group-id "order-123" \
  --message-deduplication-id "order-123-1"

SQS and SNS Best Practices

Security

# Encrypt queue at rest
aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/xxx/my-queue \
  --attributes '{"KmsMasterKeyId": "alias/aws/sqs"}'

# Encrypt SNS topic
aws sns set-topic-attributes \
  --topic-arn arn:xxx:my-topic \
  --attribute-name KmsMasterKeyId \
  --attribute-value alias/aws/sns

# Restrict access with IAM policy
aws sqs add-permission \
  --queue-url https://sqs.us-east-1.amazonaws.com/xxx/my-queue \
  --label allow-lambda \
  --aws-account-ids 123456789012 \
  --actions SendMessage ReceiveMessage DeleteMessage

Performance

Best Practices

1. Use FIFO for ordered processing
2. Implement DLQ for failed messages
3. Use message attributes for metadata
4. Set appropriate visibility timeout
5. Use long polling to reduce costs
6. Batch operations when possible
7. Monitor with CloudWatch metrics

# Key metrics to monitor
# SQS: ApproximateNumberOfMessages, ApproximateAgeOfOldestMessage
# SNS: NumberOfMessagesPublished, NumberOfNotificationsDelivered

aws cloudwatch get-metric-statistics \
  --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessages \
  --dimensions Name=QueueName,Value=my-queue \
  --period 300 \
  --statistics Average