S3 Storage

S3 Fundamentals

Amazon Simple Storage Service (S3) is an object storage service offering 99.999999999% (11 nines) durability. It stores data as objects within buckets.

Core Concepts

• Bucket: Container for objects (like a root folder)
• Object: Data + metadata (file + info about the file)
• Key: Unique identifier for an object within a bucket
• Region: Geographic location where bucket is created

Create and Manage Buckets

# Create a bucket
aws s3 mb s3://my-unique-bucket-name --region us-east-1

# List all buckets
aws s3 ls

# Enable versioning
aws s3api put-bucket-versioning \
  --bucket my-bucket \
  --versioning-configuration Status=Enabled

# Enable server access logging
aws s3api put-bucket-logging \
  --bucket my-bucket \
  --bucket-logging-status '{
    "LoggingEnabled": {
      "TargetBucket": "my-log-bucket",
      "TargetPrefix": "s3-access-logs/"
    }
  }'

# Set default encryption
aws s3api put-bucket-encryption \
  --bucket my-bucket \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "AES256"
      },
      "BucketKeyEnabled": true
    }]
  }'

# Block all public access
aws s3api put-public-access-block \
  --bucket my-bucket \
  --public-access-block-configuration {
    "BlockPublicAcls": true,
    "IgnorePublicAcls": true,
    "BlockPublicPolicy": true,
    "RestrictPublicBuckets": true
  }

S3 URL Formats

• Virtual-hosted style: https://my-bucket.s3.amazonaws.com/key
• Path style: https://s3.amazonaws.com/my-bucket/key
• Accelerated: https://my-bucket.s3-accelerate.amazonaws.com/key

S3 Storage Classes

S3 offers six storage classes optimized for different access patterns.

Storage Classes Comparison

Lifecycle Policies

# Create a lifecycle policy
aws s3api put-bucket-lifecycle-configuration \
  --bucket my-bucket \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "MoveToGlacier",
      "Status": "Enabled",
      "Filter": {"Prefix": "logs/"},
      "Transitions": [{
        "Days": 30,
        "StorageClass": "GLACIER"
      }],
      "Expiration": {
        "Days": 365
      }
    }]
  }'

# Check current lifecycle policy
aws s3api get-bucket-lifecycle-configuration --bucket my-bucket

Lifecycle Policy Example

Upload to S3 Standard
    │
    ├── 30 days → S3 Standard-IA
    │      │
    │      ├── 90 days → S3 Glacier Instant Retrieval
    │      │      │
    │      │      ├── 180 days → S3 Glacier Deep Archive
    │      │      │      │
    │      │      │      └── 365 days → Delete

S3 Intelligent-Tiering

Automatically moves objects between access tiers based on usage patterns:

• Frequent Access: Same as Standard
• Infrequent Access: Same as Standard-IA (after 30 days)
• Archive Instant Access: Same as Glacier Instant (after 90 days)
• Deep Archive Access: Same as Glacier Deep Archive (after 180 days)

S3 Access Control

Bucket Policies

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPublicRead",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket/public/*"
    },
    {
      "Sid": "DenyNonSSL",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    }
  ]
}

# Apply a bucket policy
aws s3api put-bucket-policy \
  --bucket my-bucket \
  --policy file://policy.json

# Get bucket policy
aws s3api get-bucket-policy --bucket my-bucket

# Remove bucket policy
aws s3api delete-bucket-policy --bucket my-bucket

ACLs (Access Control Lists)

# Make an object public-read
aws s3api put-object-acl \
  --bucket my-bucket \
  --key public/image.jpg \
  --acl public-read

# Get object ACL
aws s3api get-object-acl --bucket my-bucket --key public/image.jpg

Presigned URLs

# Generate a presigned URL (expires in 1 hour)
aws s3 presign s3://my-bucket/private/document.pdf --expires-in 3600

# Using Python/boto3
import boto3
s3 = boto3.client('s3')
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'private/document.pdf'},
    ExpiresIn=3600
)

S3 Access Points

# Create an Access Point
aws s3control create-access-point \
  --account-id 123456789012 \
  --name my-access-point \
  --bucket my-bucket \
  --public-access-block-configuration {
    "BlockPublicAcls": true,
    "IgnorePublicAcls": true,
    "BlockPublicPolicy": true,
    "RestrictPublicBuckets": true
  }

S3 Bucket Replication

Replicate objects between buckets for disaster recovery or compliance.

Cross-Region Replication (CRR)

# Enable versioning on both buckets
aws s3api put-bucket-versioning --bucket source-bucket --versioning-configuration Status=Enabled
aws s3api put-bucket-versioning --bucket dest-bucket --versioning-configuration Status=Enabled

# Create a replication role
aws iam create-role \
  --role-name S3ReplicationRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "s3.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

# Attach replication policy
aws iam attach-role-policy \
  --role-name S3ReplicationRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

# Configure replication
aws s3api put-bucket-replication \
  --bucket source-bucket \
  --replication-configuration '{
    "Role": "arn:aws:iam::123456789012:role/S3ReplicationRole",
    "Rules": [{
      "Status": "Enabled",
      "Prefix": "",
      "Destination": {
        "Bucket": "arn:aws:s3:::dest-bucket",
        "StorageClass": "STANDARD"
      }
    }]
  }'

S3 Batch Replication

# For existing objects (not just new ones)
aws s3api get-bucket-replication-configuration --bucket source-bucket

# Use S3 Batch Operations for existing objects
aws s3control create-job \
  --account-id 123456789012 \
  --operation '{"S3ReplicateObject": {}}' \
  --manifest '{
    "Spec": {"Format": "S3InventoryReport", "Fields": ["Bucket", "Key"]},
    "Location": {"S3Arn": "arn:aws:s3:::my-inventory-bucket/manifest.json"}
  }' \
  --priority 10 \
  --role-arn arn:aws:iam::123456789012:role/batch-role

S3 Static Website Hosting

Host a static website entirely from S3.

Enable Website Hosting

# Enable website hosting
aws s3api put-bucket-website \
  --bucket my-website-bucket \
  --website-configuration '{
    "IndexDocument": {"Suffix": "index.html"},
    "ErrorDocument": {"Key": "error.html"},
    "RoutingRules": [{
      "Condition": {
        "HttpErrorCodeReturnedEquals": "404"
      },
      "Redirect": {
        "ReplaceKeyPrefixWith": "error.html"
      }
    }]
  }'

# Upload website files
aws s3 sync ./public s3://my-website-bucket

# Make files public
aws s3api put-object-acl --bucket my-website-bucket --key index.html --acl public-read

# Website endpoint
# http://my-website-bucket.s3-website-us-east-1.amazonaws.com

Redirect All Traffic with CloudFront

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Browser    │────▶│  CloudFront  │────▶│ S3 Website   │
│              │◀────│  (HTTPS)     │◀────│ Bucket       │
└──────────────┘     └──────────────┘     └──────────────┘

S3 Request Payment

# Configure requester pays (user pays for data transfer)
aws s3api put-bucket-request-payment \
  --bucket my-bucket \
  --request-payer Requester

S3 Inventory

# Generate daily inventory report
aws s3api put-bucket-inventory-configuration \
  --bucket my-bucket \
  --id daily-inventory \
  --inventory-configuration '{
    "Destination": {
      "S3BucketDestination": {
        "Format": "CSV",
        "Bucket": "arn:aws:s3:::inventory-bucket"
      }
    },
    "IsEnabled": true,
    "Id": "daily-inventory",
    "IncludedObjectVersions": "Current",
    "Schedule": {
      "Frequency": "Daily"
    }
  }'