Skip to content
advanced Phase 80 · Scaling Advanced

Search Scaling

Search scaling with OpenSearch cluster, index sharding, search performance, and query optimization

45m
0 problems
Topic Progress 0%

OpenSearch Cluster Architecture

Cluster Topology

┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│  Master 1   │  │  Master 2   │  │  Master 3   │
│  (dedicated)│  │  (dedicated)│  │  (dedicated)│
└─────────────┘  └─────────────┘  └─────────────┘

┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│  Data 1     │  │  Data 2     │  │  Data 3     │
│  (primary)  │  │  (replica)  │  │  (replica)  │
└─────────────┘  └─────────────┘  └─────────────┘

┌─────────────┐
│  Coordinating│
│  (client)   │
└─────────────┘

Magento OpenSearch Config

// app/etc/env.php
'search' => [
    'engine' => 'elasticsearch8',
    'elasticsearch_server' => [
        'hostname' => 'opensearch-cluster.example.com',
        'port' => '9200',
        'index' => 'magento2',
        'enable_auth' => false,
        'timeout' => 15
    ]
],

Cluster Health Check

# Check cluster health
curl -X GET "http://opensearch:9200/_cluster/health?pretty"

# Key metrics:
# status: green (healthy), yellow (replicas missing), red (primary missing)
# active_shards: Number of active shards
# relocating_shards: Shards being moved
# initializing_shards: Shards starting up

Index Sharding Strategy

Sharding Configuration

PUT /magento2
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1,
    "refresh_interval": "30s",
    "translog.durability": "async",
    "translog.sync_interval": "30s"
  }
}

Shard Allocation

Shard Count Guidelines:
- Small catalog (<100K products): 2-3 shards
- Medium catalog (100K-1M): 5-8 shards
- Large catalog (>1M): 10-20 shards

Shard Size Target: 10-50GB per shard

Product Index Structure

PUT /magento2_products/_mapping
{
  "properties": {
    "entity_id": { "type": "integer" },
    "sku": { "type": "keyword" },
    "name": { "type": "text", "analyzer": "standard" },
    "description": { "type": "text" },
    "price": { "type": "float" },
    "category_ids": { "type": "integer" },
    "in_stock": { "type": "boolean" },
    "visibility": { "type": "integer" },
    "created_at": { "type": "date" },
    "updated_at": { "type": "date" }
  }
}

Search Performance Optimization

Query Optimization

// BAD: Full text search on all fields
{
  "query": {
    "query_string": {
      "query": "red shoes"
    }
  }
}

// GOOD: Targeted fields with boost
{
  "query": {
    "multi_match": {
      "query": "red shoes",
      "fields": ["name^3", "sku^2", "description"]
    }
  }
}

Caching Search Results

// Magento search cache
$cacheKey = 'search_' . md5($query . $filters . $sort);
$cachedResult = $cache->load($cacheKey);

if ($cachedResult) {
    return unserialize($cachedResult);
}

// Run search
$result = $searchClient->search($params);
$cache->save(serialize($result), $cacheKey, [], 3600);

Search Performance Metrics

# Monitor search latency
curl -X GET "http://opensearch:9200/_nodes/stats/indices/search" | jq

# Key metrics:
# search.query_total: Total queries executed
# search.query_time_in_millis: Total query time
# search.fetch_total: Total fetches
# search.fetch_time_in_millis: Total fetch time

Indexing Strategy

Bulk Indexing

POST /magento2_products/_bulk
{"index": {"_id": "1"}}
{"sku": "PRODUCT-1", "name": "Widget", "price": 29.99}
{"index": {"_id": "2"}}
{"sku": "PRODUCT-2", "name": "Gadget", "price": 49.99}

Reindex Strategy

# Magento reindex commands
bin/magento indexer:reindex catalogsearch_fulltext
bin/magento indexer:reindex catalog_product_price

# Schedule reindex via cron
bin/magento cron:run

# Monitor reindex progress
bin/magento indexer:status

Index Lifecycle

Product Save → Magento Event → Queue Indexer → OpenSearch Update

1. Product saved to MySQL
2. Magento dispatches event
3. Indexer queued in message queue
4. OpenSearch document updated
5. Search results reflect changes

Latency: ~5-30 seconds depending on queue depth

Quiz

1. What is the target shard size for OpenSearch?

Question 1 options

2. What does cluster status 'yellow' indicate?

Question 2 options

3. How often should search index refresh by default?

Question 3 options

Flashcards

Question

OpenSearch cluster status meanings?

Click to reveal answer

Answer

Green: all healthy, Yellow: replicas missing, Red: primaries missing

Question

Shard size target?

Click to reveal answer

Answer

10-50GB per shard for optimal performance

Question

Search cache strategy?

Click to reveal answer

Answer

Cache search results by query+filters+sort hash for 1 hour

Question

Product indexing latency?

Click to reveal answer

Answer

5-30 seconds from save to searchable

Revision Notes

Key Takeaways

  • 1. OpenSearch cluster needs master, data, and coordinating nodes
  • 2. Shard size should be 10-50GB for optimal performance
  • 3. Use multi_match with field boosting for better search results
  • 4. Refresh interval of 30s balances freshness and performance
  • 5. Monitor cluster health and shard allocation

Interview Tips

  • • Explain OpenSearch cluster architecture and node roles
  • • Discuss sharding strategy based on catalog size
  • • Compare query optimization techniques

Cheat Sheet

Search Scaling:
  Cluster: Master + Data + Coordinating nodes
  Shards: 10-50GB each, 5-20 per index
  Replicas: 1+ for HA

Query Optimization:
  multi_match with field boost
  Filter before full-text search
  Cache results by query hash

Indexing:
  Bulk indexing for performance
  30s refresh interval
  Queue-based async updates

Monitoring:
  Cluster health: green/yellow/red
  Query latency: <100ms target