Caching at Retrieval Layer

1. Why Caching Matters: Zipf's Law

User query behavior follows Zipf's Law: a small number of queries account for a disproportionately large share of traffic. This makes caching extraordinarily effective—cache the top 10% of queries and you serve 70% of traffic from memory.

Named after linguist George Zipf, this power-law distribution appears throughout human behavior: the most popular word ("the") appears exponentially more often than the 10th most popular word. Search queries exhibit the same pattern. On any e-commerce site, queries like "iphone" or "nike shoes" appear thousands of times per minute, while long-tail queries like "blue running shoes size 12 wide" might appear once a day. This skewed distribution is the fundamental reason caching works so well for search systems.

The practical implication is profound: by caching just a few thousand popular query results, you can serve the majority of your search traffic without touching your index at all. This reduces latency from 50-200ms (index search) to sub-millisecond (cache hit), while simultaneously reducing load on your search cluster. A well-tuned caching strategy can reduce your required Elasticsearch nodes by 50% or more, directly impacting infrastructure costs.

2. Multi-Layer Caching Architecture

Production search systems use multiple caching layers, each serving different purposes with different invalidation strategies. Understanding this stack is key to optimizing your search performance.

Think of the cache layers like a pyramid: at the top (closest to users) are CDN caches for static content like autocomplete suggestions and popular category pages. These caches are large, geographically distributed, and have long TTLs because the content rarely changes. Below that sits your application cache (Redis or Memcached), storing full query results for popular searches. The application cache is centralized and faster to invalidate when content changes.

At the Elasticsearch level, two caches work in tandem: the Request Cache stores complete response payloads (aggregations, suggestions) at the shard level, while the Query Cachestores filter clause results as bitsets at the node level. Finally, the OS filesystem cache keeps frequently accessed index segments in memory. Each layer trades off hit rate, invalidation complexity, and storage efficiency differently—the key is configuring them to complement each other.

3. Elasticsearch Query Cache

The Query Cache (also called Node Query Cache) stores filter clause results as bitsets. When the same filter is used again, Elasticsearch can skip re-evaluating documents entirely and just use the cached bitset for set operations.

A bitset is an extremely compact data structure: one bit per document in the index. If your index has 100 million documents, a filter bitset only takes ~12 MB of memory while encoding the complete set of matching document IDs. When a cached filter is combined with other filters (AND, OR, NOT), Elasticsearch performs highly optimized bitwise operations (AND = &, OR = |) that complete in microseconds rather than iterating through posting lists.

The Query Cache is particularly valuable for e-commerce faceted search where the same filters appear constantly: "category:shoes", "size:10", "color:black", "brand:nike", "in_stock:true". Each of these filters gets cached as a bitset, and subsequent queries combining them just AND the bitsets together. A query like "shoes AND size:10 AND color:black" becomes three bitwise AND operations vs. three posting list intersections—orders of magnitude faster.

4. Elasticsearch Request Cache

The Request Cache (Shard-Level Request Cache) stores entire search responses including aggregation results, suggestions, and hit counts. It's perfect for dashboard queries that run repeatedly.

Unlike the Query Cache (which stores filter bitsets), the Request Cache stores complete serialized response JSON at the shard level. The cache key is computed from the entire request body, so even small changes to the query produce cache misses. By default, it only caches requests with size: 0(no document hits) because caching actual document content would consume too much memory and become stale quickly as documents are updated.

The Request Cache is automatically invalidated when the underlying data changes (via refresh). This makes it safe for dynamic content but means it works best when your data has natural update boundaries. For example, if you batch-import data once per hour, queries executed during that hour get 100% cache hits. The cache is particularly valuable for aggregation-heavy dashboards where computing facet counts across billions of documents is expensive but the counts don't need to be real-time.

5. Cache Warming Strategies

A cold cache after deployment or cluster restart can cause severe performance degradation.Cache warming proactively populates caches before traffic arrives, ensuring users never experience the "cold start" penalty.

The "thundering herd" problem illustrates why warming matters: when a cache is empty and traffic suddenly arrives, every request becomes a cache miss simultaneously, overwhelming your search cluster with 100x the normal load. A single new server added to a load balancer without warming can bring down an entire cluster if traffic is high enough. Proper warming prevents this by ensuring cache hit rates are healthy before real traffic arrives.

The key to effective warming is replaying real query patterns from your analytics. Extract the top 1,000-10,000 most frequent queries from the past week, then replay them at a controlled rate (10-50 QPS) before routing production traffic to new instances. This populates caches with the exact queries that will drive most of your traffic, maximizing hit rate from the first real request.

6. Cache Invalidation Strategies

"There are only two hard things in Computer Science: cache invalidation and naming things." Choosing the right invalidation strategy is critical for balancing freshness with performance.

The fundamental tension is between freshness and hit rate. Aggressive invalidation (expiring entries quickly) keeps data fresh but reduces cache effectiveness. Lenient invalidation maximizes hit rate but risks serving stale data. The right balance depends entirely on your use case: a product catalog might tolerate 1-hour staleness, while stock prices need real-time accuracy.

Most production systems use hybrid strategies: short TTLs for volatile data combined with event-based invalidation for critical updates. For example, set a 5-minute TTL by default, but explicitly invalidate an item's cache when it's updated through your CMS. The TTL acts as a safety net catching any invalidation failures, while event-based invalidation ensures critical changes propagate immediately.

7. Cache Monitoring & Metrics

You can't optimize what you don't measure. Monitor cache hit rates, eviction rates, and memory usage to ensure your caching strategy is effective.

The most important metric is cache hit rate: (hits / (hits + misses)) × 100. A healthy search cache typically achieves 70-90% hit rate. If you're below 50%, your cache is likely undersized, TTLs are too short, or your queries are too diverse (long-tail dominated). Track this metric over time to detect problems: sudden drops usually indicate configuration changes, traffic pattern shifts, or cache capacity issues.

Eviction rate is your early warning system: high evictions mean the cache is undersized and frequently discarding entries to make room for new ones. This reduces hit rate and wastes the CPU time spent computing entries that get evicted before being reused. If evictions exceed 1-2% of your total entries per minute, consider increasing cache size or reducing TTLs to allow natural expiration instead.

8. Common Caching Pitfalls

Even experienced teams make caching mistakes. Here are the most common issues and how to avoid them.