The Complete Elasticsearch Developer Guide

A deep, practical reference covering Elasticsearch’s most-used features, best practices, and battle-tested patterns.

Core Concepts & Architecture
Cluster, Nodes & Shards
Index Management
Mapping & Data Types
Text Analysis (Analyzers, Tokenizers, Filters)
CRUD & Bulk Operations
Query DSL Deep Dive
Aggregations
Relevance & Scoring
Pagination Strategies
Data Modeling Patterns
Index Lifecycle Management (ILM)
Performance Best Practices
Search-as-You-Type & Autocomplete
Geo Queries
Security
Monitoring & Troubleshooting
Common Anti-Patterns
Client Code Examples
Cheat Sheet

1. Core Concepts & Architecture

Elasticsearch is a distributed, RESTful search and analytics engine built on top of Apache Lucene. It stores data as JSON documents and provides near-real-time search.

Concept	Description
Document	A JSON object stored in an index. Equivalent to a “row” in a relational DB, but schema-flexible.
Index	A collection of documents with a shared mapping. Equivalent to a “table” (loosely).
Mapping	Schema definition for a document type: field names, data types, analyzers.
Shard	A single Lucene index. Each ES index is split into one or more shards for horizontal scaling.
Replica	A copy of a shard for high availability and read throughput.
Node	A single running instance of Elasticsearch.
Cluster	A collection of nodes that together hold your entire data set.
Segment	Immutable Lucene file inside a shard. Segments are merged over time.

Why Elasticsearch?

Full-text search with relevance scoring (BM25 by default since 5.x)
Near-real-time indexing (refresh interval, default 1s)
Horizontal scalability via sharding
Powerful aggregation framework (analytics on the same data used for search)
Schema-on-write with dynamic mapping, but strict mapping is recommended for production

Document Lifecycle

Document is sent to a coordinating node.
Routed to the correct primary shard (via _routing, default is document _id hash).
Indexed into an in-memory buffer + translog (for durability).
Every refresh_interval (default 1s), the buffer is written to a new segment and becomes searchable.
Every index.translog.durability flush interval, data is fsynced to disk (flush).
Background merge process combines small segments into larger ones and purges deleted docs.

Key insight: A document is indexed immediately but not searchable until the next refresh. This is what “near real-time” means.

2. Cluster, Nodes & Shards

Node Roles

node.roles: [ master, data, ingest, ml, remote_cluster_client ]

Role	Purpose
`master`	Cluster state management, index creation/deletion, shard allocation decisions
`data` (`data_hot`, `data_warm`, `data_cold`, `data_frozen`)	Stores data, executes CRUD/search/aggregations
`ingest`	Runs ingest pipelines (pre-processing before indexing)
`ml`	Machine learning jobs
`coordinating only`	No roles set; purely routes requests (rarely needed as dedicated)
`remote_cluster_client`	Enables cross-cluster search/replication

Best practice: In production clusters (>= 6-10 nodes), dedicate master-eligible nodes (3, odd number, small instance, no data) separately from data nodes to avoid split-brain and GC pressure affecting cluster state.

Shard Sizing — The #1 Operational Decision

Recommended shard size: 10–50 GB per shard (up to 50GB is fine for logs/time-series; keep closer to 10-30GB for search-heavy use cases).
Rule of thumb: shard count ≈ total data size / target shard size.
Avoid oversharding (thousands of tiny shards) — each shard has overhead (file handles, memory, cluster state size).
Avoid undersharding — can’t scale beyond shard count, and very large shards recover/relocate slowly.
Primary shard count is fixed at index creation (cannot change without reindexing / _split / _shrink).
Replica count can be changed anytime (PUT /index/_settings).

Rough formula:
number_of_shards = ceil(expected_index_size_GB / 30GB)

Split Brain & Quorum

discovery.seed_hosts and cluster.initial_master_nodes configure master discovery.
Elasticsearch uses a quorum-based voting algorithm (since 7.x, no need to manually set minimum_master_nodes).
Always run an odd number of master-eligible nodes (3 or 5).

3. Index Management

Creating an Index with Explicit Settings

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "refresh_interval": "5s",
    "analysis": {
      "analyzer": {
        "custom_english": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stop", "english_stemmer"]
        }
      },
      "filter": {
        "english_stop": { "type": "stop", "stopwords": "_english_" },
        "english_stemmer": { "type": "stemmer", "language": "english" }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": { "type": "text", "analyzer": "custom_english" },
      "sku": { "type": "keyword" },
      "price": { "type": "scaled_float", "scaling_factor": 100 },
      "created_at": { "type": "date" }
    }
  }
}

Index Templates (Composable, since 7.8+)

Best practice: never let production indices rely purely on dynamic mapping. Use index templates for consistency.

PUT _index_template/logs_template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs_policy"
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "message": { "type": "text" }
      }
    }
  },
  "composed_of": ["component_mappings", "component_settings"],
  "priority": 200
}

Aliases — The Foundation of Zero-Downtime Reindexing

Never point applications directly at a physical index. Always use an alias.

POST /_aliases
{
  "actions": [
    { "add": { "index": "products_v2", "alias": "products" } },
    { "remove": { "index": "products_v1", "alias": "products" } }
  ]
}

Reindex pattern (zero downtime):

Create products_v2 with new mapping.
POST _reindex from products_v1 → products_v2.
Atomically swap the products alias (remove old, add new) in a single _aliases call.
Delete products_v1 once verified.

POST _reindex
{
  "source": { "index": "products_v1" },
  "dest": { "index": "products_v2" }
}

Reindex with Transformation

POST _reindex
{
  "source": { "index": "products_v1" },
  "dest": { "index": "products_v2" },
  "script": {
    "source": "ctx._source.full_name = ctx._source.first_name + ' ' + ctx._source.last_name"
  }
}

Closing / Opening / Freezing Indices

POST /index/_close — stops read/write, frees heap/file handles, keeps data on disk.
POST /index/_open — reopens.
Frozen tier (searchable snapshots) is the modern replacement for the deprecated freeze API — used for rarely-queried, cost-optimized data.

4. Mapping & Data Types

Core Field Types

Type	Use case
`text`	Full-text search, analyzed, tokenized
`keyword`	Exact match, sorting, aggregations, filtering
`long`/`integer`/`short`/`byte`	Whole numbers
`double`/`float`/`half_float`/`scaled_float`	Decimal numbers (use `scaled_float` for currency)
`date`	ISO8601 or custom formats
`boolean`	true/false
`object`	JSON object (flattened internally — loses array relationships)
`nested`	Array of objects that preserves relationships between fields
`geo_point`	Lat/lon coordinates
`geo_shape`	Complex geometries
`ip`	IPv4/IPv6
`completion`	Autocomplete suggester
`dense_vector`	Vector search / kNN / embeddings
`flattened`	Index an entire object as a single field, no explicit sub-mapping (good for arbitrary JSON)
`join`	Parent/child relationships within one index
`alias`	Points to another field name
`constant_keyword`	Same value for every document in the index
`rank_features` / `rank_feature`	Boost scoring based on numeric features
`search_as_you_type`	Optimized for autocomplete-style prefix queries

text vs keyword — The Most Important Distinction

{
  "properties": {
    "title": {
      "type": "text",
      "fields": {
        "keyword": { "type": "keyword", "ignore_above": 256 }
      }
    }
  }
}

title → full-text search (match query, relevance scoring, stemming, stopwords removal).
title.keyword → exact match, sort, terms aggregation, term query.

Rule: if you need to sort or aggregate on a string field, it MUST have a keyword sub-field or be mapped as keyword directly.

object vs nested

// object - WRONG for arrays of related fields
{ "user": [ { "first": "John", "last": "Smith" }, { "first": "Alice", "last": "Doe" } ] }

Internally flattens to user.first: [John, Alice], user.last: [Smith, Doe] — a query for first: John AND last: Doe would incorrectly match! Use nested to preserve object boundaries:

{
  "properties": {
    "user": { "type": "nested" }
  }
}

Query with nested query:

{
  "query": {
    "nested": {
      "path": "user",
      "query": {
        "bool": {
          "must": [
            { "match": { "user.first": "John" } },
            { "match": { "user.last": "Doe" } }
          ]
        }
      }
    }
  }
}

Trade-off: each nested object is indexed as a hidden separate Lucene document — more nested objects = more overhead. Avoid deeply nested or very large nested arrays (thousands of nested docs per parent).

Dynamic Mapping Control

{
  "mappings": {
    "dynamic": "strict",   // "true" | "false" | "strict"
    "properties": { ... }
  }
}

true (default): new fields auto-added to mapping.
false: new fields ignored (not indexed, not searchable, but stored in _source).
strict: new fields cause an error on index. Best for production APIs with well-known schemas.

Multi-fields Pattern (search + sort + aggregate + autocomplete on one field)

{
  "properties": {
    "title": {
      "type": "text",
      "analyzer": "standard",
      "fields": {
        "keyword": { "type": "keyword" },
        "english": { "type": "text", "analyzer": "english" },
        "autocomplete": { "type": "search_as_you_type" }
      }
    }
  }
}

Runtime Fields (define fields at query time, no reindex needed)

GET /products/_search
{
  "runtime_mappings": {
    "price_with_tax": {
      "type": "double",
      "script": "emit(doc['price'].value * 1.2)"
    }
  },
  "query": { "range": { "price_with_tax": { "gte": 100 } } }
}

Good for experimentation or rarely-used fields; costs CPU at query time vs disk at index time — don’t use for high QPS hot paths.

5. Text Analysis

Anatomy of an Analyzer

Analyzer = Character Filters (0..n) → Tokenizer (1) → Token Filters (0..n)

Character filters: modify raw text before tokenizing (html_strip, mapping, pattern_replace).
Tokenizer: splits text into tokens (standard, whitespace, ngram, edge_ngram, keyword, pattern).
Token filters: modify tokens (lowercase, stop, stemmer, synonym, asciifolding, shingle).

Testing Analyzers (always do this before deploying)

POST /_analyze
{
  "analyzer": "standard",
  "text": "The QUICK Brown-Foxes jumped!"
}

Custom Analyzer for Autocomplete (edge n-gram)

PUT /articles
{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete_analyzer": {
          "type": "custom",
          "tokenizer": "autocomplete_tokenizer",
          "filter": ["lowercase"]
        },
        "autocomplete_search_analyzer": {
          "type": "custom",
          "tokenizer": "lowercase"
        }
      },
      "tokenizer": {
        "autocomplete_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 10,
          "token_chars": ["letter", "digit"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "autocomplete_analyzer",
        "search_analyzer": "autocomplete_search_analyzer"
      }
    }
  }
}

Critical rule: use a different search_analyzer (without edge_ngram) at query time, otherwise the search query itself gets n-grammed and produces irrelevant matches.

Synonyms

"filter": {
  "synonym_filter": {
    "type": "synonym",
    "synonyms": [
      "laptop, notebook",
      "tv, television => television"
    ]
  }
}

Use synonym_graph + multiplexer for multi-word synonyms with proper phrase query support.

Language-Specific Analysis

Elasticsearch ships built-in language analyzers (english, turkish, french, etc.) that handle stemming and stopwords correctly per language. Prefer these over building custom stemmers manually.

{ "type": "text", "analyzer": "turkish" }

6. CRUD & Bulk Operations

Single Document Operations

PUT /products/_doc/1
{ "name": "Wireless Mouse", "price": 25.99 }

GET /products/_doc/1

POST /products/_update/1
{ "doc": { "price": 22.99 } }

DELETE /products/_doc/1

Optimistic Concurrency Control

PUT /products/_doc/1?if_seq_no=10&if_primary_term=1

Use _seq_no + _primary_term (modern approach, replaced version for this purpose) to prevent lost updates in concurrent write scenarios.

The Bulk API — Always Use This for Multiple Documents

POST /_bulk
{ "index": { "_index": "products", "_id": "1" } }
{ "name": "Mouse", "price": 25.99 }
{ "update": { "_index": "products", "_id": "2" } }
{ "doc": { "price": 19.99 } }
{ "delete": { "_index": "products", "_id": "3" } }

Best practices for bulk indexing:

Batch size: 5–15 MB per bulk request, or 1,000–5,000 docs — benchmark for your data.
Use multiple parallel bulk threads (but respect thread_pool.write.queue_size).
Disable replicas during massive initial bulk load, re-enable after:
```
PUT /index/_settings
{ "number_of_replicas": 0 }
```

Set refresh_interval to -1 during bulk load, restore afterward:

PUT /index/_settings
{ "index": { "refresh_interval": "-1" } }

Use _id auto-generation (omit _id) when order doesn’t matter — avoids version lookup overhead for indexing throughput.
Retry on 429 (es_rejected_execution_exception) with exponential backoff.

Update by Query / Delete by Query

POST /products/_update_by_query
{
  "query": { "term": { "category": "electronics" } },
  "script": { "source": "ctx._source.price *= 1.1" }
}

POST /products/_delete_by_query
{
  "query": { "range": { "created_at": { "lt": "now-1y" } } }
}

These are resource-intensive (essentially reindex under the hood) — throttle with requests_per_second and consider slices for parallelism.

7. Query DSL Deep Dive

Query Context vs Filter Context

Query context: “How well does this doc match?” → calculates _score.
Filter context: “Does this doc match?” → yes/no, cached, no scoring overhead.

Always put non-scoring conditions in filter, not must.

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "description": "wireless mouse" } }
      ],
      "filter": [
        { "term": { "category": "electronics" } },
        { "range": { "price": { "gte": 10, "lte": 100 } } }
      ],
      "should": [
        { "match": { "brand": "logitech" } }
      ],
      "must_not": [
        { "term": { "discontinued": true } }
      ]
    }
  }
}

Clause	Scoring	Purpose
`must`	yes	AND, contributes to score
`filter`	no (cached)	AND, no score contribution — fast
`should`	yes	OR, boosts score if matched (or acts as OR if no `must`)
`must_not`	no	NOT, cached

Full-Text Queries

// match - standard analyzed full text query
{ "match": { "title": "quick brown fox" } }

// match_phrase - exact sequence of terms
{ "match_phrase": { "title": "quick brown fox" } }

// match_phrase_prefix - phrase + prefix on last term (autocomplete)
{ "match_phrase_prefix": { "title": "quick bro" } }

// multi_match - search across multiple fields with boosting
{
  "multi_match": {
    "query": "wireless mouse",
    "fields": ["title^3", "description", "tags^2"],
    "type": "best_fields"
  }
}

// query_string / simple_query_string - user-typed query syntax (Google-like)
{ "simple_query_string": { "query": "wireless +mouse -wired", "fields": ["title", "description"] } }

multi_match types:

Type	Behavior
`best_fields` (default)	Uses the single best-matching field’s score
`most_fields`	Combines scores from all matching fields — good when analyzing same text differently across fields
`cross_fields`	Treats multiple fields as one big field (good for name search across first_name/last_name)
`phrase`	Runs `match_phrase` on each field
`bool_prefix`	Runs `match_bool_prefix` — good for search-as-you-type

Term-Level Queries (exact values, not analyzed)

{ "term": { "status.keyword": "active" } }
{ "terms": { "status.keyword": ["active", "pending"] } }
{ "range": { "price": { "gte": 10, "lt": 100 } } }
{ "exists": { "field": "email" } }
{ "prefix": { "sku": "AB-" } }
{ "wildcard": { "sku": "AB-*" } }
{ "fuzzy": { "name": { "value": "quikc", "fuzziness": "AUTO" } } }
{ "ids": { "values": ["1", "2", "3"] } }

Never run term queries on a text field — the field is analyzed/lowercased at index time, so exact-case terms won’t match. Use .keyword sub-fields.

Compound Queries

// boosting - lower score for docs matching "negative" but don't exclude
{
  "boosting": {
    "positive": { "match": { "title": "apple" } },
    "negative": { "match": { "title": "fruit" } },
    "negative_boost": 0.2
  }
}

// constant_score - wrap a filter, give a fixed score (used to skip scoring entirely)
{ "constant_score": { "filter": { "term": { "status": "active" } }, "boost": 1.2 } }

// dis_max - take max score among clauses, not sum (best for "OR across different field meanings")
{
  "dis_max": {
    "queries": [
      { "match": { "title": "star wars" } },
      { "match": { "description": "star wars" } }
    ],
    "tie_breaker": 0.3
  }
}

Sorting

{
  "sort": [
    { "price": "asc" },
    { "_score": "desc" },
    { "created_at": { "order": "desc", "missing": "_last" } }
  ]
}

Sorting on text fields is not allowed directly — use .keyword or fielddata: true (avoid fielddata, it’s memory-expensive).

Highlighting

{
  "query": { "match": { "description": "wireless mouse" } },
  "highlight": {
    "fields": { "description": { "fragment_size": 150, "number_of_fragments": 3 } }
  }
}

8. Aggregations

Metric Aggregations

{
  "aggs": {
    "avg_price": { "avg": { "field": "price" } },
    "price_stats": { "stats": { "field": "price" } },
    "unique_categories": { "cardinality": { "field": "category.keyword" } },
    "percentiles_price": { "percentiles": { "field": "price", "percents": [50, 95, 99] } }
  }
}

cardinality is approximate (HyperLogLog++) — tune precision with precision_threshold (memory trade-off).

Bucket Aggregations

{
  "aggs": {
    "by_category": {
      "terms": { "field": "category.keyword", "size": 10 },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } }
      }
    },
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "to": 50 },
          { "from": 50, "to": 200 },
          { "from": 200 }
        ]
      }
    },
    "sales_over_time": {
      "date_histogram": {
        "field": "created_at",
        "calendar_interval": "month"
      }
    }
  }
}

Pipeline Aggregations (aggregate over the results of other aggregations)

{
  "aggs": {
    "sales_per_month": {
      "date_histogram": { "field": "date", "calendar_interval": "month" },
      "aggs": { "total_sales": { "sum": { "field": "amount" } } }
    },
    "max_monthly_sales": {
      "max_bucket": { "buckets_path": "sales_per_month>total_sales" }
    },
    "cumulative_sales": {
      "cumulative_sum": { "buckets_path": "sales_per_month>total_sales" }
    }
  }
}

terms Aggregation Accuracy — Important Gotcha

terms aggregation on a sharded index is approximate by default (each shard returns its top N, then coordinator merges). For accurate results on high-cardinality fields:

{
  "terms": {
    "field": "category.keyword",
    "size": 10,
    "shard_size": 100
  }
}

Increase shard_size >> size to reduce (not eliminate) the doc_count error. Check sum_other_doc_count and doc_count_error_upper_bound in the response.

`filter`/`filters` and `composite` Aggregations

// composite - paginate through all bucket combinations (great for exporting all agg data)
{
  "aggs": {
    "my_buckets": {
      "composite": {
        "size": 1000,
        "sources": [
          { "category": { "terms": { "field": "category.keyword" } } },
          { "month": { "date_histogram": { "field": "date", "calendar_interval": "month" } } }
        ]
      }
    }
  }
}

Use composite instead of deeply nested terms aggs when you need to enumerate all combinations (it supports after for pagination, unlike regular terms).

`search.size: 0` for Aggregation-Only Queries

{ "size": 0, "aggs": { ... } }

Always set size: 0 when you only want aggregation results — avoids fetching/serializing unneeded hits.

9. Relevance & Scoring

BM25 (default similarity since ES 5.0)

Score is roughly a function of:

Term frequency (TF) — how often the term appears in the field (saturates, unlike classic TF-IDF).
Inverse document frequency (IDF) — rarer terms score higher.
Field length norm — shorter fields score higher for the same term frequency.

PUT /products
{
  "settings": {
    "index": {
      "similarity": {
        "custom_bm25": {
          "type": "BM25",
          "b": 0.75,
          "k1": 1.2
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "description": { "type": "text", "similarity": "custom_bm25" }
    }
  }
}

function_score — Combine Relevance with Business Logic

{
  "query": {
    "function_score": {
      "query": { "match": { "title": "laptop" } },
      "functions": [
        { "filter": { "term": { "featured": true } }, "weight": 2 },
        { "field_value_factor": { "field": "sales_count", "modifier": "log1p", "factor": 0.1 } },
        { "gauss": { "created_at": { "origin": "now", "scale": "10d", "decay": 0.5 } } }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

Common use: freshness decay (gauss/exp/linear decay functions), popularity boosting, manual pinning.

Rescoring (apply expensive scoring only to top-N)

{
  "query": { "match": { "title": "laptop" } },
  "rescore": {
    "window_size": 100,
    "query": {
      "rescore_query": { "match_phrase": { "title": { "query": "gaming laptop", "slop": 2 } } },
      "query_weight": 0.7,
      "rescore_query_weight": 1.2
    }
  }
}

Use rescoring to apply expensive queries (phrase matching, learning-to-rank, vector re-ranking) only on the top window instead of the whole result set — big performance win.

explain API — Debugging Relevance

GET /products/_explain/1
{ "query": { "match": { "title": "wireless mouse" } } }

Also "explain": true inside a normal _search request shows scoring breakdown per hit — essential when relevance tuning.

10. Pagination Strategies

Method	Use case	Limitation
`from` + `size`	Small result sets, UI pagination (page 1-100)	Deep pagination (`from` > 10,000) is expensive/blocked by default (`index.max_result_window`)
`search_after`	Deep pagination, real-time “next page”	Requires a stable sort (usually with `_shard_doc`/`_id` tiebreaker); no jumping to arbitrary page
Scroll API	Full data export / reindex-like batch processing	Not for user-facing pagination; keeps a point-in-time snapshot open (resource cost); being superseded by PIT
Point in Time (PIT) + `search_after`	Modern replacement for scroll; consistent view across paginated requests	Slightly more setup (open PIT, close PIT)

search_after with PIT (Recommended Modern Pattern)

POST /products/_pit?keep_alive=1m

GET /_search
{
  "size": 100,
  "query": { "match_all": {} },
  "pit": { "id": "<pit_id>", "keep_alive": "1m" },
  "sort": [ { "created_at": "asc" }, { "_shard_doc": "asc" } ]
}

Use the sort values of the last hit as the search_after value in the next request.

{
  "search_after": [1622512800000, 987654],
  "sort": [ { "created_at": "asc" }, { "_shard_doc": "asc" } ]
}

Close the PIT with DELETE /_pit when done.

Best practice: never let from + size exceed index.max_result_window (default 10,000) — it will throw an error and it’s a red flag for pagination design.

11. Data Modeling Patterns

Denormalization is Normal in Elasticsearch

Unlike relational DBs, ES has no joins across indices at scale. Denormalize related data into the document at index time.

{
  "order_id": "1001",
  "customer": { "id": "55", "name": "Jane Doe", "tier": "gold" },
  "items": [
    { "sku": "A1", "name": "Widget", "qty": 2, "price": 9.99 }
  ]
}

Parent/Child (`join` field) — When You Must Model Relationships

Use only when children are updated much more frequently than parents (avoids reindexing the parent).

PUT /forum
{
  "mappings": {
    "properties": {
      "join_field": { "type": "join", "relations": { "question": "answer" } }
    }
  }
}

Trade-off: slower queries (has_child/has_parent), requires same shard routing for parent+children. Prefer nested for read-heavy, rarely-updated relationships, and denormalization for most cases.

Nested vs Parent/Child vs Denormalized — Decision Table

Pattern	Update frequency	Query complexity	Performance	When to use
Denormalized	Data duplicated on every parent update	Simple	Fastest reads	Default choice — most cases
`nested`	Whole parent doc reindexed on any nested change	Medium (`nested` query)	Good	Small-medium arrays of related objects, mostly static
`join` (parent/child)	Children updated independently	Complex (`has_child`)	Slower	Children updated far more often than parent, large 1:many

Time-Series / Logs Pattern: Index-per-Timeframe

logs-2026.08.01
logs-2026.08.02
logs-2026.08.03

Combined with an alias (logs-write) and ILM rollover — enables easy deletion of old data (drop whole index vs delete_by_query), and isolates hot/warm/cold tiers by index age. This is the foundation of the modern data streams feature.

Data Streams (built on this pattern, since 7.9+)

PUT _index_template/logs-template
{
  "index_patterns": ["logs-myapp-*"],
  "data_stream": {},
  "template": {
    "settings": { "index.lifecycle.name": "logs-policy" }
  }
}

POST /logs-myapp-default/_doc
{ "@timestamp": "2026-08-17T10:00:00Z", "message": "hello" }

Data streams automatically manage a sequence of hidden backing indices, rollover, and simplify time-series ingestion — preferred over manually-managed index-per-day patterns for logs/metrics.

Field Explosion — Watch Mapping Growth

Default index.mapping.total_fields.limit is 1000. Dynamic mapping of arbitrary JSON (e.g., user-supplied metadata) can blow this up. Use flattened type for arbitrary/variable JSON objects instead of letting them dynamically map:

{ "metadata": { "type": "flattened" } }

12. Index Lifecycle Management (ILM)

ILM automates moving indices through hot → warm → cold → frozen → delete phases based on age/size/doc count.

PUT _ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "30gb", "max_age": "1d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": { "set_priority": { "priority": 0 }, "freeze": {} }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}

Key actions:

rollover — creates a new backing index when the current one hits size/age/doc-count thresholds (paired with an alias or data stream).
shrink — reduces shard count for older, less-queried indices (fewer shards = less overhead).
forcemerge — merges segments down (ideal for read-only warm/cold data — never forcemerge an actively-written index).
freeze / searchable snapshots — moves data to cheap object storage (S3/GCS/Azure Blob) while remaining searchable, for cold/frozen tiers.
delete — removes the index entirely.

Best practice: never manually delete time-series data with delete_by_query — always structure as one-index-per-timeframe + ILM delete phase (dropping a whole index is instant; delete_by_query is expensive and leaves tombstones).

13. Performance Best Practices

Indexing Performance

Use the Bulk API, batch by size (5–15MB) not just doc count.
Increase refresh_interval (e.g., 30s or -1) for write-heavy workloads; restore for read-heavy phases.
Set number_of_replicas: 0 during massive bulk loads, then scale up.
Use auto-generated IDs unless you need idempotent upserts (custom IDs require a version lookup before write).
Avoid deeply nested documents and huge arrays — they multiply indexing cost.
Tune indices.memory.index_buffer_size (default 10% of heap) for heavy indexing nodes.
Disable _source only for extreme space-constrained cases (drastically limits reindex/update/highlight ability — rarely worth it).
Use index.codec: best_compression for cold/archival indices to save disk (trades some CPU).

Search Performance

Use filter context wherever scoring isn’t needed — filters are cached and reused across queries.
Limit size and use pagination correctly (see section 10).
Avoid script queries / script_score in hot paths — expensive per-doc scripting.
Prefer keyword fields for aggregations/sorting over fielddata on text.

Use _source filtering to return only needed fields:

{ "_source": ["title", "price"], "query": { ... } }

Warm up caches after restart for critical indices (index.warmer, or just run representative queries).
Avoid wildcard queries starting with * (*mouse) — cannot use the term index efficiently; consider ngram fields instead.
Use routing to target specific shards when you know the partition key, avoiding scatter-gather across all shards:
```
GET /orders/_search?routing=customer_123
```
Force merge read-only indices to 1 segment for faster search (never on actively-written indices).
Increase search.max_buckets cautiously; very large aggregations increase memory pressure.

Hardware & JVM

Heap: set to 50% of available RAM, never exceed ~30-32GB (compressed oops cutoff).
Leave the other 50% RAM for the OS page cache — Lucene relies heavily on OS-level file caching.
Use SSDs; Elasticsearch is I/O sensitive.
Disable swap (bootstrap.memory_lock: true).
Monitor GC pauses — frequent long GCs indicate heap pressure or field data / query cache misuse.

Circuit Breakers

Elasticsearch has built-in circuit breakers (indices.breaker.total.limit, fielddata, request) to prevent OOM. If you frequently hit circuit_breaking_exception, investigate query patterns (huge aggregations, fielddata usage) rather than just raising limits.

14. Search-as-You-Type & Autocomplete

Option 1: `search_as_you_type` field type (simplest)

{
  "properties": {
    "title": { "type": "search_as_you_type" }
  }
}

{
  "query": {
    "multi_match": {
      "query": "wirele mo",
      "type": "bool_prefix",
      "fields": ["title", "title._2gram", "title._3gram"]
    }
  }
}

Option 2: `completion` suggester (fastest, in-memory FST structure)

{
  "properties": {
    "suggest": { "type": "completion" }
  }
}

{
  "suggest": {
    "product-suggest": {
      "prefix": "wirel",
      "completion": { "field": "suggest", "fuzzy": { "fuzziness": 1 }, "size": 5 }
    }
  }
}

Best for: dropdown-style autocomplete with millisecond latency. Limitation: less flexible ranking/filtering than a full query.

Option 3: Edge n-gram custom analyzer (see section 5)

Best for: full-text style prefix matching combined with relevance scoring / filtering.

Recommendation: use completion for pure autocomplete widgets, edge_ngram or search_as_you_type when you need it combined with other filters/scoring.

15. Geo Queries

{
  "properties": {
    "location": { "type": "geo_point" }
  }
}

// geo_distance filter
{
  "query": {
    "bool": {
      "filter": {
        "geo_distance": {
          "distance": "10km",
          "location": { "lat": 40.73, "lon": -73.99 }
        }
      }
    }
  }
}

// sort by distance
{
  "sort": [
    {
      "_geo_distance": {
        "location": { "lat": 40.73, "lon": -73.99 },
        "order": "asc",
        "unit": "km"
      }
    }
  ]
}

// geo_bounding_box - fast rectangular filter
{ "query": { "geo_bounding_box": { "location": { "top_left": { "lat": 41, "lon": -74.5 }, "bottom_right": { "lat": 40.5, "lon": -73.5 } } } } }

For complex polygons/shapes, use geo_shape field type with geo_shape query (supports intersects, within, contains, disjoint).

16. Security

Core Building Blocks (X-Pack Security, included free since 7.1 for basic features)

TLS for transport and HTTP layers — mandatory for production.
Authentication: native realm, LDAP, SAML, OIDC, Kerberos, API keys.
Role-Based Access Control (RBAC): define roles with index/cluster/field/document-level privileges.

POST /_security/role/read_only_products
{
  "indices": [
    {
      "names": ["products"],
      "privileges": ["read"],
      "field_security": { "grant": ["name", "price", "category"] },
      "query": { "term": { "public": true } }
    }
  ]
}

This combines field-level security (only expose certain fields) and document-level security (row-level filtering) in one role.

API Keys (preferred for service-to-service auth over basic auth)

POST /_security/api_key
{
  "name": "my-app-key",
  "role_descriptors": {
    "app_role": { "indices": [ { "names": ["products"], "privileges": ["read"] } ] }
  },
  "expiration": "30d"
}

Best Practices

Never expose Elasticsearch directly to the public internet.
Use API keys with minimal privileges per application, not the superuser.
Rotate credentials and set expiration on API keys.
Enable audit logging for compliance-sensitive clusters.
Use ingest pipelines/index templates to prevent field explosion attacks via dynamic mapping abuse.

17. Monitoring & Troubleshooting

Essential Cluster APIs

GET /_cluster/health?level=indices
GET /_cluster/state
GET /_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m
GET /_cat/indices?v&s=store.size:desc
GET /_cat/shards?v&h=index,shard,prirep,state,docs,store,node
GET /_nodes/stats
GET /_nodes/hot_threads
GET /_cat/thread_pool/write?v
GET /_cat/pending_tasks?v

Cluster Health Colors

Status	Meaning
Green	All primary + replica shards allocated
Yellow	All primaries allocated, some replicas not (common on single-node dev clusters)
Red	Some primary shards unallocated — data loss risk / queries failing on affected indices

Slow Log

PUT /products/_settings
{
  "index.search.slowlog.threshold.query.warn": "2s",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.indexing.slowlog.threshold.index.warn": "2s"
}

Use to identify slow queries/indexing operations in production without full profiling overhead.

Profile API (deep query performance analysis)

GET /products/_search
{
  "profile": true,
  "query": { "match": { "title": "laptop" } }
}

Shows per-clause timing breakdown (Lucene-level) — use sparingly, adds overhead, not for production hot paths.

Common Errors & Fixes

Error	Cause	Fix
`circuit_breaking_exception`	Query/agg using too much memory	Reduce agg size/cardinality, add `filter`, increase heap, check fielddata
`es_rejected_execution_exception`	Thread pool queue full (bulk/search)	Backoff + retry, reduce bulk size/concurrency, scale nodes
`mapper_parsing_exception`	Type mismatch on indexing	Fix source data or mapping; consider `ignore_malformed`
`version_conflict_engine_exception`	Concurrent update race	Retry with `retry_on_conflict`, or use optimistic concurrency correctly
`search_phase_execution_exception`	Underlying shard failures	Check `_cluster/health`, node logs, unallocated shards
Cluster stuck yellow/red	Unassigned shards	`GET _cluster/allocation/explain`

GET /_cluster/allocation/explain

This is the #1 tool for diagnosing why shards won’t allocate (disk watermark, node filtering, missing node, etc).

18. Common Anti-Patterns

Anti-Pattern	Why it’s bad	Do instead
One shard per tiny index, thousands of indices	Cluster state bloat, overhead per shard	Use data streams / ILM rollover, consolidate
Using `term` query on a `text` field	Silently never matches as expected	Use `.keyword` sub-field
Deep `from`/`size` pagination	Expensive, memory-heavy on coordinating node	`search_after` + PIT
Dynamic mapping in production with unpredictable input	Field explosion, mapping conflicts, index corruption risk	Explicit mappings + `dynamic: strict`, or `flattened` type
Using `scroll` for user-facing pagination	Resource leak, stateful, doesn’t reflect live data	`search_after` / PIT
Storing huge unbounded arrays / nested objects	Massive per-document overhead	Restructure, cap array sizes, or separate index
`wildcard` query with leading `*`	Full index scan-like cost	Use `ngram` field or restructure
Ignoring `refresh_interval` during bulk load	Unnecessary segment creation, slower indexing	Set `-1` during load
Treating ES as a system of record / primary DB	No true ACID transactions/joins; risk of data loss on misconfig	Keep a source-of-truth DB; ES as a search/analytics layer
Not monitoring disk watermarks	Cluster goes read-only unexpectedly	Alert on 85%/90%/95% watermarks
One giant `bool` query with everything in `must`	No cache reuse, unnecessary scoring	Split into `filter` where scoring isn’t needed
Using `_all` field or overly broad `multi_match` on all fields	Slow, poor relevance	Explicitly define searchable fields with appropriate boosts

19. Client Code Examples

Python (elasticsearch-py, 8.x client)

from elasticsearch import Elasticsearch

es = Elasticsearch(
    "https://localhost:9200",
    api_key="base64_api_key",
)

# Index a document
es.index(index="products", id="1", document={"name": "Mouse", "price": 25.99})

# Search
resp = es.search(
    index="products",
    query={"bool": {"must": [{"match": {"name": "mouse"}}], "filter": [{"range": {"price": {"lte": 50}}}]}},
    size=10,
)
for hit in resp["hits"]["hits"]:
    print(hit["_source"])

# Bulk
from elasticsearch.helpers import bulk

actions = [
    {"_index": "products", "_id": str(i), "_source": {"name": f"Item {i}", "price": i}}
    for i in range(1000)
]
bulk(es, actions)

Node.js (@elastic/elasticsearch)

const { Client } = require('@elastic/elasticsearch');
const client = new Client({ node: 'https://localhost:9200', auth: { apiKey: 'base64_api_key' } });

await client.index({
  index: 'products',
  id: '1',
  document: { name: 'Mouse', price: 25.99 },
});

const result = await client.search({
  index: 'products',
  query: {
    bool: {
      must: [{ match: { name: 'mouse' } }],
      filter: [{ range: { price: { lte: 50 } } }],
    },
  },
});
console.log(result.hits.hits);

Java (Java API Client, 8.x)

ElasticsearchClient client = new ElasticsearchClient(transport);

IndexResponse response = client.index(i -> i
    .index("products")
    .id("1")
    .document(new Product("Mouse", 25.99))
);

SearchResponse<Product> search = client.search(s -> s
    .index("products")
    .query(q -> q.match(m -> m.field("name").query("mouse"))),
    Product.class
);

20. Cheat Sheet

# Cluster
GET /_cluster/health
GET /_cat/indices?v
GET /_cat/nodes?v
GET /_cluster/allocation/explain

# Index management
PUT /my_index
DELETE /my_index
POST /my_index/_close
POST /my_index/_open
GET /my_index/_mapping
PUT /my_index/_settings

# Aliases
POST /_aliases
GET /_alias/my_alias

# CRUD
PUT /my_index/_doc/1
GET /my_index/_doc/1
POST /my_index/_update/1
DELETE /my_index/_doc/1
POST /_bulk

# Search
GET /my_index/_search
GET /my_index/_search?q=title:laptop
POST /my_index/_search { query, aggs, sort, size, from, _source }

# Reindex
POST /_reindex
POST /my_index/_update_by_query
POST /my_index/_delete_by_query

# ILM
GET /_ilm/policy
PUT /_ilm/policy/my_policy
POST /my_index/_ilm/retry

# Analyze
POST /_analyze { "analyzer": "standard", "text": "..." }

Table of Contents