The Complete Observability Guide — Prometheus, Grafana, Loki & Jaeger

A deep, practical reference covering the “LGTM+P” stack (Loki, Grafana, Tempo/Jaeger, Prometheus) — architecture, query languages, alerting, dashboarding, tracing, and production-grade best practices.

---

Table of Contents

1. Observability Fundamentals
2. Prometheus
3. Grafana
4. Loki
5. Jaeger & Distributed Tracing
6. Correlating Metrics, Logs & Traces
7. SLOs, SLIs & Error Budgets
8. Production Checklist

---

1. Observability Fundamentals

1.1 The Three Pillars

Pillar	Question it answers	Tool
Metrics	“Is something wrong, and how bad?”	Prometheus
Logs	“What exactly happened?”	Loki
Traces	“Where did it happen, in which service?”	Jaeger

Observability ≠ monitoring. Monitoring tells you known failure modes are occurring; observability lets you ask arbitrary new questions about system behavior without shipping new code.

1.2 RED and USE Methods

RED (for request-driven services):

• Rate — requests per second
• Errors — failed requests per second
• Duration — latency distribution (p50/p90/p99)

USE (for resources — CPU, disk, memory, network):

• Utilization — % time resource is busy
• Saturation — queue depth / work waiting
• Errors — error events

Rule of thumb: apply RED to services, USE to infrastructure resources.

1.3 The Four Golden Signals (Google SRE)

Latency, Traffic, Errors, Saturation — effectively RED + Saturation.

---

2. Prometheus

2.1 Architecture

 ┌────────────┐   scrape (pull)   ┌──────────────┐
 │ Targets/    │◄─────────────────│  Prometheus   │
 │ Exporters   │                  │  Server        │
 └────────────┘                  │  - TSDB        │
                                   │  - PromQL      │
                                   │  - Rule Engine │
                                   └──────┬─────────┘
                                          │ alerts
                                   ┌──────▼─────────┐
                                   │  Alertmanager   │
                                   └────────────────┘

• Prometheus is pull-based: it scrapes /metrics HTTP endpoints on a configured interval (default 15s/1m).
• For push-based workloads (batch jobs, lambdas) use the Pushgateway — but sparingly, it becomes a single point of failure and breaks staleness detection.
• Local TSDB stores data in 2-hour blocks on disk, compacted over time. Default retention: 15 days.
• For long-term storage / horizontal scale / multi-tenancy use Thanos, Cortex, or Grafana Mimir with remote_write.

2.2 Data Model

Every time series is uniquely identified by a metric name + a set of key-value label pairs:

http_requests_total{method="POST", handler="/api/orders", status="200"} 27

2.2.1 The Four Metric Types

Type	Description	Example
Counter	Monotonically increasing value; resets to 0 on restart	http_requests_total
Gauge	Value that can go up or down	node_memory_available_bytes
Histogram	Samples observations into configurable buckets, plus _sum and _count	http_request_duration_seconds
Summary	Like histogram but computes client-side quantiles (φ-quantiles)	rpc_duration_seconds

Histogram vs Summary — pick histograms almost always:

• Histograms allow aggregation across instances (histogram_quantile() over summed buckets); summaries do not — you can’t average pre-computed quantiles.
• Summaries are cheaper on the client, more expensive on the server for arbitrary quantiles.
• Use native histograms (Prometheus 2.40+) for high-resolution, low-cardinality-cost histograms if your client library supports them.

2.3 PromQL Deep Dive

Instant vs Range Vectors

http_requests_total                       # instant vector (current value)
http_requests_total[5m]                   # range vector (all samples in last 5m)

Rate, Increase, IRate

rate(http_requests_total[5m])       # per-second average rate over 5m window — USE FOR ALERTING/DASHBOARDS
irate(http_requests_total[5m])      # per-second rate using only last two points — spiky, use for fast-moving graphs only
increase(http_requests_total[1h])   # total increase over 1h (rate * duration, extrapolated)

Golden rule: always apply rate()/irate() to counters before aggregating with sum(). Never sum() raw counters and then rate() — you lose per-series reset detection.

# CORRECT
sum(rate(http_requests_total[5m])) by (service)

# WRONG — breaks counter reset handling
rate(sum(http_requests_total) by (service))[5m]

Aggregation Operators

sum(rate(http_requests_total[5m])) by (job)
avg(node_load1) by (instance)
max(up) by (job)
count(up == 0)                    # number of down targets
topk(5, rate(http_requests_total[5m]))
bottomk(3, node_filesystem_free_bytes)

by keeps the listed labels; without drops the listed labels and keeps everything else.

Histogram Quantiles

histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

• le (less-than-or-equal) is the bucket boundary label — must be preserved through by().
• Result is an interpolated estimate, not exact — bucket boundaries matter a lot. Choose buckets around your actual SLO thresholds (e.g., 0.1, 0.25, 0.5, 1, 2.5, 5, 10).

Binary Operators & Vector Matching

# one-to-one: label sets must match exactly (ignoring the metric name)
node_memory_MemFree_bytes / node_memory_MemTotal_bytes

# many-to-one with ignoring/group_left — enrich metrics with extra labels
sum(rate(http_requests_total[5m])) by (instance)
  * on(instance) group_left(datacenter) node_meta

# ignoring specific labels for matching
rate(a[5m]) / ignoring(instance) rate(b[5m])

Useful Functions

predict_linear(node_filesystem_free_bytes[6h], 4*3600)  # forecast disk full in 4h
deriv(some_gauge[10m])                                   # per-second derivative
delta(cpu_temp_celsius[1h])                               # difference over range (gauges)
absent(up{job="critical-service"})                        # 1 if metric doesn't exist — great for "silent failure" alerts
changes(process_start_time_seconds[1h])                   # count of restarts
resets(some_counter[1h])                                  # count of counter resets
label_replace(up, "svc", "$1", "job", "(.*)-prod")        # regex label manipulation
clamp_min(my_metric, 0)

Subqueries

max_over_time(rate(http_requests_total[5m])[1h:1m])

Powerful but expensive — use sparingly, prefer recording rules for repeated heavy queries.

2.4 Service Discovery

Static configs don’t scale. Use SD mechanisms:

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

Supported SD: Kubernetes, Consul, EC2, Azure, GCE, DNS, file-based (file_sd_configs — great for custom inventories), Docker, OpenStack.

relabel_configs run before scraping (can rewrite __address__, drop targets); metric_relabel_configs run after scraping (drop/rename high-cardinality metrics/labels).

2.5 Recording Rules

Pre-compute expensive/frequent queries into new time series:

groups:
  - name: api_slos
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)
      - record: job:http_request_errors:rate5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
      - record: job:http_error_ratio:rate5m
        expr: job:http_request_errors:rate5m / job:http_requests:rate5m

Naming convention: level:metric:operations (e.g., job:http_errors:rate5m).

2.6 Alerting Rules & Alertmanager

groups:
  - name: availability
    rules:
      - alert: HighErrorRate
        expr: job:http_error_ratio:rate5m > 0.05
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.job }}"
          description: "{{ $labels.job }} error ratio is {{ $value | humanizePercentage }} (threshold 5%)"

      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} down"

Key practices:

• Always set for: to avoid flapping alerts on transient blips.
• Alert on symptoms (SLO burn, error rate, latency), not raw causes (CPU%) — causes belong in dashboards, not pages.
• Use multi-window multi-burn-rate alerts for SLO-based paging (see §7).

Alertmanager handles routing, grouping, deduplication, silencing, inhibition:

route:
  receiver: default
  group_by: [alertname, cluster]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match: {severity: critical}
      receiver: pagerduty
    - match: {severity: warning}
      receiver: slack

inhibit_rules:
  - source_match: {severity: critical}
    target_match: {severity: warning}
    equal: [alertname, cluster, service]

2.7 Exporters

Exporter	Purpose
node_exporter	Host-level metrics (CPU, mem, disk, network)
blackbox_exporter	Probing over HTTP/DNS/TCP/ICMP — synthetic checks
kube-state-metrics	Kubernetes object states (deployments, pods, etc — NOT cAdvisor-level resource usage)
cAdvisor	Container resource usage metrics
mysqld_exporter / postgres_exporter	Database metrics
redis_exporter	Redis metrics

Prefer instrumenting your own application code with a client library (client_golang, client_python, prom-client for Node, simpleclient for Java) over wrapping it with a generic exporter — direct instrumentation gives business-level metrics.

2.8 Cardinality — the #1 Prometheus Killer

Every unique label-value combination is a new time series. Common cardinality bombs:

• User IDs, request IDs, email addresses as label values
• Unbounded URL paths (/users/12345 instead of /users/:id)
• Full stack traces or error messages as labels
• IP addresses at scale

Rule of thumb: keep total active series per instance in the low single-digit millions at most; per-metric cardinality should be predictable and bounded. Use count({__name__=~".+"}) and topk(10, count by (__name__)({__name__=~".+"})) to audit.

2.9 Federation, Remote Write & Long-Term Storage

• Federation (/federate endpoint): pull aggregated metrics from lower-level Prometheus into a global one. Good for hierarchical rollups, not for raw long-term storage.
• remote_write: push all (or filtered) samples to Thanos/Mimir/Cortex/VictoriaMetrics for durable, horizontally-scalable, long-retention storage with global query view.
• Thanos: Sidecar + Store Gateway + Compactor + Querier, backed by object storage (S3/GCS). Deduplicates across HA Prometheus replicas.
• Grafana Mimir: Fully compatible with Prometheus remote_write/PromQL, horizontally scalable, multi-tenant by design.

---

3. Grafana

3.1 Data Sources

Grafana is a visualization and alerting layer on top of many backends: Prometheus, Loki, Jaeger/Tempo, Elasticsearch, InfluxDB, MySQL/Postgres, CloudWatch, and more. Configure via UI or as code:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 30s
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: jaeger-uid

3.2 Dashboard Design Best Practices

1. Top-down layout: overview row (RED/USE summary) → drill-down rows below.
2. One dashboard, one purpose. Don’t cram unrelated services into one dashboard.
3. Use row collapsing for secondary detail panels.
4. Consistent units and thresholds across panels of the same type (latency always in s/ms, never mixed).
5. Avoid “graph soup”: cap panels per dashboard (~12–20); prefer drill-down links to a detail dashboard over adding more panels.
6. Use $__rate_interval instead of hardcoding [5m] in dashboard queries — it auto-adjusts to the selected time range and scrape interval.
7. Annotate deploys/incidents via the Annotations API so dashboards show causally relevant markers.

3.3 Variables & Templating

Variable: $datacenter   → label_values(up, datacenter)
Variable: $instance     → label_values(up{datacenter="$datacenter"}, instance)
Variable: $interval     → interval type: 1m,5m,10m,30m,1h

Use Multi-value + Include All carefully — “All” often generates a huge regex query (=~"a|b|c|...") that can be slow; consider chained variables to narrow scope first.

3.4 Provisioning as Code

Store dashboards as JSON (or Jsonnet/grafonnet) in git, provisioned via:

apiVersion: 1
providers:
  - name: default
    folder: Platform
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards

grafonnet/jsonnet lets you templatize dashboards (DRY panels across services) — widely used at scale instead of hand-editing JSON.

3.5 Unified Alerting

Grafana-managed alert rules can query any data source (not just Prometheus), unlike legacy dashboard alerts.

# Alert rule (conceptual)
condition: B
data:
  - refId: A
    query: sum(rate(http_requests_total{status=~"5.."}[5m]))
  - refId: B
    reduceExpression: last(A)
    threshold: "> 10"
for: 5m

Notification policies mirror Alertmanager’s tree-based routing (label matchers → contact points), and Grafana can proxy to an external Alertmanager if you already run Prometheus’s.

3.6 Panel Types Worth Knowing

• Time series — default, most common.
• Stat / Gauge — single current value, great for SLO/error-budget headline numbers.
• Heatmap — ideal for visualizing histogram bucket distributions over time (latency heatmaps).
• Table — for top-N breakdowns, log tail-like views.
• Node graph — service dependency visualization (pairs well with trace data).
• Logs panel — native Loki log line rendering with live tailing.
• Traces panel — renders a Jaeger/Tempo trace waterfall inline.

3.7 Exemplars

Exemplars attach a trace_id to a specific histogram observation, rendered as dots on top of a latency graph — click through straight into the trace in Jaeger/Tempo. Requires:

1. Client library support (OpenMetrics exemplar format)
2. exemplarTraceIdDestinations configured on the Prometheus data source

---

4. Loki

4.1 Philosophy

Loki indexes only metadata (labels), not full log text — unlike Elasticsearch. This makes it dramatically cheaper to run at scale, at the cost of full-text search being slower (it greps compressed chunks at query time instead of hitting an inverted index).

“Like Prometheus, but for logs” — same label model, same PromQL-inspired query language (LogQL).

4.2 Architecture

Promtail/Alloy/Fluent Bit → Distributor → Ingester → Chunks (object storage: S3/GCS)
                                              │
                                          Compactor
                                              │
                             Querier ◄── Query Frontend ◄── Grafana

• Distributor: receives log streams, validates, hashes to ingesters.
• Ingester: batches/compresses into chunks, flushes to object storage.
• Querier: executes LogQL, fetches chunks from storage + recent data from ingesters.
• Compactor: merges/deduplicates index, applies retention.
• Deployment modes: monolithic (single binary, small scale), simple scalable (read/write split), microservices (full component split, large scale).

4.3 LogQL

Log Stream Selector (like PromQL label matchers)

{app="checkout", env="prod"}
{app="checkout"} |= "error"
{app="checkout"} != "healthcheck"
{app="checkout"} |~ "error|panic|fatal"
{app="checkout"} |= "error" | json | line_format "{{.msg}}"

Operators: |= contains, != not-contains, |~ regex match, !~ regex not-match.

Parsers

{app="api"} | json                                 # parse JSON log lines into labels
{app="api"} | logfmt                                # parse logfmt lines
{app="api"} | pattern "<ip> - - <_> \"<method> <uri>"
{app="api"} | regexp "(?P<level>\\w+): (?P<msg>.*)"

Label Filters & Line Formatting

{app="api"} | json | status_code >= 500
{app="api"} | json | duration > 1s
{app="api"} | json | line_format "{{.timestamp}} {{.level}} {{.msg}}"
{app="api"} | json | label_format new_label="{{.old_label}}"

Metric Queries (aggregating logs into numbers)

sum(rate({app="api"} |= "error" [5m])) by (pod)
sum(count_over_time({app="api"}[5m])) by (level)
topk(5, sum(rate({app="api"}[5m])) by (route))

# unwrap: turn a numeric field inside the log line into a value for aggregation
quantile_over_time(0.99,
  {app="api"} | json | unwrap duration_ms [5m]
) by (route)

4.4 Promtail / Grafana Alloy Configuration

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - docker: {}
      - json:
          expressions:
            level: level
            msg: message
      - labels:
          level:
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app

Grafana Alloy is the modern successor to Promtail (also replaces the OTel Collector use case for many pipelines) — new deployments should generally prefer Alloy.

4.5 Label Cardinality — Even More Critical Than Prometheus

Loki’s index is built from label sets, and each unique label combination creates a new stream. Putting high-cardinality values (user_id, request_id, trace_id) into labels explodes the index and destroys performance/cost. Instead:

• Keep labels to low-cardinality, static dimensions: app, env, namespace, pod (bounded), level.
• Put high-cardinality fields (user_id, trace_id, request_id) into the log line body, and filter/extract them at query time with | json / | logfmt / |=.
• Avoid dynamic labels like timestamps, UUIDs, or full URLs.

4.6 Retention & Compaction

limits_config:
  retention_period: 720h   # 30 days
compactor:
  retention_enabled: true
  delete_request_store: s3

Per-tenant overrides let you retain critical audit logs longer than debug logs.

---

5. Jaeger & Distributed Tracing

5.1 Core Concepts

• Trace: the full journey of one request across services.
• Span: a single unit of work within a trace (e.g., one HTTP call, one DB query) — has a name, start/end time, tags, logs, and a parent span reference.
• Context propagation: trace/span IDs passed across process boundaries via headers (traceparent in W3C Trace Context, or Jaeger’s own uber-trace-id).
• Root span: the first span in a trace (no parent).

Trace: abc123
├── span: gateway (0-120ms)
│   ├── span: auth-service (5-20ms)
│   └── span: order-service (25-115ms)
│       ├── span: db-query (30-70ms)
│       └── span: payment-service (75-110ms)

5.2 Architecture

App (SDK/OTel) → Jaeger Agent/OTel Collector → Jaeger Collector → Storage (Elasticsearch/Cassandra/Kafka buffer) → Query Service → Jaeger UI / Grafana

• Client/SDK: instruments code, generates spans.
• Agent (legacy) or OTel Collector (modern): local sidecar/daemonset that batches and forwards spans.
• Collector: validates, processes (sampling, enrichment), writes to storage.
• Storage: Cassandra or Elasticsearch for production; Badger/in-memory for dev.
• Query: serves the UI and API for trace retrieval.

Modern recommendation: instrument with OpenTelemetry SDKs (vendor-neutral), export via OTLP to an OpenTelemetry Collector, which fans out to Jaeger (or Grafana Tempo). Don’t couple application code to Jaeger’s native client libraries anymore — they’re in maintenance mode; OTel is the standard.

5.3 Instrumentation

// Go + OpenTelemetry example
tracer := otel.Tracer("order-service")
ctx, span := tracer.Start(ctx, "ProcessOrder")
defer span.End()

span.SetAttributes(
    attribute.String("order.id", orderID),
    attribute.Int("order.item_count", len(items)),
)

if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
}

• Auto-instrumentation exists for most frameworks/languages (Java agent, Python opentelemetry-instrument, Node @opentelemetry/auto-instrumentations-node) — start here before hand-rolling spans.
• Propagate context through async boundaries (message queues, goroutines) manually — this is the #1 source of broken/orphaned traces.

5.4 Sampling Strategies

Tracing every request at scale is expensive (storage + overhead). Options:

Strategy	Description	Trade-off
Head-based, probabilistic	Sample X% of traces at trace start	Simple, but may miss rare errors
Rate limiting	N traces/sec per service	Predictable cost
Tail-based sampling	Buffer full trace, decide after seeing outcome (e.g., always keep errors/slow traces)	Better signal, needs a buffering collector (OTel Collector tail_sampling processor) — more infra
Adaptive sampling	Jaeger collector adjusts rate per-endpoint to hit a target volume	Balances low & high traffic services

Best practice: low head-based sample rate (e.g., 1-10%) combined with tail-based sampling rules that always keep error traces and traces above a latency threshold, regardless of the head decision.

5.5 Context Propagation Formats

• W3C Trace Context (traceparent, tracestate headers) — the modern standard, use this by default.
• B3 (Zipkin-originated, X-B3-* headers) — still common in older Istio/Envoy setups.
• Jaeger native (uber-trace-id) — legacy, avoid for new systems.

5.6 Correlating Traces with Metrics/Logs

• Attach trace_id and span_id as structured fields in every log line (most logging libraries have OTel-aware formatters/hooks).
• Use exemplars in Prometheus histograms to link a slow latency bucket directly to a trace.
• Grafana Tempo (and Jaeger data source) supports “Trace to logs” and “Trace to metrics” panel links configured in the data source settings.

---

6. Correlating Metrics, Logs & Traces

The real power of observability tooling comes from cross-navigation, not any single pillar in isolation.

Alert fires (Prometheus/Alertmanager)
   → Dashboard shows error rate spike (Grafana)
   → Click exemplar dot on latency panel → jump to slow Trace (Jaeger)
   → Trace shows which span/service is slow
   → Click "logs for this span" → filtered Loki query scoped to trace_id (Loki)
   → Root cause found in log line

Implementation checklist:

• Structured (JSON) logging with trace_id, span_id, service, env fields.
• Consistent label naming across Prometheus/Loki (service, namespace, pod).
• Exemplars enabled on latency histograms.
• Grafana data source links configured: Prometheus→Jaeger (exemplars), Jaeger→Loki (trace-to-logs), Loki→Jaeger (trace_id extracted via | json linked as a derived field).
• Derived fields in Loki data source config to turn trace_id text in logs into clickable trace links.

---

7. SLOs, SLIs & Error Budgets

7.1 Definitions

• SLI (Service Level Indicator): a measured metric, e.g., proportion of successful requests.
• SLO (Service Level Objective): a target for an SLI over a window, e.g., “99.9% of requests succeed over 30 days.”
• Error budget: 1 - SLO — the allowed amount of “badness” before you must stop shipping features and focus on reliability.

7.2 Example SLI Queries

# Availability SLI
sum(rate(http_requests_total{status!~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# Latency SLI (% of requests under 300ms)
sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
  / sum(rate(http_request_duration_seconds_count[5m]))

7.3 Multi-Window, Multi-Burn-Rate Alerting

Naive threshold alerts on error rate either fire too late (slow burn over days) or too often (noisy on short blips). The standard SRE pattern uses two time windows per severity:

# Fast burn: consuming 14.4x budget → exhausts 30-day budget in 2 hours
- alert: ErrorBudgetBurnFast
  expr: |
    (
      job:http_error_ratio:rate5m > 14.4 * 0.001
      and
      job:http_error_ratio:rate1h > 14.4 * 0.001
    )
  labels: {severity: critical}

# Slow burn: consuming 1x budget over 6h/3d → page only if sustained
- alert: ErrorBudgetBurnSlow
  expr: |
    (
      job:http_error_ratio:rate1h > 1 * 0.001
      and
      job:http_error_ratio:rate6h > 1 * 0.001
    )
  labels: {severity: warning}

This is directly from Google’s SRE workbook multi-window burn-rate methodology — it balances speed of detection against precision (avoiding false pages).

---

8. Production Checklist

Prometheus

• for: set on all alerts to avoid flapping
• Recording rules for expensive/frequent dashboard queries
• Cardinality audited regularly (topk(10, count by (__name__)({__name__=~".+"})))
• Remote-write to long-term storage (Thanos/Mimir/Cortex) if retention > 15 days needed
• HA pair of Prometheus servers with dedup at the Thanos/Mimir layer
• Alertmanager routing tested with amtool; inhibition rules to reduce alert storms

Grafana

• Dashboards as code, version-controlled
• $__rate_interval used instead of hardcoded windows
• Folder/permission structure matches team ownership
• Exemplars + trace-to-logs links configured

Loki

• Labels kept low-cardinality; high-cardinality fields left in log body
• Retention policy per tenant/namespace
• Alloy/Promtail pipeline stages tested for correct label extraction

Jaeger/Tracing

• OpenTelemetry SDK/auto-instrumentation in place
• Context propagation verified across async boundaries (queues, workers)
• Tail-based sampling keeps 100% of error/slow traces
• trace_id injected into structured logs

Cross-cutting

• SLOs defined with multi-window burn-rate alerts
• Runbooks linked from every alert annotation
• On-call dashboards limited to actionable, symptom-level signals

---

End of guide. Pair this with its Turkish companion document for bilingual teams.