The Complete Kong API Gateway Developer Guide

A deep, practical reference for building, securing, and operating APIs with Kong Gateway — covering core concepts, plugins, deployment models, Kubernetes integration, custom plugin development, performance, security, and production patterns.

Introduction & Architecture
Core Entities
Deployment Models
Installation & Quick Start
Admin API
Declarative Configuration & decK
Kubernetes Ingress Controller (KIC)
Authentication Plugins
Traffic Control Plugins
Transformation Plugins
Logging & Observability
Load Balancing & Health Checks
Custom Plugin Development (Lua)
Security Best Practices
Performance Tuning
Production Patterns
CI/CD & GitOps
Troubleshooting
CLI & Admin API Cheat Sheet

1. Introduction & Architecture

Kong is a cloud-native, platform-agnostic API Gateway built on top of NGINX and OpenResty (NGINX + LuaJIT). It sits between clients and upstream services, handling cross-cutting concerns so your microservices don’t have to: authentication, rate limiting, transformation, logging, load balancing, and traffic shaping.

1.1 Why a Gateway?

Decoupling: Clients never talk to backend services directly.
Centralized policy enforcement: Auth, rate limiting, and logging live in one place instead of being duplicated across services.
Protocol translation: REST ↔ gRPC ↔ WebSocket ↔ TCP.
Zero-downtime evolution: Swap, version, or scale backends without client impact.

1.2 High-Level Architecture

                     ┌────────────────────┐
Client Requests ───▶ │   Kong Gateway      │
                     │  (OpenResty/NGINX)  │
                     │  - Router            │
                     │  - Plugin Pipeline   │
                     │  - Load Balancer     │
                     └─────────┬───────────┘
                               │
              ┌────────────────┼────────────────┐
              ▼                ▼                 ▼
        Upstream A       Upstream B         Upstream C

Kong’s request lifecycle runs through phases (mirroring NGINX/OpenResty phases), and plugins hook into these phases:

Phase	Purpose
`certificate`	Handle SSL/TLS handshake (SNI-based cert selection)
`rewrite`	Rewrite request before routing decision
`access`	Authentication, authorization, rate limiting — most plugins live here
`header_filter`	Modify response headers before they’re sent
`body_filter`	Modify response body chunks
`log`	Fire-and-forget logging, analytics

Understanding phases matters when writing or ordering custom plugins — a plugin that manipulates the response body must implement body_filter, not access.

1.3 Kong Product Family

Kong Gateway (OSS/Enterprise) — the API gateway itself, self-hosted.
Kong Konnect — Kong’s SaaS control plane (hosted management, analytics, dev portal, service catalog) that can manage self-hosted or cloud-hosted data planes.
Kong Mesh / Kuma — service mesh built on Envoy, for east-west (service-to-service) traffic, complementary to Kong Gateway’s north-south (client-to-service) role.
Kong Ingress Controller (KIC) — translates Kubernetes resources (Ingress, CRDs) into Kong configuration.
Insomnia — API client/design tool from the same company, useful for testing Kong-fronted APIs.

2. Core Entities

Everything in Kong is modeled as an entity, configurable via the Admin API, declarative YAML, or Kubernetes CRDs.

2.1 Service

Represents an upstream API/microservice.

curl -i -X POST http://localhost:8001/services \
  --data name=orders-service \
  --data url=http://orders.internal:8080

2.2 Route

Defines how requests are matched and sent to a Service (by path, host, method, header, SNI).

curl -i -X POST http://localhost:8001/services/orders-service/routes \
  --data 'paths[]=/orders' \
  --data name=orders-route \
  --data strip_path=false

Key route fields:

strip_path — whether the matched path prefix is removed before proxying upstream.
preserve_host — forward the original Host header instead of the upstream’s.
protocols — restrict to http, https, grpc, grpcs, tcp, tls, udp, ws, wss.
path_handling — v0 or v1 semantics for path concatenation (matters a lot when combining strip_path with prefixed upstream paths).

2.3 Upstream & Target

An Upstream is a virtual hostname representing a pool of Targets (backend IP:port entries), enabling load balancing and health checks.

curl -i -X POST http://localhost:8001/upstreams --data name=orders-upstream
curl -i -X POST http://localhost:8001/upstreams/orders-upstream/targets \
  --data target=10.0.0.11:8080 --data weight=100
curl -i -X POST http://localhost:8001/upstreams/orders-upstream/targets \
  --data target=10.0.0.12:8080 --data weight=100

Then point a Service’s host at the upstream name (orders-upstream) instead of a raw IP.

2.4 Consumer

Represents an API client (a user, application, or partner) that Kong can attach credentials, ACL groups, and rate-limit quotas to.

curl -i -X POST http://localhost:8001/consumers --data username=mobile-app

2.5 Plugin

Attaches behavior (auth, rate limiting, transformation, logging) to a Service, Route, Consumer, or globally.

curl -i -X POST http://localhost:8001/services/orders-service/plugins \
  --data name=rate-limiting \
  --data config.minute=100 \
  --data config.policy=local

Scoping precedence (most to least specific): Route + Consumer > Route > Service > Consumer > Global. Understanding this precedence is essential for predicting which plugin instance actually fires for a given request.

2.6 Certificate & SNI

TLS certificates and their associated Server Name Indication hostnames, used for SNI-based routing and termination at the edge.

2.7 Vault (Enterprise / 3.x+)

Secret references ({vault://...}) let you store sensitive config values (API keys, TLS keys) in HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or environment variables instead of plaintext in Kong’s config.

2.8 Entity Relationship Diagram (Conceptual)

Consumer ──credentials──▶ (key-auth / jwt / oauth2 / basic-auth / hmac / mtls)
Consumer ──belongs to───▶ ACL Group
Service  ──has many─────▶ Routes
Service  ──has many─────▶ Plugins
Route    ──has many─────▶ Plugins
Service  ──points to────▶ Upstream ──has many──▶ Targets

3. Deployment Models

3.1 Traditional (DB-backed)

Kong nodes read/write config from PostgreSQL. Suitable when you need dynamic runtime changes via the Admin API (self-service dev portals, frequent consumer/plugin churn).

Pros: fully dynamic, Admin API writes take effect immediately across the cluster.
Cons: DB is a dependency; requires migrations (kong migrations up/finish) on upgrade.

3.2 DB-less (Declarative)

Kong loads its entire configuration from a static YAML/JSON file (kong.yml) at startup, or receives it via the Admin API’s /config endpoint. No database required.

Pros: GitOps-friendly, immutable infrastructure, faster startup, no DB ops burden.
Cons: Admin API becomes read-only for entities (you POST /config a full replacement rather than incrementally PATCHing).

This is the recommended default for most teams today, especially combined with decK (see Section 6).

3.3 Hybrid Mode (Control Plane / Data Plane split)

Control Plane (CP): stores config in PostgreSQL, exposes the Admin API, pushes config down to data planes over a mTLS websocket connection.
Data Plane (DP): stateless, DB-less, proxies traffic only — no direct DB or Admin API access, purely receives config from CP.

Benefits: DPs can scale horizontally and be deployed close to traffic (multi-region) without each one needing DB connectivity; reduces attack surface since only the CP touches sensitive credentials.

        ┌───────────────┐
        │ Control Plane │──── Postgres
        │ (Admin API)   │
        └───────┬───────┘
        mTLS over websocket, config sync
   ┌────────────┼────────────┐
   ▼             ▼            ▼
 DP (us-east)  DP (eu-west)  DP (ap-south)

3.4 Konnect (SaaS Control Plane)

Kong-hosted control plane; you run only data planes (self-hosted or Kong-hosted “Serverless”), and manage config, analytics, and dev portal from Konnect’s UI/API/Terraform provider.

4. Installation & Quick Start

4.1 Docker (DB-less, quickest path)

mkdir -p ~/kong-declarative
cat > ~/kong-declarative/kong.yml << 'EOF'
_format_version: "3.0"
services:
  - name: example-service
    url: https://httpbin.org
    routes:
      - name: example-route
        paths:
          - /example
EOF

docker run -d --name kong \
  -v ~/kong-declarative:/kong/declarative \
  -e "KONG_DATABASE=off" \
  -e "KONG_DECLARATIVE_CONFIG=/kong/declarative/kong.yml" \
  -e "KONG_PROXY_ACCESS_LOG=/dev/stdout" \
  -e "KONG_ADMIN_ACCESS_LOG=/dev/stdout" \
  -e "KONG_PROXY_ERROR_LOG=/dev/stderr" \
  -e "KONG_ADMIN_ERROR_LOG=/dev/stderr" \
  -e "KONG_ADMIN_LISTEN=0.0.0.0:8001" \
  -p 8000:8000 -p 8443:8443 -p 8001:8001 \
  kong:3.7

Test it:

curl http://localhost:8000/example/get

4.2 Docker Compose (DB-backed, for local dev with dynamic Admin API)

version: "3.8"
services:
  kong-database:
    image: postgres:15
    environment:
      POSTGRES_USER: kong
      POSTGRES_DB: kong
      POSTGRES_PASSWORD: kongpass
    volumes:
      - kong_data:/var/lib/postgresql/data

  kong-migrations:
    image: kong:3.7
    command: kong migrations bootstrap
    environment:
      KONG_DATABASE: postgres
      KONG_PG_HOST: kong-database
      KONG_PG_PASSWORD: kongpass
    depends_on:
      - kong-database

  kong:
    image: kong:3.7
    environment:
      KONG_DATABASE: postgres
      KONG_PG_HOST: kong-database
      KONG_PG_PASSWORD: kongpass
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
    ports:
      - "8000:8000"
      - "8443:8443"
      - "8001:8001"
    depends_on:
      - kong-migrations

volumes:
  kong_data: {}

4.3 Helm (Kubernetes)

helm repo add kong https://charts.konghq.com
helm repo update
helm install kong kong/kong \
  --namespace kong --create-namespace \
  --set ingressController.installCRDs=false \
  --set admin.enabled=true

5. Admin API

The Admin API (default port 8001) is Kong’s control surface. Every entity has a REST resource.

5.1 Core Endpoints

# Create service
curl -X POST :8001/services -d name=svc -d url=http://backend:80

# List services
curl :8001/services

# Update (PATCH partial)
curl -X PATCH :8001/services/svc -d url=http://backend:8081

# Delete
curl -X DELETE :8001/services/svc

# Nested creation (route under a service)
curl -X POST :8001/services/svc/routes -d 'paths[]=/api'

# Enable a plugin globally
curl -X POST :8001/plugins -d name=prometheus

# Validate config without applying (schema check)
curl -X POST :8001/schemas/plugins/validate -d name=rate-limiting -d config.minute=10

5.2 Securing the Admin API

Never expose port 8001 to the public internet. In production:

Bind Admin API to 127.0.0.1 or an internal-only network interface.
Put it behind a VPN, bastion, or internal load balancer with its own auth (mTLS, IP allowlist).
In Hybrid/Konnect mode, data planes don’t need Admin API access at all — restrict it to control planes only.
Consider RBAC (Kong Enterprise) to scope which teams can modify which entities.

5.3 Admin API vs. `kong.conf`

Runtime entities (services, routes, plugins) go through the Admin API or declarative config. Node-level settings (worker processes, listen addresses, log levels, plugin allowlist) go in kong.conf or KONG_* environment variables, and require a reload/restart.

6. Declarative Configuration & decK

6.1 Why decK

decK (deck) is Kong’s CLI tool for managing configuration as code — diffing, syncing, and validating kong.yml files against a running Kong (or Konnect) instance. It is the backbone of GitOps workflows for Kong.

# Dump current state into a file
deck gateway dump -o kong.yml

# Diff local file against live Kong
deck gateway diff -s kong.yml

# Apply local file (creates/updates/deletes to match)
deck gateway sync -s kong.yml

# Validate syntax/schema only, no network call
deck file validate -s kong.yml

# Lint against custom rulesets (naming conventions, required tags, etc.)
deck file lint -s kong.yml -r ruleset.yml

6.2 Example `kong.yml`

_format_version: "3.0"
_transform: true

services:
  - name: catalog-service
    url: http://catalog.internal:8080
    tags: [team-catalog, prod]
    routes:
      - name: catalog-route
        paths: ["/catalog"]
        strip_path: false
    plugins:
      - name: rate-limiting
        config:
          minute: 300
          policy: local
      - name: key-auth
        config:
          key_names: ["apikey"]

consumers:
  - username: partner-a
    keyauth_credentials:
      - key: "abc123-partner-a"
    acls:
      - group: partners

acls: []

upstreams:
  - name: catalog-upstream
    algorithm: round-robin
    healthchecks:
      active:
        http_path: /health
        healthy:
          interval: 5
          successes: 2
        unhealthy:
          interval: 5
          http_failures: 3
    targets:
      - target: 10.0.1.10:8080
        weight: 100
      - target: 10.0.1.11:8080
        weight: 100

6.3 decK Best Practices

Keep one kong.yml per environment (or split into multiple files with deck file merge).
Use tags on every entity — makes selective sync (deck gateway sync --select-tag=team-catalog) possible in shared Kong clusters.
Run deck gateway diff in CI as a PR check before deck gateway sync on merge — never sync blind.
Store secrets as Vault references ({vault://env/API_KEY}), never plaintext, inside versioned kong.yml.
Use deck file openapi2kong to bootstrap Kong config directly from an existing OpenAPI spec.

7. Kubernetes Ingress Controller (KIC)

KIC watches Kubernetes resources and converts them into Kong configuration automatically — no manual Admin API calls needed in a K8s environment.

7.1 Standard Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: orders-ingress
  annotations:
    konghq.com/strip-path: "true"
  labels:
    konghq.com/plugins: rate-limit-orders
spec:
  ingressClassName: kong
  rules:
    - http:
        paths:
          - path: /orders
            pathType: Prefix
            backend:
              service:
                name: orders-svc
                port:
                  number: 80

7.2 Kong CRDs

Kong extends Kubernetes with Custom Resource Definitions for capabilities plain Ingress can’t express:

CRD	Purpose
`KongPlugin` / `KongClusterPlugin`	Define a plugin config, attach via annotation or label
`KongConsumer`	Map a K8s identity to a Kong Consumer
`KongIngress`	Fine-grained routing/upstream config (legacy, mostly superseded)
`KongCredential`	Attach credentials (key-auth, jwt, etc.) to a `KongConsumer`
`TCPIngress` / `UDPIngress`	Route non-HTTP TCP/UDP traffic
`KongVault`	Reference external secret backends

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: rate-limit-orders
plugin: rate-limiting
config:
  minute: 100
  policy: local
---
apiVersion: v1
kind: Service
metadata:
  name: orders-svc
  annotations:
    konghq.com/plugins: rate-limit-orders

apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: mobile-app
  annotations:
    kubernetes.io/ingress.class: kong
username: mobile-app
credentials:
  - mobile-app-apikey
---
apiVersion: v1
kind: Secret
metadata:
  name: mobile-app-apikey
type: Opaque
stringData:
  kongCredType: key-auth
  key: mobile-secret-key

7.3 Gateway API Support

Modern KIC versions support the Kubernetes Gateway API (Gateway, HTTPRoute, TCPRoute, GRPCRoute) as an alternative to Ingress — the direction the K8s ecosystem is moving for more expressive, portable traffic routing (header matching, weighted backends for canary, cross-namespace routing).

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: orders-route
spec:
  parentRefs:
    - name: kong
  rules:
    - matches:
        - path: { type: PathPrefix, value: /orders }
      backendRefs:
        - name: orders-svc
          port: 80
          weight: 90
        - name: orders-svc-canary
          port: 80
          weight: 10

8. Authentication Plugins

8.1 Key Authentication

Simplest scheme — clients send an API key in a header, query param, or body.

curl -X POST :8001/services/svc/plugins -d name=key-auth -d config.key_names=apikey
curl -X POST :8001/consumers/alice/key-auth -d key=alice-secret-key

Client call: curl -H "apikey: alice-secret-key" https://api.example.com/svc

8.2 JWT

Kong validates a JWT signed by the client’s issuer (RS256/HS256), without calling an external auth server per-request.

curl -X POST :8001/services/svc/plugins -d name=jwt
curl -X POST :8001/consumers/alice/jwt \
  -d algorithm=RS256 \
  -d rsa_public_key="$(cat pubkey.pem)"

Note: Kong’s jwt plugin validates signature/expiry — it does not call an OIDC provider’s introspection endpoint. For full OAuth2/OIDC flows with token introspection, use the OpenID Connect plugin (Enterprise) instead.

8.3 OAuth2

Implements the OAuth2 framework directly inside Kong (authorization code, client credentials, implicit, password grants).

curl -X POST :8001/services/svc/plugins -d name=oauth2 \
  -d config.enable_authorization_code=true \
  -d config.scopes=read,write

For most modern setups, teams instead terminate OAuth2/OIDC at an external IdP (Auth0, Okta, Keycloak) and use Kong’s OpenID Connect plugin to validate tokens and handle discovery/introspection/JWKS.

8.4 Basic Auth / LDAP / HMAC / mTLS

basic-auth: username/password over HTTP Basic — fine for internal/service-to-service, not for public APIs without TLS.
ldap-auth: authenticate against an LDAP directory (corporate SSO scenarios).
hmac-auth: signature-based auth (client signs the request with a shared secret) — strong integrity guarantee, common for webhook/partner integrations.
mtls-auth: mutual TLS, client certificate identifies the Consumer — the strongest option, common in zero-trust/service-mesh-adjacent architectures.

8.5 Multi-Auth Pattern (OR logic)

To accept either key-auth or JWT on the same route, apply both plugins and set config.anonymous to a shared anonymous consumer on each, then chain with the request-termination or custom logic to enforce “at least one succeeded.” In Kong 3.x, use the native anonymous consumer OR-auth chaining: each auth plugin gets the same config.anonymous=<uuid>; if the first plugin fails, it falls through as anonymous, and the second plugin gets a real chance to authenticate — if both fail, the request is rejected downstream (typically by an ACL plugin blocking the anonymous group).

9. Traffic Control Plugins

9.1 Rate Limiting

curl -X POST :8001/services/svc/plugins -d name=rate-limiting \
  -d config.minute=60 \
  -d config.hour=1000 \
  -d config.policy=redis \
  -d config.redis.host=redis.internal

policy=local: counts kept per-node in memory — fast, but inconsistent across a multi-node cluster (each node enforces its own limit independently).
policy=cluster: uses the Kong database as a shared counter — accurate but adds DB load; unavailable in DB-less mode.
policy=redis: shared counter in Redis — the standard choice for multi-node production deployments needing accurate global limits.

Rate-limiting-advanced (Enterprise) adds sliding-window algorithms, multiple limits per plugin instance, and cost-based limiting.

9.2 ACL (Access Control Lists)

Combine with any auth plugin to whitelist/blacklist Consumer groups per route.

curl -X POST :8001/consumers/alice/acls -d group=partners
curl -X POST :8001/routes/orders-route/plugins -d name=acl -d config.allow=partners

9.3 IP Restriction

curl -X POST :8001/services/svc/plugins -d name=ip-restriction \
  -d config.allow=10.0.0.0/8 -d config.allow=203.0.113.5

9.4 Request Size Limiting

curl -X POST :8001/services/svc/plugins -d name=request-size-limiting \
  -d config.allowed_payload_size=10

9.5 Proxy Caching

Caches upstream responses (per method/status/vary) to reduce backend load for cacheable GET endpoints.

curl -X POST :8001/services/svc/plugins -d name=proxy-cache \
  -d config.content_type="application/json" \
  -d config.cache_ttl=300 \
  -d config.strategy=memory

9.6 Circuit Breaker Patterns

Kong doesn’t ship a first-party “circuit breaker” plugin by name, but achieves the pattern via:

Upstream active/passive health checks — automatically stop routing to unhealthy targets (see Section 12).
Enterprise proxy-cache-advanced + request-termination — serve stale cache or a fallback response when upstream is down.
Third-party/community circuit-breaker plugins built with the PDK for custom failure-threshold logic.

10. Transformation Plugins

10.1 Request Transformer

curl -X POST :8001/routes/orders-route/plugins -d name=request-transformer \
  -d config.add.headers=X-Request-Source:kong \
  -d config.remove.headers=X-Internal-Debug \
  -d config.rename.headers=X-Old-Name:X-New-Name

10.2 Response Transformer

curl -X POST :8001/routes/orders-route/plugins -d name=response-transformer \
  -d config.remove.json=internal_id \
  -d config.add.headers=X-Powered-By:Kong

10.3 Correlation ID

Injects/propagates a unique ID per request for distributed tracing correlation.

curl -X POST :8001/services/svc/plugins -d name=correlation-id \
  -d config.header_name=X-Correlation-ID \
  -d config.generator=uuid \
  -d config.echo_downstream=true

10.4 gRPC Transcoding

grpc-gateway and grpc-web plugins let HTTP/JSON clients talk to gRPC backends, and browser gRPC-Web clients talk to standard gRPC services, respectively — useful when exposing internal gRPC microservices to REST-only consumers without rewriting them.

11. Logging & Observability

11.1 Prometheus Metrics

curl -X POST :8001/plugins -d name=prometheus \
  -d config.status_code_metrics=true \
  -d config.latency_metrics=true \
  -d config.bandwidth_metrics=true

Scrape GET /metrics on the Admin API port (or a dedicated status port 8100 if configured via KONG_STATUS_LISTEN). Key metrics: kong_http_requests_total, kong_latency_bucket, kong_bandwidth_bytes, kong_upstream_target_health.

11.2 Structured Logging

Plugin	Destination
`file-log`	Local file (JSON lines)
`http-log`	HTTP endpoint (your log aggregator’s ingest API)
`tcp-log` / `udp-log`	Syslog-style forwarders
`syslog`	Local syslog
`datadog`	Datadog metrics + logs
`zipkin` / `opentelemetry`	Distributed tracing spans

curl -X POST :8001/services/svc/plugins -d name=http-log \
  -d config.http_endpoint=https://logs.example.com/ingest \
  -d config.timeout=5000 \
  -d config.keepalive=5000

11.3 OpenTelemetry

The opentelemetry plugin exports traces (and optionally logs/metrics) in OTLP format to any OTel-compatible backend (Jaeger, Tempo, Honeycomb, Datadog, etc.) — the modern default for distributed tracing, superseding the older Zipkin plugin in new deployments.

curl -X POST :8001/plugins -d name=opentelemetry \
  -d config.endpoint=http://otel-collector:4318/v1/traces \
  -d config.resource_attributes.service.name=kong-gateway

11.4 Dashboards

Kong publishes official Grafana dashboards for the Prometheus plugin output — track request rate, p99 latency, upstream health, and per-consumer usage. Pair with alerting on kong_upstream_target_health transitions and 5xx rate spikes.

12. Load Balancing & Health Checks

12.1 Load Balancing Algorithms

Algorithm	Behavior
`round-robin`	Even rotation, respects target `weight`
`consistent-hashing`	Hash on IP, header, cookie, or query param — sticky routing without server-side sessions
`least-connections`	Sends traffic to the target with fewest active connections

curl -X PATCH :8001/upstreams/orders-upstream \
  -d algorithm=consistent-hashing \
  -d hash_on=header \
  -d hash_on_header=X-Session-ID

12.2 Active Health Checks

Kong proactively probes targets on an interval.

curl -X PATCH :8001/upstreams/orders-upstream \
  -d healthchecks.active.http_path=/health \
  -d healthchecks.active.healthy.interval=5 \
  -d healthchecks.active.healthy.successes=2 \
  -d healthchecks.active.unhealthy.interval=5 \
  -d healthchecks.active.unhealthy.http_failures=3 \
  -d healthchecks.active.unhealthy.tcp_failures=3

12.3 Passive Health Checks

Kong observes real proxied traffic — if a target returns enough consecutive failures (5xx, timeouts), it’s marked unhealthy without a dedicated probe.

curl -X PATCH :8001/upstreams/orders-upstream \
  -d healthchecks.passive.unhealthy.http_failures=5 \
  -d healthchecks.passive.unhealthy.timeouts=3

Best practice: use both. Active checks catch a target that’s down before it receives live traffic; passive checks catch failures active probes might miss (e.g., a /health endpoint that’s healthy but the actual API path is broken).

12.4 Zero-Downtime Backend Swaps

Add the new target with the same or higher weight, monitor health/error rates, then reduce the old target’s weight to 0 and remove it — no route/service change needed, no client impact.

13. Custom Plugin Development (Lua)

13.1 Plugin Anatomy

A Kong plugin is a Lua module with a handler.lua (logic) and schema.lua (config validation).

my-plugin/
├── kong/plugins/my-plugin/
│   ├── handler.lua
│   └── schema.lua
└── my-plugin-1.0.0-1.rockspec

schema.lua

return {
  name = "my-plugin",
  fields = {
    { config = {
        type = "record",
        fields = {
          { header_name = { type = "string", default = "X-My-Plugin" } },
          { header_value = { type = "string", required = true } },
        },
      },
    },
  },
}

handler.lua

local MyPluginHandler = {
  PRIORITY = 1000, -- execution order relative to other plugins
  VERSION = "1.0.0",
}

function MyPluginHandler:access(conf)
  kong.service.request.set_header(conf.header_name, conf.header_value)
end

function MyPluginHandler:header_filter(conf)
  kong.response.set_header("X-Processed-By", "my-plugin")
end

return MyPluginHandler

13.2 Plugin Priority

PRIORITY is an integer; higher runs first within the same phase. Reference points from bundled plugins:

Plugin	Priority	Phase concern
`pre-function`	1000000+	Runs before nearly everything
`cors`	2000	Must run before auth to set headers on preflight
`key-auth`/`jwt`/`oauth2`	~1000-1200	Authentication
`acl`	~950	After auth, needs authenticated consumer
`rate-limiting`	~900	After auth, needs consumer identity for per-consumer limits
`request-transformer`	~800	After auth/rate-limiting decisions are made
`post-function`	-1000000	Runs after nearly everything

13.3 The Plugin Development Kit (PDK)

The PDK (kong.* namespace) is the stable API surface plugins should use instead of touching raw NGINX/OpenResty APIs — kong.request, kong.response, kong.service, kong.log, kong.client, kong.ctx, kong.vault. Using the PDK insulates your plugin from internal Kong changes across versions.

13.4 Serverless Functions (No Compiled Plugin Needed)

For quick logic without packaging a full plugin, use pre-function / post-function (OSS) or the serverless-functions (Enterprise, supports multiple languages) plugins to inject inline Lua at request time via the Admin API.

curl -X POST :8001/routes/orders-route/plugins -d name=pre-function \
  --data-urlencode 'config.access[1]=kong.service.request.set_header("X-Injected", "yes")'

13.5 Testing Plugins

Kong ships Pongo, a Docker-based dev/test environment for plugin authors:

pongo run        # start a disposable Kong + deps
pongo lint        # luacheck
pongo run spec/   # busted unit/integration tests

14. Security Best Practices

Lock down the Admin API — internal network only, mTLS, or RBAC (see 5.2). This is the single highest-impact security control.
TLS everywhere — terminate TLS at Kong, enforce protocols: [https] on public routes, redirect HTTP → HTTPS.
Rotate credentials — key-auth keys and JWT signing keys should be rotatable without downtime (create new credential, migrate clients, revoke old one).
Use Vault/secrets managers for plugin config values (Redis passwords, upstream credentials) instead of plaintext in kong.yml or the DB.
Least-privilege RBAC (Enterprise) — scope workspace/entity access per team; don’t give every engineer global Admin API rights.
Validate and sanitize with request-validator plugin (JSON Schema/OpenAPI-based request validation) to reject malformed payloads before they reach upstream.
Apply rate limiting and IP restriction defensively, even on internal services — assume the network perimeter will eventually be breached (zero-trust posture).
Enable bot-detection / ip-restriction / referrer restriction on public-facing consumer APIs to blunt scraping and credential stuffing.
Audit logging (Enterprise Admin API audit log, or a post-function capturing Admin API calls) so config changes are traceable.
Pin Kong and plugin versions, review changelogs before upgrading — Kong ships CVE fixes regularly; stay current but test in staging first.
Don’t trust client-supplied headers for identity unless explicitly stripped/overwritten by Kong — an attacker can spoof X-Consumer-* style headers if Kong doesn’t strip them before proxying (Kong does this by default for its own injected headers, but be careful with custom transformer configs).

15. Performance Tuning

15.1 Worker Processes & Connections

KONG_NGINX_WORKER_PROCESSES=auto
KONG_NGINX_WORKER_CONNECTIONS=16384

Set worker_processes to match available CPU cores (auto does this automatically).

15.2 Database Cache

DB-backed Kong caches entities in a shared-memory LRU (mem_cache_size, default 128m) — undersized caches cause frequent DB round-trips (cache misses) under load. Increase for clusters with many entities.

KONG_MEM_CACHE_SIZE=512m

15.3 DB-less/Hybrid for Scale

DB-less data planes avoid per-request DB dependency entirely, which is the single biggest lever for high-throughput, low-latency proxying at scale — this is why Hybrid mode is the recommended pattern for large production clusters.

15.4 Plugin Overhead

Every enabled plugin adds latency. Audit unused/global plugins regularly. Prefer scoping plugins to the routes/services that need them over blanket global application when the logic doesn’t need to run universally. rate-limiting with policy=redis adds a network round-trip per request — colocate Redis close to Kong nodes, or use policy=local when perfect cross-node accuracy isn’t required.

15.5 Keepalive & Upstream Connections

Tune upstream keepalive pools so Kong reuses connections to backends instead of reconnecting per request:

curl -X PATCH :8001/upstreams/orders-upstream \
  -d 'keepalive_pool_size=60' \
  -d 'keepalive_idle_timeout=60'

15.6 Load Testing

Benchmark with k6, wrk, or hey against a realistic plugin-enabled config (not a bare passthrough) — auth/rate-limiting overhead is what actually matters in production, not the raw proxy baseline.

16. Production Patterns

16.1 API Versioning

URI versioning: /v1/orders, /v2/orders as separate Routes on the same or different Services — simplest, most explicit, easiest to deprecate.
Header versioning: Accept: application/vnd.company.v2+json, matched via Route header conditions — cleaner URLs, more complex routing rules.
Run old and new versions as separate Services pointing at separate Upstreams so you can retire v1’s backend independently.

16.2 Canary / Blue-Green Deployments

Weighted targets within one Upstream give you canary releases without any client-visible change:

curl -X POST :8001/upstreams/orders-upstream/targets -d target=10.0.2.20:8080 -d weight=10   # canary, 10%
curl -X PATCH :8001/upstreams/orders-upstream/targets/10.0.1.10:8080 -d weight=90            # stable, 90%

Gradually shift weight to the canary target as confidence grows; roll back instantly by setting its weight to 0.

16.3 Backend-for-Frontend (BFF)

Expose distinct Kong Services/Routes per client type (/mobile/orders, /web/orders) each pointing at the same or different upstream composition, with tailored transformation plugins per BFF surface (trim fields for mobile bandwidth, expand for web).

16.4 Multi-Tenancy

Workspaces (Enterprise): logically isolate configuration per team/tenant within one Kong cluster, each with its own RBAC scope.
Consumer Groups (OSS 3.x+): apply differentiated plugin config (e.g., different rate limits) to segments of consumers without duplicating plugin instances per consumer.

curl -X POST :8001/consumer_groups -d name=premium-tier
curl -X POST :8001/consumer_groups/premium-tier/consumers -d consumer=alice
curl -X POST :8001/consumer_groups/premium-tier/overrides/plugins/rate-limiting-advanced \
  -d config.limit=10000 -d config.window_size=60

16.5 API Composition / Aggregation

Kong itself doesn’t do response aggregation (combining multiple backend calls into one response) — that’s a BFF/GraphQL-gateway concern. Pair Kong with a dedicated aggregation layer (or the grpc-gateway plugin for protocol bridging) rather than forcing this into a Kong plugin.

16.6 Request Validation at the Edge

Use request-validator with a JSON Schema or OpenAPI spec to reject invalid requests before they consume upstream compute — especially valuable for public-facing, high-volume APIs.

17. CI/CD & GitOps

17.1 Recommended Pipeline

1. Developer edits kong.yml (or OpenAPI spec → deck file openapi2kong)
2. PR opened → CI runs `deck file validate` + `deck file lint`
3. CI runs `deck gateway diff` against staging → posts diff as PR comment
4. On merge to main → CD runs `deck gateway sync` against staging
5. Manual/automated promotion → `deck gateway sync` against production

17.2 Example GitHub Actions Step

- name: Validate Kong config
  run: deck file validate -s kong.yml

- name: Diff against staging
  run: deck gateway diff -s kong.yml --kong-addr https://staging-admin.internal:8001

- name: Sync to staging
  if: github.ref == 'refs/heads/main'
  run: deck gateway sync -s kong.yml --kong-addr https://staging-admin.internal:8001

17.3 Terraform (Konnect Provider)

For Konnect-managed control planes, the official Terraform provider (kong/konnect) lets you manage control planes, services, routes, and plugins as Terraform resources — useful when Kong config lives alongside other infra-as-code.

17.4 Safe Rollout Discipline

Always diff before sync — never sync blind, even in automated pipelines (fail the pipeline on unexpected diff size/scope).
Tag-scope syncs (--select-tag) in shared clusters so one team’s pipeline can’t accidentally delete another team’s entities.
Keep a rollback kong.yml (previous good state) readily syncable.

18. Troubleshooting

Symptom	Likely Cause	Fix
`no Route matched` (404)	Path/host/method mismatch, wrong `strip_path`	Check `GET /routes` config, test with `X-Forwarded-*` headers if behind LB
Plugin not firing	Wrong scope (attached to wrong service/route), disabled	`GET /plugins?service.id=` to confirm scope and `enabled: true`
502/504 from Kong	Upstream unreachable, timeout too low, DNS resolution failing	Check `upstream_connect_timeout`, verify Target health, check Kong’s DNS resolver config
Inconsistent rate limits across nodes	Using `policy=local` in multi-node cluster	Switch to `policy=redis` or `cluster`
Admin API changes not taking effect	DB-less mode — Admin API is read-only for entities	`POST /config` with full declarative payload, or use `deck sync`
High latency added by plugins	Redis round-trip, external HTTP log endpoint slow	Move to async logging (`http-log` is async by default), colocate Redis, batch logs
SSL handshake failures	Wrong SNI mapping, missing intermediate cert chain	Check `GET /certificates` and `GET /snis`, verify chain with `openssl s_client`
Data plane not connecting to control plane (Hybrid)	mTLS cert mismatch, clock skew, network/firewall block on cluster port	Check `kong.conf` `cluster_cert`/`cluster_cert_key`, verify port 8005/8006 reachable
`Cannot invoke handler` after custom plugin deploy	Lua syntax error, plugin not in `KONG_PLUGINS` list	Check `kong.conf`/env for `plugins = bundled,my-plugin`, review error log

18.1 Useful Debug Commands

kong config db_export         # export current DB-backed config to file
kong check kong.yml           # validate declarative file syntax
kong health                   # basic health check of local node
curl :8001/status             # node status: connections, memory, DB reachability
curl :8001/status/ready       # readiness probe (K8s-friendly)

19. CLI & Admin API Cheat Sheet

# --- Kong CLI ---
kong start -c kong.conf
kong stop
kong reload -c kong.conf
kong migrations bootstrap      # first-time DB setup
kong migrations up             # apply pending migrations
kong migrations finish         # finalize after zero-downtime upgrade
kong version

# --- Admin API: Services & Routes ---
curl :8001/services
curl :8001/routes
curl -X POST :8001/services -d name=x -d url=http://x:80
curl -X POST :8001/services/x/routes -d 'paths[]=/x'

# --- Admin API: Plugins ---
curl :8001/plugins
curl :8001/plugins/enabled                 # list of plugins compiled into this node
curl -X POST :8001/plugins -d name=cors    # global scope

# --- Admin API: Consumers & Credentials ---
curl -X POST :8001/consumers -d username=bob
curl -X POST :8001/consumers/bob/key-auth -d key=bob-key
curl -X POST :8001/consumers/bob/acls -d group=default

# --- Admin API: Upstreams & Targets ---
curl :8001/upstreams
curl :8001/upstreams/my-upstream/health
curl -X POST :8001/upstreams/my-upstream/targets -d target=1.2.3.4:80

# --- decK ---
deck gateway dump -o kong.yml
deck gateway sync -s kong.yml
deck gateway diff -s kong.yml
deck file validate -s kong.yml
deck file openapi2kong -s openapi.yml -o kong.yml

# --- Node status ---
curl :8001/status
curl :8001/status/ready

Table of Contents