The Complete Flux CD v2 + Kustomize Developer Guide

A deep, practical, production-oriented reference covering architecture, CRDs, patterns, and best practices for GitOps with Flux v2 and Kustomize.

Core Concepts
Flux v2 Architecture
Installation & Bootstrap
Kustomize Fundamentals
Flux Source APIs
The Kustomization CRD (Flux)
Repository Structure Patterns
Multi-Environment & Multi-Tenancy
Dependency Management
Secrets Management
HelmRelease & Helm Integration
Image Automation
Health Checks, Pruning & Reconciliation
Notifications & Alerts
Advanced Kustomize Patterns
Best Practices Checklist
Troubleshooting & Debugging
CLI Cheat Sheet
Full Reference Repository Layout

1. Core Concepts

1.1 What is GitOps?

GitOps is an operating model where Git is the single source of truth for declarative infrastructure and application state. A reconciler (Flux) continuously compares the desired state (Git) with the actual state (cluster) and converges them.

Key GitOps principles:

Declarative — the entire system is described declaratively (YAML manifests).
Versioned & Immutable — the desired state is stored in Git, giving history, audit trail, and rollback.
Pulled automatically — software agents (Flux controllers) pull the desired state, rather than a CI pipeline pushing changes into the cluster.
Continuously reconciled — the agents observe actual state and correct drift automatically.

1.2 What is Kustomize?

Kustomize is a template-free configuration customization tool built into kubectl (kubectl apply -k). Instead of using placeholders/templates (like Helm), Kustomize works by:

Defining a base (a set of plain Kubernetes YAML manifests).
Defining overlays that patch/transform the base for different environments.
Using generators (ConfigMap/Secret) and transformers (namespace, labels, prefixes/suffixes, images, replicas) declared in a kustomization.yaml.

1.3 What is Flux v2?

Flux v2 (also called the “GitOps Toolkit”) is a set of Kubernetes-native controllers (CRDs + controllers) that implement GitOps continuous delivery. Unlike Flux v1 (a single monolithic binary), Flux v2 is composed of specialized, composable controllers, each responsible for one concern:

Controller	Responsibility
`source-controller`	Fetches and caches sources (Git, Helm, OCI, Bucket) as artifacts
`kustomize-controller`	Builds & applies Kustomize overlays from sources
`helm-controller`	Manages Helm releases declaratively via `HelmRelease`
`notification-controller`	Handles events/alerts to Slack, MS Teams, webhooks, etc.
`image-reflector-controller`	Scans container registries for new image tags
`image-automation-controller`	Writes image tag updates back to Git

1.4 Why Flux + Kustomize Together?

Flux natively understands Kustomize overlays as first-class citizens via the Kustomization CRD. This combination lets you:

Keep manifests template-free and readable.
Layer environment-specific patches cleanly (dev/staging/prod).
Let Flux continuously reconcile what kubectl apply -k would produce, with drift detection, pruning, health checks, and dependency ordering.

2. Flux v2 Architecture

                ┌─────────────────────┐
   Git Repo --> │  source-controller   │ --> Artifact (tarball, cached)
   OCI Repo --> │  (polls / webhooks)  │
   Helm Repo -->│                      │
   Bucket    -->│                      │
                └─────────┬────────────┘
                          │ watches Artifact
                          v
                ┌─────────────────────┐
                │ kustomize-controller │ --> kubectl apply -k (build+apply)
                └─────────┬────────────┘
                          │
                ┌─────────────────────┐
                │   helm-controller    │ --> Helm install/upgrade
                └─────────┬────────────┘
                          │
                ┌─────────────────────┐
                │ notification-controller│ --> Slack/Teams/Webhook events
                └─────────────────────┘

                ┌────────────────────────────┐
                │ image-reflector-controller  │ --> scans registries
                │ image-automation-controller │ --> commits tag bumps to Git
                └────────────────────────────┘

Key design principle: every controller works only with Kubernetes Custom Resources. There is no external database or state store — the cluster’s etcd is the state store, and Git is the desired-state store.

2.1 Toolkit API Groups

API Group	Kinds
`source.toolkit.fluxcd.io`	`GitRepository`, `OCIRepository`, `HelmRepository`, `HelmChart`, `Bucket`
`kustomize.toolkit.fluxcd.io`	`Kustomization`
`helm.toolkit.fluxcd.io`	`HelmRelease`
`notification.toolkit.fluxcd.io`	`Alert`, `Provider`, `Receiver`
`image.toolkit.fluxcd.io`	`ImageRepository`, `ImagePolicy`, `ImageUpdateAutomation`

3. Installation & Bootstrap

3.1 Install the Flux CLI

curl -s https://fluxcd.io/install.sh | sudo bash
# or
brew install fluxcd/tap/flux

3.2 Pre-flight Check

flux check --pre

3.3 Bootstrap (GitHub example)

flux bootstrap installs the controllers and commits their manifests into your Git repo, so Flux manages itself via GitOps too.

export GITHUB_TOKEN=<token>
export GITHUB_USER=<user>

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/production \
  --personal

This creates:

clusters/production/flux-system/
├── gotk-components.yaml   # controllers, CRDs, RBAC
├── gotk-sync.yaml         # GitRepository + Kustomization pointing at itself
└── kustomization.yaml

3.4 Bootstrap for GitLab / Generic Git

flux bootstrap gitlab \
  --owner=my-group \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/production \
  --token-auth

For providers without native bootstrap support (Bitbucket, Azure DevOps, on-prem Git), use flux bootstrap git:

flux bootstrap git \
  --url=ssh://git@example.com/fleet-infra.git \
  --branch=main \
  --path=clusters/production \
  --private-key-file=./id_ed25519

3.5 Multi-Cluster Bootstrap Pattern

clusters/
├── staging/
│   └── flux-system/
├── production-eu/
│   └── flux-system/
└── production-us/
    └── flux-system/

Each cluster gets its own flux-system sync path, all pointing at the same repository but different directories — enabling a “fleet” management model.

3.6 Uninstall

flux uninstall --namespace=flux-system

4. Kustomize Fundamentals

4.1 Anatomy of `kustomization.yaml`

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml
  - ../../base

namePrefix: prod-
nameSuffix: "-v1"
namespace: production

commonLabels:
  app.kubernetes.io/managed-by: flux
  environment: production

commonAnnotations:
  team: platform-engineering

images:
  - name: myapp
    newName: registry.example.com/myapp
    newTag: 1.4.2

replicas:
  - name: myapp
    count: 3

configMapGenerator:
  - name: app-config
    literals:
      - LOG_LEVEL=info
    files:
      - config.properties

secretGenerator:
  - name: app-secret
    envs:
      - secrets.env

patches:
  - path: patch-resources.yaml
    target:
      kind: Deployment
      name: myapp

components:
  - ../../components/istio-sidecar

4.2 Base & Overlay Model

base/
├── kustomization.yaml
├── deployment.yaml
├── service.yaml
└── configmap.yaml

overlays/
├── dev/
│   ├── kustomization.yaml
│   └── patch-replicas.yaml
├── staging/
│   ├── kustomization.yaml
│   └── patch-resources.yaml
└── production/
    ├── kustomization.yaml
    ├── patch-resources.yaml
    └── patch-hpa.yaml

Base contains the canonical, environment-agnostic manifests. Overlays reference the base via resources: [../../base] and apply patches/transforms.

4.3 Strategic Merge Patch vs JSON 6902

Strategic Merge Patch (preferred for most edits — merges by field, understands Kubernetes list semantics like containers by name):

# patch-resources.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
        - name: myapp
          resources:
            limits:
              cpu: "1"
              memory: 512Mi

patches:
  - path: patch-resources.yaml
    target:
      kind: Deployment
      name: myapp

JSON 6902 Patch (precise, path-based — useful for removing fields or array index operations):

patches:
  - target:
      kind: Deployment
      name: myapp
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 5
      - op: remove
        path: /spec/template/spec/containers/0/livenessProbe

4.4 Generators

ConfigMapGenerator — content-hashed, immutable ConfigMaps that trigger rollouts automatically when content changes:

configMapGenerator:
  - name: app-config
    literals:
      - ENV=production
    files:
      - application.yaml
    options:
      disableNameSuffixHash: false   # keep hash suffix for auto rollout

Generated name becomes e.g. app-config-8f92bd7c6t. Any Deployment referencing app-config gets automatically rewritten to the hashed name (nameReference transformer) — and Kubernetes triggers a rolling restart because the ConfigMap name changed.

SecretGenerator:

secretGenerator:
  - name: db-secret
    type: Opaque
    envs:
      - db.env

⚠️ Never commit plaintext secrets. Combine with SOPS or External Secrets (see Section 10).

4.5 Components (reusable partial overlays)

Components allow injecting reusable, composable pieces of config (introduced in kustomize.config.k8s.io/v1alpha1 Component kind) that can be mixed into multiple overlays:

# components/istio-sidecar/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
  - path: inject-sidecar-annotation.yaml
    target:
      kind: Deployment

Used in an overlay:

components:
  - ../../components/istio-sidecar
  - ../../components/pod-disruption-budget

Components are the recommended way to share cross-cutting concerns (e.g., sidecars, PDBs, network policies) across overlays that otherwise share little structure.

4.6 Transformers Reference

Field	Purpose
`namePrefix` / `nameSuffix`	Prepend/append string to all resource names
`namespace`	Force a namespace on all namespaced resources
`commonLabels`	Add labels + update label selectors consistently
`commonAnnotations`	Add annotations to all resources
`images`	Override image name/tag/digest
`replicas`	Override replica count by resource name
`vars` (deprecated)	Legacy variable substitution — replaced by `replacements`
`replacements`	Modern field-to-field value copying across resources
`patchesStrategicMerge` (deprecated)	Legacy — use `patches`
`patchesJson6902` (deprecated)	Legacy — use `patches`

4.7 `replacements` (modern variable substitution)

replacements:
  - source:
      kind: ConfigMap
      name: app-config
      fieldPath: data.API_URL
    targets:
      - select:
          kind: Deployment
          name: myapp
        fieldPaths:
          - spec.template.spec.containers.[name=myapp].env.[name=API_URL].value

This replaces the old vars: field, which is deprecated because it broke Kustomize’s declarative, side-effect-free build model.

4.8 Ordering & Merge Semantics

Kustomize applies transformers in a fixed internal order (generators → patches → images → replicas → labels/annotations/namespace), not the order you list them.
Patches within the patches: list are applied in the order listed, so ordering matters when patches touch the same field.
resources: order determines the apply/output order in kustomize build output — relevant for CRD-before-CR dependencies within the same Kustomization.

4.9 Validating & Building Locally

kustomize build overlays/production
kubectl kustomize overlays/production        # equivalent, uses built-in kustomize
kubectl apply -k overlays/production          # build + apply
kubectl diff -k overlays/production           # preview changes against live cluster

Always run kustomize build locally (or in CI) before merging — this is the #1 way to catch overlay errors before Flux does.

5. Flux Source APIs

5.1 GitRepository

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo
  ref:
    branch: master
  ignore: |
    /*
    !/kustomize
  secretRef:
    name: https-credentials

Key fields:

interval — polling frequency (webhooks can trigger immediate sync too).
ref — branch, tag, semver, or commit.
ignore — .gitignore-style filter to exclude files from the produced artifact.
secretRef — basic-auth or SSH credentials for private repos.
verify — Cosign/GPG commit signature verification.

5.2 OCIRepository (Flux v2 supports OCI as a source, not just Helm charts)

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/manifests/podinfo
  ref:
    tag: latest
  layerSelector:
    mediaType: "application/vnd.cncf.flux.content.v1.tar+gzip"
    operation: extract

OCI is increasingly preferred over Git as a distribution mechanism for immutable, versioned manifest bundles — separating the “build & push manifests” step from the Git source of truth.

5.3 HelmRepository

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m
  url: https://stefanprodan.github.io/podinfo
  # type: oci   # for OCI-based Helm repos

5.4 Bucket

apiVersion: source.toolkit.fluxcd.io/v1
kind: Bucket
metadata:
  name: my-artifacts
  namespace: flux-system
spec:
  interval: 5m
  provider: aws
  bucketName: my-manifests-bucket
  endpoint: s3.amazonaws.com
  region: eu-west-1
  secretRef:
    name: aws-credentials

5.5 Source Verification (Supply Chain Security)

spec:
  verify:
    provider: cosign
    secretRef:
      name: cosign-pub

Enforces that only cryptographically signed sources (commits or OCI artifacts) are reconciled — an important supply-chain security control.

6. The Kustomization CRD (Flux)

This is Flux’s own Kustomization object — not to be confused with the plain kustomization.yaml file used by the Kustomize tool. Flux’s Kustomization CRD tells the kustomize-controller to build a Kustomize overlay from a source and apply it.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m
  retryInterval: 2m
  timeout: 3m
  sourceRef:
    kind: GitRepository
    name: podinfo
  path: "./kustomize"
  prune: true
  wait: true
  targetNamespace: default
  dependsOn:
    - name: infra-controllers
  patches:
    - patch: |
        - op: add
          path: /spec/template/spec/topologySpreadConstraints
          value: []
      target:
        kind: Deployment
        name: podinfo
  postBuild:
    substitute:
      cluster_env: production
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
      - kind: Secret
        name: cluster-secrets
        optional: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: podinfo
      namespace: default
  force: false
  suspend: false

6.1 Key Fields Explained

Field	Purpose
`sourceRef`	Which source (GitRepository/OCIRepository/Bucket) to build from
`path`	Directory inside the source artifact containing the `kustomization.yaml`
`prune`	Deletes resources removed from Git (garbage collection) — essential for true GitOps
`wait`	Waits for all applied resources to become ready before marking `Ready=True`
`healthChecks`	Explicit list of resources to check (implied automatically when `wait: true`)
`dependsOn`	Ordering — this Kustomization won’t reconcile until dependencies are `Ready`
`patches`	Flux-level patches applied after kustomize build — for last-mile overrides without modifying the Git overlay
`postBuild.substitute` / `substituteFrom`	`${VAR}`-style variable substitution against rendered manifests
`targetNamespace`	Forces all resources into a namespace (like Kustomize’s `namespace:` but applied post-build)
`force`	Recreate immutable-field-conflicting resources instead of failing (use sparingly)
`timeout`	Max duration for apply + health check before marking as failed
`retryInterval`	Backoff before retrying after a failure
`suspend`	Pause reconciliation without deleting the object
`decryption`	SOPS decryption provider config
`serviceAccountName`	Impersonate a specific ServiceAccount for apply RBAC (multi-tenancy!)

6.2 `postBuild.substitute` in Practice

Because Kustomize itself has no runtime templating, Flux offers postBuild.substitute for lightweight ${VAR} substitution after the Kustomize build step — useful for cluster-specific values without duplicating overlays.

# in a manifest inside the source repo
metadata:
  annotations:
    cluster: "${cluster_name}"

spec:
  postBuild:
    substitute:
      cluster_name: "eu-prod-1"
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars

Variables not resolved will cause the build to fail unless a default is provided: ${cluster_name:=default-value}.

6.3 Impersonation for Multi-Tenancy

spec:
  serviceAccountName: tenant-a-reconciler

Combined with RBAC bound to that ServiceAccount, this scopes what a given Kustomization is allowed to apply — critical for multi-tenant clusters where teams shouldn’t have cluster-admin-equivalent apply rights.

7. Repository Structure Patterns

7.1 Monorepo (single repo, multiple clusters & apps)

fleet-infra/
├── clusters/
│   ├── staging/
│   │   ├── flux-system/
│   │   └── infrastructure.yaml   # Kustomization -> infrastructure/
│   │   └── apps.yaml             # Kustomization -> apps/staging
│   └── production/
│       ├── flux-system/
│       ├── infrastructure.yaml
│       └── apps.yaml
├── infrastructure/
│   ├── base/
│   │   ├── cert-manager/
│   │   ├── ingress-nginx/
│   │   └── monitoring/
│   └── overlays/
│       ├── staging/
│       └── production/
└── apps/
    ├── base/
    │   └── podinfo/
    └── overlays/
        ├── staging/
        └── production/

7.2 Polyrepo (apps in separate repos, referenced by a central fleet repo)

fleet-infra/                 # central control repo
└── clusters/production/
    ├── flux-system/
    └── podinfo-source.yaml  # GitRepository pointing at app repo

podinfo/                     # separate app repo
├── src/
└── deploy/
    ├── base/
    └── overlays/

Trade-offs:

Pattern	Pros	Cons
Monorepo	Simple, atomic cross-app changes, single PR review flow	Can get large; broader blast radius for permissions
Polyrepo	Clear ownership boundaries, independent release cadence	More moving parts; cross-repo coordination for shared infra

7.3 “Apps of Apps” / Fleet Pattern

A top-level Kustomization per cluster that references an apps directory, which itself is a Kustomize overlay aggregating many app-level Kustomization objects (one per microservice):

# clusters/production/apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: "./apps/production"
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  dependsOn:
    - name: infrastructure

apps/production/kustomization.yaml
resources:
  - podinfo-kustomization.yaml
  - frontend-kustomization.yaml
  - backend-kustomization.yaml

Each *-kustomization.yaml here is a Flux Kustomization CRD instance, not a Kustomize overlay — this pattern lets each app be independently tracked, health-checked, and dependency-ordered.

8. Multi-Environment & Multi-Tenancy

8.1 Environment Overlay Example

# apps/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base/podinfo
namespace: podinfo
patches:
  - path: patch-replicas.yaml
  - path: patch-resources.yaml
images:
  - name: podinfo
    newTag: 6.5.4

# patch-replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
spec:
  replicas: 5

8.2 Tenant Isolation with Flux

Pattern A — Namespace-per-tenant + RBAC impersonation:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenant-a
  namespace: flux-system
spec:
  targetNamespace: tenant-a
  serviceAccountName: tenant-a-reconciler
  sourceRef:
    kind: GitRepository
    name: tenant-a-repo
  path: "./"
  prune: true

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tenant-a-reconciler
  namespace: tenant-a
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-a-reconciler
  namespace: tenant-a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin      # scoped by namespace via RoleBinding, not ClusterRoleBinding
subjects:
  - kind: ServiceAccount
    name: tenant-a-reconciler
    namespace: tenant-a

Pattern B — flux create tenant scaffolding:

Flux ships a multi-tenancy example generator:

flux create tenant tenant-a \
  --with-namespace=tenant-a \
  --export > tenant-a.yaml

This scaffolds a Namespace, ServiceAccount, RoleBinding, and a GitRepository + Kustomization scoped to that tenant, following Flux’s official multi-tenancy guide.

8.3 Cluster API / Fleet Management

For managing many clusters, combine Flux with Cluster API or a fleet repo where each cluster directory bootstraps independently but shares common infrastructure/base and apps/base layers via overlay inheritance.

9. Dependency Management

9.1 `dependsOn` Between Flux Kustomizations

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  dependsOn:
    - name: infra-controllers
    - name: infra-configs
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: "./apps/production"
  interval: 10m
  prune: true

Flux resolves the DAG and reconciles dependencies first, waiting for their Ready condition (which itself depends on wait: true + health checks passing).

9.2 Typical Layering

infra-controllers  (CRDs: cert-manager, ingress-nginx, prometheus-operator)
        │
        v
infra-configs      (ClusterIssuer, IngressClass, Grafana dashboards — CRs that need controllers first)
        │
        v
apps               (actual workloads depending on infra being ready)

9.3 In-Kustomization Ordering (single overlay)

Within one Kustomize overlay, ordering is controlled by:

resources: list order in the final kustomize build output.
Kubernetes’ own eventual consistency (most resources don’t strictly need ordering — but CRDs before CRs do).

For CRD-then-CR ordering issues across Flux Kustomizations, split them into separate Kustomization objects with dependsOn, since kustomize build does not guarantee CRD registration completes before a CR admission webhook validates.

9.4 `wait` and Health Checks Interaction

spec:
  wait: true
  timeout: 5m

When wait: true, Flux automatically infers health checks for every applied resource (Deployments, StatefulSets, custom resources with a status.conditions[type=Ready], etc.) unless you explicitly define healthChecks: to narrow the list.

10. Secrets Management

Never commit plaintext secrets to Git. Three dominant patterns:

10.1 SOPS (Secrets OPerationS) + Flux Native Decryption

# Encrypt with SOPS using age or GPG
sops --encrypt --age <age-public-key> secret.yaml > secret.enc.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age

kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey

Flux decrypts .sops.yaml-managed files transparently at reconcile time. SOPS supports partial encryption (only data:/stringData: values), keeping the rest of the manifest diffable in Git.

10.2 Sealed Secrets (Bitnami)

Encrypt client-side with kubeseal, decrypt only inside the cluster by the sealed-secrets-controller (asymmetric encryption per-cluster):

kubeseal --format=yaml < secret.yaml > sealed-secret.yaml

Commit sealed-secret.yaml — it’s safe because only the target cluster’s controller can decrypt it.

10.3 External Secrets Operator (ESO)

Pull secrets at runtime from an external vault (AWS Secrets Manager, Vault, Azure Key Vault, GCP Secret Manager) — no encrypted material in Git at all, only a reference:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: prod/db/password

10.4 Comparison

Method	Secret in Git?	Rotation	Complexity
SOPS	Encrypted, yes	Manual re-encrypt on change	Low
Sealed Secrets	Encrypted, yes	Manual re-seal on change	Low-Medium
External Secrets Operator	No (only reference)	Automatic (poll interval)	Medium

Recommendation: SOPS with age is the most widely adopted Flux-native pattern for small-to-medium teams; ESO is preferred at scale or when a centralized secret vault already exists.

11. HelmRelease & Helm Integration

Flux can manage Helm charts declaratively via helm-controller, combining Helm’s templating power with GitOps reconciliation and Kustomize’s overlay patching.

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 30m
  url: https://charts.bitnami.com/bitnami
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: redis
  namespace: flux-system
spec:
  interval: 10m
  chart:
    spec:
      chart: redis
      version: "18.x"
      sourceRef:
        kind: HelmRepository
        name: bitnami
      interval: 10m
  values:
    architecture: replication
    auth:
      enabled: true
      existingSecret: redis-auth
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
      remediateLastFailure: true
    cleanupOnFail: true
  test:
    enable: true
  driftDetection:
    mode: enabled

11.1 Values Layering (Kustomize + Helm)

Patch a HelmRelease’s values: per environment using Kustomize’s strategic merge:

# overlays/production/patch-redis-values.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: redis
spec:
  values:
    replica:
      replicaCount: 3

patches:
  - path: patch-redis-values.yaml
    target:
      kind: HelmRelease
      name: redis

11.2 `valuesFrom` (external values sources)

spec:
  valuesFrom:
    - kind: ConfigMap
      name: redis-values
      valuesKey: values.yaml
    - kind: Secret
      name: redis-secret-values
      valuesKey: values.yaml
      optional: true

11.3 Drift Detection & Remediation

spec:
  driftDetection:
    mode: enabled       # warn | enabled | disabled
    ignore:
      - paths: ["/spec/replicas"]
        target:
          kind: Deployment

driftDetection (Flux 2.12+) detects manual kubectl edit-style changes to Helm-managed resources and can auto-correct them, closing a long-standing GitOps gap where Helm releases silently drifted.

11.4 OCI-based Helm Charts

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: my-oci-charts
spec:
  type: oci
  url: oci://ghcr.io/my-org/charts
  interval: 30m

12. Image Automation

Automatically detect new container image tags and commit the update back to Git.

12.1 ImageRepository (scan a registry)

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  image: ghcr.io/stefanprodan/podinfo
  interval: 5m

12.2 ImagePolicy (select which tag is “latest”)

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: podinfo
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: podinfo
  policy:
    semver:
      range: ">=6.0.0 <7.0.0"
  # alternatives:
  # policy:
  #   alphabetical:
  #     order: asc
  # filterTags:
  #   pattern: '^main-[a-fA-F0-9]+-(?P<ts>[0-9]+)$'
  #   extract: '$ts'

12.3 ImageUpdateAutomation (write the change back to Git)

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 30m
  sourceRef:
    kind: GitRepository
    name: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: "chore: update image {{range .Updated.Images}}{{println .}}{{end}}"
    push:
      branch: main
  update:
    path: "./apps/production"
    strategy: Setters

12.4 Marking Manifests for Automated Updates

image: ghcr.io/stefanprodan/podinfo:6.5.3 # {"$imagepolicy": "flux-system:podinfo"}

The Setters strategy annotates the image line so the automation controller knows exactly which field to update — surgical, comment-based, and diff-friendly.

13. Health Checks, Pruning & Reconciliation

13.1 Pruning (Garbage Collection)

spec:
  prune: true

When prune: true, Flux tracks which resources were applied by a given Kustomization (via inventory stored in the object’s status) and deletes any that are removed from Git on the next reconciliation. This is what makes Flux truly declarative — without it, deleted manifests would leave orphaned resources in the cluster forever.

13.2 Custom Health Checks

spec:
  wait: true
  timeout: 3m
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: podinfo
      namespace: default
    - apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      name: redis
      namespace: flux-system

Flux checks status.conditions[type=Ready] (or Available for Deployments) and blocks the Kustomization from being marked Ready until all listed resources pass — critical when downstream dependsOn chains rely on actual application readiness, not just “kubectl apply succeeded.”

13.3 Reconciliation Triggers

Interval-based: spec.interval polling.
Webhook-based: notification-controller’s Receiver API can trigger immediate reconciliation on Git push, bypassing the poll interval.
Manual: flux reconcile kustomization <name> --with-source

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: github-receiver
  namespace: flux-system
spec:
  type: github
  events:
    - "push"
  secretRef:
    name: receiver-token
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      name: flux-system

13.4 Suspend / Resume

flux suspend kustomization podinfo
flux resume kustomization podinfo

Useful during incident response to freeze a resource from being reconciled while you manually intervene, then hand control back to Git.

14. Notifications & Alerts

14.1 Provider (the destination)

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: gitops-alerts
  secretRef:
    name: slack-url

14.2 Alert (what to send, from where)

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: on-call-alerts
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: '*'
    - kind: HelmRelease
      name: '*'
  exclusionList:
    - ".*upgrade.*has.*started.*"

14.3 Supported Providers

Slack, MS Teams, Discord, Google Chat, generic Webhook, PagerDuty, Opsgenie, Prometheus Alertmanager, GitHub/GitLab commit status, Sentry, and more — all via the same Provider/Alert pattern.

15. Advanced Kustomize Patterns

15.1 `patchesStrategicMerge` vs Inline Patches vs Patch Files

Modern Kustomize favors the unified patches: field with inline YAML or JSON6902:

patches:
  - target:
      kind: Deployment
      labelSelector: "app.kubernetes.io/component=api"
    patch: |
      - op: replace
        path: /spec/replicas
        value: 2

labelSelector / annotationSelector in target: let you patch multiple resources at once without listing each by name — powerful for cross-cutting changes.

15.2 `patches` with `target` wildcard for global changes

patches:
  - patch: |
      - op: add
        path: /metadata/labels/cost-center
        value: "platform"
    target:
      kind: Deployment|StatefulSet|DaemonSet

15.3 Merging Multiple Overlays (mixins) via `resources:` + `components:`

resources:
  - ../../base/podinfo
components:
  - ../../components/network-policies
  - ../../components/pod-security
  - ../../components/monitoring

15.4 Remote Bases (use with caution)

resources:
  - https://github.com/org/repo//path/to/base?ref=v1.2.3

Remote bases work but bypass version pinning discipline unless you strictly pin ?ref=. Flux’s GitRepository + local path: is generally preferred over remote Kustomize bases for auditability and offline builds.

15.5 `configurations` for Custom Resource Behaviour (legacy, mostly superseded)

Older Kustomize versions used configurations: files to teach Kustomize how to handle CRDs for name references, var substitution, etc. Modern Kustomize auto-detects most common CRD conventions; custom transformer configs are now rarely required.

15.6 `buildMetadata` (annotate provenance)

buildMetadata: [originAnnotations, transformerAnnotations, managedByLabel]

Adds annotations like config.kubernetes.io/origin showing which file/base produced a resource — useful for debugging deeply layered overlays.

15.7 `openapi` field for custom schema validation

openapi:
  path: crd-schema.json

Lets Kustomize understand custom CRD merge-key semantics (e.g., which array field acts as a “map” keyed by name), important for correct strategic-merge behavior on CRs.

15.8 Patch Targeting by Multiple Selectors

patches:
  - patch: |-
      - op: add
        path: /spec/template/spec/nodeSelector
        value:
          workload-type: batch
    target:
      kind: Deployment
      annotationSelector: "workload=batch"
      namespace: jobs

15.9 Avoiding Common Kustomize Pitfalls

Don’t duplicate metadata.name across base and patch unless intentionally targeting the same object.
Don’t rely on vars: — it’s deprecated; use replacements:.
Do pin image tags explicitly in overlays via images: rather than editing base manifests per environment.
Do run kustomize build in CI on every PR to catch broken overlays before merge.
Avoid overly deep overlay chains (base → overlay → overlay → overlay) — two levels (base + environment) is usually sufficient; use components: for cross-cutting concerns instead of a third overlay layer.

16. Best Practices Checklist

16.1 Repository & Structure

✅ Separate infrastructure/ (cluster-wide, cert-manager, ingress, CRDs) from apps/ (workloads).
✅ One Kustomization (Flux CRD) per logical unit — don’t cram every app into a single giant Kustomization; smaller units mean smaller blast radius and faster targeted reconciliation.
✅ Keep base/ truly environment-agnostic — no hardcoded environment names, replicas, or resource limits in base.
✅ Use dependsOn for genuine ordering needs (CRDs before CRs, infra before apps) — don’t over-chain dependencies for cosmetic reasons.

16.2 Safety & Reliability

✅ Always set prune: true for true GitOps garbage collection.
✅ Set wait: true + meaningful timeout so failures surface quickly instead of silently leaving broken deployments.
✅ Use healthChecks for critical resources when automatic inference isn’t sufficient (e.g., Jobs, custom CRDs without standard Ready conditions).
✅ Set sane retryInterval (shorter than interval) so transient failures self-heal quickly.
✅ Use flux diff/kustomize build + kubectl diff -k in CI before merging to catch issues pre-merge.

16.3 Security

✅ Never commit plaintext secrets — SOPS, Sealed Secrets, or External Secrets Operator only.
✅ Use serviceAccountName + RBAC impersonation for tenant isolation — don’t let every Kustomization apply with the controller’s default cluster-admin-equivalent identity.
✅ Enable verify (Cosign/GPG) on sources handling sensitive or production workloads.
✅ Scope GitRepository/OCIRepository credentials to read-only deploy keys, never write access, unless used specifically for image automation commits.

16.4 Versioning & Change Management

✅ Pin Helm chart versions explicitly (version: "18.1.x"), never "*" or unset.
✅ Pin image tags via Kustomize images: overrides, not :latest.
✅ Use semantic commit messages; Flux notification Alerts surface these in Slack/Teams for auditability.
✅ Tag/branch strategy: main → staging (fast interval), release tags/branches → production (slower interval, manual promotion via PR/tag).

16.5 Observability

✅ Wire up Alert/Provider to a real channel from day one — silent GitOps failures are worse than no GitOps.
✅ Monitor gotk_reconcile_condition and gotk_suspend_status Prometheus metrics exposed by all toolkit controllers.
✅ Use flux get all -A and flux events regularly, not just when something breaks.

16.6 Kustomize Hygiene

✅ Run kustomize build (not just kubectl apply -k) in CI to catch build-time errors independent of cluster state.
✅ Prefer patches: (unified) over deprecated patchesStrategicMerge/patchesJson6902.
✅ Prefer replacements: over deprecated vars:.
✅ Keep overlay depth to two levels; use components: for cross-cutting concerns.
✅ Use commonLabels consistently — it also updates selectors, so retrofitting labels later is disruptive; decide your labeling scheme early.

17. Troubleshooting & Debugging

17.1 Check Overall Status

flux get all -A
flux get sources git -A
flux get kustomizations -A
flux get helmreleases -A

17.2 Inspect a Specific Kustomization

flux get kustomization podinfo -n flux-system
kubectl describe kustomization podinfo -n flux-system
kubectl get kustomization podinfo -n flux-system -o yaml

Look at status.conditions — common condition types: Ready, Reconciling, Stalled, HealthCheckFailed.

17.3 Force Reconciliation

flux reconcile source git flux-system
flux reconcile kustomization podinfo --with-source

17.4 View Events

flux events --for Kustomization/podinfo -n flux-system
kubectl get events -n flux-system --field-selector involvedObject.name=podinfo

17.5 Debug a Kustomize Build Locally

# Clone/checkout the exact ref Flux is using
git clone <repo> && cd repo && git checkout <ref>
kustomize build ./path/to/kustomization

Compare output against what’s actually in the cluster with kubectl diff -k ./path.

17.6 Common Errors & Fixes

Symptom	Likely Cause	Fix
`Kustomization` stuck `Reconciling`	Dependency not `Ready`, or resource never becoming healthy	Check `dependsOn` target and `flux get kustomization <dep>`
`HealthCheckFailed`	Deployment CrashLoopBackOff / probe failing	`kubectl logs`/`describe` the underlying workload
`build failed: ... accumulating resources`	Broken `resources:` path, typo, or missing file	`kustomize build` locally to reproduce
`field is immutable` on apply	Changed an immutable field (e.g., `selector`, PVC size shrink)	Set `force: true` cautiously, or delete+recreate manually
`context deadline exceeded`	`timeout` too short for slow-starting workloads	Increase `spec.timeout`
Secrets not decrypting	Wrong `decryption.secretRef` or expired SOPS key	Verify secret exists in `flux-system` and matches `.sops.yaml` rules
Image tag not updating	`ImagePolicy` filter doesn’t match new tags, or marker comment missing	Check `ImagePolicy.policy` regex/semver range and `{"$imagepolicy": ...}` marker
`too many open files` / rate limiting from Git provider	Polling interval too aggressive across many sources	Increase `interval`, use webhook `Receiver` instead of tight polling

17.7 Dry-Run and Diff

flux diff kustomization podinfo --path ./clusters/production

flux diff (available in recent Flux CLI versions) renders what would change without applying, similar to terraform plan — indispensable for pre-merge review.

18. CLI Cheat Sheet

# Bootstrap
flux bootstrap github --owner=org --repository=fleet-infra --branch=main --path=clusters/production

# Status
flux check
flux get all -A
flux get kustomizations -A
flux get sources all -A
flux get helmreleases -A
flux get images all -A

# Reconcile
flux reconcile source git flux-system
flux reconcile kustomization <name> --with-source
flux reconcile helmrelease <name>

# Suspend / Resume
flux suspend kustomization <name>
flux resume kustomization <name>

# Create resources imperatively (scaffolding, then export to YAML)
flux create source git podinfo --url=https://github.com/x/podinfo --branch=main --export > source.yaml
flux create kustomization podinfo --source=GitRepository/podinfo --path="./kustomize" --prune=true --export > kustomization.yaml
flux create helmrelease redis --chart=redis --source=HelmRepository/bitnami --export > helmrelease.yaml
flux create tenant tenant-a --with-namespace=tenant-a --export > tenant-a.yaml

# Events & Logs
flux events
flux logs --follow
flux logs --level=error

# Uninstall
flux uninstall --namespace=flux-system

# Kustomize (plain tool)
kustomize build overlays/production
kubectl apply -k overlays/production
kubectl diff -k overlays/production
kustomize edit set image myapp=registry.example.com/myapp:1.4.2
kustomize edit set replicas myapp=5
kustomize edit add resource deployment.yaml
kustomize edit add patch --path patch.yaml --kind Deployment

19. Full Reference Repository Layout

fleet-infra/
├── clusters/
│   ├── staging/
│   │   ├── flux-system/
│   │   │   ├── gotk-components.yaml
│   │   │   ├── gotk-sync.yaml
│   │   │   └── kustomization.yaml
│   │   ├── infrastructure.yaml        # Flux Kustomization -> infrastructure/overlays/staging
│   │   └── apps.yaml                  # Flux Kustomization -> apps/overlays/staging
│   └── production/
│       ├── flux-system/
│       ├── infrastructure.yaml
│       └── apps.yaml
│
├── infrastructure/
│   ├── base/
│   │   ├── cert-manager/
│   │   │   ├── namespace.yaml
│   │   │   ├── helmrelease.yaml
│   │   │   └── kustomization.yaml
│   │   ├── ingress-nginx/
│   │   ├── monitoring/
│   │   │   ├── kube-prometheus-stack/
│   │   │   └── grafana-dashboards/
│   │   └── kustomization.yaml
│   └── overlays/
│       ├── staging/
│       │   ├── kustomization.yaml
│       │   └── patch-resources.yaml
│       └── production/
│           ├── kustomization.yaml
│           └── patch-resources.yaml
│
├── apps/
│   ├── base/
│   │   └── podinfo/
│   │       ├── deployment.yaml
│   │       ├── service.yaml
│   │       ├── hpa.yaml
│   │       └── kustomization.yaml
│   └── overlays/
│       ├── staging/
│       │   ├── kustomization.yaml
│       │   ├── patch-replicas.yaml
│       │   └── podinfo-kustomization.yaml   # Flux Kustomization CRD
│       └── production/
│           ├── kustomization.yaml
│           ├── patch-replicas.yaml
│           ├── patch-resources.yaml
│           └── podinfo-kustomization.yaml
│
├── components/
│   ├── network-policies/
│   │   ├── kustomization.yaml         # kind: Component
│   │   └── deny-all-ingress.yaml
│   └── pod-disruption-budget/
│       ├── kustomization.yaml
│       └── pdb.yaml
│
├── tenants/
│   ├── tenant-a/
│   │   ├── namespace.yaml
│   │   ├── rbac.yaml
│   │   ├── source.yaml
│   │   └── kustomization.yaml
│   └── tenant-b/
│
└── .sops.yaml

Appendix: Quick Reference — Flux `Kustomization` vs Kustomize `kustomization.yaml`

	Flux `Kustomization` (CRD)	Kustomize `kustomization.yaml`
API	`kustomize.toolkit.fluxcd.io/v1`	`kustomize.config.k8s.io/v1beta1`
Purpose	Tells a controller what to reconcile, from where, how often	Defines how to build a set of manifests (patches, generators)
Lives in cluster?	Yes (a live Kubernetes object)	No — it’s a file consumed at build time
Contains patches?	Can add last-mile `patches:` post-build	Primary purpose is patches/generators/transformers
Health checks?	Yes	No (build-tool only, no runtime awareness)
Pruning?	Yes (`prune: true`)	No (Kustomize itself has no concept of previous state)

End of guide. For the Turkish version, see flux2-kustomize-rehberi-tr.md.

Table of Contents