Hasan OzganHasan Ozgan

Building Kubernetes Operators with Java Operator SDK (JOSDK)

Building Kubernetes operators with the Java Operator SDK: reconcilers, update controls, and the reconciliation loop.

🌱 Seedling·created: ·category:Java
tags:kubernetesjavaoperatorjosdkjdk

A Deep Dive into Reconciler<T>, UpdateControl, and the Reconciliation Loop

Java Operator SDK version referenced: v5.3.x (server-side apply by default, event filtering for self-writes) JDK version referenced: JDK 25 (LTS, released September 16, 2025)

Table of Contents

  1. What Is a Kubernetes Operator, Really?
  2. Why JDK 25 for Operator Development
  3. Project Setup
  4. The Anatomy of JOSDK
  5. The Reconciler Interface, Fully Explained
  6. How the Reconciliation Loop Actually Works
  7. UpdateControl and ErrorStatusUpdateControl in Depth
  8. Event Sources, Informers, and Caching
  9. Dependent Resources Framework
  10. Finalizers and Cleanup
  11. Error Handling & Retry Semantics
  12. Best Practices Checklist
  13. Full Working Example
  14. Testing Your Operator
  15. Observability, Leader Election, Deployment
  16. References

1. What Is a Kubernetes Operator, Really?

A Kubernetes Operator is a piece of software that extends the Kubernetes control plane with custom, domain-specific automation. It follows the same pattern that Kubernetes itself uses internally: a controller watches the state of the cluster (usually via a Custom Resource Definition, or CRD) and continuously tries to drive the actual state toward the desired state declared by the user.

The core loop underlying every controller - including the built-in ones (Deployment controller, ReplicaSet controller, etc.) - is called the control loop or reconciliation loop:

flowchart LR
    A["observe"] --> B["compare (desired vs actual)"] --> C["act"] --> D["repeat"]
    D -.-> A

This is fundamentally different from imperative scripting. You never write “create this Pod now.” Instead, you write “given this custom resource, this is what the world should look like,” and the reconciler is invoked repeatedly (on every relevant change, on a resync, after retries, etc.) until actual state converges with desired state - and it keeps being invoked afterward to correct drift.

Key properties every Operator author must internalize:

  • Level-based, not edge-based: The reconciler doesn’t process “events” like “Pod X was created.” It processes “the current state of the world,” recomputed from scratch every time. This means your logic must be idempotent and must not rely on the history of what happened, only on the current observed state.
  • Eventually consistent: A single reconciliation might not be able to reach the desired state immediately (e.g., you’re waiting for a Deployment to become Ready). That’s fine - you exit early and let a future event or reschedule trigger another attempt.
  • At-least-once execution: Your reconcile method will be called more than once for the same logical state, sometimes concurrently in effect (though JOSDK serializes reconciliations per resource by default). Never assume “this is the only time this runs.”

2. Why JDK 25 for Operator Development

JDK 25 (GA September 16, 2025) is the current LTS release (following JDK 21), guaranteed long-term support from Oracle and other vendors. For an Operator - a long-running, resource-constrained, cluster-native workload - several JDK 25 features are directly relevant:

  • Structured Concurrency (JEP 505, fifth preview): Operators frequently need to fan out to multiple Kubernetes API calls or external systems (e.g., check on 3 dependent resources) and join the results before deciding on the next state transition. Structured concurrency gives you a disciplined way to spawn and await child tasks as a unit, so failures/cancellations propagate correctly - this maps very naturally onto reconciliation logic that “waits for N sub-resources to be ready.”
  • Scoped Values (JEP 506, finalized): A safer, immutable replacement for ThreadLocal, useful for propagating per-reconciliation context (e.g., a request-scoped logger/trace ID) across the call stack without leaking state between concurrent reconciliations of different resources.
  • Virtual Threads (from JDK 21, now mature in 25): JOSDK’s executor model benefits from lightweight virtual threads when your reconcilers perform blocking I/O (Kubernetes API calls, HTTP calls to external systems). This lets you scale to many concurrently reconciling resources without exhausting a small platform-thread pool.
  • Pattern Matching for Switch with primitives (JEP 507) and Compact Source Files (JEP 512): mostly ergonomic, but they reduce boilerplate in the small helper classes/records you’ll write for status objects and spec/status diffing.
  • Records and sealed interfaces (stable since JDK 17, still central in 25): Ideal for representing your CRD’s Spec/Status POJOs and for modeling reconciliation outcomes as sealed hierarchies (e.g., sealed interface ReconcileDecision permits Requeue, Done, Waiting).

None of this replaces good architecture - but writing an Operator in JDK 25 means you can lean on structured concurrency and virtual threads to keep reconciliation logic readable and non-blocking, instead of hand-rolling CompletableFuture chains.

3. Project Setup

Minimal Maven setup (Gradle equivalent works the same way):

<properties>
  <maven.compiler.release>25</maven.compiler.release>
  <josdk.version>5.3.0</josdk.version> <!-- check for the latest patch -->
</properties>

<dependencies>
  <dependency>
    <groupId>io.javaoperatorsdk</groupId>
    <artifactId>operator-framework-core</artifactId>
    <version>${josdk.version}</version>
  </dependency>

  <!-- Optional but common: bundles fabric8 kubernetes-client + core -->
  <dependency>
    <groupId>io.javaoperatorsdk</groupId>
    <artifactId>operator-framework</artifactId>
    <version>${josdk.version}</version>
  </dependency>

  <!-- Testing -->
  <dependency>
    <groupId>io.javaoperatorsdk</groupId>
    <artifactId>operator-framework-junit-5</artifactId>
    <version>${josdk.version}</version>
    <scope>test</scope>
  </dependency>
</dependencies>

JOSDK is built on the fabric8 Kubernetes client, so you get typed access to core Kubernetes resources (Deployment, ConfigMap, Service, …) and to your own CRDs once you generate/write the model classes.

You can bootstrap a Custom Resource as a plain Java class:

@Group("example.com")
@Version("v1")
@ShortNames("wp")
public class WebPage extends CustomResource<WebPageSpec, WebPageStatus> implements Namespaced {}

public record WebPageSpec(String html, String cssStyleUrl, String javaScriptUrl) {}

public class WebPageStatus {
    private String url;
    private String observedGeneration; // convention: track processed generation
    // getters/setters
}

If you’re using Quarkus, the Quarkus Operator SDK extension wraps JOSDK with CDI injection, native-image support, and automatic CRD generation - highly recommended for production operators, but the reconciliation model underneath is identical to what’s described here.

4. The Anatomy of JOSDK

JOSDK’s runtime consists of a small number of cooperating components:

ComponentResponsibility
OperatorTop-level bootstrap object; registers reconcilers, starts informers, manages lifecycle.
ReconcilerYour business logic - the only interface you must implement.
Controller (internal)Wraps a Reconciler; owns the event processing queue for that resource type.
EventSourceWatches a resource type (primary or secondary) and emits events into the processing queue.
Informer / CacheLocal, in-memory, eventually-consistent mirror of watched resources, backed by the Kubernetes Watch API.
Workflow / DependentResourceOptional declarative layer for managing secondary resources (ConfigMaps, Deployments, etc.) that your primary resource “owns.”
RetryManager / RateLimiterGoverns backoff and retry attempts when reconciliation throws.

Everything downstream of “an event arrived for resource X” flows into a per-resource, serialized invocation of your reconciler - JOSDK guarantees that the same custom resource instance is never reconciled concurrently by two threads, which removes a whole class of race conditions from your code.

5. The Reconciler<T> Interface, Fully Explained

At its core:

public interface Reconciler<P extends HasMetadata> {
    UpdateControl<P> reconcile(P resource, Context<P> context) throws Exception;
}

That’s it - one method. Everything about your Operator’s behavior is expressed through:

  1. What you read from resource (the primary resource, as currently known - spec + status + metadata).
  2. What you do (call the Kubernetes API to create/update dependent resources, call external systems, etc.).
  3. What you return - an UpdateControl<P> that tells JOSDK what to persist and how to schedule the next reconciliation.

A minimal, idiomatic reconciler:

@ControllerConfiguration
public class WebPageReconciler implements Reconciler<WebPage> {

    private static final Logger log = LoggerFactory.getLogger(WebPageReconciler.class);

    @Override
    public UpdateControl<WebPage> reconcile(WebPage webPage, Context<WebPage> context) {
        log.info("Reconciling WebPage {}/{}",
                webPage.getMetadata().getNamespace(), webPage.getMetadata().getName());

        // 1. Compute desired state of dependent resources from webPage.getSpec()
        ConfigMap desiredConfigMap = buildConfigMap(webPage);

        // 2. Apply it declaratively (server-side apply)
        context.resourceOperations().serverSideApply(desiredConfigMap);

        // 3. Fetch fresh secondary state from context's cache (not a live GET)
        Deployment deployment = context.getSecondaryResource(Deployment.class)
                .orElseGet(() -> createDeployment(webPage));
        context.resourceOperations().serverSideApply(deployment);

        // 4. Decide whether we're "done" or still converging
        boolean ready = deployment.getStatus() != null
                && Objects.equals(deployment.getStatus().getReadyReplicas(),
                                   deployment.getSpec().getReplicas());

        // 5. Update status (never spec!) and return
        webPage.getStatus().setReady(ready);
        webPage.getStatus().setObservedGeneration(webPage.getMetadata().getGeneration());

        return UpdateControl.patchStatus(webPage);
    }
}

Key contract details of reconcile()

  • resource is a snapshot, not a live handle. It’s the version JOSDK had cached at dispatch time. Any mutation you do on it (typically to .getStatus()) is later turned into an API request by UpdateControl. You should never call client.resources(...).update(...) directly on the primary resource inside reconcile unless you have a specific reason to bypass UpdateControl - doing so breaks JOSDK’s optimistic concurrency and event-suppression logic.
  • Context<P> is your gateway to everything else: secondary resource caches (context.getSecondaryResource(...)), the Kubernetes client, controller configuration, retry info (context.isLastAttempt()), and - since v5 - context.resourceOperations() for cache-aware server-side apply calls.
  • Exceptions propagate to the retry mechanism. Throwing is a legitimate, encouraged way to say “this failed, please retry me” - you don’t need to swallow exceptions and return some sentinel UpdateControl.
  • The return value is mandatory and meaningful. There is no “no-op” by omission; you explicitly choose patchStatus, patchResource, patchResourceAndStatus, or noUpdate.

6. How the Reconciliation Loop Actually Works

This is the part most newcomers get conceptually wrong, so let’s go slowly.

6.1 Startup phase

When the Operator process starts:

  1. JOSDK starts an Informer for each registered primary resource type (and for each configured secondary/dependent resource type). Informers perform an initial LIST to populate the local cache, then open a WATCH to stream subsequent changes.
  2. As a best practice - and this is JOSDK’s default behavior - every existing resource is reconciled once at startup, because the desired state may have drifted while the Operator process was down (e.g., someone deleted a dependent ConfigMap manually, or a node crashed mid-update).
  3. Once the initial full reconciliation completes for a resource, subsequent reconciliations are event-driven.

6.2 What actually triggers a reconciliation

A reconciliation of a given custom resource instance is enqueued when any of the following happens:

  • The primary resource’s .spec changes - technically, JOSDK compares .metadata.generation, which the Kubernetes API server increments automatically whenever .spec changes (not on status or metadata-only changes). This is why the convention is to track observedGeneration in your .status - it lets your own logic (and external observers) know “has this spec change actually been reconciled yet?”
  • A secondary/dependent resource you’re watching changes (e.g., a Deployment you own transitions from NotReady to Ready - JOSDK’s event source for that Deployment enqueues a reconciliation of the owning WebPage).
  • A manual reschedule was requested via UpdateControl.rescheduleAfter(...) or the reschedule() method (v5.3+) from a previous reconciliation.
  • A retry is scheduled after a previous reconcile() call threw an exception.
  • The resource is marked for deletion and has a finalizer, triggering the cleanup() path (if you implement Cleaner<P>).

Critically: JOSDK does not reconcile on every single watch event by default. If you only change .metadata.labels on the primary resource (not .spec), no reconciliation is triggered, because .metadata.generation didn’t change. This is a deliberate optimization - most operators only care about spec changes - but you can override it with @ControllerConfiguration(triggerReconcilerOnAllEvents = true) if your logic genuinely needs to react to every metadata change too.

6.3 The dispatch and queueing model

Internally, each Controller maintains an event processing structure conceptually similar to Kubernetes’ own client-go workqueue:

  1. An event (from any event source) is translated into a ResourceID (namespace + name) to reconcile.
  2. That ID is placed onto a queue. If the same ID is already queued or currently being processed, JOSDK coalesces - you don’t get N redundant reconciliations for N rapid-fire events on the same resource; you get one reconciliation that observes the latest cached state.
  3. A worker thread pulls the ResourceID, fetches the latest version of that resource from the local informer cache (not a live API call - this is why caching correctness matters so much), and invokes your reconcile() method.
  4. Reconciliations for different resources run concurrently (bounded by a configurable thread pool - this is where JDK 25 virtual threads help a lot if your reconcile logic blocks on I/O). Reconciliations for the same resource are strictly serialized.

6.4 The “own write” feedback loop and event filtering

A subtlety that trips up many operator authors: when your reconciler calls context.resourceOperations().serverSideApply(...) on a dependent resource, that write itself produces a Kubernetes watch event. Historically, this could re-trigger a reconciliation that essentially observes “the change I just made,” wasting a cycle (or worse, causing hot loops if not handled carefully).

As of JOSDK v5.3, the framework automatically filters out events that were produced by the Operator’s own writes performed through context.resourceOperations() - so UpdateControl and ErrorStatusUpdateControl no longer trigger a redundant reconciliation from their own status patch. If you bypass the framework’s write path (e.g., raw fabric8 client calls), you lose this protection and must reason about it yourself.

6.5 Post-reconciliation: applying UpdateControl

After reconcile() returns, JOSDK:

  1. Inspects the UpdateControl to determine what to persist (status patch, resource patch, both, or nothing).
  2. Issues the corresponding Kubernetes API request(s), typically as a server-side apply PATCH.
  3. Updates its local cache optimistically with the patched object where possible, so a fast subsequent reconciliation sees fresh data without waiting for the watch round-trip.
  4. If a rescheduleAfter or reschedule() was specified, arms a timer to re-enqueue this resource after the given delay, regardless of any watch events.
  5. If the reconciler threw, the RetryManager decides whether/when to re-enqueue based on the configured retry policy (default: linear/exponential backoff, capped attempts, configurable).

7. UpdateControl and ErrorStatusUpdateControl in Depth

UpdateControl<P> is a small, immutable value object (a great fit for JDK 25 records conceptually, though JOSDK’s implementation predates broad record adoption) that encodes exactly one outcome per reconciliation:

// No changes to persist at all
UpdateControl.noUpdate();

// Persist ONLY the status subresource
UpdateControl.patchStatus(resource);

// Persist ONLY the main resource (spec/metadata), NOT status
UpdateControl.patchResource(resource);

// Persist BOTH - as two separate API requests, resource first, then status
UpdateControl.patchResourceAndStatus(resource);

Every one of these can be chained with scheduling directives:

UpdateControl.patchStatus(webPage)
    .rescheduleAfter(Duration.ofSeconds(30)); // force a re-reconciliation later, regardless of events

7.1 Why patchStatus and not a full update

Kubernetes exposes the status subresource specifically so that controllers can update observed/derived state (.status) without needing write access to - or risking accidental mutation of - the desired state (.spec), and without triggering .metadata.generation increments (status updates never bump generation). UpdateControl.patchStatus(resource):

  • Sends a PATCH scoped to /status, using server-side apply by default since v5.
  • Requires that your CRD actually declares a status subresource (subresources: { status: {} } in the CRD, or the equivalent annotation-driven generation if you use @Kep codegen / Quarkus).
  • Should be the overwhelming majority of what your Operator does on each reconciliation. If you find yourself calling patchResource frequently, ask whether you’re accidentally treating spec as mutable state that the controller should own (a common anti-pattern - spec is the user’s intent, not the controller’s scratch pad).

7.2 Server-side apply and why it changed your reconciler code

Since JOSDK v5, server-side apply (SSA) is the default update strategy, replacing the older “read-modify-write with resourceVersion-based optimistic locking” approach. SSA changes how you should construct the object you pass to patchStatus:

  • With SSA, you should submit only the fields you want to own/assert, not a full copy of the object with everything else re-stated. If you mutate the cached resource object directly and pass the whole thing back, you risk overwriting fields owned by other actors (e.g., the Kubernetes API server itself, or another controller) that legitimately write to different parts of the same status object.
  • The safe pattern many teams adopt is to construct a minimal patch object, carrying only metadata.name/namespace/resourceVersion and the status fields you’re actually setting:
WebPage statusPatch = new WebPage();
statusPatch.setMetadata(new ObjectMetaBuilder()
        .withName(webPage.getMetadata().getName())
        .withNamespace(webPage.getMetadata().getNamespace())
        .build());
statusPatch.setStatus(computeStatus(webPage));
return UpdateControl.patchStatus(statusPatch);
  • In practice, for simple operators that are the sole owner of the entire .status object, mutating the fetched resource directly and returning it is fine and is what most tutorials (including JOSDK’s own docs) show. Reach for the minimal-patch pattern once multiple writers might touch different subfields of status, or once you’re doing SSA on dependent resources with shared ownership.

7.3 The “stale cache after patchStatus” pitfall

A well-documented gotcha: after UpdateControl.patchStatus(...) executes, the freshly patched resource is not guaranteed to be immediately visible to the very next reconciliation, because the local informer cache updates asynchronously as the watch event round-trips back from the API server. Two mitigations exist:

  1. In-memory caching in your own reconciler of the last-known-good status, so your logic doesn’t regress if it re-reads a slightly stale cached object.
  2. From JOSDK v5.1 onward, a utility is provided specifically to guarantee the updated status is available for the next reconciliation without you having to hand-roll a cache - check UpdateControl/Context for context.resourceOperations()-based helpers when you need this guarantee (e.g., when writing a piece of status data that a subsequent reconciliation step depends on reading back correctly).

7.4 ErrorStatusUpdateControl<P>

When you want to record an error condition on .status even though the reconciliation is failing (and will be retried), override:

@Override
public ErrorStatusUpdateControl<WebPage> updateErrorStatus(
        WebPage resource, Context<WebPage> context, Exception e) {
    resource.getStatus().setErrorMessage(e.getMessage());
    return ErrorStatusUpdateControl.patchStatus(resource);
}

This lets users of kubectl describe see why the operator is stuck, instead of only seeing retries in the Operator’s logs. You can also opt a specific error status update out of the retry counter with .withNoRetry(), though JOSDK’s docs explicitly discourage disabling retries except for very narrow, deliberate cases (e.g., a permanently invalid spec that will never succeed without user intervention - in which case you’d rather stop hammering the API than retry forever).

8. Event Sources, Informers, and Caching

Event Sources are the mechanism by which anything - primary resources, secondary/dependent Kubernetes resources, or even non-Kubernetes systems (a message queue, a webhook, a polling job against a REST API) - can trigger a reconciliation.

  • Primary Event Source: automatically registered for your Reconciler’s resource type; backed by an Informer.
  • Secondary Event Sources: you register these for resources your reconciler depends on or manages - most commonly via the EventSourceInitializer interface or, more simply, by using the Dependent Resource abstraction (section 9), which wires this up for you automatically.
  • Custom/External Event Sources: for triggering reconciliation from outside Kubernetes (e.g., an external system’s webhook), you implement your own EventSource and register it manually.

Why prefer event sources over polling/timers: Polling wastes API server load and adds latency (you’re bound to your poll interval). An Informer-backed event source reacts within milliseconds of an actual change and lets Kubernetes’ Watch API do the heavy lifting. JOSDK’s own best-practices guidance is explicit: use rescheduleAfter sparingly, and only when there is genuinely no way to observe the awaited condition as a Kubernetes-native watchable event (e.g., waiting on wall-clock time, like a certificate expiring in 30 days).

Caching correctness matters because every reconciliation reads from the local cache, not a live GET. This is a deliberate design trade-off for scalability (you don’t hammer the API server on every reconciliation), but it means:

  • Never treat the object your reconciler receives as guaranteed 100% fresh at the instant of invocation - treat it as “fresh enough, as of the last watch event or LIST.”
  • Idempotent, convergent logic (section 12) is what makes this safe: if you act on slightly stale data, the next reconciliation (triggered by your own write, by drift detection, or by a reschedule) corrects it.

9. Dependent Resources Framework

For the extremely common case of “my custom resource owns a ConfigMap/Deployment/Service that should mirror computed state,” JOSDK provides a declarative Dependent Resource abstraction so you don’t hand-write the create-or-update-and-watch boilerplate for every secondary resource:

public class ConfigMapDependentResource
        extends CRUDKubernetesDependentResource<ConfigMap, WebPage> {

    @Override
    protected ConfigMap desired(WebPage webPage, Context<WebPage> context) {
        return new ConfigMapBuilder()
                .withNewMetadata()
                    .withName(webPage.getMetadata().getName())
                    .withNamespace(webPage.getMetadata().getNamespace())
                .endMetadata()
                .addToData("index.html", webPage.getSpec().html())
                .build();
    }
}

@ControllerConfiguration(dependents = {
        @Dependent(type = ConfigMapDependentResource.class),
        @Dependent(type = DeploymentDependentResource.class)
})
public class WebPageReconciler implements Reconciler<WebPage> {
    // reconcile() can now call context.getSecondaryResource(ConfigMap.class)
    // and the framework already reconciled it for you *before* your reconcile()
    // body runs (workflow-based dependents are reconciled as a pre-step).
}

This gives you:

  • Automatic secondary event source registration and caching for that resource type.
  • Automatic create/update (“apply”) logic each reconciliation, diffing desired vs actual.
  • Support for conditions (ReconcilePrecondition, ReadyCondition, ActivationCondition) to express “only manage this dependent if X” or “the parent isn’t Ready until this dependent reports Ready.”
  • As of v5.2, an Expectations pattern is built in (io.javaoperatorsdk.operator.processing.expectation) to avoid a classic race: acting on stale cached state before your own recent write has round-tripped back through the watch.

Whether to use full Dependent Resources or write imperative logic directly in reconcile() is a judgment call: Dependent Resources shine when you have several secondary resources with a fairly standard CRUD-and-watch lifecycle; plain imperative code is often clearer for one-off or highly conditional logic.

10. Finalizers and Cleanup

If your Operator needs to perform external cleanup when a custom resource is deleted (e.g., deprovisioning a cloud resource, revoking a certificate, removing an entry from an external system) - anything that a Kubernetes garbage-collection-based owner reference can’t handle for you - implement Cleaner<P>:

public class WebPageReconciler implements Reconciler<WebPage>, Cleaner<WebPage> {

    @Override
    public UpdateControl<WebPage> reconcile(WebPage resource, Context<WebPage> context) { ... }

    @Override
    public DeleteControl cleanup(WebPage resource, Context<WebPage> context) {
        externalSystem.deprovision(resource.getStatus().getExternalId());
        return DeleteControl.defaultDelete(); // removes the finalizer, allowing actual deletion
    }
}
  • Implementing Cleaner<P> causes JOSDK to automatically add a finalizer to your custom resource on first reconciliation (auto-generated name unless you specify one). This prevents Kubernetes from actually deleting the object (setting .metadata.deletionTimestamp is not the same as deletion) until your finalizer is removed.
  • If cleanup can’t complete yet (e.g., you’re waiting on an async deprovisioning job), return DeleteControl.noFinalizerRemoval() and reschedule - the delete-marked resource will be reconciled again.
  • If you don’t need external cleanup - e.g., all your secondary resources are Kubernetes objects with proper ownerReferences, and Kubernetes’ built-in garbage collector will delete them automatically - do not implement Cleaner<P>. Adding an unnecessary finalizer just adds operational risk (a stuck finalizer on a broken Operator instance can block deletion entirely).

11. Error Handling & Retry Semantics

  • Any exception thrown from reconcile() is caught by JOSDK, logged, and handed to the configured RetryManager, which by default uses exponential backoff with a maximum attempt count.
  • context.isLastAttempt() lets you special-case the final retry - e.g., write a terminal error status instead of a transient one:
@Override
public UpdateControl<MyResource> reconcile(MyResource resource, Context<MyResource> context) {
    if (context.isLastAttempt()) {
        resource.getStatus().setErrorMessage("Failed after all retry attempts");
        return UpdateControl.patchStatus(resource);
    }
    // normal logic
}
  • Retry exhaustion does not mean “give up forever.” Once the retry budget for a given failure streak is exhausted, JOSDK stops automatic retries, but a new watch event (spec change, dependent resource change, or manual kubectl annotate/nudge) will restart reconciliation and reset the retry counter on the next success.
  • Deactivating retries entirely is strongly discouraged by the framework’s own guidance - reserve .withNoRetry() for narrow, deliberate cases.
  • Distinguish transient errors (network blips, API server throttling - let retries handle it, maybe with backoff tuning) from permanent errors (invalid, unrecoverable spec - surface clearly in status, possibly via a Kubernetes Event, and stop pretending retries will fix it).

12. Best Practices Checklist

  1. Idempotency above all. The same observed state must always produce the same outcome. Never rely on “this is the second time I’ve seen this” - you don’t get to see call counts in the same way; you only see current state.
  2. Reconcile everything you manage, every time. Don’t try to cleverly special-case “only touch what changed.” Recompute desired state for all dependents from spec on every invocation; let JOSDK’s diffing (or your own equality checks) decide whether an actual API write is needed. Partial reconciliation logic is a common source of drift bugs.
  3. Prefer event sources over rescheduleAfter/polling. Reserve rescheduling for genuinely time-based conditions (TTLs, certificate expiry, periodic health checks against systems you can’t watch).
  4. Never block indefinitely in reconcile(). If you’re waiting on an async condition (a Pod becoming Ready, an external provisioning job finishing), exit early with UpdateControl.noUpdate() (relying on the relevant event source to re-trigger you) or a bounded rescheduleAfter. Don’t spin-wait inside the method - you’ll starve the shared reconciliation thread pool. This is exactly where JDK 25’s structured concurrency and virtual threads help if you must perform bounded async waits.
  5. Track observedGeneration in status. It’s the idiomatic way for both your own logic and external tooling (dashboards, kubectl wait, GitOps tools) to know whether the latest spec has actually been processed.
  6. Use the status subresource correctly. spec is user intent; status is controller-observed/derived truth. Never write meaningful “control” data into spec from your reconciler.
  7. Design for server-side apply. Send minimal, intentional patches for fields you own; don’t round-trip and blindly rewrite fields you don’t manage, especially for shared objects.
  8. Automatic retries are your friend - leave them on, and tune backoff/limits rather than disabling them.
  9. Use finalizers only when you truly need external cleanup. Owner references + Kubernetes GC handle in-cluster resource cleanup for free.
  10. Reconcile everything on startup. This is JOSDK’s default; don’t try to “optimize” it away - the whole point is correcting drift accumulated while the Operator was down.
  11. Keep reconcilers testable and side-effect-isolated. Push external-system calls behind interfaces you can mock; test the “given this spec+status+secondary state, produce this UpdateControl” logic in isolation.
  12. Log with resource identity (namespace/name/UID) on every log line inside reconcile - you will need to correlate logs across many concurrently reconciling resources.
  13. Emit Kubernetes Events for user-facing, human-readable state transitions (not just status fields) - this is what shows up in kubectl describe and is the idiomatic UX for operators.
  14. Version your CRD and plan conversion webhooks early if you expect the schema to evolve - retrofitting v1alpha1 → v1 conversions onto a live fleet is far harder than designing for it up front.

13. Full Working Example

@Group("example.com")
@Version("v1")
public class WebPage extends CustomResource<WebPageSpec, WebPageStatus> implements Namespaced {}

public record WebPageSpec(String html) {}

public class WebPageStatus {
    private boolean ready;
    private Long observedGeneration;
    private String errorMessage;
    // getters & setters omitted for brevity
}

@ControllerConfiguration
public class WebPageReconciler implements Reconciler<WebPage>, Cleaner<WebPage> {

    private static final Logger log = LoggerFactory.getLogger(WebPageReconciler.class);

    @Override
    public UpdateControl<WebPage> reconcile(WebPage webPage, Context<WebPage> context) {
        var name = webPage.getMetadata().getName();
        var ns = webPage.getMetadata().getNamespace();
        log.info("Reconciling {}/{}", ns, name);

        ConfigMap desired = new ConfigMapBuilder()
                .withNewMetadata().withName(name).withNamespace(ns).endMetadata()
                .addToData("index.html", webPage.getSpec().html())
                .build();

        context.resourceOperations().serverSideApply(desired);

        webPage.getStatus().setReady(true);
        webPage.getStatus().setObservedGeneration(webPage.getMetadata().getGeneration());
        webPage.getStatus().setErrorMessage(null);

        return UpdateControl.patchStatus(webPage);
    }

    @Override
    public ErrorStatusUpdateControl<WebPage> updateErrorStatus(
            WebPage resource, Context<WebPage> context, Exception e) {
        resource.getStatus().setErrorMessage(e.getMessage());
        return ErrorStatusUpdateControl.patchStatus(resource);
    }

    @Override
    public DeleteControl cleanup(WebPage resource, Context<WebPage> context) {
        log.info("Cleaning up {}/{}", resource.getMetadata().getNamespace(),
                resource.getMetadata().getName());
        return DeleteControl.defaultDelete();
    }
}

public class Main {
    public static void main(String[] args) {
        Operator operator = new Operator();
        operator.register(new WebPageReconciler());
        operator.start();
    }
}

14. Testing Your Operator

JOSDK ships operator-framework-junit-5, providing LocalizedOperatorExtension / OperatorExtension, which spins up your reconciler against a real (or envtest-style) Kubernetes API server for integration tests:

@RegisterExtension
static LocalizedOperatorExtension operator = LocalizedOperatorExtension.builder()
        .withReconciler(new WebPageReconciler())
        .build();

@Test
void createsConfigMapFromWebPage() {
    WebPage wp = new WebPage();
    wp.setMetadata(new ObjectMetaBuilder().withName("test").build());
    wp.setSpec(new WebPageSpec("<h1>Hi</h1>"));

    operator.create(wp);

    await().untilAsserted(() -> {
        WebPage updated = operator.get(WebPage.class, "test");
        assertThat(updated.getStatus().isReady()).isTrue();
    });
}

Also unit-test the pure decision logic of reconcile() separately by extracting “compute desired state” and “compute status from observed state” into plain functions you can test without any Kubernetes API at all.

15. Observability, Leader Election, Deployment

  • Leader election: for high-availability deployments (multiple Operator replicas), JOSDK supports leader election so only one instance actively reconciles at a time, avoiding duplicate work and conflicting writes; configure via LeaderElectionConfiguration.
  • Metrics: JOSDK exposes Micrometer-based metrics (reconciliation counts, durations, queue sizes) - wire these into Prometheus for dashboards/alerts on reconciliation error rates and latency.
  • RBAC: your Operator’s ServiceAccount needs RBAC rules for every resource type it watches/reads/writes - both the primary CRD and every dependent type (ConfigMaps, Deployments, etc.), plus status and finalizers subresource permissions where relevant.
  • Container image: since you’re on JDK 25, consider GraalVM native-image builds (well-supported via Quarkus Operator SDK) for fast-starting, low-memory-footprint Operator pods - a meaningful win for cluster resource budgets at scale.

16. References

This guide reflects the state of JOSDK and JDK as of early-to-mid 2026. Always check the official docs for the exact version you depend on, since the framework evolves quickly (e.g., server-side apply defaults and event-filtering behavior were both introduced in relatively recent major/minor versions).

This note is part of the Digital Garden — a collection of connected, evolving thoughts.