Tuluat: Building a Kubernetes AI Operator in Java

How we built Tuluat from scratch - a production-grade AI orchestration platform that manages LLM providers, multi-agent workflows, RAG pipelines, human-in-the-loop approvals, and multi-turn conversation memory entirely as Kubernetes Custom Resources. This is not a marketing story; it’s a step-by-step development log: what I did at each step, why, and which traps I hit and how I escaped them.

Why a Kubernetes Operator?

Hardcoding LLM configuration or wiring agents together with new in application code hits four walls the moment you scale:

Config distribution - a new model/agent means a redeploy.
Isolation - who uses which agent with which budget is anyone’s guess.
Observability - kubectl get shows you nothing.
Lifecycle - who deleted what, and why did it fail? No audit trail.

Kubernetes CRDs solve all four for free: GitOps, RBAC, audit trails, and a unified control plane. The core insight:

CRDs are the right abstraction layer for AI infrastructure. Your agents become first-class Kubernetes citizens, just like Deployments. You kubectl apply -f agent.yaml, and the operator reconciles the desired state.

Why Java?

The Kubernetes ecosystem is comfortable with Go, and for very good reasons - I use Go myself. The question I wanted to explore is different:

Can modern Java be a compelling runtime for AI infrastructure and Kubernetes-native systems?

The Java of today is not the Java many remember from a decade ago:

Virtual threads - thousands of concurrent LLM/tool calls without thread-pool anxiety.
GraalVM - native compilation for a low memory footprint and fast startup.
CRaC - checkpoint/restore opens another door toward near-zero cold starts.
JVM ecosystem - Spring AI, Embabel, Temporal are ready to use.

Projects like OpenJDK Babylon make this direction even more interesting. This is a hypothesis, not a conclusion - the real test comes when the execution/worker layer gets demanding (concurrency, resource usage, startup, isolation, scalability). This project is how I put that hypothesis under pressure.

The Architecture at a Glance

flowchart TD
    subgraph K8S["Kubernetes Control Plane"]
        CR["LlmProvider · AiAgent · McpServer<br/>AiWorkflow · WorkflowSession"]
    end

    subgraph Operator["tuluat-operator"]
        REC["AiAgentReconciler · LlmProviderReconciler · McpServerReconciler<br/>AiWorkflowReconciler · WorkflowSessionReconciler"]
    end

    subgraph Engine["tuluat-engine"]
        AES["AgentExecutionService"]
        GSM["GraphStateMachineEngine"]
        EMB["Embabel GOAP → TuluatGoalAgent"]
        RAG["RagService"]
        GRD["GuardrailPipeline"]
    end

    K8S -- "JOSDK reconciler" --> Operator
    Operator --> Engine
    AES -- "ModelGateway → Spring AI" --> SpringAI["Spring AI ChatModel"]
    GSM -- "durable execution" --> Temporal["Temporal"]
    RAG -- "pgvector + MinIO" --> Store["Vector + Object Store"]

Tuluat Architecture

Tuluat Dashboard

The project is split into six Maven modules, each with a single responsibility and a one-way dependency direction (enforced at compile time by ArchUnit):

Module	Responsibility
`tuluat-crd-domain`	Java records for all CRD specs/statuses - the single source of truth for schema
`tuluat-guardrails`	Pre/post execution filter pipeline (PII, injection, output validation)
`tuluat-protocols`	MCP client registry, A2A adapter
`tuluat-engine`	Agent execution, workflow state machine, RAG, model gateway, Embabel, Temporal
`tuluat-operator`	JOSDK reconcilers - the Kubernetes control loop
`tuluat-app`	Spring Boot entry point, REST controllers, WebSocket events

The dependency rule is strict: operator may import from engine, never the reverse; crd-domain imports nothing. We enforce this with ArchUnit, not discipline.

Step 0 - Dependencies and Versions

The heart of pom.xml:

Component	Version	Role
Java	25	Virtual threads, records, pattern matching
Spring Boot	4.1.0	Application framework
Spring AI	2.0.0	`ChatModel` abstraction
JOSDK (java-operator-sdk)	5.1.0	Reconciler framework
Fabric8 Kubernetes Client	7.8.0	Reading/writing CRs, emitting Events
Temporal	1.27.0	Durable workflow execution
Embabel	2.0.0-SNAPSHOT	GOAP-based agent planning
ArchUnit	1.4.2	Enforces the module dependency graph

Tip - SNAPSHOT dependencies: Spring AI 2.0 + Spring Boot 4.1 + Embabel 2.0 are not GA yet. You must add the Spring milestone/snapshot repositories to pom.xml or you’ll get Could not resolve artifact. This is a risk - pin to GA versions before production.

Step 1 - Define the Domain: Records as Source of Truth

Everything starts with Java records in tuluat-crd-domain. This is the single source of truth for schema - the CRD YAMLs and the Fabric8 types derive from these records.

public record AiAgentSpec(
    ProviderRef providerRef,
    String model,
    String systemPrompt,
    List<ToolDefinition> tools,
    List<SkillDefinition> skills,
    GuardrailsConfig guardrails,
    List<McpServerRef> mcpServers,
    Integer replicas,
    IngressSpec ingress,
    ...
) {
    public AiAgentSpec {
        if (tools == null) tools = List.of();
        if (skills == null) skills = List.of();
        if (systemPrompt == null) systemPrompt = "You are a helpful AI assistant.";
    }
}

Tip - Compact constructor: Records give you immutability, equals/hashCode, and toString for free. The compact constructor normalizes null lists to List.of(), so you get NPE-safe defaults. This matters for reconciler diffing: status.equals(newStatus) can only be reliable when null becomes an empty list rather than a crash.

Money fields use BigDecimal (not double - floating-point rounding silently corrupts budget tracking); token counters use long.

Step 2 - Write the CRD YAMLs

After the records, write the CRDs under manifests/crd/*.yaml. Two details are critical:

spec:
  group: ai.tuluat.com
  names:
    kind: AiWorkflow
    plural: aiworkflows
    shortNames: [workflow]     # kubectl get workflow
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: true
      subresources:
        status: {}              # ← status subresource MUST be enabled
      additionalPrinterColumns:
        - name: costSpent
          type: string
          jsonPath: .status.costSpentUsd

Tip - status subresource is non-negotiable: without subresources.status: {}, UpdateControl.patchStatus() doesn’t work. Keeping status as a separate subresource also prevents spec updates from conflicting with status updates - the operator patches only .status, the user edits only .spec.

Tip - Printer columns: additionalPrinterColumns makes kubectl get aiworkflows show cost, tokens, and session count on one line. You get platform health at a glance without kubectl describe. Every jsonPath must point at .status.<field>.

Step 3 - The Operator Skeleton: Reconciler Registration

OperatorConfig registers five reconcilers with the JOSDK Operator:

@Configuration
public class OperatorConfig {

    @Bean
    public KubernetesClient kubernetesClient() {
        return new KubernetesClientBuilder().build();
    }

    @Bean(destroyMethod = "stop")
    @ConditionalOnExpression("environment.getProperty('AGENT_NAME') == null")
    public Operator operator(KubernetesClient client,
            LlmProviderReconciler providerReconciler,
            AiAgentReconciler agentReconciler,
            McpServerReconciler mcpServerReconciler,
            AiWorkflowReconciler workflowReconciler,
            WorkflowSessionReconciler sessionReconciler) {
        Operator operator = new Operator(o -> o.withKubernetesClient(client));
        operator.register(providerReconciler);
        operator.register(agentReconciler);
        operator.register(mcpServerReconciler);
        operator.register(workflowReconciler);
        operator.register(sessionReconciler);
        operator.start();
        return operator;
    }
}

Tip - One binary, two roles: @ConditionalOnExpression(... AGENT_NAME == null) lets the same Docker image run as both operator and agent runtime. The AiAgentReconciler creates a Deployment per agent with an AGENT_NAME env var; in those pods the operator bean is never created. One image, one release pipeline - two distinct behaviors.

Step 4 - Reconciler Deep-Dive + Tips

A reconciler is a JOSDK Reconciler<T>; its single method reconcile(T resource, Context<T>) runs on every event (create/update/delete) and on periodic resync. Tuluat uses two distinct reconciler patterns.

4a. Child-resource reconciler - `AiAgentReconciler`

@Override
public UpdateControl<AiAgent> reconcile(AiAgent agent, Context<AiAgent> context) {
    try {
        var spec = agent.getSpec();
        // 1. Resolve the referenced LlmProvider (cross-namespace aware)
        LlmProvider provider = resolveProvider(spec, agent.getMetadata().getNamespace());
        if (provider == null) {
            agent.setStatus(AiAgentStatus.reconciling("Waiting for LlmProvider ...", ...));
            return UpdateControl.patchStatus(agent);
        }

        // 2. OwnerReference - for garbage collection
        OwnerReference ownerRef = new OwnerReferenceBuilder()
            .withApiVersion(agent.getApiVersion())
            .withKind(agent.getKind())
            .withName(agent.getMetadata().getName())
            .withUid(agent.getMetadata().getUid())
            .withController(true)
            .withBlockOwnerDeletion(true)
            .build();

        // 3. Reconcile Deployment, Service, Ingress
        reconcileDeployment(agent, ownerRef, ns);
        reconcileService(agent, ownerRef, ns);
        String ingressUrl = reconcileIngress(agent, ownerRef, ns);

        agent.setStatus(AiAgentStatus.ready(ingressUrl, activeSkills, activeTools, ...));
        return UpdateControl.patchStatus(agent);
    } catch (Exception e) {
        agent.setStatus(AiAgentStatus.failed("Reconciliation failure: " + e.getMessage(), ...));
        return UpdateControl.patchStatus(agent);
    }
}

Tip - OwnerReference + cascade delete: give child resources withController(true) + withBlockOwnerDeletion(true) and Kubernetes GC deletes the Deployment/Service/Ingress automatically when you kubectl delete aiagent X. Trust the platform’s GC instead of writing manual delete calls.

Tip - Immutable field drift: a Deployment’s selector.matchLabels is immutable; you can’t update it. Tuluat detects this: if labels drifted, it deletes → waitUntilCondition(Objects::isNull) → creates. Ignore it and call update(), and you’ll get spec.selector: Invalid value ... is immutable.

Tip - Idempotency: reconcile runs again on every resync. It must be idempotent - use Fabric8 helpers like createOrReplace(), never blind create. Otherwise you get resource-version conflicts and an event storm.

Tip - Catch, don’t crash: if a reconciler throws, JOSDK retries but the log gets noisy. Instead, catch the exception, write status = failed(...), and patchStatus. The loop never dies, and state stays observable.

4b. DB-aggregation reconciler - `AiWorkflowReconciler`

This reconciler creates no child resources; it aggregates real DB data from WorkflowSessionRepository + NodeExecutionRepository into status:

@ControllerConfiguration(
    maxReconciliationInterval = @MaxReconciliationInterval(interval = 30, timeUnit = TimeUnit.SECONDS))
public class AiWorkflowReconciler implements Reconciler<AiWorkflow> {

    @Override
    public UpdateControl<AiWorkflow> reconcile(AiWorkflow resource, Context<AiWorkflow> context) {
        // ... aggregate cost/tokens/agentNames from sessions ...

        AiWorkflowStatus newStatus = new AiWorkflowStatus("Ready", nodeCount,
            costSpent.toPlainString(), budget.toPlainString(), sessionCount,
            totalTokens, inputTokens, outputTokens, agentNames);

        // Skip the status PATCH when nothing changed
        if (newStatus.equals(resource.getStatus())) {
            return UpdateControl.noUpdate();
        }
        resource.setStatus(newStatus);
        eventRecorder.record(resource, TYPE_NORMAL, "WorkflowStatusUpdated", ...);
        return UpdateControl.patchStatus(resource);
    }
}

Tip - Change detection (infinite-loop guard): patchStatus triggers an update event → which triggers another reconcile → which patches again… an infinite loop. The fix is simple: if the new status equals the existing status, return UpdateControl.noUpdate(). This is the biggest payoff of records giving you equals() for free.

Tip - maxReconciliationInterval: when a session completes in the DB, no event fires on the AiWorkflow CR - because the session is a separate CR. A 30-second periodic resync keeps the AiWorkflow status eventually consistent. This is a clean example of mixing event-driven with time-driven reconciliation.

Tip - toPlainString(): serializing a BigDecimal directly yields scientific notation like 6.6e-05. Use toPlainString() for readable 0.000066 - this is why status cost fields are String (see Step 13).

Step 5 - Kubernetes Events: The Audit Trail

To surface in kubectl describe and kubectl get events, you record core/v1 Event objects. Tuluat does this in KubernetesEventRecorder:

@Component
public class KubernetesEventRecorder {
    public void record(HasMetadata resource, String type, String reason, String message) {
        if (resource == null || resource.getMetadata() == null) return;
        try {
            Event event = new EventBuilder()
                .withNewMetadata()
                    .withGenerateName(resource.getMetadata().getName() + "-")
                    .withNamespace(ns)
                .endMetadata()
                .withType(type)          // "Normal" | "Warning"
                .withReason(reason)
                .withMessage(message)
                .withNewInvolvedObject()
                    .withKind(resource.getKind())
                    .withApiVersion(resource.getApiVersion())
                    .withName(resource.getMetadata().getName())
                    .withUid(resource.getMetadata().getUid())
                .endInvolvedObject()
                .withFirstTimestamp(Instant.now().toString())
                .withLastTimestamp(Instant.now().toString())
                .withCount(1)
                .build();
            client.v1().events().inNamespace(ns).resource(event).create();
        } catch (Exception e) {
            log.warn("Failed to record event ...");   // swallow, never propagate
        }
    }
}

Tip - Event recording must never break reconciliation: it’s best-effort, wrapped in try/catch, and failures are logged and swallowed. If event emission becomes a hard dependency, even an RBAC mistake locks up the whole platform.

Tip - involvedObject: this field binds the event to the resource - always fill kind, apiVersion, name, and uid, or kubectl describe aiworkflow X won’t show it. generateName avoids name collisions.

Tuluat emits these event reasons:

Reason	Type	Meaning
`WorkflowSessionStarted`	Normal	Session began executing
`WorkflowSessionCompleted`	Normal	Session completed successfully
`WorkflowSessionFailed`	Warning	Session failed
`WorkflowSessionWaitingApproval`	Normal	Awaiting human approval
`WorkflowSessionRejected`	Warning	Session was rejected
`WorkflowSessionWorkflowNotFound`	Warning	Referenced workflow missing
`WorkflowStatusUpdated`	Normal	Workflow metrics refreshed

Step 6 - The Execution Pipeline: An Agent Call’s Journey

AgentExecutionService routes every agent call through an eight-stage pipeline:

1. Resolve model & provider    ←  AiAgent CR + LlmProvider CR
2. Pre-execution guardrails    ←  PiiMaskingFilter + PromptInjectionFilter
3. Execute tools               ←  Virtual Threads (one per tool, concurrent)
4. Inject session memory       ←  short-term conversation memory
5. Build system prompt         ←  base + tool + skill + MCP + RAG context
6. Invoke LLM                  ←  ModelGateway → Spring AI ChatModel
7. Validate output             ←  OutputValidationFilter
8. Save response to memory     ←  SessionMemoryManager

The GuardrailPipeline is a proper filter chain - each filter is a @Service implementing either PreExecutionFilter or PostExecutionFilter. Adding a guardrail is one class, zero framework changes.

Agent Logs

Step 7 - Workflows as a Sequential Graph

AiWorkflow CRs describe workflows as a sequential graph - a linear chain of typed nodes with conditional branches and maxLoops-bounded loops. This is not yet a full parallel DAG (fan-out/fan-in); parallel execution is on the roadmap:

spec:
  initialNode: research
  nodes:
    - { id: research, type: AGENT,         agentRef: researcher, outputKey: findings }
    - { id: decide,   type: CONDITION,     expression: "riskScore > 0.8" }
    - { id: approve,  type: HUMAN_APPROVAL }
  edges:
    - { from: research, to: decide }
    - { from: decide,   to: approve, condition: "true" }
    - { from: decide,   to: research, condition: "false" }   # loop - bounded by maxLoops

GraphStateMachineEngine walks the graph node-by-node, persisting WorkflowSessionEntity to PostgreSQL at every step:

public WorkflowSessionEntity executeNextStep(AiWorkflowSpec spec, WorkflowSessionEntity session, int maxLoops) {
    if (session.getLoopCount() >= maxLoops) {
        session.setStatus(SessionStatus.FAILED);   // infinite-loop guard
        return session;
    }
    // ...
    if ("AGENT".equalsIgnoreCase(node.type())) {
        AgentResponse response = agentExecutionService.executeAgent(...);
        persistNodeExecution(...);                 // per-node metric persistence
        contextData.put(node.outputKey(), response.answer());
        // validate against outputSchema if present
    } else if ("CONDITION".equalsIgnoreCase(node.type())) {
        boolean result = evaluateCondition(node.expression(), contextData);  // SpEL
        // route by edge.condition
    } else if ("HUMAN_APPROVAL".equalsIgnoreCase(node.type())) {
        if (!contextData.containsKey("approvalStatus")) {
            session.setStatus(SessionStatus.WAITING_APPROVAL);  // pause
            return session;
        }
        // advance on approve/reject
    }
    session.setLoopCount(session.getLoopCount() + 1);
    return session;
}

Tip - SpEL for conditions: CONDITION nodes evaluate Spring Expression Language (SpelExpressionParser) over contextData. Expressions like riskScore > 0.8 are managed from YAML without recompilation - change business logic without a redeploy.

Tip - maxLoops loop guard: graphs can contain loops (conditional back-edges). Without maxLoops, a bad condition spins forever and burns LLM budget. The value comes from WorkflowSession.spec.parameters.maxLoops, defaulting to 10.

Tip - per-node metric persistence: the root cause of 0 tokens/cost was that metrics were only written inside Temporal activities. When Temporal was bypassed (dev environment), nothing reached the DB. Adding NodeExecutionRepository + persistNodeExecution to GraphStateMachineEngine persisted metrics on both paths.

Workflow Graph

Step 8 - Human-in-the-Loop: The Approval Inbox

When a workflow reaches a HUMAN_APPROVAL node, the session transitions to WAITING_APPROVAL. The operator:

Publishes a WebSocket event to connected dashboards
Waits for POST /api/v1/workflows/{sessionId}/approve or /reject
If using Temporal, sends an ApprovalSignal to the running workflow

WorkflowExecutionService.processApprovalSignal writes the decision into contextData (approvalStatus, approvalFeedback) and resumes the graph from where it paused. The approval inbox UI shows pending approvals with full workflow context, the agent’s reasoning, and the accumulated contextData.

Tip - Internal vs external HITL: what’s implemented today is the in-process approval inbox (WebSocket + REST /approve//reject + the dashboard inbox). External HITL channels (email, webhooks, external actions) are on the roadmap - the Temporal ApprovalSignal infrastructure will serve as their foundation too.

Approval Inbox

Step 9 - Temporal: Durable Execution

Long-running workflows delegate execution to WorkflowSessionTemporalWorkflowImpl. TemporalConfig starts a WorkerFactory:

public static final String TASK_QUEUE = "AI_WORKFLOW_TASK_QUEUE";

@Bean
public WorkerFactory workerFactory(WorkflowClient client, GraphNodeActivitiesImpl activities) {
    WorkerFactory factory = WorkerFactory.newInstance(client);
    Worker worker = factory.newWorker(TASK_QUEUE);
    worker.registerWorkflowImplementationTypes(WorkflowSessionTemporalWorkflowImpl.class);
    worker.registerActivitiesImplementations(activities);
    factory.start();
    return factory;
}

The workflow implementation runs nodes over an ActivityStub and receives the approval signal via signalApproval:

public Map<String, Object> runSession(UUID sessionId, String workflowName, AiWorkflowSpec spec, ...) {
    Map<String, NodeDefinition> nodeIndex = spec.nodes().stream()
        .collect(Collectors.toMap(NodeDefinition::id, Function.identity()));  // O(1) lookup

    String currentNodeId = spec.initialNode();
    int loopCount = 0;
    while (currentNodeId != null && loopCount < maxLoops) {
        NodeType type = NodeType.from(nodeIndex.get(currentNodeId).type());
        var executor = executorFactory.getExecutor(type);
        currentNodeId = executor.map(e -> e.execute(...)).orElse(null);
        loopCount++;
    }
    return Map.copyOf(contextData);
}

Tip - Graceful degradation: WorkflowClient is injected as Optional<WorkflowClient>. When no Temporal cluster exists, the engine falls back to the in-process GraphStateMachineEngine. Dev needs no Temporal; production gets durable execution, distributed retry, and visibility - all through one interface.

Tip - Temporal determinism: workflow code must be deterministic - System.currentTimeMillis() and direct I/O are forbidden in the workflow body; they belong in activities. In Tuluat all LLM calls and DB access go through GraphNodeActivities; the workflow holds only orchestration. This separation is required for correct Temporal replay.

Step 10 - Embabel: GOAP-Based Agent Planning

Embabel plans agents by goal using Goal-Oriented Action Planning (GOAP). Tuluat integrates it in two parts:

1. Dynamic model registration from CRDs - CrdEmbabelConfiguration:

@Configuration(proxyBeanMethods = false)
@ConditionalOnBean(KubernetesClient.class)
@EnableScheduling
public class CrdEmbabelConfiguration {

    @Bean
    ProviderInitialization crdProviderInitialization() {
        return scanAndRegisterProviders("startup");
    }

    @Scheduled(fixedDelayString = "PT60S", initialDelayString = "PT30S")
    void reconcileCrdProviders() { ... }   // scan for new/updated LlmProvider CRs
}

Tip - Dynamic reconciliation without restart: @Scheduled re-scans the cluster every 60 seconds and registers new/updated LlmProvider CRs with Embabel. Adding a provider requires no operator restart. A registeredProviderNames set skips redundant re-registration.

2. The goal agent - TuluatGoalAgent:

@Agent(description = "Executes AI agent goals using the Tuluat agent execution pipeline")
public class TuluatGoalAgent {

    @AchievesGoal(description = "Completes the goal by executing the named AI agent")
    @Action
    public GoalResult executeGoal(GoalRequest request) {
        AgentResponse response = agentExecutionService.executeAgent(
            request.agentName(), request.goalDescription(), ...);
        return new GoalResult(request.agentName(), response.answer(), response.usage());
    }
}

Embabel’s GOAP engine discovers @Agent + @AchievesGoal + @Action and dynamically sequences multi-step goals (research → verify → report) by typed input/output contracts. For single-step goals planning is trivial; Tuluat delegates this agent to the full execution pipeline (guardrails → skills → RAG → gateway).

Step 11 - Model Gateway: Fallback and Budget

ModelGateway sits between the engine and Spring AI:

Provider routing - resolves the right ChatModel bean from the LlmProvider CRD type
Ordered fallback chains - walks spec.fallbacks[] in order when the primary fails
Budget enforcement - tracks per-agent USD spend, throws BudgetExceededException before the LLM call when the cap is exceeded

spec:
  type: OPENAI
  defaultModel: gpt-4o
  fallbacks:
    - { providerName: anthropic-provider, model: claude-3-5-sonnet-20241022 }
    - { providerName: ollama-local,      model: llama3.2 }

If every route fails, the engine degrades to a simulated response - keeping workflows alive in development without real API keys.

Tip - Placeholder: this ModelGateway is a lightweight in-process router (fallback + budget) that keeps the PoC alive. Long-term it will be replaced by a real centralized model gateway (e.g. LiteLLM), with provider routing + budget + fallback moving to that layer.

MCP Providers

Step 12 - RAG: Kubernetes-Native Document Ingestion

The RAG pipeline follows a clean ingest → embed → retrieve pattern:

Ingest - POST to /api/v1/rag/ingest with a sourceRef prefix
Chunk - RecursiveCharacterChunker splits text with configurable overlap
Embed - SpringAiEmbeddingProvider (or LocalHashEmbeddingProvider for dev) produces vectors
Store - raw document in MinIO (S3-compatible), embeddings in pgvector
Retrieve - cosine similarity top-K, injected into the system prompt with source attribution

Retrieved context injected into system prompt:
[sourceRef runbooks/incident-42 #1 (sim 0.91)]: ...
[sourceRef runbooks/incident-42 #2 (sim 0.87)]: ...

Retrieval is per-query, automatic, and invisible to the agent - it just gets a richer system prompt.

Step 13 - Persistence: PostgreSQL, Flyway, and the Transaction Trap

Tuluat does not let Spring Boot manage Flyway - migrations run in a separate Kubernetes Job before the operator starts, and Spring only runs ddl-auto=validate:

spring.datasource.url=${SPRING_DATASOURCE_URL:jdbc:postgresql://postgres-service:5432/ai_operator_db}
spring.datasource.hikari.maximum-pool-size=3
spring.jpa.hibernate.ddl-auto=validate
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect

Migrations live in one project with sequential numbers (V1__workflow_operator.sql … V6__drop_short_memory_foreign_key.sql); a Flyway Job applies them from a ConfigMap using flyway/flyway:11-alpine.

Tip - The @Transactional removal story: originally startSession held the entire workflow (slow LLM calls) in one transaction. This blocked a HikariCP connection for minutes and produced FATAL: sorry, too many clients. The fix: drop @Transactional - every repository.save already runs in its own transaction. Not holding a connection across the LLM call is the root fix; raising max_connections is only a band-aid.

Tip - PVC required: PostgreSQL data won’t survive without a PVC; a rolling update/restart deletes Flyway’s tables → Schema validation: missing table [session_short_memory]. Mount a 1Gi PVC at /var/lib/postgresql/data.

Tip - Cost → String: a BigDecimal serializes as 6.6e-05 in status. Use String in the Java record, the CRD schema (type: string), and the printer column. Change only the Java side and forget the schema, and the API server rejects the status PATCH: expected numeric, got string. All three (record + CRD schema + column) must change together.

Step 14 - Observability: Micrometer

Every workflow transition emits Micrometer counters:

ai.workflow.session.created.total{workflow="invoice-approval"}
ai.workflow.session.completed.total{workflow="invoice-approval",status="COMPLETED"}
ai.workflow.node.executed.total{workflow="...",node_type="AGENT",node_id="analyze"}

These feed directly into Prometheus + Grafana dashboards without extra instrumentation. WorkflowTelemetryService is injected as Optional at every transition point - the engine runs even when telemetry is disabled.

Step 15 - Custom Resource Summary

The project defines five Custom Resource Definitions:

CRD	Plural	Key `spec`	Key `status`
LlmProvider	`llmproviders`	`type`, `defaultModel`, `fallbacks[]`, `apiKey.secretKeyRef`	provider state
AiAgent	`aiagents`	`providerRef`, `model`, `systemPrompt`, `guardrails`, `tools[]`, `skills[]`, `mcpServers[]`, `replicas`, `ingress`	`phase`, `ingressUrl`, active skills/tools/mcp, `model`
McpServer	`mcpservers`	MCP server endpoint/config	MCP state
AiWorkflow	`aiworkflows`	`initialNode`, `nodes[]`, `edges[]`, `memoryConfig`, `budgetLimitUsd`	`state`, `nodeCount`, `costSpentUsd`, `budgetLimitUsd`, `sessionCount`, `totalTokens`, `inputTokens`, `outputTokens`, `agentNames[]`
WorkflowSession	`workflowsessions`	`workflowRef`, `input`, `parameters`	`sessionId`, `phase`, `currentNode`, `output`, `startTime`, `endTime`, `totalTokens`, `inputTokens`, `outputTokens`, `costUsd`, `durationSeconds`, `nodeExecutions[]`

The flow: a user defines an AiWorkflow → WorkflowSessionController.createSession creates a WorkflowSession CR → WorkflowSessionReconciler sees it, calls startSession, and walks the graph → on completion, AiWorkflowReconciler (30s resync) aggregates all sessions from the DB and updates AiWorkflow.status.

kubectl get aiworkflows -n tuluat-system
# NAME                       STATE  COSTSPENT  BUDGET  SESSIONS  TOTALTOKENS  INPUTTOKENS  OUTPUTTOKENS
# multi-agent-researcher     Ready  0.000132   0       2          140          80           60
# order-processing-workflow  Ready  0.000066   0       2          175          100          75

Visual Canvas

Step 16 - Java 25 + Spring Boot 4 in Practice

Virtual Threads

Tools run concurrently on virtual threads - no thread-pool sizing, no blocking:

var futures = activeDefs.stream()
    .map(def -> virtualThreadExecutor.submit(() -> tool.execute(input, def.parameters())))
    .toList();

Executors.newVirtualThreadPerTaskExecutor() means thousands of concurrent tool calls with minimal overhead. Spring’s async tasks are also wired to virtual threads:

@Bean
public AsyncTaskExecutor applicationTaskExecutor() {
    return new TaskExecutorAdapter(Executors.newVirtualThreadPerTaskExecutor());
}

Java Records

All CRD specs/statuses are records with compact constructors (Step 1). Immutability + equals/hashCode + toString come free; reconciler diffing and change detection are built on top of them.

Switch Expressions & Pattern Matching

switch (entity.getStatus()) {
    case COMPLETED        -> reason = "WorkflowSessionCompleted";
    case FAILED           -> reason = "WorkflowSessionFailed";
    case WAITING_APPROVAL -> reason = "WorkflowSessionWaitingApproval";
    case REJECTED         -> reason = "WorkflowSessionRejected";
    default               -> reason = "WorkflowSessionUpdated";
}

Optional Dependency Injection (ADR 005)

Every optional collaborator uses Optional<T> in the constructor:

public AgentExecutionService(
    ToolRegistry toolRegistry,
    Optional<SkillRegistry> skillRegistry,
    Optional<ModelGateway> modelGateway,
    Optional<RagService> ragService,
    ...
)

This makes the engine testable without Spring context, enables progressive feature enablement via @ConditionalOn*, and documents optionality at the type level.

Step 17 - The Pipeline: Kind + Kustomize + Helm

The development loop is built around three tools: Kind (local cluster), Kustomize (sample resources + secrets), and Helm (release packaging).

Local: Kind + Kustomize

./scripts/create-kind-cluster.sh sets everything up idempotently:

kind-config.yaml - extraPortMappings for Ingress (host 80/443 → container 80/443) plus the DynamicResourceAllocation=true feature gate
Create the Kind cluster (skips if present) + install the NGINX Ingress Controller
docker build -t tuluat-operator:latest .
kind load docker-image tuluat-operator:latest - load the image into the cluster
deploy-operator.sh - CRDs → RBAC → Temporal/MinIO/WireMock → sample resources → Flyway Job

./scripts/create-kind-cluster.sh

deploy-operator.sh applies sample resources with Kustomize: kubectl apply -k config/. config/kustomization.yaml bundles every sample CR (samples/*.yaml) and generates API-key secrets from environment variables via secretGenerator:

secretGenerator:
  - name: deepseek-secret
    literals:
      - api-key=${DEEPSEEK_API_KEY:-sk-placeholder-deepseek-key}
  - name: openai-secret
    literals:
      - api-key=${OPENAI_API_KEY:-sk-placeholder-openai-key}
generatorOptions:
  disableNameSuffixHash: true

Tip - kind load docker-image: you don’t need a registry in local dev - kind load docker-image copies the image straight into the node container; a zero-friction loop with no registry setup.

Tip - don’t commit secrets: secretGenerator reads API keys from env vars with a placeholder fallback. Real keys never enter the repo; disableNameSuffixHash: true keeps the secret name stable (deepseek-secret), so SecretKeyRef references don’t break.

Release: Helm

For release, the project is packaged as a single Helm chart (helm/tuluat-operator): the operator, CRDs, infrastructure (PostgreSQL+pgvector, Temporal, MinIO, WireMock, Prometheus, Grafana), and sample resources install together with one helm upgrade --install.

helm package helm/tuluat-operator -d dist
helm push dist/*.tgz "oci://ghcr.io/<org>/charts"      # GHCR OCI registry
helm upgrade --install tuluat-operator dist/*.tgz \
  --namespace tuluat-system --set wiremock.enabled=true --set samples.install=true

Tip - Helm vs Kustomize: Kustomize manages local dev and sample resources (with env-var secrets); Helm does the release packaging - one chart carries the whole stack to the GHCR OCI registry, and CI’s e2e-kind installs that chart. They’re complementary layers, not rivals.

CI: GitHub Actions

The GitHub Actions pipeline runs jobs in parallel where possible:

flowchart TD
    DC["detect-changes"]
    Build["build<br/>Spotless → Checkstyle → ArchUnit → tests → Docker image"]
    PackageHelm["package-helm"]
    PublishHelm["publish-helm<br/>GHCR OCI registry"]
    E2E["e2e-kind<br/>KinD cluster → Helm install → E2E acceptance tests"]
    Docs["build-docs<br/>MkDocs → GitHub Pages"]

    DC --> Build
    DC --> PackageHelm --> PublishHelm --> E2E
    DC --> Docs

ArchUnit enforces the module dependency graph - engine can’t import from operator, domain can’t import from engine. Architectural drift is caught at compile time (ArchitectureTest 13/13).

Tip - No wildcard imports: the codebase has no import com.foo.*; fully-qualified class-name (FQCN) usage is also banned, enforced by Checkstyle + ArchUnit. This keeps diffs small and reviews easy.

Technical Debt: An Honest Inventory

After a fast PoC, we documented known limitations to address later:

Debt	Impact
Global RAG vector store	No agent isolation - one department’s docs can leak to another
In-memory budget tracking	Resets on pod restart; doesn’t scale across replicas
Shared tool registry	No tool isolation; JAR classloader leak risk
MCP tool routing partial	`mcpServers:` spec field is partly a no-op at runtime
SNAPSHOT dependencies	Spring AI 2.0 + Spring Boot 4.1 + Embabel 2.0 not yet GA
Helm CRDs stale	`manifests/crd/` is used for deploy; Helm CRDs out of sync

Each is documented under docs/tech-debt/ with a suggested remediation path.

Try It

git clone https://github.com/netologist/tuluat
./scripts/create-kind-cluster.sh
./scripts/deploy-operator.sh          # CRDs + PostgreSQL/Flyway + Kustomize sample resources
kubectl get aiworkflows -n tuluat-system
kubectl get workflowsessions -n tuluat-system

The operator reconciles your CRDs and exposes the dashboard at http://localhost:8080.

Built with Java 25, Spring Boot 4.1, Spring AI 2.0, JOSDK 5.1, Temporal 1.27, Embabel 2.0, Fabric8 7.8, and a healthy respect for the Kubernetes control loop.