Pump Fiction: How an AI Platform Handled a Pump That Kept Running Hot

Pump Fiction

AI proposes, the decision engine governs, Temporal executes.

A pump in Manchester overheats for the fourth time in 30 days. This post walks through every step from the moment that signal enters the system, and at each step, which technology I chose and why.

Honesty note, up front: the whole chain runs end to end on a real Kubernetes cluster, but the enterprise systems are mocks and the decision engine is a local Go stub. The stub speaks the System One API and simulates the model (Laya, served by Ollaya) so CI costs nothing. The calibrated numbers below (like the 0.94) come from that stub. The architecture and the tests are real; the intelligence behind the decision is simulated. The point of this PoC is the boundaries between the layers, not the model inside any one of them.

The scene: P-104

Centrifugal pump P-104, Manchester plant. Its fourth overheat event in 30 days. For the maintenance team this is no longer “watch and wait” territory, but is it actually urgent? Is the spare part in stock? What does the manual say? Does a technician need to go on site? And most importantly: who approves it?

Each of these questions lives in a different enterprise system (EAM, PLM, ERP, FSM), and none of them knows about the others. A human working this chain by hand takes hours; handing the chain to a free-running LLM means hallucination, unbounded loops, and unauditable actions.

The system I built (optimus) charts a path between the two: AI proposes, the decision engine governs, Temporal executes.

Five steps follow the pump’s signal from arrival to work order. Three more (observability, infrastructure, verification) explain what holds the whole thing together.

P-104's journey from signal to work order

Figure 1: The whole scenario at a glance. What follows explains the “why” behind each arrow.

Part I: The flow

1. The signal arrives

The telemetry layer sends POST /v1/tenants/acme/signals with asset_id: P-104.

What I chose: Kong (ingress) → Platform API (Go) → PostgreSQL 17.

Why Go: the platform layer runs at high concurrency and wants small static binaries. The operator and the API speaking the same language is a bonus, since Kubernetes itself is Go.

Why Kong: TLS termination, rate limiting, and service-to-service JWT auth aren’t application-code concerns.

Why PostgreSQL, and RLS in particular: this system is multi-tenant. One tenant’s data leaking into another can’t be entrusted to an if check in the application layer. So I enforce it at the database layer with PostgreSQL Row-Level Security. Even if the application forgets, the database doesn’t.

How it works: every row has a tenant_id, and every table has a policy like this:

CREATE POLICY assets_tenant_isolation ON assets
    USING (tenant_id = NULLIF(current_setting('app.current_tenant', true), ''));

When a request arrives, the API establishes the context in a single line: SELECT set_config('app.current_tenant', $1, true). From that moment PostgreSQL runs every query through this filter. The NULLIF(..., '') part is critical: if the setting was never applied, the comparison returns NULL, and NULL matches no row. So accidentally writing an “open to everyone” query is impossible for the application role. The app connects as a role that neither owns the table nor has BYPASSRLS (FORCE ROW LEVEL SECURITY extends the same guarantee to the table owner). Forgetting doesn’t leak data, it returns nothing. Fail-closed.

Scoped to what, exactly: that guarantee is a property of the connection, not of the database. It holds as long as every request path runs as this role; a superuser or a role with BYPASSRLS bypasses RLS by definition. So the honest claim is the narrow one: for the application role, forgetting the tenant context fails closed.

The key point: when the signal row is written, the signal.received event is written to the outbox table in the same transaction. We’ll return to this in step 5, because the most commonly swallowed mistake in distributed systems lives there.

2. Investigation: agents that reach enterprise systems only through tools

The Temporal workflow starts an activity: POST /investigate.

On the Python side, the Pydantic AI planner first pulls the list of tools it’s allowed to call for that tenant from the Platform API, then queries the enterprise systems over MCP:

EAM → 4 overheat records for P-104
PLM → hybrid RAG finds maintenance manual PLM-COOL-4021 §4.2, a cooling-circuit failure mode
ERP → spare thermostat SP-COOL-9981, 6 in stock

Why Pydantic AI: the agent’s output had to be typed. Parsing the output of a free-text planner produces code that’s unmaintainable six months later. EvidenceContext is a schema; tests assert on that schema’s fields, not on subjective sentences.

Why MCP: MCP provides a standard discovery-and-call protocol, and I wrote one adapter per enterprise system (instead of one monolith). Changing one system doesn’t touch the others.

What is MCP, concretely? The agent first calls tools/list to get the tools exposed to that tenant (eam.get_maintenance_history, plm.search_documents, erp.get_inventory…), then calls them one by one with tools/call. The world the agent sees is exactly the contents of that list: there is no “write to the database” tool.

What MCP is not: MCP is capability discovery and invocation, not authorization enforcement. tools/list tells the agent what it can call; MCP itself does not decide who is allowed to call it, and there is no policy engine hiding inside the protocol. That decision lives one layer up, in the host/gateway that chooses which tools to expose to which tenant and with which credentials. So the security property here is narrow and precise: the agent can only invoke what its host put in the list. The audit question reduces from a full code review to which tools the host chose to expose — but that’s a property of my host configuration, not a feature MCP grants me.

Why hybrid RAG: vector search alone is weak when you’re looking for an exact identifier like PLM-COOL-4021; full-text search alone is blind to semantic queries like “cooling problem”. I combined the two, pgvector cosine similarity plus PostgreSQL full-text search, fused with RRF.

RRF (Reciprocal Rank Fusion) is simple but powerful: you take two separate rankings (vector similarity and full-text) and score each document by 1/(k + rank) summed across the two lists. It fuses ranks, not raw similarity scores, so if one method shows a document at rank 1 and the other at rank 9, the result stays balanced.

And the most important rule: the agent never touches the database, Temporal, or an enterprise system directly. It only calls tools over MCP. That single sentence carries the entire auditability of the system.

3. The decision: typed, calibrated, auditable

The evidence package goes to decision-service (POST /v1/decisions).

Why not a free-running LLM: getting a sentence back for “how serious is this failure?” is useless. I need a number I can threshold and a label I can classify. So the decision engine answers typed questions and returns calibrated probabilities in a single forward pass: no token generation, and small enough models to run on a CPU.

What “typed” buys, exactly: bounded output. A typed head can only emit a number on a fixed scale or one of a fixed set of labels, so the result is schema-valid by construction — there is no free-text channel for a fluent-but-wrong sentence to travel down. Schema-valid is not the same as semantically correct: severity: 2.1 is always a well-formed number, but it can still be the wrong number about this pump. Typing removes one failure mode (ungrounded prose); it does not remove the model being mistaken. That residual risk is precisely what the policy layer and the human approval below are for.

The typed questions come from the System One API (introduced with TypeSafe’s hosted Jev model). Laya, an open-weight model, implements the same three primitives, and Ollaya is an open-source runtime that serves such models locally behind the same /v1/systemone endpoint. Each primitive measures something different:

score (severity): a number on a “low/monitor”, “medium/schedule maintenance”, “high/P1” scale
choice (safety_risk): one of three labels: low/medium/high safety risk
noul (field_visit_required): yes/no: does a technician need to go on site?

Three types force three different decision kinds: ranking, classification, binary. None of them is free text.

Result: severity: 2.1 · safety_risk: HIGH · field_visit_required: true · confidence: 0.94.

Read that 0.94 carefully: in this repository the number is produced by the local Go stub, not by a calibrated Laya or Jev model. It demonstrates the shape of a calibrated confidence — a scalar the policy layer can threshold — not a measured model calibration. A real model would return its own value here; what’s being shown is the pipeline that consumes it, not the quality of the number itself.

Why the System One API shape, not an engine-specific client: the decision service speaks the API, not a particular engine. Today it talks to a local stub; tomorrow it could talk to Ollaya serving Laya, or to hosted Jev. What changes is configuration, not code. Not locking into a vendor is an architectural decision, recorded in ADR-003.

Why a separate policy layer: this is my favorite part of the project. The stub’s confidence is 0.94, which reads high. But what stops the system and pulls a human in is not confidence, it’s policy:

safety_risk == HIGH → approval is mandatory, regardless of confidence.

That rule doesn’t live in application code; it lives in a versioned, declarative policy. Changing the policy is a configuration change, not a code change, and every decision record notes which policy version was applied.

policy_v1 actually contains three deterministic rules, evaluated in order:

safety_risk == HIGH → approval mandatory (regardless of confidence)
severity == P1 and field dispatch required → approval mandatory
confidence < 0.75 → approval mandatory (low confidence, human verification)

Each rule produces an exact reason that gets recorded, for example "HIGH safety risk detected: human approval mandated by policy_v1". The severity score is also deterministic: score >= 2.0 → P1 (P-104’s 2.1 → P1), >= 1.0 → P2, otherwise P3.

Why kubectl describe isn’t enough: a decision’s audit trail must include the policy version, the confidence threshold, a hash of the input sent to the engine, and the probabilities that came back. That’s why there’s a queryable decision_audit table, not Kubernetes events.

4. Approval: here the workflow waits

The Temporal workflow moves into a durable wait, with a 24-hour timeout. When the supervisor approves, the Platform API sends a signal: approval-granted.

Why Temporal: an operation waiting for approval has no “desired state”; it has state. To be precise about what that means: you can express a state machine in Kubernetes — a CRD with a phase field, status transitions, and a controller that acts on them is exactly that. The claim isn’t that Kubernetes is incapable of state. It’s that the controller’s core contract is the wrong contract for this job. The CRD/reconcile model is a “desired state → converge” contract: you declare the target, and the controller repeatedly drags reality toward it, by design being safe to re-run at any moment. This workflow needs something else: wait on a signal, sleep on a durable timer, run steps in order, and compensate the step that fails. You can force that into a reconciler, but then you’re fighting the contract — re-deriving a step sequence on every reconcile — instead of using it.

Let me also preempt a second misunderstanding: Temporal retries activities too, so “Temporal runs a side effect exactly once” would be wrong. The side effect itself has to be idempotent there as well (the integration handlers in the repo are). The difference is this: Temporal owns the durable state machine that decides which side effect runs when, and it owns that across restarts, timers, and signals; idempotency lives on the side effect. A reconciler is built to converge on a target, not to carry a decision sequence through a 24-hour external wait and a compensating rollback.

5. Action: and what happens when it goes wrong

After approval:

ERP inventory reservation → RES-9981
FSM field work order → WO-10423

What if the work order can’t be created? Saga compensation kicks in: a compensating step runs the opposite operation to release the reservation and emits a work_order.failed_compensated event. Half-finished work produces an explicit record, not silent data corruption.

Why not a single “distributed transaction”: ERP and FSM are two different vendors’ HTTP APIs. You can’t get atomicity between them (both systems committing in one decision), because there’s no 2PC/XA in this world. Saga’s answer is different: every step defines a compensating action. Where a true inverse exists, compensation is that inverse — the compensation for a reservation is “release the reservation” (erp.release_inventory_reservation). Where it doesn’t, compensation is a semantic undo, not an algebraic one: you cancel forward, you don’t rewind. If a step fails, the compensations of the previously successful steps run. The system doesn’t return to “as if nothing happened”; it returns toward a consistent state, and says so explicitly with the work_order.failed_compensated event.

And there was a sneakier failure: the work succeeded, but the event couldn’t be published. Kafka went down, the publisher crashed, the network dropped.

Why transactional outbox: the work-order row and the work_order.created event are written in the same PostgreSQL transaction. A separate outbox-relay process reads unpublished rows and publishes them to Redpanda, carrying the W3C traceparent as a record header. The event can’t diverge from the row: even if the publisher crashes, the row remains, and the relay publishes it when it comes back up.

How it works, concretely:

The app writes the work_orders row; in the same transaction it adds a work_order.created row to the outbox table. They commit together or roll back together.
outbox-relay polls the table and finds unpublished rows.
It publishes the row to Redpanda, then marks it “published”.
If the publish fails, the row stays “unpublished”; the relay retries.

Result: the worst case isn’t “lost event”, it’s “duplicate event” (at-least-once delivery). A duplicate is a problem you can manage with idempotency on the consumer; a lost event you can’t manage at all.

Why Redpanda: Kafka wire-compatible, single binary, cheap to bring up in a Kind cluster, so CI tests a real event pipeline, not a mocked Kafka client.

Part II: What holds it together

6. Observability: one request’s story across eight services

In this chain, a request passes through Kong, Platform API, Temporal worker, the Python agent, the MCP server, the decision service, the decision engine, and the outbox relay.

Why OpenTelemetry + W3C TraceContext: instead of saying “it’s slow”, I wanted to open a single trace in Jaeger and see how long each step took. That’s why traceparent is carried in both HTTP calls and Redpanda record headers, so even the event pipeline doesn’t break trace continuity. Prometheus collects the metrics, Grafana shows them in one pane, and Kibana is now integrated for exploring logs alongside the traces.

Why this early: leaving observability for last is the same as never doing it.

7. Infrastructure: separating what manages from what executes

Here’s the architecture’s hardest decision.

What I chose: a Kubernetes Operator (Go, controller-runtime) and five CRDs: EnterpriseEnvironment, EnterpriseIntegration, DecisionModel, EnterpriseKnowledgeSource, and DecisionPolicy. Four have working reconcilers. DecisionPolicy is defined, but its reconciler isn’t written yet (see the end of the post).

Why CRDs: which model artifacts are pulled and warm in which environment, which integration is up in which environment, which knowledge source is being ingested: all of these are lifecycle problems. Where there’s a lifecycle, the desired-state/reconcile model is the right tool.

The reconcile loop, concretely: the operator watches the CRDs. When a DecisionModel names a model (like laya), it compares “desired” against reality: is that model pulled, are the relevant services up? If there’s a gap, it corrects it (deploys, spawns a Job), then waits again. The loop repeatedly closes the gap between “actual” and “desired”, and a reconciler can re-run at any moment by design, so idempotency is its core contract. Creating a Deployment twice is harmless. A workflow’s core contract is different: durable sequencing of steps that have side effects. That difference is why one is a CRD and the other isn’t.

Why controller-gen: hand-writing CRD YAML means the Go types and the YAML inevitably drift. It’s generated, never hand-written.

Why RunAgent, RunDecision, CreateWorkOrder are not CRDs: here’s the article’s entire thesis. Business processes belong to Temporal; infrastructure lifecycle belongs to Kubernetes. A system that mixes the two ends up hating both six months later. The operator doesn’t call the decision service, doesn’t touch Temporal, doesn’t run a workflow.

The same split applies to RAG ingestion: chunking and embedding documents is a recurring Job. The operator watches EnterpriseKnowledgeSource and spawns the ingestion Job. Not inside the workflow.

8. Verification: “it works” should cost something

This entire chain is tested end to end with Ginkgo/Gomega on a real Kind cluster. One scenario, 50 assertions, from signal ingestion to work order. What’s checked is the typed fields the decision service returns, the MCP tool-call records, the outbox rows in the database, and the event that lands in Redpanda, not the content of generated prose.

Why Kind: testing against a real Kubernetes API without cloud cost.

Local dev environment: one command, a real cluster, trusted TLS

How these steps come up on a laptop is part of the story too.

mise is a single command surface: it pins the tools (Go 1.26, Python, kubectl, kind, kustomize, sqlc, goose…) and wraps the workflow: mise run bootstrap, kind-create, build, deploy, e2e, demo. A fresh clone reaches a running demo without a “which Go version did you have?” conversation.
Kind brings up a real Kubernetes cluster and builds images into a local registry (localhost:5001). App images are built and pushed there; E2E runs on that real cluster. CI and local walk the same deployment path, so there’s no “works on my machine”.
Ingress + TLS + DNS: setup-kind-ingress.sh installs ingress-nginx, issues a real mkcert wildcard cert for *.optimus.local, and serves wildcard DNS via dnsmasq. Every service is reachable at its own address (https://grafana.optimus.local, https://jaeger.optimus.local…) with trusted TLS, no /etc/hosts hacks, and no cert exceptions. Adding a service is one Ingress.

Reproducible toolchain, a real cluster, production-shaped networking: the difference between “I ran it once” and “I engineered it” is these three.

Quick reference: which layer, which technology, why

All layers and components: the optimus architecture

Figure 2: The table below is this diagram’s “why” column: the reason each component is where it is.

Layer	Technology	Why I chose it
Ingress	Kong	TLS, rate limiting, S2S auth stay out of application code
Platform	Go 1.26	Concurrency, small binaries, same ecosystem as Kubernetes
Data access	pgx/v5 + sqlc + goose	Typed queries, no ORM magic, reproducible migrations
Storage	PostgreSQL 17 + pgvector	Relational + vector + outbox in one place, one transaction
Isolation	Row-Level Security	Tenant separation entrusted to the database, not the app
Workflow	Temporal	Durable execution, a 24h human-approval wait, compensation
Agents	Python (CI: 3.14) + Pydantic AI	Typed agent output; provider-agnostic abstraction
Tool access	MCP	Capability discovery, not authorization; per-system adapters
Decision	System One API + typed model (Laya via Ollaya, or hosted Jev)	Typed, bounded outputs; engine changes by configuration
Governance	policy_v1 (declarative, versioned)	Thresholds are config, not code; every decision records its version
Events	Transactional outbox + Redpanda	Row and event can’t diverge; Kafka-compatible, cheap in Kind
Infrastructure	Operator + 5 CRDs (controller-gen)	Lifecycle problems, not business processes
Observability	OTel + Jaeger + Prometheus + Grafana + Kibana	Unbroken trace across eight services, plus logs
Verification	Kind + Ginkgo/Gomega	Real cluster, end to end, structural assertions
Dev environment	mise + Kind + ingress-nginx + mkcert	One command surface; a real cluster; trusted wildcard TLS

What I deliberately didn’t build

This is not a production product; it’s a working architecture demo, and there’s no point hiding that:

What actually runs on the Kind cluster today

Figure 3: What actually runs on the Kind cluster today. Every green box is live. Note the decision engine in particular: an Ollaya mock server doing a deterministic forward pass, not a served model.

The enterprise systems are mocks. EAM/PLM/ERP/FSM/OI aren’t real vendor systems, but their interfaces are production-shaped.
The decision engine is a local stub. decision-service speaks the real System One API; the engine behind it is a local Go stub that simulates Laya (for zero-cost CI). The API is real.
The DecisionPolicy CRD has no reconciler yet. The CRD is defined and policy_v1 is versioned, declarative configuration, but managing policies in-cluster through the operator is still ahead of me.
NATS isn’t live yet. Its design and ADR are ready for agent telemetry; because it’s never a correctness carrier, I prioritized other layers first.
Apache Camel is in the planning stage. Designed for industrial OPC-UA/MQTT and legacy SOAP/RFC systems; today signals enter directly over REST.
Two tech debts are tracked: migrating to the official Go MCP SDK (TD-0001) and S3/MinIO-based document ingestion (TD-0002).

The target architecture

Figure 4: The target architecture — the full vision, not what’s running. Apache Camel for industrial and legacy protocols, NATS for agent telemetry, S3/MinIO for document ingestion, the official Go MCP SDK, and real vendor systems. The bullets above mark exactly where the line sits today.

I write these down because “production-grade” is easy to say; knowing where you stopped is harder.

Why this isn’t “over-engineered”: because that label assumes the goal was feature coverage. The goal of a PoC is different — put several architectural boundaries on a real Kubernetes cluster at once and watch them hold: infrastructure lifecycle in the operator, business process in Temporal, deterministic decision in the decision service, governance in policy, investigation in the agent. Breadth of components is the point only because each one marks a seam being tested. That’s also why the demo isn’t content with “the LLM answered”: it runs the chain end to end on Kind and asserts structural results — typed fields, MCP tool-call records, outbox rows, the event in Redpanda — rather than the prose a model happened to generate. A PoC that only proved the model talks would be both over-engineered and empty. Proving the seams is the contribution.

Closing

What this project taught me wasn’t the Kubernetes API. It was knowing which problem not to hand to which tool: the operator manages infrastructure, Temporal executes the work, the decision engine decides, the agent proposes. Every boundary’s rationale is written in an ADR. For a PoC, that set of deliberate non-assignments is the actual architectural deliverable — “everything as a CRD” and “let the LLM run the system” are both easy to build and both wrong.

I’m open to work, especially in these areas:

AI platform engineering and agentic integrations (MCP, tool management, agent governance)
Durable, multi-step workflows (Temporal, Saga, compensation patterns)
Distributed backend and platform engineering (Go, event-driven architecture, multi-tenant SaaS)
The Cloud Native + AI intersection: Kubernetes operators, CRD design, infrastructure automation