AI Agent Frameworks — Deep Best Practices Guide (LangGraph, PydanticAI, LlamaIndex)

LangGraph · PydanticAI · LlamaIndex (RAG)

A senior/principal-level reference for building production-grade agentic systems in Python. This guide focuses on transferable engineering concepts (state, durability, structured output, retrieval pipelines) rather than API syntax, because APIs change but these concepts don’t.

LangGraph — The Orchestration Backbone
PydanticAI — The Agent Layer
LlamaIndex — RAG as a Discipline
Cross-Framework Architecture Patterns
Final Checklist

1. LangGraph — The Orchestration Backbone

1.1 Why LangGraph Deserves the Biggest Investment

The core insight: the framework is disposable, the concepts are not. LangGraph happens to be a good current vehicle for learning:

State management
Durable execution
Checkpointing
Retries
Branching
Human-in-the-loop (HITL)
Multi-agent orchestration
Failure recovery

Even if you migrate to Temporal, AWS Step Functions, a custom actor system, or a future framework, these eight concepts remain the actual skill. Treat LangGraph as a teaching vehicle for durable, stateful distributed systems, not just “a way to call LLMs in a loop.”

1.2 State Management

Mental model: Your graph is a state machine. Every node is a pure(ish) function: f(state) -> partial_state_update. The graph engine merges updates via reducers.

Best Practices

Design your state schema first, before writing any node. Treat it like a database schema — get the shape right early, migrations are painful later.
Use TypedDict or Pydantic models for state, not raw dicts. You get IDE autocomplete, runtime validation (with Pydantic), and self-documentation.
Separate “public” state (visible to the user / other agents) from “scratch” state (internal reasoning, tool call intermediates). Don’t let internal debugging noise leak into what gets persisted/returned.
Use reducers deliberately. The default “last write wins” reducer is wrong for almost anything except scalars. For lists (e.g., messages, tool calls), use an add/append reducer. For counters, use accumulation. For sets of facts, use a merge/union reducer.
Keep state JSON-serializable. Anything you can’t cleanly serialize (open file handles, DB connections, raw exception objects) should NOT live in state — pass references/IDs instead and re-hydrate inside a node.
Version your state schema. When you add/remove/rename a field, old checkpoints in production will break unless you write a migration function. Treat this exactly like a DB schema migration.

from typing import Annotated, TypedDict
from operator import add

class AgentState(TypedDict):
    messages: Annotated[list, add]      # append-only history
    retries: Annotated[int, add]        # accumulate retry count
    plan: str | None                    # last-write-wins is fine here
    scratch: dict                       # internal-only, never returned to user

Anti-Patterns

Storing giant blobs (full documents, embeddings) directly in graph state — pass pointers/IDs to a store instead.
Mutating state in place inside a node instead of returning a delta. This breaks reducer semantics and makes debugging non-deterministic.
Using state as a dumping ground for “just in case I need it later” fields. Every field is a maintenance liability.

1.3 Durable Execution

Mental model: A long-running agent workflow should survive process restarts, deploys, and crashes exactly like a database transaction survives a server reboot.

Best Practices

Assume the process WILL die mid-execution. Design every node to be safely re-runnable (idempotent) or wrapped so re-running from the last checkpoint doesn’t duplicate side effects.
Push side effects (API calls, payments, sending emails) to the edges of nodes, and make the side-effect call itself idempotent (idempotency keys, dedupe by request ID) so a replay after a crash doesn’t double-charge a customer or double-send an email.
Distinguish “resumable” state from “ephemeral” state. Things like open socket connections cannot be checkpointed — only their intent (e.g., “call endpoint X with payload Y”) should be persisted, and the actual call re-executed on resume.
Think in terms of sagas. For multi-step workflows with external side effects, design compensating actions (undo steps) for each forward step, so failures mid-workflow can roll back cleanly instead of leaving inconsistent state.

Anti-Patterns

Treating durable execution as “just add retries.” Retries without idempotency = duplicated side effects.
Assuming your whole workflow runs in one long-lived Python process with no interruption. In production, it won’t.

1.4 Checkpointing

Mental model: A checkpoint is a snapshot of state + position-in-graph, persisted to durable storage (Postgres, Redis, SQLite for dev), that lets execution resume exactly where it left off.

Best Practices

Checkpoint after every node, not just at the end. The granularity of your checkpoints determines your blast radius on failure — checkpoint too coarsely and a crash re-does a lot of expensive work (e.g., LLM calls you already paid for).
Use a real persistent backend in production (Postgres/Redis), not the in-memory checkpointer — that’s for local dev/tests only.
Include a thread_id (conversation/session identifier) in every checkpoint key. This is what lets you resume the correct conversation instead of a random one.
Snapshot semantics matter: prefer immutable, append-only checkpoint logs over overwriting a single row. This gives you time-travel debugging (replay from any prior checkpoint) almost for free.
Periodically prune/archive old checkpoints — an unbounded checkpoint table will eventually become a performance and cost problem.

Design Pattern: Time-Travel Debugging

Because checkpoints are just persisted state snapshots, you can:

Load any historical checkpoint for a thread_id.
Fork execution from that point with a modified input.
Compare the new trajectory against the original.

This is invaluable for debugging “why did the agent do X” without re-running the entire multi-turn conversation from scratch.

1.5 Retries

Mental model: Not all failures are equal. A network timeout, a rate-limit error, a malformed LLM output, and a genuine business-logic error each need a different retry strategy.

Best Practices

Classify errors before deciding how to retry:
- Transient/infrastructure errors (timeouts, 5xx, rate limits) → exponential backoff + jitter, bounded max attempts.
- Malformed/invalid LLM output (schema validation failure) → retry with a repair prompt that includes the validation error, not a blind retry of the same prompt.
- Genuine business errors (e.g., “user not found”) → do NOT retry; surface immediately.
Cap retries with a circuit breaker. After N consecutive failures for a given tool/API, stop hammering it and escalate (to a human, to a fallback path, or fail the run cleanly).
Make retry counts part of state, not a local variable, so retries survive a process restart and don’t silently reset to zero.
Log/trace every retry with the reason. “Retried 3 times” is useless without knowing why each attempt failed — this is your primary debugging signal in production.

def call_llm_with_repair(state):
    try:
        result = llm.invoke(state["messages"])
        validated = OutputSchema.model_validate_json(result.content)
        return {"result": validated, "retries": 0}
    except ValidationError as e:
        if state["retries"] >= MAX_RETRIES:
            return {"error": "max_retries_exceeded", "retries": state["retries"]}
        repair_prompt = f"Your last output failed validation: {e}. Fix it and return valid JSON only."
        return {"messages": [repair_prompt], "retries": state["retries"] + 1}

1.6 Branching

Mental model: Real agent workflows are not linear pipelines — they’re graphs with conditional edges, parallel fan-out, and convergence points.

Best Practices

Model decisions as explicit conditional edges, not if/else buried inside a single mega-node. Explicit branches are visible in the graph visualization, debuggable, and testable in isolation.
Keep routing functions pure and side-effect-free. A router should only read state and return the next node name(s) — it shouldn’t call an LLM or mutate state itself (do that in a preceding node, then route on the result).
Use fan-out/fan-in for independent parallel subtasks (e.g., “research 3 sources in parallel, then merge”), and make sure your state reducers correctly merge concurrent updates from parallel branches.
Bound the branching factor. Unbounded dynamic fan-out (e.g., “spawn one branch per search result”) can explode cost/latency — cap it and make the cap configurable.
Design convergence nodes to handle partial failure — if 2 of 5 parallel branches fail, decide explicitly whether to proceed with partial results, retry only the failed branches, or fail the whole run.

Design Pattern: Supervisor Routing

A common, robust branching pattern: a “supervisor” node classifies the current state/intent and routes to one of several specialist nodes/subgraphs, which then return control back to the supervisor. This keeps routing logic centralized and auditable instead of scattered across many ad-hoc conditionals.

1.7 Human-in-the-Loop (HITL)

Mental model: Some decisions are too costly, ambiguous, or irreversible to fully automate. HITL is a first-class interrupt mechanism, not an afterthought bolted onto the UI layer.

Best Practices

Use interrupts at well-defined checkpoints, not by polling or sleeping. The graph should pause execution, persist state, and truly yield control back until a human responds — potentially minutes, hours, or days later.
Design the “resume” payload contract explicitly. What does a human’s approval/edit/rejection look like as a data structure? Define this schema up front (e.g., {"decision": "approve"} | {"decision": "edit", "new_value": ...} | {"decision": "reject", "reason": ...}).
Make HITL points configurable, not hardcoded. As trust in the agent grows, you’ll want to move from “approve every tool call” to “approve only high-risk actions” without a code rewrite — drive this from a policy/config, not scattered input() calls.
Always give the human enough context to decide — don’t just show the raw tool call; show the reasoning trace, the state diff, and the consequence of approval.
Log every human decision as part of the audit trail. This becomes training data for narrowing future automation and is often a compliance requirement.

Anti-Patterns

Blocking a synchronous web request while waiting on a human for an unbounded amount of time — always design HITL as async (persist, notify, resume later), never as a blocking call in a request/response cycle.
Hardcoding “always ask for confirmation” for every single action, which trains users to blindly click “approve” (approval fatigue) — be selective about which decisions truly need a human.

1.8 Multi-Agent Orchestration

Mental model: Multiple specialized agents (or subgraphs) collaborating is fundamentally a distributed systems problem: message passing, shared state, ownership boundaries, and failure isolation.

Best Practices

Give each sub-agent a narrow, well-defined responsibility and explicit input/output contract. A “does everything” agent is un-debuggable and un-testable.
Choose your topology deliberately:
- Supervisor/worker: one orchestrator routes to specialists — easiest to reason about and debug, good default.
- Peer-to-peer/swarm: agents hand off directly to each other — more flexible, much harder to debug and bound (watch for infinite hand-off loops).
- Hierarchical: supervisors of supervisors — needed for genuinely large systems, adds real operational complexity.
Isolate each sub-agent’s scratch state from shared state. Sub-agents should only read/write what’s explicitly in their contract, not the entire global state (prevents accidental coupling).
Put a hard cap on inter-agent hand-offs / turns to prevent infinite loops between agents that keep deferring to each other.
Design for partial failure. If one sub-agent fails, the orchestrator needs an explicit policy: retry that sub-agent, substitute a fallback, or fail the overall run — never let a silent exception in one sub-agent corrupt shared state for the rest.

Design Pattern: Contract-First Sub-Agents

Before writing any sub-agent’s prompt or logic, write its interface contract: exact input schema, exact output schema, and its failure modes. Treat every sub-agent like a microservice with an API contract — this is what actually makes multi-agent systems maintainable at scale.

1.9 Failure Recovery

Mental model: Failure recovery is what separates a demo from a production system. Assume nodes, tools, and models WILL fail; design for graceful degradation, not just the happy path.

Best Practices

Define an explicit failure taxonomy for your system: transient/retryable, degraded (partial success acceptable), and fatal (must halt and escalate). Route each differently.
Always have a fallback path. If your primary LLM/tool fails after retries, what’s the fallback? A cheaper/simpler model? A cached/stale response with a disclaimer? A human handoff? Decide this explicitly per node, not implicitly.
Make partial success a real, modeled outcome — not everything is binary success/failure. A research agent that got 3 of 5 sources should be able to proceed with a documented gap, rather than failing the entire run.
Persist enough context on failure to debug later — the full state at time of failure, the error, and the retry history. Treat this like a stack trace for a distributed system.
Alert on failure-rate anomalies, not individual failures. Individual transient failures are normal; a spike in failure rate for a given node/tool is the actionable signal.

2. PydanticAI — The Agent Layer

2.1 Why This Combination Works

If you’re already in Python + Pydantic, the value isn’t PydanticAI’s specific API — it’s that it forces you to think correctly about:

Tool design
Structured output
Dependency injection
Validation
Agent boundaries
Context management
Testing

These are the same concerns you’d have building any typed, testable agent layer — PydanticAI just gives you good defaults and guardrails for them.

2.2 Tool Design

Mental model: A tool is a typed function contract you’re handing to a non-deterministic caller (the LLM). Design it the way you’d design a public API for an untrusted, occasionally-confused client.

Best Practices

Every tool parameter should be strongly typed with Pydantic, including constraints (e.g., Field(ge=0, le=100), enums for closed sets of choices instead of free-text strings). This turns “the LLM passed a garbage value” into “the LLM gets a clear validation error it can self-correct from.”
Write docstrings for the LLM, not for other developers. The docstring IS the tool’s prompt — be explicit about when to use it, what it returns, and edge cases (“Returns an empty list if no results found, never raises for a valid query”).
Keep tools narrow and composable rather than one mega-tool with 15 optional parameters. An LLM reasons better about “search_flights” + “book_flight” than about one “travel_action” tool with a mode discriminator.
Make tools idempotent where possible, especially any tool with side effects — since the LLM may call it more than once (retries, re-planning), duplicate calls shouldn’t duplicate real-world effects.
Return structured, typed results from tools, not raw strings — the calling agent code and any downstream tool should get a Pydantic model back, not a string it has to re-parse.
Design tool errors to be self-correcting. When a tool raises, the error message becomes part of what the LLM sees next — write error messages that tell the model exactly what to fix.

from pydantic import BaseModel, Field
from pydantic_ai import Agent, RunContext

class FlightSearchParams(BaseModel):
    origin: str = Field(description="IATA airport code, e.g. 'JFK'")
    destination: str = Field(description="IATA airport code, e.g. 'LAX'")
    date: str = Field(description="ISO 8601 date, e.g. '2026-09-01'")
    max_results: int = Field(default=5, ge=1, le=20)

class Flight(BaseModel):
    flight_number: str
    price_usd: float
    departure_time: str

agent = Agent("openai:gpt-4o", deps_type=FlightAPI)

@agent.tool
def search_flights(ctx: RunContext[FlightAPI], params: FlightSearchParams) -> list[Flight]:
    """Search available flights between two airports on a given date.
    Returns an empty list if none are found. Never raises for a valid,
    well-formed search — only raises on malformed IATA codes."""
    return ctx.deps.search(params)

Anti-Patterns

Tools that return giant unstructured text blobs the model has to “parse” mentally — that’s re-introducing the exact fragility structured tools were meant to remove.
Tools with ambiguous names/descriptions that overlap in purpose (get_data vs fetch_data vs retrieve_data) — the model will pick inconsistently.

2.3 Structured Output

Mental model: Treat the LLM’s final answer like an API response, not like chat text — validate it the same way you’d validate any external input, because it is one.

Best Practices

Always define an explicit Pydantic output schema for the agent’s final answer, even when the “product” feels conversational — you can render a structured object as prose in the UI layer, but internally you want guarantees.
Use nested models + enums to close down ambiguity. status: Literal["approved", "rejected", "needs_review"] is far safer than status: str.
Add field-level descriptions — these become part of the schema the model sees and materially improve output quality, they aren’t just documentation.
Validate confidence/uncertainty explicitly if relevant — e.g., include a confidence: float or needs_human_review: bool field so downstream logic (like HITL routing) has something concrete to branch on, instead of trying to sniff hedging language out of free text.
Handle validation failures with automatic re-ask, bounded by a retry limit (PydanticAI supports this natively via result retries) — don’t let a single malformed output crash the whole run.
Don’t over-nest. Deeply nested output schemas (5+ levels) are harder for models to fill in correctly and harder for you to validate/debug — flatten where you reasonably can.

from typing import Literal
from pydantic import BaseModel, Field

class TicketTriage(BaseModel):
    category: Literal["billing", "technical", "account", "other"]
    priority: Literal["low", "medium", "high", "urgent"]
    summary: str = Field(description="One-sentence summary of the issue")
    needs_human_review: bool = Field(description="True if ambiguous or high-risk")
    confidence: float = Field(ge=0.0, le=1.0)

triage_agent = Agent("openai:gpt-4o", result_type=TicketTriage, retries=2)

2.4 Dependency Injection

Mental model: Your agent’s behavior is code (deterministic), your agent’s reasoning is the LLM (non-deterministic). Dependency injection is how you keep the deterministic parts testable, swappable, and free of hidden global state.

Best Practices

Pass all external dependencies (DB connections, API clients, config, the current user/tenant) through a typed deps object, never through module-level globals or environment variable lookups scattered across tool functions.
Make deps a Pydantic model or a plain dataclass with clear types — this is your agent’s “environment,” and it should be as explicit as a function’s parameter list.
Inject fakes/mocks for dependencies in tests — this is the entire point of DI: your tools call ctx.deps.db.query(...), and in tests you swap in an in-memory fake, never touching a real database or real API.
Scope dependencies correctly — per-request (a specific user’s auth token), per-session (conversation memory store), and per-process (a shared connection pool) are different lifetimes; don’t conflate them into one global bag.
Never put secrets directly in deps as plain strings if avoidable — inject a client that’s already authenticated, rather than passing raw API keys through agent state where they might get logged or serialized into a checkpoint.

from dataclasses import dataclass

@dataclass
class AppDeps:
    db: DatabaseClient
    payments_api: PaymentsClient
    current_user_id: str

agent = Agent("openai:gpt-4o", deps_type=AppDeps)

@agent.tool
def get_account_balance(ctx: RunContext[AppDeps]) -> float:
    """Return the current user's account balance."""
    return ctx.deps.db.get_balance(ctx.deps.current_user_id)

2.5 Validation

Mental model: Validation is not just “does the JSON parse” — it’s your primary defense against silent semantic drift between what the model thinks it returned and what your system needs.

Best Practices

Validate at every boundary: tool inputs, tool outputs, and final agent output — not just the final answer.
Prefer semantic validators over type-only validators where correctness matters. A @field_validator that checks a date isn’t in the past, or that a total matches the sum of line items, catches errors type-checking alone never will.
Fail loudly and specifically. A validation error message should tell the model (or the developer) exactly which field and why, not just “invalid input.”
Distinguish “recoverable via re-ask” validation errors from “hard” ones. A missing optional field can prompt a re-ask; a security-relevant field failing validation (e.g., an amount that’s negative for a payment) should hard-fail, not silently retry into a corner case.
Unit test your Pydantic models independently of the agent. Validators should have their own test suite — you’re testing business rules, not LLM behavior, at this layer.

2.6 Agent Boundaries

Mental model: An “agent” is not a magic all-powerful actor — it’s a bounded component with a defined scope of authority, exactly like a microservice.

Best Practices

Define explicitly what an agent is allowed to decide vs. what must be hardcoded/deterministic. E.g., an agent can decide which refund tier applies, but the actual refund amount calculation should be deterministic code, not something the LLM “computes” in free text.
Never let an agent’s tools exceed its stated authority — if an agent is scoped to “customer support triage,” don’t give it a tool that can delete a production database record, even if it would technically work; scope tools to match the agent’s actual job.
Draw a hard line between “agent decides” and “agent executes.” Keep irreversible/high-stakes execution (payments, deletions, sending real emails) behind an explicit approval or deterministic guard rail, not solely behind LLM judgment.
Document each agent’s boundary like an API contract: what it will do, what it will refuse, what it hands off, and to whom.
Compose narrow agents rather than building one broad agent — this is the same principle as microservices vs. a monolith, and it applies for the same reasons (testability, replaceability, blast-radius containment).

2.7 Context Management

Mental model: Context window is a scarce, expensive resource — treat what goes into it with the same discipline you’d apply to what goes into a function’s arguments, not as an infinite scratchpad.

Best Practices

Summarize/compact conversation history rather than letting it grow unbounded — implement a policy (e.g., keep last N turns verbatim + a running summary of everything older) rather than truncating blindly or sending everything forever.
Don’t put entire tool outputs into the context verbatim if they’re large — extract/summarize the relevant parts, and only expand on demand (e.g., the model can call a “get_full_document” tool if it decides it actually needs the whole thing).
Separate system/instruction context from dynamic per-turn context clearly, so your prompt structure stays stable and cache-friendly (important for prompt caching cost savings).
Track token budgets explicitly — know roughly how many tokens your system prompt + tools + recent history consume before you even get to the current turn, and leave headroom.
Be deliberate about what dependency/state data actually needs to be in the prompt vs. just available to tools via deps. Not everything in deps should be serialized into context — most of it should just be accessible to tool code.

2.8 Testing

Mental model: You’re testing two different things — deterministic code (tools, validators, routing) and non-deterministic behavior (the LLM’s choices) — and they need different testing strategies.

Best Practices

Unit test tools and validators in complete isolation from the LLM — these are plain Python functions/Pydantic models; test them exactly like any other code, with 100% determinism expected.
Test agent behavior with recorded/replayed LLM responses (cassettes) for fast, deterministic CI — don’t hit a real LLM API on every CI run; that’s slow, costly, and flaky.
Maintain a small suite of real, live-model “eval” tests separately from your fast CI suite — these check that the actual model still behaves as expected (prompt regressions, model version changes), run less frequently (nightly/on-demand), and tolerate some non-determinism (assert on properties/schemas, not exact strings).
Test the DI boundary directly — inject fake deps and assert your tools behave correctly against a fully controlled fake environment.
Test validation failure paths explicitly — write tests that feed malformed model output through your Pydantic schema and confirm the re-ask/retry/hard-fail behavior triggers as designed.
Assert on structure, not exact wording, for LLM-generated free text — check schema conformance, field constraints, and key facts, not string equality.

3. LlamaIndex — RAG as a Discipline

3.1 Why Learn RAG as a Pipeline, Not a Library

You don’t need to commit to LlamaIndex as a permanent dependency. What you actually need is to deeply understand the pipeline stages below — once you do, moving between LlamaIndex, LangChain, Haystack, or a hand-rolled RAG stack becomes trivial, because it’s the same seven stages everywhere.

Document ingestion
       ↓
Chunking
       ↓
Embedding
       ↓
Index
       ↓
Retrieval
       ↓
Reranking
       ↓
Context construction
       ↓
LLM

3.2 Document Ingestion

Mental model: Garbage in, garbage retrieved. This stage is underrated and is where most real-world RAG quality is won or lost.

Best Practices

Preserve structure during parsing (headings, tables, lists, page numbers) — don’t flatten a PDF into one undifferentiated text blob; structure is retrieval signal.
Extract and separately handle tables and images — naive text extraction mangles tabular data; consider a dedicated table parser or vision model for scanned/complex documents.
Attach rich metadata at ingestion time: source URI, section/page number, author, date, document version, access-control tags. This metadata becomes your filtering/reranking signal later — it’s much harder to backfill than to capture at ingestion.
Deduplicate near-identical documents before indexing — duplicate content pollutes retrieval results and wastes embedding cost.
Version your ingestion pipeline itself. If you change your parser or chunking logic, you generally need to re-ingest and re-embed everything — treat this like a migration, not a hot patch.
Handle ingestion failures explicitly (corrupted files, unsupported formats, OCR failures) — log and quarantine failed documents rather than silently dropping them.

3.3 Chunking

Mental model: A chunk is the atomic unit of retrieval — its size and boundaries directly determine what context the LLM eventually sees. This is arguably the single highest-leverage tuning knob in RAG.

Best Practices

Chunk along semantic boundaries, not fixed character counts — split on headings, paragraphs, or sentence groups rather than blindly slicing every N characters, which routinely cuts a sentence or table row in half.
Use overlap between chunks (e.g., 10-20% of chunk size) so information near a boundary isn’t lost from either chunk’s context.
Tune chunk size to your retrieval + generation task, not a universal default — smaller chunks (100-300 tokens) improve retrieval precision for fact lookup; larger chunks (500-1000+ tokens) preserve more context for synthesis-heavy answers. Test both.
Consider hierarchical/parent-child chunking: embed and search over small chunks for precision, but retrieve the larger parent chunk/section for the LLM’s context — best of both worlds.
Preserve chunk-to-source traceability — every chunk needs a stable ID back to its source document/section for citations and for debugging bad retrievals.
Re-chunk when your embedding model changes — chunk size sweet-spots are somewhat model-dependent (context length, how the model was trained to represent text).

3.4 Embedding

Mental model: An embedding model is a lossy compression function into a fixed-size vector — its quality bounds your entire retrieval ceiling; no amount of downstream reranking fully compensates for a bad embedding.

Best Practices

Match the embedding model to your domain. General-purpose embedding models underperform on highly specialized domains (legal, medical, code) — evaluate domain-specific or fine-tuned embedding models where retrieval quality matters.
Never mix embedding models within one index without re-embedding everything — vectors from different models are not comparable, and silently mixing them corrupts similarity search results.
Cache embeddings — re-embedding unchanged documents on every pipeline run is wasted cost; hash content and skip unchanged chunks.
Batch embedding calls for throughput and cost efficiency rather than embedding one chunk at a time.
Track embedding model + version as index metadata so you know exactly when a full re-embed is required (e.g., after upgrading to a new model version).
Consider asymmetric embedding strategies (different encoding for queries vs. documents) if your embedding provider supports it — some models are specifically trained for this and it measurably improves retrieval.

3.5 Index

Mental model: The index is your retrieval data structure — the tradeoff space here is speed vs. recall vs. cost vs. update-flexibility, and different index types make very different tradeoffs.

Best Practices

Pick the index type deliberately: flat/brute-force for small corpora (exact, simplest, slow at scale), HNSW/graph-based for approximate nearest-neighbor at scale (fast, tunable recall), IVF for very large corpora with acceptable recall tradeoffs.
Design for incremental updates, not just full rebuilds — a production knowledge base changes continuously; your index strategy needs an “add/update/delete single document” path, not just “rebuild everything nightly,” unless your corpus is genuinely static.
Combine dense (embedding) and sparse (keyword/BM25) indexes — hybrid search — dense retrieval alone often misses exact-match needs (product SKUs, error codes, proper nouns); hybrid search consistently outperforms either alone in practice.
Store metadata alongside vectors for filtering (date ranges, document type, access control) — pre-filtering by metadata before/during vector search is usually far more effective than trying to encode all of that into the embedding itself.
Plan your index’s access-control model up front — retrieval-time permission filtering (so a user never gets chunks from documents they can’t access) is a security requirement, not a nice-to-have, and it’s much harder to retrofit.

3.6 Retrieval

Mental model: Retrieval is a search-relevance problem, and you should apply the same rigor here that a search engineer would — this is not “just call .similarity_search() and move on.”

Best Practices

Tune top_k deliberately — too few risks missing relevant context, too many dilutes the context window with noise (and increases cost/latency); this is a per-use-case tuning parameter, not a universal constant.
Use hybrid retrieval (dense + sparse) as your default, not pure vector search, for most real-world corpora.
Add metadata filters at retrieval time (date, source, access rights) rather than relying on the embedding alone to encode these constraints.
Consider query transformation techniques: query expansion, HyDE (hypothetical document embeddings), or multi-query retrieval (generate several rephrasings of the user’s question, retrieve for each, merge results) — these materially improve recall for ambiguous or under-specified queries.
Evaluate retrieval independently from generation — measure retrieval precision/recall (does the right chunk show up in top-k?) as its own metric, separate from whether the final LLM answer “looks good,” because a good LLM can mask bad retrieval and a bad LLM can waste great retrieval.
Log every retrieval (query, retrieved chunk IDs, scores) for offline evaluation and debugging — you cannot improve what you don’t measure.

3.7 Reranking

Mental model: Initial retrieval optimizes for recall at scale (fast, approximate, over the whole corpus); reranking optimizes for precision on a small candidate set (slow, exact, only on the top-k) — use each for what it’s good at.

Best Practices

Always retrieve more than you need, then rerank down — e.g., retrieve top-50 with a fast method, rerank to top-5 with a more expensive cross-encoder model, rather than trying to get the fast method to be precise directly.
Use cross-encoder rerankers, not bi-encoder similarity, for the reranking stage — cross-encoders jointly attend to query and document and are meaningfully more accurate at relevance judgment, at the cost of being too slow to run over an entire corpus (hence: only on the small candidate set).
Consider reranking on multiple signals, not just semantic relevance — recency, source authority/trust, and metadata match can all be blended into a final ranking score depending on your use case.
Measure whether reranking actually helps for YOUR corpus/query distribution — it’s not universally a large win; benchmark before assuming it’s worth the added latency/cost.

3.8 Context Construction

Mental model: This is prompt engineering applied to retrieved content — how you assemble retrieved chunks into the LLM’s context materially affects answer quality, independent of retrieval quality itself.

Best Practices

Order matters — many models attend more reliably to the beginning and end of long contexts (“lost in the middle” effect); put your most relevant chunk(s) first and/or last, not buried in the center.
Always include source attribution inline with each chunk (e.g., [Source: policy_doc.pdf, p.12]) so the LLM can cite sources and so you can trace hallucinations back to a specific gap or error in retrieval.
De-duplicate near-identical retrieved chunks before constructing context — redundant chunks waste context budget and can reinforce a wrong answer if one source has an error that gets repeated.
Explicitly instruct the model on how to handle insufficient context (“If the context doesn’t contain the answer, say so — do not guess”) — this is one of the highest-leverage single interventions against hallucination in RAG systems.
Keep the retrieved-context section structurally distinct from the user’s question and system instructions (clear delimiters/headers) so the model doesn’t confuse retrieved content with instructions (this also matters for prompt-injection resistance, since retrieved documents are, from a trust perspective, untrusted user-adjacent input).

3.9 LLM (Generation)

Mental model: By the time content reaches the LLM, most of the quality ceiling has already been set by ingestion → chunking → embedding → retrieval → reranking → context construction. The generation step is where remaining errors get exposed, not usually where they’re created.

Best Practices

Instruct the model to ground every claim in the provided context and to explicitly flag when it’s extrapolating beyond it.
Ask for citations as a structured part of the output (e.g., a sources: list[str] field), not just prose citations — this is both more useful downstream and easier to verify programmatically.
Evaluate the full pipeline end-to-end, not just generation quality — use metrics like faithfulness (is the answer supported by the retrieved context?) and answer relevance, separately from retrieval metrics, so you can localize where a bad answer actually came from.
Iterate on the earlier pipeline stages first when quality is poor — a common mistake is prompt-engineering harder at the generation stage to compensate for bad retrieval; fix retrieval, don’t paper over it with prompting.

4. Cross-Framework Architecture Patterns

These three learning tracks compose naturally in a production system:

LangGraph as the outer orchestration layer: manages the overall workflow, state, checkpoints, HITL interrupts, and multi-agent routing.
PydanticAI as the agent/tool layer inside individual LangGraph nodes: each node that needs to “reason” wraps a PydanticAI agent with typed tools, typed output, and DI for its dependencies.
LlamaIndex-style RAG pipeline as a tool/service that any agent can call: expose retrieval as a typed tool (search_knowledge_base(query: str) -> list[RetrievedChunk]) that a PydanticAI agent calls, orchestrated by LangGraph when the workflow needs to branch on retrieval results (e.g., “insufficient context found → escalate to human”).

LangGraph (workflow, state, durability, HITL)
   └── Node: "answer_question"
         └── PydanticAI Agent (typed tools, structured output)
                └── Tool: search_knowledge_base()
                       └── RAG pipeline (ingest → chunk → embed → index →
                                          retrieve → rerank → context → LLM)

General Cross-Cutting Best Practices

Observability first. Instrument every layer (graph node entry/exit, tool calls, retrieval queries, LLM calls) with structured tracing (e.g., OpenTelemetry) before you need to debug a production incident, not after.
Cost tracking per component. Know your per-request cost breakdown across LLM calls, embedding calls, and reranking calls — this is what lets you make informed tradeoffs (e.g., “is reranking worth its added cost for this use case?”).
Version everything that affects output determinism: prompts, model versions, embedding model versions, chunking parameters. Treat prompt/config changes with the same rigor as code changes (PRs, review, changelogs) — they affect production behavior just as much as code does.
Design for evaluation from day one. Build a small, curated evaluation dataset (queries + expected properties of good answers) early, and re-run it on every meaningful change to prompts, models, or pipeline stages — this is your regression test suite for a fundamentally non-deterministic system.

5. Final Checklist

LangGraph

State schema is typed and versioned
Reducers are chosen deliberately per field
Checkpointer is a durable backend in production
Retries classify error types and are capped with backoff
Branching uses explicit, pure routing functions
HITL interrupts are async, with a defined resume contract
Multi-agent contracts are documented like microservice APIs
Failure recovery has explicit fallback paths and partial-success handling

PydanticAI

Tools have strongly typed, constrained parameters and clear docstrings
Output schemas are explicit, flat where possible, with field descriptions
Dependencies are injected via a typed deps object, never globals
Validation happens at every boundary, with recoverable vs. hard failures distinguished
Agent authority/scope is documented and enforced via tool availability
Context window usage is actively managed (summarization, selective inclusion)
Tests split cleanly into deterministic unit tests and periodic live-model evals

LlamaIndex / RAG

Ingestion preserves structure and captures rich metadata
Chunking follows semantic boundaries with deliberate overlap and size
Embedding model matches domain and is versioned
Index supports hybrid (dense + sparse) search and incremental updates
Retrieval is tuned and evaluated independently of generation
Reranking is benchmarked, not assumed to help
Context construction includes source attribution and “don’t know” instructions
Full pipeline is evaluated end-to-end with faithfulness/relevance metrics

LangGraph · PydanticAI · LlamaIndex (RAG)

Table of Contents

1. LangGraph — The Orchestration Backbone

1.1 Why LangGraph Deserves the Biggest Investment

1.2 State Management

Best Practices

Anti-Patterns

1.3 Durable Execution

Best Practices

Anti-Patterns

1.4 Checkpointing

Best Practices

Design Pattern: Time-Travel Debugging

1.5 Retries

Best Practices

1.6 Branching

Best Practices

Design Pattern: Supervisor Routing

1.7 Human-in-the-Loop (HITL)

Best Practices

Anti-Patterns

1.8 Multi-Agent Orchestration

Best Practices

Design Pattern: Contract-First Sub-Agents

1.9 Failure Recovery

Best Practices

2. PydanticAI — The Agent Layer

2.1 Why This Combination Works

2.2 Tool Design

Best Practices

Anti-Patterns

2.3 Structured Output

Best Practices

2.4 Dependency Injection

Best Practices

2.5 Validation

Best Practices

2.6 Agent Boundaries

Best Practices

2.7 Context Management

Best Practices

2.8 Testing

Best Practices

3. LlamaIndex — RAG as a Discipline

3.1 Why Learn RAG as a Pipeline, Not a Library

3.2 Document Ingestion

Best Practices

3.3 Chunking

Best Practices

3.4 Embedding

Best Practices

3.5 Index

Best Practices

3.6 Retrieval

Best Practices

3.7 Reranking

Best Practices

3.8 Context Construction

Best Practices

3.9 LLM (Generation)

Best Practices

4. Cross-Framework Architecture Patterns

General Cross-Cutting Best Practices

5. Final Checklist