LLM-Based Agentic Architecture

1. Executive Summary

LLM-based multi-agent systems have moved from research prototypes to production infrastructure. A well-designed agent system decomposes complex, multi-step goals into specialized sub-tasks handled by purpose-built agents, each with a focused role, bounded authority, and clear communication contract. The result is a system that can tackle tasks no single model call could accomplish reliably: long-horizon research, iterative code generation, multi-source data synthesis, and any workflow requiring memory, tool use, and adaptive decision-making.

This document defines a production-ready architecture for deploying LLM-based multi-agent systems on Google Kubernetes Engine (GKE). It covers the full engineering stack: prompt construction, context window management, the agent harness, memory systems, orchestration patterns, reliability controls, security, observability, and the framework landscape as of mid-2026. It is framework-agnostic in principle but references specific open-source frameworks — including OpenClaw, DeerFlow, LangGraph, AutoGen, CrewAI, and Nanobots as design references.

Design Goals

• Reliability over capability: A system that consistently delivers correct, verifiable outputs at 95% accuracy is more valuable than one that occasionally achieves 99% but fails unpredictably.
• Harness-first thinking: The agent harness — not the underlying model — is the binding constraint on production agent quality. Model selection matters less than how the system wraps, validates, and recovers from model calls.
• Bounded autonomy: Agents operate within explicitly declared authority levels. No agent takes consequential action without either deterministic authorization logic or human approval.
• Cost-conscious scaling: Tiered model routing, prompt caching, and context compression reduce inference costs by 60–80% compared to naïvely using frontier models for all tasks.
• Full auditability: Every agent decision — reasoning trace, retrieved context, tool calls, outputs, validation results — is logged immutably and indexed for replay.

2. The 2026 LLM Agent Paradigm

The discipline of building reliable LLM agents has passed through three distinct paradigm shifts since 2022. Understanding this evolution is essential for making the right architectural investments.

The Agent as a System

A single LLM agent is best understood not as “a model with a prompt” but as a closed-loop reasoning system composed of five interacting subsystems:

• Perception: What the agent receives — user input, tool results, retrieved memory, inter-agent messages.
• Context assembly: How the agent’s working context is constructed from perception inputs, constrained by the token budget.
• Reasoning: The LLM inference call that produces structured outputs, tool calls, or sub-task delegations.
• Action: Tool execution, state mutation, or message dispatch — always mediated by the harness, never directly by the model.
• Memory: What persists across turns and sessions — short-term context, long-term knowledge, episodic decision history.

Agent quality degrades when any of these subsystems is under-engineered. The most common failure mode is treating context assembly and harness as afterthoughts while over-investing in prompt wording.

3. Agent Design Patterns & Taxonomy

Rather than designing agents around specific domain tasks, production systems should be built from a small vocabulary of generic agent archetypes. Domain specialization is delivered through the agent’s system prompt, knowledge base, and tool set — not through bespoke code per use case.

Core Agent Archetypes

Authority Levels

Authority levels are enforced by the harness, not the prompt. A prompt-only authority declaration can be overridden by prompt injection or model drift; harness-enforced authority cannot.

• Fully Autonomous: Output is delivered or action is taken directly. Reserved for low-risk, reversible operations within a pre-authorized scope (e.g., reading data, sending a draft message for review).
• Human-in-Loop Required: The agent produces a decision and reasoning trace; a human must explicitly approve before any action is taken. Applied to all irreversible or high-impact actions.
• Advisory Only: Agent output feeds a human dashboard or downstream system but is never acted upon autonomously. The agent has no tool access that mutates state.

Coordination Patterns

Inter-Agent Communication Contract

Regardless of coordination pattern, every inter-agent message must carry a structured envelope with: originating agent identity, correlation/trace ID, the specific question or sub-task being delegated, the accumulated decision context so far, a confidence score, and a time-to-live. The harness enforces a maximum delegation depth (typically 2–3 hops) before escalating to the Coordinator, preventing unbounded recursive delegation.

When to Use Multi-Agent (and When Not To)

Staged Evolution (Recommended)

1. Stage 1 — Single agent + tools: One agent with RAG, memory, tool calling, and a workflow engine. Handles the majority of enterprise use cases. Low operational burden, easy to audit, straightforward to debug.
2. Stage 2 — Add a Critic/Validator agent: When output quality concerns justify it, add a separate validation pass. This single addition captures most of the reliability benefit of multi-agent architectures at a fraction of the coordination overhead.
3. Stage 3 — Add Planner + parallel Executors: Only when tasks are reliably too large for one context window, have independently parallelizable sub-tasks, or require distinct access permission scopes per sub-task.
4. Stage 4 — Full multi-agent system: For large-scale autonomous workflows where Stages 1–3 are demonstrably insufficient. This document’s full architecture applies at this stage.

Decision Framework

4. Prompt Engineering

Prompt engineering is the deliberate design of the instructions, constraints, and examples that constitute an agent’s behavioral specification. While the context window is broader than the prompt alone, the prompt provides the agent’s stable identity — the constitution that governs how it interprets every piece of context it receives.

System Prompt Architecture

All agent system prompts follow a four-block hierarchy with clearly marked delimiters. This structure is consistent across all agent types — only the content of each block changes per agent. User input and retrieved context are never placed in the system prompt; they arrive in separate, clearly labeled turns or context blocks, preventing the model from conflating instructions with data.

• Role block: Declares the agent’s identity, single primary responsibility, and explicit scope boundary. Instructs the agent to refuse requests outside its scope rather than attempt them.
• Rules block: Contains absolute, non-overridable constraints — behaviors that must hold regardless of any instruction in the conversation, including user input. Examples: “always cite sources for factual claims,” “never take irreversible actions without explicit approval,” “when uncertain, escalate rather than fabricate.”
• Context block: Populated at runtime by the retrieval pipeline. Contains relevant knowledge base chunks, retrieved documents, few-shot examples, and any agent-specific state. This block is dynamic and varies per request.
• Task block: Specifies the current request and the required structured output format, including schema constraints the model must satisfy.

Few-Shot Strategy

Few-shot examples dramatically improve structured output quality and constraint adherence. The key design decision is whether examples are static (hardcoded in the system prompt) or dynamic (retrieved from the knowledge base at runtime).

• Dynamic retrieval is strongly preferred for production systems. Examples are stored in the knowledge base and retrieved by semantic similarity to the current request. This allows the example pool to be updated, curated, and improved without a deployment.
• Negative examples are as important as positive ones. Show the model what a bad or out-of-scope response looks like, paired with the correct refusal or escalation.
• “I don’t know” examples are mandatory for any agent operating in a domain with bounded knowledge. The model must see examples where the correct answer is explicit escalation or acknowledgment of uncertainty — not a fabricated response.
• Format examples as structured reasoning traces (chain-of-thought), not just input/output pairs. This trains the model to show its work, which is essential for validation and audit.

Prompt Versioning & Governance

System prompts are first-class software artifacts. They must be version-controlled, reviewed, tested, and deployed with the same rigor as code.

• Prompts are stored in a Git-based registry as versioned, tagged files — reviewed via pull request by the AI platform team before merging.
• Each deployed agent references a specific prompt version. The harness rejects agent startup if the referenced prompt version is not in the approved registry.
• Changes to the rules block require a higher-level approval gate than changes to the task or context blocks.
• New prompt versions are validated against the full regression test suite before promotion to production.
• Deployed prompts are loaded via configuration at agent startup; hot-reload is supported without pod restarts for low-risk updates.

Prompt Injection Mitigation

5. Context Engineering & Window Management

Context engineering is the discipline of deciding what information to place in an agent’s context window, how much space to allocate to each category, how to compress when the budget is exceeded, and how to keep the most relevant information at the top of the window where models attend most strongly.

Research consistently shows that context quality — the relevance and density of information in the window — has a larger impact on output quality than prompt wording. Filling the window with low-relevance retrieved chunks actively degrades performance. Context engineering is therefore not a minor optimization; it is central to agent reliability.

Context Window as a Managed Resource

Every agent’s context window is divided into explicit, budget-controlled slots. The prompt assembly layer enforces these budgets before each LLM call — never relying on the model to manage its own attention.

Context Assembly Pipeline

Before each LLM call, the context assembly pipeline runs in this order:

1. Query analysis: Extract intent, entities, and required knowledge domains from the incoming request.
2. Parallel retrieval: Simultaneously execute sparse keyword search (BM25) and dense embedding search over the knowledge base. Run graph queries for entity relationships where applicable.
3. Re-ranking: Apply Reciprocal Rank Fusion (RRF) to merge sparse and dense results. Apply a domain-specific re-ranker model if available. Select the top-K chunks by relevance score.
4. History compression: If conversation history exceeds the allocated slot, run a lightweight summarization pass (using a low-cost model) to condense older turns.
5. Slot assembly: Fill each slot in priority order: rules → retrieved context → compressed history → current request. Enforce byte-level token counts per slot.
6. Overflow handling: If assembled context still exceeds the model’s window after compression, drop lowest-scoring retrieved chunks first, then older history. If after two rounds the window is still exceeded, reject the request and escalate — never silently truncate.

Retrieval Quality

• Embedding model selection: Choose an embedding model trained on content similar to the target domain. General-purpose embeddings perform well for broad tasks; domain-specific fine-tuned embeddings improve recall on specialized corpora.
• Chunk sizing: Chunks of 256–512 tokens generally outperform longer chunks for retrieval precision. Long documents should be chunked with sentence-boundary awareness and overlapping windows to avoid splitting reasoning context.
• Query expansion: A lightweight pre-retrieval step generates 2–3 query variants (paraphrases, related terms) and retrieves against all variants. Dramatically improves recall for ambiguous or terse user queries.
• Contextual compression: After retrieval, a compression pass extracts only the sentences from each chunk that are directly relevant to the query, reducing noise in the context window without losing key information.

Long-Running Agent Context Management

Agents that operate over extended periods (hours to days) face a unique challenge: context accumulated across thousands of turns cannot fit in any context window. Frameworks like OpenClaw address this with a Memory Dreaming pattern — during idle periods, a background consolidator model distills short-term conversation history into structured long-term memory entries, which are then retrievable via semantic search. This pattern is recommended for any agent designed to maintain coherence across sessions.

A complementary approach, used by Claude Code and similar developer agents, is a virtualized filesystem: the agent externalizes working state to structured artifacts (plan files, task lists, decision logs) that persist independently of the context window and are selectively loaded into context as needed.

6. Agent Harness Design

The agent harness is the execution layer that wraps every LLM inference call. It is responsible for context assembly, tool execution, output validation, retry logic, memory persistence, and observability instrumentation. In 2026, industry consensus has converged on a key principle: the harness is the binding constraint on agent reliability, not the underlying model.

Harness Responsibilities

The Agent Reasoning Loop

The harness implements the agent’s core reasoning loop — a multi-turn cycle that runs until the model produces a final text-only response (no further tool calls):

1. Assemble context from memory, retrieval pipeline, and current conversation state.
2. Call the LLM with the assembled context.
3. If the model returns a tool call: validate authorization → execute in sandbox → inject result → return to step 1.
4. If the model returns a delegation request: emit message to target agent queue → await response (async) or block (sync, with timeout) → inject result → return to step 1.
5. If the model returns a final response: run output validation → check confidence → persist to memory → deliver or escalate.
6. If any step produces an unrecoverable error after retries: emit escalation event → persist failure record → return error response.

Bounded, Deterministic Workflows

Production experience across the industry has strongly validated the Supervisor Pattern over unconstrained agent swarms. Key principles:

• Phase-gating: Complex workflows are divided into explicit phases (plan → gather → draft → review → deliver). No phase begins until the previous phase’s output has been validated. This prevents cascading errors from propagating through the entire workflow.
• Bounded loops: Every reasoning loop has a hard maximum iteration count enforced by the harness, not the prompt. Unconstrained loops are the most common cause of runaway cost and infinite regress.
• Explicit termination conditions: The harness defines success and failure criteria independently of the model. A task completes when the harness validates the output satisfies the success criteria — not when the model claims it is done.
• Prefer simpler graphs: Resist the temptation to model every possible execution path. A flat, sequential pipeline with explicit branching at known decision points is more reliable and easier to debug than a fully dynamic agent graph.

Model Routing

The harness selects the most cost-efficient model that meets the task’s requirements at runtime:

Tool Integration via MCP (Model Context Protocol)

The Model Context Protocol (MCP) has become the standard abstraction for connecting agents to external tools, data sources, and capabilities. Rather than each agent implementing bespoke tool integrations, MCP-compatible servers expose capabilities through a standardized interface that any MCP-aware harness can discover and invoke. This approach is now adopted by Anthropic’s Claude, OpenAI, Hermes Agent, Nanobots, and major enterprise platforms.

• MCP servers: Each tool category (database access, file system, web search, API calls, code execution) is exposed as a separate MCP server with a declared capability manifest. Agents discover available tools from the manifest at startup — no hardcoded integrations in the harness.
• MCP governance: Server manifests declare required permissions and data access scopes. The harness validates that the requesting agent’s allowlist includes the specific MCP server and capability before establishing a connection. Unsigned or unregistered MCP servers are rejected at the harness layer.
• MCP security model: Each MCP server runs in its own sandboxed process with declared network egress, filesystem access, and API scopes — enforced by the container runtime, not by the server’s own code. Tool results returned from MCP servers are sanitized (size limits, executable content stripped) before injection into agent context. A compromised MCP server cannot affect the harness process.
• MCP in production on GKE: Host each MCP server as a separate GKE Deployment with its own service account, resource limits, and network policy. This enables independent scaling, versioning, rollback, and security policy per tool category — far more operationally manageable than bundled tool integrations.

7. Memory Architecture

Agent memory is organized into three tiers by temporal scope, with distinct storage backends and access patterns for each. A well-designed memory system lets agents maintain coherence across long sessions and leverage accumulated knowledge without ever overflowing their context window.

Memory Tiers

Short-Term Memory Management

The default working context retains the most recent N turns, where N is determined by the agent’s allocated history slot in the token budget. When the history slot fills to 70% of capacity, the harness automatically triggers a summarization pass using a low-cost model, condensing older turns into a compact summary object that replaces them in context. This keeps the context window from filling with stale conversational scaffolding while preserving the essential reasoning thread.

Long-Term Memory & Memory Dreaming

Inspired by OpenClaw’s Memory Dreaming architecture, long-term memory consolidation runs as a background process during idle periods. A lightweight consolidator model reviews the agent’s recent short-term memory, extracts key facts, decisions, preferences, and outcomes, and writes structured entries to the long-term vector store. This mimics biological memory consolidation during sleep and enables agents operating over days or weeks to maintain coherence without ever exceeding their context window.

All consequential decisions — task completions, escalations, tool call outcomes, user approvals — are persisted as immutable records to an append-only decision log before the response is returned.

Storage Architecture

Knowledge Base Governance

Knowledge base entries are version-controlled with effective-from and effective-until markers. Superseded entries are retired (excluded from retrieval) but never deleted — they remain accessible for audit and historical context. All writes to the knowledge base require an authenticated, audited write path; agents have read-only access. Knowledge base entries are stored with content integrity hashes; retrieval re-verifies hashes before injecting content into agent context.

8. Multi-Agent Orchestration

Recommended Pattern: Hierarchical Supervisor

For the majority of enterprise agent deployments, the hierarchical supervisor pattern is the right choice. A Coordinator agent receives all inbound requests, classifies intent, and routes to the appropriate specialist agents. Specialists may sub-delegate once (at most twice) before returning results to the Coordinator. This produces a tree-shaped execution graph that is auditable, debuggable, and controllable.

Workflow Engine

Complex multi-step agent workflows (involving tool calls, human-in-loop waits, retries, and conditional branching) must be backed by a durable workflow engine — not simple async code. The workflow engine guarantees that workflows survive process crashes, network failures, and model errors by replaying from the last successful checkpoint. Apache Temporal is the recommended engine for GKE deployments; it supports:

• Durable, retryable activities with exponential backoff.
• Long-running human-approval activities that pause indefinitely until a human approval signal is received.
• Full execution history as an immutable audit trail — Temporal’s event history is the authoritative record of what each agent did and when.
• Deterministic workflow replay for debugging and investigation.

Communication Protocol

• Asynchronous (Kafka): Default for high-volume internal agent-to-agent messaging within the cluster. A dedicated topic per agent archetype, with consumer group scaling. Messages carry the full delegation envelope (Section 3). Dead-letter queues capture messages that fail processing after the maximum retry count.
• Synchronous (gRPC / HTTP): Used only when the Coordinator must block on a sub-agent result before proceeding — e.g., a fast lookup on the critical path. Hard timeout enforced; failure routes to the async fallback path.
• A2A Protocol: The interoperability layer for agent-to-agent communication across framework and vendor boundaries. See below.

Agent-to-Agent Interoperability: A2A Protocol

The Agent-to-Agent (A2A) Protocol is the Linux Foundation open standard for inter-agent communication. Originally developed by Google and donated to LF AI & Data, A2A v1.0 was released in early 2026 with over 150 supporting organizations: Google, Microsoft, AWS, Oracle, Databricks, Snowflake, Salesforce, ServiceNow, SAP, and IBM. It is natively integrated into Google ADK, Microsoft Agent Framework, LangGraph, Semantic Kernel, CrewAI, Amazon Bedrock AgentCore, and Azure AI Foundry. Financial services is a confirmed production adoption vertical.

A2A and MCP have distinct, complementary roles — one is not a replacement for the other:

Key A2A production properties relevant to this architecture:

• Signed Agent Cards: Each agent publishes a cryptographically signed capability manifest (Agent Card) declaring its skills, required permissions, and communication endpoint. Agents verify cards before accepting delegated tasks — preventing spoofing and capability misrepresentation.
• Cross-framework interoperability: A Coordinator built on Google ADK can delegate to a specialist agent built on LangGraph over A2A without bespoke integration code. This is significant for organizations that run multiple teams with different framework preferences, and for evolving the architecture incrementally.
• Enterprise security: A2A v1.0 includes authenticated agent discovery, enterprise-grade multi-tenancy, and VPC-compatible deployment — required properties for banking infrastructure.
• Kafka as A2A transport: A2A defines the protocol layer (message envelope, capability discovery, authentication). For high-volume internal messaging within the GKE cluster, Kafka remains the recommended transport implementation, carrying A2A-formatted messages.

Conflict Resolution

When two agents produce contradictory outputs on the same question, the Coordinator runs a synthesis step: both reasoning traces are presented to a high-tier model with the explicit instruction to identify the root cause of disagreement. If the disagreement is resolvable by evidence quality (one agent has higher-confidence retrieval), the higher-confidence output wins. If it is genuinely unresolvable, the decision is escalated to a human, and both outputs plus the synthesis analysis are presented together. Human resolutions are logged and fed back into the labeled training dataset.

State Consistency

Shared state (task status, accumulated results, entity state) lives exclusively in the authoritative data store. Agents do not cache shared state locally between decisions — they query the source of truth on each reasoning step. Optimistic concurrency control prevents race conditions when multiple agents act on the same entity simultaneously. The workflow engine checkpoints task progress so that any agent can be restarted and resume from the last known-good state.

9. Hallucination & Reliability Controls

Grounding Requirements

Any agent output that makes a factual claim must cite the source identifiers from the knowledge base that support that claim. The output validation layer rejects outputs that make factual assertions without citations before they are delivered. Agents are instructed in the rules block to use explicitly hedged language for all claims (“evidence suggests,” “based on retrieved data”) rather than asserting facts as certainties.

Confidence Scoring

A composite confidence score is computed from objective, externally verifiable signals. Any signal below its individual threshold contributes to a lower composite score; the composite falling below 0.75 triggers escalation:

• Citation coverage: Fraction of factual claims in the output supported by a retrieved knowledge base source identifier. Uncited claims directly lower this signal. This is the highest-weight signal.
• Retrieval quality: Mean similarity score of the top retrieved chunks against the query embedding. Low scores indicate the knowledge base may not cover the requested topic — the agent is reasoning from poor grounding.
• Validation pass rate: Whether the output passed the schema validator, constraint checker, and PII scanner on the first generation attempt. Re-generation required → lower score.
• Critic agreement: For decisions reviewed by a Critic agent, the degree of agreement between the Critic’s assessment and the Executor’s output. Disagreement lowers the score proportionally to its severity.
• Historical task accuracy: Rolling accuracy of this agent on the same task type over the past N runs, derived from the audit log and labeled decision corpus — not from the model’s own self-assessment.

When the composite score falls below the configured threshold (default: 0.75), the decision is automatically flagged for human escalation. Threshold values must be calibrated per task type against the labeled decision corpus; a single global threshold is insufficient for systems handling diverse task categories.

Fact Verification Pipeline

1. A lightweight extraction pass (using a low-cost model) identifies all factual claims in the agent’s output.
2. Each claim is checked against the knowledge base for supporting or contradicting evidence.
3. Claims with no supporting evidence: confidence is downgraded; a disclaimer is added; the output is flagged for review.
4. Claims directly contradicted by the knowledge base: output is rejected; the incident is logged as a hallucination event; the agent retries with additional grounding context injected.

Self-Correction Loop

After initial generation, a second verification pass checks the output for internal consistency: does the conclusion follow from the cited evidence? Do claims contradict each other? Detected inconsistencies trigger a regeneration request with the specific issues highlighted as additional context. After two failed regeneration attempts, the request is escalated — it is never silently delivered in an inconsistent state.

Human-in-Loop Triggers

• Blended confidence score below the configured threshold
• Any factual claim with no supporting knowledge base citation
• Any irreversible action (deletion, external message send, state mutation on critical data)
• Agent output flagged by the consistency verification pass
• Three consecutive decisions by the same agent with declining confidence (potential drift signal)
• Any decision in a use-case category explicitly marked “human-in-loop required” in the agent configuration

10. Security & Governance

OWASP Top 10 for LLM Applications (2025)

Model Governance

• Model Registry: All approved model versions are tracked in a governed registry with approval timestamps, approver identity, and declared use-case scope. Agents may only invoke models listed in the registry for their specific use case.
• Rollout Strategy: New model versions follow a canary deployment: 5% traffic → 25% → 100%, with automatic rollback triggered if accuracy or confidence metrics regress more than 5% from the baseline at any stage.
• Prompt Change Control: System prompt changes require a Git pull request reviewed by the AI platform team. Merged prompt versions are tagged, immutable, and must pass the full regression suite before production promotion.
• Fine-tuning Governance: Training datasets for any fine-tuning run are reviewed for quality, bias, and provenance before use. Fine-tuned models must complete the same canary rollout as any model change.

Data Isolation & PII

• Each agent runs with its own service identity; database access is scoped to the minimum required grants for that agent’s declared tool set. Row-level security enforces tenant scoping on every query.
• PII (names, contact details, credentials, any regulated personal data) is tokenized at system ingestion before any agent ever sees it. Detokenization occurs only in the delivery layer, outside the agent reasoning path.
• All agent outputs pass through a PII scanner before delivery or logging. Any detected PII is redacted and an alert is emitted.

Audit Trail

Every agent decision produces an immutable audit record persisted to an append-only log before the response is returned. This record includes the agent’s identity, model version, a hash of the prompt used, identifiers of retrieved context chunks, the model’s full output, all tool calls made, validation results, confidence score, and delivery timestamp. Hard deletes on this log are blocked at the database layer.

11. Observability & Monitoring

Agent Health Metrics

Reasoning Traces & Audit Records

The authoritative audit record for each agent decision is a structured decision artifact — not the raw reasoning trace — persisted to the append-only audit log before the response is returned:

• Agent identity + model version + prompt hash: Which agent, which model, which exact prompt version governed this decision
• Retrieved context identifiers + similarity scores: Which knowledge base sources the agent was grounded on, with retrieval quality signal
• Tool calls and results: Every tool invocation with declared parameters and structured output (never raw execution logs)
• Final output: The actual response delivered to the caller
• Validation results: Schema check, citation coverage score, PII scan result, composite confidence score, Critic assessment if applicable
• Workflow state snapshot: Phase, iteration count, delegation depth, Temporal workflow ID

Raw reasoning traces, when stored for debugging or model improvement purposes, must be written to a separate, encrypted, restricted-access log path. Access requires elevated authorization and is audited. Raw traces must never appear in general application logs, never be returned via API responses, and never be used as the sole basis for audit or compliance evidence.

Model Drift Detection

• Decision accuracy drift: Weekly automated regression against the labeled decision corpus. Alert if accuracy drops more than 5% from the rolling 30-day baseline.
• Semantic drift: Monthly embedding similarity analysis comparing the centroid of agent output embeddings this month against the 3-month rolling average. Significant divergence triggers a governance review.
• Confidence drift: Rolling 7-day mean confidence per agent monitored continuously. A sustained decline from baseline triggers investigation into potential model degradation or knowledge base staleness.

Self-Healing & Auto-Remediation

• Circuit Breakers: Per-LLM-provider circuit breaker. On consecutive timeouts, the circuit opens and traffic routes to the configured fallback model. Circuit probes every 30 seconds; closes after three consecutive successes.
• Failed Agent Recovery: After three consecutive validation failures, the agent is automatically restarted with a sanitized (empty) context. If failures continue post-restart, the agent is isolated and an incident is created.
• Context Corruption Detection: Knowledge base entries retrieved with a hash mismatch are quarantined, excluded from future retrieval, and flagged for manual review. The agent proceeds with reduced context rather than poisoned context.
• Resource Exhaustion: Vector index size and cache memory are monitored continuously. Alerts fire at 70% capacity to allow proactive scaling before exhaustion.

Tooling Stack

• Metrics: Prometheus + Grafana. Custom dashboards for per-agent cost tracking and accuracy burn-down.
• Logs & Traces: Structured JSON to Loki; distributed tracing to Tempo via OpenTelemetry. All LLM spans annotated with model identity, token counts, cost, and confidence.
• LLM-specific monitoring: Arize AI or equivalent (hallucination detection, embedding drift, decision accuracy tracking). Async export — never on the critical inference path.
• Prompt management: Git-based registry; Grafana panels show per-prompt-version accuracy and cost trends to guide promotion decisions.

12. Framework Landscape

For a regulated enterprise (large financial institution) deploying on GKE, framework selection must satisfy additional criteria beyond technical capability: active vendor support, enterprise security posture, longevity guarantees, and regulatory acceptability. The frameworks below are divided into two tiers accordingly. Communication protocols (MCP for tool access, A2A for agent-to-agent — covered in Sections 6 and 8) are framework-agnostic and should be adopted regardless of which framework tier you choose.

Enterprise-Grade Frameworks

Actively maintained, production-proven, with enterprise support and security commitments. Acceptable for regulated financial services deployments.

Design Pattern References

Framework Selection Guidance

13. Testing Strategy

Unit Testing

• Prompt constraint adherence: Assert that system prompts with adversarial user inputs do not produce outputs that violate declared rules. Use a lightweight mock LLM that returns scripted responses for specific input patterns — not a live API call.
• Context assembly: Assert correct token counts, slot allocation, overflow handling, and summarization triggers for known input sizes. These are deterministic tests that do not require LLM calls.
• Retrieval pipeline: Unit tests for embedding search, BM25 indexing, RRF fusion, and re-ranking logic with fixed fixtures.
• Output validation: Tests for the schema validator, PII scanner, confidence threshold logic, and fact-verification pipeline in isolation.
• Harness authorization: Assert that tool calls not in the agent’s allowlist are rejected before reaching the execution sandbox.

Integration Testing

• End-to-end workflow: Full agent decision flow against real backing services (vector DB, Redis, knowledge base) with a mock LLM via a local OpenAI-compatible server.
• Multi-agent coordination: Test the full delegation chain (Coordinator → Executor → Critic) with mock LLM. Assert correct message routing, delegation depth enforcement, and escalation on conflict.
• Workflow durability: Kill a worker mid-workflow; verify Temporal replays and resumes from the last checkpoint without data loss.
• PII non-leakage: Inject synthetic PII into test inputs; assert it is absent from all message queue events, log entries, and agent outputs.

Hallucination Testing

• Out-of-knowledge-base queries: Present agents with questions whose answers are not in the knowledge base. Assert output contains an explicit uncertainty signal, not a fabricated answer.
• Adversarial prompt library: 50+ curated jailbreak and injection attempts. Assert all are detected and rejected before reaching the LLM API.
• Consistency stress tests: Feed contradictory premises; assert the agent detects the contradiction and escalates rather than synthesizing a fabricated resolution.
• Production regression: Every hallucination event in production generates a new test case. Target: 100% of known failure modes covered.

Performance Testing

• Latency profiling: Measure P50/P95/P99 per agent archetype under 1×, 2×, 5×, and 10× normal load. Identify which component (inference API, retrieval, orchestration) limits throughput.
• Token efficiency: Measure actual vs. budgeted token usage per slot. Any agent consistently using more than 90% of its budget needs configuration review.
• Cache effectiveness: Measure embedding cache and prompt cache hit rates under realistic query distributions. Target >70% for stable knowledge base content.

Canary & A/B Testing

• New prompt versions are deployed to 5% of traffic. Promotion criteria: accuracy ≥95% of control, latency within 110%, cost within 105%, measured over a minimum 48-hour window.
• Model version changes: single-replica canary first. Workflow history provides the ground truth for comparison.
• Retrieval strategy A/B: BM25-only vs. hybrid search — decision quality measured over 2-week windows before committing to a change.

Evaluation Framework

LLM agent evaluation operates at four distinct layers that require separate metrics and tooling. Conflating them produces misleading quality signals — a model that scores well on benchmarks can still produce a failing agent workflow, and a passing workflow can still miss business targets.

LLM-as-Judge

LLM-as-judge uses a separate LLM to evaluate agent output quality at scale, enabling evaluation that would be prohibitively expensive with human reviewers alone. MLflow ships built-in judges for grounding, correctness, safety, relevance, and custom guidelines. Key design rules for reliable judges in production:

• Use a stronger or equal model as judge: A weaker judge cannot reliably evaluate a stronger model’s output. Use the same tier or higher.
• Provide reference answers: Pointwise scoring against a known-good reference is significantly more reliable than comparative ranking between two outputs.
• LLM judges have well-documented biases: Verbosity bias (preferring longer outputs), self-preference bias (preferring outputs resembling their own training style), and sensitivity to prompt template wording are confirmed in research. Use deterministic rule-based metrics alongside LLM judges — not instead of them.
• Calibrate against human labels: MLflow supports human feedback loops where reviewer labels improve judge accuracy over time. Establish this calibration pipeline before relying on judges for production promotion decisions.
• LLM judges cannot detect subtle factual hallucinations: Judges share the same knowledge limitations as the models they evaluate. They are reliable for assessing structure, format, relevance, and citation presence — not for verifying factual correctness in specialized domains.

MLflow Integration

MLflow is the recommended evaluation and experiment tracking platform (30M+ monthly downloads; top-rated agent eval platform in 2026). Key integrations for this architecture:

• Log every agent run as an MLflow experiment with token counts, latency, confidence scores, tool calls, citation coverage, and validation results as tracked parameters.
• Use MLflow’s LLM Evaluate API to run batch evaluations against the labeled decision corpus before promoting new prompt or model versions to production.
• Compare prompt and model versions side-by-side in the MLflow comparison UI — accuracy, cost, latency, citation coverage — enabling data-driven promotion decisions.
• Export human reviewer labels (approve/reject/correct) back into MLflow as ground-truth feedback, closing the loop between production outcomes and offline evaluation accuracy.
• Use MLflow’s experiment tracking for A/B tests on retrieval strategies, prompt variants, and model tier assignments — the comparison must be evidence-based, not intuition-based.

Test Data & Fixtures

• Labeled decision corpus: 500+ decisions with human-approved ground-truth outcomes, maintained in a dedicated regression test store and used for the weekly accuracy regression run.
• Edge case library: Every production incident generates a synthetic test case. Retained indefinitely.
• Synthetic stress data: Programmatically generated inputs covering boundary conditions, unusual-but-valid inputs, and known failure modes.

14. Deployment & Scaling (GKE)

GKE Node Pools

Auto-Scaling

Standard CPU/memory HPA is insufficient for LLM agent workloads whose resource usage is driven by model inference latency rather than compute. Custom metrics-based HPA is required:

• Message queue depth per agent topic: Consumer pods scale when backlog exceeds a configured threshold, keeping delegation latency predictable under load spikes.
• P95 decision latency: Additional replicas are provisioned when latency exceeds 80% of the per-agent budget, providing headroom before SLA breach.
• GPU nodes: Scale-to-zero for batch inference pools; a ~3-minute cold start is acceptable for non-real-time workloads. Real-time agents must run on always-on compute.

Google ADK Deployment on GKE

For architectures built on Google ADK, Google provides an official GKE deployment path with first-party tooling. ADK agents deploy to GKE via the adk deploy command or the Vertex AI SDK, and on deployment automatically inherit: managed infrastructure, built-in VPC Service Controls, Cloud Trace integration, CMEK encryption, and Vertex AI Agent Runtime management. ADK agents deployed to GKE Autopilot can be exposed as A2A endpoints with automatic Agent Card serving and authenticated access — enabling cross-framework agent interoperability from day one. Refer to the official ADK GKE tutorial (linked in References) for the full deployment specification.

GitOps with ArgoCD

• All agent Kubernetes manifests, prompt ConfigMaps, and knowledge base update jobs are managed via ArgoCD. Infrastructure state is always derivable from the Git repository.
• Prompt updates create a new ConfigMap version; ArgoCD’s rolling update strategy delivers zero-downtime hot-reload without agent pod restarts.
• Model version changes go through a separate ArgoCD Application with a manual sync gate requiring AI governance team approval before applying to production.

Deployment Strategies

• Agent services: Blue-green deployment via ArgoCD Rollouts. New version is deployed alongside existing; traffic shifts only after health checks pass.
• LLM model versions: Canary via a feature flag in the harness model router. The flag is controlled by a ConfigMap — no code deployment required for model version changes.
• Knowledge base updates: Incremental batch jobs that append new entries and retire superseded ones. No downtime; retrieval continues against the stable existing index during the update.

15. Cost Optimization

Token Cost Levers (Highest Impact First)

Cost Monitoring & Governance

• Per-agent, per-model, and per-tenant cost counters updated at inference time feed real-time Grafana dashboards.
• Budget burn-rate alerts fire when projected monthly spend exceeds 80% of budget, leaving time to adjust routing rules before overrun.
• Weekly cost efficiency reports compare actual token usage against budget per agent and flag agents where actual usage consistently exceeds 90% of budget.
• Model routing rules are reviewed quarterly against current provider pricing and workload accuracy data to re-optimize the tier assignments.

16. Architecture Diagrams

System Architecture Overview

flowchart TB
    subgraph Clients["Client Layer"]
        UI["Web / Mobile UI"]
        API["API Consumers"]
        MSG["Messaging Channels\n(WhatsApp · Slack · Telegram)"]
    end

    subgraph GW["API Gateway (GKE)"]
        APIGW["Nginx / FastAPI Gateway\nRate Limiting · Auth · PII Tokenization"]
    end

    subgraph Orchestration["Orchestration Layer (GKE)"]
        COORD["Coordinator Agent\nIntent Classification · Routing"]
        TEMPORAL["Apache Temporal\nDurable Workflow Engine"]
    end

    subgraph Harness["Agent Harness (GKE)"]
        CTX["Context Assembly\nRetrieval · Compression · Slot Filling"]
        EXEC["Tool Execution Sandbox\nSchema Validation · Authorization"]
        VALID["Output Validation\nConstraint Check · PII Scan · Confidence"]
    end

    subgraph Agents["Specialist Agent Pool (GKE)"]
        PLANNER["Planner Agent\nHigh-tier model"]
        EXECUTOR["Executor Agents\nMid-tier model"]
        CRITIC["Critic / Verifier\nLow-to-mid-tier"]
        SYNTH["Synthesizer\nHigh-tier model"]
        MEMORY_CONS["Memory Consolidator\nLow-tier · Background"]
    end

    subgraph Memory["Memory and Knowledge (GKE)"]
        REDIS["Redis\nHot Cache · Sessions"]
        PG["PostgreSQL + pgvector\nVector Search · Audit Log"]
        NEO4J["Neo4j\nKnowledge Graph"]
        GCS["Object Storage\nCold Archive"]
    end

    subgraph LLM["LLM Providers"]
        ANTHROPIC["Anthropic API\nHaiku · Sonnet · Opus"]
        OPENAI["OpenAI API\nGPT-4o"]
        VLLM["vLLM (GKE GPU Pool)\nOpen-Source Models"]
    end

    subgraph Obs["Observability"]
        PROM["Prometheus + Grafana"]
        LOKI["Loki — Reasoning Traces"]
        ARIZE["Arize AI — Drift Detection"]
        TEMPO["Tempo — Distributed Tracing"]
    end

    KAFKA["Apache Kafka\nAgent Messaging · DLQ"]

    Clients --> GW --> COORD
    COORD <--> TEMPORAL
    COORD --> KAFKA --> Agents
    Agents --> Harness
    Harness --> Memory
    Harness --> LLM
    Agents --> Obs
    COORD --> Obs
        

Agent Decision Flow (Single Agent)

flowchart LR
    INPUT["Inbound Request"] --> SANITIZE

    subgraph Prep["Harness — Context Assembly"]
        SANITIZE["Input Sanitization\nPII tokenization · Injection detection"]
        RETRIEVE["Retrieval Pipeline\nBM25 + Dense embedding + RRF re-ranking"]
        COMPRESS["Contextual Compression\nRemove low-relevance sentences"]
        ASSEMBLE["Slot Assembly\nEnforce token budgets per slot"]
    end

    SANITIZE --> RETRIEVE --> COMPRESS --> ASSEMBLE

    subgraph Inference["LLM Inference"]
        CACHE["Cache Check\nPrompt cache + Embedding cache"]
        LLM["LLM API Call\nmodel selected by router"]
        CACHE -- "cache miss" --> LLM
        CACHE -- "cache hit" --> VALID
    end

    ASSEMBLE --> CACHE

    subgraph ToolLoop["Tool Execution Loop"]
        TOOL_AUTH["Tool Authorization\nAllowlist check · Schema validate"]
        TOOL_EXEC["Sandboxed Execution\nResult injected then return to LLM"]
    end

    LLM -- "tool call" --> TOOL_AUTH --> TOOL_EXEC --> LLM

    subgraph Validation["Output Validation"]
        VALID["Schema + Constraint Check"]
        FACT["Fact Verification\nClaims vs. knowledge base"]
        PII["PII Scanner"]
        CONSISTENCY["Consistency Verification\n2nd-pass lightweight model"]
        CONFIDENCE["Confidence Threshold"]
    end

    LLM -- "final response" --> VALID --> FACT --> PII --> CONSISTENCY --> CONFIDENCE

    CONFIDENCE -- "pass" --> AUDIT["Append-Only Audit Record"]
    CONFIDENCE -- "below threshold" --> HUMAN["Human Review Queue\nTemporal long-running activity"]
    AUDIT --> RESPONSE["Deliver Response"]
    HUMAN --> RESPONSE
        

Multi-Agent Collaboration: Planner to Executors to Critic to Synthesizer

sequenceDiagram
    participant USER as User / Upstream System
    participant COORD as Coordinator
    participant PLANNER as Planner Agent
    participant KAFKA as Kafka
    participant EX1 as Executor A
    participant EX2 as Executor B
    participant CRITIC as Critic / Verifier
    participant SYNTH as Synthesizer
    participant TEMPORAL as Temporal Workflow

    USER->>COORD: Submit complex goal
    COORD->>TEMPORAL: Start durable workflow
    TEMPORAL->>KAFKA: Dispatch to Planner queue
    KAFKA->>PLANNER: Consume goal
    PLANNER->>PLANNER: Decompose into sub-tasks and select tools
    PLANNER-->>KAFKA: Publish execution plan
    KAFKA-->>TEMPORAL: Consume plan

    par Parallel execution
        TEMPORAL->>KAFKA: Dispatch sub-task A to Executor A queue
        KAFKA->>EX1: Consume sub-task A
        EX1->>EX1: Retrieve context, LLM inference, tool calls
        EX1-->>KAFKA: Publish result A
    and
        TEMPORAL->>KAFKA: Dispatch sub-task B to Executor B queue
        KAFKA->>EX2: Consume sub-task B
        EX2->>EX2: Retrieve context, LLM inference, tool calls
        EX2-->>KAFKA: Publish result B
    end

    KAFKA-->>TEMPORAL: Both results received
    TEMPORAL->>KAFKA: Dispatch to Critic queue
    KAFKA->>CRITIC: Consume results A and B
    CRITIC->>CRITIC: Verify claims, check consistency, score confidence
    CRITIC-->>KAFKA: Verification report pass or issues found
    KAFKA-->>TEMPORAL: Consume verification

    alt Issues found
        TEMPORAL->>KAFKA: Re-dispatch failing sub-task with critic feedback
    end

    TEMPORAL->>KAFKA: Dispatch to Synthesizer queue
    KAFKA->>SYNTH: Consume verified results
    SYNTH->>SYNTH: Combine, resolve conflicts, generate final output
    SYNTH-->>USER: Final response plus reasoning trace plus audit trail ID
        

Memory Retrieval Interaction

sequenceDiagram
    participant AGENT as Agent (any archetype)
    participant REDIS as Redis (Hot Cache)
    participant PG as PostgreSQL + pgvector
    participant NEO4J as Neo4j (Knowledge Graph)
    participant ASSEMBLER as Context Assembler

    AGENT->>REDIS: Check embedding cache (query hash)
    alt Cache hit (TTL valid)
        REDIS-->>AGENT: Cached chunks
    else Cache miss
        AGENT->>PG: Dense embedding search (top-10 chunks)
        AGENT->>PG: BM25 keyword search (top-10 chunks)
        AGENT->>PG: RRF fusion to top-5 ranked chunks
        PG-->>AGENT: Retrieved chunks + source identifiers
        AGENT->>NEO4J: Graph query (entity relationships, 2-hop)
        NEO4J-->>AGENT: Related entity context
        AGENT->>PG: Query expansion variants (2-3 paraphrases)
        PG-->>AGENT: Additional chunks (merged and de-duplicated)
        AGENT->>REDIS: Cache merged result (configurable TTL)
    end
    AGENT->>PG: Fetch recent episodic decisions (last N entries)
    PG-->>AGENT: Decision history (summarized if needed)
    AGENT->>ASSEMBLER: Combine rules, retrieved chunks, graph context, history, request
    ASSEMBLER->>ASSEMBLER: Enforce slot budgets and apply contextual compression
    ASSEMBLER-->>AGENT: Final prompt within context window
        

Context Window Budget (Long-Context Synthesizer Agent)

pie title Context Budget — Synthesizer Agent (30,000 tokens)
    "System Prompt + Rules" : 3000
    "Retrieved Context (multi-source)" : 20000
    "Conversation History" : 2000
    "Output Reservation" : 5000
        

Security & Audit Data Flow

flowchart LR
    subgraph Ingress["Ingress"]
        INPUT["Raw User Input / External Data"]
        PII_TOK["PII Tokenizer\nvault — agents see tokens only"]
    end

    subgraph AgentExec["Agent Execution"]
        HARNESS["Agent Harness\nauthorization · sandboxing · retry"]
        LLM_CALL["LLM API Call\ntokens, never raw PII"]
        TOOL["Tool Execution\nsandboxed · allowlisted"]
    end

    subgraph OutputProc["Output Processing"]
        PII_SCAN["PII Scanner\nredact any leaked sensitive data"]
        SCHEMA_VAL["Schema + Constraint Validator"]
        AUDIT_WRITE["Append-Only Audit Log\nimmutable — deletes blocked"]
    end

    subgraph Delivery["Delivery and Observability"]
        CLIENT["Client / Downstream System"]
        LOKI_LOG["Loki — Reasoning Trace"]
        ARIZE["Arize AI — Async Export"]
    end

    INPUT --> PII_TOK --> HARNESS --> LLM_CALL --> TOOL --> HARNESS
    HARNESS --> PII_SCAN --> SCHEMA_VAL --> AUDIT_WRITE
    AUDIT_WRITE --> CLIENT
    AUDIT_WRITE --> LOKI_LOG
    AUDIT_WRITE --> ARIZE
        

Appendix A: Glossary

Appendix B: Model Comparison

Appendix C: References & Further Reading

• OWASP Top 10 for Large Language Model Applications (2025 Edition) — owasp.org/www-project-top-10-for-large-language-model-applications
• Google Agent Development Kit (ADK) — GKE Deployment Guide — cloud.google.com/kubernetes-engine/docs/tutorials/agentic-adk-vertex
• Google Agent Development Kit Documentation — adk.dev
• Microsoft Agent Framework 1.0 GA Announcement — devblogs.microsoft.com/agent-framework
• Microsoft Agent Framework Overview — learn.microsoft.com/en-us/agent-framework/overview
• A2A Protocol v1.0 — Linux Foundation AI & Data — a2a-protocol.org
• A2A Protocol: One Year Milestones (150+ Organizations) — linuxfoundation.org, April 2026
• LangGraph Production Patterns — langchain-ai.github.io/langgraph
• Agent Harness Engineering: The Rise of the AI Control Plane — Adnan Masood, Medium, April 2026
• From Prompts to Harnesses — Four Years of AI Agentic Patterns — bits-bytes-nn.github.io, April 2026
• Agent Harness for Large Language Model Agents: A Survey — Preprints.org, April 2026
• Lewis et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS 2020.
• Ji et al. (2023). "Survey of Hallucination in Natural Language Generation." ACM Computing Surveys.
• Yao et al. (2023). "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR 2023.
• Apache Temporal Workflow Engine Documentation — docs.temporal.io
• pgvector: Open-source vector similarity search for PostgreSQL — github.com/pgvector/pgvector
• NIST AI Risk Management Framework (AI RMF 1.0) — nist.gov/artificial-intelligence
• Anthropic Responsible Scaling Policy — anthropic.com/rsp
• vLLM: Easy, Fast, and Cheap LLM Serving — github.com/vllm-project/vllm