AI

How to Evaluate AI Agents: An Engineering Framework

Single-turn prompts tell you almost nothing about how an autonomous agent will behave in the wild. Here is the architecture — sandboxed environments, agent loops, graded by runtime execution — that actually measures what matters.

Comparison of single-turn vs multi-turn agent evaluation architecture

By Editorial · Published Jul 15, 2026 · 8 min read

The capabilities that make AI agents useful — autonomy, tool use, multi-turn reasoning — are the same capabilities that make them hard to evaluate. Standard benchmarks were built for a world where a model takes one input and produces one output. An agent that spins up a server, reads a file, calls an API, and decides what to do next based on the result doesn't fit that model.

Most teams handle this by reaching for what they already have: static prompts, human review, or vibe checks after a demo. None of these scale, and none of them catch the failure modes that actually matter in production — the agent that loops forever, the agent that writes code that looks right but produces a malformed response, the agent that completes nine of ten steps and stops without flagging the gap.

This is an engineering blueprint for evaluation infrastructure that actually works: isolated sandbox environments, a controlled agent loop, and a grading pipeline that tests runtime behavior rather than surface-level output. The benchmark task throughout is building a Model Context Protocol (MCP) server — a real engineering deliverable with a crisp, machine-verifiable pass condition.

Why static evals fail for agents

The gap between single-turn and multi-turn evaluation is architectural, not cosmetic.

Comparison of single-turn vs multi-turn agent evaluation
DimensionSingle-Turn EvalMulti-Turn Agent Eval
Agent inputStatic text promptTask description + tool definitions + ephemeral workspace
Agent behaviorOne generationReason → call tool → process output → repeat
EnvironmentNone (stateless)State maintained across turns (filesystem, processes, network)
Grader inputModel's text responseFinal environment state: artifacts, logs, process output
Grading methodExact match, regex, LLM-as-judgeRuntime execution: unit tests, integration tests, protocol compliance

The crucial shift is in what gets graded. A single-turn eval grades a string. An agent eval grades a world state — what the agent did to the environment across the full trajectory of its actions.

This matters because agent failure modes don't show up in text. An agent can write a well-structured file with coherent prose that fails to parse. It can produce code that compiles but mishandles edge cases the grader will hit. It can partially complete a task and produce an artifact that looks done but isn't. Text-similarity scoring misses all of this.

The correct primitive is runtime execution: boot the artifact, exercise it, observe whether it behaves correctly.

The benchmark task: building an MCP server

MCP is an open standard introduced by Anthropic in November 2024 that defines how AI agents communicate with external tools through a uniform JSON-RPC interface. An agent connecting to an MCP-compliant server can call any tool the server exposes — file reads, database queries, API calls — without knowing anything about the tool's implementation.

Building an MCP server is a good benchmark task for several reasons:

Crisp, machine-verifiable pass condition. An MCP server either responds correctly to an initialization handshake or it doesn't. There's no rubric to debate.

Requires multi-step reasoning. The agent must understand a protocol specification, scaffold the right project structure, implement the request/response handling, and verify its work — in the right order.

Representative of real engineering tasks. This isn't a toy problem. The same capabilities an agent needs to build an MCP server — reading specs, writing and running code, debugging, iterating — are the capabilities that matter for production agentic systems.

Extensible. Once you have the harness running, you can add sub-tests incrementally: tool listing, schema validation, error handling, resource endpoints. The framework scales with the task.

System architecture

Agent evaluation system architecture diagram
Prompt
agent-eval-harness/
├── docker/
│   └── sandbox.Dockerfile      # Ephemeral sandbox image
├── tasks/
│   └── mcp-sqlite-server/
│       ├── task.json           # Metadata, constraints, instructions
│       ├── scaffold/           # Boilerplate files given to the agent
│       └── tests/              # Hidden tests used by the grader only
├── src/
│   ├── sandbox.ts              # Docker container lifecycle manager
│   ├── agent.ts                # Agent loop: LLM + tools interface
│   ├── grader.ts               # Test executor and protocol compliance checker
│   └── run.ts                  # Main entrypoint
├── package.json
└── tsconfig.json

The architecture has three logical components: the sandbox that isolates execution, the agent loop that drives behavior, and the grader that evaluates the result. Each component has a single responsibility and a clean interface to the others.

1. Sandboxed environment (sandbox.ts)

Every eval run gets a fresh Docker container. No state bleeds between runs. The container gets the task scaffold on startup and is destroyed when the grader finishes.

TYPESCRIPT
import { execSync } from 'child_process';
import * as fs from 'fs';
import * as path from 'path';

export interface SandboxConfig {
  taskId: string;
  scaffoldPath: string;
}

export class Sandbox {
  private containerName: string;
  private hostVolumePath: string;

  constructor(config: SandboxConfig) {
    this.containerName = `agent-sandbox-${config.taskId}-${Date.now()}`;
    this.hostVolumePath = path.resolve(`/tmp/evals/${this.containerName}`);

    // Copy scaffold into an isolated runtime directory
    fs.mkdirSync(this.hostVolumePath, { recursive: true });
    execSync(`cp -R ${config.scaffoldPath}/* ${this.hostVolumePath}/`);
  }

  public async start(): Promise<void> {
    const cmd = `
      docker run -d \
        --name ${this.containerName} \
        -v ${this.hostVolumePath}:/workspace \
        -w /workspace \
        --network none \
        node:20-alpine \
        tail -f /dev/null
    `.trim();
    execSync(cmd);
  }

  public exec(command: string, timeoutMs = 15_000): string {
    try {
      return execSync(
        `docker exec ${this.containerName} sh -c ${JSON.stringify(command)}`,
        { timeout: timeoutMs, encoding: 'utf-8' }
      );
    } catch (error: any) {
      // Surface stdout even on non-zero exit so the agent can read compiler errors
      return error.stdout || error.message;
    }
  }

  public async cleanup(): Promise<void> {
    try {
      execSync(`docker rm -f ${this.containerName}`);
      fs.rmSync(this.hostVolumePath, { recursive: true, force: true });
    } catch (e) {
      console.error(`Cleanup failed for ${this.containerName}:`, e);
    }
  }
}

Two details worth noting relative to the naive implementation. First, --network none on the container: agents should complete the task using their provided tools, not by fetching arbitrary resources from the internet during execution. Remove this flag only if the task explicitly requires network access, and scope it tightly. Second, exec returns error.stdout on failure rather than throwing — compiler errors live on stdout, and the agent needs to read them to iterate.

2. The agent loop (agent.ts)

The agent runs in a bounded loop: observe context, select a tool, execute it, update context, repeat. The loop terminates when the agent calls final_submit or when it hits the step ceiling.

TYPESCRIPT
export interface Tool {
  name: string;
  description: string;
  execute: (args: Record<string, unknown>) => Promise<string>;
}

interface AgentAction {
  reasoning: string;
  toolName: string;
  toolArgs: Record<string, unknown>;
}

export class AgentScaffold {
  private tools: Map<string, Tool>;
  private readonly maxSteps: number;

  constructor(tools: Tool[], maxSteps = 20) {
    this.tools = new Map(tools.map(t => [t.name, t]));
    this.maxSteps = maxSteps;
  }

  public async executeTask(instructions: string): Promise<'completed' | 'halted'> {
    let step = 0;
    let context = `Task:\n${instructions}\n\nAvailable tools: ${[...this.tools.keys()].join(', ')}\n`;

    while (step < this.maxSteps) {
      step++;

      const action = await this.callLLM(context);

      if (action.toolName === 'final_submit') {
        console.log(`Agent completed at step ${step}.`);
        return 'completed';
      }

      const tool = this.tools.get(action.toolName);
      const toolOutput = tool
        ? await tool.execute(action.toolArgs).catch((e: Error) => `Tool error: ${e.message}`)
        : `Unknown tool: ${action.toolName}`;

      context += `\n[Step ${step}] Tool: ${action.toolName}\nArgs: ${JSON.stringify(action.toolArgs)}\nOutput:\n${toolOutput}\n`;
    }

    console.warn(`Agent halted at step limit (${this.maxSteps}).`);
    return 'halted';
  }

  private async callLLM(context: string): Promise<AgentAction> {
    // In production: send context to the model, parse tool_calls from the response.
    // The model receives the full conversation history on every turn.
    throw new Error('callLLM not implemented — wire in your model client here.');
  }
}

The step ceiling is not a soft guideline. It's the mechanism that distinguishes "the agent is working" from "the agent is stuck." Without it, a looping agent consumes unbounded compute. With it, you can measure what fraction of agents complete the task within budget, which is itself a useful signal about task complexity and model capability.

The context-window strategy matters here too. Because the agent receives the full history on every turn, very long tasks can overflow the model's context before completion. For tasks expected to exceed ~50 steps, consider a sliding window or a summarization step that compresses earlier tool outputs. For most engineering tasks at the 20-step ceiling, the full history fits comfortably.

3. Tool definitions

The tools available to the agent should match what a developer would actually use for the task. For an MCP server implementation, the minimum useful set is:

TYPESCRIPT
import { Sandbox } from './sandbox';

export function buildTools(sandbox: Sandbox) {
  return [
    {
      name: 'run_command',
      description: 'Execute a shell command in the workspace and return stdout/stderr.',
      execute: async ({ command }: { command: string }) =>
        sandbox.exec(command),
    },
    {
      name: 'write_file',
      description: 'Write content to a file at the given path, creating directories as needed.',
      execute: async ({ path: filePath, content }: { path: string; content: string }) => {
        sandbox.exec(`mkdir -p $(dirname ${JSON.stringify(filePath)})`);
        sandbox.exec(`cat > ${JSON.stringify(filePath)} << 'EVAL_EOF'\n${content}\nEVAL_EOF`);
        return `Written: ${filePath}`;
      },
    },
    {
      name: 'read_file',
      description: 'Read and return the content of a file.',
      execute: async ({ path: filePath }: { path: string }) =>
        sandbox.exec(`cat ${JSON.stringify(filePath)}`),
    },
    {
      name: 'list_directory',
      description: 'List files and directories at a given path.',
      execute: async ({ path: dirPath = '.' }: { path?: string }) =>
        sandbox.exec(`find ${JSON.stringify(dirPath)} -maxdepth 2 | sort`),
    },
    {
      name: 'final_submit',
      description: 'Signal that the implementation is complete and ready for grading.',
      execute: async () => 'Submitted.',
    },
  ];
}

Keep tool descriptions precise. Vague descriptions produce vague tool calls. The agent infers what to do from the description — "Execute a shell command in the workspace and return stdout/stderr" is more actionable than "run stuff."

4. Automated grading (grader.ts)

Key components of automated grading for agent evaluation

The grader runs after the agent exits. It doesn't read the agent's reasoning or evaluate its code style — it boots the artifact and tests whether it behaves correctly.

For an MCP server, the critical test is the initialization handshake: send a JSON-RPC initialize request over stdin, assert that the server responds with a valid protocolVersion. This is the minimum viable compliance check.

TYPESCRIPT
import { execSync, spawn, ChildProcess } from 'child_process';
import * as path from 'path';
import * as fs from 'fs';

interface TestResult {
  name: string;
  passed: boolean;
  message: string;
}

export class MCPGrader {
  constructor(private readonly workspacePath: string) {}

  public async grade(): Promise<{ score: number; results: TestResult[] }> {
    const results: TestResult[] = [];

    results.push(this.testDependenciesInstalled());
    results.push(this.testCompilation());
    results.push(this.testEntryPointExists());
    results.push(await this.testMCPHandshake());

    const passed = results.filter(r => r.passed).length;
    const score = Math.round((passed / results.length) * 100);

    return { score, results };
  }

  private testDependenciesInstalled(): TestResult {
    try {
      execSync(`cd ${this.workspacePath} && npm install`, { stdio: 'ignore', timeout: 60_000 });
      return { name: 'npm install', passed: true, message: 'Dependencies installed.' };
    } catch {
      return { name: 'npm install', passed: false, message: 'npm install failed.' };
    }
  }

  private testCompilation(): TestResult {
    try {
      const pkg = JSON.parse(
        fs.readFileSync(path.join(this.workspacePath, 'package.json'), 'utf-8')
      );
      const buildCmd = pkg.scripts?.build ?? 'npx tsc --noEmit';
      execSync(`cd ${this.workspacePath} && ${buildCmd}`, { stdio: 'ignore', timeout: 30_000 });
      return { name: 'compilation', passed: true, message: 'Compiled without errors.' };
    } catch {
      return { name: 'compilation', passed: false, message: 'Compilation failed.' };
    }
  }

  private testEntryPointExists(): TestResult {
    const candidates = ['build/index.js', 'dist/index.js', 'index.js'];
    const found = candidates.find(p =>
      fs.existsSync(path.join(this.workspacePath, p))
    );
    return found
      ? { name: 'entry point', passed: true, message: `Entry point found: ${found}` }
      : { name: 'entry point', passed: false, message: `No entry point found. Checked: ${candidates.join(', ')}` };
  }

  private testMCPHandshake(): Promise<TestResult> {
    return new Promise(resolve => {
      const entryPoint = ['build/index.js', 'dist/index.js', 'index.js']
        .map(p => path.join(this.workspacePath, p))
        .find(p => fs.existsSync(p));

      if (!entryPoint) {
        resolve({ name: 'MCP handshake', passed: false, message: 'No entry point to launch.' });
        return;
      }

      const server: ChildProcess = spawn('node', [entryPoint], {
        cwd: this.workspacePath,
      });

      let buffer = '';

      const timeout = setTimeout(() => {
        server.kill();
        resolve({
          name: 'MCP handshake',
          passed: false,
          message: 'Timed out after 5s. Server did not respond to initialize.',
        });
      }, 5_000);

      server.stdout?.on('data', (chunk: Buffer) => {
        buffer += chunk.toString();
        // MCP responses are newline-delimited JSON — scan for a complete object
        for (const line of buffer.split('\n')) {
          try {
            const msg = JSON.parse(line.trim());
            if (msg.id === 1 && msg.result?.protocolVersion) {
              clearTimeout(timeout);
              server.kill();
              resolve({
                name: 'MCP handshake',
                passed: true,
                message: `Handshake succeeded. Protocol version: ${msg.result.protocolVersion}`,
              });
            }
          } catch {
            // Incomplete chunk — keep accumulating
          }
        }
      });

      server.stderr?.on('data', (d: Buffer) => process.stderr.write(d));

      const initRequest = JSON.stringify({
        jsonrpc: '2.0',
        id: 1,
        method: 'initialize',
        params: {
          protocolVersion: '2024-11-05',
          capabilities: {},
          clientInfo: { name: 'eval-harness', version: '1.0.0' },
        },
      });

      server.stdin?.write(initRequest + '\n');
    });
  }
}

The handshake test is deliberately strict about what it checks: the protocolVersion field in the response, per the MCP specification. This is the minimum bar for a compliant server. As you extend the benchmark, add sub-tests for tool listing (tools/list), schema validation on tool definitions, and correct error responses on malformed requests.

5. The main entrypoint (run.ts)

The orchestrator ties the three components together:

TYPESCRIPT
import { Sandbox } from './sandbox';
import { AgentScaffold } from './agent';
import { MCPGrader } from './grader';
import { buildTools } from './tools';
import * as path from 'path';
import * as fs from 'fs';

async function run() {
  const taskDir = path.resolve(__dirname, '../tasks/mcp-sqlite-server');
  const taskConfig = JSON.parse(fs.readFileSync(path.join(taskDir, 'task.json'), 'utf-8'));

  const sandbox = new Sandbox({
    taskId: taskConfig.id,
    scaffoldPath: path.join(taskDir, 'scaffold'),
  });

  console.log('Provisioning sandbox...');
  await sandbox.start();

  let exitCode = 0;

  try {
    const tools = buildTools(sandbox);
    const agent = new AgentScaffold(tools, 20);

    console.log('Starting agent...');
    const outcome = await agent.executeTask(taskConfig.instructions);
    console.log(`Agent outcome: ${outcome}`);

    console.log('\nGrading...');
    const grader = new MCPGrader(sandbox['hostVolumePath']);
    const { score, results } = await grader.grade();

    console.log('\n--- Results ---');
    results.forEach(r => console.log(`[${r.passed ? 'PASS' : 'FAIL'}] ${r.name}: ${r.message}`));
    console.log(`\nScore: ${score}/100`);

    exitCode = score === 100 ? 0 : 1;
  } finally {
    console.log('\nCleaning up...');
    await sandbox.cleanup();
    process.exit(exitCode);
  }
}

run().catch(err => {
  console.error('Harness error:', err);
  process.exit(2);
});

The finally block is non-negotiable. If the grader throws — or the agent does — the sandbox still gets destroyed. Leaked containers accumulate disk usage silently and produce false positives on the next run if their volumes aren't cleaned up.

Execution flow

Prompt
[1. Task setup]          Load task.json, resolve scaffold path
       ↓
[2. Sandbox provision]   docker run with mounted volume, --network none
       ↓
[3. Agent loop]          Reason → tool call → observe output → repeat (≤20 steps)
       ↓
[4. Agent exit]          final_submit or step ceiling reached
       ↓
[5. Grader runs]         npm install → compile → entry point check → MCP handshake
       ↓
[6. Report + teardown]   Print results, docker rm -f, rm -rf workspace

The pipeline is synchronous by design. Running one eval at a time is slower than parallelism but produces clean results and makes debugging straightforward. Once you have confidence in the harness, parallelize at the run level — spin up N sandboxes concurrently, each with a different agent configuration or task variant.

Extending the benchmark

The framework above handles the core case. Production evaluation infrastructure typically needs several extensions:

Additional sub-tests. The handshake is a necessary but not sufficient compliance check. Add tools/list verification, schema conformance on tool definitions, error handling for malformed requests, and any domain-specific behavior the MCP server is supposed to implement (database queries, file operations, etc.). Each sub-test should be binary and independently gradable.

Task variants. Once the evaluation infrastructure exists, the interesting question is how models perform across a range of tasks, not just one. Structure tasks/ as a directory of task definitions, each with its own scaffold and test suite. A single run.ts loop over all tasks in the directory gives you a complete benchmark.

Trajectory logging. Store the full agent context — every tool call, every tool output — alongside the grade. This is the data you need to understand why agents fail. An agent that compiles but fails the handshake is failing differently than an agent that hits the step ceiling before compiling. The trajectory distinguishes them.

Multiple agent configurations. The same task run against different models, different system prompts, or different tool sets gives you signal about what drives capability. Structure the harness to accept an agent configuration object and vary it across runs.

Regression tracking. A score on one run is a data point. A score on the same task across ten runs is a distribution. Run each configuration multiple times and report mean and variance — model outputs are stochastic, and a single run can flatter or penalize a configuration for reasons unrelated to its actual capability.

What this framework is measuring

It's worth being precise about what this kind of evaluation does and doesn't tell you.

It measures task completion rate under constrained conditions — a fixed step budget, a specific set of tools, a specific task with a machine-verifiable pass condition. That's a useful, reproducible signal about agent capability on the benchmark task.

It does not measure how the agent will perform on open-ended tasks without a crisp pass condition, on tasks that require more than 20 steps, or in environments where the tool set differs significantly from what was available during evaluation. Benchmark performance and deployment performance are correlated but not identical.

The useful framing is that evals are a measurement instrument, not a certification. A 100% score on this benchmark tells you the agent can implement a compliant MCP server under these conditions. It tells you less about what it will do on the next task it hasn't seen. Build the harness, run the evals, use the signal — but treat it as one input into a broader picture of agent capability rather than a final verdict.

The MCP specification and TypeScript SDK are the primary references for the protocol details covered here. The evaluation harness pattern applies to any agentic task with a machine-verifiable completion condition, not only MCP server construction.

Explore Related Concepts
Frequently Asked Questions
What is the difference between a single-turn and a multi-turn agent evaluation?+

A single-turn eval sends one prompt and scores the response — measuring text quality, factual accuracy, or instruction-following on a fixed input. A multi-turn agent eval drops the agent into an environment with tools, lets it act across many steps, and grades the final state of the environment rather than any individual response. The latter is the only approach that surfaces real agentic failure modes like tool misuse, infinite loops, or partial task completion.

Why use Docker containers for agent evaluation sandboxes?+

Docker gives you reproducibility and isolation in a single primitive. Every eval run starts from the same image — same OS, same Node version, same directory structure — so any difference in outcomes is attributable to the agent, not the environment. Isolation prevents agents from polluting the host filesystem or leaking state between runs, both of which produce false positives or difficult-to-debug failures.

What is the Model Context Protocol (MCP)?+

MCP is an open standard, introduced by Anthropic in November 2024, that defines how AI agents communicate with external tools and data sources through a uniform JSON-RPC interface over stdio or HTTP. Instead of each tool requiring a custom integration, MCP lets any compliant client connect to any compliant server. Building an MCP server is a useful benchmark task because it requires the agent to understand a protocol specification, implement it correctly, and produce a working server that passes a live handshake test.

How should I score an agent eval with partial task completion?+

Decompose the task into independently gradable sub-tests, each returning a binary pass/fail. The aggregate score is the fraction of sub-tests passed. This avoids both all-or-nothing scoring (which discards signal from agents that complete 90% of a task) and vague rubrics (which introduce grader variance). For complex tasks, weight sub-tests by difficulty or criticality — a successful protocol handshake should count more than correct file naming.

What should the step ceiling be for an agent eval?+

It depends on expected task complexity, but 20 steps is a reasonable default for a moderately complex engineering task like implementing an MCP server. The ceiling serves two purposes: it bounds compute cost per run, and it surfaces agents that loop or stall rather than converging. If your benchmark tasks consistently hit the ceiling on correct solutions, raise it — but do not remove it.