Scoring in the Agent Container

Overview

By default (evaluation.container: dedicated), Bunsen evaluates experiment results in a dedicated scorer container — a fresh instance of the same Docker image with the agent's final /workspace extracted from the agent container and /workspace-source — an immutable snapshot of the initial seeded inputs — mounted alongside it. This provides isolation but loses system-level changes: installed packages, files outside /workspace, and system configuration.

evaluation.container: agent opts into running scorers inside the agent's container after the agent finishes. The agent's filesystem state is fully preserved: packages, system files, configuration changes.

Important caveat: this mode is what makes an experiment's verifiers/ directory visible to the agent. Because Docker cannot add mounts to a running container, the mount must exist up front — so in evaluation.container: agent mode /bunsen/verifiers is mounted (read-only) into the agent container before the agent runs, and the agent can read it during the run. Don't put held-out fixtures or answer keys in verifiers/ for an agent-mode experiment. In the default dedicated mode only the scorer container mounts verifiers/, so those assets stay hidden from the agent.

Key flow: LLM-backed criteria (judge, agent, browser-agent) get their provider's key delivered to each scorer exec as a one-time key file the scorer reads and deletes (never an environment variable) — so even in this mode the agent under test never sees the platform's key. See Trust Model.

When the agent ran as the non-root bunsen user, scorers in the agent container also run as bunsen with HOME=/home/bunsen. If the agent ran as root (environment.user: root), scorers run as root. This matters for user-scoped state like conda environments, virtualenvs, and per-user config directories.

What It Preserves (and What It Doesn't)

evaluation.container: agent preserves state — not necessarily processes.

Why processes may not survive: Background processes started by the agent can survive into the scoring phase, but only if the agent uses proper shell backgrounding (nohup ... &). Some agent frameworks (e.g., Claude Code's run_in_background tool parameter) manage background tasks internally and kill them on exit — those processes won't survive. See Scoring Service Tasks for the detailed analysis.

For service experiments, whether the agent properly daemonizes a service is meaningful benchmark signal, not a platform bug. An agent that can't leave a persistent service running has a real capability gap.

When to Use

Set evaluation.container: agent when the experiment involves:

• Package installation — agent installs/upgrades packages that scorers need to verify.
• System configuration — agent modifies files outside /workspace (e.g. /etc, /opt).
• Environment setup — agent creates conda environments, virtualenvs, or other runtime environments.
• Service experiments — agent starts servers or daemons that scorers need to test — with the understanding that process survival depends on agent behavior.

Usage

$schema: https://schemas.bunsen.dev/experiment.v1.json
version: v1
name: my-package-task
description: Fix a package version
task:
  prompt: Upgrade pandas to support dtype_backend.
workspace:
  sources:
    - path: ./workspace
environment:
  image:
    base: bunsen/headless
evaluation:
  container: agent
  criteria:
    - id: tests-pass
      title: Tests pass
      type: script
      run: pytest /bunsen/verifiers/test_outputs.py -v
      scores: [0, 1]

How It Works

Default flow (dedicated scorer container)

Agent runs in Container A
  → extract /workspace
  → extract /workspace-source
  → create Container B (fresh image)
  → run scorers in B
  → destroy B
  → destroy A

With evaluation.container: agent

Agent runs in Container A (with scorer mounts pre-attached)
  → agent finishes
  → run scorers in Container A (no extraction, no new container)
  → destroy A

What this means for you:

• No separate container — the workspace is already in place, so scoring saves container-creation overhead and preserves all state.
• Same workspace contract — scorers still see a mutable /workspace and /workspace-source — an immutable snapshot of the initial seeded inputs.
• Same user context — scorers run as the same user as the agent, so user-scoped environments resolve the same way during scoring.
• Same criterion types — script, judge, agent, and browser-agent criteria all work unchanged. Browser criteria need Playwright's browsers to be readable and executable by the scorer's user. Updated bunsen/visual images install them in /ms-playwright with read and execute access for non-root users, so both bunsen and root runs can use browser criteria in this mode.
• Verifier visibility — /bunsen/verifiers must be mounted up front, so the agent can read verifier assets before scoring begins. This is specific to this mode: in the default dedicated mode only the scorer container mounts them. Don't put scoring secrets in verifiers/ when you opt into agent-container scoring.
• Bring your own grader — because the agent's defaults.passEnv keys (e.g. ANTHROPIC_API_KEY) are present in this container, a type: script criterion that calls a model itself and writes result.json works here. In the default dedicated mode no provider key reaches script criteria, by design.

Updating an older visual image

Older bunsen/visual images install Chromium under /root/.cache/ms-playwright, which non-root scorers cannot access. Rebuild the image from the updated Dockerfile:

docker build -t bunsen/visual ./images/visual
bash images/visual/test.sh

Published images need to be rebuilt and republished before registry users receive this fix. After publication, refresh an existing local image with:

docker pull ghcr.io/bunsen-dev/bunsen-visual:latest
docker tag ghcr.io/bunsen-dev/bunsen-visual:latest bunsen/visual:latest

If an experiment adds package layers on top of bunsen/visual, remove its cached bunsen-experiment-... Docker image so the next run rebuilds those layers from the updated base. Custom images must likewise install browsers outside a private home directory and give the scorer's user read and execute access.

Trade-offs

Examples

Two illustrative cases for when evaluation.container: agent is the right fit:

• Package/system state (reliable): the agent upgrades pandas, and the scorer imports pandas 2.0 features that only exist after the upgrade. The agent-installed package must be present at scoring time.
• Service/daemon (depends on agent behavior): the agent starts a fibonacci-server, and the scorer issues a request against it. Scoring succeeds only if the agent daemonized the service so it survives into the scoring phase.

Scores land in the run manifest the same way regardless of evaluation mode — see Run Manifest & Events.

See also

• Scorers & Evaluation — criterion types and how scoring works
• Scoring Service Tasks — process survival for service/daemon experiments
• Running as Root (environment.user) — how the agent's execution user affects scoring
• experiment.yaml Reference — the full evaluation block