Four Eval Engineering Lessons From Building Agent Plugins: Grade the Trajectory, Not the Summary
Google Cloud's Casey West shared four lessons from evaluating the Google Cloud Developer plugin for AI coding agents. The first is the most widely applicable: an LLM judge that reads only the agent's final reply cannot see what the agent did. In an 11-prompt suite, three prompts were flagged as failing, yet the raw tool-call logs showed all three had passed — in one Cloud Run cleanup task, the agent correctly ran the required help command first, but its final message focused on data-loss warnings and the grader penalized it. The recommendation is to score the execution trajectory: which commands ran, were validated and were scoped.
The second lesson concerns asynchronous tool dispatch. Flattening telemetry into a sequential list hid the fact that the harness sent a skill-loading call and two documentation searches in one tool_calls payload — so the skill's instructions arrived after the searches had already run and could not govern them. Preserving turn structure in telemetry is what exposes this class of bug.
Third, run live and plan-only tests side by side: live runs with sandboxed credentials show whether syntax and parameters actually work and whether the agent exercises restraint before destructive actions, but can abort on permission errors; plan-only runs with credentials stripped expose the model's routing and reasoning. In one onboarding prompt the live run stalled on a permissions error while the plan-only run produced an 8,000-character plan that ignored org policies and approvals — roughly half the most important findings came from the difference. Fourth, treat human-curated golden sets of whole-system behaviour (harness, model and plugins) as durable baselines that survive model changes.
Read more — Google Cloud Blog
Grafana Applies SLOs and Error Budgets to AI Agent Behaviour
Grafana Labs outlined an approach that borrows SRE practice for agent quality. Behavioural evaluations — groundedness, task fulfilment, toxicity, personal-data exposure, prompt injection — are computed per interaction, using either an LLM judge or deterministic checks such as regular expressions. The pass rate becomes a service-level indicator measured against an SLO, for example "95% of conversations judged fulfilled over 30 days", and the remaining 5% is the error budget.
The budget then drives decisions the same way it does for latency or availability: while budget remains, teams can experiment with prompts, models and tools; as it burns, quality fixes take priority. Grafana's argument is that conventional metrics — response time, error rate, token use — can show a perfectly "available" agent that is quietly becoming less trustworthy, and that burn-rate alerts catch gradual drift from model, prompt, retrieval or user-behaviour changes that failure alerts miss. The work grew out of monitoring Grafana Assistant.
Grafana recommends keeping quality dimensions as separate SLOs rather than one aggregate score, baselining targets from historical evaluations and tightening them as the agent matures. InfoQ notes the obvious caveat: an SLO is only as reliable as its measurement, so misclassifying LLM judges create false precision, and teams still need representative datasets, human review of ambiguous cases and ongoing validation of the evaluators themselves.
Read more — InfoQ
IBM Granite Switch 4.1 Serves Chat, Decisions and Speech From One Endpoint
IBM released Granite Switch 4.1 in 8B and 30B sizes, a modular composition that serves chat, decision-making and speech recognition from a single endpoint. Each model stacks a general-purpose Decision adapter, a Guardian adapter for safety decisions and the ~470M-parameter Granite-Speech-5.0-Turbo-CTC ASR model on a base Granite model. Because the adapters are Activated LoRAs that reuse the base model's KV cache, the composed model uses roughly the same GPU memory as the base model alone.
This is a notable step in the decision-model trend of recent weeks: rather than deploying a separate decision model next to a chat model, one endpoint answers typed choice/score questions and also serves normal chat. IBM says it can replace calls to the /v1/systemone decision endpoint, runs in vLLM via a plug-in from the Granite Switch repository, and has native llama.cpp support. IBM reports that the 30B model finished a 5-step RAG workflow faster than Jev-27B and that a 3B custom Switch model made 60–100 decisions per second in a demo, though precise timings were not published.
Users can compose their own Switch models from Granite Libraries adapters, and IBM plans an Adapter SDK in its Mellea framework. The post does not state a license; check the model cards before production use.
Read more — Hugging Face
Stack Overflow 2026 Survey: Only 7% of Developers Trust AI Output for Important Decisions
Docker highlighted the AI findings of the 2026 Stack Overflow Developer Survey (more than 30,000 respondents across 169 countries). Only 7% of developers trust AI output for important work decisions, and 48% trust it only when they can verify it themselves. Security and privacy ranked second only to accuracy as the factor that decides whether teams adopt or expand AI tools, and nearly one in four developers said AI-related security or privacy issues cost their team significant time in the past year.
The numbers reinforce a pattern visible across this digest: developer workflows are shifting from "trust the agent" to "verify the agent" — trajectory-level evals, behavioural SLOs and isolation by default. Docker's own conclusion, unsurprisingly, is that autonomous agents need microVM-style sandboxes rather than plain containers, and it points to its Cloud Sandboxes and the Sandbox Kit specification it intends to bring to the CNCF; that part is vendor positioning, but the survey figures stand on their own.
Read more — Docker
Safe & Secure AI Agent Practices
Chainguard Launches Workspaces and Checks: microVM Sandboxes and Attested CI for Coding Agents
At Assemble London, Chainguard introduced two betas aimed at AI coding agents. Chainguard Workspaces gives each developer or agent a dedicated remote environment on hardened microVMs; chainctl develop syncs a local working tree, including uncommitted changes, and detached sessions are snapshotted with files, processes and shell state. Multiple isolated Workspaces per repository keep parallel agents from sharing ports, caches or processes. A host-side proxy injects short-lived credentials into approved outbound requests so long-lived secrets never enter the guest, optional egress rules block and record connections to unapproved endpoints, and commands, code changes and network calls are logged.
Chainguard Checks runs CI steps in warm microVMs built from Chainguard packages, prepared from the project's lockfile. Agents can invoke it through an MCP server or SDK; a failed check returns structured logs so the agent can retry without rebuilding, and a successful run produces a signed attestation bound to the environment, result and commit SHA that merge gates can verify instead of rerunning tests. Workspaces currently supports Claude, with other agents planned.
InfoQ flags open questions: no performance comparison with established CI services, no explanation of how customers verify the logs' tamper resistance, and a trust dependency on Chainguard's execution and signing system. The design pattern — credential injection at a proxy, default-deny egress, attested results — is nonetheless a useful template regardless of vendor.
Read more — InfoQ
AWS Strands Box: Kernel-Enforced Agent Isolation With Dogwood Policies Outside the Model
AWS released Strands Box as a developer preview: an open-source sandbox that routes an agent's shell commands, Python execution, MCP tool calls, filesystem operations and network access through a single policy layer, combining OS-level kernel-enforced isolation with policies written in Dogwood, a new open policy language.
What distinguishes it from container or Firecracker sandboxes, which mainly restrict filesystem, process and network access, is that Dogwood policies can decide based on tool arguments and execution history. Examples include requiring passing tests before a code push, or rate-limiting external API calls. Because these checks run outside the language model, they do not depend on prompt instructions being followed — a prompt-injected agent still cannot push untested code.
AWS says Strands Box works with different models and agent frameworks, locally or in the cloud. For teams designing agent guardrails, it is a concrete example of moving from static allow/deny lists toward stateful, history-aware policy enforcement.
Read more — InfoQ
Anthropic Gives METR Employee-Like Access and Publishes a Three-Tier Model for Independent Evaluators
Anthropic announced that a six-person METR team now has "employee-like access": company hardware, office space and internal tools roughly matching Anthropic's own risk-assessment teams. Anthropic staff are encouraged to share information with them except for defined categories — customer data, privileged legal information and model architecture details. METR can publish its analysis; Anthropic cannot veto publication but can require redaction of those categories, and METR may comment publicly on its level of access. The engaged team is walled off from other METR staff under enforceable agreements.
Anthropic also described a three-tier evaluator model: Tier 1 for narrowly scoped investigations where Anthropic sets access and approves publication; Tier 2 with more information and publication without approval, subject to category redactions that evaluators can contest; and Tier 3, the METR arrangement. Other evaluators named include SecureBio (reviewing the chem/bio chapter of the August Risk Report), Accenture with Faculty (evaluation, red-teaming and safeguard testing), and Hugging Face in early discussions. Stated criteria are frontier expertise, financial and operational independence, sufficient access and freedom to publish.
For practitioners, the tiered structure is a reusable reference for anyone designing third-party audits of AI systems: separate the question of what auditors can see from what they can say, and make redaction categories explicit up front.
Read more — Anthropic
GitHub Actions Workflow Lockfiles Pin Actions and Reusable Workflows to Exact Commits
GitHub put workflow lockfiles into public preview. A lockfile at .github/workflows/actions.lock records exact commits for every action, their transitive dependencies and reusable workflows; once a workflow is onboarded, every dependency must have a lockfile entry and Actions runs only the recorded commits. The rationale is the well-known tag-moving attack: tags and branches can move after a workflow is reviewed, so a compromised upstream action can inject code without any change to your repository.
The gh actions-lock CLI extension generates reviewable lockfiles, onboards workflows one at a time and checks for staleness in CI without modifying files. Dependabot regenerates the matching lockfile entry in the same PR when it bumps an action, and administrators can enforce a "require lockfile" policy at enterprise, organization or repository level. It is available on Enterprise Cloud (including data residency), Team and Free plans.
This matters more as coding agents increasingly author and modify CI workflows: a lockfile gives reviewers one diffable artifact showing exactly what code will run. GitHub is explicit that locking does not make a dependency safe — review before you lock.
Read more — GitHub Changelog
Links & Sources
- Four evaluation engineering lessons learned from building agent plugins
- Grafana Applies SLOs and Error Budgets to AI Agent Behaviour
- Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint
- From Containers to Sandboxes: Developer Trust in the Agentic Era
- Chainguard Launches Secure Workspaces and Checks for AI Coding Agents
- AWS Introduces Strands Decider 2B and Strands Box for AI Agent Development
- An update on independent evaluators
- GitHub Actions workflow lockfiles in public preview
You May Also Enjoy
