ScarfBench Shows AI Agents Still Fail Most Enterprise Java Migration Tasks
IBM Research published ScarfBench (Self-Contained Application Refactoring Benchmark), an open, expert-validated benchmark for evaluating how well AI agents handle cross-framework migration in enterprise Java — for example, moving an application from Spring to Quarkus, or between Jakarta EE implementations. The benchmark comprises 34 application families spanning Spring, Jakarta EE, and Quarkus, yielding 102 application variants (roughly 151,000 lines of paired Java code) and 204 directed migration tasks with a public leaderboard.
The headline finding is sobering for anyone assuming agentic coding tools have solved framework migration: the strongest agent evaluated achieved only 15.3% aggregate test-pass rate on focused-layer migrations and 12.2% on whole-application migrations. Out of all 204 tasks, only a single one produced a fully behaviorally equivalent target application. ScarfBench is deliberately designed to catch what surface-level code generation benchmarks miss — a migration has to preserve build configuration, dependency injection wiring, persistence mappings, and request-handling behavior simultaneously, not just produce code that compiles.
For engineering teams evaluating whether to trust an AI agent with a framework migration project, ScarfBench is a useful reality check: current agents can meaningfully assist with individual file-level changes during a migration, but end-to-end automated migration of a nontrivial enterprise application is not yet a solved problem. The benchmark and reference implementations are available on GitHub for teams that want to run their own agent tooling against it.
Read more — Hugging Face / IBM Research
Why Specialization Is Inevitable: The Case Against General-Purpose Models
A research essay published on Hugging Face, drawing on a 2026 paper by Goldfeder, Wyder, LeCun, and Shwartz-Ziv, argues that specialization — not ever-larger general-purpose models — is the more durable path for effective AI systems in production. The argument spans optimization theory, biology, and organizational economics: narrower, task-targeted architectures and training regimes consistently outperform broader-coverage systems on the specific tasks that matter, even when the resulting system's outputs look like evidence of general intelligence.
The piece points to AlphaFold as the canonical example — its breakthrough in protein structure prediction came from a task-specific architecture and training pipeline aimed squarely at that one problem, not from scaling a general-purpose model further. The authors extend this into a practical claim for AI procurement and system design: teams building production AI systems should weight vertical, domain-specific models more heavily than the current default assumption that the biggest general-purpose frontier model is always the right starting point.
For teams making build-vs-buy and model-selection decisions, this is a useful counterweight to "just use the biggest model" defaults — it suggests actively evaluating fine-tuned or purpose-built smaller models against general-purpose frontier models for well-scoped, repetitive tasks (classification, extraction, domain-specific code transformation), where a specialized model may deliver better reliability and lower cost than a general-purpose one.
Read more — Hugging Face
Safe & Secure AI Agent Practices
Docker Makes the Case for Sandbox Isolation With SBX and Sandbox Kits
Docker published a detailed explainer on why AI coding agents need runtime isolation, framing the underlying shift plainly: AI agents have moved from passive assistants to active participants that execute commands, install packages, and modify files — meaning developers now supervise AI-generated actions rather than directly controlling every action themselves. Because agent outputs are generated probabilistically, an agent can delete important files, expose credentials, install a malicious dependency, or access data it shouldn't, even without any adversarial input.
Docker's answer is Docker SBX (Sandboxes), which isolates agent execution using microVM-based protection — stronger than standard container isolation — across Windows, Mac, and Linux. Rather than giving an agent direct access to real credentials, SBX routes outgoing network requests through a proxy that injects credentials at the network boundary, so secrets never live inside the sandboxed environment the agent operates in. Outbound network access is restricted by policy, and the sandbox environment itself is fully disposable — Docker's example scenario is deliberately running sudo rm -rf /* inside a sandbox with zero impact on the host.
To make this practical to standardize across a team, Docker introduces Sandbox Kits: reusable, runtime-enforced configurations rather than passive templates. Mixin Kits extend an existing agent setup — installing linters, injecting team configuration, or granting access to approved internal services — while Agent Kits define a complete environment from scratch, including container image, entrypoint, networking, credential handling, and persistence behavior. Kits can also append project-specific instructions (AGENTS.md, CLAUDE.md-style files) and stack multiple Mixin Kits for layered functionality, letting an organization distribute a consistent, pre-approved sandboxed setup across every developer running agentic tools. As agentic coding tools gain more autonomy, this kind of infrastructure-level containment is becoming as foundational to safe AI-assisted development as containerization itself was for reproducible deployments.
Read more — Docker Blog
