Angular and Vaadin: complete Codex and Claude results

Pass rates, with harnesses kept separate

Each bar contains four model/effort combinations × five trials. Angular was run with vanilla only. Compare vanilla rows for the technology comparison; MCP-only and skills + MCP measure additional configurations. Codex and Claude use different models and agent harnesses, so their results are not pooled.

Results, running time and cost — all completed trials

Codex

Claude Code

Averages include successes and failed tests. Agent time excludes setup and verification; full-trial time includes both. Queue waits are excluded. Token totals equal input + output; cached input is already included in input. Costs are recorded USD estimates, not subscription invoices.

Comparison for each model and effort

Codex

Claude Code

Each cell contains five trials. Time, tokens and costs are means over all five; cached input is a subset of input. The CSV contains full-precision counts, including cache-creation tokens where available.

Token usage and estimated cost

Codex

Claude Code

Costs are available for 160 completed trials. Values come from each trial’s agent_result.cost_usd, not a recalculation using current prices. Harbor’s Codex adapter can estimate costs from pricing tables; Claude reports session costs. Both harnesses use subscription authentication. These estimates are not invoices and exclude subscription fees, infrastructure and verification costs. Different pricing/recording paths limit direct cross-harness cost comparisons.

Cost / pass = total recorded cost of completed trials ÷ passes, including spend on failures. It describes this sample, not a forecast. Cached reads and cache-creation tokens are included in input; they must not be added to it again. Output already includes recorded reasoning tokens. Counts sum usage over requests, including repeated context, rather than counting unique prompt text. K = thousand; M = million.

Passing trials only, grouped by task

Only completed trials with reward = 1 contribute to this section’s times, tokens and costs. Each row has one task, one harness, one model, one effort and one configuration. No averages combine tasks or models. Passed is shown out of five scheduled trials.

Angular Petclinic task

angular-petclinic-vets-simple

Codex

Claude Code

Vaadin Petclinic task

flow-petclinic-vets-simple

Codex

Claude Code

Compare matching model/effort rows across the two tasks. These successful-run subsets are small and unequal. A dash indicates no passing trials, not zero cost or time. Average cost here is the mean cost of a passing run, while the main report’s cost / pass also includes failed-run spending. Failed trials and their spending are excluded here. Data export: passing-trials.csv.

What the completed results show

• The vanilla comparison favors Vaadin in both harnesses. The observed gap is 55 percentage points with Codex and 25 points with Claude Code.
• Tooling effects differ. Codex scores 16/20 vanilla, 13/20 with MCP only and 19/20 with skills + MCP. Claude scores 17/20, 18/20 and 16/20 respectively. MCP only does not improve both harnesses in this sample.
• A combined tooling advantage is not consistent across harnesses. Skills + MCP adds three passes over vanilla for Codex but loses one for Claude. These are small samples; they do not establish a general causal effect.
• Keep model-level comparisons visible. Model prices, effort levels and successful-run subsets differ. Use the per-model tables rather than pooled successful-run cost or timing averages.

Comparability and limits

Both tasks use the same one-line instruction and identical UC-002 use-case text and shared business rules in the inspected submissions. Both allow 2 CPUs, 4 GB RAM, a 60-minute agent phase and a 20-minute verifier phase. Fixture code, the 1,000-vet dataset, a 200-result-row loading-step budget and 20-second browser interaction timeouts are matched.

The complete stacks differ: Angular + Quarkus + JPA versus Vaadin Flow + Spring Boot + jOOQ. Angular needs frontend/backend integration and its own paging UI; Vaadin supplies a Grid component and backend data-provider integration. Angular checks browser-visible behavior; Vaadin combines server-side component tests with browser tests and requires a Vaadin Grid. Their scroll-driving mechanisms differ. Those differences are part of this benchmark.

There is one task pair and five repetitions per model/effort/configuration. A single trial changes a model cell by 20 percentage points. Results are descriptive scores under these verifiers, not a general ranking of frontend technologies or application runtime performance. MCP-only trials ran later in a separate suite; order and provider state are not randomized.

One original Claude attempt ended with a revoked OAuth token and was replaced after authentication recovered. The 160 scored results use the valid replacement and exclude the interrupted attempt’s time and cost. This report covers scored trial estimates, not all money or quota consumed while operating the suites. No benchmark tests or submitted solutions were modified for this report.