Brings the rubric_agent package up to parity with agento_next/main
(post-#1071 architecture), adapted to webeval:
- Adds `CriticalPointAgent` (Step -1) that classifies the task against
a critical-point taxonomy (`critical_point_types.yaml`) and threads
the result through rubric generation, action-only scoring, outcome
verification, and a new CP-violation check.
- Adds `VerifierAgent` (Steps 9a/9b/10): first-point-of-failure
analysis with the error taxonomy, trajectory-informed task
verification, and unified task verification. Step 11 (synthetic
human-voice feedback) is intentionally dropped — not needed in webeval.
- Mirrors upstream #1071's DRY refactor: extracts shared helpers
(`format_action_history`, `call_llm`, `encode_image_b64`,
`get_init_url_context`, `build_scored_rubric_summary`,
`build_all_screenshot_evidence_text`) into `formatting.py` so
`MMRubricAgent` and `VerifierAgent` don't duplicate them. Also
removes the lazy `MMRubricAgent._format_action_history` import
from `critical_point_classifier.py`.
- Picks up upstream #889 `run_command` support: adds
`StepSummary.tool_output`, populates it from post-action
`ToolOutput` observations, renders a `Command Output:` line in the
action history, and teaches the prompts to treat that output as
ground truth (so an unchanged desktop after `run_command` is not
read as failure) while sanity-checking that the command isn't a
fake (`echo "success"`).
- Adds missing runtime deps (`imagehash`, `jinja2`) to
`webeval/pyproject.toml` — both are imported by the new modules.
Architecture note: the branch keeps `MMRubricAgent` and `VerifierAgent`
as independent agents orchestrated by the caller, rather than upstream
#1071's compose pattern. This is the same direction (decoupling) and
goes a step further by also stripping Step 11. `verify_trajectories.py`
drives them in sequence.
All 28 webeval unit tests pass; the live-LLM end-to-end test
(`test_verify_trajectories_live_llm`, opt-in via `FARA_VERIFY_LIVE_TEST=1`)
asserts the new CP-aware fields (`cp_type_used`, `cp_violation`,
`error_taxonomy.first_point_of_failure.failure_points[].error_code`,
Steps 9b/10 `is_ambiguous`/`is_invalid`) hit the score file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the cuaverifierbench/ build script (now hosted alongside the
dataset on HuggingFace) and checks in a full WebTailBench trajectory
under webeval/data/example_trajectory/ (web_surfer.log, final_answer,
screenshots, core.log, times.json, task_data.json, rubric score file).
webeval/README.md now walks through each file against that concrete
example and corrects several field-level inaccuracies (times.json keys,
emitted event types, webtailbench score payload, auto-0 vs excluded
semantics). Adds test_trajectory_loading and test_verify_trajectories
coverage; repairs HuggingFace dataset URLs and doubled /path/to paths
in the root README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes in one PR:
1. Remove webeval's dependency on ``autogen-core`` / ``autogen-ext``.
All chat completion clients, message types, and the graceful-retry
layer now live under ``webeval/src/webeval/oai_clients/`` —
self-contained wrappers around openai / azure-identity. Install no
longer needs the autogen submodule; just ``pip install -e .[vllm]``
then ``cd webeval; pip install -e .``.
2. Incorporate the initial (now-stale) WebTailBench benchmark into the
codebase. ``webeval/src/webeval/benchmarks/webtailbench/`` +
``webeval/scripts/webtailbench.py``. Loader auto-downloads
``WebTailBench-v1-rubrics.tsv`` from
``huggingface.co/datasets/microsoft/WebTailBench`` and threads each
task's published ``precomputed_rubric`` through to the verifier so
rubrics never get regenerated.
3. Release the Universal Verifier (``MMRubricAgent``) as the official
judge for WebTailBench. Multimodal, rubric-grounded, two-model
ensemble (``gpt-5.2`` + ``o4-mini``) with per-criterion scoring,
outcome verification, ambiguity / invalid-task classification, and
first-point-of-failure analysis. ``webeval/scripts/verify_trajectories.py``
is a stand-alone parallel runner that re-scores any directory of
webeval-shaped trajectories without touching the solver.
Documentation: repo-root README ``Updates`` section + Reproducibility
CLI block; ``webeval/README.md`` documents the Trajectory / FinalAnswer
schema, the ``<no_answer>`` semantics, and per-benchmark score-file
shape.
Tests: 18 passing, 1 skipped (opt-in HF download).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hussein Mozannar