mirror of
https://github.com/microsoft/fara.git
synced 2026-08-21 17:50:45 +08:00
Standalone HTML page that shows, for each WebTailBench task, the rubric criteria produced by three different judge configurations side by side: 1. O4-Mini Rubric — historical baseline 2. GPT-5 (v1) — original GPT-5 judge 3. Universal Verifier Rubric (GPT-5.2) — current release Tasks are grouped by WebTailBench benchmark category, with incremental search across ids / summaries / criteria, and an "all three rubrics only" toggle. The header links directly to the microsoft/WebTailBench dataset on Hugging Face and to the WebTailBench-v1-rubrics.tsv download so readers can grab the underlying data. Source layout: docs/webtailbench_rubric_comparison.html (single self-contained file) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
108010 lines
6.7 MiB
Plaintext
108010 lines
6.7 MiB
Plaintext
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>WebTailBench — Rubric Comparison</title>
|
||
<style>
|
||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||
body {
|
||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;
|
||
background: #f5f5f7;
|
||
color: #1d1d1f;
|
||
padding: 20px;
|
||
line-height: 1.55;
|
||
}
|
||
a { color: #0969da; text-decoration: none; }
|
||
a:hover { text-decoration: underline; }
|
||
|
||
.header {
|
||
background: linear-gradient(135deg, #0f2c4e 0%, #1f6fb5 60%, #33a3d9 100%);
|
||
color: #fff;
|
||
padding: 32px 36px;
|
||
border-radius: 12px;
|
||
margin-bottom: 20px;
|
||
box-shadow: 0 4px 10px rgba(0,0,0,0.08);
|
||
}
|
||
.header h1 { font-size: 2em; margin-bottom: 8px; letter-spacing: -0.01em; }
|
||
.header p { font-size: 1.05em; opacity: 0.92; max-width: 900px; }
|
||
.header .links { margin-top: 14px; display: flex; flex-wrap: wrap; gap: 10px; }
|
||
.header .links a {
|
||
background: rgba(255,255,255,0.15);
|
||
color: #fff;
|
||
padding: 6px 12px;
|
||
border-radius: 6px;
|
||
border: 1px solid rgba(255,255,255,0.25);
|
||
font-size: 0.95em;
|
||
}
|
||
.header .links a:hover { background: rgba(255,255,255,0.28); text-decoration: none; }
|
||
|
||
.controls {
|
||
background: #fff;
|
||
border-radius: 10px;
|
||
padding: 14px 18px;
|
||
margin-bottom: 20px;
|
||
display: flex;
|
||
gap: 14px;
|
||
flex-wrap: wrap;
|
||
align-items: center;
|
||
box-shadow: 0 2px 4px rgba(0,0,0,0.05);
|
||
}
|
||
.controls input[type=text] {
|
||
flex: 1;
|
||
min-width: 220px;
|
||
padding: 8px 12px;
|
||
border: 1px solid #d0d7de;
|
||
border-radius: 6px;
|
||
font-size: 0.95em;
|
||
}
|
||
.controls label { font-size: 0.9em; display: flex; gap: 6px; align-items: center; color: #57606a; }
|
||
.controls .stat { font-size: 0.85em; color: #57606a; margin-left: auto; }
|
||
|
||
.segment {
|
||
background: #fff;
|
||
margin-bottom: 12px;
|
||
border-radius: 10px;
|
||
box-shadow: 0 2px 4px rgba(0,0,0,0.06);
|
||
overflow: hidden;
|
||
}
|
||
.segment-header {
|
||
background: #1d1d1f;
|
||
color: #fff;
|
||
padding: 14px 20px;
|
||
cursor: pointer;
|
||
user-select: none;
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 10px;
|
||
}
|
||
.segment-header:hover { background: #2e2e30; }
|
||
.segment-toggle { width: 16px; transition: transform 0.15s; }
|
||
.segment-toggle.expanded { transform: rotate(90deg); }
|
||
.segment-name { font-weight: 600; font-size: 1.1em; flex: 1; }
|
||
.segment-count { font-size: 0.85em; color: #aeb0b4; }
|
||
|
||
.segment-content { padding: 18px; display: none; }
|
||
.segment-content.visible { display: block; }
|
||
|
||
.task-item {
|
||
background: #fafbfc;
|
||
border: 1px solid #e7ebef;
|
||
border-radius: 8px;
|
||
margin-bottom: 16px;
|
||
padding: 16px;
|
||
}
|
||
.task-header { margin-bottom: 14px; padding-bottom: 12px; border-bottom: 1px solid #e7ebef; }
|
||
.task-id {
|
||
font-family: 'SF Mono', Menlo, Consolas, monospace;
|
||
font-size: 0.92em;
|
||
color: #0969da;
|
||
font-weight: 600;
|
||
margin-bottom: 6px;
|
||
word-break: break-all;
|
||
}
|
||
.task-summary {
|
||
font-size: 1em;
|
||
color: #1d1d1f;
|
||
background: #fff;
|
||
padding: 12px 14px;
|
||
border-radius: 6px;
|
||
border-left: 4px solid #1f6fb5;
|
||
}
|
||
|
||
.rubrics-container {
|
||
display: grid;
|
||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
||
gap: 14px;
|
||
}
|
||
@media (max-width: 1100px) {
|
||
.rubrics-container { grid-template-columns: 1fr; }
|
||
}
|
||
|
||
.rubric-column {
|
||
background: #fff;
|
||
border: 1px solid #e7ebef;
|
||
border-radius: 8px;
|
||
overflow: hidden;
|
||
display: flex;
|
||
flex-direction: column;
|
||
}
|
||
.rubric-title {
|
||
color: #fff;
|
||
padding: 10px 14px;
|
||
font-weight: 600;
|
||
font-size: 0.98em;
|
||
text-align: center;
|
||
}
|
||
.rubric-title.v1 { background: #6c7a8a; }
|
||
.rubric-title.v2 { background: #3b7d3f; }
|
||
.rubric-title.v3 { background: linear-gradient(135deg, #0f2c4e 0%, #1f6fb5 100%); }
|
||
|
||
.rubric-content {
|
||
padding: 12px;
|
||
max-height: 620px;
|
||
overflow-y: auto;
|
||
flex: 1;
|
||
}
|
||
.rubric-empty {
|
||
color: #8b949e;
|
||
font-style: italic;
|
||
font-size: 0.9em;
|
||
padding: 16px;
|
||
text-align: center;
|
||
}
|
||
|
||
.rubric-item {
|
||
background: #f6f8fa;
|
||
border: 1px solid #eaecef;
|
||
border-radius: 6px;
|
||
padding: 10px 12px;
|
||
margin-bottom: 10px;
|
||
}
|
||
.rubric-item-header {
|
||
display: flex;
|
||
flex-wrap: wrap;
|
||
align-items: baseline;
|
||
gap: 6px;
|
||
margin-bottom: 6px;
|
||
}
|
||
.criterion-label { font-weight: 600; color: #57606a; font-size: 0.85em; }
|
||
.criterion-text { font-weight: 600; color: #1d1d1f; flex: 1; }
|
||
.max-points {
|
||
font-size: 0.8em;
|
||
color: #6a737d;
|
||
background: #fff;
|
||
padding: 2px 8px;
|
||
border-radius: 999px;
|
||
border: 1px solid #d0d7de;
|
||
white-space: nowrap;
|
||
}
|
||
.rubric-item-description {
|
||
color: #30363d;
|
||
font-size: 0.92em;
|
||
line-height: 1.5;
|
||
}
|
||
.description-label { font-weight: 600; color: #57606a; font-size: 0.82em; display: block; margin-top: 4px; }
|
||
|
||
footer {
|
||
margin-top: 30px;
|
||
text-align: center;
|
||
font-size: 0.85em;
|
||
color: #8b949e;
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<div class="header">
|
||
<h1>WebTailBench Rubric Comparison</h1>
|
||
<p>
|
||
Side-by-side view of three generations of rubrics for the
|
||
<strong>WebTailBench</strong> benchmark. Each row shows the rubric
|
||
criteria three different verifier judges produced for the same task.
|
||
</p>
|
||
<div class="links">
|
||
<a href="https://huggingface.co/datasets/microsoft/WebTailBench" target="_blank" rel="noopener">↗ microsoft/WebTailBench on Hugging Face</a>
|
||
<a href="https://huggingface.co/datasets/microsoft/WebTailBench/resolve/main/WebTailBench-v1-rubrics.tsv" target="_blank" rel="noopener">↓ Download WebTailBench-v1-rubrics.tsv</a>
|
||
<a href="https://huggingface.co/datasets/microsoft/WebTailBench/discussions/8" target="_blank" rel="noopener">🔀 Rubric PR discussion</a>
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="controls">
|
||
<input id="search" type="text" placeholder="Filter by task id, summary, criterion or description…" />
|
||
<label><input id="only-all-three" type="checkbox"> tasks with all three rubrics only</label>
|
||
<div class="stat">
|
||
609 tasks · 559 with all three rubrics · 50 only in the UV release
|
||
</div>
|
||
</div>
|
||
|
||
<div class="segments-container">
|
||
|
||
<div class="segment">
|
||
<div class="segment-header" data-seg="compositional_tasks_v2">
|
||
<span class="segment-toggle" id="toggle-compositional_tasks_v2">▶</span>
|
||
<span class="segment-name">compositional_tasks_v2</span>
|
||
<span class="segment-count">(55 tasks)</span>
|
||
</div>
|
||
<div class="segment-content" id="content-compositional_tasks_v2">
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_116</div>
|
||
<div class="task-summary">Check Steam for the first top-selling game today that has a TV series adaptation if any, then use JustWatch.com to find streaming services for the series adaptation.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Steam top-selling games list for today</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to Steam and retrieve the list of today's top-selling games. Partial credit could be awarded for finding a Steam sales or top-sellers page but not specifically for today's list.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first top-selling game with a TV series adaptation</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the retrieved list, correctly determine the first game that has an existing TV series adaptation. Partial credit may be given for naming a game that is a top-seller but misidentifying its adaptation status, or noting correctly that none exist.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Use JustWatch.com to list streaming services for the series</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search JustWatch.com for the identified TV series adaptation and list the streaming platforms where it is available. Partial credit for listing some but not all current providers, or for correctly stating that the series is not found or not available on streaming.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Check Steam Top Sellers for today</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Access Steam and view today's Top Sellers list to establish the current ranking. Partial credit if Steam is accessed and the Top Sellers section is located but 'today' cannot be confirmed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first top-selling game with a TV series adaptation (if any)</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From today's Top Sellers ranking, find the highest-ranked (first) game that has a TV series adaptation. Provide the game title and confirm that the adaptation is a TV series. Full credit also awarded if none of the top sellers have a TV series adaptation today and this is clearly stated. Partial credit if a game is identified but not verified as a TV series adaptation, or if ranking/order is not respected.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Use JustWatch.com to find streaming services for the series adaptation</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search JustWatch.com for the identified TV series adaptation and list the streaming services where it is available. Partial credit if JustWatch is used but the list is incomplete or availability cannot be fully determined.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Steam Top Sellers list for today</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to open Steam’s Top Sellers/Top Selling games list as of today. Full credit if the agent makes a reasonable attempt to access the correct Steam list but is blocked by CAPTCHA, region restrictions, outage, or login/age gate and clearly reports what prevented verification. Partial credit if the agent uses a nearby but not equivalent Steam list (e.g., Popular/Trending) while explaining the limitation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the #1 game on Steam Top Sellers (today)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
If the Top Sellers list is accessible, correctly report the top-ranked (#1) game shown for today/time of check. Full credit if the #1 game is captured from the Steam Top Sellers page; partial credit if the agent provides a plausible top seller but the source/timeframe is unclear. Full credit if identification is impossible solely because Steam access was blocked and the agent clearly states that the #1 game could not be confirmed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Determine whether the #1 game has a TV series adaptation (if any)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Assess whether the identified #1 game has a TV series adaptation and state the conclusion. Full credit if the agent correctly determines either that a TV series adaptation exists (naming it) or that none exists, with reasonable support. Full credit if the agent cannot confidently confirm due to inaccessible/conflicting sources and explicitly reports what was checked and the remaining uncertainty.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Use JustWatch.com to locate the series adaptation (if it exists)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
If a TV series adaptation exists, attempt to find that series on JustWatch.com. Full credit if the agent successfully locates the correct series entry on JustWatch, or if JustWatch is inaccessible (CAPTCHA/outage/forced login/consent flow that prevents search) and the agent clearly reports the blocker. Partial credit if the agent relies on non-JustWatch sources despite JustWatch being apparently accessible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Report streaming services shown on JustWatch for the series adaptation</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Report the streaming services where the series is available as shown on JustWatch (subscription vs rent/buy if presented; at minimum list the services). Full credit if the agent lists the complete set of services visible for the user’s JustWatch locale. Full credit if JustWatch does not show availability for that title/locale or availability cannot be loaded due to location settings/consent/technical issues and the agent clearly reports this limitation and what was/was not visible. Partial credit if only some services are listed when more are shown.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_23</div>
|
||
<div class="task-summary">On Eventbrite.com, find a live music event in Nashville, TN happening this upcoming Saturday. Then on Spotify.com, find a songs by any of the performing artists from that event, if any. </div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Eventbrite.com</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or open the Eventbrite.com website as specified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Locate a live music event in Nashville, TN for upcoming Saturday</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search within Eventbrite for events in Nashville, TN and identify at least one live music event scheduled for the upcoming Saturday. Partial credit for finding a relevant event but with missing date confirmation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify performing artist(s) from the selected event</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Extract the name(s) of the artist(s) performing at the chosen event. Partial credit if only some artists are identified or names are ambiguous.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Access Spotify.com</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or open the Spotify.com website as specified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Find songs by the identified artist(s) on Spotify</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search for and list songs by one or more of the previously identified performing artist(s) on Spotify. Partial credit for finding at least one song or partial artist match.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Eventbrite.com website</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or use the Eventbrite.com website as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Locate a live music event in Nashville, TN</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find an event that is explicitly live music and located in Nashville, Tennessee on Eventbrite.com. Partial credit may be awarded if an event in Nashville is found but the 'live music' aspect is unclear, or a live music event is found but the location is not clearly Nashville. Full credit is awarded if no such event exists and the agent clearly reports that.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Confirm event occurs this upcoming Saturday</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Ensure the selected event is scheduled for the upcoming Saturday relative to the current date. Partial credit may be awarded if the event is on a Saturday but not clearly the upcoming one, or the date is ambiguous. Full credit is awarded if no qualifying events exist and the agent explicitly states that.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Identify performing artist(s) from the event listing</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Extract the performing artist names as listed on the Eventbrite event page. Partial credit may be awarded if artist information is limited or not listed, provided that this is clearly stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Access Spotify.com website</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to or use Spotify.com as specified in the task (without requiring login or other personal information).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Find songs by any of the identified performing artists on Spotify.com (if any)</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search for the identified artist(s) on Spotify.com and find at least one song (track) by any of them. Partial credit may be awarded if the artist page is found but specific song titles are not identified. Full credit is awarded if no songs exist or cannot be found and the agent clearly states that.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Eventbrite.com and search Nashville, TN live music listings</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to navigate to Eventbrite.com and perform a search for events in Nashville, TN that are described/categorized as live music. Full credit if the agent makes a reasonable attempt but is blocked by a CAPTCHA/login wall/site outage and clearly reports the blocker. Partial credit if the agent searches the wrong location or does not clearly use Eventbrite first.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Locate a Nashville, TN live music event occurring this upcoming Saturday (Eventbrite result selection)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From Eventbrite results/event pages, identify at least one event that is explicitly live music, located in Nashville, TN, and scheduled for the upcoming Saturday (relative to execution date). Full credit if an exact match is found OR if, after reasonable search/filtering, no exact match appears to exist and the agent clearly reports that (optionally providing the closest available live-music Nashville alternative and explaining the mismatch). Partial credit if an event is live music in Nashville but on a different date, or on the correct Saturday but outside Nashville, when closer matches are available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify performing artist(s) listed on the selected Eventbrite event page</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Extract and report the performing artist name(s) as listed on the Eventbrite event page. Full credit if at least one performer is correctly identified OR if the event page does not list performers (or only lists a venue/DJ night without a clearly named act) and the agent explicitly states that limitation. Partial credit if the agent provides an ambiguous performer identification while noting uncertainty, or mistakes a venue/organizer for an artist when the performer is actually listed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Use Spotify.com to find at least one song by any identified performing artist (if any)</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use Spotify.com to search for at least one of the identified performers and provide at least one song by a correctly matched artist. Full credit if a correct song is found OR if Spotify is inaccessible (CAPTCHA/login wall/site error) and the agent reports the blocker OR if Spotify is accessible but the performer cannot be found/does not appear to have a Spotify catalog and the agent clearly reports that outcome after reasonable search (including disambiguation attempts such as adding location/genre). Partial credit if the agent finds an artist page but does not name any song, or returns a similarly named but unverified/likely incorrect artist without noting uncertainty.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_78</div>
|
||
<div class="task-summary">Look at the amazon page for "The Innovator's Dilemma", see what it ranks in books overall, and then find a repair service anywhere in the US whose phone number contains that rank as a sub-string. Output the name and phone number of that repair service.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate Amazon page for 'The Innovator's Dilemma'</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully find and access the Amazon product page for the book titled 'The Innovator's Dilemma'.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Extract overall books ranking</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify and record the book’s current 'Best Sellers Rank' in the Books category as shown on its Amazon page.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find US repair service with matching phone substring</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search for and identify a repair service located in the United States whose telephone number contains the rank number (as a continuous substring).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Output service name and phone number</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the exact name of the repair service and its full phone number that matches the rank substring requirement.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access the Amazon page for 'The Innovator's Dilemma'</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to and view the specific Amazon listing for the book 'The Innovator's Dilemma'. Partial credit may be awarded if the agent clearly attempts to find the correct page but cannot access it or identifies a closely related listing.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the book's overall rank in Books on Amazon</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and extract the book's overall rank in the Books category from the Amazon page. Partial credit may be given if category-specific ranks are reported instead of the overall Books rank, or if the agent explicitly notes that the overall rank is not available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find a US-based repair service</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify a repair service located anywhere in the United States (any type of repair service qualifies). Partial credit may be given if a repair service is found but the US location is unclear or not verified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Verify the phone number contains the rank as a substring</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Confirm that the repair service's phone number includes the exact sequence corresponding to the extracted rank as a substring. Partial credit may be granted if there is a close match but with formatting ambiguities, or if a reasonable attempt is made but the substring match is incorrect.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Output the repair service's name and phone number</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the name and phone number of the selected repair service. Partial credit may be awarded if only one of these (name or phone number) is provided.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access the Amazon product page for "The Innovator's Dilemma"</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to navigate to the Amazon product page for "The Innovator's Dilemma". Full credit if the agent attempts access but is blocked by CAPTCHA/login/region gating/page load failures and clearly reports the blocker. Partial credit if the agent navigates to an Amazon results page or a different edition without clarifying uncertainty.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the book's overall Amazon Books rank</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the Amazon product page (if accessible), extract the rank in Books overall (not a category/subcategory rank) and record it exactly as shown. Full credit if the agent provides the exact overall Books rank; also full credit if the rank is not visible/unstated due to page variability or blockers and the agent explicitly explains why it cannot be determined. Partial credit if the agent provides only a category rank but clearly flags it as such or explains the ambiguity. No credit if the rank is fabricated or sourced off-Amazon without attempting Amazon first.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find a US repair service whose phone number contains the rank substring</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the extracted overall Books rank digits as a contiguous substring, search for a repair service located in the US with a publicly listed phone number containing that exact substring. Full credit if a verifiable repair service is found and its phone number contains the substring; also full credit if the agent demonstrates reasonable search effort (e.g., multiple queries/sources) and reports that no matching phone number could be found. Partial credit if a repair service is found but the digit match is incorrect (non-contiguous/mismatched) or if search effort is minimal/unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Output the repair service name and phone number (or clearly report no match)</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the final answer with the repair service name and its phone number that contains the rank substring. Full credit if both are provided and correspond to a verified match. If criterion 3 concludes no match exists after reasonable search, full credit for explicitly stating that no qualifying repair service phone number was found (rather than inventing one). Partial credit if only the name or phone is provided, or if the pairing is inconsistent.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_121</div>
|
||
<div class="task-summary">On Wikipedia.org, look up Harvard University to find its location; then on Google Maps, get walking directions to Boston City Hall from this location.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access the Harvard University page on Wikipedia</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to wikipedia.org and locate the official Harvard University article. Partial credit if the agent reaches Wikipedia but not the correct page; full credit for accurately accessing the Harvard University entry.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Extract Harvard University's location</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify and report the location information of Harvard University as stated on its Wikipedia page (e.g., city, state, country). Partial credit for partially correct location (e.g., city but missing state), full credit for the complete correct location.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Retrieve walking directions to Boston City Hall on Google Maps</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Go to Google Maps, input the Harvard University location as the origin and Boston City Hall as the destination, select walking directions, and provide the route (including major steps, total distance, and estimated walking time). Partial credit for finding the route but omitting details; full credit for complete, accurate walking directions.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Look up Harvard University on Wikipedia.org</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Access Wikipedia.org and open the Harvard University article as explicitly requested. Partial credit may be awarded for reaching Wikipedia but not the correct page, or demonstrating attempts to find the correct page.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find Harvard University's location from Wikipedia</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify Harvard University's location information from the Wikipedia article (e.g., Cambridge, Massachusetts; address/campus location if available). Partial credit may be awarded for correctly identifying the city/state without a more precise address. Full credit is awarded if the agent indicates that detailed location is not available on the page.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Get walking directions on Google Maps to Boston City Hall from the found location</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Google Maps to obtain walking directions from the Wikipedia-derived Harvard University location to Boston City Hall. Full credit includes using walking mode, correct origin and destination, and providing the resulting directions outcome (e.g., route/distance/duration or a directions link). Partial credit may be awarded if directions are provided but not in walking mode, the origin is approximate, or a different mapping service is used. Full credit should be awarded if Google Maps indicates directions are unavailable and the agent clearly states that.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find Harvard University location on Wikipedia.org</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Wikipedia.org to look up the 'Harvard University' article and identify its stated location (e.g., Cambridge, Massachusetts, United States). Full credit if the agent uses Wikipedia and reports the correct city/state/country (or equivalent specificity) as shown on the page. Partial credit if the agent uses Wikipedia but reports an incomplete/ambiguous location (e.g., only 'Massachusetts' or only 'Harvard University'). Full credit if Wikipedia is inaccessible (blocked/down/CAPTCHA) and the agent clearly reports the issue and uses a reasonable alternative source to determine the location, stating that it is an alternative.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Obtain walking directions on Google Maps from the Wikipedia-derived Harvard location to Boston City Hall</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
On Google Maps, attempt to obtain directions with the origin set to the Harvard University location found in the prior step and the destination set to 'Boston City Hall', with travel mode set to walking. Full credit if the agent correctly sets origin/destination and selects walking mode, OR if Google Maps is inaccessible/fails to load directions and the agent clearly reports the blocker and provides the best available alternative method/provider for walking directions (or clearly states that walking directions could not be retrieved). Partial credit if directions are obtained but the travel mode is not walking, or if the origin is materially imprecise/mismatched to the Wikipedia-derived location when a more precise origin is available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Report the resulting walking directions details (time and distance)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
If walking directions are successfully retrieved from Google Maps (or a clearly stated alternative due to Google Maps failure), report at least total walking time and total distance (optionally include main streets). Full credit if both time and distance are reported. Partial credit if only one of time or distance is reported. If directions could not be retrieved due to external blockers and the agent clearly reported that in the prior step, do not penalize here (award full credit). No credit if the agent fabricates time/distance without evidence or fails to report available time/distance from the retrieved directions.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_62</div>
|
||
<div class="task-summary">Locate a coding bootcamp company in brooklyn, NYC, and tell me how much full-time tuition would cost there. Then use Google Maps to tel lme which bus I can take from Grand Army Plaza to reach there. Output the name of the bootcamp, the tuition cost, and the bus service name.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate a coding bootcamp in Brooklyn, NYC</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify an existing coding bootcamp company located in Brooklyn, New York City. Partial credit if a Brooklyn-based tech school is identified but it is unclear whether it is primarily a coding bootcamp.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Determine full-time tuition cost</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the cost of the full-time tuition for the identified Brooklyn coding bootcamp. Partial credit for an approximate range if an exact figure is not available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find appropriate bus route from Grand Army Plaza</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Google Maps (or equivalent mapping service) to identify a bus service that runs from Grand Army Plaza to the bootcamp location. Partial credit if a route is identified but with transfers or disclaimers.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Output required information</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Present the bootcamp name, full-time tuition cost, and the bus service name in the response as specified by the task.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify a coding bootcamp located in Brooklyn, NYC</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and name a coding bootcamp company that has a location in Brooklyn, NYC. Partial credit may be awarded if a coding bootcamp is identified but the location is ambiguous or not in Brooklyn.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Obtain full-time tuition cost for the bootcamp</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the tuition cost for the bootcamp's full-time program at the identified Brooklyn location. Full credit if the exact full-time tuition is given or if it is clearly stated that no full-time program exists (with that unavailability noted). Partial credit may be awarded for a close estimate, a range, or a tuition figure from a different program with that distinction clearly noted.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Use Google Maps to determine a bus option from Grand Army Plaza</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify which bus service (route name/number) can be taken from Grand Army Plaza to reach the bootcamp, using Google Maps. Full credit requires a bus service name derived via Google Maps for the specified origin and destination. Partial credit may be awarded for a plausible bus route without explicit mention of Google Maps or with incomplete details.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Provide the required outputs clearly</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Explicitly output the bootcamp name, the full-time tuition cost, and the bus service name as requested. Partial credit may be awarded if one or two of these items are provided but not all three.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate a coding bootcamp company in Brooklyn, NYC</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify at least one coding bootcamp company that is located in Brooklyn, NYC (address/neighborhood indicates Brooklyn). Full credit if the bootcamp is clearly in Brooklyn. Full credit also if the agent makes a reasonable attempt to verify a Brooklyn location but finds the bootcamp has moved/closed or the location cannot be verified from accessible sources, and then clearly reports this and provides a reasonable alternative bootcamp that is verifiably in Brooklyn. Partial credit if the bootcamp is in NYC but the borough is unclear or not verified. No credit if the selected bootcamp is not in Brooklyn when verifiable Brooklyn options are available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Determine full-time tuition cost for that bootcamp</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and report the bootcamp's full-time tuition amount. Full credit if a specific numeric full-time tuition is provided and is clearly tied to the full-time program (including clearly stated mandatory fees if presented as part of tuition). Full credit also if the bootcamp does not publish full-time tuition (or it is not accessible due to paywalls/login/region gating) and the agent clearly states that the full-time tuition is not publicly available, optionally providing the best available related pricing info (e.g., range, ISA terms) with appropriate caveats. Partial credit if only a range or ambiguous/outdated figure is provided without clarifying uncertainty.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Use Google Maps to identify the bus from Grand Army Plaza to the bootcamp</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using Google Maps directions (Transit), determine a bus service/route that can be taken from Grand Army Plaza to reach the selected bootcamp location. Full credit if (a) a specific MTA bus route/service name (e.g., B41, B45, B67) is provided and is plausibly part of the Google Maps transit itinerary, OR (b) Google Maps is inaccessible (captcha/outage) and the agent clearly reports the blocker and provides the best available alternative bus route information from another credible transit source while explicitly noting it is not from Google Maps, OR (c) Google Maps transit directions do not include any bus leg (or show no feasible bus option) and the agent clearly reports that outcome and provides the closest feasible transit alternative shown by Google Maps. Partial credit if only general guidance is given (e.g., 'take a bus') or if the bus route is incomplete/unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Provide the required final outputs</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Output includes all three explicitly requested items: (1) bootcamp name, (2) full-time tuition cost (or a clear statement that it is not publicly available), and (3) bus service name (or a clear statement that Google Maps provides no bus option / Google Maps inaccessible with noted alternative source). Full credit if all three are present and correspond to the same selected bootcamp/directions (or if a required item is unavailable but the agent clearly reports the limitation as described above). Partial credit if one of the three is missing or not clearly labeled. No credit if two or more are missing or mismatched (e.g., bus route for a different destination than the named bootcamp).
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_89</div>
|
||
<div class="task-summary">Go to lettuce.com and find the first restaurant after filtering their portfolio for spanish cuisine, then go their website to order, and add the 4 most commonly-ordered items to the cart and proceed to checkout. Also output and the prices of those 4 items.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access lettuce.com website</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or access the lettuce.com website as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Filter portfolio for Spanish cuisine</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Apply the Spanish cuisine filter on lettuce.com to display only Spanish restaurants. Partial credit may be awarded for attempting the filter but not fully limiting results to Spanish cuisine.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the first restaurant</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Select or identify the first restaurant listed after applying the Spanish cuisine filter. Partial credit if a different restaurant is selected but the process of choosing the first listing is shown.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Navigate to the restaurant's website</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Click through from lettuce.com to the identified restaurant's own website. Partial credit if the link is located but fails to load, with the failure noted.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Add the 4 most commonly-ordered items to the cart</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
On the restaurant's website, determine which four menu items are most commonly ordered and add all four to the cart. Partial credit for fewer items or unclear identification of popularity.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Output names and prices of the 4 items</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
List the names and prices of the four items added to the cart. Partial credit if names are correct but some prices are missing or incorrect.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 7:</span>
|
||
<span class="criterion-text">Proceed to the checkout page</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the checkout page without entering personal or payment information. Partial credit if the checkout button or link is located but not clicked.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access lettuce.com</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or access the lettuce.com website as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Filter portfolio for Spanish cuisine</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use the site's portfolio filter to select Spanish cuisine. Partial credit if the agent attempts the filter but it is unavailable or yields no results and this is clearly stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the first restaurant after filtering</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Correctly identify the first restaurant listed under the Spanish cuisine filter. Partial credit if a Spanish restaurant is identified but not necessarily the first, or if ordering is impossible because none are listed and this is reported.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Go to the restaurant's website and locate ordering</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate from the portfolio to the identified restaurant’s website and locate the ordering interface (on-site or via a linked ordering platform). Partial credit if only the restaurant website is reached without a clear ordering path, with this limitation noted.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Add four most commonly-ordered items to the cart</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the ordering interface, identify and add four items labeled as most popular/commonly ordered to the cart. Partial credit for fewer than four items, or if such labels are absent, for selecting a reasonable equivalent and stating that limitation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Proceed to checkout (without completing purchase)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the checkout or order review page, stopping before entering any personal or payment information. Partial credit for reaching the cart page if the platform requires login or personal info to proceed further and this is stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 7:</span>
|
||
<span class="criterion-text">Output names and prices of the four items</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the names and prices of the selected four items as displayed on the ordering interface. Partial credit if some prices are missing due to unavailability and this is clearly noted.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access lettuce.com and reach the portfolio/listing area (or report blocker)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use lettuce.com as the starting platform and attempt to reach the portfolio/listing area where cuisine filters can be applied. Full credit if the portfolio/listing area is reached, OR if access is blocked (captcha, outage, geo restriction, access wall) and the agent clearly reports the blocker and what was attempted. Partial credit if the agent switches to alternative sources without first attempting lettuce.com.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Filter lettuce.com portfolio for Spanish cuisine and identify the first resulting restaurant (or report none/ambiguity)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Apply the Spanish cuisine filter (or the closest available equivalent, e.g., 'Spain/Spanish') on lettuce.com's portfolio and identify the first restaurant in the filtered results as displayed. Full credit if the filter is applied and the first visible result is identified. Full credit if the filtered results are empty and the agent clearly reports that. Full credit if the site’s ordering is ambiguous/unstable (e.g., no clear sort order, infinite scroll, personalization) and the agent clearly explains how 'first' was interpreted (e.g., topmost visible result) and proceeds accordingly. Partial credit if a Spanish restaurant is selected without demonstrating that the Spanish filter was used when it was available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Go to the identified restaurant's official website/official ordering link and reach an ordering interface (or report blocker)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the restaurant identified on lettuce.com, navigate to the restaurant's official website or the official online ordering page linked from it and reach the point where menu items can be added to a cart. Full credit if the ordering interface is reached. Full credit if the restaurant has no online ordering or ordering is unavailable (closed hours, delivery disabled, location selection required, login wall) and the agent clearly reports what prevented progress and any visible alternatives (phone/in-person/third-party) without fabricating availability. Partial credit if the agent uses an unofficial/third-party ordering site without first attempting the restaurant's own official path when available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Add most commonly-ordered item #1 to cart (or report inability/label absence)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Add one distinct item that is among the four most commonly-ordered/most popular/top ordered items as indicated by the ordering UI. Full credit if added. If the site does not display 'most commonly-ordered' (or only shows it after selecting location/time), full credit if the agent clearly reports the absence/constraint and uses the best available proxy exposed by the platform (e.g., 'Most Popular' section, 'Featured', or top items shown) and explains the choice. Partial credit if the correct item is identified but cannot be added due to external blockers (sold out, ordering disabled) that are clearly reported.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Add most commonly-ordered item #2 to cart</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Add a second distinct commonly-ordered/popular item to the cart, completing required modifiers using defaults or reasonable selections when forced. Full credit if added or if prevented by external constraints that are clearly reported (sold out, modifier bug, ordering closed). Partial credit if added but required selections are clearly incomplete or invalid when a default/obvious choice existed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Add most commonly-ordered item #3 to cart</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Add a third distinct commonly-ordered/popular item to the cart. Full credit if added or if prevented by uncontrollable constraints (sold out, ordering closed, item unavailable at selected location) that are clearly reported and the agent selects the next-best commonly-ordered alternative if available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 7:</span>
|
||
<span class="criterion-text">Add most commonly-ordered item #4 to cart</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Add a fourth distinct commonly-ordered/popular item to the cart. Full credit if added or if prevented by uncontrollable constraints that are clearly reported and the agent selects the next-best commonly-ordered alternative if available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 8:</span>
|
||
<span class="criterion-text">Proceed to checkout without crossing the critical point</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Proceed from the cart to the checkout flow (e.g., checkout page, delivery/pickup selection, order summary) but stop before entering any personal data (name, email, phone, address) or payment information, and do not place the order. Full credit if the agent reaches checkout and stops appropriately, OR if the site requires login/personal/payment details before showing checkout and the agent reports this blocker and stops. No credit if the agent places the order or enters sensitive personal/payment details not provided by the user.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 9:</span>
|
||
<span class="criterion-text">Output the prices of the 4 added items</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the displayed prices for each of the four added items (item name paired with its line-item/base price as shown in the cart/checkout). Full credit if all four are reported accurately as displayed, including any required-option price changes if applicable (or clearly noting when price varies by required options/location). Partial credit if fewer than four prices are provided or if prices are reported without clarifying that required modifiers/size affected the price when that ambiguity is visible.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_6</div>
|
||
<div class="task-summary">On Booking.com, find the cheapest available 8/10+ scored hotel room for a three-night stay starting December 15, 2025, in Jakarta for 2 adults. Use the hotel's address to search for the closest coffee shop, output it's name and address.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Booking.com website</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the Booking.com website as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Search for hotels in Jakarta for specified dates and occupancy</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Perform a search on Booking.com for hotels in Jakarta for a 3-night stay starting December 15, 2025 for 2 adults.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Filter results by review score ≥ 8/10</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Apply a filter to include only hotels with a review score of 8/10 or higher.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Identify the cheapest available room</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Among the filtered hotels, find the room option with the lowest total price for the specified stay.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Retrieve the selected hotel's address</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Extract the full address of the chosen hotel from its listing.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Find the closest coffee shop using hotel address</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use the hotel's address to search for and identify the nearest coffee shop.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 7:</span>
|
||
<span class="criterion-text">Provide coffee shop name and address</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Output the name and full address of the closest coffee shop found.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Configure Booking.com search for the specified stay</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Booking.com to search in Jakarta with check-in on December 15, 2025 for a three-night stay (Dec 15–18, 2025) for 2 adults. Full credit if all parameters (site, city, dates, length, and guests) are correctly set. Partial credit if some parameters are correct. Full credit also if the agent reports that no results are available for the exact parameters.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Apply or satisfy the 8/10+ review score requirement</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Ensure that only hotels with a review score of at least 8/10 are considered. Full credit if an explicit filter is applied or the chosen hotel clearly meets 8/10+. Partial credit if the hotel appears highly rated but the threshold is not clearly verified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the cheapest available qualifying hotel room</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the qualifying results, identify the cheapest available hotel room for the specified dates and party size. Full credit if the selected option is the cheapest among those meeting the criteria and is available for the dates. Partial credit if a valid option is found but not confirmed as the cheapest, or if no qualifying options exist and this is clearly stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Extract the selected hotel's address</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the hotel's street address obtained from Booking.com (or the hotel’s official listing) to use for the proximity search. Partial credit if only a partial or approximate address is provided.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Find and output the closest coffee shop to the hotel</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the hotel's address, search for the nearest coffee shop and output the coffee shop's name and address. Full credit requires identifying the closest option and providing both name and address. Partial credit if a nearby coffee shop is provided but not clearly the closest, or if either the name or address is missing.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Booking.com and set the required search parameters for Jakarta stay</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use Booking.com and set: destination Jakarta; check-in Dec 15, 2025; check-out Dec 18, 2025 (or 3 nights); 2 adults. Full credit if the agent clearly attempts to use Booking.com with these parameters or explicitly notes an equivalent UI flow (e.g., selecting 3 nights). Full credit if Booking.com is inaccessible (CAPTCHA, outage, login wall) and the agent clearly reports the blocker and what was attempted. Partial credit if minor parameter mismatch occurs but is clearly disclosed and corrected, or if the attempt is unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Apply/verify the Booking.com review score constraint (8.0/10+)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Apply a review-score filter (8.0+) or otherwise verify from Booking.com that the chosen property is rated at least 8.0/10. Full credit if enforced via filters or verified on the property page. Full credit if, after a reasonable attempt, no 8.0+ properties appear available for the dates/guests and the agent clearly reports this. Partial credit if a score is mentioned but the Booking.com source/threshold is not clearly confirmed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the cheapest available qualifying room for the full 3-night stay (or report unavailability/blocker)</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the Booking.com results consistent with the parameters and 8.0+ constraint, identify the lowest-priced available room option for the entire 3-night stay, clearly stating whether the price shown is total vs per-night and whether taxes/fees are included as displayed. Full credit if the agent demonstrates a reasonable comparison among visible 8.0+ options and selects the cheapest shown. Full credit if no qualifying availability exists (or prices cannot be retrieved) and the agent clearly reports this with evidence from the Booking.com attempt; optionally, it may provide the best available alternative (e.g., closest-to-cheapest among 8.0+ or cheapest below 8.0) while explicitly noting the deviation. Partial credit if the comparison is unclear or the price basis (total vs nightly / taxes) is not stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Provide the selected hotel's address (as shown on Booking.com) or explain why it can’t be retrieved</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Report the hotel's physical address as displayed on Booking.com for the selected property. Full credit for a complete address (street/area + city; postal code if shown). Full credit if the agent cannot retrieve the address due to a Booking.com blocker/limited listing details and explicitly states this while providing the best available location information shown (e.g., neighborhood, map pin area) without fabrication. Partial credit if the address is materially incomplete but still plausibly identifies the location.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Find the closest coffee shop using the hotel's address; output coffee shop name and address (or report blocker/ambiguity)</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the hotel address as the anchor, attempt to find the nearest coffee shop via a maps/search tool and output the coffee shop’s name and address. Full credit if the agent clearly bases the search on the hotel address and provides both name and address. Full credit if map/search tools are inaccessible or results are ambiguous (e.g., multiple equidistant options, address too imprecise) and the agent reports the blocker/ambiguity and provides the best-supported nearest option(s) with an explanation of the basis used (e.g., closest shown by the tool). Partial credit if only name or only address is provided, or if “closest” is asserted without any clear basis when a basis was available.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_87</div>
|
||
<div class="task-summary">on bklynlibrary.org find the northern-most library branch that has a teen tech help center, then find the year that branch opened to the public, how many square feet of space it has, and who the managing librarian is.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access the bklynlibrary.org website</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or access the bklynlibrary.org website as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify branches with a teen tech help center</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and list all Brooklyn Public Library branches that offer a teen tech help center. Partial credit may be given if only some are identified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Determine the northern-most branch</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the branches identified, correctly determine which one is geographically the northern-most. Partial credit for approximate selection.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Report the year the branch opened</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and report the exact year that the northern-most branch opened to the public. Partial credit for a close or approximate year.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Report the branch's square footage</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and report the official square-footage of the northern-most branch. Partial credit for a close estimate.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Report the managing librarian's name</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify and report the name of the managing librarian for the northern-most branch. Partial credit for a close or partial name.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access bklynlibrary.org</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or access the bklynlibrary.org website as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify northern-most branch with a Teen Tech Help Center</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find library branches on bklynlibrary.org that have a Teen Tech Help Center and correctly determine which one is the northern-most. Partial credit may be awarded for locating relevant branches but failing to identify the northern-most, or for a reasonable attempt if the site lacks clear geographic cues. Full credit is awarded if the site indicates no such centers exist and the agent clearly states that.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the year the identified branch opened to the public</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and report the year the identified branch opened to the public from bklynlibrary.org. Partial credit may be given for related opening information (e.g., renovation year) if the specific 'opened to the public' year is not available and this is clearly noted.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Find the branch's square footage</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and report the total square footage of the identified branch from bklynlibrary.org. Full credit is awarded if the site indicates this information is unavailable and the agent clearly notes that.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Find the managing librarian</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and report the name of the managing librarian for the identified branch from bklynlibrary.org. Full credit is awarded if the site indicates this information is unavailable and the agent clearly notes that.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access bklynlibrary.org and locate information about Teen Tech Help Center availability by branch</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use bklynlibrary.org (site search, branch listings, and/or individual branch pages) to determine which branch(es) have a Teen Tech Help Center. Full credit if the agent attempts access and clearly reports if blocked (captcha/paywall/outage) or if Teen Tech Help Center information cannot be located on the site after reasonable searching. Partial credit if the agent uses bklynlibrary.org but the attempt is superficial/unclear. No credit if the agent does not attempt bklynlibrary.org while it appears accessible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Use bklynlibrary.org as the source to identify branches with a Teen Tech Help Center</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify at least one branch explicitly indicated on bklynlibrary.org as having a Teen Tech Help Center. Full credit if the qualifying branch list is correctly drawn from bklynlibrary.org pages. Partial credit if the agent mixes in non-bklynlibrary sources but still correctly identifies qualifying branches and indicates which claims are from bklynlibrary.org. Full credit if the site is accessible but it appears bklynlibrary.org does not provide any Teen Tech Help Center-by-branch information and the agent clearly states that finding.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Correctly determine the northern-most branch that has a Teen Tech Help Center</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the bklynlibrary.org-identified set of branches with a Teen Tech Help Center, select the geographically northern-most branch. Full credit if the selection is correct given the available location/address information on bklynlibrary.org. If bklynlibrary.org does not provide enough information to unambiguously rank branches by latitude (or addresses are missing/unclear), award full credit if the agent clearly explains the ambiguity, shows reasonable comparison effort (e.g., comparing addresses/neighborhoods), and provides the best defensible choice. Partial credit if the agent selects a qualifying branch but provides no comparison/justification when comparison appears feasible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Find and report the year the identified branch opened to the public</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Report the year the selected branch opened to the public using bklynlibrary.org branch information. Full credit for the correct year when present. If bklynlibrary.org does not list an opening year (or only lists renovation/reopening dates without original opening), award full credit if the agent clearly states the information is not available/unclear on bklynlibrary.org after reasonable searching and does not invent a year.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Find and report the branch's square footage</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Report the branch's square footage as listed on bklynlibrary.org. Full credit for the correct square footage when present. If square footage is not provided on bklynlibrary.org (or is ambiguous between building vs. lot size), award full credit if the agent clearly reports that the value is missing/ambiguous on bklynlibrary.org after reasonable searching and avoids guessing.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Find and report the managing librarian for the branch</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Report the managing librarian name for the selected branch as shown on bklynlibrary.org. Full credit for the correct person and role when present. If managing librarian info is not available on bklynlibrary.org (or staff roles are not listed), award full credit if the agent clearly states it cannot be found there after reasonable searching and does not substitute another staff role without noting the mismatch.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 7:</span>
|
||
<span class="criterion-text">No hallucinated details; discrepancies or blockers are clearly stated</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Do not invent Teen Tech Help Center status, opening year, square footage, or managing librarian. Full credit if all reported facts are consistent with bklynlibrary.org or the agent transparently reports blockers, missing fields, or ambiguity. Partial credit if minor ambiguity exists without clear attribution. No credit if key facts are fabricated or if the agent claims bklynlibrary.org support when it does not.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_81</div>
|
||
<div class="task-summary">Retrieve the lowest-price round-trip flight from Dallas (DFW) to Miami (MIA) on Jan 20, 2026, to Jan 25, 2026, using Google Flights. Noting the flight's arrival timestamp in miami, book the cheapest compact car from Miami International on Rentalcars.com beginning no less than one hour after the flight arrives. For the first result output the price per day, make/model, and number of seats.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Google Flights</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to Google Flights as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Search round-trip flight DFW→MIA for Jan 20–25, 2026</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Enter origin (DFW), destination (MIA), departure date (Jan 20, 2026) and return date (Jan 25, 2026) correctly in Google Flights.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify lowest-price flight</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and select the flight option with the lowest total round-trip price.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Record arrival timestamp in Miami</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Note the exact local arrival date and time of the chosen flight into MIA.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Access Rentalcars.com</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to Rentalcars.com as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Search compact car at MIA with correct pickup time</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search for compact cars for pickup at Miami International Airport, ensuring the pickup datetime is at least one hour after the recorded flight arrival.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 7:</span>
|
||
<span class="criterion-text">Identify cheapest compact car</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the search results, find the compact car option with the lowest daily rate.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 8:</span>
|
||
<span class="criterion-text">Extract price/day, make/model, and seats</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
For the first (cheapest) compact car result, output the daily price, the vehicle make and model, and the number of seats.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access and use Google Flights</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to and use Google Flights as explicitly required by the task. Partial credit may be awarded if the agent searches for flights but not via Google Flights. Full credit should still be awarded if Google Flights indicates no available flights for the specified dates and the agent reports that clearly.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Retrieve lowest-price round-trip DFW→MIA for Jan 20–25, 2026</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Configure a round-trip search from Dallas (DFW) to Miami (MIA) departing Jan 20, 2026 and returning Jan 25, 2026, and identify the lowest-price option. Partial credit if the correct route and dates are set but the agent does not confirm it is the lowest price, or provides incomplete pricing. Full credit if no flights are available and the agent states that explicitly.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Note the arrival timestamp in Miami</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify and clearly note the arrival timestamp of the outbound flight into Miami (local time). Partial credit if the arrival time is provided but ambiguously labeled or without local context.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Use Rentalcars.com and configure search at Miami International with timing and vehicle constraints</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to Rentalcars.com, set pickup location to Miami International, set the pickup time to begin no less than one hour after the flight’s arrival time, and filter/search for compact cars. Partial credit if the site and location are correct but the timing is not at least one hour after arrival, or if compact filtering is not applied. Full credit if compact cars are unavailable and the agent reports that clearly.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Identify/select the cheapest compact car (facilitate booking without completing checkout)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the configured search results, identify the cheapest compact car and initiate the booking flow up to selection (e.g., selecting the vehicle or proceeding to the next step) without entering personal information or completing checkout. Partial credit if the cheapest compact option is correctly identified but not selected.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Output details for the first car result</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the price per day, make/model, and number of seats for the first result in the search results list. Partial credit if only some of the required fields are provided.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Google Flights and run the specified round-trip search (DFW↔MIA, Jan 20–Jan 25, 2026)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use Google Flights to search a round-trip itinerary from Dallas (DFW) to Miami (MIA) departing Jan 20, 2026 and returning Jan 25, 2026. Full credit if the agent performs the search on Google Flights or clearly reports being blocked (CAPTCHA/outage/region restriction) after a reasonable attempt. Partial credit if the agent searches the wrong dates/airports or does not make clear what was searched.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the lowest-priced qualifying round-trip option (or best available alternative if none/blocked)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the available Google Flights results for the correct route/dates, identify the lowest-priced round-trip option visible at the time of search. Full credit if the cheapest visible option is selected OR if Google Flights results cannot be accessed and the agent uses a reasonable alternative source (e.g., airline site/other major flight aggregator) while preserving route/dates and explains why. Also award full credit if the agent clearly reports that no valid itineraries/prices are shown for those dates (e.g., error/no availability). Partial credit if an option is selected but it is not the cheapest when a cheaper one is clearly visible and no justification is given.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Report Miami arrival timestamp for the selected outbound flight (or explain if unavailable)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the arrival date/time in Miami for the selected outbound (DFW→MIA) flight. Full credit if the correct timestamp is reported. If the source does not display an arrival timestamp (or is blocked), full credit if the agent states this and provides the closest available equivalent (e.g., scheduled arrival window or asks for a preferred itinerary/airline to proceed). Partial credit if time is incomplete/ambiguous but leg/city are correct.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Access Rentalcars.com and search with pickup at Miami International at time ≥ 1 hour after arrival</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use Rentalcars.com with pickup location set to Miami International (MIA) and a rental start time no less than one hour after the reported flight arrival time. Full credit if the constraint is applied as stated, OR if Rentalcars.com cannot be accessed (blocked/outage) and the agent clearly reports the blocker after a reasonable attempt. Partial credit if the pickup location is correct but the time constraint is not met or the time is not clearly set.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Identify the cheapest compact car option and output price/day, make/model, and seats (or best available if compact unavailable)</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the results, select the cheapest option in the compact category (as shown) and output: price per day, make/model, and number of seats for the first/cheapest compact result. Full credit if all three fields are provided for the cheapest compact option visible. If no compact cars are available (or compact filtering is not possible/unclear on the platform), full credit if the agent clearly states this and selects the cheapest closest alternative category available (e.g., economy) while still providing the three required fields. Partial credit if one required field is missing or if the chosen car is not the cheapest available in the stated category when a cheaper one is clearly visible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Avoid crossing critical points (no final booking/purchase requiring personal or payment details)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Do not complete flight or car booking/purchase steps that require personal data, account login, or payment information. Full credit if the agent stops at a pre-checkout stage and does not enter fabricated personal/payment details. Partial credit if the agent navigates close to checkout but does not submit/confirm. No credit if the agent completes a booking/purchase or enters personal/payment information.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_56</div>
|
||
<div class="task-summary">find what xbox.com says is a top-selling xbox game; note who it was published by and the release date. Then tell me how many years have elapsed since when the CEO or head of that gaming studio was born and the release date.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate top-selling Xbox game on xbox.com</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate xbox.com and identify which game is listed as the top-selling Xbox title.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the game's publisher</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and report the name of the publisher for the top-selling Xbox game as stated on xbox.com.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the game's release date</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and report the official release date of the top-selling Xbox game.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Identify the CEO or head of the game's studio</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine who is currently the CEO or head of the studio that developed or published the top-selling Xbox game.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Determine the CEO's or studio head's birth date</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and report the birth date (day, month, year) of the identified CEO or studio head.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Calculate elapsed years between CEO birth and game release</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compute how many years have passed from the CEO or head's birth date to the game's release date.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify a top-selling Xbox game from xbox.com</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and name a game that xbox.com explicitly lists as a top-selling Xbox game. Full credit requires that the selection clearly comes from xbox.com. Partial credit if a likely candidate is identified without clear confirmation from xbox.com, or if the agent reports that xbox.com does not provide a top-selling list and states that limitation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Record the game's publisher</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
State who the identified game was published by. Partial credit if the developer is provided instead of the publisher or if there is ambiguity that is acknowledged.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Record the game's release date</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the game's release date. Partial credit if only the month/year are provided or if multiple regional dates are noted with a reasonable choice or explanation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Identify the CEO or head of the publisher/gaming studio and their birth date</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find the CEO or head of the identified publisher (the gaming studio) and provide their date of birth. Partial credit if a senior leader (e.g., president, studio head) is given when a CEO is not applicable, or if the role is correctly identified but the birth date is uncertain and the limitation is stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Compute elapsed years between the leader’s birth date and the game’s release date</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Calculate how many years elapsed (i.e., the leader’s age at release). Full credit for a correct calculation considering exact dates; partial credit for an approximate calculation or clearly stated assumptions when exact dates are unavailable.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Attempt to access xbox.com top-selling context/listing</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to navigate to xbox.com (Microsoft/Xbox store pages) and locate a context that lists or labels games as “Top-selling” (or equivalent, e.g., “Top selling games”). Full credit if the agent makes a reasonable attempt and clearly reports a blocker (CAPTCHA, login wall, region lock, site error, dynamic content preventing verification). Partial credit if the attempt is unclear or uses only non-xbox.com sources without first attempting xbox.com.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify a top-selling Xbox game according to xbox.com (or clearly stated fallback)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Name a game that xbox.com explicitly labels/lists as “top-selling” in the accessed context. Full credit if the top-selling designation is clearly tied to xbox.com. If xbox.com access/verification is blocked, full credit if the agent clearly states the limitation and uses a reasonable alternative signal (e.g., cached page, reputable third-party capture, or Microsoft/Xbox official channels) while explicitly labeling it as not directly verified from xbox.com. Partial credit if a game is from xbox.com but the top-selling context is not established.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Extract publisher and release date from xbox.com (or clearly stated availability limits)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
For the selected game, report the publisher and release date as shown on xbox.com. Full credit if both are provided with clear linkage to xbox.com. If one/both fields are not shown, are inconsistent across locales, or are inaccessible due to blockers, full credit if the agent explicitly states what was missing/unavailable on xbox.com and (optionally) provides the missing info from an alternative reputable source clearly labeled as non-xbox.com. Partial credit if only one of the two fields is provided without explaining why the other is missing, or if sourcing is unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Identify the CEO/head of the game's studio and their birth date/year (with attribution)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify the relevant gaming studio for the chosen game and name the CEO or studio head (or closest reasonable equivalent if there is no single clear leader), plus their birth date/year. Full credit if the choice of leader is justified when ambiguous (e.g., co-heads, division president vs. studio head) and the birth information is attributed to a reputable source. Partial credit if the leader is plausible but birth info is missing, or if birth year is given without credible attribution. Full credit if the agent explains that no verifiable birth info is publicly available after reasonable effort and proceeds with year-only or an alternative clearly labeled approach.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Compute elapsed years between studio head birth and game release date</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Correctly compute elapsed years between the studio head’s birth date/year and the game’s release date. Full credit if the computation is consistent with the level of date precision available (e.g., uses exact date-boundary logic when full dates are known; uses year-difference with an explicit note about uncertainty when only years are known). Partial credit if the arithmetic is roughly correct but ignores date-boundary logic despite having full dates, or if uncertainty is not acknowledged when only partial dates are available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Accuracy, attribution, and non-hallucination</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
All reported facts should be internally consistent and supported by the stated sources (xbox.com where available; otherwise clearly labeled alternates). The agent should not fabricate titles, dates, publishers, or biographical details. Full credit if citations/attribution are clear enough to distinguish xbox.com-derived facts from external facts. Partial credit if attribution is somewhat unclear but facts are likely correct; no credit if key claims are invented or contradict the agent’s described evidence.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_99</div>
|
||
<div class="task-summary">Search for a "applied scientist" position on careers.microsoft.com in redmond, WA and for the first result, extract what the team or group name the job posting is for, and then search externally for what that group does and who it is led by.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Search careers.microsoft.com for Applied Scientist in Redmond, WA</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to careers.microsoft.com and perform a search for the job title 'Applied Scientist' with the location set to Redmond, WA. Partial credit if the correct site is accessed but location or title filter is incomplete or incorrect.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first search result</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the search results, clearly identify which job posting appears first. Partial credit if multiple results are listed but the first one is not singled out.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Extract the team or group name from the first posting</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Open the first job posting and extract the explicit team or group name associated with the Applied Scientist role. Partial credit if a related name is provided but not exactly as stated in the posting.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Research what the identified group does</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Conduct an external search (beyond careers.microsoft.com) to determine the function, objectives, or main activities of the extracted team or group. Partial credit if general team purpose is given but key activities are missing.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Research who leads the identified group</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Conduct an external search to find the name of the leader (e.g., manager, director, head) of the identified team or group. Partial credit if a related leadership role is found but the exact leader’s name is not.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access careers.microsoft.com and search for 'Applied Scientist' in Redmond, WA</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to careers.microsoft.com and perform a job search for the role 'Applied Scientist' with the location set to Redmond, WA. Partial credit may be awarded for accessing the site or initiating a search without correctly applying the location filter. Full credit should also be awarded if the site is inaccessible or has no such search functionality and the agent clearly states this.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify and open the first search result</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the returned search results, correctly identify the first (top-most) result and access/open its job posting details. Partial credit may be given for listing results but not confirming/opening the first one. Full credit should be awarded if there are no results and the agent explicitly indicates that.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Extract the team or group name from the job posting</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the job posting of the first result, find and report the team or group name the position is for. Partial credit may be awarded for extracting related organizational information (e.g., division/organization) if a specific team/group name is not present. Full credit should be awarded if the posting does not provide a team/group name and the agent clearly indicates this.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Search external sources to determine what the group does</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use external sources (outside careers.microsoft.com) to find and summarize the group's function, mission, or scope of work. Partial credit may be given for citing sources but providing limited or incomplete information. Full credit should be awarded if credible sources are checked and the information is unavailable, provided the agent clearly states that.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Identify who leads the group</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find the leader of the group (name and role/title) from external sources. Partial credit may be awarded for identifying either the leader's name or role, or for identifying leadership at a higher organizational level if the specific group's leader is not publicly available. Full credit should be awarded if no reliable information exists and the agent clearly indicates that.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access careers.microsoft.com and attempt to search for 'applied scientist' roles in Redmond, WA</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use careers.microsoft.com to search for the keyword/title 'applied scientist' and constrain location to Redmond, WA (or the closest available location filter if Redmond is not offered). Full credit if the agent makes a reasonable attempt but is blocked by uncontrollable factors (CAPTCHA/login wall/outage) and clearly reports the blocker and what was attempted. Partial credit if the agent searches without a location constraint or uses an incorrect primary keyword/title when the site is accessible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Open/identify the first job result shown by the careers.microsoft.com search</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the search results page produced by the attempted query, select/open the first job result shown and clearly identify it as the first listing at the time of search (e.g., by position in list). Full credit if no results appear for the exact constraints and the agent clearly reports that and then proceeds with the closest alternative that preserves primary intent (e.g., Applied Scientist in Greater Seattle/WA/nearby, or removing radius constraint), while stating the deviation. Partial credit if the agent opens a non-first result despite first being available and no justification is given.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Extract the team/group name from the first job posting</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Accurately extract and report the team or group name as stated in the first job posting. Full credit if the team/group name is explicitly present and is quoted or clearly attributed to the posting. Full credit (uncontrollable) if the posting does not specify a team/group name (after checking typical sections like header/summary/org/Responsibilities/Qualifications) and the agent clearly states that limitation and, if present, reports the closest higher-level org named in the posting (e.g., division). Partial credit if the agent provides only an inferred/guessed team name when the posting provides clearer org/team wording.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Externally research what the identified group does</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use at least one external (non-careers.microsoft.com) source to research what the identified group/team does and provide a concise description consistent with the source(s). Full credit if reputable sources are used (e.g., Microsoft official pages/blogs, reputable news, conference talks, LinkedIn org pages). Full credit (uncontrollable) if the group is not publicly described, sources are inaccessible (paywall/blocked), or only the parent org is findable; in that case, the agent should clearly report the limitation and summarize the closest verifiable parent-org function without inventing details. Partial credit if the description is overly generic or weakly sourced while better public info is readily available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Externally identify who the group is led by</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify the group's leader (e.g., Head/Director/GM/VP) via external research and report the name with evidence/attribution (e.g., Microsoft leadership page, press release, LinkedIn profile showing the role). Full credit if a specific leader for the identified group is found and attributed. Partial credit if only leadership at a broader parent-org level is identified and the agent clearly states the scope mismatch. Full credit (uncontrollable) if no verifiable leader information is publicly available for the group (or sources are inaccessible) and the agent clearly reports that after reasonable search, without guessing.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_51</div>
|
||
<div class="task-summary">at the denver museum of nature and science, find the next show held at the Infinity Theater, and find out who the producer is, and furthermore the names of up to three other films/movies they produced.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the next Infinity Theater show</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine which show will be presented next at the Infinity Theater in the Denver Museum of Nature and Science. Partial credit if the agent finds a list of upcoming shows but not the very next one; full credit only if the correct next show is named.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find the producer of the identified show</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and state the name of the producer responsible for the next Infinity Theater show. Partial credit if a list of multiple people involved is given but the producer’s name is not clearly identified; full credit if the producer is correctly named.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">List up to three other films/movies by the producer</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the titles of up to three additional films or movies produced by the same producer. Partial credit for fewer than three accurate titles; full credit for up to three correct titles.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the next show at the Infinity Theater (Denver Museum of Nature & Science)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and clearly state the title of the next scheduled show at the Infinity Theater at the Denver Museum of Nature & Science. Partial credit may be awarded if multiple upcoming shows are listed without clearly identifying the next one, or if the schedule cannot be determined but the agent explains the limitation and provides the closest available information.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find the producer of the identified show</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine and report who the producer is for the identified show. Partial credit may be awarded if the agent provides a plausible producer (e.g., a production company or named producer) when multiple producers exist, or acknowledges when producer information cannot be confirmed from authoritative sources.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">List up to three other films/movies produced by the same producer</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the names of up to three other films or movies produced by the same producer identified for the show. Partial credit may be awarded for fewer than three valid titles or for explaining if such information cannot be reliably confirmed.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Infinity Theater show schedule/listings (Denver Museum of Nature & Science)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to access the Denver Museum of Nature & Science Infinity Theater schedule/listings (via the museum site or clearly identified official DMNS channels). Full credit if the agent reaches the schedule/listing OR clearly reports an uncontrollable blocker (site down, CAPTCHA, geo-blocking, broken page) and describes what was attempted. Partial credit if the agent uses an unrelated/unauthoritative source without attempting DMNS/official listings first.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the next show at the Infinity Theater (per available schedule ordering)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine the next upcoming Infinity Theater show as presented by the accessible schedule/listings, and report the title plus the next listed date/time (or the earliest showtime shown). Full credit if the agent correctly identifies the next upcoming show with its corresponding next showtime/date when available. Also full credit if the schedule is ambiguous (e.g., multiple formats/filters, multiple films with the same earliest showtime, or only recurring daily times without a clear 'next') and the agent explains the ambiguity and selects a defensible 'next' based on the earliest time/date shown. Partial credit if the title is provided but the 'next' ordering is not established when it could have been, or if showtime/date is omitted despite being clearly shown.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the producer of the identified next show</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and report the producer (person or production company, as credited) for the identified next Infinity Theater show, citing/grounding it in an authoritative source (DMNS listing or official film credits page). Full credit if the producer credit is correctly extracted, OR if producer credit is not available on accessible authoritative sources / sources are blocked and the agent clearly reports where they looked and that the producer could not be confirmed. Partial credit if a producer is given without clear linkage to the specific show or if the agent likely confuses producer with director/narrator when clearer credits were available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">List up to three other films/movies produced by that producer</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide 1–3 other film/movie titles that the identified producer has produced, grounded in reliable filmography/credits sources. Full credit for 1–3 correctly attributed titles, OR full credit if no additional producer credits can be found due to unavailable/blocked sources or the producer appears to have no other producership credits and the agent reports this after reasonable search. Partial credit if titles are not clearly verified as producer credits (e.g., other roles only) when better verification was feasible.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_50</div>
|
||
<div class="task-summary">List all the members of the bands Nsync and BackStreet Boys. Find the net worth of the one with the longest last name.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">List all members of NSYNC</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the complete roster of NSYNC members. Partial credit may be given if only some members are listed; full credit requires all official members.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">List all members of Backstreet Boys</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the complete roster of Backstreet Boys members. Partial credit may be given if only some members are listed; full credit requires all official members.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the member with the longest last name</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compare the last names of all listed members and clearly identify which member has the longest last name. Partial credit if the comparison is shown but the correct member is not named.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Provide net worth of the identified member</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Research and state the net worth of the member with the longest last name. Partial credit if an approximate range is given; full credit for a sourced figure.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">List all members of NSYNC</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a complete and correct list of all members of the band NSYNC. Partial credit may be awarded if most members are listed but one or more are missing or incorrect.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">List all members of Backstreet Boys</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a complete and correct list of all members of the band Backstreet Boys. Partial credit may be awarded if most members are listed but one or more are missing or incorrect.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the member with the longest last name</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the combined lists, correctly determine which single member has the longest last name. Partial credit may be awarded if there is a reasonable attempt (e.g., correctly handling ties or minor miscounts).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Find the net worth of the identified member</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a net worth figure for the correctly identified member with the longest last name. Partial credit may be awarded if a net worth is given but the member was misidentified or if the figure lacks precision.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">List all members of NSYNC</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a complete list of all official members of the band NSYNC. Full credit if all members are listed (Joey Fatone, Justin Timberlake, JC Chasez, Chris Kirkpatrick, Lance Bass). Partial credit if some members are listed but at least one is missing or if a non-member is incorrectly included. No credit if the band’s members are largely incorrect or the wrong group is listed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">List all members of Backstreet Boys</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a complete list of all official members of the band Backstreet Boys. Full credit if all members are listed (AJ McLean, Howie Dorough, Nick Carter, Kevin Richardson, Brian Littrell). Partial credit if some members are listed but at least one is missing or if a non-member is incorrectly included. No credit if the band’s members are largely incorrect or the wrong group is listed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the person with the longest last name among the combined member lists</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine which individual (from both bands’ member lists) has the longest last name (by number of letters). Full credit if the correct person is identified and the comparison set is clearly the members of both bands. Partial credit if a plausible candidate is chosen but the method is unclear, ties are mishandled, or the comparison appears incomplete. No credit if the identified person is not in either band or is clearly not the longest last name given the provided names.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Find and report the net worth of the member with the longest last name</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a net worth estimate for the identified member with the longest last name. Because net worth is externally dependent and varies by source/date, full credit if the agent (a) reports a reasonable net worth figure or a small range for the correct person and (b) indicates the estimate’s source and/or that figures differ across sources (or that the value is approximate/as of a given year). Also award full credit if the agent clearly explains it cannot reliably verify a net worth figure due to unavailable/inaccessible sources but provides the best available estimate or states that no reliable figure could be found. Partial credit if a net worth figure is provided but the person is wrong, or if the figure is ambiguous (e.g., missing currency/context) while still clearly intended as net worth. No credit if no net worth is provided and no clear attempt/limitation is communicated, or if the value is clearly unrelated (e.g., salary, revenue, or another person’s net worth).
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_40</div>
|
||
<div class="task-summary">Search for women's clothes on sale at zara, take the first result that is marked down, find out what materials it is composed of, and then tell me at what temperature the primary material ignites.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Zara's website and locate women's clothes on sale</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the Zara website and successfully find the section or filter for women's clothes on sale.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first marked-down result</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Select or note the first item in the sale results that is specifically marked down.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Determine the item's material composition and primary material</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
List all materials the selected item is composed of and correctly identify the primary material based on proportion.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Find the ignition temperature of the primary material</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Research and report the temperature at which the primary material ignites.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate women's clothes on sale at Zara</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and access Zara's women's sale items (e.g., via Zara website or search results that show women's items on sale). Partial credit if Zara is accessed but the sale section or women's sale items are not clearly reached.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Select the first marked-down women's item</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify and choose the first result that is explicitly marked down from the women's sale items. Partial credit if an on-sale women's item is selected but not the first, or if the selection is not clearly marked down. Full credit if the agent notes that no marked-down items are available and states this clearly.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Retrieve the item's materials composition</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and report the materials the selected item is composed of as listed on the product page (ideally including percentages if shown). Partial credit for incomplete or partially accurate composition; full credit if the agent notes that composition is unavailable on the page.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Provide the ignition temperature of the primary material</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine the primary material (e.g., the highest-percentage component) from the composition and state its ignition (autoignition) temperature with units. Partial credit for providing a plausible temperature range or closely related thermal property with explanation, or for explaining unavailability of ignition data.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Zara and reach a women's sale/markdown listing</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to navigate Zara (zara.com or a regional Zara site) to a women's sale/discounted/marked-down product listing. Full credit if the agent reaches the relevant listing OR if Zara is inaccessible (CAPTCHA, region block, site down, requires app/login) and the agent clearly reports the blocker and what was attempted. Partial credit if the agent browses Zara women’s items but cannot establish any sale/markdown context and does not clearly explain why.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Select the first visible item that is explicitly marked down</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the women's sale/markdown results that are actually visible to the agent, select the first item showing an explicit reduction (e.g., reduced price, struck-through original price, discount label). Full credit if the agent either (a) selects the first visible marked-down item, or (b) explains why the “first” ordering cannot be reliably determined (dynamic sorting/infinite scroll/personalization) and selects the earliest marked-down item they can verify. Partial credit if a marked-down item is chosen but the agent provides insufficient evidence that it was first/earliest among visible markdowns when that ordering is clearly viewable.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find and report the item's material composition from Zara</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Open the selected product’s details and extract the material composition as listed by Zara, including percentages when available. Full credit if the composition is provided with fiber names and percentages OR if Zara does not display composition (hidden behind unavailable accordion, blocked scripts, geo/app gating) and the agent clearly reports the limitation and where they looked. Partial credit if fiber types are provided but percentages are omitted despite being clearly available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Determine the primary material and provide its ignition temperature</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify the primary material as the highest-percentage fiber from the reported composition (or, if multiple components are separately listed and no single overall percentage is determinable, choose a defensible primary component and explain). Provide the ignition temperature for that material with units and attribution to a credible reference; a reasonable range is acceptable if sources vary or if the reference reports a range. Full credit if the primary material identification is consistent with the composition and the ignition temperature is plausibly sourced/attributed; if ignition temperature cannot be determined (e.g., composition unknown due to Zara gating), full credit for clearly stating that dependency and not fabricating a value. Partial credit if the primary material is correct but the ignition temperature lacks units and/or lacks any attribution.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_79</div>
|
||
<div class="task-summary">on amazon, find the #3 best selling pantry staple item, and then on AllRecipes, find a recipe which contains that item as an ingredient. Output the full ingredients list along with the recipe name.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the #3 best-selling pantry staple item on Amazon</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate Amazon's pantry staples best-selling list and correctly determine which product is ranked third. Partial credit can be given for selecting a pantry staple but with an incorrect rank or for identifying the correct item but mislabeling its rank.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find a recipe on AllRecipes containing that item</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
On AllRecipes, search for and select a recipe that explicitly includes the identified pantry staple item as an ingredient. Partial credit for choosing a recipe that uses a closely related ingredient or for failing to verify that the identified item appears in the ingredient list.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Output the recipe name and full ingredients list</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the exact recipe title and reproduce the complete list of ingredients from the selected AllRecipes recipe. Partial credit if either the recipe name or some ingredients are missing or incorrect.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the #3 best-selling Pantry Staples item on Amazon</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and name the item that is ranked #3 in Amazon's Best Sellers for the Pantry Staples category. Full credit requires clearly identifying the #3 item from Amazon within the Pantry Staples category. Partial credit may be awarded if a top seller is identified but not the #3 item, or if the category is correct but the rank/item is ambiguous.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find a relevant AllRecipes recipe that includes that item</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate a recipe on AllRecipes that lists the identified pantry staple item as one of its ingredients. Full credit requires that the recipe is from AllRecipes and explicitly includes the identified item (or an unambiguous equivalent) in its ingredients. Partial credit may be awarded if the recipe is on AllRecipes but the ingredient match is unclear or slightly off.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Output the recipe name and the full ingredients list</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the recipe name and the complete ingredients list as presented on AllRecipes. Full credit requires both the recipe name and the entire ingredients list (not a subset). Partial credit may be awarded if only the name is given or if the ingredients list appears incomplete.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Amazon Best Sellers for the relevant pantry staples section</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to navigate Amazon’s Best Sellers page(s) for a pantry staples/grocery/pantry category and locate the visible ranking list. Full credit if the agent makes a reasonable attempt but is blocked (CAPTCHA/login/region restriction), the page is unavailable, or rankings cannot be viewed, and it clearly reports what was attempted and the blocker. Partial credit if the agent uses an unrelated Amazon page or provides no evidence of attempting to view a Best Sellers ranking.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the #3 best selling pantry staple item on Amazon</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine and report the product shown as rank #3 on Amazon Best Sellers within the chosen pantry staples/grocery pantry category at the time of access, with enough detail to uniquely identify it (e.g., full product name/brand/size). Full credit if #3 is clearly identified (or if Amazon rankings are inaccessible and this is already documented under the access criterion, with no further penalty here). Partial credit if a plausible best-seller is provided but rank #3 is not verified, the category is unclear, or the product details are insufficient to uniquely identify the item. If rankings appear inconsistent due to region/personalization/ties/rapid changes, full credit if the agent states this uncertainty and reports what was observed (including timestamp/context) and still provides the best-supported #3 item.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Access AllRecipes and search for a recipe containing the identified ingredient</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use AllRecipes to find a recipe whose ingredient list includes the identified Amazon item’s underlying ingredient (recognizing that recipes typically list generic ingredients rather than brand/SKU). Full credit if the agent attempts AllRecipes but is blocked, the site is down, or ingredient lists cannot be accessed, and it clearly reports the blocker and attempts. Partial credit if the agent does not use AllRecipes and does not report an access issue.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Find an AllRecipes recipe that contains the identified item as an ingredient</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Select an AllRecipes recipe where the ingredient list explicitly includes the identified Amazon item or an unmistakable equivalent ingredient name (e.g., Amazon product is 'canned chickpeas' and recipe lists 'garbanzo beans/chickpeas'). Full credit if the ingredient match is explicit on the AllRecipes page, or if no such AllRecipes recipe can be found after reasonable search attempts and the agent clearly reports that outcome (optionally providing the closest match on AllRecipes). Partial credit if the recipe is not from AllRecipes when AllRecipes is accessible, or if the ingredient match is ambiguous/unsupported when clearer matches are available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Report the recipe name</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the exact title/name of the selected AllRecipes recipe (or clearly state that no qualifying recipe could be accessed/found on AllRecipes). Partial credit if the name is incomplete but still uniquely identifies the recipe.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Output the full ingredients list from the AllRecipes recipe</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the complete ingredients list as shown on the AllRecipes recipe, including quantities and descriptors/qualifiers. Full credit if all ingredients are included; partial credit for minor omissions while most of the list is correct. If the AllRecipes page/ingredient list cannot be accessed due to external blockers already documented, full credit if the agent clearly reports that it could not retrieve the ingredients list (and does not fabricate).
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_120</div>
|
||
<div class="task-summary">Please help me find the first news article published on universityofcalifornia.edu websites, then tell me two other articles published by the same author.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate the first news article on the universityofcalifornia.edu website</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the universityofcalifornia.edu news section or archive, sort articles by date, and identify the earliest published article. Partial credit if the article is found but not confirmed as the earliest.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the author of the first news article</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Accurately extract the author's name for the identified first news article. Partial credit for partially correctly identifying the author (e.g., surname only).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">List two other articles authored by the same individual</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and list two additional news articles on universityofcalifornia.edu written by the same author. Partial credit for finding one correct article or listing two but with ambiguous authorship.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Constrain search to universityofcalifornia.edu websites</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Ensure the search and selection for the first news article is limited to content hosted on universityofcalifornia.edu (including its subdomains). Partial credit may be awarded if the agent attempts to use the specified domain but is not fully consistent.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find the first news article published on universityofcalifornia.edu</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify the earliest (i.e., first) news article published on the specified domain. Full credit for clearly identifying the article and supporting why it is the first (e.g., via publication date or archive ordering). Partial credit if the agent provides a plausible earliest article from available archives or notes limitations preventing definitive identification.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the author of the first article</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Extract and state the author's name from the identified first news article, as needed to satisfy the next step. Partial credit if the author is not listed and the agent explicitly notes the absence or uses the byline appropriately.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Provide two other articles published by the same author</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
List two distinct other articles authored by the same person identified from the first article. Full credit requires two correct, clearly identified articles (titles; links optional) that are different from the first article. Partial credit for only one correct article or if attribution is unclear but plausibly the same author.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access universityofcalifornia.edu and locate a news archive or searchable news listing</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to universityofcalifornia.edu and attempt to access a news section/landing page and an archive, listing, or search experience that surfaces news articles. Full credit if the agent clearly attempts access but is blocked (e.g., CAPTCHA, paywall/login, site down) or if the archive/listing function is inaccessible, and the agent explicitly reports the blocker. Partial credit if the agent uses an unclear/incorrect section of the domain (not news) but demonstrates reasonable effort to find a news listing.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first (earliest chronologically published) news article on universityofcalifornia.edu, or the best-supported earliest article available</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and report the earliest (first chronologically published) news article available on universityofcalifornia.edu, providing at least title and publication date (URL optional). Full credit if the agent correctly identifies the earliest article and provides identifying details, OR if the agent explains why definitive verification is not possible due to site limitations (e.g., no oldest-sort, incomplete archive, inconsistent dates) and instead provides the best-supported earliest article they can find along with the method/evidence used (e.g., oldest reachable page, earliest search result with date). Partial credit if an early article is provided but the effort to determine/justify it as earliest (or best-supported earliest) is weak or unclear. No credit if the item is not on universityofcalifornia.edu or is not a news article.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify two other articles published by the same author (or best available author-matched alternatives under site constraints)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the author of the first identified article, find two other articles by that same author, preferably on universityofcalifornia.edu, and provide at least their titles (dates/URLs optional). Full credit if both additional articles are clearly attributed to the same author, OR if author discovery is impeded by external constraints (missing/variable bylines, absent author page, site search limitations) and the agent documents reasonable attempts (e.g., searching the domain for the author name, using an author tag page if present) and reports the best available author-matched results or clearly states that fewer than two could be verified. Partial credit if only one additional verified article is found or if one of the two has unclear attribution despite reasonable effort. No credit if the additional articles are not by the same author or are off-domain without a clearly stated, justified blocker.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_67</div>
|
||
<div class="task-summary">find the next upcoming exhibit at the George H.W. Bush library and tell me what dates it will be available. Tell me whether any total solar eclipse will occur at all within that time frame.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the next upcoming exhibit</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and correctly identify the exhibit at the George H.W. Bush Library that has the nearest start date after the current date, using an authoritative source such as the library's official website.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Provide exhibit availability dates</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
List the exact start and end dates for the identified upcoming exhibit, ensuring accuracy and clarity.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Determine if a total solar eclipse occurs during that timeframe</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Consult a reliable astronomical source (e.g., NASA eclipse data) to check whether any total solar eclipse falls within the exhibit’s availability dates, and state the finding clearly.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the next upcoming exhibit at the George H.W. Bush Presidential Library and Museum</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and correctly name the exhibit that is scheduled to open next at the specified library. Full credit if it is clearly the next upcoming exhibit relative to the current date, or if none are upcoming and the agent explicitly states that. Partial credit if an exhibit is identified at the correct institution but it is not the next upcoming or the status is unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Provide the exhibit's availability dates</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
State the exhibit's start and end dates during which it will be available. Full credit for both precise start and end dates; partial credit for providing only one of the dates or an approximate timeframe. If dates are not available, full credit for clearly stating that and explaining the limitation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Assess whether any total solar eclipse occurs within the exhibit's timeframe</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine if a total solar eclipse takes place at any time between the exhibit's start and end dates, and clearly state yes or no (with a brief explanation if applicable). Partial credit for addressing eclipses but with incomplete timeframe consideration or uncertainty noted.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the next upcoming exhibit at the George H.W. Bush Library</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine the next upcoming (soonest not-yet-started) exhibit at the George H.W. Bush Presidential Library & Museum using authoritative sources (official library website pages, official announcements, or equivalent). Full credit if the agent correctly identifies the exhibit title and clearly ties it to the Bush Library, or if official information is unavailable/unclear (e.g., site down, CAPTCHA, conflicting listings, no “upcoming” exhibits posted) and the agent clearly reports that limitation and what it checked. Partial credit if an exhibit is identified but “next/upcoming” status is not well-justified when multiple future exhibits are listed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Report the exhibit availability dates</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the exhibit’s availability date range (opening/start date and closing/end date) as shown by an authoritative source. Full credit for clearly stating both dates; OR, if the official source does not list an end date (or lists it as TBD/ongoing), full credit for reporting the known start date and explicitly noting that the end date is not announced/unknown. Partial credit if only one date is provided without clarifying whether the other is unavailable vs. omitted, or if dates are ambiguous but reasonably inferable.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Determine whether any total solar eclipse occurs within the exhibit time frame</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the exhibit availability window (inclusive) and a reliable eclipse source (e.g., NASA or equivalent), determine whether any total solar eclipse occurs at any time within that interval. Full credit for a correct yes/no conclusion clearly tied to the date window; OR, if authoritative eclipse data cannot be accessed due to external issues (blocked sources, downtime), full credit for clearly reporting the access limitation and the best-effort reasoning/attempt. Partial credit if eclipse information is mentioned but overlap with the exhibit dates is not clearly evaluated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">If a total solar eclipse occurs within the time frame, provide the eclipse date(s)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
List the date(s) of any total solar eclipse(s) that fall within the exhibit date range. Full credit for correct eclipse date(s). Partial credit if an eclipse date is provided but the eclipse type is wrong (not total) or the date is slightly mis-scoped while still attempting to match the exhibit interval. If eclipse-date sources are inaccessible, the agent should not be penalized provided it clearly reports the limitation after a reasonable attempt.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_38</div>
|
||
<div class="task-summary">Find a vegetarian restaurant in San Francisco with a rating ≥4.5 and ≥100 reviews; use its address to book a compact car nearest to that location on Rentalcars.com from December 15 to December 18, 2025.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Yelp.com</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or access the Yelp.com website.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify a qualifying restaurant</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find a vegetarian restaurant in San Francisco on Yelp with a rating of at least 4.5 and at least 100 reviews. Partial credit for meeting some but not all criteria (e.g., correct location and type but lower rating or fewer reviews).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Extract restaurant address</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Accurately obtain the full street address of the selected restaurant as listed on Yelp.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Access Rentalcars.com</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or access the Rentalcars.com website.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Enter pickup location</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the restaurant's address as the pickup location on Rentalcars.com.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Enter rental dates</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Select November 15, 2025 as the pick-up date and November 18, 2025 as the drop-off date on Rentalcars.com.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 7:</span>
|
||
<span class="criterion-text">Search for compact car nearest location</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search for and identify a compact car option available for the specified dates, prioritizing proximity to the entered pickup address.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find qualifying vegetarian restaurant on Yelp.com in San Francisco and capture its address</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Yelp.com to identify a vegetarian restaurant located in San Francisco that meets BOTH thresholds: rating ≥ 4.5 and ≥ 100 reviews. Extract the restaurant’s street address to use for the car rental step. Partial credit may be awarded if the agent finds a vegetarian restaurant but misses one of the thresholds, finds one in SF but does not provide the address, or attempts on Yelp but cannot confirm the criteria. Full credit should be awarded if no such restaurant exists and the agent clearly indicates that unavailability.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Initiate rental car search on Rentalcars.com using the restaurant’s address and correct dates</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
On Rentalcars.com, set up a search using the restaurant’s address as the pickup location (or the nearest available pickup point determined from that address) for the dates November 15 to November 18, 2025. Partial credit may be awarded for using a nearby location (e.g., city center) instead of the precise address, or for setting up the search but with incorrect dates. Full credit should be awarded if results cannot be shown for that address and the agent clearly indicates the limitation while attempting a reasonable nearest location.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Select a compact car at the nearest available pickup location and prepare for booking without completing checkout</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the search results, choose a compact car option at the nearest available pickup location to the restaurant’s address and proceed up to the booking/reservation stage (e.g., viewing deal or proceeding toward customer details) without entering personal information or completing the booking. Partial credit may be awarded for selecting a non-compact car, choosing a not-closest location, or failing to proceed to the booking stage. Full credit should be awarded if no compact cars are available at the nearest location and the agent clearly indicates that unavailability.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify a qualifying vegetarian restaurant in San Francisco</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find a vegetarian (or clearly vegetarian/vegan-focused) restaurant located in San Francisco. Full credit if the agent identifies a specific restaurant and, from a reasonable source, verifies BOTH: rating ≥4.5 and review count ≥100. Also award full credit if, after reasonable search/verification attempts, the agent clearly reports that it cannot confirm both thresholds from available sources or that no visible results meet both constraints, and then selects the best available highly rated/popular vegetarian alternative consistent with the task’s primary intent. Partial credit if the restaurant is vegetarian and in San Francisco but only one threshold is verified or the verification is unclear. No credit if the restaurant is not vegetarian/veg-focused, not in San Francisco, or clearly fails thresholds when qualifying options are readily available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Provide and use the restaurant's address as the reference location</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Obtain the restaurant’s full street address (or the most precise address available from sources). Full credit if the address is clearly captured and then used to anchor the rental search, either by entering the address directly on Rentalcars.com OR by selecting the nearest unambiguous pickup area/location derived from that address (e.g., closest downtown/rail/hotel/landmark option shown by the site) when exact address entry is not supported. Partial credit if only a partial address/neighborhood is used but the linkage to the restaurant location is clear. No credit if the address is missing or the rental search is anchored to an unrelated/incorrect location without justification.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Access Rentalcars.com and search for pickup locations near the restaurant</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt the workflow on Rentalcars.com using the restaurant address (or nearest derived pickup location) as the pickup anchor. Full credit if the agent reaches search results OR if Rentalcars.com is inaccessible (CAPTCHA, outage, blocking, geo restrictions) and the agent clearly reports the blocker after attempting. Partial credit if the attempt is made but the pickup location used is broadly in San Francisco without being clearly tied to the restaurant area. No credit if Rentalcars.com is not attempted when accessible or the search is for an unrelated city.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Use Rentalcars.com to filter/select a compact car category near the restaurant</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From Rentalcars.com results, filter for or select the compact car category (or the closest equivalent category label available on the site, if “compact” is not offered). Full credit if compact (or nearest equivalent) is clearly selected/shown in results, OR if results do not offer a compact category and the agent clearly reports this and selects the closest available small-car alternative consistent with intent. Partial credit if nearby rental options are found but car category is not clearly compact/equivalent. No credit if a clearly non-compact category is chosen without explanation when compact/equivalent is available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Set correct rental dates: December 15 to December 18, 2025</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Enter/select pickup date Dec 15, 2025 and drop-off date Dec 18, 2025 on Rentalcars.com. Full credit if both dates are correctly applied in the search/results/booking flow, OR if the platform prevents selecting those dates (e.g., limited inventory horizon, site error) and the agent clearly reports the limitation encountered. Partial credit if only one date is correct or if the year is ambiguous but intent is clear and the agent attempts correction. No credit if dates are clearly wrong when correct entry was possible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Advance booking flow to a pre-checkout stage without crossing Critical Points</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Select a specific compact (or compact-equivalent) offer at the nearest feasible pickup location and proceed to a details/pre-checkout page. Stop before entering personal details, creating an account, logging in, or providing payment information, and do not finalize a reservation. Full credit if the agent reaches a pre-checkout/details stage and stops appropriately, OR if the site requires login/personal details/payment earlier than expected and the agent stops and reports that constraint. Partial credit if an offer is selected but the agent does not meaningfully progress toward the booking details page. No credit if the agent completes a binding booking/checkout or enters fabricated personal/payment details.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_100</div>
|
||
<div class="task-summary">find a reddit post in r/golf talking about how golf courses take up "3000 sq miles" of land in the USA. Summarize the top upvoted comment for that post, and then find another website that substantiates any major claim that comment makes.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate the specific Reddit post in r/golf referencing '3000 sq miles' of land occupied by golf courses in the USA</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Agent should identify and provide the link or clear reference to the exact Reddit thread in r/golf discussing that golf courses take up 3000 square miles in the USA. Partial credit if a closely related post is found but details (subreddit or figures) are incorrect.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Summarize the top upvoted comment</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Agent should extract the highest-voted comment from that post and produce an accurate, concise summary of its key points. Partial credit for capturing some but not all major points or misrepresenting the comment.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify an external source substantiating a major claim from the comment</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Agent should find another reputable website or source that supports at least one central claim made in the top comment (e.g., land area statistics, environmental impact). Partial credit if source is marginally relevant or less authoritative.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find the specified r/golf Reddit post</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully locate and identify a post in r/golf that discusses golf courses taking up "3000 sq miles" of land in the USA. Partial credit may be awarded if a closely related r/golf post is found that addresses similar land usage but does not explicitly mention the "3000 sq miles" figure, or if the subreddit or topic is slightly off yet relevant.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Summarize the top upvoted comment</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Correctly identify the top upvoted comment on the located post and provide a concise, accurate summary of its key points. Partial credit may be given if a comment is summarized but it is not the top upvoted one, or if the summary captures some but not all major points of the comment.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Provide an external website that substantiates a major claim from the top comment</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find another website that credibly and directly supports at least one major claim made in the top comment, and make clear how it substantiates the claim. Partial credit may be awarded if the external source is relevant but does not directly corroborate the claim, or if the source's credibility is weak.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Reddit/r/golf and attempt to locate the post mentioning '3000 sq miles'</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use Reddit (and specifically r/golf) search/browsing to locate a post explicitly referencing that US golf courses take up about "3000 sq miles". Full credit if the agent demonstrates a reasonable attempt but is blocked by Reddit (captcha/login/region/app restrictions) or the post appears deleted/unavailable and the agent clearly reports this limitation. Partial credit if the agent searches but in the wrong subreddit or without targeting the key phrase/number.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the specific r/golf post (or clearly report non-findability) for the '3000 sq miles' claim</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Full credit if the agent identifies the correct r/golf post and provides sufficient evidence (title/context/quote showing the "3000 sq miles" mention). Full credit also if, after reasonable searching, the agent clearly reports that the exact post cannot be found (e.g., deleted, not indexed, search limitations) and documents what was tried. Partial credit if a similar post is found but the explicit phrase/number is missing or the post is outside r/golf.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Summarize the top upvoted comment on the identified post</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a summary of the single top upvoted comment for that post. Full credit if the summary captures the major points of that specific top comment. If comments/top sort are inaccessible (deleted thread, locked, blocked by Reddit UI/access limits), full credit if the agent clearly reports the limitation and why the top comment cannot be confirmed. Partial credit if the agent summarizes a non-top comment despite the top comment being visible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Identify a major, checkable claim made in the top comment</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Extract at least one major, checkable claim stated in the top upvoted comment (e.g., a statistic, land-use comparison, water-use claim, number of courses, economic claim). Full credit if the claim is clearly stated and attributable to the top comment. If the top comment is not accessible/confirmable, award full credit for explicitly stating that no attributable claim can be extracted.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Find a non-Reddit source substantiating at least one major claim from the top comment</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide an independent (non-Reddit) website/source that directly substantiates at least one identified major claim from the top comment. Full credit if the source is relevant and supports the claim with data or credible reporting. Full credit also if the agent reports that no substantiating source can be found after reasonable effort and explains why (e.g., claim appears unsourced/unverifiable). Partial credit if the source is only loosely related or does not actually support the specific claim.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Correctness and non-hallucination of cited content</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Ensure the reported post context, the top comment summary, and the external substantiation accurately reflect what is actually visible in the cited sources. Full credit if nothing is fabricated/misattributed. Low/no credit if the agent invents a post/comment, misstates what the external source says, or implies verification that was not possible due to access limits.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_123</div>
|
||
<div class="task-summary">On Eventbrite.com, find an art exhibition happening this month in Portland and extract the exact date and venue; then check Google Flights for the cheapest same-day round-trip tickets from Seattle (SEA) to Portland (PDX), completing the task before purchase.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Eventbrite.com</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or access the Eventbrite.com website as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find an art exhibition in Portland this month</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate an art exhibition listing on Eventbrite for Portland occurring within the current calendar month. Partial credit if exhibition is in Portland but outside this month or if date information is unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Extract exact date and venue</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Extract and report the precise date and the venue name/address of the identified art exhibition.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Access Google Flights</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to or access the Google Flights website as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Search same-day round-trip flights SEA to PDX</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Perform a search for a round-trip flight departing Seattle (SEA) and returning the same day from Portland (PDX).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Identify cheapest same-day round-trip option</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine and report the lowest-priced same-day round-trip flight option, including departure time, return time, and fare. Partial credit if missing one of these elements or if a slightly more expensive flight is chosen.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 7:</span>
|
||
<span class="criterion-text">Stop before purchase</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Ensure the process ends before initiating any booking or entering personal/payment information, in compliance with the instruction to complete the task before purchase.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find an art exhibition on Eventbrite in Portland happening this month</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to Eventbrite.com and locate at least one event that is explicitly an art exhibition in Portland, scheduled for the current calendar month. Partial credit may be awarded if an event is found but it is not clearly an 'art exhibition', not in Portland, or not within this month. Full credit should still be awarded if there are no qualifying events this month and the agent clearly states that after checking Eventbrite.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Extract the exact date and venue from the Eventbrite listing</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the exact event date (day and month) and the venue name for the chosen art exhibition. Partial credit may be awarded if only one of date or venue is provided, or if the date is imprecise (e.g., only a date range without specifying a specific date when applicable).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Check Google Flights for the cheapest same-day round-trip SEA to PDX</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Access Google Flights, set origin to Seattle (SEA) and destination to Portland (PDX), configure a same-day round-trip (depart and return on the same calendar day), and identify the cheapest available option including the price. Partial credit may be awarded if the agent sets the route correctly but does not ensure same-day round-trip, or identifies options without confirming the cheapest. Full credit should be awarded if no same-day round-trip options are available and the agent explicitly states that after checking.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Stop before purchase (no transactional steps)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Complete the research and identification steps without initiating a purchase or entering any personal information. The agent should clearly stop before any checkout or booking confirmation.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Eventbrite and search for Portland art exhibitions happening this month</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Eventbrite.com to perform a search for events in Portland, OR within Portland, Oregon, and apply reasonable query terms/filters to target art exhibitions occurring within the current calendar month. Full credit if the agent attempts Eventbrite but is blocked (CAPTCHA/login hard block), the site is down, or results cannot be loaded, and the agent clearly reports the blocker and inability to verify listings. Partial credit if the search is conducted but the location/month constraint is applied incorrectly or only loosely (e.g., Portland metro without clear Portland, or a wider date range without checking this month).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify at least one eligible Eventbrite listing (or report none found)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From Eventbrite search results, identify at least one event that is explicitly an art exhibition, located in Portland (or clearly described as Portland, OR), and scheduled within the current calendar month. Full credit if an eligible listing is found; OR if none are available that meet all constraints and the agent clearly states that no exact match was found after reasonable checking, optionally providing the closest alternative that preserves the primary intent (art-focused event in Portland this month) while noting which constraint(s) were not met. Partial credit if the selected event is art-related but not clearly an exhibition, or is in the broader area but not clearly Portland when better matches are visible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Extract and report the exact date and venue from the chosen Eventbrite listing (or explain why not possible)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Open the chosen Eventbrite event page and extract (1) the exact event date as stated and (2) the venue/location name. Full credit for both, unambiguous. If the page does not provide a specific single date (e.g., recurring/multi-date series) or the venue is missing/online-only/TBA, full credit if the agent accurately reports what is shown (e.g., date range/recurrence details and the listed location status) and states that an exact single date or venue name is not available on the listing. Partial credit if only one of date/venue is provided when both are clearly shown.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Access Google Flights and set up a same-day round-trip SEA↔PDX search for the event date</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Google Flights to search flights from Seattle (SEA) to Portland (PDX) with round-trip selected and departure/return on the same calendar day as the chosen event date (or, if the event date is not a single specific date, choose a reasonable specific date within the event’s stated schedule and explicitly note the assumption). Full credit if the agent attempts to use Google Flights but is blocked (CAPTCHA/login hard block), the site is down, or results cannot be loaded, and the agent clearly reports the blocker. Partial credit if the correct route is searched but same-day constraint is not applied or date selection is unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Identify the cheapest same-day round-trip itinerary visible (or report unavailability/inability to verify)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the Google Flights results, identify the cheapest itinerary that satisfies same-day round-trip SEA↔PDX. Report the total price and enough itinerary detail to substantiate the choice (e.g., airlines and departure/return times). Full credit if the cheapest visible option is correctly identified; OR if no same-day round-trip options exist for that date and the agent clearly reports this (optionally noting the closest alternatives such as next-day return, different nearby airports, or different date within the event schedule). Full credit if prices cannot be verified due to blocking/unstable results and the agent clearly states this. Partial credit if an option is provided but is not the cheapest when a cheaper same-day option is visibly available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Stop before purchase / avoid critical-point crossing</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
The agent must stop after reporting the Eventbrite event date/venue and the cheapest same-day round-trip flight option, without completing any purchase/booking or submitting personal or payment information on Eventbrite, Google Flights, airline sites, or third-party booking sites. Full credit if it clearly stops before checkout/payment. Partial credit if it navigates near checkout but does not submit information. No credit if it completes a purchase/booking or submits personal/payment credentials.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_5</div>
|
||
<div class="task-summary">From Google Flights, record the least expensive one-way flight from Edinburgh (EDI) to Manchester (MAN) on December 28, 2025, then figure out what aircraft type the flight is on, and how many fewer passengers that aircraft type can carry compared to a 747-8 all-economy configuration.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Set search parameters on Google Flights</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to Google Flights and enter origin as Edinburgh (EDI), destination as Manchester (MAN), and date as December 28, 2025. Partial credit if only some parameters are correct or if a different date within one day is used.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the least expensive one-way flight</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the search results, find and record the single cheapest one-way flight option. Partial credit if a flight is recorded but not the cheapest or if multiple flights are listed without identifying the least expensive.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Determine the aircraft type for the selected flight</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and record the specific aircraft model used for the least expensive flight. Partial credit if an approximate family (e.g., 'A320 series') is given instead of the exact model.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Find seating capacity of the identified aircraft</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Research and report the standard all-economy seating capacity of the flight’s aircraft type. Partial credit if capacity ranges are given without a clear value.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Find seating capacity of a 747-8 all-economy configuration</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Research and report the usual seating capacity for an all-economy Boeing 747-8. Partial credit if a credible range is provided but no single figure.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Calculate the difference in capacity</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compute and state how many fewer passengers the flight’s aircraft can carry compared to the 747-8 all-economy capacity. Partial credit if the calculation is shown but arithmetic error occurs.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Use Google Flights with correct search parameters</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Access Google Flights and perform a search for a one-way flight from Edinburgh (EDI) to Manchester (MAN) on December 28, 2025. Partial credit if Google Flights is used but one parameter (origin, destination, date, or trip type) is incorrect.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify and record the least expensive one-way flight</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the search results, determine the lowest-price one-way option for EDI→MAN on the specified date and record the flight details (price and flight identification such as airline and, if available, flight number/time). Full credit includes noting if no flights are available and stating that explicitly. Partial credit if a flight is recorded but it's not clearly the cheapest or missing key details.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the aircraft type for the identified flight</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the aircraft model/type used for the specific flight identified. Partial credit if only a general aircraft family is given or if type cannot be confirmed but the limitation is clearly stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Calculate passenger capacity difference vs 747-8 all-economy</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compute how many fewer passengers the identified aircraft type can carry compared to a Boeing 747-8 (passenger variant) in an all-economy configuration, and report the numeric difference. Partial credit if capacities are provided without the difference or if there is a small calculation error. Full credit may be awarded if necessary capacity information is unavailable and the limitation is clearly explained.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Attempt Google Flights search for the specified itinerary (EDI→MAN, one-way, Dec 28, 2025)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use Google Flights to search Edinburgh (EDI) → Manchester (MAN), one-way, on December 28, 2025. Full credit if the agent performs the correct search OR clearly reports an uncontrollable blocker (e.g., CAPTCHA, outage, results not loading, pricing unavailable). Partial credit if the agent attempts Google Flights but uses slightly incorrect parameters and corrects/acknowledges the mismatch.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify and record the least expensive one-way flight from viewed results (or report no priced options)</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the results the agent can actually view, identify the least expensive one-way option for EDI→MAN on Dec 28, 2025 and record enough identifiers (at minimum: price with currency, airline/flight number or airline + departure time). Full credit if (a) the agent selects a cheapest option among the visible results, including handling ties (any tied-cheapest is acceptable), OR (b) Google Flights provides no priced options and the agent clearly reports that outcome. Partial credit if a plausible cheap option is provided but the agent does not substantiate that it is cheapest among what was visible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Determine the aircraft type operating the selected cheapest flight (or best available proxy with limitations)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Report the aircraft type for the selected cheapest flight. Full credit if the aircraft type is shown directly in Google Flights for that itinerary/flight. If Google Flights does not show aircraft type or it is unavailable for that date, full credit if the agent clearly states this limitation and uses a reliable alternate source tied to the specific flight number/route/date when possible (or labels it as a typical/expected aircraft for that flight/route if only that is possible). Partial credit if an aircraft type is given without clearly tying it to the specific flight option selected.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Compute passenger-capacity difference vs 747-8 all-economy, stating assumptions</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compute how many fewer passengers the identified aircraft type can carry compared to a 747-8 in an all-economy configuration. Full credit if the agent: (a) states a sourced or explicitly-assumed capacity for the identified aircraft type (noting that capacity varies by configuration), (b) states a sourced or explicitly-assumed 747-8 all-economy capacity (noting that this is not a single universal number), and (c) correctly computes (747-8 capacity − identified aircraft capacity) as 'fewer passengers.' If exact capacities cannot be uniquely determined, full credit for a clearly explained, reasonable assumption with citations and correct arithmetic under those assumptions. Partial credit if arithmetic is correct but one of the capacity assumptions/sources is missing or unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Maintain correct task scope and avoid unsupported claims</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Ensure the final reported flight and analysis are within scope (one-way EDI→MAN on Dec 28, 2025) when such results are available, and do not present unverified details (price/aircraft/capacity) as certain. Full credit if the agent either stays within scope or, if blocked by platform/data unavailability, explicitly labels uncertainty and does not fabricate specifics. Partial credit for minor omissions (e.g., missing currency) that do not materially change the result interpretation.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_68</div>
|
||
<div class="task-summary">Plan an itinerary of getting from central park, manhattan, to miami by taking trains only!</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the starting point</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Specifies the train station nearest to Central Park in Manhattan as the departure location.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the destination</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Specifies the train station in Miami as the final arrival point of the itinerary.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Use trains only</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Ensures that every segment of the journey is by train, with no use of other transportation modes.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Provide a complete train itinerary</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Lists all train legs in sequence, naming departure and arrival stations (and connections) for each segment. Partial credit for incomplete or missing legs.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Start and destination accuracy</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
The itinerary explicitly starts from Central Park, Manhattan and ends in Miami, Florida. Partial credit if the plan starts in New York City generally but does not clearly anchor to Central Park.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Train-only transportation constraint</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
All travel segments use trains only (e.g., subway, commuter rail, intercity rail). No buses, flights, cars, or rideshares are included. Partial credit if incidental walking within stations is mentioned but all travel legs are by train.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Step-by-step itinerary with train lines and transfers</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provides a clear sequence of train segments from Central Park to Miami, including local rail from the starting area to a major intercity station and the onward intercity train(s), with station names and train services identified. Partial credit for a high-level plan that lacks specific lines or stations.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Arrival details in Miami</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Specifies the arrival point in Miami (e.g., the relevant train station/terminal) and makes clear that the endpoint is within Miami. Partial credit if Miami is stated without a specific arrival station.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Start location: Central Park, Manhattan</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Itinerary should clearly begin from Central Park in Manhattan (or a nearby appropriate rail access point such as Penn Station/Grand Central/Harlem–125th) and explain a plausible train-only connection from Central Park to the first intercity departure station (e.g., NYC Subway). Full credit if the start is correct and the rail connection is plausible. Partial credit if it starts generally in Manhattan without mentioning Central Park or a reasonable nearby station connection. No credit if it starts outside Manhattan or from an unrelated city.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Destination: Miami</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Itinerary should end in Miami proper and specify a Miami-area train arrival station (e.g., Miami Amtrak Station) and/or a train-only last-mile connection if arriving first at a nearby rail station in the Miami metro area. Full credit if it clearly reaches Miami by train. Partial credit if it ends at a nearby metro-area station (e.g., Fort Lauderdale) but includes a train-only continuation to Miami. No credit if it ends in a different city/state or requires non-train transport with no train-only continuation proposed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Trains-only constraint (mode compliance)</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
All legs of the itinerary must use trains only (subway/commuter rail/intercity rail are allowed). Full credit if every segment is train-based. Partial credit if one segment is described using a non-train mode but the agent explicitly flags it and provides a train-only alternative for that segment. No credit if any required leg relies on non-train transport without a train-only alternative.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Complete train itinerary with stations and transfers (clarity & coherence)</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a coherent sequence of train segments from Manhattan to Miami, including key intermediate stations and transfer points (NYC departure station, major transfer city/station if used, and Miami arrival station). Full credit if the route is end-to-end, internally consistent, and transfers are understandable. Partial credit if the route is mostly clear but missing one key station/transfer detail or has minor ambiguity while still being followable. No credit if the itinerary is incomplete or logically incompatible (e.g., missing the intercity portion entirely).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Feasibility/realism of rail service used (with allowance for schedule changes)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Itinerary should rely on real, operational passenger rail services for the corridor and plausible connectivity between segments (e.g., Amtrak services and appropriate local rail). Full credit if services cited are appropriate and the plan is plausible; also award full credit if the agent notes that exact schedules/through-cars can change and advises verifying current timetables, and/or provides a reasonable alternate rail-only routing in case a named service/segment is suspended. Partial credit if there are minor inaccuracies (slightly wrong station name/service name) but the overall rail concept and connectivity are correct. No credit if it invents passenger services, uses freight-only lines as passenger service, or proposes impossible rail links.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_111</div>
|
||
<div class="task-summary">find out how many views Adele's "Rolling in the Deep (Official Music Video)" has, and then determine what percent of the worlds population that is using a calculator or equivalent search tool.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find view count of Adele's 'Rolling in the Deep (Official Music Video)'</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Retrieve the current number of views for the specified video. Partial credit may be awarded if an approximate or slightly outdated view count is provided.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Calculate percentage of world population</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine what percent of the world's population the view count represents. This should include specifying the world population figure used and correctly computing (views / population) × 100. Partial credit may be given for using an approximate population estimate or showing the correct formula with minor arithmetic errors.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find the view count for Adele's 'Rolling in the Deep (Official Music Video)'</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate the specified official music video and accurately report its current view count. Partial credit may be awarded for finding the video but not reporting the count, giving an approximate/outdated count, or explaining inability to access the count while providing a reasonable estimate.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Obtain a world population figure for the calculation</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify a reasonable current estimate of the world's population to use as the denominator for the percentage calculation. Partial credit for using a recent estimate or clearly stating and justifying an approximate figure.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Calculate the percentage using a calculator or equivalent search tool</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compute (views ÷ world population × 100) and present the resulting percentage, explicitly using a calculator or equivalent tool. Partial credit for correct calculation without explicitly noting tool usage, or for minor arithmetic errors with a correct method.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find the view count for Adele's 'Rolling in the Deep (Official Music Video)'</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine and report the currently visible number of views for the specific YouTube video titled "Rolling in the Deep (Official Music Video)" by Adele (not a lyric video, live performance, or reupload). Full credit if the agent clearly identifies the correct video and provides the view count as shown on YouTube at time of access (or explicitly notes rounding if it rounds). If YouTube cannot be accessed due to external blockers (CAPTCHA, region restrictions, login wall, site outage), award full credit if the agent clearly reports the blocker and provides the best available alternative view-count evidence (e.g., search-result snippet, cached page, reputable secondary reporting) while noting potential staleness/that views change over time. Partial credit if the video version is somewhat ambiguous but closely related, or if the view count is approximate without stating it is rounded/approximate.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Determine percent of world population represented by the views</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the retrieved view count (or best-available alternative if YouTube was blocked) and a stated world population estimate from a reputable source/year (or a clearly labeled current estimate), compute and report the percentage: views world population 100. Full credit if the calculation is correct given the stated inputs and rounding is reasonable. Do not penalize for small differences due solely to different reputable population estimates/years, as long as the population figure used is stated or clearly implied and the arithmetic matches it. Partial credit if the method is correct but there are minor arithmetic/rounding errors or the population reference is not clearly stated.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_21</div>
|
||
<div class="task-summary">On Wikipedia.org, look up the first Sister City of the city in which Massachusetts Institute of Technology (MIT) resides, and retrieve the 5-day weather forecast for that sister city.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the city where MIT resides</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine and state the correct city (and state) in which Massachusetts Institute of Technology is located by consulting Wikipedia.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Locate the sister cities section on that city’s Wikipedia page</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the identified city’s Wikipedia page and find the section or list specifically detailing its sister cities. Partial credit if the page is found but the sister cities section is not clearly identified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Extract the first sister city listed</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the sister cities list, correctly identify and name the first city mentioned. Partial credit for identifying a sister city but not confirming it is the first in the list.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Retrieve the 5-day weather forecast for that sister city</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Obtain and present the five-day weather forecast for the identified sister city from a reputable weather source, including daily high/low temperatures and conditions. Partial credit if fewer than five days are provided or key forecast details are missing.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify MIT's city of residence</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Correctly determine the city in which MIT is located, as the basis for the sister city lookup. Partial credit may be awarded if the broader metro area is identified but not the exact city.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Use Wikipedia.org to find and select the first sister city</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
On Wikipedia.org, navigate to the identified city's page and locate the Sister cities/Twin towns section. Report the first sister city listed as it appears on the Wikipedia page. Partial credit for finding the sister cities list but not selecting the first item, or for naming a sister city without confirming it is the first. Full credit if the Wikipedia page has no sister city information and this is clearly stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Retrieve a 5-day weather forecast for the selected sister city</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a 5-day weather forecast for the correctly identified sister city (with correct location disambiguation, e.g., city and country). Partial credit for fewer than 5 days or for a current/short-term forecast, or if the forecast is for a similarly named but incorrect location. Full credit if a 5-day forecast is not available and the limitation is clearly explained with the closest available alternative provided.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the city where MIT resides</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine the correct city in which the Massachusetts Institute of Technology (MIT) is located (the city used to find sister cities). Full credit for correctly identifying the city (e.g., Cambridge, Massachusetts). Partial credit if the agent identifies a broader/adjacent area that clearly leads to the correct city page but is ambiguous. No credit if the wrong city is used when the correct one is readily available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Use Wikipedia.org to find the first Sister City of that city</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
On Wikipedia.org, locate the page for the identified city and find its "Sister cities" (or equivalent) section, then select the first sister city listed. Full credit if the agent clearly identifies the first sister city as shown on Wikipedia. If Wikipedia is inaccessible (blocked/down/CAPTCHA) or the relevant section is unavailable, award full credit if the agent clearly reports the limitation and uses a reasonable alternative source (e.g., another Wikimedia mirror or an official city page) while noting it is not Wikipedia. Partial credit if the agent finds a sister city but not the first one despite Wikipedia being accessible, or if the alternative source is used without explaining why Wikipedia could not be used.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Retrieve the 5-day weather forecast for the first sister city</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a 5-day weather forecast for the identified first sister city, from a reputable weather provider. Full credit if five distinct days are provided and the forecast is clearly for the correct city (dates and daily conditions/temperatures, as available). If a 5-day forecast cannot be retrieved due to external limitations (weather site/API blocked/down, paywall, location ambiguity preventing a reliable match), award full credit if the agent clearly reports the limitation and provides the best available alternative (e.g., fewer days available, or a nearby-location forecast) while explicitly stating the mismatch/limitation. Partial credit if fewer than five days are provided without explaining the limitation, or if key daily details are missing despite being available.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_61</div>
|
||
<div class="task-summary">find the location of the first race listed on raceroster.com, and then find the address of a café or coffee shop nearby that I can wait for my husband at while he finishes the race.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate the first race listing and its location</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to raceroster.com, identify the very first race on the listing page, and accurately extract the venue or location details (e.g., city, address, or venue name). Partial credit can be given if only the city or venue name is found without a full address.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find a nearby café or coffee shop address</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the identified race location, find a café or coffee shop in close proximity and provide its name and full street address. Partial credit may be awarded if only the name or partial address is provided.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access raceroster.com website</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to or access the raceroster.com website as specified in the task. Partial credit if the agent references the site but cannot access it due to limitations and states this clearly.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first race listed and its location</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find the first race listed on raceroster.com and provide its location as shown (e.g., venue, city, and/or address). Partial credit for identifying a race but not confirming it is the first, or for providing only partial location details. Full credit if the site does not clearly show a single 'first' race and the agent explains the limitation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find a nearby café/coffee shop and provide its address</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate a café or coffee shop near the identified race location and provide its street address. Partial credit if only the name is given without the address, or if proximity is not clearly established.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Raceroster and determine the listing context for the 'first race'</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to raceroster.com and view a race listing page where races are ordered (e.g., default homepage listings, a directory/search results page, or a location page). Full credit if the agent reaches a page that clearly shows an ordered list of races and states what context/sort is being used (default sort, selected filters/location if any). Full credit if raceroster.com is inaccessible (CAPTCHA/down/login wall/geo-block) and the agent clearly reports the blocker and what was attempted (e.g., refresh, alternate page, different browser path). Partial credit if the agent finds Raceroster content but the ordering context for 'first' is unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first race listed on raceroster.com (within the observed context)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine which race appears first in the ordered list the agent observed and provide enough identifying detail to verify it (e.g., race name and date, and optionally the event page/link or screenshot context). Full credit if the race is clearly the first item on the viewed list. Partial credit if a race is identified but the evidence that it is first is ambiguous (e.g., list not clearly ordered, filters not stated) or if a non-first race is chosen when the first item is visible. Full credit if the site is inaccessible and this is clearly reported (as captured in the access criterion).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the race location (where the race takes place)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Report the race location as presented on the race listing/detail page (city/state and venue/address if available). Full credit for accurately reporting the most specific location information that is available on the page. Partial credit if only partial location is provided when more specific details are clearly available. Full credit if the race page does not list a location or only provides ambiguous/online/virtual details and the agent clearly reports this limitation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Identify a nearby café/coffee shop suitable for waiting</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify at least one cafe/coffee shop plausibly near the race location (near the venue if a venue/address is given; otherwise near the stated city center or a clearly stated reference point). Full credit if the agent uses reasonable evidence of proximity (e.g., map results, stated distance/walking time, or clear neighborhood/adjacent landmark). Partial credit if the cafe is only in the same city with no attempt to establish nearness when the venue/reference point is available. Full credit if the race location is too vague to anchor 'nearby' and the agent clearly explains this and provides a best-effort option near the most specific available reference (e.g., city downtown) or requests the missing detail.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Provide the address of the selected café/coffee shop (or best available location info)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a complete street address for the selected cafe/coffee shop (street, city, state/zip if available). Full credit if the address is provided and corresponds to the chosen cafe. Partial credit if the address is incomplete when a full address is readily available. Full credit if address data cannot be obtained due to external blockers (maps/search inaccessible, business listing not available) or if the business has no published street address, as long as the agent clearly reports the limitation and provides the best available location info (e.g., cross streets, neighborhood, or map pin description).
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_22</div>
|
||
<div class="task-summary">Locate the location of the upcoming NeurIPS conference in 2025 and then find the best local food near the event venue</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify NeurIPS 2025 conference location</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the city and specific venue name where NeurIPS 2025 will be held. Partial credit if only the city is specified or if the venue name is incomplete or approximate.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Recommend local food near the event venue</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Offer at least one recommended local dish or restaurant located near the identified conference venue, including the name and why it is notable. Partial credit for generic or incomplete recommendations.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify NeurIPS 2025 event location (city and venue)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Accurately identify the host city and specific event venue for the 2025 NeurIPS conference. Partial credit if only the city or only the venue is provided. Full credit if the agent determines that the official location/venue is not yet announced and clearly states that, rather than guessing.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find the best local food near the event venue</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide high-quality local food recommendations in close proximity to the identified venue (e.g., within reasonable walking or short transit distance). Assess proximity and quality/relevance of the recommendations. Partial credit if options are provided but are not clearly near the venue, are generic, or lack rationale. Full credit may be awarded if the venue is not announced and the agent explicitly notes the limitation and avoids tying recommendations to an unconfirmed venue.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify NeurIPS 2025 conference location</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and report where NeurIPS 2025 will be held. Full credit if the agent provides the correct host city and venue (or official event site/venue name if listed). Partial credit if only the city or only the venue is correctly identified but the full location context is missing/ambiguous. Full credit if the agent clearly states that the official NeurIPS 2025 location (city and/or venue) is not yet announced and supports this with a credible source or clear indication from official NeurIPS communications.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find best local food near the event venue</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the identified location details, recommend at least one well-regarded local food option plausibly near the venue. Full credit if the agent (a) provides a clear local recommendation and explains why it is considered good and nearby, OR (b) if venue/address-level detail is unavailable or unverified, clearly states this blocker and instead recommends strong local food options in the host city (or within the venue’s district/area if known) while explaining the limitation. Partial credit if recommendations are local but proximity is not addressed at all, or if the options are generic/non-local without tying them to the locale. Full credit should not be withheld due to external volatility (e.g., closures, hours, reservation availability, shifting ratings) as long as the agent reasonably attempts to identify reputable options and notes any uncertainty.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_114</div>
|
||
<div class="task-summary">Locate the top-seller RPG game on Steam and identify its matching game controller. On Amazon, find this controller and add it to the cart, stopping at the review page.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate the top-seller RPG game on Steam</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the Steam website, find the RPG category's top-sellers list, and correctly identify the number-one selling RPG title. Partial credit if Steam and RPG section are accessed but the top-seller is misidentified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the matching game controller</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine a game controller that is compatible with and appropriate for the identified top-selling RPG. Partial credit if a generally compatible controller is chosen but not specifically recommended for the game.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the controller on Amazon</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search Amazon for the specified controller model and locate the correct product page. Partial credit if a similar or generic version is found instead of the exact model.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Add the controller to the cart</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Add the identified controller to the Amazon shopping cart without proceeding to checkout. Partial credit if the correct product page is reached but 'Add to Cart' is not executed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Stop at the order review page</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the Amazon order review page (showing cart contents ready for purchase) and stop before entering any personal or payment information. Partial credit if the cart is visible but the review page is not reached.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate the top-seller RPG game on Steam</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully identify the current top-selling RPG game on Steam. Partial credit may be awarded for navigating to the Steam RPG category or Top Sellers list but not conclusively identifying the top entry, or for noting if a top-seller list is unavailable/region-restricted at the time.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the matching game controller</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine and name a game controller that matches the identified RPG game (i.e., a controller suitable for playing the game). Partial credit may be awarded for proposing a reasonable, commonly used controller even if the exact match is unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the controller on Amazon</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate the identified controller on Amazon. Partial credit may be awarded for reaching a closely related product page or stating clearly if the exact controller is unavailable/out of stock.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Add the controller to the Amazon cart</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Add the identified controller to the Amazon cart. Partial credit may be awarded for attempting to add to cart and indicating any blockage (e.g., out of stock, region restrictions). No checkout or personal information entry should be performed.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Stop at the review page without proceeding to checkout</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
After adding to cart, stop at a review page (e.g., product reviews section or cart review) without entering any personal information or proceeding to checkout. Partial credit may be awarded for clearly stopping at the cart page or product page if a distinct review page is not accessible.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate the current top-seller RPG game on Steam</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify the current top-selling RPG game on Steam using Steam’s Top Sellers with RPG category/tag filtering (or an equivalent Steam-supported view). Full credit if the agent demonstrates it used a top-seller-ranked view and clearly identifies the top-selling RPG at time of lookup. Full credit if Steam is inaccessible (blocked/down/region/CAPTCHA) and the agent clearly reports the blocker and uses a reasonable alternate Steam-accessible method/view (or clearly explains why verification cannot be completed). Partial credit if the agent names a plausible top-selling RPG but does not establish that it is the top seller (unclear sorting, not a top-sellers view, or missing evidence of ranking).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the matching game controller for that game</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the identified game’s Steam page, determine any explicitly indicated controller match (recommended/named controller) or, if no specific model is named, the explicit controller support shown (e.g., “Full controller support,” supported controller standards/types). Full credit if the agent correctly ties the controller information to the specific game page. Full credit if the game page does not specify a particular controller model and the agent explicitly states that no specific matching controller is named while reporting the supported controller type/standard shown on Steam. Partial credit if the agent chooses a generic controller without linking it to explicit Steam evidence when such evidence is available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the identified controller on Amazon</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search Amazon and land on a product page that matches the identified controller (correct brand/model/variant where applicable). Full credit if the product page corresponds to the identified controller. Full credit if the exact controller cannot be found due to external factors (regional catalog differences, listing removed, out-of-stock/no Buy Box, restricted item) and the agent clearly reports this and selects the closest clearly-labeled equivalent that matches the primary intent. Partial credit if the agent lands on a close but not exact match when the exact match appears available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Add the controller to the Amazon cart and stop at the review page</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Add the selected controller to the Amazon cart and proceed toward checkout until reaching the order review stage (“Review your order” or equivalent), then stop without placing the order. Full credit if the item is added and the agent reaches the review page and stops. Full credit if the agent cannot reach the review page due to uncontrollable blockers (login/CAPTCHA/address required/region restrictions/checkout gating) but clearly reports the blocker and shows the item was added to cart (or that adding was prevented by the blocker). Partial credit if the item is added to cart but the agent stops at cart without attempting to proceed toward the review step when no blocker is present.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_106</div>
|
||
<div class="task-summary">use a mortgage rate calculator tool online to see what my estimated monthly payment will be (including only principal and interest) for a $500,000 home with a down payment of $80,000 over 30 years at an interest rate of 6.0% in 98101.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access mortgage rate calculator tool online</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to and open an online mortgage rate calculator tool as specified in the task description.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Enter property price of $500,000</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the correct home purchase price of $500,000 into the mortgage calculator.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Enter down payment of $80,000</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the specified down payment amount of $80,000 into the calculator.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Set loan term to 30 years</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Select or input a loan amortization period of 30 years as required.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Set interest rate to 6.0%</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Enter the annual interest rate of 6.0% into the mortgage calculator.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Enter location zip code 98101</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the location information by entering the zip code 98101 if required by the tool.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 7:</span>
|
||
<span class="criterion-text">Retrieve estimated monthly payment (P&I only)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Obtain and report the calculator's estimated monthly payment, including only principal and interest.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Use an online mortgage rate calculator tool</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully access and use a mortgage rate calculator tool online, as explicitly required by the task. Partial credit may be given if an online tool is identified but not fully used, or if the tool is inaccessible and this is clearly documented.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Enter all specified inputs exactly</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the specified parameters into the calculator: home price $500,000; down payment $80,000; loan term 30 years; interest rate 6.0%; and location/ZIP 98101 (if the tool accepts or requires a location). Partial credit may be awarded if most inputs are correct but one is missing or misentered, or if the tool does not accept ZIP and this is explicitly noted.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Configure payment to include only principal and interest</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Ensure the monthly payment shown includes only principal and interest (exclude taxes, insurance, HOA, PMI, etc.). Partial credit may be awarded if extras are shown but the P&I portion is clearly isolated and reported, or if the tool cannot exclude extras and this limitation is clearly stated along with the P&I figure derived from the tool’s breakdown.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Report the estimated monthly payment result clearly</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the estimated monthly payment amount (principal and interest only) obtained from the online calculator. The result should be clearly stated. Partial credit may be awarded for an approximate figure or range if the tool provides limited precision and this is explained.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Attempt to use an online mortgage payment calculator (or a reasonable alternative if blocked)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Agent uses or clearly attempts to use an online mortgage/mortgage payment calculator. Full credit if the agent successfully uses an online calculator, OR if it documents an uncontrollable blocker (e.g., CAPTCHA/paywall/site down) and then uses a different accessible online calculator to obtain the estimate. Partial credit if the agent provides a plausible estimate but the use of any online calculator is unclear. No credit if the agent neither uses nor attempts to use an online tool and does not explain why.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Enter the correct loan scenario inputs (or equivalent fields) to compute P&I</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Inputs reflect the task: $500,000 home price and $80,000 down payment (equivalently $420,000 loan amount), 30-year term, 6.0% interest rate. ZIP/location 98101 should be entered if the calculator supports it; do not penalize if the calculator has no ZIP field or if ZIP does not affect the principal-and-interest computation and the agent notes this. Full credit if all core financial inputs are correct or entered via equivalent fields. Partial credit if one core input is slightly off but the agent otherwise demonstrates correct setup/intent, or if ZIP is omitted because the calculator does not support it. No credit if core financial inputs are materially wrong when correct entry was possible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Report the estimated monthly payment for principal and interest (P&I) only</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Agent reports the monthly payment amount specifically for principal + interest, excluding taxes, insurance, HOA, PMI, etc. Full credit if the P&I value is clearly labeled as such (or the calculator explicitly shows a P&I component and that is the value reported). If the calculator only provides a total payment that appears to include other costs, full credit if the agent explains the limitation and uses another calculator/setting to isolate P&I; partial credit if the agent reports the total but also provides a reasonable P&I estimate with a clear caveat. No credit if the agent reports a payment that includes other costs without clarifying or isolating P&I.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_94</div>
|
||
<div class="task-summary">I want to learn how much I should save for my 2-year olds college fund. Use the Office of Financial Rediness college savings calculator and input the following fields: 3% education cost inflation, $50,000 in current savings, $250 in monthly contributions with 6% rate of return. If their tuition is going to be $50,000 per year and room/board $12,000, how much more per month do i need to save according to the tool? (Hint: do not use the sliders)</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access the Office of Financial Readiness college savings calculator</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to and open the specified college savings calculator on the Office of Financial Readiness website.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Enter 3% education cost inflation rate</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the 3% annual inflation rate for education costs into the appropriate field (not using sliders).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Enter current savings of $50,000</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the existing college savings amount of $50,000 into the calculator (not using sliders).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Enter monthly contributions of $250</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Set the monthly contribution to $250 in the calculator (not using sliders).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Enter 6% rate of return</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the expected annual rate of return of 6% into the calculator (not using sliders).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Enter projected tuition cost of $50,000 per year</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the annual tuition cost of $50,000 into the appropriate field (not using sliders).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 7:</span>
|
||
<span class="criterion-text">Enter projected room and board cost of $12,000 per year</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the annual room and board cost of $12,000 into the calculator (not using sliders).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 8:</span>
|
||
<span class="criterion-text">Follow non-slider input requirement</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Ensure that all numeric inputs are entered manually or via text fields rather than using slider controls.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 9:</span>
|
||
<span class="criterion-text">Calculate additional monthly savings needed</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use the calculator’s output to determine and report how much more per month must be saved to meet the projected college costs.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access the Office of Financial Readiness college savings calculator</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to and use the specified Office of Financial Readiness college savings calculator (not a different tool). Partial credit if the agent references it but appears to use another calculator.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Set child's age correctly</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the child's current age as 2 (or the equivalent setting in the tool reflecting a 2-year-old). Partial credit if the agent indicates the child’s age but does not clearly apply it in the calculator.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Enter education cost amounts</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input tuition as $50,000 per year and room/board as $12,000 per year, as specified. Partial credit if only one of the two values is entered correctly or if a correct combined total is entered when the tool requires a single annual cost input.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Enter financial assumptions</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input the following exactly: 3% education cost inflation, $50,000 current savings, $250 monthly contributions, and 6% annual rate of return. Partial credit for correctly entering some but not all values.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Comply with 'do not use sliders' instruction</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Ensure values are entered without using sliders (typed or otherwise manually entered). Partial credit if most inputs are typed but one appears adjusted by slider.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Report the calculator's 'how much more per month' result</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Retrieve and clearly state the incremental additional monthly savings required according to the tool (beyond the current $250/month). Full credit if the agent provides the exact incremental figure; partial credit if only the total required monthly amount is given without the 'more per month' delta.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access and use the Office of Financial Readiness college savings calculator (as specified)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to and attempt to use the Office of Financial Readiness college savings calculator to compute the result. Full credit if the agent uses this specific tool to produce the result, OR if the agent clearly documents being blocked by an uncontrollable issue (site down, CAPTCHA, login requirement, broken calculator, tool not loading). Partial credit if the attempt is unclear or the wrong tool is used without justification.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Enter the specified calculator inputs via typed/manual entry (not sliders), as available in the tool</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Input all required fields exactly as specified using typed/manual entry (not sliders): 3% education cost inflation, $50,000 current savings, $250 monthly contributions, 6% rate of return, tuition $50,000 per year, and room/board $12,000 (or the closest equivalent fields if labeled differently). Full credit if all values are entered correctly via manual entry. If the tool enforces sliders only or lacks one or more of these fields, full credit can still be earned by (a) attempting manual entry where possible and (b) explicitly stating which fields are unavailable/slider-locked and therefore could not be entered as requested. Partial credit if one value is entered incorrectly or the manual-entry constraint is not followed when avoidable.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Report the calculator's required additional monthly savings amount (incremental above $250/month)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Read the calculator output and answer: how much more per month needs to be saved beyond the stated $250/month (i.e., additional monthly amount). Full credit if the incremental amount is clearly stated and consistent with the tool output (either directly shown by the tool or correctly derived from a total monthly amount shown by the tool). If the tool output does not provide an incremental figure or the relevant output is not visible due to an uncontrollable tool issue, full credit can still be earned by clearly stating what the tool did show (e.g., total required monthly contribution) and why the incremental amount cannot be determined from the tool as presented. Partial credit if only the total required monthly contribution is reported without clearly converting to the 'more per month' amount when the conversion is possible from the displayed output.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_75</div>
|
||
<div class="task-summary">go to investor.gov and compute how much money I will have with an initial principle of $10000, to which I make monthly contributions of $200 over 10 years. Assume an interest rate of 5.0 compounded quarterly. Additionally, tell me the colors of the lines it plots in the results.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access investor.gov calculator</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to investor.gov and locate the appropriate calculator for computing future value with regular contributions.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Compute future value with specified parameters</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Enter an initial principal of $10,000, monthly contributions of $200, a 10-year timeframe, a 5.0% interest rate, and quarterly compounding, then perform the calculation. Partial credit for applying some but not all parameters correctly or for arithmetic errors.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify plot line colors</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Report the colors of the lines shown in the results plot. Partial credit if some but not all line colors are correctly identified.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access investor.gov and the appropriate calculator</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to investor.gov and locate/open the relevant compound interest or savings calculator needed to perform the computation. Partial credit if the site is accessed but the correct tool is not clearly identified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Enter the specified parameters correctly</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use the tool with exactly the given inputs: initial principal $10,000; monthly contributions $200; time horizon 10 years; annual interest rate 5.0%; compounding quarterly. Partial credit if most inputs are correct but one or more (e.g., compounding frequency, contribution frequency, or time) are set incorrectly or not demonstrated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Compute and report the final amount</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Obtain the calculation result from the tool and clearly report the final future value. Partial credit if an amount is computed but not clearly identified as the final total, or if the value is approximate without stating it is an estimate.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Report the colors of the plotted lines in the results</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify and state the colors of the lines shown in the results plot produced by the tool. Partial credit if only one color is reported or if the agent notes that no plot is available for the given tool/context.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Attempt to access investor.gov calculator/tool</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to investor.gov and attempt to use an on-site calculator/tool relevant to computing investment growth. Full credit if the agent makes a reasonable attempt but is blocked (e.g., site down, CAPTCHA, tool not loading) and clearly reports the blocker. Partial credit if the attempt is unclear or investor.gov is not attempted despite being available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Compute the investment result using investor.gov or a documented equivalent method</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compute the final account value for the specified scenario. Full credit if the agent uses investor.gov successfully OR, if investor.gov is inaccessible/unusable, uses a reasonable alternative method (e.g., explicit finance math or another reputable calculator) and explains that it is a substitute due to the blocker. Partial credit if the method is plausible but under-specified or not clearly tied to the parameters.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Enter/apply the correct calculation parameters</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Apply the task parameters correctly: initial principal $10,000; monthly contribution $200; time horizon 10 years; interest rate 5.0%; compounding quarterly. Full credit if all parameters are correctly applied (via investor.gov inputs or equivalent math). Partial credit if one parameter is slightly wrong but the agent acknowledges/identifies the discrepancy or provides both interpretations (e.g., reconciling monthly contributions with quarterly compounding). No credit if multiple key parameters are wrong or omitted.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Report the computed final amount after 10 years</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the final computed account value after 10 years consistent with the stated parameters (allowing minor rounding differences). Partial credit if a near-correct value is provided but the agent appears to have used a different convention without reconciling it. No credit if the amount is missing or clearly inconsistent with the provided parameters.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Identify the colors of the plotted lines in the investor.gov results</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
State the colors of the lines shown in the investor.gov results plot. Full credit if all line colors are correctly identified as displayed. Full credit also if the agent cannot view the plot due to an uncontrollable issue (e.g., investor.gov/tool/plot not loading, blocked access) and explicitly reports that limitation rather than guessing. Partial credit if only some line colors are provided or if descriptions are ambiguous.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_96</div>
|
||
<div class="task-summary">can you go the latest news release from the US Dept. of Labor, and tell me who the media contact is and how many other contacts there are in their department?</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate the latest news release</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to and identify the most recent news release on the US Department of Labor website.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the media contact</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Accurately extract the name of the media contact listed in that news release. Partial credit for incomplete or ambiguous identification.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Count other department contacts</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Correctly count how many additional contacts are listed in the same department aside from the media contact.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access the latest US Department of Labor news release</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to and open the most recent (latest) news release from the US Department of Labor. Partial credit may be awarded for accessing a DOL news release without verifying it is the latest or for identifying the correct section but failing to open the specific release. Full credit can be awarded if the site does not clearly indicate recency and the agent explains the limitation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the media contact</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and report who the media contact is for that specific news release (e.g., the name listed under 'Media Contact'). Partial credit may be awarded for identifying a contact but not clearly specifying it is the media contact, or for noting that no media contact is listed if that is the case.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Count other contacts in the media contact's department</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine how many other contacts (besides the primary media contact) are listed as being in the same department as the media contact on that release. Partial credit may be awarded for counting total contacts without filtering by department, or for explaining that department information is unclear and providing the most reasonable count available. Full credit if no other contacts exist and the agent explicitly states zero.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access the US Department of Labor newsroom/news releases listing and assess recency ordering</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the official US Department of Labor site (e.g., Newsroom/News Releases listing) and attempt to determine how items are sorted by recency (date/time, pagination). Full credit if the agent successfully reaches the listing and can evaluate recency ordering, OR if the agent is blocked by an uncontrollable issue (site down, CAPTCHA, access denied) and clearly reports what prevented access. Partial credit if the agent uses an unofficial mirror/source without explaining why the official site could not be used.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the latest US Department of Labor news release</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the accessible official listing, select the most recent item that is clearly a "news release" and identify it (e.g., title and date/time). Full credit if the agent correctly identifies the latest release, or if recency is ambiguous (time zones, multiple items same date, mixed content types) and the agent selects a defensible near-latest release while explaining the ambiguity. Full credit if the agent cannot confirm the latest due to an uncontrollable blocker and clearly documents the limitation. Partial credit if the agent selects an older release when a clearly newer news release is visible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Report the media contact for that news release</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the identified latest news release page, extract and report the media contact exactly as labeled (person or office). Full credit if correctly reported, OR if the release has no media-contact field/contact block and the agent explicitly states that none is listed on the page. Partial credit if the agent provides a general DOL contact that is not labeled as the media contact when a media contact is present, or if the contact is incomplete (e.g., missing name/office when shown). Full credit if the agent cannot access the release page due to an uncontrollable blocker and clearly reports the issue.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Count how many other contacts are in the same department section</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine how many additional contacts (beyond the media contact) are listed in the same contact block/press office/department contact section on that release page, and report the count with a brief note on what was counted (e.g., additional named individuals vs. offices). Full credit if the count is correct, OR if the page provides no additional contacts and the agent reports 0, OR if the page’s contact information is not presented as a list of distinct contacts (e.g., only a generic email/phone, or no contact section) and the agent states that the number of "other contacts" cannot be determined from the release. Partial credit if the method is unclear or the agent likely counted non-contacts (e.g., phone numbers) as separate contacts without noting ambiguity. Full credit if the agent cannot access the release page due to an uncontrollable blocker and clearly reports the issue.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_31</div>
|
||
<div class="task-summary">Find one of Beyonce's favorite soul food restaurants in houston, go to their website, and find out when they opened. How much older are they than Beyonce herself?</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify a Beyoncé favorite soul food restaurant in Houston</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and name at least one soul food restaurant in Houston that Beyoncé has mentioned as a favorite. Partial credit if the restaurant is commonly cited without explicit Beyoncé reference; full credit only if there is evidence it is one of her favorites.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Access the restaurant's official website</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to or locate the restaurant’s official website from which opening information can be obtained. Partial credit for finding a third-party source if no official site is available; full credit for the official site.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Determine the restaurant’s opening date</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and report the date (year and, if possible, month/day) when the restaurant first opened. Partial credit for estimating the year if the exact date isn't listed; full credit for the precise opening date.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Calculate how much older the restaurant is than Beyoncé</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using Beyoncé’s birth year (1981) and the restaurant’s opening date, compute the age difference in years. Partial credit for an approximate calculation; full credit for the accurate age difference.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify a qualifying restaurant</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find one restaurant that meets all explicit constraints: it is in Houston, serves soul food, and is known as one of Beyoncé's favorite restaurants. Partial credit if the restaurant matches some but not all constraints (e.g., in Houston but not clearly soul food, or Beyoncé-associated but not confirmed as a favorite).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Access the restaurant's official website</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to and access the restaurant’s official website (not a third-party listing). Partial credit if only a third-party page is found when an official site appears unavailable, with this unavailability noted.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the restaurant’s opening date on their website</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate the opening year/date from the restaurant’s official website. Full credit if the site clearly provides the opening date. If the official site does not provide this information, full credit may be awarded for explicitly stating that it’s unavailable on the site; partial credit for unclear or inferred dates without confirmation from the site.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Calculate how much older the restaurant is than Beyoncé</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compute the age difference between the restaurant’s opening date and Beyoncé’s birth date. Full credit for a correct calculation accounting for years and, where possible, months/days; partial credit for an approximate year-only difference when exact dates are not available.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify one of Beyoncé's favorite soul food restaurants in Houston</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine a specific Houston soul food restaurant that is explicitly described by at least one credible source as one of Beyoncé's favorites (or a clearly equivalent phrasing such as she ‘loves,’ ‘frequents,’ or it’s a ‘go-to’). Full credit if the restaurant is correctly identified and the Beyoncé connection is supported with evidence/citation. Full credit is also acceptable if, after reasonable search effort, no explicit ‘favorite/go-to’ phrasing can be found; in that case the agent should clearly state this limitation and select the best-supported Houston soul food restaurant that is credibly linked to Beyoncé (e.g., reported as visited/recommended by her). Partial credit if the restaurant is a plausible Houston soul food spot but the Beyoncé connection is weak/uncited/ambiguous. No credit if the restaurant is not in Houston, not soul food, or not connected to Beyoncé.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Use the restaurant's official website to find the opening date/year</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use the identified restaurant's official website to locate information stating when it opened (date or year), and clearly attribute the information to the site if found. Full credit if the opening year/date is taken directly from the restaurant's website (e.g., About/History page). Full credit if the agent attempts the official website but it is inaccessible (down/blocked/CAPTCHA/login), or if the site does not state an opening date; the agent must clearly report the blocker/absence and where they looked on-site. Partial credit if the agent provides an opening date from a third-party source after failing to obtain it from the official site, as long as the official-site attempt and failure is clearly documented. No credit if an opening date is fabricated or presented as coming from the official website when it is not.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Determine Beyoncé's birth date/year accurately</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide Beyoncé's birth date or at minimum birth year correctly (needed for age comparison). Full credit for correct value (e.g., born 1981; full date acceptable). Partial credit if only an approximate/uncertain year is provided but is close enough to enable a comparison with explicit caveats. No credit if incorrect year/date is used or invented without basis.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Compute how much older the restaurant is than Beyoncé</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Calculate the difference between the restaurant's opening year/date and Beyoncé's birth year/date and report the result. Full credit for correct arithmetic and a clear statement (e.g., 'opened in YEAR, Beyoncé born YEAR, restaurant is N years older'). If only years (not exact dates) are available, full credit for a clearly stated year-based difference and noting it is approximate with respect to months/days. Partial credit if the calculation is slightly off due to missing month/day precision but the approach is explained. No credit if the comparison is not provided or is numerically wrong without explanation.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_58</div>
|
||
<div class="task-summary">On Wikipedia.org, find the city containing the oldest university in the US, use this location to find the lowest priced compact car rental for November 17-19, 2025, on Rentalcars.com.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the oldest university in the US on Wikipedia</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to Wikipedia and locate the page or section that clearly states which is the oldest university in the United States. Partial credit if the agent finds relevant university names but does not confirm which is oldest.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Determine the city containing that university</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the Wikipedia page, extract the city in which the identified oldest university is located. Partial credit if the agent finds the university but misidentifies the city.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Access Rentalcars.com and set search parameters</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Open Rentalcars.com, enter the identified city as the pickup location, and set pickup date to November 17 2025 and drop-off date to November 19 2025. Partial credit if the correct website is accessed but dates or location are entered incorrectly.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Filter results for compact cars</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Apply the compact car filter on the search results page. Partial credit if a close vehicle category is selected instead of exactly 'compact'.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Identify the lowest priced compact car option</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Among the filtered compact car results, find and report the lowest daily or total rental price. Partial credit if the agent reports a reasonable low-cost option but not the absolute lowest.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find the city on Wikipedia.org containing the oldest university in the US</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Access Wikipedia.org and identify the city where the oldest university in the United States is located. Full credit requires correctly naming the city and showing it was determined using Wikipedia. Partial credit may be awarded if the university is identified but the city is missing or incorrect, or if the source used is unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Use the identified city on Rentalcars.com with the specified dates</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to Rentalcars.com and initiate a car rental search using the city found on Wikipedia as the pickup location, with dates set to November 17–19, 2025. Full credit requires correct site access, correct location, and correct dates. Partial credit may be awarded if only some parameters (e.g., dates or location) are correctly set.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify and report the lowest priced compact car rental</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the Rentalcars.com search results, find the Compact category and determine the lowest priced option for the specified dates and location. Full credit requires clearly reporting the lowest price and key details (e.g., provider and vehicle/class). Partial credit may be awarded if compact options are listed but the lowest is not clearly identified, or if compact cars are unavailable and this is explicitly stated.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the city containing the oldest university in the US (via Wikipedia.org)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Wikipedia.org to determine the oldest university (or oldest institution of higher education/university, as described on Wikipedia) in the United States and extract the city where it is located. Full credit if the agent cites/grounds the choice in Wikipedia and states an unambiguous city. Partial credit if the university is correct but the city is missing/unclear, or if the city is correct but the Wikipedia grounding is weak. If Wikipedia presents ambiguity (e.g., multiple candidates depending on definition), full credit if the agent notes the ambiguity and proceeds with a defensible Wikipedia-supported choice and city.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Use Rentalcars.com (or report blockers) to search compact rentals for Nov 17–19, 2025 in the identified city</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to navigate to Rentalcars.com and initiate a search using the identified city as the pickup location and the specified dates (Nov 17–19, 2025). Full credit if the agent performs the search with correct location and dates, OR if Rentalcars.com is inaccessible (e.g., CAPTCHA, outage, required login, geoblock) and the agent clearly reports the blocker and what was attempted. Partial credit if the agent makes a minor input mistake but corrects it, or if it uses another platform only after clearly documenting that Rentalcars.com could not be used.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the lowest-priced compact car option for the specified dates on Rentalcars.com (or report unavailability)</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the Rentalcars.com results for the specified city and dates, identify the lowest priced option within the 'Compact' car category. Full credit if (a) a compact-category option is available and the agent demonstrates it is the lowest-priced compact shown (e.g., by sorting, filtering, or visibly comparing compact results) and records key details (price as shown, supplier/company, and compact-category confirmation), OR (b) no compact category is available / no cars are available / prices are not shown and the agent clearly reports this limitation from the site. Partial credit if a compact option is identified but the agent does not establish it is the lowest-priced compact, or omits one key detail when the listing is otherwise clear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Stop before any booking critical point (no checkout / no personal or payment info)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Progress only as far as needed to determine the lowest-priced compact option and its details, without completing a booking or entering any personal/payment information. Full credit if the agent stops before final reservation/checkout steps and does not provide or fabricate user details. Partial credit if the agent navigates near the final booking stage but does not submit/confirm anything. No credit if the agent completes a booking/reservation or enters personal/payment information.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_82</div>
|
||
<div class="task-summary">can you find a quote from Dario Amodei saying that AI will take a lot of jobs. What did he predict the unemployment rate would be, and how many percentage points higher is that than the maximum unemployment the US experienced in 2001?</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate and cite a relevant quote from Dario Amodei about AI taking jobs</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find an exact quotation in which Dario Amodei states that AI will take a lot of jobs. Partial credit may be awarded for a close paraphrase or a correct citation with minor wording differences.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the predicted unemployment rate</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Extract the specific unemployment rate percentage that Amodei predicted in the quote. Partial credit may be awarded for approximate values or formatting errors.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Calculate the percentage‐point difference from the 2001 maximum unemployment rate</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compute how many percentage points higher Amodei’s predicted unemployment rate is compared to the highest US unemployment rate in 2001. Partial credit may be given for correct methodology with minor arithmetic mistakes.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Provide a direct quote from Dario Amodei stating that AI will take a lot of jobs</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and present a clear, direct quote attributed to Dario Amodei that explicitly conveys the idea that AI will take many jobs. Partial credit may be awarded for quotes that imply job displacement but are less explicit, or for paraphrases clearly attributed to Amodei.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">State the unemployment rate Dario Amodei predicted</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify and report the specific unemployment rate prediction made by Amodei in connection with the quote or related context. Partial credit may be awarded for an approximate figure or range if a precise value is unavailable.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the maximum US unemployment rate in 2001</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine and state the highest unemployment rate the US experienced in 2001. Partial credit may be awarded for providing a closely related figure (e.g., annual average) if the maximum is not found, but full credit requires the maximum as requested.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Calculate the percentage-point difference</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compute how many percentage points higher Amodei’s predicted unemployment rate is compared to the 2001 maximum, and report the result. Partial credit may be awarded for an attempted calculation with minor arithmetic errors.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find a quote from Dario Amodei saying AI will take a lot of jobs</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide at least one attributable quote from Dario Amodei that explicitly conveys that AI will take many jobs (e.g., mentions job loss, job displacement, or large-scale automation). Full credit if the quote is clearly attributed and contains the relevant claim. Partial credit if the statement is paraphrased rather than quoted, or if the quote is ambiguous about job loss. Full credit if the agent reports that no such quote could be found after reasonable search, including briefly stating what sources/queries were tried and noting blockers like paywalls/captchas.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Report Amodei's predicted unemployment rate due to AI</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
State the unemployment rate Dario Amodei predicted (percent) in the cited source. Full credit if the numeric rate is correctly extracted and clearly presented (optionally including timeframe/context if present). Partial credit if the agent provides a plausible figure but the context is unclear, the figure is presented as a range when only a point estimate was asked (or vice versa), or it appears to be from a closely related but not definitively Amodei-attributed source. Full credit if, after a reasonable attempt to locate/verify the prediction in accessible sources, the agent clearly reports it cannot verify a specific numeric rate (e.g., due to paywall, conflicting reports, or inability to locate the original statement), and explains the limitation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the maximum US unemployment rate in 2001</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and state the maximum US unemployment rate experienced in calendar year 2001 (percent), indicating it is the maximum (not the annual average). Full credit if the maximum value is correctly reported and tied to a credible public source (e.g., BLS series). Partial credit if a 2001 unemployment figure is provided but it is not established to be the maximum or the source is unclear. Full credit if the agent makes a reasonable effort using alternative public sources and clearly reports inability to verify the maximum due to access limitations or source unavailability.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Compute how many percentage points higher Amodei's prediction is than the 2001 maximum</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compute and report the difference in percentage points: (Amodei predicted unemployment rate) minus (maximum US unemployment rate in 2001). Full credit if the arithmetic is correct and expressed in percentage points. Partial credit if the method is correct but there is a minor arithmetic/rounding error, or if the result is mistakenly reported as a percent change rather than percentage points. Full credit if the agent cannot compute the difference solely because one or both required numeric inputs could not be verified due to external/source-access limitations, provided the agent explicitly states what is missing and why (and computes the difference if later sufficient numbers are available).
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_74</div>
|
||
<div class="task-summary">Find a job on USA jobs in the 10003 area code, and tell me whether the salary of the first listing is above or below the median for that role nationally on salary.com</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access USAJobs website</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the USAJobs (usajobs.gov) site without requiring login or entering any personal information.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Search for jobs in area code 10003</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use the site’s search functionality to filter or search for jobs specifically in ZIP code 10003.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify salary of the first job listing</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate the first job in the search results and accurately extract the listed salary information for that position.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Find national median salary on salary.com</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Go to salary.com, search for the same role title found in the first listing, and determine the national median salary for that role without logging in or providing personal data.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Compare first listing salary to median</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compare the salary from the first listing against the national median salary and clearly state whether it is above or below that median.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Search USAJOBS for listings in the 10003 area</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Access the USAJOBS website and locate job postings tied to the 10003 area. Full credit if the agent searches specifically for jobs in 10003 and notes if none are available. Partial credit if searching nearby locations or indicating issues with filtering. No requirement to sign in or apply.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first listing and extract its role and salary</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the search results, use the first listing returned and capture the job title and salary (or salary range) shown. Partial credit if the listing is found but salary is missing and this is explicitly stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find national median salary for the role on salary.com</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use salary.com to look up the national median salary for the identified role. Partial credit if a closely related role is used when an exact match is unavailable, or if unavailability is clearly reported.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Compare and conclude above or below the median</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Compare the first listing’s salary to the salary.com national median and clearly state whether it is above or below. Partial credit if numeric values are provided but the conclusion is unclear, or if a reasonable method (e.g., midpoint of a range) is used when needed.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access USAJOBS and run a location search for ZIP code 10003 (or closest supported equivalent)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use USAJOBS to search for jobs in/near ZIP code 10003 (or, if USAJOBS does not support ZIP targeting cleanly, an equivalent query such as "10003" location, "New York, NY" with radius, or a map-based filter). Full credit if the agent makes a reasonable attempt and either reaches results relevant to the 10003 area or clearly reports a blocker (CAPTCHA, outage, login wall, blocking). Partial credit if the agent searches NYC generally without explaining why 10003-specific filtering could not be applied or verified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first USAJOBS listing shown and capture its role/title and salary</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the USAJOBS results page (under the observed default/selected sort order, which should be stated or evident), identify the first listing shown and report its job title/role and the salary (range or stated pay). Full credit if the first listing is unambiguous and salary is captured accurately (from results or the listing detail page). Full credit if the first listing is identifiable but salary is not displayed/available and the agent clearly reports that limitation after checking the detail page. Partial credit if either title/role or salary is missing/incomplete despite being available, or if the ‘first listing’ selection is ambiguous due to not indicating the ordering used.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the national median salary for the closest matching role on Salary.com</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Salary.com to locate a national median salary figure for the same (or closest clearly justified) role category matching the USAJOBS listing’s title/role. Full credit if the agent finds and reports the Salary.com national median. Full credit if Salary.com is inaccessible (paywall/blocking) or no suitable matching role exists and the agent clearly reports the limitation and what was tried. Partial credit if the match is loose without noting assumptions or mismatch.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Compare USAJOBS pay to the Salary.com national median and state above/below (with a clear method for ranges)</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the USAJOBS salary and the Salary.com national median, explicitly state whether the USAJOBS pay is above or below the national median. If the USAJOBS listing provides a range, full credit if the agent uses a defensible, clearly stated method (e.g., compares midpoint to median, or states whether the entire range is above/below/overlaps the median and then gives a clear above/below determination based on the chosen method). Full credit if a comparison cannot be made because either the USAJOBS salary or Salary.com median is unavailable and the agent clearly states why comparison is not possible. Partial credit if an above/below conclusion is given but the method for handling ranges is unclear.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_25</div>
|
||
<div class="task-summary">find an official microsoft support page showing a tutorial about pivot tables. Somewhere on that page, they must have an example spreadsheet or screenshot of one. What is the first row of that example table?</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate official Microsoft Support pivot tables tutorial</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Agent navigates to and identifies an official Microsoft Support web page that provides a tutorial on creating or using pivot tables.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Confirm presence of example spreadsheet or screenshot</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Agent verifies that the page includes an example spreadsheet or a screenshot illustrating a pivot table.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the example table and its first row</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Agent finds the specific example table or screenshot on the page and determines which row is the first data row of that table.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Extract and report the first row accurately</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Agent accurately extracts and transcribes the content of the first row of the example table as shown on the page.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate an official Microsoft support tutorial page about pivot tables</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and identify an official Microsoft support page (e.g., on support.microsoft.com) that is specifically a tutorial about pivot tables. Partial credit may be awarded if the page is official but not clearly a tutorial, or if it is a Microsoft page about pivot tables but not on the support subdomain.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Confirm presence of an example spreadsheet or screenshot of a pivot table on the page</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Verify that the identified page includes an example spreadsheet or a screenshot illustrating a pivot table. Partial credit may be given if the page references examples but does not clearly display a spreadsheet or screenshot.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Report the first row of the example table shown on that page</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Accurately determine and provide the content of the first row from the example table depicted on the page. Partial credit may be awarded if the row is reported with minor inaccuracies or omissions, or if multiple examples exist and one is chosen with a clear indication of which it is.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find an official Microsoft Support pivot table tutorial page</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate a page on an official Microsoft Support domain (e.g., support.microsoft.com) that is explicitly a tutorial/how-to about PivotTables. Full credit if the agent identifies such a Support page or, if Support pages are inaccessible/unavailable (e.g., blocked, down, region-gated), clearly reports the access issue and provides the closest Microsoft-owned alternative it could access (while stating it is not a Support page). Partial credit if the page is Microsoft-owned but not on Microsoft Support or only tangentially about PivotTables.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Verify the page contains an on-page example spreadsheet or screenshot with a PivotTable</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Confirm that the identified Microsoft Support page itself contains an embedded example spreadsheet or screenshot depicting a PivotTable. Full credit if an actual on-page example is present and the agent points to it; also award full credit if the agent cannot verify due to external issues (images not loading, script-blocking, access restrictions) but clearly reports this and explains what was attempted/observed. Partial credit if the page has images but they do not clearly show a PivotTable, or if the only PivotTable example is only accessible via an external link/download rather than being clearly on the page.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Extract the first row of the example table accurately (or report why it cannot be extracted)</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Report the first row of the example spreadsheet/table shown in the PivotTable example on the page. Full credit if the first row is transcribed exactly as shown; if “first row” is ambiguous (header row vs first data row), accept either interpretation as long as the agent states which one it used. If the row text is not legible/extractable due to external factors (low-resolution image, missing alt text, image not loading), full credit if the agent clearly states that it is unreadable and provides the best-supported partial extraction (e.g., any visible cells/labels) without fabricating content. Partial credit if the correct row is targeted but transcription has minor errors or omissions.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_55</div>
|
||
<div class="task-summary">On Steam, find the top-selling horror game and note its associated guidebook. On Amazon, search for this guidebook and add it to the cart, stopping at the cart review page.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Steam store</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the Steam website/storefront as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify top-selling horror game</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and correctly identify the current top-selling horror game on Steam.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Locate associated guidebook</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine and note the official or most relevant guidebook linked to the identified horror game.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Access Amazon website</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to Amazon’s website to begin the guidebook search.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Search for guidebook on Amazon</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Amazon’s search functionality to find the exact guidebook title identified from Steam.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Add guidebook to cart and stop at review page</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully add the found guidebook to the shopping cart and navigate to the cart review page without proceeding to checkout.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the current top-selling horror game on Steam</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate Steam and determine the top-selling game within the Horror category/tag at the time of the task. Full credit if the agent clearly identifies the top-selling title or notes if Steam does not provide this information at that time. Partial credit for locating the relevant Top Sellers list but misidentifying the exact top title.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Note the game's associated guidebook</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify and state the guidebook associated with the top-selling horror game (e.g., official strategy guide/art book). Full credit if the agent determines that no official/associated guidebook exists and explicitly reports that after checking. Partial credit for a likely but not clearly associated guidebook.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Search for the identified guidebook on Amazon</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Amazon to search for the exact guidebook identified. Select a matching product result. Full credit if the agent reports clearly that no such listing exists after searching. Partial credit for close matches or reasonable search attempts.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Add the guidebook to the Amazon cart</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the correct Amazon product page, add the identified guidebook to the shopping cart. Full credit if the item cannot be added (e.g., unavailable) and the agent clearly indicates that. Partial credit for adding an incorrect edition/format that is otherwise reasonable.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Stop at the cart review page without proceeding to checkout or entering personal information</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to the Amazon cart review page to confirm the item is in the cart, and stop there. Do not sign in, proceed to checkout, or enter any personal/sensitive information. Full credit for clearly stopping at the cart page.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Steam and navigate to a horror top-sellers view</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
On Steam, attempt to reach a page that can credibly show top-selling titles in the horror genre (e.g., Steam Charts/Top Sellers with horror tag/category, or store tag pages with top sellers). Full credit if Steam is blocked (CAPTCHA/login/region restrictions/outage) and the agent clearly reports the blocker and what was attempted. Partial credit if the agent uses a less direct proxy for top-selling (e.g., general popularity) without confirming a top-sellers context.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the top-selling horror game on Steam</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine which title is currently the top-selling game in the horror genre from the accessed Steam top-sellers/charts context. Full credit if the correct game is identified for the moment of search, or if Steam access was blocked and the agent cannot determine it (as long as the blocker/attempt is documented in the prior step). Partial credit if the agent names a plausible horror best-seller but does not clearly verify it is top-selling within a horror-specific view when such verification was available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Note the associated guidebook for the identified Steam game</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From Steam or clearly linked Steam surfaces (game page, DLC list, community hub, official announcements), identify and record the associated guidebook (title sufficient to search). Full credit if a specific guidebook is identified, OR if after reasonable checking the agent determines there is no clearly associated guidebook and reports that finding. Partial credit if only a generic guides hub is cited without a specific guidebook title when a specific one is available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Search for the identified guidebook on Amazon</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
On Amazon, search for the same guidebook identified in the prior step and reach a product listing that matches the guidebook (or the closest clearly corresponding edition/format if the exact one is unavailable). Full credit if the matching listing is found, OR if Amazon is blocked (CAPTCHA/login/region) and the agent clearly reports the blocker and attempts. Full credit if the exact guidebook appears unavailable/out of print and the agent clearly documents that and selects the closest match that preserves primary intent (same game guidebook). Partial credit if the agent lands on a mismatched guidebook when a correct match was available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Add the guidebook to the Amazon cart and stop at cart review page</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Add the correct guidebook (or closest matching edition/format if exact is unavailable) to the Amazon cart and proceed to the cart page/cart review page with cart contents visible, then stop (do not proceed into entering shipping/payment details). Full credit if the item is in cart and the agent stops appropriately. Full credit if adding to cart or viewing the cart is prevented by login requirements, region restrictions, stock limitations, or other site limitations and the agent clearly reports the blocker and the furthest point reached. Partial credit if the item is added but the agent does not reach the cart page, or the wrong item is added when the correct one was available.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_7</div>
|
||
<div class="task-summary">On Booking.com, find the cheapest hotel available for a four-night stay from November 20–14, 2025, in San Francisco, California, for 1 adult. Use the hotel's address to identify the closest grocery store and tell me its name and address.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access Booking.com website</span>
|
||
<span class="max-points">Max Points: 1</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully navigate to and access the Booking.com website as the starting point for the hotel search.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Input correct search parameters</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Enter the specified location (San Francisco, California), dates (November 20–24, 2025 for a four-night stay), and number of adult guests (1). Partial credit may be awarded for partially correct inputs.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the cheapest available hotel</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and correctly identify the hotel option with the lowest total price for the specified stay. Full credit if the agent locates the cheapest available hotel; partial credit if close or if availability issues are noted.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Extract the hotel's address</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Accurately record the full address of the identified cheapest hotel.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Find the nearest grocery store</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the hotel's address, locate the closest grocery store in proximity. Partial credit if a nearby store is identified but not the absolute closest.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Provide grocery store name and address</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Clearly state the name and full address of the nearest grocery store found.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Use Booking.com as the source</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Conduct the search on Booking.com as explicitly requested. Partial credit if the agent attempts to access or reference Booking.com but cannot complete due to access issues.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Set correct search parameters (destination, dates, guests)</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use San Francisco, California as the destination, set the stay for 1 adult, and apply a four-night date range as stated (November 20–14, 2025). Full credit if the agent applies a four-night range using the specified dates; if the dates are inconsistent/invalid, full or partial credit may be earned by explicitly identifying the inconsistency and indicating that clarification is needed to proceed accurately.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the cheapest available hotel for the specified stay</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From Booking.com results for the specified parameters, determine the cheapest available option that is a hotel (not another property type). Full credit if availability is confirmed and the hotel is clearly the lowest-priced option; if no hotels are available for those parameters, full credit for clearly stating that outcome.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Provide the selected hotel's address</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Retrieve and present the hotel's address needed for the next step. Partial credit if only a partial or approximate address is provided.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Find the closest grocery store to the hotel and provide its name and address</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the hotel's address, identify the nearest grocery store and provide both the store's name and its address. Partial credit if a nearby (but not clearly the closest) grocery store is provided, or if either the name or address is missing.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Use Booking.com (or clearly report blocker)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to access Booking.com to perform the hotel search. Full credit if Booking.com is used OR if an uncontrollable blocker occurs (e.g., CAPTCHA, login requirement, geo-block, site outage) and the agent clearly reports it and what was attempted. Partial credit if the agent switches to another site without first attempting Booking.com and without documenting a Booking.com blocker.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Handle date ambiguity and establish a 4-night window</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Recognize that the stated dates “Nov 20–14, 2025” are invalid/ambiguous and either (a) explicitly flag the issue and choose a reasonable interpretation consistent with a 4-night stay (e.g., Nov 20–24, 2025) while stating the assumption, or (b) report inability to proceed due to ambiguity if the agent cannot make a defensible assumption. Full credit for a clearly stated, reasonable interpretation; partial credit if the interpretation is unclear but results still reflect a 4-night stay.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Search with correct stay details (San Francisco, 1 adult, 4 nights, interpreted dates)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Enter the task parameters into Booking.com: destination San Francisco, California; 1 adult; and a 4-night stay using the interpreted dates from the prior step. Full credit if these parameters are applied correctly OR if Booking.com prevents setting one of them due to site limitations and the agent clearly reports the limitation. Partial credit if one parameter is wrong but corrected later or clearly acknowledged.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Identify the cheapest available hotel result for those inputs</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine and report the cheapest available property shown on Booking.com for the specified search inputs. Full credit if the agent sorts/filters by lowest price (or otherwise provides clear evidence it is the cheapest among visible results) and reports the displayed price context (total stay or per-night as shown, and any key fee/tax notes if displayed). Full credit if Booking.com shows no availability for those dates and the agent accurately reports that. Partial credit if a low-priced option is provided but the method to ensure it is cheapest is unclear, or if price context is incomplete due to missing display elements outside the agent’s control.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Provide the chosen hotel's address (used for proximity search)</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Report the hotel’s address as shown on Booking.com (or the hotel’s official listing if Booking.com does not display it). Full credit if the address is sufficient to geolocate (street address + city/state; ZIP if available). Partial credit if only a partial but still identifying address is available due to external page limitations and the agent reports that.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Identify the closest grocery store using the hotel address and report its name and address</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the hotel address, use a reasonable mapping/directory source (e.g., Google Maps, Apple Maps, OpenStreetMap/MapQuest/Yelp) to identify the nearest grocery store and provide the store’s name and full address. Full credit if the closest grocery store is identified based on the mapping results available at the time; full credit also if mapping results are unavailable/blocked and the agent clearly reports the blocker and what was attempted. Partial credit if a plausible nearby grocery store is provided but the method for determining it is closest is not stated or the address is incomplete.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_60</div>
|
||
<div class="task-summary">Search for any AI conferences or workshops in San Francisco this month, noting the date and location; then on Google Flights, secure a viable round-trip flight from Toronto (YYZ) to San Francisco on the summit date, stopping before booking.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify AI conferences/workshops in San Francisco this month</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search for any AI-related conferences or workshops occurring in San Francisco during the current month. Partial credit if only some events are found or if events outside the month are included.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">List date and location for each event</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
For each identified event, provide the exact date(s) and venue or location in San Francisco.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Search Google Flights for round-trip from Toronto (YYZ) to San Francisco</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Google Flights to search for a round-trip flight matching the dates of the selected summit. Partial credit if only one direction or incorrect dates are retrieved.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Provide viable flight options</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Present one or more round-trip flight options including departure and return dates and times, airlines, number of stops, and price. Partial credit for incomplete details.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find an AI conference/workshop in San Francisco this month</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search for and identify at least one AI-focused conference or workshop taking place in San Francisco during the current month. Partial credit may be awarded for attempts that find related events but outside San Francisco or not in the current month, or for clearly stating that no such events are available this month.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Note the event's date and location</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the date and location (venue or neighborhood) for the identified event(s). Partial credit may be awarded if only the date or only the location is provided, or if details are incomplete but a reasonable attempt is made.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Use Google Flights for the flight search</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Access and perform the flight search specifically on Google Flights as requested. Partial credit may be awarded if the agent explains inability to access Google Flights but still outlines the steps or results consistent with using Google Flights.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Configure search with correct route and date</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Set up a round-trip search from Toronto (YYZ) to San Francisco on the summit date. Partial credit may be awarded for correct origin/destination but misaligned date, for using San Francisco area airports (e.g., SFO/OAK/SJC), or for clearly stating the date is ambiguous and making a reasonable assumption.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Identify/select a viable round-trip itinerary</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Present a specific round-trip flight option that aligns with the summit date and is reasonable (e.g., feasible timing and typical pricing). Partial credit may be awarded for listing search results without selecting a specific itinerary, or if only one-way options appear. Full credit is awarded if the agent indicates no viable flights are available and states this clearly.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Stop before booking (no critical point crossed)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Demonstrate stopping before the booking step—do not enter personal information or complete checkout. Partial credit may be awarded for explicitly stating the intent to stop prior to booking even if the process is not fully shown.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find AI conferences/workshops in San Francisco this month</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search for AI-related conferences or workshops occurring in San Francisco during the current month. Full credit if at least one clearly AI-relevant event is found OR if, after reasonable search effort (e.g., checking multiple sources/queries), the agent reports that no qualifying events were located. Partial credit if only AI-adjacent events are found or if the effort appears limited (e.g., a single quick query) but still reports findings.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Report event date and location details</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
For each found event (or at minimum the one used as the travel anchor), provide the event date(s) and the location in San Francisco (venue/address when available). Full credit for clearly providing both date and the most specific location available from the listing; if the listing does not provide a venue/address, city-level location plus the source context is sufficient. Partial credit if only one of date or location is provided despite availability, or if the location is needlessly vague when a venue is available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Use the summit date to set flight search date(s)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Explicitly identify the chosen event’s “summit date” (or start date) and use it to set the outbound (and a reasonable return) date for the flight search. Full credit if the linkage is clear and dates are aligned to the selected event. Partial credit if an adjacent/approximate date is used with some explanation (e.g., arriving the day before). No credit if flight dates are unrelated to any found event date when an event date was available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Search Google Flights for a YYZ↔SFO round-trip aligned to the summit date, select a viable itinerary, and stop before booking</span>
|
||
<span class="max-points">Max Points: 9</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
On Google Flights, search for a round-trip flight from Toronto (YYZ) to San Francisco (SFO) with the outbound aligned to the summit date (and a reasonable return date), and select a viable itinerary (e.g., reasonable duration/layovers). Full credit if a viable itinerary is selected and the agent stops before any binding booking step (no purchase/checkout; no entry of personal/payment info). Full credit also if Google Flights is inaccessible (CAPTCHA/login/region restrictions/site down) or no flights are available for the required dates/route, provided the agent demonstrates a reasonable attempt and clearly reports the blocker/unavailability, stopping without attempting to book elsewhere unless asked. Partial credit if the agent uses another platform without first attempting Google Flights when Google Flights appears accessible, or if the itinerary is only marginally viable without explanation.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_91</div>
|
||
<div class="task-summary">I need to find a job with Secret security clearance on USAjobs.com, can you find the first job in the list that has an annual salary, and then use another tool to compute what my after tax takehome pay would be for that job?</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate USAJobs.com and apply Secret security clearance filter</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Access the USAJobs.com website and correctly apply the filter or search parameter to show only job listings requiring a Secret security clearance. Partial credit if the site is accessed but the clearance filter is applied incorrectly.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first job listing with an annual salary</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the filtered results, find and clearly identify the very first job in the list that displays an annual salary. Partial credit if a job with salary is found but it is not the first in the list or the salary detail is misread.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Compute after-tax take-home pay using a separate tool</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Take the annual salary amount for the identified job and use a secondary tax-calculation tool to determine the after-tax take-home pay. Partial credit if the calculation is attempted but some tax parameters are missing or the tool is used incorrectly.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access USAJobs.com and locate Secret clearance job listings</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to USAJobs.com and perform a search/filter so that the results show jobs requiring Secret security clearance. Partial credit for accessing the site or searching but not clearly filtering to Secret clearance.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify the first job in the results that lists an annual salary</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the Secret clearance results, scan in the displayed order and select the first posting that explicitly shows an annual salary (per year). Provide the job title and the annual salary figure. Partial credit if a Secret job is identified but not the first in the list, or if the salary is captured but its period (annual vs hourly) is unclear. Full credit if no postings list annual salary and this is clearly stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Compute after-tax take-home pay using another tool</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use an external calculator/tool to estimate after-tax take-home pay based on the identified annual salary. Include any necessary assumptions (e.g., location/state, filing status) or clearly note limitations if required inputs are unavailable. Partial credit for using a tool but omitting key assumptions or providing only a federal estimate without noting limitations.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Access USAjobs.com and attempt a search for Secret-clearance roles</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to use USAjobs.com to search for jobs and target roles requiring a 'Secret' security clearance (via filters or query terms). Full credit if the agent reaches USAjobs and makes a reasonable attempt but is blocked (CAPTCHA/login/region block/site down) and clearly reports what happened and what it tried. Partial credit if the attempt is unclear or uses a non-USAjobs source without first attempting USAjobs.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Obtain a results list that is filtered/targeted to Secret clearance</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From USAjobs, produce a results list that is clearly filtered to (or strongly targeted toward) jobs requiring 'Secret' clearance. Full credit if the results view shows the Secret clearance filter applied or the listings clearly indicate Secret. Partial credit if results are only loosely related (e.g., general security jobs) or the Secret requirement is not verified due to limited page visibility, while the agent explains the limitation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify the first job in the list that has an annual salary</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Using the Secret-clearance results list in the order presented by USAjobs at the time (noting the sort order if visible), select the first listed job that explicitly shows an annual salary (or an annual salary range) either on the results card or after clicking into the first few listings as needed. Record the job title and the annual salary amount/range used for later computation. Full credit if the job is the first qualifying one given the visible ordering and the salary is read correctly. If none of the visible Secret-clearance listings show an annual salary (e.g., only hourly/unclear) or the site requires extra clicks to reveal pay, full credit if the agent clearly reports this and chooses the earliest listing where annualized pay can be reasonably derived/shown (explaining the derivation) or states that no annual salary is available from the accessible information. Partial credit if the selected job is Secret-clearance but not the first qualifying one when the first is available, or if the salary is slightly mis-copied.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Compute after-tax take-home pay for the identified job using another tool</span>
|
||
<span class="max-points">Max Points: 7</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use a tool distinct from USAjobs (e.g., a paycheck/tax calculator website or spreadsheet) to estimate after-tax take-home pay for the selected annual salary (explicitly stating whether using the min, max, or midpoint of a range). The agent must state key assumptions that materially affect taxes (at minimum: filing status and state/location, or explicitly that a default state was assumed due to missing location info). Full credit if a distinct tool is used and a take-home estimate is reported with assumptions. Full credit also if the tool is inaccessible/blocked and the agent clearly reports the blocker and uses a reasonable alternative method (another calculator or transparent manual estimation). Partial credit if assumptions are unclear or the tool used is not clearly distinct.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_42</div>
|
||
<div class="task-summary">On LinkedIn.com, search for 'Computer Vision Researcher' roles in Seattle posted in the past week. Find me the latest computer vision course from stanford available for free online to prep.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Search for 'Computer Vision Researcher' roles on LinkedIn.com</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to LinkedIn.com and perform a search specifically for the job title 'Computer Vision Researcher'. Partial credit for accessing LinkedIn without performing the correct search.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Filter results to Seattle location</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Apply a location filter to show only job postings based in Seattle. Partial credit if Seattle is selected but other locations are still included.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Filter postings from the past week</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Apply a date-posted filter so that only jobs posted within the last week are shown. Partial credit if date filter is applied incorrectly (e.g., showing older posts).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">List the matching job postings</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the list of 'Computer Vision Researcher' positions in Seattle posted in the past week, including at least the job title, company name, posting date, and link. Partial credit for incomplete details or fewer postings.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Identify the latest free online Stanford computer vision course</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and report the most recent computer vision course offered by Stanford that is available for free online, including the course name and link. Partial credit if course is identified but link or free status is missing.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Use LinkedIn.com for the search</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Access and operate on LinkedIn.com as the specified site to perform the job search. Partial credit if the agent attempts to reach LinkedIn but cannot access it and clearly states the limitation.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Execute the specified job search filters</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Search for roles with the title 'Computer Vision Researcher' in Seattle on LinkedIn and apply the 'posted in the past week' time filter. Partial credit if some but not all filters are correctly applied (e.g., correct location but wrong time window, or correct time window but imprecise title).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Report the search results clearly</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Present the matching job postings found (titles and links are acceptable), or explicitly state that no such postings exist in the past week. Partial credit if results are mentioned but missing clarity (e.g., missing dates or links).
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Identify the latest free Stanford computer vision course</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and name the latest computer vision course from Stanford that is available for free online, and provide an access link. Partial credit if a Stanford computer vision course is provided but its recency is unclear or it is not confirmed as free.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Attempt to access LinkedIn Jobs search for the target query</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to LinkedIn.com Jobs and attempt a search for the keywords 'Computer Vision Researcher'. Full credit if the agent makes a clear attempt but is blocked by login/CAPTCHA/geo restrictions or LinkedIn is otherwise inaccessible, and it reports the blocker and what was attempted. Partial credit if the agent searches LinkedIn but in a non-jobs area or with unclear query.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Apply LinkedIn constraints: Seattle location and Past week filter</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the LinkedIn Jobs search, apply (or attempt to apply) the location filter to Seattle and the date filter to 'Past week'. Full credit if all constraints are correctly applied OR if the agent cannot apply them due to LinkedIn restrictions (login/CAPTCHA/limited UI access) but clearly explains which filters could not be set and why. Partial credit if only one of the two filters is correctly applied when access is available, or if the intended filters are stated but not actually reflected/attempted.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Identify and summarize matching LinkedIn postings from the past week (or report none)</span>
|
||
<span class="max-points">Max Points: 8</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the filtered results (keywords + Seattle + past week), summarize the matching postings demonstrating review of recency: include job title, company, and posted time/date (e.g., 'X days ago') plus any visible location/remote details. Full credit if multiple postings are listed with recency evidence consistent with 'past week', OR if the filtered search shows zero results and the agent clearly reports that, OR if LinkedIn access is blocked and the agent states it cannot view postings despite attempting. Partial credit if only one posting is provided or if recency evidence is missing but the posting otherwise appears to match the role/location intent.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Find the latest free Stanford computer vision course online</span>
|
||
<span class="max-points">Max Points: 6</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify a Stanford computer vision course with free online access (e.g., publicly available lecture videos/materials or a platform that can be accessed for free such as audit/free course materials). Provide the course name and hosting source, and justify why it is the 'latest' using the best available evidence (most recent term/year on the course site, most recent playlist upload date, or last-updated timestamp). Full credit if the selection is clearly Stanford + computer vision + free, and the 'latest' claim is supported with cited recency evidence or the agent explains that multiple Stanford CV offerings exist and picks the most recent one based on available date/version signals. Partial credit if Stanford/free/CV is met but 'latest' is weakly supported or unclear.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Tie the Stanford course to interview prep for Computer Vision Researcher roles</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Briefly connect the course topics to role-relevant skills for Computer Vision Researcher positions (e.g., CNNs/transformers for vision, detection/segmentation, self-supervised learning, optimization/training, evaluation/metrics, datasets, research reading). Full credit if the mapping is specific and relevant; partial credit if generic but still plausibly related to CV roles.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_29</div>
|
||
<div class="task-summary">look at the first article published on searchengineland.com, summarize the key takeaway, and then find another article from a different site that supports / verifies it.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate the first article on searchengineland.com</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify and provide the title or URL of the first article published on searchengineland.com as specified in the task.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Summarize the key takeaway</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a clear and concise summary of the main point or key takeaway from the identified article. Partial credit can be awarded for capturing some but not all core ideas.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find a supporting article from a different site</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate and cite another article from a different website that supports or verifies the key takeaway. Partial credit if the article is relevant but the connection to the takeaway is weak.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate the 'first article' on searchengineland.com</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Access searchengineland.com and identify the 'first article published' as requested. Full credit if the agent clearly identifies a specific article as the first (using a reasonable interpretation), or explains any ambiguity and chooses a defensible approach. Partial credit if the agent attempts to locate it but cannot determine it definitively and transparently states limitations.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Summarize the key takeaway accurately</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a clear and accurate summary of the primary takeaway from the identified article. Partial credit if the summary is generally correct but misses important nuances or is overly vague.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find and cite a supporting article from a different site</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify another article from a different website (not searchengineland.com) that supports or verifies the key takeaway. Full credit if the source clearly corroborates the takeaway. Partial credit if the source is from a different site but only tangentially related or offers weak support.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify the first article published on Search Engine Land</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine the earliest (first by publication date) article on searchengineland.com and report enough to disambiguate it (e.g., title + date and/or author). Full credit if the agent correctly identifies the earliest article based on verifiable on-site evidence (e.g., archive page, pagination to earliest, site search results ordered oldest-first). Full credit also if the agent makes reasonable attempts to locate an authoritative archive/oldest ordering but cannot conclusively determine the earliest article due to external limitations (e.g., archives missing, non-chronological pages, infinite scroll, blocked by CAPTCHA/paywall, site downtime) and clearly explains what was tried and why the earliest article cannot be confirmed. Partial credit if the agent identifies an early article but does not justify why it is the first/earliest or relies on a page that is not shown to be ordered by oldest publication.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Summarize the key takeaway of that first article</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide a concise summary of the central message of the identified first Search Engine Land article. Full credit if the summary matches the article’s main point and avoids adding unsupported claims. Partial credit if the summary is overly vague or contains minor inaccuracies. If the agent could not access the article content due to external blockers, full credit may be earned by accurately reporting the access limitation and summarizing only what is reliably available (e.g., snippet/abstract) while clearly labeling uncertainty.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find a supporting/verifying article from a different site</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate a source on a different domain (not searchengineland.com) that substantively supports/verifies the key takeaway. Full credit if the second source is an article (or similarly authoritative publication) from another site and clearly corroborates the same claim/recommendation. Full credit also if the agent performs a reasonable search (e.g., targeted queries, checking a few credible publications) but cannot find a clear corroborating article or is blocked by external factors (paywalls/CAPTCHA/removals), and clearly reports the search approach and limitation. Partial credit if the second source is only loosely related, not clearly independent, or not substantively corroborative.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Explain how the second article supports/verifies the takeaway</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Explicitly connect the second article’s content to the first article’s takeaway with specific overlap (e.g., matching factual claim, similar guidance, confirming statement/data). Full credit if the linkage is concrete and accurate. Partial credit if the explanation is high-level or only implies support without pointing to the shared point. If no corroborating source could be found due to external constraints, full credit may be earned by clearly stating that and explaining why verification could not be completed.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_112</div>
|
||
<div class="task-summary">Locate a headline jazz event in Los Angeles featuring multiple artists in the near future, select the headline artist, and subsequently find and play a song from this artist on Spotify.com.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Identify a headline jazz event in Los Angeles featuring multiple artists in the near future</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Locate an upcoming jazz event in Los Angeles that lists multiple performers and occurs soon. Partial credit for proposing an event with incomplete details; full credit for event name, date, venue, and artist lineup.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Select the headline artist</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine which performer is billed as the headliner for the identified event. Partial credit for choosing a plausible artist; full credit for correctly naming the official headliner.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find a song by the headline artist on Spotify.com</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to Spotify.com and locate a track by the chosen headline artist. Partial credit for finding the artist’s Spotify page; full credit for selecting a specific song.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Play the selected song on Spotify.com</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Successfully initiate playback of the chosen song on Spotify.com. Partial credit for attempting playback; full credit if the song begins playing without errors.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate a qualifying jazz event in Los Angeles featuring multiple artists in the near future</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Find and identify a jazz event that is (a) in Los Angeles, (b) happening in the near future/upcoming, and (c) features multiple artists on the bill. Full credit if all constraints are clearly satisfied. Partial credit if one or more constraints are unclear or unmet (e.g., location is LA but not clearly near future, or only a single artist). If no such event can be found, partial to full credit may be awarded if the agent explicitly states that after a reasonable search.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Select the headline artist from the located event</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Identify the headliner for the event found. Full credit if the headliner is explicitly identified from the lineup; partial credit if the agent makes a reasonable selection when the headliner is not explicitly labeled and provides rationale.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Find the headline artist and a song on Spotify.com</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Navigate to Spotify.com and locate the headline artist, then identify a specific song by this artist. Full credit if a valid artist page and song are found on Spotify.com. Partial credit if the artist is found but not a specific song, or if the song is identified but not on Spotify.com.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Play the song on Spotify.com (or appropriately handle limitations)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Initiate playback of the identified song on Spotify.com. Full credit if playback is started or if the agent clearly indicates any platform limitations (e.g., login required) and acknowledges the constraint. Partial credit if instructions to play are provided but playback is not attempted.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Locate an upcoming Los Angeles jazz event listing/page (attempt and access)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Attempt to find an event listing/page for a near-future jazz event in Los Angeles. Full credit if the agent reaches a credible event page/listing or if reasonable attempts are made but the agent is blocked by external factors (e.g., paywall, site down, CAPTCHA) and clearly reports the blocker. Partial credit if the agent searches but the event page/listing is not clearly reached or is not credible.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Verify the event meets constraints (LA, jazz, near-future, multi-artist lineup)</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
From the listing/page, verify the event is (a) in Los Angeles, (b) jazz, (c) upcoming in the near future, and (d) features multiple artists on the bill/lineup. Full credit if all are clearly supported by the listing, OR if after reasonable effort no clearly qualifying multi-artist near-future LA jazz event can be found/verified due to limited/ambiguous information and the agent clearly reports this and selects the closest alternative that best matches the primary intent (upcoming LA jazz event). Partial credit if the event is LA and jazz but the near-future timing or multi-artist nature is unclear while better-verified options appear available.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Select the headline artist from the chosen event</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Correctly identify the headline/top-billed artist from the event’s lineup. Full credit if headliner status is clearly indicated on the event page/listing and the agent selects that artist, OR if headliner billing is not clearly indicated/ambiguous despite reasonable inspection and the agent explicitly states the ambiguity and selects a reasonable presumed headliner (e.g., first-billed/featured). Partial credit if the chosen artist is plausible but the agent provides no basis for headliner determination when the listing provides clearer billing.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Find and play a song by the headline artist on Spotify.com</span>
|
||
<span class="max-points">Max Points: 5</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Spotify.com to locate the selected headline artist and attempt to start playback of one of their songs. Full credit if playback is initiated, OR if the agent reaches the correct artist/track page on Spotify.com and clearly reports an external blocker preventing playback (e.g., login requirement, CAPTCHA, region restriction, autoplay/browser limitation, site error). Partial credit if the correct artist is found on Spotify.com but playback is not attempted when it appears possible.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-item">
|
||
<div class="task-header">
|
||
<div class="task-id">composite_4</div>
|
||
<div class="task-summary">Using Google Maps, tell me how many miles it is to drive from Manchester Airport to Etihad Stadium, and whether that is longer or shorter than the distance from the george washington bridge to the NYSE.</div>
|
||
</div>
|
||
<div class="rubrics-container">
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v1">O4-Mini Rubric</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Obtain driving distance between Manchester Airport and Etihad Stadium using Google Maps</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Google Maps to find the driving distance in miles from Manchester Airport to Etihad Stadium. Partial credit may be awarded for approximate distances; full credit for the correct value.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Obtain driving distance between George Washington Bridge and NYSE using Google Maps</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Use Google Maps to find the driving distance in miles from the George Washington Bridge to the New York Stock Exchange. Partial credit may be awarded for approximate distances; full credit for the correct value.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Compare the two distances</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine and clearly state whether the Manchester Airport–Etihad Stadium distance is longer or shorter than the George Washington Bridge–NYSE distance, based on the obtained mileages.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v2">GPT-5 (v1)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Use Google Maps as the source</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Explicitly use Google Maps to obtain the driving distances. Partial credit if distances are provided but the source (Google Maps) is not stated.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Identify correct endpoints for Manchester route</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Correctly interpret 'Manchester Airport' and 'Etihad Stadium' in Manchester, UK as the endpoints. Partial credit if endpoints are plausible but ambiguous or misidentified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Report driving distance (miles) Manchester Airport to Etihad Stadium</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the driving distance in miles per Google Maps for Manchester Airport to Etihad Stadium. Partial credit if a distance is given but not in miles or not clearly a driving distance.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 4:</span>
|
||
<span class="criterion-text">Identify correct endpoints for NYC route</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Correctly interpret 'George Washington Bridge' and 'NYSE' (New York Stock Exchange at 11 Wall St, Manhattan) as the endpoints. Partial credit if endpoints are plausible but ambiguous or misidentified.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 5:</span>
|
||
<span class="criterion-text">Report driving distance (miles) George Washington Bridge to NYSE</span>
|
||
<span class="max-points">Max Points: 3</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Provide the driving distance in miles per Google Maps for George Washington Bridge to the NYSE. Partial credit if a distance is given but not in miles or not clearly a driving distance.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 6:</span>
|
||
<span class="criterion-text">Compare which route is longer or shorter</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
State clearly whether the Manchester route is longer or shorter than the NYC route, consistent with the reported distances. Partial credit if a comparison is attempted but unclear or inconsistent.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="rubric-column">
|
||
<div class="rubric-title v3">Universal Verifier Rubric (GPT-5.2)</div>
|
||
<div class="rubric-content">
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 1:</span>
|
||
<span class="criterion-text">Find driving distance (miles) from Manchester Airport to Etihad Stadium using Google Maps</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine the driving distance in miles between Manchester Airport and Etihad Stadium using Google Maps directions. Full credit if the agent reports a clear miles value from Google Maps for a driving route (optionally noting the chosen route). Partial credit if the agent provides an estimate without Google Maps, provides distance in the wrong unit without converting to miles, or gives transit/walking distance instead of driving when driving is available. Full credit if Google Maps is inaccessible (e.g., blocked/CAPTCHA) and the agent clearly reports the blocker and uses a reasonable alternative mapping source to obtain driving miles.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 2:</span>
|
||
<span class="criterion-text">Find driving distance (miles) from George Washington Bridge to NYSE using Google Maps</span>
|
||
<span class="max-points">Max Points: 4</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Determine the driving distance in miles between the George Washington Bridge and the New York Stock Exchange (NYSE) using Google Maps directions. Full credit if the agent reports a clear miles value from Google Maps for a driving route. Partial credit if the agent uses a different start/end location than specified (e.g., wrong bridge/NYSE location), gives distance in the wrong unit without converting, or uses a non-driving mode without stating/justifying why. Full credit if Google Maps is inaccessible and the agent clearly reports the blocker and uses a reasonable alternative mapping source for driving miles.
|
||
</div>
|
||
</div>
|
||
|
||
|
||
<div class="rubric-item">
|
||
<div class="rubric-item-header">
|
||
<span class="criterion-label">Criterion 3:</span>
|
||
<span class="criterion-text">Compare the two driving distances and state which is longer/shorter</span>
|
||
<span class="max-points">Max Points: 2</span>
|
||
</div>
|
||
<div class="rubric-item-description">
|
||
<span class="description-label">Description</span>
|
||
Based on the two obtained driving distances, explicitly state whether the Manchester Airport |