Files
Microsoft-fara/webeval/tests
corbyandClaude Opus 4.8 19224e97da graceful_client: guarantee create() terminates; fix infinite retry hang
A live verifier test wedged for ~19h: a judge endpoint returning HTTP 401
(mapped by the openai SDK to AuthenticationError) hit the AuthenticationError
branch, which neither blocklisted the endpoint nor consumed the `tries`
budget — so next_client() kept handing back the same failing endpoint and
the loop spun forever, opening connections the whole time.

- AuthenticationError and the check-access-response-enc branch now decrement
  `tries` (and AuthenticationError backs off 1s) so a persistent auth/access
  failure on a small pool can't loop forever.
- Add a hard `max_total_attempts` cap (default max_retries + 2*n_endpoints)
  as a backstop: create() always terminates regardless of which branch fires,
  including blocklisting paths that intentionally don't spend the budget.
- Exhaustion error now reports total_attempts + both caps.
- Tests: 3 regression cases (persistent-auth termination across 1 and 3
  endpoints, all-endpoints-blocklisted termination), each guarded by a 30s
  asyncio.wait_for so any future regression fails loudly instead of hanging.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 12:52:36 -07:00
..