fix(transport): normalize lifecycle incident failures

[tode-rl]

Summary

• expose stable normalized details on API status, connection, and timeout errors
• preserve HTTPX failures as cause/__cause__ and prevent unsafe partial-transfer retries
• add bounded sync/async tunnel readiness helpers with injected clocks/backoff
• add a hermetic sync/async MockTransport incident reproduction

Verification

• uv run pytest tests/test_transport_error_contract.py -n 0
• uv run pytest tests/sdk/devbox/test_interfaces.py tests/sdk/async_devbox/test_interfaces.py -n 0
• uv run pytest tests/test_client.py -n 0
• uv run ruff check .
• uv run ruff format --check .
• uv run mypy src/runloop_api_client
• uv run pyright src/runloop_api_client
• uv build

Full repository type checking additionally inspects docs/conf.py; local docs dependencies (sphinx, sphinx-toolbox) are not installed in the provided environment.

[reflex-loop]

⚠️ Reflex agent status: Error

The agent hit an error on its last turn.

This PR was created by Reflex.

View the agent run →

_{This comment updates in place as the agent works.}

[tode-rl]

Validated full review decision: request_changes

Pinned base: e8c3d90cc565a39733abe449e50330cb11000170
Pinned head: 3443be0cc66b6dafd67c3c04d0d930da9701d622
Checkout and remote PR head were confirmed at the pinned head before review.

Observed scope: stable API/transport error normalization, retry-safety changes, sync/async high-level tunnel readiness polling, and hermetic incident regressions.

File partition

• In scope (fully reviewed): _base_client.py, _exceptions.py, lib/error_contract.py, lib/tunnel_readiness.py, sdk/devbox.py, sdk/async_devbox.py, tests/test_transport_error_contract.py
• Drive-by: none
• Merge-ghost: none

Deep pass (completed against pinned base...head)

• critical / blocks merge — sdk/devbox.py:866-869 and sdk/async_devbox.py:860-863 explicitly follow redirects while carrying X-Runloop-Tunnel-Authorization; HTTPX preserves that custom header across a cross-origin redirect, exposing the tunnel token to the redirect target. An injected client can also contribute a default API Authorization header despite the sync docstring's non-forwarding guarantee.
• important / blocks merge — lib/tunnel_readiness.py:70-75 re-raises the final retry-deadline error without from error.cause, so a final retried ConnectTimeout retains .cause but loses the required original exception in __cause__.
• important / blocks merge — _base_client.py:1063-1070 tries to inspect the structured error body before streamed error responses are read, while _base_client.py:1442-1455 and 2152-2165 only read after the retry decision. Body-only ambiguous transfer codes can therefore fall through to the generic 5xx retry path.
• important / blocks merge — lib/error_contract.py:95-102 declares every non-HTTPX exception safe to retry; the broad catches at _base_client.py:1412-1426 and 2122-2136 include exceptions from post-receipt hooks, so an ambiguous non-idempotent/execute request can still be replayed.

Readiness pass (independently completed against the same pinned base...head)

• critical / blocks merge — sdk/devbox.py:866-869 and async counterpart forward the tunnel credential across followed redirects, violating authenticated-readiness safety.
• important / blocks merge — lib/tunnel_readiness.py:82-85 honors X-Should-Retry: false but ignores X-Should-Retry: true for readiness/connect server failures other than tunnel_service_not_ready; tunnel_unavailable should remain the explicit terminal exception.
• important / blocks merge — lib/tunnel_readiness.py:70-75 does not preserve the final retried transport failure through __cause__, contrary to the public error contract.

Consolidated accepted findings

All five root causes above were re-read at their cited lines and accepted. Both passes completed against the same pinned SHAs with verifiable evidence; no partial coverage was used. The decision is request_changes because each accepted finding is in scope and blocks merge.

[tode-rl]

Review-and-fix handoff

Validated review decision: request_changes against base e8c3d90cc565a39733abe449e50330cb11000170 and reviewed head 3443be0cc66b6dafd67c3c04d0d930da9701d622. Both the deep and readiness passes completed; all five accepted blocking findings were fixed in 43fe1f1.

Fixed findings

• Prevented tunnel-token and injected API-authorization disclosure: probes now strip Authorization, disable client auth, and do not follow redirects (lib/tunnel_readiness.py:95-116; integration at sdk/devbox.py:866-875 and async counterpart).
• Preserved the last retried connect failure through both .cause and __cause__ when the readiness deadline expires (lib/tunnel_readiness.py:70-77).
• Honored both values of X-Should-Retry while retaining tunnel_unavailable as terminal (lib/tunnel_readiness.py:84-92).
• Read streamed structured error envelopes before retry classification and normalized failures while reading those envelopes (_base_client.py:1442-1479, _base_client.py:2173-2210).
• Stopped treating arbitrary non-HTTPX exceptions as proven-safe transport retries (lib/error_contract.py:96-100); the existing retry-count regression now uses an explicit pre-write ConnectError.

Tests added/changed

• Authenticated established-tunnel URL polling and Retry-After behavior.
• Cross-origin redirect/token-leak prevention and injected API-header stripping.
• Terminal tunnel_unavailable behavior.
• Explicit X-Should-Retry: true readiness retry.
• Deadline port/path diagnostics with transport cause preservation.
• Generic ambiguous exception and streamed body-only ambiguous-code no-retry regressions (sync and async where transport paths differ).
• Existing sync/async retry-count tests now inject httpx.ConnectError rather than an unclassified runtime exception.

Verification

uv run pytest tests/test_transport_error_contract.py tests/test_client.py -n 0
  173 passed
uv run pytest tests/api_resources/devboxes/test_executions.py -n 0 -k 'await_completed_timeout_handling or await_completed_server_error_handling'
  4 passed
uv run mypy src/runloop_api_client
  success (300 source files)
uv run pyright src/runloop_api_client
  0 errors
uv run ruff check .
  passed
uv run ruff format --check .
  459 files formatted
uv build
  sdist and wheel built
GitHub CI build / lint / full test
  passed

Deferred/out of scope: none.

[reflex-loop]

✅ Reflex agent status: Completed

The agent completed its work.

This PR was created by Reflex.

View the agent run →

_{This comment updates in place as the agent works.}

[tode-rl]

Validated follow-up review decision: approve

Pinned parent: 43fe1f1b7193baeb787a97a07da55b2cd59d7563
Pinned head: 3d59e3d4dab9ca834238f9e37439e3ad30cf9e25

Two independent follow-up passes completed against the pinned diff:

• Readiness pass: approved with no critical, important, or blocking findings. The shared sync/async predicate now admits only tunnel_service_not_ready (while retaining the existing X-Should-Retry: false veto), so generic retry headers cannot widen readiness polling to terminal structured codes.
• Deep pass: approved with no critical, important, or blocking findings. The change remains in the handwritten generator-safe library, does not alter public error fields or inheritance, and the sync/async regressions prove one request, attempts == 1, and no sleep for tunnel_backend_connect_timeout with X-Should-Retry: true and Retry-After: 0.
• Existing tests continue to cover tunnel_service_not_ready Retry-After delay and bounded deadline behavior.

Verification

uv run pytest tests/test_transport_error_contract.py -n 0
  25 passed
uv run ruff check .
  passed
uv run ruff format --check .
  459 files already formatted
uv run mypy src
  success (300 source files)
uv run pyright
  0 errors, 0 warnings
uv build
  sdist and wheel built
GitHub CI build / lint / full test
  passed

Accepted follow-up findings: none.
Deferred/out of scope: none.

[tode-rl]

Additional independent verification at 3d59e3d4dab9ca834238f9e37439e3ad30cf9e25: UV_PYTHON=3.9.18 ./scripts/test passed with 3789 passed, 23 skipped in 9.90s using the required Steady mock server. Raw pytest without scripts/test is not a valid harness result because the mock service on port 4010 is absent. This supplements the already-passing focused transport, Ruff, mypy, pyright, build, and GitHub CI checks.

[tode-rl]

@gautam-rl Final head 3d59e3d4dab9ca834238f9e37439e3ad30cf9e25 has completed the validated follow-up review with no blocking findings. Canonical UV_PYTHON=3.9.18 ./scripts/test passes (3789 passed, 23 skipped), focused acceptance passes (7/7), local Ruff/mypy/pyright/build pass, and GitHub build/lint/test are green. The sole remaining branch-protection requirement is one approving review; please review/approve when available. Auto-merge is already enabled.

[gautam-rl]

Please close this PR. The retry change points the wrong way, and the rest is unrelated scope.

What went wrong. Stainless already retries connect failures and (for most routes) timeouts / protocol errors. That is correct for connect: nothing was sent, always retry. After connect, retry is safe if the call is idempotent, not if we can prove the body was never sent. This PR inverts that: is_safe_transport_retry only allows ConnectTimeout / ConnectError on every request(), so read timeouts, pool waits, and HTTP/2 GOAWAY stop retrying even for GETs and for APIs that already have state-machine protection.

Timeout is not a special "do not retry" class. It just means we do not know if the server finished. If repeating the call cannot change the outcome, retry. If it can, do not.

Direction. Move toward idempotency, not fewer retries. This client already mints Stainless idempotency keys on non-GET; execute already has command_id; enable-tunnel is one-tunnel-per-devbox. Use those so a mid-flight timeout is safe to retry. Denylist only the few non-idempotent body calls (upload stream, execute without command_id). Do not teach the SDK that timeouts are radioactive.

The new APIError contract fields, wait_for_tunnel_ready, and clock/sleep test seams are unrelated to that incident and should not ride along.

If the Aug 16 issue still needs a client change, open a small follow-up later. Do not reuse this branch.