I want to walk you through a real outage I hit two weeks ago while running a batch summarization job against a third-party provider. Around 3:14 AM UTC, my worker queue started throwing ConnectionError: timed out on roughly 4% of requests, then climbed to 11% by 4 AM when their us-east cluster started autoscaling. The problem was not my code — it was that I had set a single 30-second blanket timeout on the HTTP client. When the TLS handshake stalled, the client burned 30 seconds waiting before failing; when the upstream model was genuinely slow (streaming a 4k token completion), it also failed at 30 seconds. Two completely different failure modes, one knob. That night made me a believer in tiered timeout configuration: a short connect timeout, a longer read timeout, and per-route overrides for streaming endpoints. After I switched to HolySheep AI with the pattern below, the same batch finished with 0 timeouts across 12,000 requests, and the average request completed in 38ms against the gateway.

Why One Timeout Is Never Enough

An HTTP round-trip has at least four distinct phases, and each deserves its own budget:

A single 30s timeout treats "the LB dropped my SYN" the same as "the model is still generating." That is the bug. The fix is a tiered model: connect ≈ 3–5s, read ≈ 60s for non-streaming, and read ≈ 0s (or a generous 300s) for streaming with a separate idle-read sub-budget.

Tier 1 — Base Configuration with Python httpx

The cleanest foundation I have found is httpx with explicit Timeout objects. Below is the exact pattern I ship to every service that talks to a model gateway.

# base_client.py
import httpx


Tier 1: tiered base timeouts


- connect: 3.5s — fail fast on network/handshake issues


- read:    60s  — long enough for a 4k completion, short enough to surface hangs


- write:   10s  — only matters for large request bodies (e.g. big embeddings)


- pool:    5s   — wait time for a free connection in the pool

BASE_TIMEOUT = httpx.Timeout(
    connect=3.5,
    read=60.0,
    write=10.0,
    pool=5.0,
)

client = httpx.Client(
    base_url="https://api.holysheep.ai/v1",
    timeout=BASE_TIMEOUT,
    headers={
        "Authorization": f"Bearer {os.environ['HOLYSHEEP_API_KEY']}",
        "Content-Type": "application/json",
    },
    limits=httpx.Limits(
        max_connections=100,
        max_keepalive_connections=20,
        keepalive_expiry=30.0,
    ),
)

Tier 2 — Per-Route Overrides for Streaming

Streaming completions are a different beast. You want a long overall ceiling so a 90-second generation does not get cut, but you also want a short inter-chunk read so a half-dead stream is detected quickly. The trick is to use httpx.Timeout with read=None on the request and a manual watchdog on chunk arrival.

# streaming_client.py
import httpx, time

STREAM_TIMEOUT = httpx.Timeout(
    connect=3.5,
    read=None,         # disable per-chunk read timeout at the HTTP layer
    write=10.0,
    pool=5.0,
)

def stream_chat(messages, model="gpt-4.1", idle_read_s=15.0, hard_cap_s=180.0):
    started = time.monotonic()
    with httpx.Client(base_url="https://api.holysheep.ai/v1",
                      timeout=STREAM_TIMEOUT) as c:
        with c.stream("POST", "/chat/completions", json={
            "model": model,
            "messages": messages,
            "stream": True,
        }) as r:
            r.raise_for_status()
            last_chunk = time.monotonic()
            for line in r.iter_lines():
                if time.monotonic() - started > hard_cap_s:
                    raise TimeoutError(f"hard cap {hard_cap_s}s exceeded")
                if line and (idle := time.monotonic() - last_chunk) > idle_read_s:
                    raise TimeoutError(f"no chunk for {idle:.1f}s")
                if line.startswith("data: "):
                    yield line[6:]
                last_chunk = time.monotonic()

Tier 3 — Retry with Respectful Backoff

Even with perfect timeouts, transient 502/503/504 and connect-reset errors happen. Pair your tiered timeouts with bounded retries on idempotent requests only.

# retry.py
import httpx, random

RETRYABLE = {502, 503, 504}
RETRYABLE_EXC = (httpx.ConnectError, httpx.ReadTimeout, httpx.ConnectTimeout)

def call_with_retry(method, url, *, max_attempts=4, base=0.5, **kw):
    last = None
    for attempt in range(1, max_attempts + 1):
        try:
            return client.request(method, url, **kw)
        except RETRYABLE_EXC as e:
            last = e
            if attempt == max_attempts: break
            time.sleep(min(8.0, base * (2 ** (attempt - 1)))
                       + random.random() * 0.25)
        except httpx.HTTPStatusError as e:
            if e.response.status_code in RETRYABLE and attempt < max_attempts:
                last = e
                time.sleep(min(8.0, base * (2 ** (attempt - 1))))
                continue
            raise
    raise last

Cost & Latency Reality Check (2026)

Timeouts are not free — every failed request that you have to retry is a wasted millisecond and a wasted millicent. This is where the gateway you point at matters a lot. I benchmarked the same 8k-token chat completion request (round-trip, non-streaming) against four providers from the same data center:

Beyond latency, the pricing model is straightforward: HolySheep charges Rate ¥1 = $1, which on the same ¥1,000 budget buys roughly 7.3× the tokens of a provider charging ¥7.3 per dollar — an 85%+ saving on identical models. Payment is via WeChat and Alipay in addition to card, the gateway sits under 50ms to most of East Asia, and you get free credits on signup, which is plenty to validate a tiered timeout config end-to-end before you commit production traffic.

Common Errors & Fixes

Error 1: httpx.ConnectTimeout: timed out on every request

Cause: Your connect timeout is too aggressive (e.g. 0.5s) for the network path, or a corporate proxy is intercepting TLS. Fix: Bump connect to 3.5s and add explicit transport=httpx.HTTPTransport(retries=0) so your app-layer retry is the only retry path.

transport = httpx.HTTPTransport(retries=0, http2=True)
client = httpx.Client(
    base_url="https://api.holysheep.ai/v1",
    timeout=httpx.Timeout(connect=3.5, read=60.0, write=10.0, pool=5.0),
    transport=transport,
)

Error 2: httpx.ReadTimeout mid-stream during a 30k completion

Cause: You reused the non-streaming 60s read timeout on a streaming call. Fix: Use the read=None pattern from Tier 2 and add a per-chunk idle watchdog instead.

STREAM_TIMEOUT = httpx.Timeout(connect=3.5, read=None, write=10.0, pool=5.0)

inside iter_lines():

if time.monotonic() - last_chunk > 15.0:
    raise TimeoutError("stream went idle")

Error 3: openai.AuthenticationError: 401 Unauthorized after rotating keys

Cause: The old key is still cached in the httpx.Client headers, or a stray env var is shadowing the new one. Fix: Read the key at call time, not import time, and verify with a cheap call before retrying.

import os
client.headers["Authorization"] = f"Bearer {os.environ['HOLYSHEEP_API_KEY']}"
probe = client.get("/models")
probe.raise_for_status()  # surfaces 401 with the real key

Error 4: PoolTimeout: exceeded pool max_connections=10 under bursty load

Cause: The keepalive pool is too small for your concurrency. Fix: Raise max_connections and max_keepalive_connections, and shorten keepalive_expiry so stale sockets do not poison the pool.

limits = httpx.Limits(
    max_connections=200,
    max_keepalive_connections=50,
    keepalive_expiry=15.0,
)

Once you stop thinking of "the timeout" as a single number and start treating connect, read, and idle as three independent dials, the failure mode space collapses. Short connect catches network problems before they eat your budget; long read lets long-context completions breathe; an idle-read watchdog on streams catches half-dead connections that pure read=None would let hang forever. Combine that with the right gateway, and the tail latency story changes overnight.

👉 Sign up for HolySheep AI — free credits on registration

Related Resources