The Model Replied in 900ms. My Pipeline Still Timed Out.

It was 3:17 AM and my nightly summarization job had failed again. The log showed a connection timeout, the model provider's status page was green, and when I ran the same request from my laptop, the model answered in 900 milliseconds. So where did those thirty seconds actually go? That question turned out to be more interesting than the error message itself.

The setup sounds simple: MonkeyCode is an open-source project that offers free model access and a free server option, and I used both to run a nightly job that summarizes the day's articles and posts the result to a database. Free model access and a free server sound like a dream until your pipeline wakes up at 3 AM and discovers that "free" comes with a lifecycle you never read about. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The symptom and the obvious suspects

The cron log had one line I have seen a hundred times:

requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='api.example.com', port=443): Max retries exceeded with url: /v1/chat/completions

My first suspects were the usual lineup: the model API was down, the network was flaky, or DNS was lying to me again. I had already been burned by a broken DNS resolver on a free server once, so I checked that first. DNS was fine this time. The status page was green. The API answered in 900ms from my laptop. Every obvious suspect had an alibi.

Reproducing it from the right host

The mistake I almost made was trusting my laptop's measurement. The pipeline runs on the free server, not on my desk, so I reran the failing request from the server itself with curl's timing output:

curl -w "dns: %{time_namelookup}s connect: %{time_connect}s total: %{time_total}s\n" \
  -o /dev/null -s https://api.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"free-model","messages":[{"role":"user","content":"ping"}]}'

At 3 PM it returned in about a second. At 3 AM it hung for thirty seconds and died. Same server, same endpoint, same payload, different time of day. That pattern is the clue: the model was not the bottleneck, the server hosting my job was. The free server sleeps after a period of inactivity, and the 3 AM cron job was the first request after hours of silence. The cold start took longer than my client's timeout allowed.

Why my retry loop made it worse

Here is the part that hurt. My job did not fail once; it failed three times in a row, because I had written a naive retry loop:

for attempt in range(3):
    try:
        requests.post(url, json=payload, timeout=30)
        break
    except requests.RequestException:
        time.sleep(1)  # surely one second is enough?

Every retry landed in the same cold-start window. The server was still warming up, so each attempt consumed the same thirty seconds, and the job gave up before the server ever became ready. The retries were not a safety net; they were a second and third copy of the same failure. The assumption I never wrote down was "the server is always awake," and that unexamined assumption was the real bug. I have started keeping a small reasoning ledger for exactly this kind of decision: what did I assume, and when will that assumption break?

The fix: warm up, wait, and back off

The fix has three parts, and none of them require a bigger machine. First, add a health check that waits for the server to wake up before calling the model. Second, give the first request of the day a longer timeout. Third, replace immediate retries with exponential backoff plus jitter:

import random
import time

import requests

def wait_for_server(health_url, attempts=12, delay=5):
    for _ in range(attempts):
        try:
            if requests.get(health_url, timeout=3).status_code < 500:
                return True
        except requests.RequestException:
            pass
        time.sleep(delay)
    return False

def call_model_with_backoff(url, payload, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            return requests.post(url, json=payload, timeout=90)
        except requests.RequestException:
            if attempt == max_attempts - 1:
                raise
            sleep_time = min(2 ** attempt, 30) + random.uniform(0, 1)
            time.sleep(sleep_time)

The health check turns a blind timeout into a deliberate wait. The longer timeout accepts that the first call of the day is special. The backoff with jitter stops the retry stampede, because if the server is still warming up, retrying one second later is just organized impatience.

A small decision table for timeout debugging

Limitations and who should not use this approach

This workflow is honest about its limits. A free server's idle policy is not a contract; it can change, and my fix assumes the cold start is measured in seconds, not minutes. The free tier's token allowance is also a moving target: MonkeyCode currently advertises a 10M token allowance for its free model access, but verify the current numbers on the project page before you build a pipeline around them. And if your workload cannot tolerate occasional multi-second cold starts, or if a missed nightly job would cost real money, then a free server is the wrong tool; pay for an always-on instance and stop reading this article.

The real lesson

The model replied in 900ms, and my pipeline still timed out, because I measured the wrong host, trusted an unexamined assumption, and let a retry loop multiply a single failure into three. The next time your log screams "timeout," ask where the time actually went before you blame the API. If you want to experiment with a free model endpoint and a free server without touching a credit card, MonkeyCode's free tier is a reasonable place to start, but go in with a stopwatch, a health check, and a timeout plan.