Three thousand lines in, the process died. Not with a Python traceback — with a silent OOM kill. The input file was untouched. The 3,000 records already sent to the model were unmarked. Re-running meant paying for them twice.
Cron does not solve this. A for loop does not solve this. A queue does.
This article walks through a small log-normalization job — 10,000 unstructured error lines turned into structured JSON — using SQLite as the queue, Python as the worker, and MonkeyCode's free model access plus free server option as the infrastructure.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The Job
A service writes raw error logs to a file. Each line is unstructured:
2026-08-25 03:12:44 ERROR conn=42 op=checkout msg="timeout waiting for lock" ctx=order:9912
The goal: turn each line into structured JSON — timestamp, severity, module, error code, and a one-line summary — then store it in a table for querying.
10,000 lines. A few hundred tokens each. A few million tokens total. Well within a free allowance.
The constraint: the worker runs on a free server. It can be killed at any time. It can be restarted. It must not re-process what already succeeded.
Why Not Just a Script?
A script is a queue that forgets. If it dies at line 3,000, it has no memory of line 3,000. You can add a "processed" marker file, but then you are building a queue anyway — badly.
SQLite gives you the queue for free. No Redis, no RabbitMQ, no extra service to run and monitor. One file, ACID transactions, and a UNIQUE constraint that does the deduplication for you.
The Schema
Three fields matter: an idempotency key, a status, and an attempt counter.
CREATE TABLE IF NOT EXISTS jobs (
id INTEGER PRIMARY KEY,
idempotency_key TEXT UNIQUE,
payload TEXT NOT NULL,
status TEXT DEFAULT 'pending',
attempts INTEGER DEFAULT 0,
next_retry_at REAL DEFAULT 0,
result TEXT,
created_at REAL DEFAULT (unixepoch())
);
CREATE INDEX IF NOT EXISTS idx_pending
ON jobs(status, next_retry_at);
The idempotency key is a hash of the payload. Inserting the same line twice becomes a no-op:
def enqueue(conn, payload: str) -> None:
key = hashlib.sha256(payload.encode()).hexdigest()
conn.execute(
"INSERT OR IGNORE INTO jobs (idempotency_key, payload) VALUES (?, ?)",
(key, payload),
)
conn.commit()
If the process crashes after 3,000 lines and you re-run the enqueue step, INSERT OR IGNORE skips what is already there. The queue is the memory.
Claiming Work Atomically
The worker needs a batch of jobs that are pending, and it needs to mark them as running in the same operation. Otherwise two workers can claim the same row.
SQLite's UPDATE ... RETURNING does this in one statement:
def claim_batch(conn, limit: int = 10, now: float | None = None) -> list[tuple[int, str]]:
now = now or time.time()
rows = conn.execute(
"""
UPDATE jobs
SET status = 'running', attempts = attempts + 1
WHERE id IN (
SELECT id FROM jobs
WHERE status = 'pending' AND next_retry_at <= ?
ORDER BY id
LIMIT ?
)
RETURNING id, payload
""",
(now, limit),
).fetchall()
conn.commit()
return rows
The next_retry_at check is what makes retries possible. A failed job is not pending until its backoff window has passed.
The Worker Loop
The worker is deliberately boring. Claim a batch, call the model, store the result, repeat.
def worker_loop(conn, batch_size: int = 10) -> None:
while True:
batch = claim_batch(conn, limit=batch_size)
if not batch:
time.sleep(5)
continue
for job_id, payload in batch:
try:
result = process_log_line(payload)
complete(conn, job_id, result)
except Exception as exc:
retry_or_fail(conn, job_id, error=str(exc))
The model call is isolated in one function. In this project it used MonkeyCode's free model access; the function signature is the only thing the rest of the code depends on.
def process_log_line(line: str) -> dict:
# Returns {"timestamp": ..., "severity": ..., "module": ...}
# The provider SDK call lives here.
...
Retry With Backoff, Then Fail Loud
A free model endpoint can return 429, 500, or just hang. The queue needs a retry budget.
def retry_or_fail(conn, job_id: int, error: str, max_attempts: int = 5) -> None:
row = conn.execute(
"SELECT attempts FROM jobs WHERE id = ?", (job_id,)
).fetchone()
if row["attempts"] >= max_attempts:
conn.execute(
"UPDATE jobs SET status = 'failed', result = ? WHERE id = ?",
(error, job_id),
)
else:
backoff = (2 ** row["attempts"]) + random.uniform(0, 1)
conn.execute(
"""
UPDATE jobs
SET status = 'pending', next_retry_at = ?
WHERE id = ?
""",
(time.time() + backoff, job_id),
)
conn.commit()
Exponential backoff with jitter. After five attempts, the job is marked failed and the error is stored. You can inspect it later instead of guessing.
Results
A test run with 10,000 lines finished in about 40 minutes on the free server. The numbers that mattered:
- 0 duplicate model calls. The idempotency key held across three worker restarts.
- 312 retries. Mostly rate-limit responses, handled silently by backoff.
-
7 permanent failures. All malformed lines that no retry would fix, visible in the
failedstatus.
These are numbers from a single test run, not a benchmark. Your token usage and failure rates will differ.
The free allowance covered the full run with room to spare. A few million tokens for 10,000 structured records is a reasonable trade — if the queue prevents you from paying for the same record twice.
Querying the results is plain SQL:
SELECT status, count(*) FROM jobs GROUP BY status;
SELECT json_extract(result, '$.severity') AS severity, count(*)
FROM jobs WHERE status = 'done' GROUP BY severity;
Limitations
SQLite is not Postgres. Concurrent writers will hit database is locked; the fix is WAL mode and a single writer process.
conn.execute("PRAGMA journal_mode=WAL;")
conn.execute("PRAGMA busy_timeout = 5000;")
The free server has memory limits. A batch size of 10 keeps the working set small. Do not load 10,000 lines into memory and expect the OOM killer to be kind.
Free tiers change. The token allowance and server specs in this article are what the project used; check the current limits before you build on top of them.
Who should skip this? Anyone processing more than a few hundred thousand records, anyone with strict latency requirements, and anyone storing sensitive data on a free server. This is a small-job queue, and it is honest about that.
What the Queue Taught Me
The first lesson: the queue is the product. The model call is the easy part. The hard part is knowing what has been done, what has failed, and what is worth retrying.
The second lesson: idempotency keys are cheaper than apologies. One hash per line saved thousands of duplicate model calls.
The third lesson: a free server is enough when the job is sized to it. 10,000 lines, a few million tokens, one SQLite file. No Kubernetes, no bill, no drama.
If you have a batch job that keeps dying halfway, start with a queue, not a bigger server. SQLite will hold up longer than you expect.
