Thinking Like a Platform Engineer: Celery, Docker, CI/CD, and Getting SokoFlow Into the Cloud

SokoFlow Build Log — Month 2 of 4

Welcome back to another SokoFlow build log. If you're new here — I'm an IT student running a structured, project-based learning plan to grow into a production-grade backend engineer. Last semester I built SimPesa, a local-first STK Push simulator for testing M-Pesa Daraja payment workflows without touching a live sandbox. This semester, the theme is from controlled environments to the messy real world.

SokoFlow is my flagship project for this semester: a conversational ERP for small Kenyan shopkeepers that lets them track inventory and record sales entirely through chat — no app to download, no onboarding session, just natural language over WhatsApp. I covered the architecture and core business logic in the Month 1 build log.

Month 2 wasn't about adding features. It was about making everything built in Month 1 deployable, scalable, and production-ready — so that every subsequent feature ships into an environment that actually mirrors the real world. That meant Celery, Docker, CI/CD, and a live staging deployment. Let's get into it.

Where We Left Off

Month 1 ended with a solid core: CRUD operations for the key models, business rules enforced through a strict TDD cycle, and a combined test suite of 40+ tests at 92% coverage. The foundation was sound — but it only ran on my machine.

Month 2's goals were concrete and measurable:

• Celery processes a test job, with worker logs confirming execution.
• A green badge on the main branch; pull requests blocked if coverage drops below 85%.
• A production Dockerfile with build logs showing cache hits on unchanged layers.
• A CD pipeline that deploys to staging on merge to main, with both the Swagger UI and /health endpoint publicly accessible.

Week 5 — Celery Integration: Broker Config, Task Definition, Worker Startup

Before writing a single line of code, I had to build the right mental model. This week revolved entirely around one idea: background jobs.

In Month 1, I built CRUD for sales. Now imagine a shopkeeper sends the system a message at the end of the day: "Generate a daily sales report."

The naive approach is to handle that inside the normal HTTP request-response cycle:

The problems are immediate:

• The API is blocked for the entire duration.
• The request is likely to time out.
• The server can't handle any other users while it waits.

The better pattern is to offload the heavy work to a background task:

The user gets an immediate response. The hard work happens asynchronously. The API stays free to serve other requests.

Where Each Component Fits

Celery is not the broker — it's the framework that ties everything together. It gives you a client (inside the API) to publish jobs, and a worker process to consume and execute them. The broker — Redis in SokoFlow's case — is the transport layer that carries messages between the two.

The Mental Model That Cleared My Confusion

Coming from SimPesa, I had worked extensively with BullMQ. The name similarity between BullMQ and RabbitMQ made me mentally group them as peers — which led to real confusion when setting Celery up.

The hierarchy that finally cleared it up:

Application Layer      → Your business logic (generate_report)
Task Framework Layer   → BullMQ (Node.js) | Celery (Python)      ← peers
Messaging/Storage      → RabbitMQ | Redis | Amazon SQS           ← peers
Infrastructure         → TCP | Disk | Memory | Network

BullMQ and Celery sit at the same layer. RabbitMQ and Redis sit at the same layer. They are not interchangeable across layers.

Biggest Takeaway

Offloading slow operations to background workers keeps the API fast and improves scalability — but it's not free. It introduces real complexity in error handling, state management, and debugging, because execution now happens entirely outside the main request flow. Design for that from the start.

Week 6 — GitHub Actions: Setting Up the CI Pipeline

With Celery integrated, Week 6 was about automation. Specifically, making sure the test suite runs on every push and that a failing test physically blocks deployment.

Why CI Exists: The Problem It Solved

To understand why CI pipelines matter, it helps to understand what software development looked like before them.

Imagine a team of ten developers, each working on an isolated feature branch for several weeks. When everyone finally tries to merge their work at once — different assumptions, different dependencies, overlapping changes — the result is what engineers call Integration Hell: a codebase that can barely compile, let alone pass tests.

The breakthrough insight was deceptively simple: merge frequently. If you merge three months of work at once, you inherit a thousand conflicts. Finding the root cause is nearly impossible. If you merge three hours of work, you get one or two conflicts. Root cause analysis takes minutes.

But merging daily created a new bottleneck — human fatigue. Every merge meant someone had to manually compile the code, install dependencies, and run thousands of tests. Steps got skipped. Mistakes got missed.

The solution was to automate the checklist entirely: CI pipelines. Every push triggers an isolated server to compile the code, run the tests, and return a clear pass or fail signal. Jenkins, GitHub Actions, GitLab CI — they all do this, just with different configuration syntax.

Challenges

Database dependencies in CI. Locally, the test suite runs against a Postgres and Redis instance I already have configured. The CI runner starts with a completely clean machine. How does it get those?

The modern answer is service containers. When the CI job starts, the runner spins up lightweight Docker containers for Postgres and Redis alongside the main test container, connects them on a private local network, and exposes them via hostnames like localhost or postgres. The test code connects to them exactly the same way it does locally.

Credentials and security. Having database credentials hardcoded in a YAML file sounds alarming, but test database credentials carry essentially zero risk. The containers live on a private, ephemeral network that's unreachable from the public internet, and the database is permanently destroyed the moment the tests finish.

Sensitive credentials — production deployment keys, third-party API keys — are a different matter entirely. Those go into the platform's encrypted secrets store (GitHub's "Secrets" settings panel), and the pipeline injects them into the runner's memory as environment variables at runtime.

The Mental Model: GitHub Actions Runner = A Rented Computer

The most useful reframe for understanding GitHub Actions:

The only real difference: your laptop persists its state. The runner is ephemeral — it starts completely clean, does exactly what you scripted, and is destroyed the moment it finishes.

Biggest Takeaway

A GitHub Actions runner is not abstract magic. It is a regular computer in a data center, running the same commands you would run manually in a terminal. Once that clicked, everything about configuring CI became far more intuitive.

Week 7 — Docker Image Optimization

With CI in place, Week 7 was about containerization. My targets: build time under 90 seconds and a final image size under 200MB. Given the number of services in SokoFlow's stack, I was skeptical.

The Problem With Unoptimized Images

The typical first Dockerfile — pull a full OS, install every tool, copy all the code, run it — produces images that are 1–2 GB. That creates real operational problems:

1. High infrastructure costs. A team deploying 20 times a day and pushing a 2 GB image each time wastes enormous bandwidth and storage.
2. Slow deployments. Auto-scaling events that require pulling a 2 GB image are measurably slower than pulling a 100 MB one.
3. Expanded attack surface. Bundling compilers, package managers, and text editors into a production image gives an attacker more tools to work with if they ever get in.

Solution 1: Multi-Stage Builds

A multi-stage build is like a professional kitchen. You use the full counter space — heavy tools, cutting boards, prep mess — to build the dish. But when it's time to serve, only the final plate goes out. The kitchen stays in the kitchen.

In Dockerfile terms:

1. Build stage: Start with a heavy base image that includes compilers and package managers. Install all dependencies, compile what needs compiling.
2. Production stage: Start fresh with a minimal base image (Alpine Linux is around 5 MB). Copy only the compiled application artifacts from the build stage. Discard everything else.

The result: a final image that might be 50–80 MB instead of 1 GB.

Solution 2: Respecting the Layer Cache

Every instruction in a Dockerfile creates a new immutable layer. Docker caches these layers and reuses them if their inputs haven't changed. The key implication: layer order determines cache efficiency.

# Layer 1 — Base image (rarely changes)
FROM python:3.12-alpine

# Layer 2 — Dependencies (changes when requirements.txt changes)
COPY requirements.txt .
RUN pip install -r requirements.txt

# Layer 3 — Application code (changes constantly)
COPY . .

If you only change application code, Docker rebuilds only Layer 3. Layers 1 and 2 are pulled straight from cache. Copy your code before your dependencies and you forfeit that optimization entirely — every code change forces a full dependency reinstall.

Challenges

Chaining RUN commands. I noticed the recommended pattern for apt-get installs chains everything into a single command:

# Three separate layers — the cleanup layer doesn't actually remove the cached package index
RUN apt-get update
RUN apt-get install -y build-essential
RUN rm -rf /var/lib/apt/lists/*

# One layer — download, install, and cleanup happen before Docker takes the snapshot
RUN apt-get update && \
    apt-get install -y --no-install-recommends build-essential libpq-dev && \
    rm -rf /var/lib/apt/lists/*

The subtlety: deleting files in a later layer doesn't remove them from earlier layers — it just hides them. The data is still baked into the image history. Chaining with && ensures everything happens inside a single container state before Docker freezes it into a layer.

Why does a Python app need C build tools? Adding build-essential and libpq-dev to a Python Dockerfile initially confused me. I was writing Python — why did I need a C compiler?

The answer is in how psycopg2 works. Python is a high-level, dynamically typed language — fast to write, but comparatively slow to execute. PostgreSQL is written in C and expects high-speed communication. psycopg2 bridges that gap by wrapping a C extension that handles network sockets, memory management, and binary data streams directly. When Python calls psycopg2.connect(), it hands the heavy lifting to that underlying C layer.

• build-essential provides gcc, the C compiler needed to build that extension.
• libpq-dev provides the PostgreSQL header files — the "dictionary" the C compiler needs to understand how to communicate with Postgres.

Python Wheels. Historically, pip install psycopg2 downloaded raw C source and compiled it locally — slow builds and cryptic errors if the build tools weren't present. Python's solution is Wheels (.whl files): pre-compiled binary packages built by library maintainers for common operating systems and uploaded to PyPI. When a matching Wheel exists, pip install just downloads and unpacks it. No C compilation required.

Biggest Takeaway

Docker is not magic. It is mostly Linux. The Dockerfile is a sequence of Linux commands executed inside a container environment:

RUN apt-get update

is literally running apt-get update inside the image. Once I stopped thinking of Docker as a separate abstraction and started reading it as a shell script with layers, the mental overhead dropped significantly.

Week 8 — Deployment: Shipping to Staging

Week 8 was the capstone. The goal: a CD pipeline that deploys to staging automatically on every successful merge to main, with the Swagger UI and /health endpoint publicly accessible.

Connecting CI to CD

The interesting design question here wasn't the deployment itself — it was the sequencing. You only want to deploy when CI passes. A failing test suite means questionable code, and questionable code should never deploy automatically.

The simplest pattern is a single workflow file using the needs keyword to create a strict job dependency — Job B won't start until Job A finishes with a green checkmark.

I opted for two separate files (ci.yml and deploy.yml) because separate files communicate intent more clearly and are easier to manage independently. The tradeoff is that the needs keyword doesn't work across files — they don't share context. The solution is workflow_run, which lets the deployment workflow listen for the CI workflow to complete:

on:
  workflow_run:
    workflows: ["CI"]
    types: [completed]

jobs:
  deploy:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}

This means the deployment job only wakes up when the CI workflow finishes, and only proceeds if it finished successfully.

Challenges

Database driver mismatch. Railway provided a standard PostgreSQL connection URL, but SokoFlow's SQLAlchemy engine is configured for async operation using asyncpg. A plain postgresql:// URL defaults to the synchronous psycopg2 driver, which fails immediately in an async context. The fix is to normalize the driver on startup:

if url.drivername in ("postgresql", "postgres", "postgresql+psycopg2"):
    url = url.set(drivername="postgresql+asyncpg")

The Full Deployment Journey

Here's how a single git push to main travels from a local machine to a live staging environment:

1. CI pipeline — GitHub Actions spins up an isolated runner, installs dependencies, and runs the full test suite. If anything fails, the pipeline stops here.
2. CD pipeline — On CI success, the deployment workflow triggers and connects to Railway.
3. Build phase — Railway reads the Dockerfile and builds the production image.
4. Pre-deploy phase — Railway runs alembic upgrade head in a temporary container, applying any pending database migrations before the new application code goes live.
5. Live deploy — Railway starts the new FastAPI containers and performs a health check. Once they respond successfully, traffic is switched from the old containers to the new ones with minimal downtime.

Biggest Takeaway

The cloud is not magic either. Before Week 8, SokoFlow looked like this:

FastAPI → localhost → Postgres, Redis

After Week 8:

FastAPI → Environment Variables → Railway PostgreSQL, Railway Redis

The application code is identical. It only knows connection strings. Environment variables replace .env in production — same variable names, different values, and the application has no idea where they came from.

And staging is not "production lite." It is a production-identical environment used to validate deployment mechanics, infrastructure configuration, and integration behavior before anything touches real users.

Month 1 → Month 2: What Changed

Looking Ahead to Month 3: Conversation Engine and Webhook Simulation

Month 3 is the feature I've been looking forward to most since this project started — and probably the hardest one.

A WhatsApp message looks like this:

"sold 3 milks"
"Add uji 24 pcs @ 85"
"How much stock for soda?"
"Today's report"

But the system needs structured actions like this:

{
  "intent": "RECORD_SALE",
  "product": "Milk 500ml",
  "quantity": 3
}

The core challenge: how do you turn unpredictable human language into deterministic system commands — without the whole thing collapsing into spaghetti at scale?

SokoFlow's answer is Finite State Machines (FSMs) + controlled intent parsing + conversational context. The easy path would be to drop an LLM into the middle of this and let it handle everything. I didn't take that path — and Month 3's build log explains exactly why.

Stay tuned.