Solid Queue 1.6.0 Fiber Workers: How Rails Turned Concurrency Primitives into Agent Job Orchestration

Solid Queue 1.6.0 ships with fiber-based workers, replacing the traditional thread-per-job model with cooperative multitasking. This is not a cosmetic change. Fibers let you run hundreds of concurrent jobs in a single OS thread, cutting memory overhead and context-switching costs. For agent workflows that spawn many lightweight tasks (API calls, LLM token streaming, webhook dispatches), this changes the economics of job orchestration.

Rails already had Active Job and Solid Queue for database-backed job queues. The new fiber mode integrates the Async gem, which provides non-blocking I/O primitives. Instead of blocking a thread while waiting for HTTP responses or database queries, a fiber yields control back to the scheduler. The scheduler picks another ready fiber, keeping the CPU busy without spawning new threads.

Why Fibers Matter for Agent Orchestration

Agent systems often need to coordinate many small tasks: fetching context from vector stores, calling LLM APIs, updating state in a database, sending notifications. Each task involves I/O waits. Traditional thread pools handle this by allocating one thread per job. If you have 100 concurrent jobs, you need 100 threads. Each thread consumes 1-2 MB of stack memory, and the OS scheduler juggles them with preemptive context switches.

Fibers flip the model. You can run 1,000 fibers on a handful of threads. Each fiber has a small stack (kilobytes, not megabytes). Context switches happen cooperatively when a fiber explicitly yields, not when the OS preempts it. This reduces memory pressure and eliminates the overhead of kernel-level thread scheduling.

The trade-off: fibers only work if your I/O operations are non-blocking. If a fiber calls a blocking system call, it stalls the entire thread. Solid Queue 1.6.0 solves this by integrating Async, which wraps network and file I/O in non-blocking primitives.

Architecture: How Fiber Workers Execute Jobs

Solid Queue's fiber mode runs on top of the Async gem. Here's the execution flow:

1. Supervisor process polls the database for ready jobs.
2. Fiber scheduler spawns a fiber for each job.
3. Job fiber runs the job's perform method. When it hits I/O (HTTP request, database query), it yields control.
4. Scheduler picks another ready fiber and resumes it.
5. I/O completion wakes the original fiber, which resumes from where it yielded.

The supervisor process itself runs in a fiber. This means polling the job queue is non-blocking. If no jobs are ready, the supervisor fiber yields, and other fibers (like health checks or metrics exporters) can run.

Configuration

You enable fiber mode in config/queue.yml:

production:
  dispatchers:
    - polling_interval: 1
      batch_size: 50
  workers:
    - queues: default
      threads: 4
      processes: 2
      mode: fiber
      max_fibers: 100

Key parameters:

• mode: fiber activates the fiber scheduler.
• max_fibers caps the number of concurrent fibers per worker process.
• threads still matters: each thread runs its own fiber scheduler. More threads mean more parallel I/O operations.

Database Connection Pooling

This is where fiber concurrency gets tricky. Active Record's connection pool expects one connection per thread. If you have 4 threads and 100 fibers, you still only have 4 database connections. If 10 fibers try to query the database simultaneously on the same thread, 6 of them will block waiting for a connection.

Solid Queue handles this by making database queries fiber-aware. When a fiber requests a connection and none are available, it yields. The scheduler moves to another fiber. When a connection is released, the waiting fiber resumes.

This works because Async patches Active Record's connection adapter to use non-blocking I/O. If you use raw SQL or a gem that bypasses Active Record, you lose this benefit. The fiber will block, stalling the thread.

Failure Modes and Backpressure

Fiber-based concurrency introduces new failure modes:

Backpressure is critical. If jobs arrive faster than fibers can process them, the queue grows unbounded. Solid Queue's max_fibers setting acts as a concurrency limiter. Once you hit the cap, new jobs wait in the database until a fiber finishes.

For agent workflows, this means you need to tune max_fibers based on your bottleneck. If your agents spend most of their time waiting for LLM API responses, you can run many fibers. If they do CPU-heavy parsing, fewer fibers per thread makes sense.

State Management in Long-Running Agent Workflows

Agents often maintain state across multiple job executions: conversation history, tool call results, partial outputs. In a thread-per-job model, you can store state in thread-local variables. In fiber mode, thread-locals are shared across all fibers on that thread. You need fiber-local storage.

Ruby provides Fiber[] for this:

class AgentJob < ApplicationJob
  def perform(agent_id, task)
    Fiber[:agent_context] = load_context(agent_id)

    result = execute_task(task)

    save_context(agent_id, Fiber[:agent_context])
    result
  ensure
    Fiber[:agent_context] = nil
  end
end

Fiber-local variables are isolated per fiber. When the scheduler switches fibers, each fiber sees its own Fiber[:agent_context].

The risk: if you forget to clear fiber-locals, they persist until the fiber is garbage-collected. In a long-running worker process with a fiber pool, this leaks memory.

Observability: Tracing Fiber Execution

Traditional thread-based observability tools break in fiber mode. Thread IDs no longer map one-to-one with job executions. You need fiber-aware instrumentation.

Solid Queue emits Active Support notifications for job lifecycle events:

• enqueue.active_job
• perform_start.active_job
• perform.active_job

These events include job metadata but not fiber IDs. To trace fiber execution, you need to inject fiber context into your logs:

class AgentJob < ApplicationJob
  around_perform do |job, block|
    fiber_id = Fiber.current.object_id

    Rails.logger.tagged("fiber:#{fiber_id}") do
      block.call
    end
  end
end

This tags all log lines from a fiber with its object ID. You can correlate logs across yields and resumes.

For distributed tracing (OpenTelemetry, Datadog), you need to propagate trace context through fiber yields. The Async gem provides hooks for this, but you must configure them explicitly.

When Fiber Workers Make Sense

Fiber mode shines when your jobs are I/O-bound and you need high concurrency. Typical agent workloads that benefit:

• LLM API orchestration: Hundreds of concurrent streaming requests, each waiting for tokens.
• Webhook fan-out: Dispatching notifications to many endpoints in parallel.
• Multi-source data fetching: Querying vector stores, knowledge graphs, and APIs simultaneously.

Fiber mode is a poor fit when:

• Jobs are CPU-bound: Parsing large documents, running embeddings locally. Fibers don't parallelize CPU work.
• Dependencies use blocking I/O: Legacy gems that call IO.read or Net::HTTP without Async wrappers.
• You need thread isolation: Jobs that rely on thread-locals or thread-specific resources.

Deployment Shape

Solid Queue runs as a standalone process, separate from your web server. In production, you typically deploy:

• Web dynos: Handle HTTP requests, enqueue jobs.
• Worker dynos: Run Solid Queue with fiber mode enabled.
• Database: Stores job queue, shared by web and worker dynos.

For agent systems, you might split workers by job type:

• Fast lane: High max_fibers, short timeout, for quick API calls.
• Slow lane: Fewer fibers, longer timeout, for complex agent reasoning.

This prevents slow jobs from starving fast ones.

Technical Verdict

Use Solid Queue with fiber workers when you need to orchestrate hundreds of lightweight, I/O-bound agent tasks on a single Rails instance. The memory savings and reduced context-switching overhead make it viable to run high-concurrency workloads without scaling horizontally.

Avoid fiber mode if your jobs are CPU-intensive, your dependencies use blocking I/O, or you need strict thread isolation. The cooperative multitasking model requires careful auditing of your job code and its dependencies. One blocking call can stall an entire thread's worth of fibers.

For agent workflows that mix LLM API calls, database queries, and webhook dispatches, fiber workers are a practical upgrade. You get better resource utilization without rewriting your job logic. Just watch your connection pool size and tune max_fibers based on your actual I/O wait times.

Source Links

• Solid Queue 1.6.0 Release
• Hacker News Discussion