I Built InferWeave: A Python SDK for Deploying AI Models Across Cloud GPU Providers

Deploying an open-source AI model should be straightforward.

You choose a model, select a GPU provider, deploy it, and start making inference requests.

In practice, things get complicated quickly.

Different providers have different APIs, GPU configurations, authentication mechanisms, deployment lifecycles, and pricing models. Moving an inference workload from one provider to another often means rewriting infrastructure-specific code.

I wanted a simpler way to manage this.

So I started building InferWeave, an open-source Python SDK that provides a unified interface for deploying and managing AI workloads across cloud GPU providers.

GitHub: https://github.com/AmaLS367/InferWeave

The problem

Suppose you're building an application that uses several AI models:

A language model served through vLLM.
A text-to-speech model running Fish Speech.
An image generation model using FLUX.
A video generation workload.

You might want to deploy one workload on Modal, another on RunPod, and experiment with Lightning AI or Vast.ai.

Each provider introduces its own requirements.

You need to manage GPU selection, container environments, credentials, readiness checks, deployment state, and cleanup.

These problems become especially frustrating when working with multiple accounts or switching providers.

I wanted a single interface that could handle much of this orchestration while leaving the choice of infrastructure to the developer.

Introducing InferWeave

InferWeave is a Python SDK designed to simplify AI inference deployment across cloud GPU platforms.

Its architecture separates the model registry, runtime configuration, compute providers, inference clients, and deployment lifecycle management.

The basic idea is simple:

Your application → InferWeave → GPU provider → Model endpoint

Instead of managing every provider through a separate integration, you interact with a common deployment API.

Current integrations include:

Modal — serverless GPU deployment.
Lightning AI — container-based deployments.
RunPod and Vast.ai — through SkyPilot.
AWS, GCP, Azure, and other SkyPilot-supported providers — through the multi-cloud backend.

Provider-specific requirements and supported operations still vary.

Deploying a text-to-speech model

Here's an example using Fish Speech S2 Pro on Modal.

First, install the Modal integration:

pip install "inferweave[modal]"

After configuring your Modal account credentials and separate endpoint Proxy Token credentials, you can deploy a model and generate speech:

import asyncio
from pathlib import Path

from inferweave import InferWeave


async def main():
    weave = InferWeave()
    deployment = None

    try:
        deployment = await weave.deploy(
            model="fish-s2-pro",
            provider="modal",
            custom_args={"cleanup_on_failure": True},
        )

        audio = await deployment.synthesize(
            "Hello from InferWeave",
            format="wav",
        )

        Path("speech.wav").write_bytes(audio)

    finally:
        try:
            if deployment is not None:
                await deployment.stop()
        finally:
            await weave.close()


asyncio.run(main())

InferWeave handles the deployment configuration, runtime preparation, readiness polling, and authenticated inference transport.

The example explicitly stops the deployment to avoid leaving cloud resources running.

The first deployment can take several minutes while the runtime is prepared and model weights are loaded. GPU usage is billed by the underlying provider.

A complete setup walkthrough is available in the Modal TTS tutorial.

More than a deployment wrapper

A major goal of InferWeave is to address the operational problems that appear after the initial deployment.

1. GPU-aware model configuration

Different models have different hardware requirements.

InferWeave uses a model registry with workload metadata, runtime specifications, and GPU requirements.

Its routing layer can evaluate candidate hardware and supports automatic provider selection strategies, including cost-oriented routing.

Actual GPU availability and final billing remain provider-dependent.

2. Deployment recovery

What happens when your application restarts while a GPU deployment is still running?

InferWeave persists deployment records using SQLite.

A new SDK instance can recover an existing deployment handle:

deployment = await weave.attach(deployment_id)

You can also find a uniquely matching deployment by model and provider.

This helps applications manage running infrastructure across process restarts without automatically creating duplicate deployments.

3. Multiple accounts per provider

This is one of the areas I've spent significant effort on.

InferWeave 0.3.0 integrates with CredWeave, another open-source project, to manage multiple provider accounts.

It supports:

Round-robin and weighted account selection.
Priority-based failover.
Cooldowns for rate limits and exhausted quotas.
Deployment ownership tied to the originating account.
Credential isolation between provider accounts.

An important design rule is that an existing deployment must remain associated with the account that created it.

Recovery and cleanup should never silently switch to another account.

The multi-account implementation has extensive offline testing, with further live cloud validation still planned.

4. Lifecycle management

Leaving GPU resources running unnecessarily can become expensive.

InferWeave provides health checks, lifecycle monitoring, idle shutdown policies, and explicit deployment cleanup.

On Modal, scale-to-zero can release an idle GPU container while preserving the application for future use.

Stopping the entire deployment is handled separately.

5. Modular installation

You can install only the provider integrations you need.

pip install inferweave
pip install "inferweave[modal]"
pip install "inferweave[runpod]"
pip install "inferweave[vast]"

The SDK also includes a CLI:

inferweave models
inferweave providers
inferweave list

You can preview a deployment configuration without provisioning a GPU:

inferweave deploy fish-s2-pro --provider modal --dry-run

Current state and limitations

InferWeave is currently in beta (v0.3.0).

The built-in model registry includes audio, language, image, and video workloads.

However, the higher-level inference API is currently implemented for audio (synthesize()) and image generation (render()).

LLM and video workloads can be deployed, but inference requires interacting directly with their exposed endpoints.

Live deployment testing has been performed for selected configurations, including Modal GPU workloads. Other provider, model, and multi-account combinations still need additional real-cloud verification.

The project is actively evolving, and I want to keep its limitations visible rather than imply that every possible configuration has been tested.

An unexpected bit of recognition

Recently, the founder of XERJ, an open-source search engine for AI agents, reached out after independently reviewing part of InferWeave's container specification implementation.

That implementation was selected for planned inclusion in the XERJ Corpus Hub, a public index of reference code that coding agents can retrieve when solving related problems.

It was encouraging to see another open-source project discover the work independently.