Deploying large language models in production requires more than selecting weights from a leaderboard. Engineering teams must navigate GPU provisioning, driver compatibility, inference server tuning, and autoscaling logic before the first token reaches a client. This guide walks through the standard workflow for self-hosting LLMs on major cloud providers, then contrasts that path with managed inference alternatives for teams that prioritize velocity over infrastructure ownership.
1. Provision GPU Infrastructure
Start by selecting an instance type that matches your model's memory footprint. A 70B parameter model in FP16 requires roughly 140 GB of VRAM, which means two NVIDIA A100 80GB GPUs or a single H100 node with quantization. AWS offers P4d and P5 instances, GCP provides A2 and G2 families, and Azure has NC and ND series VMs. Reserve capacity if you run in production, because GPU shortages are common in popular regions.
2. Configure the Runtime Environment
Once the instance is live, install the NVIDIA driver, CUDA toolkit, and Docker. The NVIDIA Container Toolkit is mandatory if you plan to serve inside containers. A typical Ubuntu setup looks like this:
# Install NVIDIA drivers and Docker (simplified)
sudo apt update && sudo apt install -y nvidia-driver-535 docker.io
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo systemctl restart dockerVerify that nvidia-smi reports the correct devices and that Docker can access them with docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi.
3. Serve the Model with an Inference Engine
Most production teams use an inference engine rather than raw PyTorch. vLLM, Hugging Face TGI, and TensorRT-LLM each offer continuous batching, PagedAttention, or optimized kernels that improve throughput. For a general-purpose model like Llama 3.3 70B, vLLM provides a fast path to an OpenAI-compatible server:
docker run --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
vllm/vllm-openai:latest \
--model meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 2 \
--dtype halfDownload times for 70B weights can exceed thirty minutes depending on bandwidth. You also need to handle Hugging Face authentication for gated models, configure quantization if VRAM is tight, and validate that your batch size and max sequence length fit within available memory.
4. Expose and Secure the Endpoint
A local port is not a production endpoint. You need TLS termination, request authentication, and rate limiting. Common patterns include placing an Nginx reverse proxy in front of the container, terminating TLS with certificates from Let's Encrypt, and injecting an API key validation layer. If you run multiple replicas behind a load balancer, ensure that health checks probe the model server's readiness endpoint, not just the root path. Otherwise you will route traffic to a node that is still downloading weights or warming up.
5. Monitor, Scale, and Optimize
Self-hosted inference demands observability. Export GPU utilization, memory usage, KV cache hit rates, and request latency to Prometheus, then build Grafana dashboards around them. The real cost is often hidden in poor utilization. GPU instances accumulate hourly charges regardless of request volume, which can quickly exceed managed endpoint costs for sporadic traffic. Autoscaling GPU workloads is also non-trivial. Scaling to zero saves money but introduces cold starts that last minutes. Keeping warm instances burns budget. Finding the right trade-off requires constant tuning.
6. The Managed Alternative: Oxlo.ai
Owning the stack makes sense when you have strict data residency requirements, custom quantization pipelines, or existing Kubernetes fleets. For every other scenario, managed inference removes infrastructure debt and lets engineers focus on application logic. Oxlo.ai is a developer-first AI inference platform built around a simple idea: one flat cost per API request, regardless of prompt length. Unlike token-based providers, Oxlo.ai does not scale costs with input size, which makes it significantly cheaper for long-context and agentic workloads.
Oxlo.ai runs 45+ open-source and proprietary models, including Llama 3.3 70B, DeepSeek R
