Traditional cloud infrastructure treats compute as fungible. You request CPU or memory, the scheduler finds capacity, and your workload runs. AI workloads break this model in three ways: GPU affinity matters, workload types have wildly different resource profiles, and agentic systems spawn unpredictable execution graphs that don't fit batch job semantics.
Corey Sanders, SVP of Product at CoreWeave, recently outlined why AI-native infrastructure requires different primitives. The conversation focused on how GPU orchestration diverges when you're serving inference requests, running multi-day training jobs, or handling agents that call tools in non-deterministic sequences.
Why Traditional Cloud Scheduling Fails for AI
Standard cloud schedulers optimize for bin packing and utilization. They assume workloads are stateless, short-lived, or at least predictable. AI workloads violate all three assumptions.
Training runs hold GPUs for days or weeks. You can't migrate them mid-job without checkpointing state, and checkpointing a 70B parameter model takes minutes. Preemption is expensive.
Inference serving needs low-latency GPU access but has bursty traffic patterns. A chatbot might sit idle for seconds, then need 8 GPUs to handle a spike. Traditional autoscaling is too slow because spinning up a GPU instance and loading a model takes 30-90 seconds.
Agentic workloads are worse. An agent might call a vision model, wait for a user response, invoke a code interpreter, then fan out to three parallel tool calls. The execution graph is a DAG with variable depth and unknown width. You can't reserve resources upfront because you don't know what the agent will do next.
AI-Native Orchestration Primitives
CoreWeave's approach exposes infrastructure primitives that traditional clouds abstract away. The key differences:
Model-aware routing. The scheduler knows which models are loaded on which GPUs. When an inference request arrives, it routes to a warm instance instead of cold-starting. This cuts P99 latency from seconds to milliseconds.
Stateful session affinity. For agentic workflows, the orchestrator can pin a session to a GPU pool. If an agent loads a 13B model for the first tool call, subsequent calls in that session hit the same warm cache. You're not paying load time on every step.
Elastic GPU pools. Instead of fixed instance sizes (8 GPUs, 16 GPUs), you allocate from a pool and scale within the job. A training run might start with 64 GPUs, scale to 128 during the compute-heavy phase, then drop to 32 for validation. The orchestrator handles topology and interconnect constraints.
Workload-specific QoS. Training jobs tolerate higher latency but need guaranteed throughput. Inference needs low P99 but can shed load. Agents need bounded tail latency on tool calls but can queue planning steps. The scheduler applies different policies per workload type.
Resource Allocation for Unpredictable Agent Graphs
The hardest problem is agents that spawn tool-calling graphs you can't predict. A customer support agent might:
- Call a retrieval model (needs 1 GPU, 200ms)
- Invoke a code interpreter (needs CPU, 2 seconds)
- Fan out to 5 parallel API calls (needs network, 500ms)
- Synthesize results with a reasoning model (needs 4 GPUs, 1 second)
You can't reserve 4 GPUs for the entire session because step 1 only needs 1. You can't release after step 1 because step 4 will wait in the cold-start queue. Traditional clouds would either over-provision (wasteful) or under-provision (slow).
AI-native infrastructure solves this with speculative resource hints and priority queues. The agent runtime signals likely next steps based on the current state. The orchestrator pre-warms capacity in a shared pool. If the agent takes a different path, the capacity gets reassigned. If it follows the hint, the GPU is already loaded.
Priority queues let you trade cost for latency. High-priority agent sessions get dedicated capacity. Low-priority sessions share a pool and tolerate queuing. The scheduler dynamically adjusts based on load.
Training vs. Inference vs. Agentic: A Comparison
| Workload Type | GPU Tenure | Latency Tolerance | Resource Predictability | Scheduling Strategy |
|---|---|---|---|---|
| Training | Hours to weeks | High (batch jobs) | Predictable, static | Reserve fixed topology, allow preemption with checkpointing |
| Inference | Milliseconds to seconds | Low (user-facing) | Bursty, spiky | Model-aware routing, warm pools, autoscale on queue depth |
| Agentic | Seconds to minutes per session | Medium (tool calls need <1s, planning can wait) | Unpredictable DAG | Session affinity, speculative warming, priority-based queuing |
Observability and Failure Modes
AI-native infrastructure needs different telemetry. Traditional metrics (CPU, memory, disk) don't expose the bottlenecks.
GPU utilization is misleading. A GPU at 80% might be memory-bound, not compute-bound. You need tensor core utilization, memory bandwidth, and PCIe throughput. CoreWeave's stack exposes these per-job.
Model load time is a hidden cost. If you're cold-starting models on every request, you're paying 10-100x the inference cost in load overhead. Track cache hit rate and time-to-first-token separately.
Agent session length matters. Long sessions hold resources. If an agent waits 30 seconds for a user response while pinning 4 GPUs, you're burning money. The orchestrator should support session pause/resume with state serialization.
Failure modes are different. Training jobs fail from OOM or hardware faults. Inference fails from cold starts or rate limits. Agents fail from tool timeouts, malformed outputs, or infinite loops. You need per-workload circuit breakers and retry policies.
Deployment Shape
AI-native infrastructure typically runs as a control plane on top of bare-metal GPU clusters. The stack looks like:
# Simplified orchestration config for agentic workload
apiVersion: coreweave.com/v1
kind: AgentSession
metadata:
name: support-agent-session
spec:
priority: high
affinityHints:
- modelId: llama-3-70b
gpuCount: 4
preload: true
- modelId: clip-vit-large
gpuCount: 1
preload: false
resourcePolicy:
maxGPUs: 8
idleTimeout: 30s
pauseOnIdle: true
toolCallPolicy:
maxConcurrency: 5
timeout: 10s
retries: 2
The control plane:
- Tracks which models are loaded on which nodes
- Maintains a warm pool based on historical access patterns
- Routes requests to nodes with cache hits
- Handles session lifecycle (create, pause, resume, terminate)
- Exposes metrics on GPU utilization, cache hit rate, and queue depth
The data plane is bare metal with NVLink or InfiniBand for multi-GPU jobs. No virtualization overhead. The orchestrator schedules at the pod level but understands GPU topology.
When to Use AI-Native Infrastructure
Use it when:
- You're running multi-GPU training jobs that need high-bandwidth interconnects
- Inference latency matters and you can't tolerate cold starts
- You're building agentic systems with unpredictable tool-calling patterns
- You need fine-grained control over GPU allocation and model caching
- Your workload mix includes training, inference, and agents on shared capacity
Avoid it when:
- You're running small models that fit on CPU or single-GPU instances
- Your inference traffic is steady and predictable (traditional autoscaling works fine)
- You don't have the operational expertise to manage GPU orchestration
- Your agents are simple linear chains with known resource needs
- Cost predictability matters more than performance (reserved instances are simpler)
Technical Verdict
AI-native infrastructure is not a better version of traditional cloud. It's a different abstraction layer optimized for workloads that traditional schedulers can't handle efficiently. The win is in workload-specific primitives: model-aware routing, stateful session affinity, and speculative resource allocation.
The trade-off is operational complexity. You're managing GPU topology, model caching, and session lifecycle instead of just requesting compute. If your workload fits the AI-native profile (multi-GPU training, low-latency inference, or agentic tool-calling graphs), the performance and cost gains are real. If you're running simple inference or batch jobs, traditional cloud is simpler and cheaper.
The key insight is that agentic workloads are a third category. They're not training (long-running, predictable) or inference (short, bursty). They're session-based, stateful, and spawn unpredictable execution graphs. Infrastructure that treats them as either training or inference will over-provision or under-perform.
