Fractional GPU on ECS: Architecture Decision Record for AI Inference

In August 2026, Amazon ECS gained support for fractional GPU scheduling on EC2 G6f instances — the first concrete signal that AWS is treating GPU as a first-class resource in the ECS scheduling plane, not just a passthrough device. For architects of AI inference platforms in financial environments, where cost-per-inference and workload isolation are real constraints, this change demands an explicit architecture decision: when to adopt, how to configure, and what operational consequences nobody documents at launch time.

Context and Forces: Why This Matters Now

Before this feature, ECS treated GPU as a binary resource: either you allocated an entire GPU to a container, or you allocated none. In practice, this meant a credit risk classification model with 1.2 billion parameters — which fits comfortably in 4 GB of VRAM — consumed an entire 24 GB L4 GPU, wasting 83% of accelerator memory capacity. For inference platforms serving dozens of different models with asymmetric traffic patterns, that waste translated directly into unnecessary instance cost and cold-start latency because auto-scaling had to provision full instances for traffic spikes on small models.

G6f instances were announced in GA in July 2025 with native GPU partitioning support via NVIDIA MIG-like partitioning on L4 Tensor Core GPUs. The integration with the ECS scheduling plane, however, only arrived in August 2026 — a 13-month gap that forced teams to use workarounds like custom device plugins on EKS or manual GPU allocation via environment variables. Now, with GPU=0.125, GPU=0.25, and GPU=0.5 as valid values in ECS task definitions, the scheduler achieves real bin-packing: up to 8 tasks on a single G6f instance, each with 3 GB of isolated VRAM.

The financial context adds layers of complexity. Regulators like the Brazilian Central Bank and the SEC require traceability of ML model decisions used in credit, fraud detection, and derivatives pricing. This means multiple models must coexist with auditable memory isolation — exactly what GPU partitioning offers, provided the configuration is done correctly and monitored continuously.

Forces in Tension: What Makes This Decision Non-Trivial

The decision to adopt fractional GPU on ECS is not simply 'enable and save'. There are genuinely competing architectural forces that must be resolved before any production change.

Density vs. Blast Radius Isolation: When you place 8 tasks on a single G6f instance, a GPU driver failure or a hung CUDA kernel can affect all 8 partitions simultaneously. ECS Managed Instances includes automatic health monitoring that detects GPU hardware failures and replaces unhealthy instances, but the detection and replacement time — typically 2-5 minutes — is unacceptable for real-time fraud detection workloads with P99 latency SLOs below 200ms.

Cost per Inference vs. Reserved Capacity Cost: A G6f instance with 8 one-eighth GPU partitions can serve 8 small models simultaneously, but if traffic is seasonal and only 2-3 partitions are active most of the time, the idle instance cost outweighs the bin-packing benefit. The correct calculation requires real GPU utilization data per model, not estimates.

VRAM Memory Management vs. Runtime Overhead: With 3 GB of VRAM per 1/8 partition, models using aggressive INT8 or FP16 quantization fit easily. But frameworks like TensorRT and PyTorch Serve have runtime overheads that consume 200-400 MB of VRAM before even loading model weights. On 3 GB partitions, that overhead represents 7-13% of total capacity — a factor that must enter the sizing calculation.

ECS Managed Instances vs. ECS on EC2 Self-Managed: The choice between the two operating modes has direct implications for operational control, AMI customization, and management cost. For financial environments with OS hardening requirements and CIS Benchmark compliance, the ECS Managed Instances managed AMI may not satisfy requirements without additional configuration.

Options Considered for GPU Inference Platform

ECS + Fractional GPU (G6f, GPU=0.125/0.25/0.5)

Pros

• Native bin-packing in ECS scheduler — up to 8 tasks per G6f instance
• ECS Managed Instances with automatic GPU health monitoring and metrics via CloudWatch Container Insights
• No Kubernetes control plane overhead — lower operational complexity for teams without EKS expertise
• Native integration with IAM task roles, Secrets Manager, and VPC networking without additional configuration

Cons

• Only 3 fraction values supported (0.125, 0.25, 0.5) — no intermediate granularity
• Per-instance blast radius: hardware failure affects all partitions on the same G6f
• Available only in regions where G6f is available — regional coverage still limited

Verdict: Recommended for small models with VRAM ≤ 10 GB and traffic patterns that justify bin-packing

EKS + NVIDIA Device Plugin + MIG (Fractional GPU via Kubernetes)

Pros

• Full control over MIG partitioning — granularity of 1g.3gb, 2g.10gb, 4g.20gb on A100/H100
• Support for GPU Time-Slicing and MPS for workloads that don't support native MIG
• Mature ecosystem: DCGM Exporter, Prometheus, Grafana for GPU observability

Cons

• Significant operational overhead: MIG configuration, DaemonSets, node labeling, taint/toleration management
• EKS control plane cost + upgrade complexity with GPU node pools
• Time-Slicing does not offer real memory isolation — models can interfere with each other

Verdict: Preferable when the team already operates EKS and needs MIG granularity beyond 1/8, or A100/H100 support

Amazon Bedrock Inference Endpoints (Serverless/Provisioned)

Pros

• Zero GPU infrastructure management — fully serverless for supported models
• Native tenant isolation, SOC2/ISO27001 compliance managed by AWS

Cons

• No support for custom or fine-tuned models outside the Bedrock catalog
• Significantly higher cost per token than self-hosted inference at high scale
• Serverless mode cold-start latency incompatible with real-time fraud detection SLOs

Verdict: Suitable for experimentation and low-volume models; not for proprietary inference at financial scale

EC2 G6f Self-Managed (no ECS/EKS, manual allocation)

Pros

• Maximum control over driver configuration, CUDA toolkit version, and kernel optimizations

Cons

• No automatic scheduling, bin-packing, or health monitoring — all management is manual
• High operational cost; does not scale to dozens of models in production

Verdict: Only for research or isolated proof of concept

The Decision: ECS + Fractional GPU with ECS Managed Instances for Small Models

For an AI inference platform in a financial environment serving credit scoring, document classification, and anomaly detection models — all with maximum VRAM of 8 GB and throughput of 50-500 req/s per model — the decision is to adopt ECS with fractional GPU on G6f instances using ECS Managed Instances, with the following specific configurations.

Each inference task definition declares GPU=0.25 as default (6 GB of VRAM, 4 tasks per instance), reserving headroom for TensorRT runtime overhead (≈350 MB) and batch size spikes. Models with less than 2 GB of INT8-quantized weights can use GPU=0.125, but only on services with minimumHealthyPercent=100 and maximumPercent=200 to ensure that an instance replacement does not reduce available capacity below the minimum SLO.

The capacity provider is configured with a dedicated Auto Scaling Group (ASG) for G6f instances, with targetCapacityPercent=70 — maintaining 30% headroom to absorb spikes without provisioning latency. managedTerminationProtection=ENABLED on the capacity provider ensures that instances with active tasks are not terminated during scale-in, avoiding interruptions to in-flight inferences.

For IAM, each task role has minimum permissions via condition keys: aws:RequestedRegion fixed to the operating region, aws:SourceVpc restricted to the inference VPC, and S3 access for model artifact reads limited by prefix via s3:prefix condition. Dedicated KMS CMK per environment (dev/staging/prod) for model artifact encryption at rest, with key policy allowing only the ECS task role and the model deployment CI/CD pipeline.

The decision to use ECS Managed Instances over self-managed ECS on EC2 is motivated by automatic GPU health monitoring — specifically hardware failure detection that replaces instances without manual intervention. In a financial environment with 99.9% SLA for inference, that automation is more valuable than the additional AMI control that self-managed mode offers.

Fractional GPU Inference Architecture on ECS

Flow of a financial inference request from API Gateway to the ECS task with isolated GPU partition, showing scheduling, observability, and security isolation.

🌐 Edge / Ingress

• API Gateway REST + WAF (edge)
• NLB Internal (network)

🧠 ECS Inference Cluster (G6f)

• ECS Service Capacity Provider (compute)
• Task A GPU=0.25 (6 GB VRAM) (ai)
• Task B GPU=0.125 (3 GB VRAM) (ai)
• EC2 G6f NVIDIA L4 24GB (Managed Instance) (compute)

📦 Model Artifacts

• S3 Bucket Model Artifacts (KMS CMK) (storage)
• ECR Inference Image (Immutable Tag) (storage)

🔒 Security & IAM

• IAM Task Role VPC + Region Conditions (security)
• KMS CMK Per-Env Key (security)

📊 Observability

• CloudWatch Container Insights GPU Metrics (data)
• CloudWatch Alarm GPU Util < 20% → Scale-in (data)

Flows

• client -> apigw: HTTPS + JWT
• apigw -> nlb: WAF filtered
• nlb -> ecs_svc: internal routing
• ecs_svc -> task_a: schedule GPU=0.25
• ecs_svc -> task_b: schedule GPU=0.125
• task_a -> g6f: isolated L4 partition
• task_b -> g6f: isolated L4 partition
• task_a -> s3_models: artifact read (KMS)
• task_a -> ecr: image pull
• task_role -> task_a: assume role
• kms -> s3_models: decrypt artifact
• g6f -> cw_insights: GPU util / mem metrics
• cw_insights -> cw_alarm: threshold trigger

Specific Configuration: Task Definition, Capacity Provider, and Observability

Correct configuration of a task definition for fractional GPU requires attention to details not obvious in the launch documentation. The resourceRequirements field in the container definition accepts type: GPU with value as a string representing the fraction: "0.125", "0.25", or "0.5". Important: the value is a string, not a float — a common mistake that results in silent validation failure and task placement on instances without GPU.

For the capacity provider with G6f, the ASG configuration must include instanceWarmupPeriod of at least 180 seconds to accommodate NVIDIA driver and container runtime initialization time. With managedScaling enabled, targetCapacityPercent=70 on the capacity provider ensures ECS maintains enough instances to absorb a 43% spike above current capacity without provisioning latency. For fraud detection workloads with P99 SLO < 200ms, any instance cold-start latency is unacceptable — the 30% headroom is the insurance cost for that SLO.

GPU observability via CloudWatch Container Insights with ECS Managed Instances exposes metrics like GPUUtilization, GPUMemoryUsed, and GPUMemoryTotal per task and per instance. Alarm configuration must include: (1) GPUMemoryUsed / GPUMemoryTotal > 90% for 5 minutes as a memory pressure signal that precedes OOM kills; (2) GPUUtilization < 15% for 15 minutes as a safe scale-in signal; (3) TaskCount < desiredCount for 2 minutes as a service degradation alarm that triggers PagerDuty.

For regulatory traceability, each inference request must include in the response header the X-Model-Version and X-Inference-TaskArn — the ECS Task ARN allows correlating the model decision with the exact instance and GPU partition that executed it, which is required for credit model audits under Central Bank guidelines.

Consequences and Operational Risks: Per-instance blast radius is real and underestimated. With 4-8 tasks on a single G6f, an NVIDIA driver failure — not rare after kernel updates — brings down all partitions simultaneously. ECS Managed Instances health monitoring detects the failure and replaces the instance, but replacement time (provisioning + warmup + image pull + model initialization) can be 4-8 minutes. For critical financial workloads, this means redundancy design must ensure no single G6f instance serves more than 40% of total capacity for a model — implying a minimum of 3 active G6f instances per critical model, even if utilization does not justify it.

VRAM overflow is silent without configured alerts. If a model exceeds partition VRAM (3 GB for GPU=0.125), the default driver behavior is not a clean failure — it can swap to host memory with severe latency degradation (10-100x) before eventual OOM kill. Configure GPUMemoryUsed > 85% as WARNING alarm and > 95% as CRITICAL with automatic task replacement action.

Savings Plans cost coverage does not automatically cover G6f. G6f instances are covered by Compute Savings Plans, but not by EC2 Instance Savings Plans specific to other families. Verify existing coverage before migrating workloads to avoid end-of-month cost surprises.

Data compliance on shared partitions. Even with VRAM memory isolation between partitions, data in transit on the PCIe bus is shared. For models processing PII data from financial customers, evaluate whether partition isolation meets LGPD and PCI-DSS requirements before placing multiple tenants on the same G6f instance.

Cost Analysis: When Bin-Packing Actually Pays Off

The cost reduction promise of fractional GPU is real, but conditional. The calculation must be done with real utilization data, not the optimistic premise that all partitions will be occupied all the time.

Consider a concrete scenario: a financial platform with 12 small inference models, each with a peak of 100 req/s and 4 GB VRAM. Under the previous model (full GPU per task), you would need 12 G6 instances (one GPU per model), at approximately $X per hour. With fractional GPU using GPU=0.25 (4 tasks per G6f instance), you need a minimum of 3 G6f instances for the 12 models — a theoretical 75% reduction in instance count.

But the real calculation includes: (1) 30% headroom to absorb spikes without cold-start → 3 instances become 4; (2) minimum redundancy of 3 instances per critical model for blast radius → critical models need dedicated instances; (3) instanceWarmupPeriod of 180s means the ASG cannot scale fast enough for abrupt spikes without pre-warming.

In practice, for 12 models with mixed distribution (4 critical + 8 experimental), the optimal configuration is: 4 dedicated G6f instances for critical models (one model per instance, using GPU=0.25 to leave headroom for simultaneous A/B versions) + 2 shared G6f instances for the 8 experimental models. This results in 6 G6f instances versus 12 previous G6 instances — a 50% reduction, not 75%, but still significant. The real benefit is that experimental models move from dedicated to shared instances, freeing capacity for critical ones without cost increase.

To calculate break-even, the correct metric is cost per 1000 inferences per model, not instance cost. With bin-packing, the denominator increases proportionally to the number of tasks per instance, but the numerator (instance cost) remains fixed — which improves cost per inference only when aggregate partition utilization is high enough to justify the shared instance cost.

Well-Architected Review: Fractional GPU on ECS

• security: IAM task roles with VPC and region condition keys; per-environment KMS CMK for model artifacts; no EC2 instance metadata access via disableNetworking or mandatory IMDSv2; dedicated SecurityGroup for the GPU ECS cluster with egress restricted to S3 endpoint and ECR endpoint via VPC Endpoints.
• reliability: Minimum of 3 active G6f instances per critical model to tolerate one instance failure without SLO degradation; minimumHealthyPercent=100 on critical services; automatic ECS Managed Instances health monitoring for GPU-failed instance replacement; deployment circuit breaker enabled to prevent rollout of images with model initialization failure.
• performance: TensorRT for inference optimization with FP16/INT8 precision; GPU=0.25 as default for VRAM headroom; instanceWarmupPeriod=180s on ASG; instance pre-warming via scheduled scaling for predictable spikes (e.g., financial market open at 9am); P50/P95/P99 latency monitoring per model via CloudWatch Embedded Metrics Format.
• sustainability: Fractional GPU bin-packing reduces the number of active instances, lowering energy consumption per inference; INT8-quantized models reduce FLOPs per inference without significant accuracy degradation for credit scoring; automatic idle instance shutdown via scale-to-zero for development and staging environments.

Anti-Patterns to Avoid with Fractional GPU on ECS

• Using GPU=0.125 for critical production models without instance redundancy — 3 GB VRAM is sufficient for small models, but the blast radius of an instance failure is unacceptable without multiple replicas on different instances.
• Mixing models from different tenants (financial clients) on the same G6f instance without data compliance evaluation — VRAM isolation does not guarantee data isolation across all attack vectors.
• Setting targetCapacityPercent=100 on the capacity provider to maximize utilization — this eliminates the headroom needed to absorb spikes and results in instance cold-start latency during traffic spikes.
• Using self-managed ECS on EC2 without implementing GPU health monitoring equivalent to ECS Managed Instances — GPU hardware failures without automatic detection result in stuck tasks serving silent errors.
• Assuming the GPU value in the task definition is a float — it is a string; GPU=0.25 works, GPU=0.250 may fail depending on the ECS agent version.
• Ignoring inference framework runtime overhead in VRAM sizing — TensorRT, PyTorch Serve, and Triton consume 200-500 MB of VRAM before loading any model weights.

Architect's Note: In my experience with inference platforms in financial environments, the most expensive mistake is not choosing the wrong GPU fraction — it is underestimating the blast radius of hardware failures on high-density task instances. I always start with GPU=0.25 and never place more than 40% of a critical model's capacity on a single G6f instance, regardless of headroom cost. The second most common mistake I see is skipping VRAM validation in the CI/CD pipeline: a model that grew from 2.8 GB to 3.1 GB of VRAM after a silent fine-tune will cause latency degradation in production before any alarm fires. Automate that validation as a deploy gate. Finally, ECS Managed Instances is the right choice for 90% of cases — the additional control of self-managed mode rarely justifies the operational cost for teams without a dedicated NVIDIA driver management specialist.

Verdict: Adopt with Sizing Discipline and Redundancy

Fractional GPU on ECS with G6f instances is a genuinely useful feature for AI inference platforms operating multiple small models — especially in financial environments where cost per inference and regulatory traceability are real constraints. Native integration with the ECS scheduler eliminates the operational complexity of custom device plugins that teams had to maintain on EKS, and ECS Managed Instances with automatic GPU health monitoring reduces the operational risk of undetected hardware failures.

The recommendation is to adopt GPU=0.25 as default for production models (VRAM headroom), GPU=0.125 only for experimental or low-volume models with explicit redundancy, and GPU=0.5 for models that need 10-12 GB of VRAM after runtime overhead. Configure targetCapacityPercent=70 on the capacity provider, implement GPUMemoryUsed > 85% and GPUUtilization < 15% alarms, and never concentrate more than 40% of a critical model's capacity on a single G6f instance.

What this feature does not solve: models requiring more than 12 GB of VRAM (use GPU=0.5 or G6/G5 instances with full GPU), workloads requiring hardware-level tenant isolation between partitions, and environments where G6f regional coverage is not yet available. For those cases, EKS with manually configured MIG or dedicated instances per model remain the correct choices.

Rating: Recommended with Conditions

References

• Amazon ECS Fractional GPU Scheduling — AWS What's New (Aug 2026)
• Amazon EC2 G6f Instances GA with Fractional GPUs — AWS What's New (Jul 2025)
• Amazon ECS Fractional GPU Documentation
• ECS Managed Instances — AWS Documentation
• CloudWatch Container Insights Enhanced Observability for ECS
• Amazon EC2 G6 Instance Page
• AWS Well-Architected Framework — Performance Efficiency Pillar
• NVIDIA L4 Tensor Core GPU Product Brief

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.