Two worlds, one discipline:

HOSTED-MODEL APPS (LLMOps)             You call a managed model (Azure OpenAI and similar). You ship: application code, prompts, retrieval indexes, tool definitions, and
                                       configuration. You operate: quotas, gateways, evaluation gates, cost, model-version pinning, provider changes.

SELF-HOSTED MODELS (classic MLOps)     You run the model: ONNX/ML.NET models in your process, or model servers in containers on CPU/GPU nodes. You ship and operate all of the above PLUS
                                       model artifacts, container images, GPU capacity, scaling, and model drift.

Most real systems are a mix — a hosted LLM for generation, a self-hosted embedding or vision model, a managed search index — so this guide covers both.

MANAGED MODEL ENDPOINT        A provider hosts the model; you create a "deployment" and call it (Azure OpenAI deployments, Azure AI Foundry model deployments, other hosted APIs).
                              + No GPUs to manage, fastest path, provider handles scaling.   - Quotas, per-token cost, provider-driven model lifecycle, data leaves your boundary (unless private networking/terms suffice).

IN-PROCESS MODEL              The model is loaded inside your app: ONNX Runtime or ML.NET in an ASP.NET Core service (classifiers, detectors, embedding models, small LMs).
                              + Lowest latency, no network hop, simplest topology, scales with the app.   - Memory per replica, startup time, CPU/GPU contention with the app, tied to app release cadence.

SEPARATE MODEL SERVICE        A dedicated inference service over HTTP/gRPC (your own container, NVIDIA Triton, vLLM for LLMs, KServe, Azure ML managed online endpoints, Ollama for development).
                              + Independent scaling, GPU isolation, multiple clients, model updates decoupled from the app.   - Another service to operate, network latency, more moving parts.

SERVERLESS / SCALE-TO-ZERO    Azure Container Apps (including serverless GPU workload profiles) or Functions with a model.
                              + Pay only when used, good for spiky or low traffic.   - Cold starts (model load!), GPU availability, limits on size and duration.

EDGE / ON-DEVICE              Models shipped to devices or local servers (ONNX Runtime, DirectML, NPUs).
                              + Offline, private, no per-call cost.   - Update distribution, device variability, no central monitoring unless you build it.

General LLM, no special data-residency or custom-model needs            -> managed endpoint (Azure OpenAI)
A small model (classifier, detector, embedder) with tight latency       -> in-process ONNX/ML.NET
A big or GPU-hungry model shared by several apps                        -> separate model service on GPU nodes (AKS / Azure ML / Container Apps GPU)
Occasional, bursty batch jobs                                           -> scale-to-zero jobs or serverless; batch APIs for hosted models
Strict offline/privacy requirements                                     -> self-hosted open-weight model or on-device

// An Azure OpenAI deployment declared as infrastructure (illustrative — verify the API version, model name, version, SKU names, and capacity units)
resource chatDeployment 'Microsoft.CognitiveServices/accounts/deployments@2024-10-01' = {
  parent: openAiAccount
  name: 'chat-prod'                                   // the deployment name your app uses (the app never hard-codes the underlying model name)
  sku: {
    name: 'Standard'                                  // pay-per-token; provisioned throughput SKUs exist for predictable latency at steady load
    capacity: 100                                     // quota units (thousands of tokens per minute)
  }
  properties: {
    model: { format: 'OpenAI', name: 'gpt-4o-mini', version: '2024-07-18' }
    versionUpgradeOption: 'NoAutoUpgrade'            // PIN the model version: upgrades happen when YOU choose, after evaluation
    raiPolicyName: 'prod-content-policy'              // server-side content filter policy
  }
}

Why this matters:
  - PINNED VERSION: a silent upgrade can change behavior (and quality) under you. Pin, evaluate the new version in a parallel deployment, then switch.
  - DEPLOYMENT NAMES decouple code from model names: moving "chat-prod" to a new model is an infrastructure change, not a code deploy.
  - ENVIRONMENT PARITY: dev/test/prod deployments created from the same template.
  - SERVER-SIDE POLICY: content filters and quotas set where callers can't weaken them (see AI Security, Section 12).

PACKAGE      Export to a portable format (ONNX from PyTorch/TensorFlow; an ML.NET model .zip), optimize and optionally quantize, and record a manifest (Section 2).
STORE        Put artifacts in a model registry or versioned blob storage (not in source control for large files, not only inside a container image).
LOAD         At startup, fetch the model, verify its checksum, create ONE inference session, and run a warm-up inference.
GATE         Don't accept traffic until the model is loaded and warm: readiness probes (below). A pod that serves requests while still loading just returns errors.
ROLL OUT     Blue/green, canary, or shadow — compare the new model to the old on real traffic before cutting over (Section 3).

using Microsoft.ML.OnnxRuntime;

var options = new SessionOptions
{
    GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_ALL,       // graph-level optimizations (operator fusion, constant folding)
    IntraOpNumThreads = int.Parse(Environment.GetEnvironmentVariable("ORT_INTRA_OP_THREADS") ?? "4"),   // match the container's CPU limit — don't oversubscribe
    ExecutionMode = ExecutionMode.ORT_SEQUENTIAL,
};

// GPU (requires the Microsoft.ML.OnnxRuntime.Gpu package, plus matching CUDA/cuDNN libraries in the image):
// options.AppendExecutionProvider_CUDA(deviceId: 0);

var session = new InferenceSession(modelPath, options);                   // create ONCE per process; Run() is safe to call concurrently

public sealed class ModelState { public volatile bool Ready; }

public sealed class ModelLoader(ModelState state, ObjectDetector detector, ILogger<ModelLoader> log) : BackgroundService
{
    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // Loading and a dummy inference happen at startup, off the request path
        detector.WarmUp();                                                // run a few inferences on a synthetic image to initialize kernels and allocators
        state.Ready = true;
        log.LogInformation("Model loaded and warm");
        return Task.CompletedTask;
    }
}

public sealed class ModelReadyCheck(ModelState state) : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default) =>
        Task.FromResult(state.Ready ? HealthCheckResult.Healthy() : HealthCheckResult.Unhealthy("Model not ready"));
}

builder.Services.AddSingleton<ModelState>();
builder.Services.AddHostedService<ModelLoader>();
builder.Services.AddHealthChecks().AddCheck<ModelReadyCheck>("model", tags: ["ready"]);

app.MapHealthChecks("/health/ready", new HealthCheckOptions { Predicate = c => c.Tags.Contains("ready") });   // Kubernetes readiness probe
app.MapHealthChecks("/health/live",  new HealthCheckOptions { Predicate = _ => false });                      // liveness: "the process is up" — does NOT depend on the model

BLUE/GREEN     Run old and new side by side; switch traffic at once; instant rollback by switching back. Doubles capacity during the switch.
CANARY         Send a small share (1%, 5%, 25%...) to the new version, watch metrics (quality, latency, errors, cost), and expand or roll back automatically.
SHADOW         Mirror real traffic to the new version, discard its responses, and compare offline (quality, cost, latency). Zero user risk; best for validating a new model or prompt.
A/B TEST       Randomly split users between versions and compare outcomes (feedback, task success). Needs enough traffic and a clear success metric.
ROLLING        Replace replicas gradually (Kubernetes default) — fine for app code, but for models make sure old and new replicas can coexist and produce compatible outputs.
ALWAYS         Keep the previous version deployable, record which version served each request (trace attribute), and define rollback criteria BEFORE the rollout.

MODEL WEIGHTS            The ONNX file, ML.NET .zip, or fine-tuned checkpoint (and, for hosted models, the provider's model + version string)
TOKENIZER / VOCAB / LABELS  Must match the weights exactly — a tokenizer mismatch silently destroys accuracy
PREPROCESSING / POSTPROCESSING CONFIG  Input size, normalization, thresholds, NMS settings, label maps
PROMPTS                  System prompts, templates, few-shot examples, judge rubrics (a prompt change is a behavior change)
RETRIEVAL INDEX          Index contents version, chunking strategy, and the EMBEDDING MODEL used (changing the embedding model invalidates every stored vector)
TOOL DEFINITIONS         Names, descriptions, schemas, and permissions
EVALUATION SETS          The golden set and its grading rubrics — so scores are comparable over time
INFRASTRUCTURE / CONFIG  Deployment templates, quotas, content-filter policies, price tables
DATA (for training)      The dataset version, to make training reproducible
CODE                     Application and pipeline code (git commit)

public record ModelManifest(
    string Name,                    // "vision-detector"
    string Version,                 // semantic version: bump MAJOR when outputs/labels/contract change, MINOR for retrains with the same contract, PATCH for fixes
    string Sha256,                  // checksum of the model artifact — verified at load time
    string InputSpec,               // "rgb 640x640 float32 0..1 NCHW, letterboxed"
    string[] Labels,
    string TrainingDataVersion,     // lineage: which data
    string GitCommit,               // lineage: which code
    DateTimeOffset TrainedAt,
    Dictionary<string, double> EvalMetrics);     // what it scored when it was registered (mAP, accuracy, etc.)

public static class ModelVerifier
{
    // Refuse to load an artifact whose checksum doesn't match its manifest — protects against corruption, wrong files, and tampering
    public static async Task VerifyAsync(string modelPath, ModelManifest manifest)
    {
        await using FileStream stream = File.OpenRead(modelPath);
        byte[] hash = await SHA256.HashDataAsync(stream);

        if (!Convert.ToHexString(hash).Equals(manifest.Sha256, StringComparison.OrdinalIgnoreCase))
            throw new InvalidOperationException($"Model {manifest.Name}@{manifest.Version} failed integrity verification.");
    }
}

A MODEL REGISTRY stores immutable, versioned artifacts with metadata and lets you promote versions through stages.
  Options: Azure Machine Learning model registry · MLflow Model Registry · Hugging Face Hub (private repos) · OCI registries (artifacts in Azure Container Registry) · versioned Blob Storage with a manifest.

PRINCIPLES
  IMMUTABLE        A published version is never overwritten. A fix is a NEW version.
  LINEAGE          Each version records its training data version, code commit, hyperparameters, and evaluation metrics.
  PROMOTION        dev -> staging -> production via stages, aliases, or tags (for example "champion" and "challenger"), gated by evaluation and approval — and audited.
  ROLLBACK         Re-pointing the production alias to the previous version is a one-line, instant operation.
  ACCESS CONTROL   Who can register, promote, and read models — models are valuable and, if poisoned, dangerous (see AI Security, Section 12).

Providers retire model versions on a published schedule, and some "latest" aliases move without notice. For hosted models:
  PIN the version in your deployment (see the Bicep above); never rely on an unpinned alias for a production workload that you've evaluated.
  TRACK RETIREMENT DATES for every model you use and put the migration on your calendar well ahead of time.
  MIGRATE DELIBERATELY: stand up the new version as a parallel deployment, run your evaluation suite against it, shadow/canary it, then switch — treat it as a release, not a config tweak.
  RECORD the actual model/version from each response (response.ModelId) on traces and logs so a provider-side change is visible (see AI Observability, Section 8).

PROMPTS    Files in source control (prompts/support-answer/v12.md), loaded by name and version, changed through pull requests with evaluation results attached. Tag every trace with the prompt name
           and version. Support deploying a prompt change independently of code when needed — but through the same review, evaluation, and rollout discipline.

INDEXES    Build a NEW index for any change to chunking, the embedding model, or the schema; validate it with the golden set; then switch an ALIAS or configuration pointer (blue/green for indexes).
           Keep the old index until the new one is proven. Store the embedding model name and index version in each record and in the index metadata.

COMPATIBILITY  Keep a matrix of which model version works with which tokenizer, preprocessing, labels, prompt version, and downstream consumers. A model that adds a class or changes an output
           shape is a breaking change for everything that consumes it — treat it as a MAJOR version.

DATA DRIFT       The distribution of inputs changes (new customer segment, a new camera, a new product line, seasonal language, a different mix of languages).
CONCEPT DRIFT    The relationship between inputs and the right answer changes (fraud tactics evolve, "good" customer behavior changes, a policy changes the labels).
PREDICTION DRIFT The distribution of outputs shifts (the classifier suddenly predicts one class far more often) — often the earliest visible symptom.
UPSTREAM CHANGE  A feature pipeline, tokenizer, data source, or the provider's model changes without notice.
CORPUS CHANGE    (RAG) The documents change or the index is rebuilt; retrieval quality shifts with them.
A model that was excellent at launch quietly gets worse. Without monitoring, you learn about it from customers.

1. SYSTEM HEALTH        Availability, error classes, latency, throughput, GPU/CPU/memory, queue depth (Section 9 and AI Observability).
2. INPUT / DATA QUALITY Schema and range checks, missing values, new categories, language mix, image size and format distribution, text-length distribution.
3. DATA DRIFT           Statistical comparison of live input distributions against a reference (the training or validation data, or a recent healthy window).
4. PREDICTION DRIFT     Distribution of predicted classes, scores, and confidence over time; rising share of low-confidence or "unknown" outputs.
5. MODEL PERFORMANCE    Actual accuracy/precision/recall/F1 (or task success) — which requires ground truth, often delayed (below).
6. BUSINESS IMPACT      The KPI the model exists to move: conversion, fraud caught, handling time, deflection, defect escape rate.

Performance metrics need labels, and in production labels arrive late, partially, or never.
  DELAYED LABELS     Outcomes known later (a loan defaults in 12 months, a return occurs in 30 days). Monitor leading indicators (drift, confidence) in the meantime, then backfill performance.
  HUMAN REVIEW       Route a sample (and all low-confidence cases) to reviewers; their decisions are labeled data and a live accuracy estimate.
  USER FEEDBACK      Corrections, overrides, thumbs, escalations — biased and sparse, but valuable signals.
  CANARY SETS        A fixed labeled set re-scored on a schedule detects model/pipeline regressions even without fresh labels.
  PROXY METRICS      Confidence distributions, agreement between the model and a stronger offline model, and consistency checks.

// PSI compares a feature's (or a score's) live distribution to a reference. Rules of thumb often used: < 0.1 stable, 0.1-0.25 moderate shift, > 0.25 significant shift.
public static double PopulationStabilityIndex(double[] reference, double[] live, int bins = 10)
{
    // Bin edges are the reference distribution's quantiles, so each reference bin holds ~equal mass
    double[] sorted = reference.OrderBy(x => x).ToArray();
    double[] edges = Enumerable.Range(1, bins - 1)
        .Select(i => sorted[(int)(i * sorted.Length / (double)bins)])
        .ToArray();

    double[] Distribution(double[] data)
    {
        var counts = new double[bins];
        foreach (double value in data)
        {
            int bin = Array.FindIndex(edges, edge => value <= edge);
            counts[bin < 0 ? bins - 1 : bin]++;
        }
        return counts.Select(c => Math.Max(c / data.Length, 1e-6)).ToArray();         // floor avoids log(0)
    }

    double[] expected = Distribution(reference);
    double[] actual   = Distribution(live);

    return expected.Zip(actual, (e, a) => (a - e) * Math.Log(a / e)).Sum();
}

Other tools: Kolmogorov-Smirnov test (continuous features), chi-square (categorical), Jensen-Shannon / Wasserstein distances, and embedding-based drift (cluster live inputs and watch cluster shares —
useful for text and images). Platforms such as Azure Machine Learning model monitoring and open-source libraries (for example Evidently, a Python library) can compute these; you can also run them from
.NET jobs. Compute drift on a schedule (hourly/daily windows), emit the scores as metrics, and alert on thresholds.

You can't observe weights or training data, so monitor behavior:
  MODEL VERSION      Alert when response.ModelId changes (provider update) — record it on every trace.
  CANARY GOLDEN SET  Run it on a schedule against production configuration; alert on score drops (see AI Observability, Section 8).
  OUTPUT STATISTICS  Length, refusal rate, format-failure rate, finish-reason mix, language, sentiment — sudden shifts mean something changed.
  INPUT STATISTICS   Prompt length, language mix, topic clusters, new intents.
  QUALITY SAMPLING   Sampled LLM-judge scores and user feedback by prompt/model/index version.
  COST AND LATENCY   Tokens per request, cost per outcome, TTFT percentiles.

TRIGGERS     Drift above threshold · performance below SLO · new data volume · schedule (for example quarterly) · a significant upstream or business change.
RESPONSE     Investigate (is it real drift or a pipeline bug?) -> retrain or fine-tune on fresh data, or update prompts/index -> evaluate against the golden set AND recent real data ->
             shadow/canary -> promote with approval -> monitor. Rollback to the previous version if the new one underperforms.
AUTOMATION   Automate the pipeline (data -> train -> evaluate -> register), but keep a HUMAN approval gate before promotion to production for anything consequential.
GUARDRAIL    Beware feedback loops: if model predictions influence the data you later train on (recommendations shape clicks), monitor for self-reinforcing bias.

CODE CHANGE     build · unit tests (fake chat client) · static analysis · dependency + secret scanning · container build + scan · deploy · smoke test
PROMPT CHANGE   lint · EVALUATION against the golden set · security (injection) suite · cost/latency check · review · promote · canary
MODEL CHANGE    registry registration · EVALUATION on held-out + recent data · performance benchmark (latency/throughput on target hardware) · shadow/canary · promote with approval
INDEX CHANGE    build new index · retrieval evaluation (Recall@k, MRR) · switch alias · monitor · keep the old index for rollback
INFRA CHANGE    IaC plan/what-if · policy checks · deploy to staging · verify quotas and content-filter policy

UNIT TESTS (fast, deterministic, free)    Business logic, prompt assembly, history trimming, tool wiring and authorization, validators, parsers — against a FAKE IChatClient
                                          (see Microsoft.Extensions.AI, Section 10). Run on every commit.

INTEGRATION TESTS                         Real vector store/database in a container (Testcontainers), real HTTP pipeline, still a fake or recorded model. Verifies plumbing.

EVALUATION TESTS (slower, statistical, costs money)  The golden set run against the REAL model/prompt/index: retrieval metrics, judged faithfulness, citation validity, task success, refusal behavior, cost
                                          and latency. Repeat each case several times (non-determinism). Compare to a baseline. Run when prompts, models, retrieval, or tools change — not necessarily on every commit.

SECURITY TESTS                            The adversarial suite: injection, exfiltration, cross-tenant access, tool misuse (see AI Security, Section 14). A release gate.

LOAD / PERFORMANCE TESTS                  Throughput, latency percentiles, saturation, and cost at target traffic — on production-like hardware.

SMOKE / SYNTHETIC CHECKS                  After deploy and continuously in production: a handful of real end-to-end requests.

[Trait("Category", "Eval")]
public class SupportAssistantEvals(EvalFixture fixture) : IClassFixture<EvalFixture>
{
    [Fact]
    public async Task Release_candidate_meets_quality_safety_and_cost_thresholds()
    {
        EvalReport report = await fixture.RunGoldenSetAsync(repetitions: 3);          // every case 3x because outputs vary

        Assert.True(report.RetrievalHitRateAtK >= 0.90,  $"Retrieval hit rate {report.RetrievalHitRateAtK:P1} below 90%");
        Assert.True(report.MeanFaithfulness     >= 4.2,  $"Faithfulness {report.MeanFaithfulness:F2} below 4.2");
        Assert.True(report.CitationValidityRate >= 0.97, $"Citation validity {report.CitationValidityRate:P1} below 97%");
        Assert.True(report.CorrectRefusalRate   >= 0.95, "Unanswerable questions must be refused");

        Assert.Equal(0, report.InjectionAttacksSucceeded);                            // security gate: any successful injection fails the build
        Assert.Equal(0, report.CrossTenantLeaks);

        Assert.True(report.P95LatencySeconds    <= 12,   $"p95 latency {report.P95LatencySeconds:F1}s above budget");
        Assert.True(report.MeanCostPerRequestUsd <= 0.015m, $"Cost per request ${report.MeanCostPerRequestUsd:F4} above budget");

        // Regression vs. the last released version: no metric may drop by more than the noise tolerance
        Assert.True(report.MeanFaithfulness >= fixture.Baseline.MeanFaithfulness - 0.15, "Faithfulness regressed vs. baseline");
    }
}

Practical notes on evaluation in CI:
  - NON-DETERMINISM: use repetitions and tolerance bands; a single failure on a single run is noise, a shift in the mean is signal.
  - COST: eval runs call real models. Cache responses (the Microsoft.Extensions.AI.Evaluation reporting libraries support response caching), run on a sampled or smaller "smoke" subset for pull requests, and the
    full suite on merge, nightly, or when prompt/model/retrieval paths change (use path filters).
  - BASELINES: store results of the production version and compare against it, rather than only absolute thresholds.
  - JUDGE STABILITY: pin the judge model/version and prompts; changing the judge changes the scores.
  - DATA HYGIENE: keep the golden set out of prompts and fine-tuning data; version it; refresh it with real failures.
  - SECRETS: evaluation needs model access — use short-lived federated credentials (below), not stored keys.

name: ai-app
on:
  pull_request:
  push: { branches: [main] }

permissions:
  id-token: write        # OIDC federation to Azure — no stored cloud credentials
  contents: read

jobs:
  build-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with: { dotnet-version: '9.0.x' }
      - run: dotnet restore
      - run: dotnet build --no-restore -c Release
      - run: dotnet test --no-build -c Release --filter "Category!=Eval&Category!=Load"     # fast unit + integration tests with fakes
      - name: Secret scan
        uses: gitleaks/gitleaks-action@v2

  evaluate:
    needs: build-test
    runs-on: ubuntu-latest
    environment: ai-eval                              # protected environment; scoped federated credential with inference-only rights
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
      - uses: actions/setup-dotnet@v4
        with: { dotnet-version: '9.0.x' }
      - name: Evaluation + security gate
        run: dotnet test -c Release --filter "Category=Eval|Category=Security"
        env:
          AI_ENDPOINT: ${{ vars.EVAL_AI_ENDPOINT }}                 # an evaluation deployment, with its own quota and budget alerts

  image:
    needs: [build-test, evaluate]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
      - run: az acr build --registry myregistry --image ai-app:${{ github.sha }} .
      - name: Vulnerability scan
        uses: aquasecurity/trivy-action@master            # pin to a specific version or commit SHA in real pipelines
        with: { image-ref: 'myregistry.azurecr.io/ai-app:${{ github.sha }}', severity: 'CRITICAL,HIGH', exit-code: '1' }

  deploy-staging:
    needs: image
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - run: echo "Deploy via IaC / GitOps (Flux or Argo CD), run smoke tests, then start the canary"

FEDERATED CREDENTIALS     CI authenticates to Azure with OIDC federation (workload identity) — never a stored client secret or API key. Evaluation identities get inference-only roles.
SEPARATE EVAL RESOURCES   Use dedicated deployments/quotas/budgets for evaluation so test traffic can't starve production or surprise finance.
PATH FILTERS              Run expensive evaluation only when relevant paths change (prompts/, retrieval code, model references, tool definitions).
ARTIFACT PROMOTION        Build once, promote the SAME image and model/prompt versions through environments; don't rebuild per environment.
CONFIG-DRIVEN CHANGES     Allow prompt and model-version changes as reviewed configuration rollouts with audit trails — and still require the evaluation gate.
SUPPLY CHAIN              Pin base images and dependencies, scan images, generate an SBOM, and sign images (for example with Sigstore cosign); verify model checksums at load.
GITOPS                    Declare desired deployment state in git (Flux or Argo CD) so rollouts and rollbacks are reviewed, auditable commits.
AUTOMATED ROLLBACK        Tie canary analysis to SLO and quality metrics; roll back automatically on breach.
TRAINING PIPELINES        For self-trained models: data validation -> train -> evaluate -> register -> approve -> deploy, in an orchestrator (Azure ML pipelines, or workflow tools), with lineage recorded.

AZURE OPENAI / AZURE AI FOUNDRY      Hosted OpenAI-family (and, through the model catalog, other) models; deployments, content filtering, evaluation, tracing, an agent service, and project/hub organization
                                     for AI work. The default managed home for LLM applications on Azure.
AZURE AI SEARCH                      Managed full-text + vector + hybrid search with semantic ranking and filtering — the usual retrieval layer for RAG.
AZURE AI LANGUAGE / VISION / SPEECH / TRANSLATOR / DOCUMENT INTELLIGENCE    Prebuilt task APIs (sentiment, entities, PII, summarization; image analysis and OCR; speech-to-text; translation; document extraction).
AZURE AI CONTENT SAFETY              Moderation and prompt-attack detection for text and images.
AZURE MACHINE LEARNING               Training, experiment tracking, pipelines, model registry, managed online/batch endpoints, data and model monitoring, MLflow integration.
AZURE CONTAINER APPS                 Serverless containers with KEDA-based scaling (including scale to zero) and GPU workload profiles; simplest path for containerized AI services.
AZURE KUBERNETES SERVICE (AKS)       Full Kubernetes with GPU node pools; operators for model serving (for example the Kubernetes AI Toolchain Operator, KAITO); maximum control.
AZURE API MANAGEMENT                 An "AI gateway": authentication, token-based rate limits, token metrics, semantic caching, load balancing and circuit breaking across model backends.
AZURE FUNCTIONS                      Event-driven glue and light inference; mind cold starts for model loading.
AZURE KEY VAULT / ENTRA ID / MANAGED IDENTITY    Secrets, keyless authentication, workload identity (see AI Security, Section 10).
AZURE MONITOR / APPLICATION INSIGHTS / LOG ANALYTICS    Telemetry backend (see AI Observability, Section 9).
AZURE CONTAINER REGISTRY (ACR)       Image (and OCI artifact) registry, with scanning and geo-replication options.
AZURE REDIS / COSMOS DB / POSTGRESQL    Caching, document storage, and vector stores (pgvector).

DEPLOYMENT TYPES (names and options change — check current docs)
  STANDARD (pay-as-you-go)      Billed per token; shared capacity; subject to quotas; the default. Latency can vary with load.
  PROVISIONED THROUGHPUT        Reserved capacity (units) for predictable latency and throughput at steady, high volume; billed for the reservation. Cost-effective only when well utilized.
  GLOBAL / DATA ZONE / REGIONAL  Where inference may be processed — trade-offs between capacity/cost and data-residency requirements.
  BATCH                         Asynchronous processing of large request sets at a discount, with results within a window — right for offline classification, summarization, and embedding jobs.

QUOTAS         Tokens-per-minute (and requests-per-minute) limits per subscription, region, and model; allocated across your deployments. Hitting them returns 429 (see AI APIs with .NET, Section 11).
PLANNING       Estimate TPM from expected requests x (average input + output tokens); request quota increases early; spread load across deployments/regions; combine provisioned capacity for the steady
               baseline with standard capacity (or a spillover/fallback) for bursts.
LIFECYCLE      Models have retirement dates; pin versions and migrate deliberately (Section 2).
POLICIES       Content filtering and safety policies are configured per deployment; managed identity + RBAC for access; private endpoints for network isolation; diagnostic logs to Azure Monitor.

Put Azure API Management (or another gateway) in front of model endpoints. It becomes the single policy-enforcement point:
  - AUTHENTICATION of callers (Entra ID/JWT), so applications never hold provider keys; the gateway uses managed identity to reach the backend.
  - TOKEN-BASED RATE LIMITS and quotas per application, team, or user — not just request counts.
  - TOKEN METRICS emitted per caller for cost attribution and chargeback.
  - SEMANTIC CACHING of responses (with the caveats in Section 8).
  - LOAD BALANCING AND FAILOVER across multiple deployments/regions with circuit breakers — absorbing 429s and regional trouble.
  - CENTRAL LOGGING and request/response size limits; header hygiene.

<!-- Illustrative API Management policy fragment: per-caller token limit and token metrics (check the current policy names and attributes in the API Management reference) -->
<inbound>
    <base />
    <llm-token-limit counter-key="@(context.Subscription.Id)" tokens-per-minute="50000" estimate-prompt-tokens="true" />
    <llm-emit-token-metric namespace="ai-gateway">
        <dimension name="Subscription ID" />
        <dimension name="API ID" />
    </llm-emit-token-metric>
</inbound>

AZURE CONTAINER APPS     Simplest operations; scale-to-zero and event-driven scaling; GPU workload profiles for inference. Good for small and medium services and spiky traffic.
AKS                      Maximum control: GPU node pools, custom schedulers, serving stacks (Triton, vLLM, KServe), multi-model clusters. Needs Kubernetes expertise.
AZURE MACHINE LEARNING   Managed online endpoints (autoscaling, traffic splitting for canary/blue-green, monitoring) and batch endpoints; strong if your team trains models there too.
VIRTUAL MACHINES         GPU VMs for single-purpose or long-running workloads; the most manual option.
Choose the simplest option that meets your scale and control needs; move up only when you must.

TAGS AND SEPARATE RESOURCES   Tag resources by application, environment, and owner; use separate deployments/projects per application so billing exports attribute spend cleanly.
BUDGETS AND ALERTS            Azure Cost Management budgets with alerts; provider-side spending limits where available.
POLICY                        Azure Policy to enforce private endpoints, disable key authentication, require content-filter policies, and restrict allowed regions and models.
PRIVATE NETWORKING            Private endpoints and virtual-network integration for model, search, storage, and Key Vault; disable public access where feasible (see AI Security, Section 12).

# ---- build stage ----
FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build
WORKDIR /src

# Copy project files first so the restore layer is cached until dependencies change
COPY *.sln ./
COPY src/VisionApi/VisionApi.csproj src/VisionApi/
RUN dotnet restore src/VisionApi/VisionApi.csproj

COPY . .
RUN dotnet publish src/VisionApi/VisionApi.csproj -c Release -o /app/publish --no-restore /p:UseAppHost=false

# ---- runtime stage ----
FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS runtime
WORKDIR /app
COPY --from=build /app/publish .

# The model is NOT baked into the image: it is mounted or fetched at startup (see below)
ENV ASPNETCORE_URLS=http://+:8080 \
    MODEL_PATH=/models/model.onnx \
    ORT_INTRA_OP_THREADS=4

EXPOSE 8080
USER $APP_UID                       # run as non-root (the base image defines APP_UID)
ENTRYPOINT ["dotnet", "VisionApi.dll"]

BAKED INTO THE IMAGE       + Self-contained, simplest, versioned together with the app, fast startup (no download).
                           - Huge images (slow pulls and pushes, registry cost), every model update rebuilds and redeploys the image, secrets/licensing concerns if the model is sensitive.
                           Good for: small models (tens of MB) that change rarely.

MOUNTED VOLUME             + Decouples model from image; many pods can share a read-only volume (Azure Files / Blob CSI); update the model without rebuilding.
                           - Volume performance and availability become dependencies; versioning is a convention you must enforce.

DOWNLOADED AT STARTUP      + Image stays small; the model version comes from configuration; works with a registry or blob storage and checksum verification (Section 2).
(INIT CONTAINER OR APP)    - Cold start includes the download — mitigate with node-local caching, an init container, and a startup probe that waits for readiness.
                           Good for: large or frequently updated models. The usual choice for GPU and LLM-scale artifacts.

RULE OF THUMB   Small + stable: bake in.  Large or changing: fetch/mount, verify the checksum, cache on the node.

GLIBC vs. MUSL     Native ML libraries (ONNX Runtime, OpenCV, CUDA) are built for glibc. Alpine's musl libc often breaks them or lacks official builds. Prefer Debian/Ubuntu-based .NET images for AI workloads.
CHISELED IMAGES    Minimal, non-root, shell-less Ubuntu images shrink size and attack surface — but they omit packages some native libraries need. Test thoroughly before adopting.
GPU IMAGES         Use an NVIDIA CUDA runtime base (with cuDNN) matching the CUDA/cuDNN versions your ONNX Runtime GPU package requires, install the .NET runtime on top (or copy it in), and run with the NVIDIA
                   Container Toolkit (--gpus all locally; the device plugin in Kubernetes). A version mismatch is the classic cause of silent fallback to CPU (Section 10).
OPENCV             Include the matching OpenCvSharp runtime package for your OS and any required system libraries; test in the final image, not on your dev machine.
KEEP IT SMALL      Multi-stage builds; no SDK in the runtime image; a .dockerignore; don't copy test data or model files you don't need.

NON-ROOT            Run as an unprivileged user; read-only root filesystem where possible; drop capabilities.
NO SECRETS IN IMAGES  Inject at runtime (environment from Key Vault, mounted secrets, workload identity). Images are copied around and scanned; assume they leak.
SCAN, SBOM, SIGN    Vulnerability-scan (for example Trivy or Defender for Containers), produce an SBOM, sign images, and verify signatures at admission.
PIN EVERYTHING      Base image digests, package versions, model checksums.
RESOURCE AWARENESS  .NET respects container CPU and memory limits; set ONNX Runtime thread counts from the CPU limit, and size memory for the model plus request buffers (large image decodes, long prompts).
GRACEFUL SHUTDOWN   Handle SIGTERM: stop accepting new work, let in-flight inference finish (HostOptions.ShutdownTimeout), then exit. Align with Kubernetes terminationGracePeriodSeconds.
LOGS TO STDOUT      Structured logs and OpenTelemetry export — no local log files.
HEALTH ENDPOINTS    /health/live (process up) and /health/ready (model loaded) for orchestrators (Section 1).

# docker-compose.yml — the app plus the dependencies it needs, and an OpenTelemetry viewer
services:
  api:
    build: .
    ports: ["8080:8080"]
    environment:
      MODEL_PATH: /models/model.onnx
      OTEL_EXPORTER_OTLP_ENDPOINT: http://aspire-dashboard:18889
    volumes: ["./models:/models:ro"]
    depends_on: [postgres]

  postgres:
    image: pgvector/pgvector:pg17          # PostgreSQL with the pgvector extension
    environment: { POSTGRES_PASSWORD: dev-only-password }
    ports: ["5432:5432"]

  aspire-dashboard:                         # a lightweight OTLP viewer for traces, metrics, and logs during development
    image: mcr.microsoft.com/dotnet/aspire-dashboard:latest
    ports: ["18888:18888"]
    environment: { DASHBOARD__FRONTEND__AUTHMODE: Unsecured }     # local development ONLY

  # ollama:                                 # optional: a local model server for offline development
  #   image: ollama/ollama
  #   ports: ["11434:11434"]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vision-api
  labels: { app: vision-api }
spec:
  replicas: 2
  selector:
    matchLabels: { app: vision-api }
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }       # keep capacity during rollouts — GPUs are scarce, so surge only if a spare GPU exists
  template:
    metadata:
      labels:
        app: vision-api
        azure.workload.identity/use: "true"                 # AKS workload identity: keyless access to Azure services
    spec:
      serviceAccountName: vision-api
      terminationGracePeriodSeconds: 60                     # let in-flight inference finish
      tolerations:
        - { key: "sku", operator: "Equal", value: "gpu", effect: "NoSchedule" }     # allow scheduling onto tainted GPU nodes
      initContainers:
        - name: fetch-model                                  # download + verify the model before the app starts
          image: myregistry.azurecr.io/model-fetcher:1.2
          env:
            - { name: MODEL_URI, value: "https://mystorage.blob.core.windows.net/models/detector/1.3.0/model.onnx" }
            - { name: MODEL_SHA256, value: "<expected checksum>" }
          volumeMounts:
            - { name: models, mountPath: /models }
      containers:
        - name: api
          image: myregistry.azurecr.io/vision-api:2.4.1      # immutable tag or digest — never :latest
          ports:
            - containerPort: 8080
          env:
            - { name: MODEL_PATH, value: /models/model.onnx }
          resources:
            requests: { cpu: "2", memory: 4Gi }
            limits:   { memory: 6Gi, nvidia.com/gpu: 1 }     # the GPU is requested via the extended-resource limit
          startupProbe:                                       # tolerate a slow model load without being killed
            httpGet: { path: /health/live, port: 8080 }
            failureThreshold: 30
            periodSeconds: 10
          readinessProbe:                                     # only receive traffic once the model is loaded and warm
            httpGet: { path: /health/ready, port: 8080 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /health/live, port: 8080 }
            periodSeconds: 15
          volumeMounts:
            - { name: models, mountPath: /models, readOnly: true }
      volumes:
        - name: models
          emptyDir: {}

GPU SCHEDULING     The NVIDIA device plugin exposes nvidia.com/gpu as a schedulable resource. Use dedicated GPU node pools, TAINT them so only GPU workloads land there (and tolerate the taint in your pods),
                   and use node selectors/affinity for specific GPU types. GPUs aren't overcommitted: each requested GPU is exclusive unless you enable sharing.
GPU SHARING        Time-slicing, MPS, or MIG (on supported GPUs) split a GPU among pods — improving utilization for small models at the cost of isolation and predictability. Evaluate carefully.
PROBES             startupProbe (slow load), readinessProbe (model warm), livenessProbe (process alive and NOT dependent on the model). Wrong probes cause restart loops and traffic to cold pods.
MODEL DELIVERY     Init container download + checksum, or a shared read-only volume (Azure Files/Blob CSI), or node-local cache (Section 8). Avoid re-downloading gigabytes on every pod start.
IMAGE PULLS        GPU/CUDA images are large; use ACR with artifact streaming or pre-pulled/cached images on nodes to cut cold start.
IDENTITY & SECRETS Workload identity for keyless Azure access; Key Vault via the Secrets Store CSI driver; no keys in manifests.
NETWORK POLICY     Default-deny and explicit egress allow-lists — essential for agents and tools (see AI Security, Section 9).
DISRUPTION         PodDisruptionBudgets so node upgrades don't remove all replicas; topology spread constraints for zone resilience.
QUOTAS             ResourceQuota/LimitRange per namespace to stop one team from consuming all GPUs.
OBSERVABILITY      OpenTelemetry Collector (as a Deployment or DaemonSet), GPU metrics via the NVIDIA DCGM exporter into Prometheus/Azure Monitor, and request-level metrics from the app.

CPU-BASED HPA IS OFTEN THE WRONG SIGNAL. A GPU-bound or LLM-calling service can be saturated while CPU sits idle, and a service waiting on a remote model does little CPU work at all.
BETTER SIGNALS
  - Concurrent in-flight requests (or requests per second per pod) — exported as a custom metric
  - Queue depth / message backlog (for asynchronous workloads)
  - GPU utilization or GPU memory (from DCGM)
  - Latency percentiles / SLO burn (as a guard, not usually the primary scaler)
TOOLS
  - HPA with custom/external metrics (Prometheus adapter or Azure Monitor metrics)
  - KEDA — event-driven autoscaling from queues, Prometheus queries, and more; supports scale to ZERO
  - Cluster Autoscaler / node auto-provisioning to add (and remove) GPU nodes — but node provisioning takes minutes, so combine with warm capacity for latency-sensitive services

# KEDA: scale an embedding worker on queue backlog, down to zero when idle
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: embed-worker
spec:
  scaleTargetRef:
    name: embed-worker
  minReplicaCount: 0
  maxReplicaCount: 20
  cooldownPeriod: 300
  triggers:
    - type: azure-servicebus
      metadata:
        queueName: embed-jobs
        messageCount: "50"                      # target backlog per replica
      authenticationRef:
        name: servicebus-auth                   # or workload-identity-based authentication

JOBS / CRONJOBS   Offline inference, embedding refresh, ingestion, nightly canary evaluations, drift computation — run as Kubernetes Jobs/CronJobs (on cheaper or spot capacity where interruptions are acceptable),
                  with retries, idempotent processing, and results written to durable storage.
PIPELINES         Multi-step training/evaluation/registration workflows run in an orchestrator (Azure ML pipelines, Argo Workflows, Kubeflow) with artifact and lineage tracking.
SPOT / PREEMPTIBLE  Significant savings for interruptible batch and training work if you checkpoint and tolerate eviction; avoid for latency-sensitive serving unless you over-provision.

CUSTOM ASP.NET CORE SERVICE   Simplest for ONNX/ML.NET models you control (as in this series).
NVIDIA TRITON                 A multi-framework, multi-model inference server with dynamic batching, concurrent model execution, and metrics.
vLLM / TEXT-GENERATION SERVERS  High-throughput serving of open-weight LLMs with continuous batching and efficient KV-cache management.
KServe / AZURE ML ENDPOINTS   Standardized model-serving abstractions with autoscaling and canary traffic splitting.
KAITO (on AKS)                Automates provisioning of GPU nodes and deployment of open-weight models.
OLLAMA / LLAMA.CPP            Convenient for local development and small deployments, not usually the production-scale answer.

1. RESPONSE CACHE (EXACT)      Same inputs -> stored output. Skips the model entirely.
2. SEMANTIC CACHE              "Similar enough" questions share a stored answer, matched by embedding similarity.
3. PROVIDER PROMPT CACHE       The provider discounts and speeds up requests that begin with a recently processed prompt prefix.
4. EMBEDDING CACHE             Content hash -> vector. Avoids re-embedding unchanged text or repeated queries.
5. KV CACHE (self-hosted LLMs) The attention key/value cache inside the inference server, including prefix caching across requests.
6. MODEL ARTIFACT CACHE        Model files cached on nodes/volumes so pods don't re-download them; image layers cached; warm pools of loaded models.
7. IN-PROCESS MODEL CACHE      The loaded model session held in memory (singleton), or an LRU of several models when you serve many.
8. PREDICTION CACHE            For deterministic classical models: input features -> prediction.

// HybridCache (.NET 9+): a two-level cache (in-memory + distributed) with built-in stampede protection — concurrent callers for the same key trigger ONE model call
public sealed class SummaryService(HybridCache cache, IChatClient chat)
{
    public async Task<string> SummarizeAsync(string text, CancellationToken ct)
    {
        // The key must include EVERYTHING that changes the answer: prompt version, model, parameters, and the input hash
        string key = $"summary:{Prompts.SummaryVersion}:{Models.SummaryDeployment}:{Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(text)))}";

        return await cache.GetOrCreateAsync(
            key,
            async token => (await chat.GetResponseAsync($"Summarize in two sentences:\n\n{text}", new ChatOptions { Temperature = 0 }, token)).Text,
            new HybridCacheEntryOptions { Expiration = TimeSpan.FromHours(12), LocalCacheExpiration = TimeSpan.FromMinutes(5) },
            cancellationToken: ct);
    }
}

RULES FOR RESPONSE CACHING
  CACHE ONLY WHERE IT'S SAFE: deterministic-ish, non-personalized, not time-sensitive requests (temperature 0 summaries, classifications, FAQ answers). Never cache answers that depend on the user's
    permissions or live data unless the key includes the user/tenant and the freshness requirement is met.
  KEY EVERYTHING: prompt version, model/deployment, parameters, input hash, tenant (when relevant). A stale key serves old-prompt answers after a deployment.
  INVALIDATE: version the key on prompt/model/index changes; use TTLs; use cache tags to evict related entries.
  MIND PRIVACY: cached responses are stored data — apply the same access control, encryption, and retention as the source (see AI Security, Section 4).
  MEASURE: hit rate, saved tokens/cost, and staleness complaints (see AI Observability).

// Match by MEANING: "How do I reset my password?" and "password reset steps?" can share one answer
public sealed class SemanticCache(IEmbeddingGenerator<string, Embedding<float>> embeddings, ICacheVectorStore store)
{
    public async Task<string?> TryGetAsync(string question, string tenantId, string promptVersion, CancellationToken ct)
    {
        ReadOnlyMemory<float> vector = await embeddings.GenerateVectorAsync(question, cancellationToken: ct);

        // Scope the lookup: SAME tenant, SAME prompt version/model — otherwise users see each other's answers, or stale-prompt answers
        CacheHit? hit = await store.NearestAsync(vector, tenantId, promptVersion, ct);

        return hit is { Score: >= 0.95f } ? hit.Answer : null;        // a HIGH threshold, tuned on real traffic
    }
}

SEMANTIC CACHE HAZARDS
  FALSE HITS      "Cancel my order" and "Don't cancel my order" embed very close together (embeddings blur negation and numbers). A wrong cache hit is a confidently wrong answer. Use a high threshold, restrict to
                  low-risk, FAQ-style domains, and consider exact-match verification or a lightweight re-check.
  TENANT LEAKS    A cache keyed only on similarity serves one user's answer to another. Always scope by tenant/permissions (see AI Security, Section 4).
  STALENESS       Cached answers outlive the documents, prompts, or policies they were based on. Invalidate on corpus, prompt, and model changes, and keep TTLs short.
  PERSONALIZATION Don't semantically cache answers that depend on the individual user.
  COST/BENEFIT    Every lookup costs an embedding call; it pays off only with meaningful hit rates on repetitive traffic. Measure before and after.
Managed options exist (for example API Management's semantic caching policies backed by Redis) — the same caveats apply.

Providers can automatically discount and speed up requests whose beginning matches a recently processed prompt (typically applying above a minimum prompt length — check current thresholds and discounts).
TO BENEFIT:
  - Put STATIC content FIRST: system prompt, instructions, tool definitions, reference documents. Put VARIABLE content LAST: the user's question, retrieved passages that change per request.
  - Keep the static prefix BYTE-IDENTICAL across requests (no timestamps, request IDs, or reordered tools at the top).
  - Track the cached-token share (see AI Observability, Section 3): a drop means your prefix changed.

EMBEDDINGS   Key: hash(text) + embedding model/version. Embedding the same document chunk or popular query repeatedly is pure waste — Microsoft.Extensions.AI's UseDistributedCache middleware handles this for
             embedding generators (see Microsoft.Extensions.AI, Section 3). Include the model version in the key.
PREDICTIONS  For deterministic models (classifiers, scorers) a cache from input features to prediction works; invalidate on model version change (include the version in the key).

IN-PROCESS         Load once (singleton), warm up, reuse. Don't load per request.
MANY MODELS        Keep an LRU of loaded models bounded by memory; load on demand, evict the least recently used; protect against load stampedes (lock per model).
NODE-LOCAL CACHE   Download model artifacts to node-local disk (or a shared volume) once per node and reuse across pods and restarts, verifying checksums. Avoid every pod pulling gigabytes.
IMAGE LAYERS       Order Dockerfile layers so large, rarely changing layers (CUDA, native libraries) are cached; use registry features that stream or pre-pull images to nodes.
WARM CAPACITY      Keep a minimum number of warm replicas for latency-sensitive services; scale-to-zero trades cost for cold-start latency.
KV / PREFIX CACHE  For self-hosted LLM servers, enable prefix caching and size GPU memory for KV cache — it often determines concurrency more than model weights do.

LITTLE'S LAW:   concurrency = throughput x latency
  A service handling 50 requests/second at 0.4 s per request has ~20 requests in flight on average.
  If one replica can handle 5 concurrent requests before latency degrades, you need ~4 replicas at average load — plus headroom for peaks (commonly 30-50%) and for losing a replica.

For LLM calls to a hosted provider, also compute TOKEN throughput: requests/s x (input + output tokens) must fit your tokens-per-minute quota.
Always LOAD TEST: measure latency percentiles at increasing load on production-like hardware, find the knee where latency climbs, and set capacity and autoscaling targets below it.

1. DON'T DO THE WORK         Caching (Section 8), deduplication, cheaper routing — the fastest request is the one you skip.
2. MAKE EACH REQUEST CHEAPER Smaller or distilled models, quantization (FP16/INT8), shorter prompts and outputs, graph optimization, efficient runtimes (ONNX Runtime, TensorRT, OpenVINO, DirectML).
3. BATCH                     Process several requests per model invocation — raises throughput substantially on GPUs (and helps CPUs), at a latency cost (below).
4. SCALE OUT                 More stateless replicas behind a load balancer, autoscaled on the right signals (Section 7).
5. SCALE UP                  Bigger or faster hardware (more cores, a GPU, a larger GPU) when one model invocation itself is the bottleneck.
6. SHIFT WORK OFF THE REQUEST PATH  Queues and background workers for anything users aren't waiting on.

Inference services should be stateless (conversation and cache state live in external stores) so any replica can serve any request and replicas can be added or removed freely.

LOAD BALANCING GOTCHA: inference request durations vary enormously (a 50-token answer vs. a 2,000-token answer). Plain round-robin can pile long requests on one replica. Prefer
least-connections or queue-aware balancing, and cap per-replica concurrency so overload is rejected quickly instead of queueing indefinitely.

// A bounded queue between the API and the workers: when the system is saturated, say so (503 + Retry-After) instead of accepting unbounded work and failing slowly
public sealed class InferenceQueue
{
    private readonly Channel<InferenceJob> _channel = Channel.CreateBounded<InferenceJob>(new BoundedChannelOptions(200)
    {
        SingleReader = false, SingleWriter = false
    });

    public bool TryEnqueue(InferenceJob job) => _channel.Writer.TryWrite(job);           // returns false immediately when the queue is full
    public IAsyncEnumerable<InferenceJob> ReadAllAsync(CancellationToken ct) => _channel.Reader.ReadAllAsync(ct);
}

app.MapPost("/analyze", async (AnalyzeRequest req, InferenceQueue queue) =>
{
    var job = new InferenceJob(Guid.NewGuid(), req.ImageBytes);

    if (!queue.TryEnqueue(job))
        return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);               // shed load; the client retries after the Retry-After interval

    return Results.Accepted($"/results/{job.Id}", new { job.Id });                         // asynchronous pattern: poll /results/{id} or get a callback
});

ADMISSION CONTROL     Reject early when overloaded (429/503 with Retry-After), cap concurrent requests per replica, and give each request a deadline. Slow failure under overload is worse than fast failure.
ASYNC WORKFLOWS       For slow work (large documents, long video, big batches), accept the job, return an ID, process with background workers (KEDA-scaled on queue depth), and notify by polling, webhook, or
                      push. Make jobs idempotent and retries safe.

public sealed class MicroBatcher<TIn, TOut>(
    Func<IReadOnlyList<TIn>, Task<IReadOnlyList<TOut>>> runBatch, int maxBatchSize, TimeSpan maxWait)
{
    private readonly record struct Item(TIn Input, TaskCompletionSource<TOut> Completion);
    private readonly Channel<Item> _channel = Channel.CreateUnbounded<Item>();

    // Callers await this; the result arrives when the batch containing their item has run
    public Task<TOut> SubmitAsync(TIn input)
    {
        var tcs = new TaskCompletionSource<TOut>(TaskCreationOptions.RunContinuationsAsynchronously);
        _channel.Writer.TryWrite(new Item(input, tcs));
        return tcs.Task;
    }

    // Run this as a hosted background service
    public async Task RunAsync(CancellationToken ct)
    {
        ChannelReader<Item> reader = _channel.Reader;

        while (await reader.WaitToReadAsync(ct))
        {
            var batch = new List<Item>(maxBatchSize);

            // The first item has arrived: allow up to maxWait for more to join, or until the batch is full
            using var window = CancellationTokenSource.CreateLinkedTokenSource(ct);
            window.CancelAfter(maxWait);
            try
            {
                while (batch.Count < maxBatchSize && await reader.WaitToReadAsync(window.Token))
                    while (batch.Count < maxBatchSize && reader.TryRead(out Item item))
                        batch.Add(item);
            }
            catch (OperationCanceledException) when (!ct.IsCancellationRequested) { /* the wait window elapsed: run what we have */ }

            if (batch.Count == 0) continue;

            try
            {
                IReadOnlyList<TOut> outputs = await runBatch(batch.Select(b => b.Input).ToList());
                for (int i = 0; i < batch.Count; i++) batch[i].Completion.SetResult(outputs[i]);
            }
            catch (Exception ex)
            {
                foreach (Item item in batch) item.Completion.SetException(ex);       // one failed batch fails its callers, not the whole service
            }
        }
    }
}

THE TRADE-OFF: batching raises throughput but adds up to maxWait of latency per request. Tune maxBatchSize and maxWait against your latency SLO and measure throughput per dollar. Dedicated servers
(Triton's dynamic batching; vLLM's continuous batching for LLMs) implement this far more elaborately — use them for serious GPU serving, and the pattern above for simpler in-process cases.

QUANTIZATION      Reduce numeric precision (FP32 -> FP16 -> INT8, and lower for LLMs) to cut memory and increase speed, usually with a small accuracy cost — always re-evaluate quality after quantizing.
DISTILLATION      Train a small "student" model to mimic a large "teacher" — often the best accuracy-per-cost for a specific task.
PRUNING / SMALLER ARCHITECTURES   Remove redundancy or pick a more efficient architecture for the task.
GRAPH OPTIMIZATION  ONNX Runtime operator fusion and constant folding; hardware-specific runtimes (TensorRT for NVIDIA GPUs, OpenVINO for Intel CPUs, DirectML on Windows GPUs/NPUs).
SHORTER SEQUENCES Trim prompts and outputs; resize images to the smallest resolution that preserves accuracy.
RIGHT-SIZE THE MODEL  Use the smallest model that meets the quality bar — then route harder requests to a bigger one (see LLM Application Development, Section 9).

QUOTAS ARE THE CEILING     Your throughput is capped by tokens-per-minute and requests-per-minute quotas. Plan capacity from token estimates, not request counts.
MULTIPLE DEPLOYMENTS/REGIONS  Spread load across several deployments (and regions where data-residency allows) behind a gateway that load-balances, honors Retry-After, and trips circuit breakers on 429s and 5xx.
PROVISIONED + STANDARD     Reserved (provisioned) capacity for the steady baseline, with pay-as-you-go capacity as burst or fallback.
BATCH API FOR OFFLINE WORK Move non-interactive jobs off the real-time quota and onto discounted asynchronous batch processing.
RESILIENCE                 Retries with jittered backoff, timeouts, circuit breakers, and fallbacks (cheaper model, cached answer, degraded feature) — see AI APIs with .NET, Section 10.
CLIENT-SIDE LIMITS         Bound concurrency and token rates in your app so you degrade gracefully before the provider rejects you (see AI Security, Section 11).

ACTIVE-ACTIVE OR ACTIVE-PASSIVE across regions for the application, the index, and model endpoints, behind a global load balancer (Azure Front Door/Traffic Manager). Replicate indexes and artifacts;
keep model versions and prompts consistent across regions; test failover regularly; and account for data-residency constraints (not every region may be eligible for every tenant).

LARGE MODELS          Transformers, large vision models, and LLMs with many millions or billions of parameters.
HIGH, STEADY THROUGHPUT  Enough traffic to keep the GPU busy, especially with batching.
LATENCY-CRITICAL HEAVY MODELS   When a CPU simply can't meet the latency target for the model size.
BATCH EMBEDDING / BULK INFERENCE   Large offline jobs where throughput per dollar dominates.
SELF-HOSTED LLM SERVING   Practically requires GPUs (or accelerators) for useful speed at scale.

SMALL MODELS          Classical ML, gradient-boosted trees, small neural nets, small embedding/classification models (tens to low hundreds of millions of parameters, quantized).
LOW OR BURSTY TRAFFIC An idle GPU burns money; CPU instances scale to zero or run cheaply at low utilization.
BATCH SIZE OF ONE, LOW LATENCY   Small single-item inferences often run just as fast on CPU (no host-to-device transfer overhead).
OPERATIONAL SIMPLICITY No driver/CUDA/image-version matrix, easier scheduling and autoscaling, broader availability.
QUANTIZED MODELS      INT8/INT4 on modern CPUs (AVX2/AVX-512/VNNI instructions) can be surprisingly fast for the right models.
ARM CPUs              Often the best price/performance for CPU-bound inference where supported.

MODEL MEMORY (rule of thumb):  parameters x bytes per parameter
  7-billion-parameter model:   FP16 (2 bytes) ≈ 14 GB;  INT8 ≈ 7 GB;  4-bit ≈ 3.5-4 GB  — PLUS the KV cache and activations
KV CACHE grows with context length and concurrent sequences; for LLM serving it frequently limits concurrency more than the weights do.
A model that doesn't fit in GPU memory doesn't run (or must be split across GPUs/offloaded to CPU at a heavy speed penalty).
Quantization is therefore both a cost lever and a feasibility lever. For LLM decoding, speed is often limited by memory BANDWIDTH rather than raw compute.

Don't decide from spec sheets. Measure, for each candidate hardware option (and a few batch sizes):
  THROUGHPUT         items (or tokens) per second at saturation
  LATENCY            p50/p95/p99 at your expected concurrency
  COST PER 1,000 INFERENCES  = instance cost per hour / (throughput per hour / 1000), at realistic UTILIZATION (not 100%)
  MEMORY             peak usage; headroom for your largest inputs
  QUALITY            after quantization or precision changes — re-run your evaluation set
Then compare cost per inference at the utilization you will actually achieve. A GPU at 15% utilization can cost several times more per inference than a CPU fleet — and a CPU fleet that can't meet
your latency target is not an option.

PACKAGES      Microsoft.ML.OnnxRuntime (CPU), .Gpu (CUDA), .DirectML (Windows GPUs/NPUs). Use the package matching your hardware and keep CUDA/cuDNN versions aligned with it.
SILENT FALLBACK  If the CUDA libraries are missing or mismatched, ONNX Runtime may quietly run on the CPU — and you will pay GPU prices for CPU speed. VERIFY at startup and in tests:

// Fail fast (or at least alert loudly) if the GPU execution provider isn't actually available
string[] providers = OrtEnv.Instance().GetAvailableProviders();           // e.g. ["CUDAExecutionProvider", "CPUExecutionProvider"]

if (config.RequireGpu && !providers.Contains("CUDAExecutionProvider"))
    throw new InvalidOperationException($"GPU required but unavailable. Providers: {string.Join(", ", providers)}");

var options = new SessionOptions();
if (providers.Contains("CUDAExecutionProvider") && config.UseGpu)
    options.AppendExecutionProvider_CUDA(deviceId: 0);

log.LogInformation("Inference providers available: {Providers}; GPU enabled: {Gpu}", providers, config.UseGpu);

ALSO
  - Emit GPU utilization and memory metrics (DCGM exporter) and alert on persistently LOW utilization (wasted money) as well as saturation.
  - Pre/post-processing (image decode, resize, NMS) often runs on the CPU and can bottleneck a fast GPU — profile the whole pipeline, and parallelize or move steps to the GPU where it helps.
  - Keep a CPU-only code path and image for development, CI, and fallback.
  - Small language models can run locally with ONNX Runtime GenAI or similar runtimes on CPU/GPU/NPU — useful for edge and privacy scenarios.

 DEVELOPER ──> GitHub (code · prompts · IaC · eval sets · manifests)
                   │  pull request
                   v
        ┌── CI/CD (GitHub Actions, OIDC) ──────────────────────────────────────────────┐
        │  build · unit tests (fakes) · scans · EVAL + SECURITY GATES · image build/scan/sign · IaC what-if │
        └───────────────┬──────────────────────────────────────────────┬────────────────┘
                        v                                              v
              AZURE CONTAINER REGISTRY                          MODEL REGISTRY / BLOB (versioned, checksummed models)
                        │                                              │
                        v                                              v
   ┌──────────── RUNTIME (AKS or Container Apps) ───────────────────────────────────────────┐
   │  AI API (ASP.NET Core)  ──>  Redis (cache) · Azure AI Search / pgvector (retrieval)    │
   │      │                                                                                  │
   │      ├──> inference service(s) on GPU/CPU pools (ONNX/Triton) — init-container model fetch, warm readiness │
   │      └──> API MANAGEMENT (AI GATEWAY): auth · token limits · metrics · semantic cache · failover          │
   │                    │                                                                    │
   │                    v                                                                    │
   │            AZURE OPENAI deployments (pinned versions; provisioned + standard; multi-region)             │
   │  Workers (KEDA on queues): ingestion · embedding · batch inference · scheduled canary evaluations        │
   └──────────────────────┬──────────────────────────────────────────────────────────────────┘
                          v
        OBSERVABILITY: OpenTelemetry → Collector → Azure Monitor / Prometheus / Grafana
        (traces · token/cost/latency metrics · drift · quality scores)  →  alerts · SLOs · dashboards
        Key Vault + managed/workload identity everywhere · private endpoints · Azure Policy

MONITOR (drift, quality, cost, latency)  ->  ALERT / REVIEW  ->  DIAGNOSE (trace-level debugging)  ->  FIX (prompt, index, model, data, code)
   ^                                                                                                         │
   └── PROMOTE ← CANARY ← EVALUATE (golden set + recent production samples + security suite) ← VERSIONED CHANGE ┘
Every production failure becomes a test case; every model/prompt/index change is versioned, evaluated, rolled out gradually, and reversible.

RELIABILITY
  [ ] SLOs defined (availability, latency incl. TTFT, quality, cost) with error budgets and burn-rate alerts
  [ ] Load-tested at expected peak + headroom on production-like hardware; capacity and autoscaling targets set below the latency knee
  [ ] Timeouts, retries with jittered backoff, circuit breakers, and fallbacks (cheaper model / cached answer / degraded mode) implemented and tested
  [ ] Backpressure and load shedding (bounded queues, 429/503 with Retry-After), per-replica concurrency caps
  [ ] Readiness/liveness/startup probes correct (liveness independent of the model); graceful shutdown drains in-flight work
  [ ] Multi-zone (and where needed multi-region) deployment; failover tested; PodDisruptionBudgets in place
  [ ] Provider quotas sized from token estimates; quota-increase requests filed; multiple deployments/regions behind a gateway

VERSIONING AND RELEASES
  [ ] Models, tokenizers/labels, prompts, indexes, tools, eval sets, and IaC all versioned; versions recorded on every trace
  [ ] Hosted-model versions pinned; retirement dates tracked; migration plan with evaluation
  [ ] Model checksums verified at load; registry with promotion stages and audit trail
  [ ] Canary/shadow/blue-green rollout with automatic rollback criteria defined BEFORE release; previous version always deployable

QUALITY AND EVALUATION
  [ ] Golden set (with real failures and adversarial cases) run as a CI gate; baselines stored; repetitions to handle non-determinism
  [ ] Scheduled canary evaluation in production; model-version-change alerts; drift monitoring on inputs and predictions
  [ ] Ground-truth pipeline (human review, feedback, delayed labels) feeding performance measurement and retraining

SECURITY AND PRIVACY (see AI Security)
  [ ] Keyless/managed identity; no secrets in images, prompts, clients, or logs; secret scanning in CI
  [ ] Private networking; model access through a gateway; server-side model/deployment selection and content policies
  [ ] Authorization at the data layer; tenant isolation tests; adversarial suite as a release gate
  [ ] Image scanning, SBOM, signing; non-root containers; network policies with egress allow-lists
  [ ] Telemetry privacy: metadata-only by default, redaction, retention and deletion procedures

COST
  [ ] Cost attribution by feature/tenant/model; budgets, burn-rate and anomaly alerts; provider-side spend limits
  [ ] Caching strategy with correct keys and invalidation; routing to right-sized models; batch API for offline work
  [ ] GPU utilization monitored (alert on idle waste); hardware choice justified by cost per inference at real utilization

OPERATIONS
  [ ] Dashboards for ops, cost, quality, and security; alerts with owners and runbooks; on-call rotation
  [ ] Kill switches (per feature/tool/model) and credential-rotation runbooks; incident process and postmortems
  [ ] Documented architecture, data flows, dependencies, and DR plan; regular game days