Image / video frame
  ↓
Decode + preprocess        (ImageSharp / SkiaSharp / OpenCV: resize, normalize, orient)
  ↓
Model                      (cloud API  |  ONNX Runtime  |  ML.NET  |  multimodal LLM)
  ↓
Postprocess                (softmax, boxes + non-max suppression, text assembly, thresholds)
  ↓
Business logic             (store, search, flag for review, trigger a workflow)

CLASSIFICATION      What is in this image?                      "cat", "defective", "invoice"      one label (or a few) per image
OBJECT DETECTION    What objects are where?                     boxes + labels + confidence         "3 cars at these coordinates"
SEGMENTATION        Which pixels belong to which object?        per-pixel masks                     medical imaging, background removal
OCR                 What text is in this image?                 text + positions                    receipts, IDs, signs, scanned PDFs
DOCUMENT ANALYSIS   Structure and fields of a document          tables, key-value pairs             invoices, forms
FACE TASKS          Detect, verify, identify faces              boxes, embeddings, match scores     (heavily regulated — Section 5)
EMBEDDINGS          A vector capturing the image's meaning      float[ ]                            visual search, dedupe, recommendations
CAPTIONING / VQA    Describe or answer questions about image    natural language                    accessibility, search, assistants

1. MANAGED CLOUD API       Azure AI Vision (Image Analysis, Read OCR), Azure AI Document Intelligence, Azure AI Face.
                           + No ML expertise, fast to build, maintained and improved for you.
                           - Per-call cost, data leaves your boundary (unless containers), limited customization,
                             vendor features change and retire.

2. PRE-TRAINED MODEL ON    Download an ONNX model (ResNet, MobileNet, YOLO, CLIP, ArcFace...) and run it in-process
   ONNX RUNTIME            with Microsoft.ML.OnnxRuntime.
                           + Full control, no per-call fee, data stays local, runs offline/at the edge, can use GPU.
                           - You own preprocessing/postprocessing, model hosting, performance, and licensing checks.

3. TRAIN YOUR OWN          Fine-tune a model on your labeled images — ML.NET transfer learning for classification, or
                           train in PyTorch/TensorFlow/Azure ML and export to ONNX.
                           + Best accuracy on your specific domain (defect types, your products).
                           - Needs labeled data, evaluation discipline, and MLOps.

4. MULTIMODAL LLM          Send the image to a vision-capable chat model (see Section 10).
                           + Extremely flexible, zero-shot, handles messy real-world images and open-ended questions.
                           - Cost, latency, non-determinism, hallucination risk, weaker at precise localization and
                             counting, and data-privacy considerations.

Microsoft.ML.OnnxRuntime (+ .Gpu / .DirectML)   Run ONNX models in-process (CPU, CUDA, DirectML)
Microsoft.ML + Microsoft.ML.Vision + Microsoft.ML.ImageAnalytics   ML.NET image classification via transfer learning
SixLabors.ImageSharp (+ .Drawing)               Managed, cross-platform image processing
SkiaSharp                                        Fast 2D graphics and image processing (Skia)
OpenCvSharp4 (+ runtime packages)               OpenCV bindings: classic CV, video, DNN
Azure.AI.Vision.ImageAnalysis                   Azure AI Vision Image Analysis 4.0 (captions, tags, objects, read OCR, people)
Azure.AI.DocumentIntelligence                   Document OCR, layout, and prebuilt/custom extractors
Tesseract                                        Open-source OCR engine wrapper
Microsoft.Extensions.AI                          Vision-capable chat models via IChatClient (Section 10)
System.Numerics.Tensors                          Vector math (cosine similarity) for embeddings

"Read the text/fields from documents"                 -> Azure AI Document Intelligence (or Vision Read for plain OCR)
"Describe / tag / caption general photos"             -> Azure AI Vision Image Analysis, or a multimodal LLM
"Detect common objects (people, cars, ...)"           -> Azure AI Vision Objects/People, or a YOLO ONNX model
"Classify MY specific categories (defects, SKUs)"     -> ML.NET transfer learning or a fine-tuned model exported to ONNX
"Find similar images / search images by text"         -> image embeddings (CLIP-style) + vector search
"Identify people by face"                             -> stop and read Section 5 first (legal, ethical, and access constraints)
"Process / resize / crop / draw on images"            -> ImageSharp, SkiaSharp, or OpenCvSharp

// dotnet add package Microsoft.ML.OnnxRuntime
// dotnet add package SixLabors.ImageSharp
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;

public sealed class ImageClassifier : IDisposable
{
    private readonly InferenceSession _session;                  // create ONCE and reuse — loading a model is expensive
    private readonly string _inputName;
    private readonly string[] _labels;

    // ImageNet normalization used by many torchvision-trained models — MUST match YOUR model's training
    private static readonly float[] Mean = [0.485f, 0.456f, 0.406f];
    private static readonly float[] Std  = [0.229f, 0.224f, 0.225f];

    public ImageClassifier(string modelPath, string labelsPath)
    {
        _session = new InferenceSession(modelPath);
        _inputName = _session.InputMetadata.Keys.First();
        _labels = File.ReadAllLines(labelsPath);
    }

    public IReadOnlyList<(string Label, float Probability)> Classify(Image<Rgb24> image, int topK = 3)
    {
        // 1. PREPROCESS: resize + center-crop to 224x224, scale to 0..1, normalize, order as NCHW (batch, channel, height, width)
        using var resized = image.Clone(x => x.Resize(new ResizeOptions { Size = new Size(224, 224), Mode = ResizeMode.Crop }));

        var tensor = new DenseTensor<float>([1, 3, 224, 224]);
        resized.ProcessPixelRows(accessor =>
        {
            for (int y = 0; y < accessor.Height; y++)
            {
                Span<Rgb24> row = accessor.GetRowSpan(y);
                for (int x = 0; x < row.Length; x++)
                {
                    tensor[0, 0, y, x] = (row[x].R / 255f - Mean[0]) / Std[0];
                    tensor[0, 1, y, x] = (row[x].G / 255f - Mean[1]) / Std[1];
                    tensor[0, 2, y, x] = (row[x].B / 255f - Mean[2]) / Std[2];
                }
            }
        });

        // 2. INFERENCE
        using var results = _session.Run([NamedOnnxValue.CreateFromTensor(_inputName, tensor)]);
        float[] logits = results.First().AsEnumerable<float>().ToArray();

        // 3. POSTPROCESS: softmax turns raw scores (logits) into probabilities that sum to 1
        float max = logits.Max();
        float[] exp = logits.Select(l => MathF.Exp(l - max)).ToArray();     // subtract max for numerical stability
        float sum = exp.Sum();

        return exp.Select((e, i) => (Label: _labels[i], Probability: e / sum))
                  .OrderByDescending(x => x.Probability)
                  .Take(topK)
                  .ToList();
    }

    public void Dispose() => _session.Dispose();
}

THE PREPROCESSING MUST MATCH TRAINING. Check your model's documentation for:
  - INPUT SIZE        (224x224, 256x256, 384x384 ...)
  - CHANNEL ORDER     RGB vs BGR (OpenCV loads BGR; most deep-learning models expect RGB)
  - LAYOUT            NCHW (batch, channels, height, width) vs NHWC
  - SCALING           0..1 vs 0..255 vs -1..1
  - NORMALIZATION     per-channel mean/std (or none)
  - RESIZE POLICY     stretch vs. preserve aspect ratio with crop or padding
A mismatch doesn't throw an error — it just silently degrades accuracy. This is the #1 cause of "the model works in Python
but gives garbage in .NET."

// dotnet add package Microsoft.ML
// dotnet add package Microsoft.ML.Vision
// dotnet add package Microsoft.ML.ImageAnalytics
// dotnet add package SciSharp.TensorFlow.Redist        (the TensorFlow runtime ML.NET's vision trainer uses)
using Microsoft.ML;
using Microsoft.ML.Vision;

public class ImageData
{
    public string ImagePath { get; set; } = "";
    public string Label { get; set; } = "";              // e.g. the name of the folder the image sits in
}

public class ImagePrediction
{
    public string PredictedLabel { get; set; } = "";
    public float[] Score { get; set; } = [];
}

var mlContext = new MLContext(seed: 1);

// Folder layout:  images/<label>/<file>.jpg   — one subfolder per class
IEnumerable<ImageData> images = Directory.EnumerateFiles("images", "*.jpg", SearchOption.AllDirectories)
    .Select(path => new ImageData { ImagePath = path, Label = Directory.GetParent(path)!.Name });

IDataView data = mlContext.Data.ShuffleRows(mlContext.Data.LoadFromEnumerable(images));

// Convert the string label to a numeric "key" and load the raw image bytes
var preprocess = mlContext.Transforms.Conversion.MapValueToKey("LabelAsKey", nameof(ImageData.Label))
    .Append(mlContext.Transforms.LoadRawImageBytes("Image", null, nameof(ImageData.ImagePath)));

IDataView processed = preprocess.Fit(data).Transform(data);

var split = mlContext.Data.TrainTestSplit(processed, testFraction: 0.3, seed: 1);
var validation = mlContext.Data.TrainTestSplit(split.TestSet, testFraction: 0.5, seed: 1);
IDataView trainSet = split.TrainSet, validationSet = validation.TrainSet, testSet = validation.TestSet;

var options = new ImageClassificationTrainer.Options
{
    FeatureColumnName = "Image",
    LabelColumnName = "LabelAsKey",
    ValidationSet = validationSet,
    Arch = ImageClassificationTrainer.Architecture.ResnetV2101,      // the pre-trained network to build on
    MetricsCallback = Console.WriteLine,                             // progress and validation accuracy per epoch
    TestOnTrainSet = false,
    ReuseTrainSetBottleneckCachedValues = true,                      // cache the expensive feature extraction between epochs
    ReuseValidationSetBottleneckCachedValues = true,
};

var pipeline = mlContext.MulticlassClassification.Trainers.ImageClassification(options)
    .Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel"));

ITransformer model = pipeline.Fit(trainSet);

// Evaluate on images the model never trained on (see AI/ML Fundamentals, Section 8)
var predictions = model.Transform(testSet);
var metrics = mlContext.MulticlassClassification.Evaluate(predictions, labelColumnName: "LabelAsKey", predictedLabelColumnName: "PredictedLabel");
Console.WriteLine($"Micro-accuracy: {metrics.MicroAccuracy:P1}   Macro-accuracy: {metrics.MacroAccuracy:P1}");

mlContext.Model.Save(model, trainSet.Schema, "imageClassifier.zip");

CONFIDENCE THRESHOLDS     A classifier always returns SOME label, even for an image unlike anything it has seen. Apply a threshold
                          and add an "unknown / needs human review" path — don't act on a 41% prediction as if it were certain.
CLASS IMBALANCE           If 95% of images are "OK," accuracy is misleading (see AI/ML Fundamentals, Section 9). Track precision, recall,
                          and per-class results, and use a confusion matrix.
DATA QUALITY OVER MODEL   Consistent labels and images representative of PRODUCTION conditions (lighting, angle, camera, backgrounds) matter
                          more than the architecture. Models trained on clean studio photos fail on blurry phone snapshots.
SPLIT BY SOURCE           Near-duplicate images (frames of the same video, photos of the same object) must stay on one side of the
                          train/test split, or you'll measure memorization.
AUGMENTATION              Flips, crops, brightness/contrast changes, and small rotations expand your data and improve robustness.
MULTI-LABEL               If an image can have several labels at once, you need a multi-label setup (independent probabilities), not softmax.

Output of a detector for one image:
  [ { label: "person", confidence: 0.91, box: (x=120, y=64, w=88, h=210) },
    { label: "car",    confidence: 0.84, box: (x=410, y=190, w=260, h=130) }, ... ]

using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;
using RectangleF = System.Drawing.RectangleF;       // avoid ambiguity with ImageSharp's own RectangleF

public record Detection(string Label, float Confidence, RectangleF Box);

public sealed class ObjectDetector : IDisposable
{
    private const int InputSize = 640;
    private readonly InferenceSession _session;
    private readonly string _inputName;
    private readonly string[] _labels;

    public ObjectDetector(string modelPath, string[] labels)
    {
        _session = new InferenceSession(modelPath);
        _inputName = _session.InputMetadata.Keys.First();
        _labels = labels;
    }

    public IReadOnlyList<Detection> Detect(Image<Rgb24> original, float confThreshold = 0.25f, float iouThreshold = 0.45f)
    {
        // 1. LETTERBOX: scale to fit 640x640 while PRESERVING aspect ratio, padding the rest with gray
        float scale = Math.Min((float)InputSize / original.Width, (float)InputSize / original.Height);
        int newW = (int)(original.Width * scale), newH = (int)(original.Height * scale);
        int padX = (InputSize - newW) / 2, padY = (InputSize - newH) / 2;

        using var resized = original.Clone(x => x.Resize(newW, newH));

        var input = new DenseTensor<float>([1, 3, InputSize, InputSize]);
        input.Fill(114f / 255f);                                           // padding color
        resized.ProcessPixelRows(accessor =>
        {
            for (int y = 0; y < accessor.Height; y++)
            {
                Span<Rgb24> row = accessor.GetRowSpan(y);
                for (int x = 0; x < row.Length; x++)
                {
                    input[0, 0, y + padY, x + padX] = row[x].R / 255f;     // YOLO-style models typically want RGB scaled 0..1
                    input[0, 1, y + padY, x + padX] = row[x].G / 255f;
                    input[0, 2, y + padY, x + padX] = row[x].B / 255f;
                }
            }
        });

        // 2. INFERENCE
        using var results = _session.Run([NamedOnnxValue.CreateFromTensor(_inputName, input)]);
        Tensor<float> output = results.First().AsTensor<float>();         // YOLOv8-style layout: [1, 4 + numClasses, numAnchors]

        int numClasses = output.Dimensions[1] - 4;
        int numAnchors = output.Dimensions[2];

        // 3. POSTPROCESS: decode candidate boxes
        var candidates = new List<Detection>();
        for (int i = 0; i < numAnchors; i++)
        {
            int bestClass = 0; float bestScore = 0;
            for (int c = 0; c < numClasses; c++)
            {
                float score = output[0, 4 + c, i];
                if (score > bestScore) { bestScore = score; bestClass = c; }
            }
            if (bestScore < confThreshold) continue;

            float cx = output[0, 0, i], cy = output[0, 1, i], w = output[0, 2, i], h = output[0, 3, i];

            // Undo the letterbox: model coordinates -> original image coordinates
            float x1 = (cx - w / 2 - padX) / scale;
            float y1 = (cy - h / 2 - padY) / scale;
            candidates.Add(new Detection(_labels[bestClass], bestScore, new RectangleF(x1, y1, w / scale, h / scale)));
        }

        // 4. NON-MAXIMUM SUPPRESSION: remove duplicate overlapping boxes for the same object
        return NonMaxSuppression(candidates, iouThreshold);
    }

    private static List<Detection> NonMaxSuppression(List<Detection> detections, float iouThreshold)
    {
        var kept = new List<Detection>();

        foreach (var group in detections.GroupBy(d => d.Label))           // per-class NMS
        {
            var remaining = group.OrderByDescending(d => d.Confidence).ToList();
            while (remaining.Count > 0)
            {
                Detection best = remaining[0];
                kept.Add(best);
                remaining.RemoveAt(0);
                remaining.RemoveAll(d => IoU(best.Box, d.Box) > iouThreshold);
            }
        }
        return kept;
    }

    private static float IoU(RectangleF a, RectangleF b)
    {
        RectangleF inter = RectangleF.Intersect(a, b);
        float interArea = inter.IsEmpty ? 0 : inter.Width * inter.Height;
        float union = a.Width * a.Height + b.Width * b.Height - interArea;
        return union <= 0 ? 0 : interArea / union;
    }

    public void Dispose() => _session.Dispose();
}

BOUNDING BOX        A rectangle: (x, y, width, height) or (x1, y1, x2, y2). Models may output center-based (cx, cy, w, h) —
                    convert carefully.
CONFIDENCE          The model's score for the detection; filter by a threshold. Lower = more detections and more false
                    positives; higher = fewer false positives but more misses.
IoU                 "Intersection over Union": the overlap between two boxes divided by their combined area (0 = no overlap,
                    1 = identical). Used both for NMS and for evaluating detections against ground truth.
NMS                 Detectors emit many overlapping boxes per object. NMS keeps the most confident box and discards others
                    that overlap it above the IoU threshold.
LETTERBOXING        Preserving aspect ratio (padding) instead of stretching — stretched images distort object shapes and hurt
                    accuracy, and you MUST map boxes back to the original image's coordinates afterward.

OUTPUT LAYOUT   Differs by model family and export settings: some outputs include a separate "objectness" score, some are transposed
                ([1, numAnchors, 4 + classes]), and some exports bake NMS into the model. Inspect the output shape
                (session.OutputMetadata) and the model's documentation — the code above matches the YOLOv8-style layout.
LABELS          The class-name list must match the model's training (80 COCO classes for common pretrained checkpoints).
LICENSE         Check the license of the model and its weights before commercial use — some popular detectors use licenses with
                obligations (such as copyleft terms) that matter for products.
PERFORMANCE     Indexing a Tensor per element is convenient but slow; for throughput, copy to a float[] and index with arithmetic.
                Use GPU execution providers (CUDA, DirectML) and smaller or quantized (INT8) models where latency matters.

AZURE AI VISION    The Image Analysis API returns detected objects, people, and tags with bounding boxes — no model to host,
                   good for common categories (Section 7).
CUSTOM DETECTORS   For YOUR objects (a specific part, a defect type), fine-tune a detector on labeled images (label with a tool like
                   CVAT or Label Studio), train in PyTorch or Azure ML, export to ONNX, and run with the code above.
ML.NET             Offers object detection via Model Builder and Azure-based training workflows; check current documentation for
                   what it supports.

COUNTING            Count boxes per class — but beware occlusion, tiny objects, and duplicates; validate counts against ground truth.
CROPPING            Crop each box and send it to a second model (a classifier, OCR, or a face model) — a common multi-stage pipeline.
REGION RULES        "Alert when a person is inside this polygon" — geometry on top of detections.
TRACKING (VIDEO)    Detection works per frame. To follow an object across frames you need a tracking layer (e.g. SORT/ByteTrack-style
                    algorithms) that associates detections over time.
THRESHOLDS PER CLASS  Tune confidence per class — a safety-critical class may deserve a lower threshold (accepting false alarms)
                    than a convenience feature.

AZURE AI VISION READ        Cloud OCR for printed and handwritten text in photos and documents, many languages. Part of the
                            Image Analysis 4.0 API. Good for general "get me the text" needs.

AZURE AI DOCUMENT           Document-focused: layout (paragraphs, tables, selection marks), prebuilt models (invoices, receipts,
INTELLIGENCE                IDs, business cards, tax forms) that return structured FIELDS, and custom models trained on your
                            forms. The right tool for documents rather than photos.

TESSERACT                   Open-source engine, runs locally. Free and offline, but accuracy depends heavily on image quality and
                            preprocessing; weaker on photos, handwriting, and complex layouts.

OTHER LOCAL ENGINES         PaddleOCR (via .NET wrappers) and platform OCR (such as Windows' built-in OCR) are options with
                            different accuracy, language, and portability trade-offs.

MULTIMODAL LLMs             Vision-capable chat models can read text and extract structured fields from messy images
                            (Section 10) — flexible, but slower and costlier, and they can hallucinate text.

// dotnet add package Azure.AI.Vision.ImageAnalysis
using Azure;
using Azure.AI.Vision.ImageAnalysis;

var client = new ImageAnalysisClient(new Uri(endpoint), new AzureKeyCredential(key));     // or an Entra ID TokenCredential

ImageAnalysisResult result = await client.AnalyzeAsync(
    BinaryData.FromBytes(imageBytes),               // or pass a Uri to an image the service can fetch
    VisualFeatures.Read);

foreach (DetectedTextBlock block in result.Read.Blocks)
    foreach (DetectedTextLine line in block.Lines)
    {
        Console.WriteLine(line.Text);

        foreach (DetectedTextWord word in line.Words)
            Console.WriteLine($"   {word.Text}  (confidence {word.Confidence:P0})");
    }

// dotnet add package Azure.AI.DocumentIntelligence
using Azure;
using Azure.AI.DocumentIntelligence;

var docClient = new DocumentIntelligenceClient(new Uri(endpoint), new AzureKeyCredential(key));

// "prebuilt-layout" returns text, tables, and structure; "prebuilt-invoice", "prebuilt-receipt", "prebuilt-idDocument"... return typed fields
Operation<AnalyzeResult> operation = await docClient.AnalyzeDocumentAsync(
    WaitUntil.Completed, "prebuilt-invoice", BinaryData.FromBytes(pdfOrImageBytes));

AnalyzeResult analysis = operation.Value;

foreach (AnalyzedDocument doc in analysis.Documents)
{
    if (doc.Fields.TryGetValue("VendorName", out DocumentField? vendor))
        Console.WriteLine($"Vendor: {vendor.ValueString}  (confidence {vendor.Confidence:P0})");

    if (doc.Fields.TryGetValue("InvoiceTotal", out DocumentField? total))
        Console.WriteLine($"Total: {total.ValueCurrency?.Amount}");
}

// dotnet add package Tesseract     (also download the trained-data files, e.g. eng.traineddata, into ./tessdata)
using Tesseract;

using var engine = new TesseractEngine(@"./tessdata", "eng", EngineMode.Default);
using var pix = Pix.LoadFromFile("scan.png");
using var page = engine.Process(pix);

Console.WriteLine(page.GetText());
Console.WriteLine($"Mean confidence: {page.GetMeanConfidence():P0}");

RESOLUTION     Aim for roughly 300 DPI equivalent for documents; upscale tiny text before OCR.
CONTRAST       Convert to grayscale and apply adaptive thresholding (binarization) to separate text from uneven backgrounds.
DESKEW         Rotate skewed scans so lines are horizontal. Correct orientation (EXIF rotation, upside-down pages).
DENOISE        Remove speckle and scanner noise; avoid blurring text itself.
CROP           Remove borders, shadows, and non-text regions.
LIGHTING       Phone photos often have shadows and glare — normalize illumination.

// Typical OpenCvSharp cleanup before OCR (Section 8)
using OpenCvSharp;

using var src  = Cv2.ImRead("receipt.jpg", ImreadModes.Color);
using var gray = new Mat();
Cv2.CvtColor(src, gray, ColorConversionCodes.BGR2GRAY);

using var denoised = new Mat();
Cv2.GaussianBlur(gray, denoised, new Size(3, 3), 0);

using var binary = new Mat();
Cv2.AdaptiveThreshold(denoised, binary, 255, AdaptiveThresholdTypes.GaussianC, ThresholdTypes.Binary, blockSize: 31, c: 15);

Cv2.ImWrite("receipt-clean.png", binary);

CONFIDENCE         Use word/line/field confidence to route low-confidence results to human review rather than trusting them blindly.
VALIDATE           Cross-check extracted values: totals equal the sum of line items; dates parse; account numbers pass checksums;
                   VAT numbers match a pattern. Validation catches OCR errors that confidence scores miss.
STRUCTURE          For tables and forms use a layout-aware service rather than reconstructing columns from raw text.
LANGUAGES / SCRIPTS  Verify support for your languages and scripts, including right-to-left text and handwriting.
RAG PIPELINES      OCR is the first stage of ingesting scanned PDFs for retrieval (see this series' RAG guide, Section 3): detect low-text PDF
                   pages and route them through OCR.
SENSITIVE DATA     IDs, medical forms, and financial documents are sensitive. Control where images are sent, how long they're retained, and who
                   can see the extracted text (see the security notes in Section 11).

static double CharacterErrorRate(string reference, string hypothesis)
{
    if (reference.Length == 0) return hypothesis.Length == 0 ? 0 : 1;
    return (double)Levenshtein(reference, hypothesis) / reference.Length;
}

static int Levenshtein(string a, string b)
{
    var previous = Enumerable.Range(0, b.Length + 1).ToArray();
    for (int i = 1; i <= a.Length; i++)
    {
        var current = new int[b.Length + 1];
        current[0] = i;
        for (int j = 1; j <= b.Length; j++)
            current[j] = Math.Min(Math.Min(current[j - 1] + 1, previous[j] + 1), previous[j - 1] + (a[i - 1] == b[j - 1] ? 0 : 1));
        previous = current;
    }
    return previous[b.Length];
}

FACE DETECTION      "Is there a face, and where?" — boxes (and optionally landmarks such as eyes and nose). The least sensitive
                    task; used for auto-cropping, blurring, camera focus, and counting.

FACE VERIFICATION   "Are these two images the same person?" (1:1). Used for unlocking a device or matching a selfie to an ID photo.

FACE IDENTIFICATION "Who is this, among a set of known people?" (1:N). Searches a gallery of enrolled faces. The most sensitive task.

FACE ATTRIBUTES     Inferring age, gender, emotion, or other traits from a face. Scientifically contested and discontinued or
                    restricted by major providers (below). Avoid.

LAW AND REGULATION     Biometric identification is treated as sensitive or special-category data under laws such as the GDPR, and some
                       jurisdictions have dedicated biometric statutes (for example Illinois' BIPA, with consent, retention, and
                       private-right-of-action provisions). The EU AI Act restricts or prohibits certain biometric uses — for example
                       real-time remote biometric identification in public spaces by law enforcement (with narrow exceptions), untargeted
                       scraping of facial images to build databases, and emotion recognition in workplaces and schools. Rules differ by
                       country and are evolving: consult legal counsel before building anything that identifies people by face.

CONSENT AND PURPOSE    Obtain informed consent where required, use faces only for the stated purpose, and don't repurpose them.

DATA PROTECTION        Store face embeddings/templates as sensitive data: encryption, strict access control, minimal retention,
                       deletion on request, and a clear policy for what happens when a person withdraws consent.

BIAS AND ACCURACY      Face recognition accuracy has been shown to vary across demographic groups and imaging conditions. A system that
                       works well on average can fail badly — and unfairly — on specific groups. Evaluate by subgroup, set conservative
                       thresholds, and never make high-stakes decisions on a face match alone.

PLATFORM POLICIES      Microsoft's Azure Face service applies a Limited Access policy: identification and verification features require an
                       approved application and use case, and the earlier capabilities for inferring attributes like emotion, gender, and age
                       were retired. Other vendors have their own restrictions. Read the current terms.

ALTERNATIVES           Often a less invasive design meets the need: a badge or PIN plus a face DETECTION step for privacy blurring, 1:1
                       verification against a document the user provides, or on-device matching where templates never leave the user's device.

1. DETECT    Find faces in the image (a face detector returns boxes and landmarks).
2. ALIGN     Crop and rotate so eyes and mouth sit in standard positions — recognition models expect aligned faces.
3. EMBED     A face-recognition network (ArcFace/FaceNet-style, available as ONNX models) converts each aligned face to a vector
             (commonly 128-512 numbers) such that the same person's photos land close together.
4. COMPARE   Verification: cosine similarity (or distance) between two embeddings vs. a threshold.
             Identification: nearest-neighbor search over a gallery of enrolled embeddings (vector search — see this series'
             Embeddings & Vector Search guide), then apply the threshold to the best match.
5. DECIDE    A threshold trades false accepts against false rejects (below). Add liveness detection to defeat photos and replays.

// Verification with embeddings from a face-recognition ONNX model (embedding extraction code is analogous to the classifier above)
float[] a = faceEmbedder.Embed(alignedFaceA);
float[] b = faceEmbedder.Embed(alignedFaceB);

float similarity = TensorPrimitives.CosineSimilarity(a, b);
bool samePerson = similarity >= threshold;       // choose the threshold from a labeled validation set, NOT from a blog post

FALSE ACCEPT RATE (FAR)   Different people judged to be the same — the SECURITY failure.
FALSE REJECT RATE (FRR)   The same person judged different — the USABILITY failure.
The threshold trades one for the other. Pick it on a labeled validation set representative of your users and conditions, to meet a
target FAR for your risk level, and re-measure by subgroup.

LIVENESS / ANTI-SPOOFING  Without it, a printed photo or a screen replay can fool recognition. Use liveness checks for any
                          authentication use.

// Classic detector via OpenCvSharp — simple and dependency-light, but less accurate than modern deep-learning detectors
using OpenCvSharp;

using var cascade = new CascadeClassifier("haarcascade_frontalface_default.xml");
using var image = Cv2.ImRead("group.jpg");
using var gray = new Mat();
Cv2.CvtColor(image, gray, ColorConversionCodes.BGR2GRAY);

Rect[] faces = cascade.DetectMultiScale(gray, scaleFactor: 1.1, minNeighbors: 5, flags: HaarDetectionTypes.ScaleImage, minSize: new Size(40, 40));

foreach (Rect face in faces)
    Cv2.Rectangle(image, face, Scalar.Red, 2);        // or: blur the region to anonymize people in an image

CATEGORY RECOGNITION    "That is a laptop" — classification/detection (Sections 2-3).
INSTANCE RECOGNITION    "That is THIS exact product (SKU 4471)" or "this is a photo of the Eiffel Tower" — distinguishing specific instances.
                        The modern approach is EMBEDDING-BASED: embed a reference image of each known item, embed the query image, and
                        find the nearest reference via vector search (Section 6). Adding a new product means adding an embedding — no retraining.

IMAGE → vector     image embedding model
TEXT  → vector     text embedding model (the PAIRED model of the SAME family)
Compare with cosine similarity: text-to-image search, image-to-image search, zero-shot classification.

AZURE AI VISION MULTIMODAL EMBEDDINGS   A managed API that embeds images AND text into a shared space (REST endpoints for vectorizing images and text).
                                        No model to host.

CLIP-STYLE MODELS VIA ONNX              Open CLIP-family models exported to ONNX run locally with ONNX Runtime — an image encoder and a text
                                        encoder (the text side needs the matching tokenizer, which is the fiddly part).

VISION BACKBONES                        Features from image models such as ResNet or DINO-style networks work well for image-to-image similarity
                                        (not text search) — export the model so its feature (pre-classification) layer is exposed.

MULTIMODAL LLM PROVIDERS                Some providers offer multimodal embedding endpoints; check availability.

public interface IImageEmbedder
{
    Task<float[]> EmbedImageAsync(byte[] imageBytes, CancellationToken ct = default);
    Task<float[]> EmbedTextAsync(string text, CancellationToken ct = default);       // only meaningful for multimodal (CLIP-style) models
}

public sealed class AzureVisionEmbedder(HttpClient http, string endpoint, string key) : IImageEmbedder
{
    // Illustrative: confirm the route, api-version, and model-version in the current Azure AI Vision documentation
    public async Task<float[]> EmbedImageAsync(byte[] imageBytes, CancellationToken ct = default)
    {
        using var request = new HttpRequestMessage(HttpMethod.Post,
            $"{endpoint}/computervision/retrieval:vectorizeImage?api-version=2024-02-01&model-version=2023-04-15")
        {
            Content = new ByteArrayContent(imageBytes) { Headers = { ContentType = new("application/octet-stream") } }
        };
        request.Headers.Add("Ocp-Apim-Subscription-Key", key);

        using var response = await http.SendAsync(request, ct);
        response.EnsureSuccessStatusCode();

        var json = await response.Content.ReadFromJsonAsync<EmbeddingResponse>(cancellationToken: ct);
        return json!.Vector;
    }

    public async Task<float[]> EmbedTextAsync(string text, CancellationToken ct = default) { /* the vectorizeText counterpart */ throw new NotImplementedException(); }

    private record EmbeddingResponse(float[] Vector);
}

// 1. INDEX: embed every catalog image once and store the vector (with the image id and metadata) in a vector store
foreach (var product in catalog)
    await store.UpsertAsync(product.Id, await embedder.EmbedImageAsync(product.ImageBytes), product.Metadata);

// 2. SEARCH BY IMAGE: "find products that look like this photo"
float[] queryVector = await embedder.EmbedImageAsync(uploadedPhotoBytes);
var similar = await store.SearchAsync(queryVector, top: 10, filter: tenantFilter);

// 3. SEARCH BY TEXT (multimodal models only): "red leather handbag"
float[] textVector = await embedder.EmbedTextAsync("red leather handbag");
var matches = await store.SearchAsync(textVector, top: 10, filter: tenantFilter);

// 4. ZERO-SHOT CLASSIFICATION: compare the image to text prompts for each candidate label — no training data
string[] labels = ["cat", "dog", "bird"];
var labelVectors = await Task.WhenAll(labels.Select(l => embedder.EmbedTextAsync($"a photo of a {l}")));
float[] imageVector = await embedder.EmbedImageAsync(imageBytes);
int best = Enumerable.Range(0, labels.Length).MaxBy(i => TensorPrimitives.CosineSimilarity(imageVector, labelVectors[i]));

USE CASES
  Visual search ("shop the look"), reverse image search          Duplicate and near-duplicate detection (moderation, DAM systems)
  Recommendations ("more like this")                              Instance recognition (match a photo to a known product)
  Clustering/organizing photo collections                         Zero-shot classification and tagging
  Retrieval for multimodal RAG (find relevant diagrams or figures)

SAME MODEL for the index and the queries — different models' vectors are not comparable. Record the model name/version with stored vectors; changing
  models means re-embedding everything.
NORMALIZE / USE COSINE per the model's guidance; choose the vector index metric to match.
CACHE embeddings by content hash — re-embedding unchanged images is pure waste.
FILTER BY METADATA and permissions INSIDE the vector query (tenant, category, date) — see the vector search guide.
EVALUATE retrieval with real queries and labeled relevance (Recall@k), not by eyeballing a few results.
LIMITS: embeddings capture overall semantic/visual similarity, not fine detail. They can miss small text in an image, exact counts, or subtle defects —
  pair them with OCR or detection when those matter.
PRIVACY: embeddings derived from sensitive images (faces, medical, documents) are sensitive data.

AZURE AI VISION — IMAGE ANALYSIS 4.0   General image understanding: CAPTIONS (a sentence describing the image), DENSE CAPTIONS (regions),
                                       TAGS, OBJECT DETECTION, PEOPLE DETECTION, SMART CROPS, and READ (OCR). One API, choose features per call.

AZURE AI VISION — MULTIMODAL           Image and text embeddings for vector search (Section 6).
EMBEDDINGS

AZURE AI DOCUMENT INTELLIGENCE         Document OCR, layout, prebuilt extractors (invoice, receipt, ID, ...) and custom models (Section 4).

AZURE AI FACE                          Face detection and, for approved use cases, verification/identification (Section 5; Limited Access applies).

AZURE OPENAI (VISION-CAPABLE MODELS)   Multimodal chat models that understand images (Section 10).

AZURE AI CONTENT SAFETY                Detects harmful content in images and text — useful for moderating user uploads.

CONTAINERS                             Some Vision and Document Intelligence capabilities are available as containers you can run in your own
                                       environment for data-residency or disconnected scenarios (check which features and licensing apply).

// dotnet add package Azure.AI.Vision.ImageAnalysis
using Azure;
using Azure.AI.Vision.ImageAnalysis;

var client = new ImageAnalysisClient(new Uri(endpoint), new AzureKeyCredential(key));

ImageAnalysisResult result = await client.AnalyzeAsync(
    BinaryData.FromBytes(imageBytes),
    VisualFeatures.Caption | VisualFeatures.Tags | VisualFeatures.Objects | VisualFeatures.People | VisualFeatures.Read,
    new ImageAnalysisOptions { GenderNeutralCaption = true, Language = "en" });

Console.WriteLine($"Caption: {result.Caption.Text} ({result.Caption.Confidence:P0})");

foreach (DetectedTag tag in result.Tags.Values)
    Console.WriteLine($"Tag: {tag.Name} ({tag.Confidence:P0})");

foreach (DetectedObject obj in result.Objects.Values)
    Console.WriteLine($"Object: {obj.Tags[0].Name} ({obj.Tags[0].Confidence:P0}) at X={obj.BoundingBox.X}, Y={obj.BoundingBox.Y}, W={obj.BoundingBox.Width}, H={obj.BoundingBox.Height}");

foreach (DetectedPerson person in result.People.Values.Where(p => p.Confidence > 0.6))
    Console.WriteLine($"Person at {person.BoundingBox.X},{person.BoundingBox.Y} ({person.Confidence:P0})");

AUTH            Prefer Microsoft Entra ID (DefaultAzureCredential, managed identity) over API keys; if you use keys, keep them in Key Vault.
                (The SDK clients accept either.)
NETWORKING      Private endpoints and virtual-network restrictions are available for enterprise deployments.
LIMITS          Image size and dimension limits, supported formats, and requests-per-second quotas apply — resize before upload, and handle
                HTTP 429 with backoff (see AI APIs with .NET, Sections 10-11). The SDK clients include retry policies for transient errors.
COST            Billed per transaction (per feature, per image). Cache results by image hash, don't re-analyze unchanged images, and request only
                needed features.
DATA            Understand what the service retains and where it processes data; images can be sensitive. Use containers or private networking if policy requires.

public interface IImageAnalyzer
{
    Task<ImageSummary> AnalyzeAsync(byte[] image, CancellationToken ct);
}

public record ImageSummary(string Caption, IReadOnlyList<string> Tags, IReadOnlyList<(string Label, double Confidence, int X, int Y, int W, int H)> Objects, string Text);

dotnet add package OpenCvSharp4
dotnet add package OpenCvSharp4.runtime.win          # native OpenCV binaries — choose the runtime package for YOUR platform:
                                                     #   .runtime.win, .runtime.ubuntu.20.04-x64, .runtime.linux-arm, .runtime.osx ...
dotnet add package OpenCvSharp4.Extensions            # conversions to/from System.Drawing bitmaps (Windows-oriented)

using OpenCvSharp;

// Mat is OpenCV's image/matrix type. It owns NATIVE memory that the .NET garbage collector cannot see —
// ALWAYS dispose it (using), especially in loops and servers, or you will leak memory.
using Mat image = Cv2.ImRead("photo.jpg", ImreadModes.Color);       // loads as BGR (not RGB!)
using Mat gray  = new();
Cv2.CvtColor(image, gray, ColorConversionCodes.BGR2GRAY);

using Mat blurred = new();
Cv2.GaussianBlur(gray, blurred, new Size(5, 5), 0);

using Mat edges = new();
Cv2.Canny(blurred, edges, threshold1: 50, threshold2: 150);

Cv2.ImWrite("edges.png", edges);

OpenCV stores color images as BLUE-GREEN-RED, while most deep-learning models (and most other imaging libraries, including ImageSharp and
SkiaSharp) use RGB. Feed an unconverted BGR image to an RGB-trained model and accuracy silently drops. Convert with
Cv2.CvtColor(src, dst, ColorConversionCodes.BGR2RGB) at the boundary.

Cv2.Resize(image, resized, new Size(640, 480));                                   // resize
Cv2.Rotate(image, rotated, RotateFlags.Rotate90Clockwise);                         // rotate
using var roi = new Mat(image, new Rect(100, 50, 200, 200));                       // crop (a view into the same data)

// Find contours (shapes) in a binary image
Cv2.FindContours(binary, out Point[][] contours, out HierarchyIndex[] hierarchy, RetrievalModes.External, ContourApproximationModes.ApproxSimple);
foreach (var contour in contours.Where(c => Cv2.ContourArea(c) > 500))
    Cv2.Rectangle(image, Cv2.BoundingRect(contour), Scalar.Lime, 2);

// Draw and label detections from Section 3
foreach (var d in detections)
{
    var rect = new Rect((int)d.Box.X, (int)d.Box.Y, (int)d.Box.Width, (int)d.Box.Height);
    Cv2.Rectangle(image, rect, Scalar.Red, 2);
    Cv2.PutText(image, $"{d.Label} {d.Confidence:P0}", new Point(rect.X, Math.Max(15, rect.Y - 5)),
                HersheyFonts.HersheySimplex, 0.6, Scalar.Red, 2);
}

// Encode/decode in memory (no temp files) — the way to move images over HTTP or through queues
Cv2.ImEncode(".jpg", image, out byte[] jpegBytes);
using Mat decoded = Cv2.ImDecode(jpegBytes, ImreadModes.Color);

// To feed a neural-network input tensor from a Mat, copy to a managed array and convert BGR -> RGB deliberately (not by accident)

using var capture = new VideoCapture("traffic.mp4");          // a file path, an RTSP/HTTP URL, or a camera index (0)
using var frame = new Mat();

int frameIndex = 0;
while (capture.Read(frame) && !frame.Empty())
{
    // Detection is expensive: run it every Nth frame (and track in between) instead of every frame
    if (frameIndex++ % 5 == 0)
    {
        // convert and run a detector (Section 3) ...
    }
}

VIDEO PRACTICALITIES
  - Decode on a worker thread; keep a small bounded queue between capture, inference, and output so a slow model drops frames instead of
    accumulating latency and memory.
  - Process every Nth frame or at reduced resolution for real-time budgets.
  - Use hardware decoding and GPU inference when throughput matters.
  - Reconnect automatically for network streams; they drop.
  - Watch resource use: each frame is a large native allocation.

using OpenCvSharp.Dnn;

using Net net = CvDnn.ReadNetFromOnnx("model.onnx");
using Mat blob = CvDnn.BlobFromImage(image, scaleFactor: 1 / 255.0, size: new Size(640, 640), mean: new Scalar(), swapRB: true, crop: false);
net.SetInput(blob);
using Mat output = net.Forward();

VIDEO AND CAMERAS            capture, decode, encode, streams
GEOMETRIC / CLASSICAL CV     perspective correction, deskewing, contour and shape analysis, feature matching, calibration, background subtraction
PREPROCESSING FOR OCR        grayscale, thresholding, denoising, deskew (Section 4)
FAST NATIVE IMAGE OPS        heavy per-pixel work where native performance matters
Prefer a managed library (Section 9) for simple resizing, format conversion, and drawing in web back ends — fewer native dependencies to deploy.

SixLabors.ImageSharp      Fully managed, cross-platform, modern API, no native dependencies, good for web servers and containers.
                          Drawing lives in the separate ImageSharp.Drawing package. NOTE: it uses a "split license" — free for open-source and
                          qualifying small organizations, but commercial use may require a paid license; check the current terms.

SkiaSharp                 Bindings to Google's Skia graphics engine; fast, widely used (MAUI, Avalonia), strong 2D drawing and text rendering. Native
                          binaries per platform; permissive MIT license.

System.Drawing.Common     GDI+ based. On .NET 6+ it is supported on WINDOWS ONLY. Don't use it for cross-platform or Linux-container services.

Magick.NET                Wrapper over ImageMagick: enormous format support and many operations; native dependencies; Apache-style license.

NetVips                   Wrapper over libvips: very fast and memory-efficient for large images and pipelines; native dependency.

OpenCvSharp               Computer vision and video, with image I/O and drawing (Section 8). BGR channel order.

Windows imaging APIs      WIC/Windows.Graphics.Imaging for Windows-only apps.

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.Formats.Jpeg;
using SixLabors.ImageSharp.Processing;

// Load -> fix orientation -> resize (never enlarge) -> strip metadata -> save
using Image image = await Image.LoadAsync(inputStream);

image.Mutate(x => x
    .AutoOrient()                                                                  // apply the EXIF rotation phones store instead of rotating pixels
    .Resize(new ResizeOptions { Size = new Size(1600, 1600), Mode = ResizeMode.Max }));   // fit within 1600x1600, keep aspect ratio

image.Metadata.ExifProfile = null;                                                 // remove EXIF (GPS location, device info) before storing or sharing

await image.SaveAsJpegAsync(outputStream, new JpegEncoder { Quality = 85 });

EXIF ORIENTATION   Phone photos are often stored sideways with an orientation flag. If you ignore it, the model sees a rotated image (and OCR fails).
                   Apply orientation (AutoOrient) BEFORE inference.

EXIF PRIVACY       Photos carry GPS coordinates, timestamps, and device details. Strip metadata before storing or publishing images you didn't create.

COLOR ORDER        RGB (ImageSharp, SkiaSharp, most models) vs BGR (OpenCV). Convert deliberately.

COLOR SPACES       Some images are CMYK, grayscale, 16-bit, or have alpha. Convert to a known format (e.g. Rgb24) before building tensors.

RESIZE QUALITY     Choose a good resampler (bicubic/Lanczos) for downscaling; the choice can measurably affect model accuracy, and it should match
                   training if the model is sensitive.

MEMORY             A 50-megapixel image decodes to ~150 MB of pixels. Cap dimensions BEFORE full decode (read just the header with Image.Identify) and
                   resize early.

THREAD SAFETY      Image/Mat objects are not thread-safe; don't share one across concurrent requests.

FORMAT SUPPORT     JPEG, PNG, WebP, GIF, BMP, TIFF are common; HEIC (iPhone) and RAW formats often need extra libraries — plan for them if users upload from phones.

// SkiaSharp: draw labeled boxes on a decoded bitmap
using SkiaSharp;

using SKBitmap bitmap = SKBitmap.Decode(imageBytes);
using var canvas = new SKCanvas(bitmap);
using var boxPaint = new SKPaint { Color = SKColors.Red, Style = SKPaintStyle.Stroke, StrokeWidth = 3 };

foreach (var d in detections)
    canvas.DrawRect(d.Box.X, d.Box.Y, d.Box.Width, d.Box.Height, boxPaint);

using var image = SKImage.FromBitmap(bitmap);
using var data = image.Encode(SKEncodedImageFormat.Jpeg, 85);
await File.WriteAllBytesAsync("annotated.jpg", data.ToArray());

// Read dimensions from the header WITHOUT decoding the whole image, and reject abusive inputs early
ImageInfo info = await Image.IdentifyAsync(stream);                  // throws on unrecognized formats
if (info.Width > 10_000 || info.Height > 10_000 || (long)info.Width * info.Height > 50_000_000)
    return Results.BadRequest("Image dimensions are too large.");
stream.Position = 0;

Treat image uploads like any untrusted file: enforce a maximum byte size (and request-body limits); verify the actual format from content, not the file extension
or Content-Type header; limit decoded dimensions (decompression bombs — a tiny file that expands to gigabytes of pixels); process in a resource-limited
context; re-encode to a safe format rather than storing originals blindly; and scan for malware where policy requires. Keep image-decoding libraries updated —
image codecs are a historically common source of vulnerabilities.

using Microsoft.Extensions.AI;

var message = new ChatMessage(ChatRole.User,
[
    new TextContent("Extract the vendor name, invoice number, date, and total from this invoice. Respond as JSON."),
    new DataContent(imageBytes, "image/jpeg"),
]);

ChatResponse response = await chatClient.GetResponseAsync([message]);

// Or get a typed result directly
public record InvoiceFields(string Vendor, string InvoiceNumber, string Date, decimal Total);
ChatResponse<InvoiceFields> typed = await chatClient.GetResponseAsync<InvoiceFields>([message]);
InvoiceFields invoice = typed.Result;                          // validate it — shape is guaranteed, truth is not

OPEN-ENDED QUESTIONS     "Is anything unsafe in this photo?" "Describe what happened in this screenshot." — no model training or fixed label set.
ZERO-SHOT FLEXIBILITY    New categories or extraction schemas by changing the prompt, not retraining.
MESSY DOCUMENTS          Irregular layouts, handwriting, receipts photographed at angles, charts and diagrams.
REASONING OVER IMAGES    Combining visual content with text instructions and context.
RAPID PROTOTYPING        Proof of concept in an afternoon.

HALLUCINATION         They can confidently invent text, numbers, or objects that aren't in the image. For OCR-like extraction this is dangerous: a plausible but
                      wrong total looks exactly like a right one.
PRECISE LOCALIZATION  Bounding boxes, pixel-accurate positions, and exact counting of many objects are unreliable compared with dedicated detectors.
CONSISTENCY           Non-deterministic; the same image can yield different outputs.
COST AND LATENCY      Each image consumes many tokens and takes seconds — painful at video frame rates or millions of images.
THROUGHPUT            Rate limits and per-call pricing don't suit high-volume, real-time pipelines.
PRIVACY               Images are sent to a hosted model; sensitive content (faces, IDs, medical images) raises data-protection questions.
LIMITS                Image size and count per request, and resolution that may downscale small text.

Classical + LLM, each doing what it's best at:
  - OCR / Document Intelligence extracts the text and fields WITH confidence; an LLM interprets or normalizes them — or only handles the low-confidence cases.
  - A detector finds and crops regions; an LLM describes or reads each crop.
  - A cheap classifier screens images; only the uncertain ones go to the expensive multimodal model.
  - An LLM proposes; deterministic code VALIDATES (totals add up, dates parse, values within ranges).

// Program.cs
builder.Services.AddSingleton(sp => new ObjectDetector("models/yolov8n.onnx", CocoLabels.All));   // load the model ONCE

builder.Services.AddRateLimiter(o => o.AddPolicy("vision", ctx => RateLimitPartition.GetFixedWindowLimiter(
    ctx.User.Identity?.Name ?? "anonymous", _ => new FixedWindowRateLimiterOptions { PermitLimit = 30, Window = TimeSpan.FromMinutes(1) })));

builder.WebHost.ConfigureKestrel(k => k.Limits.MaxRequestBodySize = 10_000_000);                   // cap upload size

var inference = new SemaphoreSlim(Environment.ProcessorCount);                                     // bound concurrent CPU-heavy inference

app.MapPost("/detect", async (IFormFile file, ObjectDetector detector, CancellationToken ct) =>
{
    if (file.Length is 0 or > 10_000_000) return Results.BadRequest("Image must be between 1 byte and 10 MB.");

    await using Stream stream = file.OpenReadStream();

    ImageInfo info;
    try { info = await Image.IdentifyAsync(stream, ct); }                                          // verify it's a real image; read dimensions cheaply
    catch (Exception) { return Results.BadRequest("Unsupported or corrupt image."); }

    if ((long)info.Width * info.Height > 50_000_000) return Results.BadRequest("Image dimensions are too large.");
    stream.Position = 0;

    using Image<Rgb24> image = await Image.LoadAsync<Rgb24>(stream, ct);
    image.Mutate(x => x.AutoOrient());                                                             // honor EXIF rotation BEFORE inference

    await inference.WaitAsync(ct);
    try
    {
        var detections = await Task.Run(() => detector.Detect(image), ct);                         // keep the request thread free; bound the work
        return Results.Ok(detections.Select(d => new { d.Label, d.Confidence, d.Box.X, d.Box.Y, d.Box.Width, d.Box.Height }));
    }
    finally { inference.Release(); }
})
.RequireAuthorization()
.RequireRateLimiting("vision")
.DisableAntiforgery();                                                                             // an API endpoint, not a browser form post

LOAD MODELS ONCE            Register the inference session as a singleton; loading a model per request is slow and memory-hungry. InferenceSession.Run
                            is safe to call concurrently; image objects are not shared.

WARM UP                     The first inference is slower (graph optimization, allocations). Run a dummy inference at startup so users don't pay for it.

BOUND CONCURRENCY           CPU/GPU inference is resource-intensive; limit parallel work (SemaphoreSlim) and queue the rest rather than thrashing.

ASYNC AND BACKGROUND        For slow or bulk jobs (video, large batches, heavy OCR), accept the upload, enqueue a job, return an ID, and let a background worker process it
                            (BackgroundService / queue / Azure Functions), with status polling or a webhook/SignalR notification.

GPU                         Microsoft.ML.OnnxRuntime.Gpu (CUDA) or .DirectML (Windows) can accelerate inference substantially; build and test GPU container images
                            deliberately (drivers, CUDA/cuDNN versions).

MODEL SIZE VS. SPEED        Smaller or quantized models (INT8) and lower input resolutions cut latency and memory at some accuracy cost; measure the trade-off.

BATCHING                    Batching several images into one inference call raises GPU throughput at the cost of per-request latency.

CACHE                       Key results by a hash of the image bytes (plus model version) — repeated images are common (duplicates, retries, shared files).

MODEL MANAGEMENT            Version models like code; store them in blob storage or the image, record the model version with each result, and use a rolling
                            rollout so you can compare and roll back.

OBSERVABILITY               Log latency, input sizes, confidence distributions, and error rates; export traces/metrics (OpenTelemetry). Drift in confidence
                            distributions is an early warning that production images have changed.

DOCKER                      Include native dependencies (OpenCV runtime packages, ONNX Runtime native libs) and test the actual image; keep images small.

UNTRUSTED UPLOADS       Size/dimension limits, format verification, resource-limited processing, updated codecs (Section 9).
AUTHORIZATION           Users may only analyze and retrieve THEIR images/results; scope storage and queries per user/tenant.
SENSITIVE CONTENT       Images may contain faces, IDs, medical or financial documents. Minimize retention, encrypt at rest and in transit, restrict who and what
                        can see them, and be explicit about when images leave your environment (cloud APIs, LLMs).
EXIF AND METADATA       Strip location and device data before storing or sharing.
MODERATION              If users upload content others will see, screen it (for example with Azure AI Content Safety).
ADVERSARIAL INPUTS      Models can be fooled by crafted images (adversarial examples); don't rely on vision output alone for security-critical decisions. Images can
                        also carry hidden instructions aimed at multimodal LLMs.
COSTS AND ABUSE         Rate limit, cap sizes, and authenticate — an open image-analysis endpoint (especially one calling a paid API) is a cost-abuse target.

CLASSIFICATION    Accuracy (careful with imbalance), precision, recall, F1, per-class results, confusion matrix, calibration of confidence scores.
                  (See AI/ML Fundamentals, Section 9.)

OBJECT DETECTION  IoU between predicted and true boxes; a detection is correct if the label matches and IoU exceeds a threshold (commonly 0.5).
                    PRECISION = correct detections / all detections;   RECALL = correct detections / all true objects.
                    AP (average precision) per class summarizes the precision-recall curve; mAP averages across classes (and sometimes IoU thresholds).
                  Also check counting error and per-size performance (small objects are hardest).

OCR               Character error rate (CER) and word error rate (WER) for text; FIELD-LEVEL accuracy and per-field confidence calibration for extraction;
                  end-to-end "is the final record correct?" rate.

FACE VERIFICATION FAR and FRR at the chosen threshold, ROC curves, and results BROKEN DOWN BY DEMOGRAPHIC GROUP and imaging condition.

EMBEDDINGS /      Recall@k and MRR on labeled query-to-relevant-image pairs (see Embeddings & Vector Search and RAG, Section 12).
VISUAL SEARCH

LLM VISION        Task-specific accuracy against ground truth, plus hallucination checks (does the extracted value actually appear in the image?),
                  consistency across repeated runs, and cost/latency.

REPRESENTATIVE TEST SET    Collect images from the REAL production environment: your cameras, lighting, devices, angles, resolutions, and edge cases (blur, glare,
                           occlusion, odd orientations). Models validated on clean benchmark photos routinely disappoint in the field.
SPLIT BY SOURCE            Keep near-duplicates (frames from one video, shots of the same object, pages of one document) on one side of the split.
CONSISTENT LABELS          Ambiguous or noisy labels cap achievable accuracy. Write labeling guidelines, measure inter-annotator agreement, and review disputed cases.
STRATIFY AND SLICE         Report results by condition (day/night, indoor/outdoor, device type), by class, and by object size — averages hide failures.
FAIRNESS                   Where people appear, evaluate across demographic groups; large gaps are a defect even when the average looks good.
NEGATIVE EXAMPLES          Include images with NONE of the target objects, and out-of-distribution images, to measure false positives and unknown handling.
FROZEN TEST SET            Don't tune on it. Keep a held-out set untouched until the end, and refresh it as the world changes.

CONFIDENCE DISTRIBUTIONS   Track them over time; shifts signal that production images no longer match what the model was validated on.
HUMAN REVIEW QUEUE         Route low-confidence or high-stakes results to people; use their corrections as new labeled data.
SAMPLED AUDIT              Regularly review random samples of automated decisions, not just the flagged ones.
FEEDBACK LOOP              Capture errors and overrides, label them, and fold them into the next training and evaluation set.
VERSIONED REGRESSION SUITE Re-run your evaluation set whenever you change models, thresholds, preprocessing, or a provider's model version.
COST AND LATENCY BUDGETS   Track p50/p95 latency and per-image cost alongside accuracy.

START WITH THE REQUIREMENTS, NOT THE MODEL
  What decision does this feed? What does a mistake cost? Is it real-time? How many images per day? Do images contain sensitive data? Must it run offline or
  inside a boundary? Are the categories common or specific to you?

THEN WALK THE LADDER, STOPPING AS EARLY AS POSSIBLE
  1. A managed API handles it (Azure AI Vision, Document Intelligence)        -> build in days; pay per call
  2. A multimodal LLM handles it well enough                                  -> flexible and fast to build; check cost, latency, and hallucination risk
  3. A pre-trained ONNX model handles it                                      -> run locally; no per-call fee; you own pre/postprocessing
  4. You need custom accuracy on your own data                                -> labeled dataset, transfer learning or fine-tuning, evaluation, MLOps

THEN HARDEN
  Evaluate on realistic images, set thresholds and a human-review path, add validation, secure the pipeline, monitor drift, and plan for model and service lifecycle changes.