The LLM engineering stack, from the inside out:

  TEXT ──> TOKENIZER ──> EMBEDDINGS + POSITIONS ──> [ TRANSFORMER BLOCK x N ] ──> LOGITS ──> SAMPLER ──> next token ──┐
                                                       │  self-attention (with KV cache)                              │
                                                       │  feed-forward                                                │
                                                       └───────────────────────────── repeat, token by token <───────┘

  And around it: quantization (shrink it) · distillation (compress its skill) · LoRA (adapt it cheaply) · RAG (give it knowledge) ·
  routing (pick the right model) · fallback (survive failures) · multimodal inputs (see more than text).

ENCODER-ONLY (BERT-style)    Each token attends to ALL tokens (bidirectional). Produces rich representations; great for classification, embeddings, NER, reranking. Doesn't generate text.
DECODER-ONLY (GPT-style)     Each token attends only to PREVIOUS tokens (causal mask). Trained to predict the next token; generates text autoregressively. The architecture of today's chat LLMs.
ENCODER-DECODER (T5-style)   An encoder reads the input; a decoder generates output while attending to the encoder. Common in translation and some summarization systems.

 input text
   │  tokenizer
   v
 token IDs  [1042, 318, 257, ...]
   │  embedding table (vocab_size x d_model)
   v
 token vectors  + positional information (Section 3)
   │
   v
 ┌─────────────────────  TRANSFORMER BLOCK  (repeated N times, e.g. 32 – 120)  ─────────────────────┐
 │   x ──> Normalize ──> MULTI-HEAD SELF-ATTENTION (causal) ──> + residual ──┐                        │
 │   ┌─────────────────────────────────────────────────────────────────────┘                        │
 │   └──> Normalize ──> FEED-FORWARD NETWORK (MLP) ──────────────────────> + residual ──> out        │
 └─────────────────────────────────────────────────────────────────────────────────────────────────┘
   │
   v
 final normalization ──> LM head (linear layer, d_model → vocab_size) ──> LOGITS (one score per vocabulary token)
   │  softmax (with temperature, top-k, top-p — Section 7)
   v
 next-token probabilities ──> sample ──> append to the sequence ──> repeat

EMBEDDINGS         A learned table mapping each token ID to a vector of size d_model (for example 4,096). The LM head often reuses (ties) the same matrix.
RESIDUAL CONNECTIONS  Each sub-layer's output is ADDED to its input (x + f(x)), letting gradients flow and letting layers refine rather than replace the representation.
NORMALIZATION      LayerNorm or RMSNorm stabilizes activations. Modern models usually normalize BEFORE each sub-layer ("pre-norm"), which trains more stably at depth.
ATTENTION          Mixes information ACROSS tokens (Section 2).
FEED-FORWARD (MLP) Transforms each token's vector INDEPENDENTLY: expand to a larger hidden size, apply a nonlinearity (modern models often use gated variants such as SwiGLU), project back.
                   The MLP holds a large share of the parameters (roughly two-thirds in classic designs) and much of the model's "knowledge."
LM HEAD            Projects the final vector to one score (logit) per vocabulary entry.

Per block (classic design, MLP expansion of 4x):
  attention projections (Q, K, V, output):   4 x d²
  feed-forward (two matrices, d -> 4d -> d): 8 x d²
  ≈ 12 x d² parameters per block

Total ≈ 12 x layers x d²  (+ embeddings: vocab x d)

Example (GPT-3-scale): 96 layers, d = 12,288  ->  12 x 96 x 12,288² ≈ 174 billion  (published figure: ~175B).

In an MoE model the single feed-forward network in a block is replaced by many "expert" networks plus a small ROUTER that sends each token to just a few of them (for example 2 of 64).
  + Total parameters (knowledge capacity) can be huge while ACTIVE parameters per token — and therefore compute per token — stay modest.
  - ALL experts must still be held in memory; routing adds engineering complexity; serving is harder (expert parallelism, load balancing).
When you see "47B total / 13B active," that's MoE: memory cost follows total parameters, compute cost follows active parameters.

1. PRETRAINING      Predict the next token over trillions of tokens of text/code. Produces a "base" model with broad knowledge but no instruction-following manners. Enormously expensive.
2. SUPERVISED       Fine-tune on curated (instruction, ideal response) examples so the model follows instructions and uses chat formats.
   FINE-TUNING (SFT)
3. PREFERENCE       Align behavior with human or AI preferences (RLHF — reinforcement learning from human feedback; DPO and related methods optimize preferences more directly): helpfulness, honesty,
   TUNING           safety, tone.
4. REASONING / RL   Newer "reasoning" models add reinforcement learning on verifiable tasks to produce extended step-by-step reasoning before answering — trading latency and cost for accuracy.
Application developers almost never pretrain; they use base-to-chat models as-is, adapt them cheaply (Sections 10-11), or call hosted models.

- The model is a NEXT-TOKEN PREDICTOR; everything else (reasoning, tool use, JSON) is behavior learned or elicited on top. That's why validation, grounding, and constraints matter.
- Cost scales with PARAMETERS (memory) and with TOKENS processed (compute), which is why prompt length, output length, and model size drive your bill and latency.
- The context window is bounded by how the architecture handles position and by the memory cost of attention (Sections 3, 6).

QUERY (Q)   "What am I looking for?"
KEY   (K)   "What do I contain / what can I be matched on?"
VALUE (V)   "What information do I hand over if I'm attended to?"

For a sequence X:     Q = X·W_Q     K = X·W_K     V = X·W_V

                              Q · Kᵀ
  Attention(Q, K, V) = softmax( ────── + mask ) · V
                              √d_k

STEP BY STEP
  1. SCORES      Each token's query is dot-producted with every key: how well does token i match token j?   (an n x n matrix)
  2. SCALE       Divide by √d_k so scores don't grow with dimension and saturate the softmax.
  3. MASK        In decoder-only models, token i may not attend to tokens AFTER it: set those scores to -infinity (the "causal mask").
  4. SOFTMAX     Turn each row of scores into weights that sum to 1.
  5. MIX         Output for token i = weighted sum of the VALUE vectors, using those weights.

// Educational single-head causal self-attention. Q, K, V are [sequenceLength][headDim].
// Real implementations use batched, fused GPU kernels (and avoid materializing the n x n score matrix).
static float[][] CausalSelfAttention(float[][] Q, float[][] K, float[][] V)
{
    int n = Q.Length;
    int d = Q[0].Length;
    var output = new float[n][];

    for (int i = 0; i < n; i++)
    {
        // Token i may attend to tokens 0..i only (the causal mask)
        var scores = new float[i + 1];
        for (int j = 0; j <= i; j++)
        {
            float dot = 0;
            for (int k = 0; k < d; k++) dot += Q[i][k] * K[j][k];
            scores[j] = dot / MathF.Sqrt(d);                          // scaled dot product
        }

        // Softmax (subtract the max for numerical stability)
        float max = scores.Max();
        float sum = 0;
        for (int j = 0; j < scores.Length; j++) { scores[j] = MathF.Exp(scores[j] - max); sum += scores[j]; }
        for (int j = 0; j < scores.Length; j++) scores[j] /= sum;

        // Weighted sum of value vectors
        var o = new float[V[0].Length];
        for (int j = 0; j <= i; j++)
            for (int k = 0; k < o.Length; k++) o[k] += scores[j] * V[j][k];

        output[i] = o;
    }
    return output;
}

d_model = 4,096,  heads h = 32  ->  each head works in head_dim = d_model / h = 128 dimensions.
  head_1 .. head_32 each compute attention independently  ->  concatenate (back to 4,096)  ->  output projection W_O.
Different heads learn different roles (interpretability research finds heads that track previous tokens, match brackets, copy earlier text, and more).

COMPUTE AND MEMORY:   the score matrix is n x n  ->  cost grows QUADRATICALLY with sequence length n.
  Double the context and attention work roughly quadruples (the MLP part grows linearly).
  This is the fundamental reason long contexts are expensive and slow, and why so much engineering goes into attention efficiency.

MULTI-QUERY ATTENTION (MQA)    All query heads SHARE one key/value head -> the KV cache (Section 6) shrinks ~h-fold; slight quality cost.
GROUPED-QUERY ATTENTION (GQA)  Query heads are split into groups, each group sharing one K/V head. A middle path (for example 32 query heads, 8 KV heads) that most modern open models use:
                               near-MHA quality with a much smaller KV cache and faster decoding.
MULTI-HEAD LATENT ATTENTION    Compresses keys/values into a low-dimensional latent to shrink the cache even further (used in some recent models).
FLASHATTENTION                 NOT an approximation — an exact attention algorithm reorganized to minimize slow GPU memory traffic (tiling in fast on-chip memory), giving large speedups and
                               avoiding the n x n matrix in memory. Standard in modern training and serving.
SLIDING-WINDOW / LOCAL / SPARSE  Each token attends only to a recent window (or a pattern), cutting cost from quadratic toward linear, at the price of direct long-range access (often combined
                               with some global layers or "attention sinks").
CROSS-ATTENTION                Queries from one sequence attend to keys/values from ANOTHER (decoder over encoder output; text over image features in some multimodal designs).

SINUSOIDAL (ABSOLUTE)    The original transformer added fixed sine/cosine waves of different frequencies to each token embedding:
                           PE(pos, 2i)   = sin(pos / 10000^(2i/d))
                           PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
                         Each position gets a unique pattern; no learned parameters.

LEARNED ABSOLUTE         A trainable embedding per position (used by earlier GPT models). Simple, but can't extrapolate beyond the lengths seen in training.

RELATIVE POSITION BIAS   Encode the DISTANCE between tokens (added to attention scores), as in T5-style models.

ALiBi                    Add a fixed penalty to attention scores that grows linearly with distance, with no position embeddings at all — good length extrapolation.

ROTARY (RoPE)            Rotate the query and key vectors by an angle proportional to their position. Because the dot product of two rotated vectors depends on the DIFFERENCE of their rotation
                         angles, attention scores naturally depend on RELATIVE position. The dominant choice in modern open models.

static float[][] SinusoidalPositionalEncoding(int maxPositions, int dModel)
{
    var pe = new float[maxPositions][];
    for (int pos = 0; pos < maxPositions; pos++)
    {
        pe[pos] = new float[dModel];
        for (int i = 0; i < dModel; i += 2)
        {
            double angle = pos / Math.Pow(10000, (double)i / dModel);
            pe[pos][i] = (float)Math.Sin(angle);
            if (i + 1 < dModel) pe[pos][i + 1] = (float)Math.Cos(angle);
        }
    }
    return pe;
}

// Rotate consecutive pairs of dimensions by a position-dependent angle. Applied to Q and K (not V) before the dot product.
static void ApplyRope(float[] vector, int position, float theta = 10000f)
{
    int d = vector.Length;
    for (int i = 0; i < d; i += 2)
    {
        float frequency = MathF.Pow(theta, -(float)i / d);          // low dimensions rotate fast, high dimensions slowly
        float angle = position * frequency;
        float cos = MathF.Cos(angle), sin = MathF.Sin(angle);

        float x0 = vector[i], x1 = vector[i + 1];
        vector[i]     = x0 * cos - x1 * sin;
        vector[i + 1] = x0 * sin + x1 * cos;
    }
}

A model only reliably handles positions it saw during training. Pushing beyond that range degrades quality. Techniques extend context after training:
  POSITION INTERPOLATION   Compress positions so a longer sequence maps into the trained range, then briefly fine-tune.
  NTK-AWARE / YaRN SCALING Rescale RoPE frequencies so high-frequency detail is preserved while low frequencies stretch.
  LONG-CONTEXT TRAINING    Continue training on longer sequences.
Advertised context length is therefore a property of the TRAINING and POSITION SCHEME, not just a config number — and quality at the far end of a long window is often weaker than the headline suggests
(see "lost in the middle," Section 5).

CHARACTERS  Tiny vocabulary, no unknown words — but sequences become very long (expensive attention) and each token carries little meaning.
WORDS       Short sequences — but a huge vocabulary, no way to represent unseen words, typos, or new names ("out-of-vocabulary").
SUBWORDS    The compromise: common words are single tokens; rare words split into reusable pieces ("un" + "believ" + "ably"). Open vocabulary, moderate length. All modern LLMs use subword tokenization.

BYTE-PAIR ENCODING (BPE)
  Start from individual symbols (characters or BYTES). Repeatedly find the most frequent adjacent pair in the training corpus and MERGE it into a new symbol. Stop at the target vocabulary size.
  Encoding applies the learned merges in order. BYTE-LEVEL BPE (GPT-2 onward) starts from the 256 bytes, so ANY text — any language, emoji, binary — can be encoded with no "unknown" token.
  Used by GPT-family models (via the tiktoken implementation), Llama 3 and others.

WORDPIECE
  Like BPE, but chooses merges that maximize the likelihood of the training data rather than raw frequency. Continuation pieces are marked ("##ing"). Used by BERT-style models.

UNIGRAM LANGUAGE MODEL
  Start with a LARGE candidate vocabulary and repeatedly PRUNE the pieces that hurt corpus likelihood the least. Encoding finds the most probable segmentation; it can also SAMPLE alternative
  segmentations (useful as regularization). Implemented in SentencePiece.

SENTENCEPIECE
  A library/framework (supporting BPE and Unigram) that treats text as a raw character stream — no language-specific pre-splitting; spaces become a visible marker (▁). Used by Llama 2, T5, Gemma and many others.

PRE-TOKENIZATION
  Most tokenizers first split text with a regular expression (separating words, numbers, punctuation, whitespace) so merges don't cross certain boundaries. The regex is part of the tokenizer's identity.

// Learns merge rules from a word list — the heart of BPE. (Educational: real trainers work on bytes, at corpus scale, with careful data structures.)
public static List<(string A, string B)> LearnBpeMerges(IEnumerable<string> words, int numMerges)
{
    // Represent each word as space-separated symbols plus an end-of-word marker, with its frequency
    var vocab = words.GroupBy(w => w).ToDictionary(
        g => string.Join(" ", g.Key.Select(c => c.ToString())) + " </w>",
        g => g.Count());

    var merges = new List<(string A, string B)>();

    for (int step = 0; step < numMerges; step++)
    {
        // Count every adjacent symbol pair, weighted by word frequency
        var pairCounts = new Dictionary<(string, string), int>();
        foreach (var (word, freq) in vocab)
        {
            string[] symbols = word.Split(' ');
            for (int i = 0; i < symbols.Length - 1; i++)
            {
                var pair = (symbols[i], symbols[i + 1]);
                pairCounts[pair] = pairCounts.GetValueOrDefault(pair) + freq;
            }
        }
        if (pairCounts.Count == 0) break;

        // Merge the most frequent pair everywhere it occurs
        var best = pairCounts.MaxBy(kv => kv.Value).Key;
        merges.Add(best);

        string pattern = $@"(?<!\S){Regex.Escape(best.Item1 + " " + best.Item2)}(?!\S)";      // whole-symbol matches only
        string replacement = best.Item1 + best.Item2;
        vocab = vocab.ToDictionary(kv => Regex.Replace(kv.Key, pattern, replacement), kv => kv.Value);
    }
    return merges;
}

// LearnBpeMerges(["low", "low", "lower", "newest", "newest", "widest"], 10) learns merges like ("e","s") -> "es" -> ("es","t") -> "est" -> ...
// i.e. frequent character sequences become single vocabulary entries.

// dotnet add package Microsoft.ML.Tokenizers
using Microsoft.ML.Tokenizers;

Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4o");       // BPE tokenizer matching an OpenAI model family

IReadOnlyList<int> ids = tokenizer.EncodeToIds("Tokenization matters for cost.");
int count = tokenizer.CountTokens("Tokenization matters for cost.");
string roundTrip = tokenizer.Decode(ids);

foreach (EncodedToken token in tokenizer.EncodeToTokens("unbelievably", out string? _))
    Console.WriteLine($"{token.Id,8}  '{token.Value}'");                  // see exactly how a word is split into pieces

COST AND LIMITS ARE IN TOKENS    "1,000 words" is not a fixed price: ~750 tokens in English, noticeably more in many other languages and in code.
THE MULTILINGUAL "TOKEN TAX"     Tokenizers trained mostly on English split other languages (and scripts such as Devanagari, Thai, Arabic) into more, smaller pieces — so the same content costs more, fills the
                                 context faster, and can run slower. Measure token counts on YOUR languages before pricing or sizing.
ARITHMETIC AND SPELLING          The model sees tokens, not letters or digits as such: "strawberry" may be 2-3 tokens, so counting letters is genuinely hard; numbers split inconsistently ("12345" vs "1 2345"),
                                 hurting arithmetic. Do these tasks in code (tools).
WHITESPACE AND CASE MATTER       " hello" and "hello" and "Hello" are different tokens; leading spaces and trailing whitespace in prompts change behavior and token counts.
CODE AND STRUCTURED TEXT         Indentation, long identifiers, and JSON punctuation consume many tokens; minified or compact formats can save cost.
SPECIAL TOKENS AND CHAT TEMPLATES  Roles (system/user/assistant), end-of-turn, and tool markers are special tokens inserted by a chat template. Getting the template wrong for an open-weight model
                                 (a mismatched template) is a classic cause of poor output. Never let user text inject special-token strings.
GLITCH TOKENS                    Rare or undertrained tokens can cause erratic behavior.
TOKEN HEALING / BOUNDARIES       Cutting a prompt mid-word (for completion) can place a token boundary the model rarely saw in training and degrade results.

- Count tokens with the right tokenizer BEFORE sending (budgets, rate limits, cost estimates).
- Test token efficiency for your languages and data formats; consider compact formats for large structured payloads.
- Keep tokenizer, vocabulary, and chat template pinned with the model version (see AI Production / MLOps, Section 2).
- Don't ask the model to do character-level or digit-level work it can't see — give it a tool.

POSITION SCHEME     The model is reliable only over position ranges seen in (long-context) training (Section 3).
COMPUTE             Attention cost grows quadratically with length; prefill for a 100k-token prompt is slow and expensive.
MEMORY              The KV cache grows linearly with length x concurrent sequences (Section 6) — often the binding constraint when serving many users.
ATTENTION QUALITY   More tokens means attention is spread thinner, and irrelevant text competes with relevant text.

NEEDLE-IN-A-HAYSTACK TESTS    Check whether a model can retrieve one planted fact from a long document. Many models do well — but a single easy lookup doesn't predict performance on tasks that need to
                              combine many pieces, reason over the whole text, or ignore distractors.
"LOST IN THE MIDDLE"          Research has shown that information in the middle of a long prompt is often used less reliably than information near the beginning or end. Improved in newer models, but
                              don't assume it's gone.
CONTEXT ROT / DISTRACTION     Quality on a fixed task can DEGRADE as irrelevant or conflicting text is added — even well within the window.
INSTRUCTION DILUTION          Instructions at the top can lose force after tens of thousands of tokens of content; restate critical rules near the end.
SO:  measure YOUR task at YOUR typical and maximum context lengths. The usable window is whatever length your evaluation shows holds quality.

LONG CONTEXT ("just put it all in")         + Simple; the model sees everything; good for tasks needing the whole document (summarize, review, cross-reference).
                                            - Cost and latency scale with length on EVERY request (mitigated by prompt caching); quality can degrade; no access control per passage; bounded by the window.
RAG (retrieve what's relevant)              + Cheap per request, scales to huge corpora, supports citations and per-user permissions and deletion.
                                            - Retrieval can miss; needs infrastructure and evaluation (see the RAG guide).
HYBRID                                      Retrieve broadly, then feed a generous (not unlimited) context; cache the stable prefix.
Choose per task; large windows make "stuff it in" viable for moderately sized corpora, but they don't remove the need for evaluation.

BUDGET EXPLICITLY       window = system + tools + history + retrieved context + user input + RESERVED OUTPUT + safety margin. Reserve output tokens first.
RANK AND PACK           Fill the budget with the highest-value items first (below).
COMPACT HISTORY         Summarize older turns; keep recent turns verbatim; keep a small structured scratchpad (goals, decisions, key facts).
TRIM TOOL OUTPUT        Tool results dominate agent contexts — return only what's needed; truncate and summarize large results.
PLACE WELL              Stable instructions first (also enables prefix caching), the question and critical constraints near the END, reference material clearly delimited.
CACHE THE PREFIX        Keep the static prefix byte-identical so providers (and your own servers) can reuse its KV cache (Section 6).
ISOLATE WITH SUB-AGENTS Give sub-tasks fresh, small contexts and take back summaries (see AI Agents, Section 7).
RE-ASK, DON'T RE-STUFF  For long documents, run focused queries over chunks (map-reduce) rather than one giant prompt (see NLP with .NET, Section 5).
MONITOR                 Track input tokens per request, context utilization, truncation (finish_reason = length), and quality vs. context length (see AI Observability).

public sealed record ContextItem(string Id, string Text, int Priority, int Tokens);

public static class ContextPacker
{
    // Choose the most valuable items that fit the token budget. (Greedy by priority; ties favor smaller items.)
    public static IReadOnlyList<ContextItem> Pack(IEnumerable<ContextItem> candidates, int budgetTokens)
    {
        var chosen = new List<ContextItem>();
        int used = 0;

        foreach (ContextItem item in candidates.OrderByDescending(i => i.Priority).ThenBy(i => i.Tokens))
        {
            if (used + item.Tokens > budgetTokens) continue;      // skip items that don't fit; smaller, lower-priority ones may still fit
            chosen.Add(item);
            used += item.Tokens;
        }
        return chosen;
    }
}

// Budget = window - reservedOutput - system - tools - safetyMargin
int available = 128_000 - 4_000 - systemTokens - toolTokens - 1_000;
var packed = ContextPacker.Pack(retrievedPassages.Select(p => new ContextItem(p.Id, p.Text, Priority: p.RerankScore >= 0.8 ? 3 : 2, Tokens: tokenizer.CountTokens(p.Text))), available);

PREFILL (processing the prompt):   all prompt tokens are processed IN PARALLEL; K and V are computed and stored. Compute-bound. Determines TIME TO FIRST TOKEN (TTFT).
DECODE  (generating output):       one token per step; compute Q, K, V for just that token; attend over the cached K/V; append the new K/V. Memory-bandwidth-bound (the whole model's weights and
                                   the growing cache must be read for every single token). Determines TIME PER OUTPUT TOKEN.

Without a cache, decode work per token grows with sequence length; with it, each step does a constant amount of new projection work plus a read of the cache.

KV cache bytes  =  2 (K and V)  x  layers  x  kv_heads  x  head_dim  x  sequence_length  x  batch_size  x  bytes_per_element

Example — a 7B-class model with full multi-head attention (32 layers, 32 KV heads, head_dim 128, FP16 = 2 bytes):
  per token:   2 x 32 x 32 x 128 x 2 B = 524,288 B = 0.5 MiB
  4,096 tokens, 1 sequence:  2 GiB      ->   16 concurrent sequences:  32 GiB   (more than the ~13 GiB of weights!)

Same model with GROUPED-QUERY attention (8 KV heads):  per token 128 KiB;  4,096 tokens = 0.5 GiB per sequence.
A 70B-class GQA model (80 layers, 8 KV heads, head_dim 128, FP16): 320 KiB per token;  8,192 tokens ≈ 2.5 GiB per sequence.

GROUPED-QUERY / MULTI-QUERY ATTENTION   Fewer KV heads -> proportionally smaller cache (Section 2). Chosen at model-design time.
KV-CACHE QUANTIZATION                   Store keys/values in FP8 or INT8 instead of FP16 -> roughly half the memory, small quality impact (measure it).
PAGEDATTENTION (vLLM and similar)       Manage the cache in fixed-size blocks like virtual-memory pages instead of one contiguous buffer per sequence -> far less fragmentation and waste -> more concurrent sequences.
PREFIX CACHING                          Reuse the cached K/V of a shared prefix (system prompt, tool definitions, a reference document) across requests, so only the NEW suffix needs prefill.
                                        This is what providers' "prompt caching" discounts reflect.
SLIDING WINDOW / ATTENTION SINKS        Keep only recent tokens (plus a few initial "sink" tokens) for streaming-style use with bounded memory, accepting loss of distant context.
EVICTION / COMPRESSION                  Drop or merge cache entries judged unimportant; research-stage trade-offs between memory and fidelity.
OFFLOADING                              Move cold cache blocks to CPU memory or disk and fetch as needed (adds latency).

PUT STATIC CONTENT FIRST        A prompt whose first N tokens are identical across requests can reuse the cache for that prefix — big TTFT and cost wins. Any change in the prefix (a timestamp, a reordered tool
                                list, a per-request ID at the top) invalidates the reuse from that point on. Variable content goes LAST.
LONG PROMPTS COST TTFT          Prefill time grows with prompt length; a huge prompt delays the first token even if the answer is short.
OUTPUT LENGTH COSTS DECODE      Every generated token is a full pass through the model: long answers are slow regardless of prompt size.
CONCURRENCY IS MEMORY-BOUND     Self-hosted: size GPU memory for weights + cache x expected concurrency; this, not FLOPs, often decides how many users a GPU serves.
HOSTED MODELS                   You don't manage the cache, but you pay for it: cached-prefix discounts, TTLs, and minimum prefix lengths are the visible effects (see AI Observability, Section 3).

CONTINUOUS BATCHING     Instead of waiting for a whole batch to finish, the server adds new requests and removes finished ones at every decoding step -> much higher GPU utilization (vLLM, TGI, TensorRT-LLM).
CHUNKED PREFILL         Split long prompts so they don't stall other requests' decoding.
SPECULATIVE DECODING    A small "draft" model proposes several tokens; the big model VERIFIES them in one parallel pass, accepting the matching prefix. Same output distribution, faster decoding when the
                        draft is usually right (and a natural use for a small language model, Section 12).
TENSOR / PIPELINE PARALLELISM   Split a model that doesn't fit on one GPU across several.
DECODE IS BANDWIDTH-BOUND       Tokens/sec per sequence ≈ memory bandwidth / bytes of weights read per token (Section 16) — which is why quantization (Section 8) speeds decoding, not just shrinks memory.

softmax with temperature T:     p_i = exp(z_i / T) / Σ_j exp(z_j / T)

Example logits [2.0, 1.0, 0.1] for three candidate tokens:
  T = 0.5   ->  [0.864, 0.117, 0.019]     sharper: the leader dominates
  T = 1.0   ->  [0.659, 0.242, 0.099]     the model's "natural" distribution
  T = 2.0   ->  [0.502, 0.304, 0.194]     flatter: unlikely tokens get a real chance
  T -> 0                                   greedy: always the highest-probability token (deterministic, in principle)

GREEDY (T = 0)         Always pick the single most likely token. Deterministic in theory; can be repetitive and can get stuck in loops. (In practice, floating-point and batching effects on GPUs can
                       still cause tiny variations — "temperature 0" is not a guarantee of identical outputs.)

TEMPERATURE            Rescales the distribution BEFORE sampling. Low = focused and consistent (extraction, code, factual Q&A); high = varied and surprising (brainstorming, creative writing).

TOP-K                  Keep only the K most probable tokens, renormalize, sample among them. Simple but blunt: K=40 is too many when the model is certain and too few when many continuations are reasonable.

TOP-P (NUCLEUS)        Keep the SMALLEST set of tokens whose cumulative probability reaches P (for example 0.9), renormalize, sample. Adapts to the distribution's shape: few candidates when the model is
                       confident, many when it's uncertain. With the example distribution at T=1 and P=0.9, only the top two tokens (0.659 + 0.242 ≈ 0.90) survive.

MIN-P                  Keep tokens whose probability is at least a fraction of the TOP token's probability — another adaptive truncation offered by some inference engines.

PENALTIES              Frequency penalty (discourage tokens in proportion to how often they've appeared), presence penalty (discourage any token that has appeared), and repetition penalties reduce loops.

LOGIT BIAS / MASKING   Add or subtract from specific tokens' logits to encourage or forbid them. Constrained decoding (JSON schema, grammars) masks every token that would violate the format at each step —
                       the mechanism behind structured outputs.

BEAM SEARCH            Tracks several candidate sequences and picks the highest-scoring overall. Common in translation; rarely used for open-ended chat (it favors bland, repetitive text).

SEED                   Some APIs accept a seed to make sampling reproducible "best effort."

// Educational sampler: temperature -> (top-k) -> (top-p) -> sample. Real engines use partial selection on GPU instead of a full sort over a 100k+ vocabulary.
public static int SampleNextToken(ReadOnlySpan<float> logits, float temperature, int topK, float topP, Random rng)
{
    int n = logits.Length;

    if (temperature <= 0)                                                  // greedy
    {
        int best = 0;
        for (int i = 1; i < n; i++) if (logits[i] > logits[best]) best = i;
        return best;
    }

    // 1. Temperature-scaled softmax
    float max = float.NegativeInfinity;
    foreach (float z in logits) max = Math.Max(max, z);

    var probs = new float[n];
    float sum = 0;
    for (int i = 0; i < n; i++) { probs[i] = MathF.Exp((logits[i] - max) / temperature); sum += probs[i]; }
    for (int i = 0; i < n; i++) probs[i] /= sum;

    // 2. Rank tokens by probability
    int[] order = Enumerable.Range(0, n).OrderByDescending(i => probs[i]).ToArray();

    // 3. Top-k: at most k candidates (0 = disabled)
    int keep = topK > 0 ? Math.Min(topK, n) : n;

    // 4. Top-p: the smallest prefix whose cumulative probability reaches p
    float cumulative = 0;
    int cutoff = 0;
    while (cutoff < keep)
    {
        cumulative += probs[order[cutoff]];
        cutoff++;
        if (cumulative >= topP) break;
    }

    // 5. Renormalize over the surviving candidates and sample
    float kept = 0;
    for (int i = 0; i < cutoff; i++) kept += probs[order[i]];

    float r = (float)rng.NextDouble() * kept, acc = 0;
    for (int i = 0; i < cutoff; i++)
    {
        acc += probs[order[i]];
        if (r <= acc) return order[i];
    }
    return order[cutoff - 1];
}

CHANGE ONE: adjust temperature OR top_p, not both at once — they overlap in purpose, and changing both makes behavior hard to reason about.
TASKS:      extraction, classification, code, tool calls, structured output -> low temperature (0-0.3).   Chat/Q&A -> low to moderate.   Creative writing, brainstorming -> higher.
STRUCTURED OUTPUT: use schema-constrained decoding (structured outputs) rather than hoping low temperature yields valid JSON.
REASONING MODELS: many restrict or ignore sampling parameters and use other controls (reasoning effort); check the model's documentation.
DETERMINISM: temperature 0 + a seed reduces but doesn't eliminate variation; design evaluation and caching with that in mind (repeat runs; tolerance bands).
DIVERSITY:  if you need N different candidates (for self-consistency or ranking), sample at moderate temperature rather than re-running greedy.
EVALUATE:   tune on a representative test set, not a handful of prompts.

PRECISION            BYTES/PARAM   7B MODEL (weights only)
FP32                 4             ≈ 26 GiB
FP16 / BF16          2             ≈ 13 GiB     <- the common "full" precision for inference
INT8 / FP8           1             ≈ 6.5 GiB
INT4 (4-bit)         0.5           ≈ 3.3 GiB    (+ small overhead for scales)

SYMMETRIC INT8 QUANTIZATION:     scale = max(|x|) / 127          q = round(x / scale)  (clamped to -127..127)         x̂ = q x scale (dequantize)
AFFINE (ASYMMETRIC):             q = round(x / scale) + zero_point   — handles ranges not centered on zero.
The rounding error per value is at most scale/2; the question is how much that error hurts the model's output.

public static (sbyte[] Quantized, float Scale) QuantizeInt8(float[] values)
{
    float maxAbs = values.Select(MathF.Abs).Max();
    float scale = maxAbs == 0 ? 1f : maxAbs / 127f;

    sbyte[] q = values.Select(v => (sbyte)Math.Clamp((int)MathF.Round(v / scale), -127, 127)).ToArray();
    return (q, scale);
}

public static float[] Dequantize(sbyte[] q, float scale) => q.Select(v => v * scale).ToArray();

// Group-wise: each block of (say) 64 weights gets its OWN scale, so one outlier only degrades its own group instead of the whole tensor
public static (sbyte[] Q, float[] Scales) QuantizeInt8Grouped(float[] values, int groupSize = 64)
{
    var q = new sbyte[values.Length];
    var scales = new float[(values.Length + groupSize - 1) / groupSize];

    for (int g = 0; g < scales.Length; g++)
    {
        int start = g * groupSize, len = Math.Min(groupSize, values.Length - start);
        float maxAbs = 0;
        for (int i = 0; i < len; i++) maxAbs = Math.Max(maxAbs, MathF.Abs(values[start + i]));

        scales[g] = maxAbs == 0 ? 1f : maxAbs / 127f;
        for (int i = 0; i < len; i++)
            q[start + i] = (sbyte)Math.Clamp((int)MathF.Round(values[start + i] / scales[g]), -127, 127);
    }
    return (q, scales);
}

WHAT IS QUANTIZED
  WEIGHT-ONLY (e.g. "W4A16")   Weights stored in 4/8 bits, dequantized on the fly; activations stay in 16-bit. The most common for LLM inference: big memory win, modest quality loss.
  WEIGHTS + ACTIVATIONS ("W8A8") Both in 8-bit so matrix multiplies run in integer/FP8 hardware — faster compute, but activations have OUTLIERS that make this harder (techniques such as
                               SmoothQuant and LLM.int8() handle them).
  KV CACHE                     Quantize cached keys/values (Section 6) to raise concurrency.

GRANULARITY
  PER-TENSOR (coarse), PER-CHANNEL, PER-GROUP (for example groups of 32/64/128 weights) — finer granularity means better accuracy at slightly more storage for scales.

WHEN
  POST-TRAINING QUANTIZATION (PTQ)    Quantize a finished model, often with a small calibration dataset. Cheap and by far the most common for LLMs.
  QUANTIZATION-AWARE TRAINING (QAT)   Simulate quantization during training so the model adapts. Better quality at very low bit-widths; far more expensive.

POPULAR METHODS AND FORMATS (names you will encounter)
  GPTQ, AWQ                    Post-training 4-bit weight quantization methods that choose roundings/scales to minimize output error, protecting the most important weights.
  bitsandbytes (NF4, int8)     Library supporting 8-bit and 4-bit "NormalFloat" quantization — the basis of QLoRA (Section 11).
  GGUF (llama.cpp k-quants)    File format and mixed-precision quantization schemes for CPU/consumer-GPU inference.
  FP8                          8-bit floating point supported by recent datacenter GPUs for both weights and activations.
  ONNX Runtime quantization    Tooling to quantize ONNX models (INT8 for classic models; INT4 weight-only paths for LLMs via ONNX Runtime GenAI model builders).

GAINS    Memory (fit a bigger model, or more concurrent sequences), speed (less data to read per token; faster integer math on supported hardware), cost, and edge/CPU deployment.
RISKS    Quality degradation — small at 8-bit, noticeable but often acceptable at 4-bit for larger models, and increasingly severe below that; larger models tolerate quantization better than small ones.
         Some tasks (math, code, long-context reasoning, low-resource languages, precise formatting) degrade sooner than general chat. Outlier weights/activations are the usual culprit.

1. Start from the highest-quality precision you can afford; step down (FP16 -> INT8 -> INT4) while measuring.
2. EVALUATE ON YOUR TASK with your evaluation set (not just perplexity, which can look fine while task accuracy drops). Include tail cases: long inputs, structured output, tool calls, other languages.
3. Compare cost per inference and latency at each level (see AI Production / MLOps, Section 10) — a quantized model may let you drop to a cheaper GPU or CPU.
4. Prefer a larger model quantized to 4-bit over a smaller model at 16-bit when memory is the constraint — this often wins on quality per byte, but verify for your task.
5. Version quantized artifacts like any model (method, bit-width, group size, calibration data) and record them on traces.

CLASSIC (LOGIT) DISTILLATION   The student learns from the teacher's full output distribution ("soft targets") rather than just the correct label. A temperature T softens both distributions
                               so the student also learns the teacher's relative beliefs about wrong answers ("dark knowledge": a cat photo is more like a dog than a truck).

       L = α · CE(y, student)  +  (1 − α) · T² · KL( teacher_T ‖ student_T )

SEQUENCE-LEVEL / SYNTHETIC-DATA DISTILLATION   The most common approach for LLMs today: have the teacher GENERATE training data — answers, rationales, tool-use traces, classifications — and fine-tune the
                               student on it with ordinary supervised learning. No access to teacher logits is needed, so it works even when the teacher is an API.

REASONING-TRACE DISTILLATION   Train the student on the teacher's step-by-step reasoning, transferring problem-solving patterns, not just final answers.

FEATURE / HIDDEN-STATE DISTILLATION   Match intermediate representations (used for models like DistilBERT); requires access to both models' internals.

ON-POLICY DISTILLATION         The student generates, the teacher scores or corrects, and the student learns from the feedback — closing the gap between training and inference behavior.

// KL divergence between teacher and student next-token distributions, both softened by temperature T
public static double DistillationLoss(float[] teacherLogits, float[] studentLogits, int trueTokenId, double temperature, double alpha)
{
    double[] pt = Softmax(teacherLogits, temperature);
    double[] ps = Softmax(studentLogits, temperature);

    double kl = 0;
    for (int i = 0; i < pt.Length; i++)
        if (pt[i] > 0) kl += pt[i] * (Math.Log(pt[i]) - Math.Log(ps[i]));

    double[] psHard = Softmax(studentLogits, 1.0);
    double crossEntropy = -Math.Log(psHard[trueTokenId]);

    return alpha * crossEntropy + (1 - alpha) * temperature * temperature * kl;     // T² keeps gradient magnitudes comparable across temperatures
}

static double[] Softmax(float[] logits, double t)
{
    double max = logits.Max();
    double[] exp = logits.Select(z => Math.Exp((z - max) / t)).ToArray();
    double sum = exp.Sum();
    return exp.Select(e => e / sum).ToArray();
}

HIGH-VOLUME, NARROW TASK    Classification, extraction, routing, summarization of a specific document type, structured output — where a frontier model is overkill and cost/latency matter at scale.
LATENCY-CRITICAL           A student small enough to meet a tight response-time or on-device constraint.
COST REDUCTION             Replace most calls to an expensive model with a cheap specialist, keeping the big model for hard cases (see routing, Section 15).
PRIVACY / OFFLINE          Run a distilled open-weight student in your environment.
THE PRACTICAL RECIPE       1) Collect representative inputs (real, anonymized, consented). 2) Have the teacher label/generate outputs, with human spot-checking and filtering for quality. 3) Fine-tune the student
                           (often with LoRA, Section 11). 4) Evaluate against the teacher AND against ground truth on a held-out set. 5) Deploy behind a router with the teacher as fallback.

CAPABILITY CEILING   A student rarely exceeds its teacher and usually generalizes worse outside the distribution it was distilled on; broad world knowledge doesn't compress well into few parameters.
ERROR INHERITANCE    The student copies the teacher's mistakes and biases — and can amplify them; filter synthetic data and verify against ground truth where possible.
DATA QUALITY         Distilled models are only as good as the (synthetic) data: diversity, correctness, and coverage of edge cases matter more than volume.
TERMS OF USE         Many proprietary providers' terms restrict using model OUTPUTS to train competing models. Read the license/terms of any teacher whose outputs you use, and check data rights generally.
EVALUATION           Measure the student on YOUR task — a small model can look fine on average and fail on the rare, important cases.

FINE-TUNING   Changes the MODEL'S WEIGHTS (or adds trainable adapters). Best at teaching BEHAVIOR: tone, format, task skill, domain conventions, tool-calling style.
RAG           Changes the PROMPT by retrieving relevant documents at request time. Best at supplying KNOWLEDGE: facts that are private, changing, or too large to memorize — with citations.

A useful slogan: fine-tuning teaches the model HOW to respond; RAG tells it WHAT to respond with.

"IT DOESN'T KNOW OUR DOCS / PRODUCTS / POLICIES"       -> RAG.
"ANSWERS MUST BE UP TO DATE OR CITE SOURCES"           -> RAG.
"DIFFERENT USERS MAY SEE DIFFERENT DATA"               -> RAG with permission filters.
"IT WON'T FOLLOW OUR FORMAT / TONE / HOUSE STYLE"      -> prompting first; fine-tune if prompts get long, brittle, or costly.
"WE NEED CONSISTENT STRUCTURED EXTRACTION AT SCALE"    -> fine-tune a small model (often via distillation) or use structured outputs with a good prompt.
"A SPECIALIZED CLASSIFICATION / DOMAIN TASK"           -> fine-tuning (or a classical/embedding classifier).
"TOO SLOW / TOO EXPENSIVE AT VOLUME"                   -> distill or fine-tune a smaller model; cache; route.
"IT USES OUR TOOLS BADLY"                              -> better tool descriptions and few-shot examples first; fine-tune on tool-use traces if needed.
"WE WANT BOTH"                                         -> combine: RAG for facts + a fine-tuned model that is good at using retrieved context and your format (sometimes called RAFT-style training).

1. PROMPT ENGINEERING       Clear instructions, structure, few-shot examples.            (minutes; free; reversible)
2. STRUCTURED OUTPUTS / TOOLS  Constrain format; let code do exact work.                  (hours)
3. RAG                      Add retrieval for knowledge.                                  (days-weeks; infrastructure)
4. FINE-TUNING (PEFT/LoRA)  Adapt behavior on curated examples.                           (weeks; needs data + evaluation)
5. CONTINUED PRETRAINING    Teach a base model a new domain at scale (specialized fields, new languages).   (rare; very expensive)
6. TRAIN FROM SCRATCH       Almost never justified for an application.
Move down the ladder only when you can show — with an evaluation set — that the previous rung is insufficient.

FULL FINE-TUNING          Update all weights. Strongest adaptation, highest memory/compute cost, a full copy of the model per variant, higher forgetting risk.
PARAMETER-EFFICIENT (PEFT)  Train a small number of added or selected parameters (LoRA and relatives, Section 11) while freezing the base. Cheap, fast, swappable. The default for most teams.
SFT                       Supervised fine-tuning on (prompt, ideal response) pairs.
PREFERENCE TUNING         DPO, RLHF-style methods that optimize preferred over rejected responses — to shape style, safety, and judgment.

QUALITY OVER QUANTITY     Hundreds to a few thousand carefully curated, consistent, representative examples often beat tens of thousands of noisy ones. Wrong or inconsistent labels teach the wrong thing.
FORMAT = INFERENCE FORMAT Train with the same chat template, system prompt, and output format you'll use in production.
COVER EDGE CASES          Include refusals, "not found" answers, unusual inputs, and hard negatives — not only happy paths.
HOLD OUT A TEST SET       Never evaluate on training examples; split by source to avoid leakage.
GUARD PRIVACY AND LICENSES  Training data can be memorized and regurgitated; scrub PII and secrets; confirm you have rights to the data and the base model's license allows your use.
DON'T TRAIN FACTS YOU'LL CHANGE  Facts that change belong in retrieval, not weights.

CATASTROPHIC FORGETTING  Narrow training erodes general abilities; evaluate general behavior too, and prefer PEFT with modest learning rates and epochs.
OVERFITTING              The model memorizes training examples; watch the validation loss and held-out task metrics.
TEACHING HALLUCINATION   Training on answers requiring knowledge the model lacks teaches it to answer confidently anyway.
STALE EXPERTISE          Weights freeze a snapshot; re-tune on a schedule or pair with RAG.
EVALUATION GAP           Without an evaluation set you can't tell whether fine-tuning helped. Build it BEFORE you train.
OPERATIONAL BURDEN       Fine-tuned models need versioning, deployment, monitoring, and retraining pipelines (see AI Production / MLOps).

HOSTED FINE-TUNING   Some providers (including Azure OpenAI for supported models) let you upload training data and get a private fine-tuned deployment: simplest operations, but limited model choice, per-token
                     training and (often higher) inference pricing, and provider-specific lifecycle. Check current model support and costs.
SELF-MANAGED         Fine-tune open-weight models yourself (Hugging Face PEFT/TRL, Axolotl, Unsloth and similar tooling — mostly Python), on GPUs you rent or own, then export and serve the result
                     (Section 11). Maximum control, with the operational burden that implies. The .NET role is usually orchestration, evaluation, and serving.

FREEZE the pretrained weight matrix W (d x k).
LEARN a low-rank update:      ΔW = B · A        with   A: (r x k),   B: (d x r),   rank r ≪ min(d, k)
Effective weight:             W' = W + (α / r) · B · A         (α is a scaling hyperparameter)

INITIALIZATION: A ~ small random values, B = 0  ->  ΔW = 0 at the start, so training begins exactly at the pretrained model.

PARAMETER COUNT (one matrix, d = k = 4,096, rank r = 16):
  full matrix:   4,096 x 4,096              = 16,777,216
  LoRA (A + B):  16 x (4,096 + 4,096)        =    131,072          ≈ 0.78 % of the matrix

WHERE TO APPLY IT     Typically the attention projections (query and value at minimum; often all of q, k, v, o) and frequently the MLP's linear layers too. More modules = more capacity, more parameters.
KEY HYPERPARAMETERS   rank r (commonly 8-64: higher for harder, more diverse tasks), alpha (often set around r to 2r), dropout (small, e.g. 0.05), learning rate (higher than full fine-tuning, often ~1e-4 to 2e-4),
                      epochs (usually 1-3), and target modules.
RESULT                A tiny "adapter" file (megabytes, not gigabytes) that sits on top of the unchanged base model.

MEMORY & COST        Gradients and optimizer states exist only for the small adapter, slashing training memory — fine-tuning big models on modest hardware becomes possible.
SWAPPABLE ADAPTERS   One base model, many adapters (per task, customer, language, or tone). Serving stacks can load several adapters on a single base model and select one per request ("multi-LoRA serving"),
                     which is far cheaper than hosting many full models.
NO INFERENCE PENALTY IF MERGED   Fold ΔW into W (W' = W + (α/r)·BA) and ship an ordinary model with zero added latency — or keep adapters separate for flexibility, accepting a small overhead.
LESS FORGETTING      The base is frozen, so general capabilities are better preserved (though not guaranteed).
REVERSIBLE           Remove the adapter to restore the original model.

4-BIT NORMALFLOAT (NF4)   A 4-bit data type designed for normally distributed weights, more accurate than naive 4-bit integers.
DOUBLE QUANTIZATION       Quantize the quantization constants too, saving a bit more memory.
PAGED OPTIMIZERS          Avoid out-of-memory spikes by paging optimizer state between GPU and CPU.
EFFECT                    Weights cost ~0.5 byte/parameter instead of 2, so a large model that normally needs a multi-GPU server can be fine-tuned on a single GPU — with quality close to 16-bit LoRA
                          in the reported results. Training is somewhat slower due to dequantization overhead.

# Illustrative: Hugging Face transformers + peft + bitsandbytes (+ trl for the training loop). Verify current APIs and versions.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

base = AutoModelForCausalLM.from_pretrained("<base-model-id>", quantization_config=bnb, device_map="auto")
base = prepare_model_for_kbit_training(base)

lora = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)

model = get_peft_model(base, lora)
model.print_trainable_parameters()      # typically well under 1-2% of all parameters
# ... train with your formatted dataset (use the SAME chat template you'll use at inference), evaluate, then save the adapter

1. EVALUATE the adapter on a held-out set against the base model and the prompting/RAG alternatives.
2. MERGE the adapter into the base (optional) and, if desired, QUANTIZE for serving; or keep the adapter separate for multi-LoRA serving.
3. SERVE the result:
     - An OpenAI-compatible server (vLLM, Ollama, and similar) — then your .NET code is just another IChatClient (below).
     - ONNX Runtime GenAI, after exporting the merged model to ONNX — in-process or local inference from .NET (Section 12).
4. VERSION the base model, adapter, training data version, and hyperparameters; record the version on traces (AI Production / MLOps, Section 2).
5. MONITOR drift and quality like any model; re-tune as needs change.

// A self-hosted open-weight (or LoRA-adapted) model behind an OpenAI-compatible endpoint is consumed like any other chat model
IChatClient chat = new OpenAIClient(
        new ApiKeyCredential("not-needed-locally"),
        new OpenAIClientOptions { Endpoint = new Uri("http://llm.internal:8000/v1") })       // vLLM / Ollama / similar
    .GetChatClient("my-finetuned-model")
    .AsIChatClient();

ChatResponse response = await chat.GetResponseAsync("Classify this ticket: 'I was charged twice.'");

Variants of LoRA (DoRA, rsLoRA, LoRA+, and others) and other PEFT families (adapters, prefix/prompt tuning, IA3) exist; LoRA/QLoRA remain the most widely used. Preference tuning (DPO) is commonly run
on top of LoRA as well. Treat "which variant" as an experiment to evaluate, not a given.

COST                 A fraction of the per-token (or per-GPU) cost of frontier models.
LATENCY              Faster time to first token and tokens per second; viable for real-time, interactive, and high-throughput pipelines.
PRIVACY & RESIDENCY  Run entirely inside your boundary, on-device, or offline — data never leaves.
EDGE & OFFLINE       Laptops, phones, factory floors, vehicles, retail devices, with NPUs/GPUs/CPUs.
FINE-TUNING          Cheap to specialize with LoRA/QLoRA (Section 11), so a tuned SLM can beat a general giant on a narrow task.
SPECULATIVE DECODING A natural "draft" model to accelerate a larger model (Section 6).
COMPONENTS           Routers, guard/moderation models, classifiers, extractors, query rewriters, rerankers inside larger systems.

GOOD AT      Classification, extraction, routing and intent detection, short summarization, structured output, rewriting, FAQ-style Q&A grounded by retrieval, simple tool use, on-device assistants,
             and narrow domains after fine-tuning.
WEAKER AT    Broad world knowledge (they can't store as many facts — pair with RAG), complex multi-step reasoning, long-context tasks, subtle instruction-following in messy prompts, robustness to
             adversarial input, multilingual breadth (varies), and safety behavior (verify; add guardrails).
So: evaluate on YOUR task; use SLMs where the task is narrow and well-defined, and fall back to a larger model when they struggle (Section 15).

ONNX RUNTIME GenAI    The Microsoft.ML.OnnxRuntimeGenAI package runs generative models exported to ONNX (CPU, CUDA, DirectML, and NPU paths), handling tokenization, the KV cache, and sampling.
OPENAI-COMPATIBLE LOCAL SERVERS   Ollama, llama.cpp-based servers, vLLM, and local runtimes such as Microsoft's Foundry Local expose an OpenAI-style API consumed through IChatClient.
LLAMA.CPP BINDINGS    .NET bindings (for example LLamaSharp) run GGUF models in-process.

// dotnet add package Microsoft.ML.OnnxRuntimeGenAI   (APIs have changed across versions — check the current documentation)
using Microsoft.ML.OnnxRuntimeGenAI;

using var model = new Model(@"C:\models\phi-onnx-int4");
using var tokenizer = new Tokenizer(model);
using var stream = tokenizer.CreateStream();

string prompt = "<|user|>\nExplain dependency injection in one sentence.<|end|>\n<|assistant|>\n";      // the model's chat template matters!

using var generatorParams = new GeneratorParams(model);
generatorParams.SetSearchOption("max_length", 512);

using var generator = new Generator(model, generatorParams);
generator.AppendTokens(tokenizer.Encode(prompt)[0]);               // newer API shape; older versions used SetInputSequences + ComputeLogits

while (!generator.IsDone())
{
    generator.GenerateNextToken();
    Console.Write(stream.Decode(generator.GetSequence(0)[^1]));    // stream tokens as they are produced
}

// The package also offers an IChatClient implementation so a local model plugs into the same abstractions as hosted ones
// (check current availability and constructor shape):
IChatClient local = new OnnxRuntimeGenAIChatClient(@"C:\models\phi-onnx-int4");

PICK A QUANTIZED BUILD  4-bit weights make multi-billion-parameter models practical on CPUs and small GPUs (Section 8); measure quality loss on your task.
MATCH THE CHAT TEMPLATE The wrong role/special-token format is the commonest cause of "the small model is dumb."
CONSTRAIN OUTPUT        Use grammar/schema-constrained decoding or validation for structured results — small models drift off-format more often.
COMBINE WITH RAG        Let retrieval supply facts; let the SLM do language work.
GUARDRAILS              Add input/output moderation and validation; don't assume the safety behavior of a larger model.
BENCHMARK ON TARGET HARDWARE  Tokens/sec and memory on the actual CPU/GPU/NPU, at realistic concurrency (see AI Production / MLOps, Section 10).
USE AS A CASCADE TIER   Handle the easy majority with the SLM; escalate uncertain or hard requests to a larger model (Section 15).

PROPRIETARY (CLOSED)   Accessed through an API or managed service; weights and training details are not released (many frontier models).
OPEN-WEIGHT            The trained WEIGHTS are downloadable, so you can run, fine-tune, and quantize the model yourself. The license varies and may restrict use.
OPEN-SOURCE            A stricter notion: permissive license plus (under emerging definitions) openness about training data and code. Many "open" models are open-weight only.

LICENSES MATTER      Some open-weight models use permissive licenses (for example Apache-2.0 or MIT); others use custom "community" licenses with conditions — acceptable-use restrictions, attribution,
                     limits for very large companies, or restrictions on using outputs to train other models. READ THE LICENSE and have legal review it for commercial use.

HOSTED:       monthly cost ≈ tokens per month x price per token  (plus any reserved-capacity fees)
SELF-HOSTED:  monthly cost ≈ GPU instance cost x hours x replicas  +  engineering and operations time  +  storage/networking
BREAK-EVEN:   depends almost entirely on UTILIZATION. A GPU that is 15% busy is expensive per token; one at 70% utilization on a quantized, well-batched model can be far cheaper than API pricing.
              Include the people cost honestly — and the cost of quality differences (a cheaper model that needs more retries, review, or escalation isn't cheaper).

START HOSTED when you're proving value, traffic is modest or spiky, you need top quality fast, and data terms are acceptable.
CONSIDER OPEN-WEIGHT when: data cannot leave your boundary; volume is high and steady enough to keep GPUs busy; you need deep customization (fine-tuning, constrained decoding, adapters); you need offline/edge
   operation; you want to avoid dependency on a provider's pricing and deprecations; or a small tuned model meets the quality bar far more cheaply.
MIX:  route by task and sensitivity — a self-hosted SLM for classification, routing, and PII-bearing requests; a hosted frontier model for hard reasoning and generation (Section 15).
   Cloud model catalogs (such as Azure AI Foundry's) host both kinds, letting you consume open models as managed endpoints without running GPUs yourself.
ABSTRACT the model behind IChatClient so the choice is reversible (see Microsoft.Extensions.AI), and keep an evaluation set so you can compare models on YOUR task.

TYPICAL ARCHITECTURE (vision-language models)
  IMAGE ──> VISION ENCODER (a Vision Transformer, ViT: splits the image into patches, e.g. 14x14 pixels, and embeds each)
        ──> PROJECTOR / ADAPTER (maps image-patch embeddings into the language model's embedding space)
        ──> the resulting "image tokens" are placed in the sequence alongside text tokens
  The LLM then attends over the combined sequence exactly as it does over text.

DESIGN VARIANTS
  EARLY FUSION / UNIFIED   Image tokens go directly into the decoder's input sequence (common today).
  CROSS-ATTENTION          The text decoder cross-attends to image features in dedicated layers (Flamingo-style).
  NATIVE MULTIMODAL        Trained from the start on interleaved text, image, and audio data (some frontier models), rather than bolting a vision encoder onto a text model.
AUDIO    Audio encoders (Whisper-style) feed speech into the model, or natively multimodal models take audio directly and may emit speech.
VIDEO    Sampled frames (plus audio/time information) treated as sequences of images — expensive in tokens.
OUTPUT   Many "multimodal" models only OUTPUT text; image or audio generation is often handled by separate generative models (diffusion models for images) or a model with native generation.

An image consumes tokens in proportion to its size/detail — a high-resolution image can cost hundreds to thousands of tokens (providers often tile images and charge per tile, with a low-detail mode that
uses a small fixed cost). Video and multi-page documents multiply this. So image-heavy workloads dominate input cost and fill the context window quickly. Check your provider's image-token accounting, downscale
images to the smallest size that preserves what the task needs, and cache or pre-process repeated images.

using Microsoft.Extensions.AI;

public record InvoiceFields(string Vendor, string InvoiceNumber, string Date, decimal Total);

var message = new ChatMessage(ChatRole.User,
[
    new TextContent("Extract the vendor, invoice number, date, and total from this invoice."),
    new DataContent(imageBytes, "image/jpeg"),                       // or: new UriContent(new Uri("https://..."), "image/jpeg")
]);

ChatResponse<InvoiceFields> response = await chat.GetResponseAsync<InvoiceFields>([message]);
InvoiceFields invoice = response.Result;                              // shape is guaranteed; VALIDATE the values (does the total equal the sum of the lines?)

STRENGTHS     Flexible zero-shot understanding of screenshots, charts, diagrams, handwriting, receipts, UI, and photos; question answering and reasoning that combines text and image; messy real-world documents.

LIMITS
  FINE DETAIL & SMALL TEXT   Downscaled images lose small print; crop and zoom to regions that matter.
  COUNTING & SPATIAL REASONING  Counting many objects and precise positions/boxes are unreliable; use a dedicated detector (see Computer Vision with .NET) and let the LLM interpret.
  HALLUCINATION              Models can invent text or details not in the image — especially in OCR-like extraction. Validate against business rules and, where possible, a classical OCR cross-check.
  CHARTS & TABLES            Reading exact values from charts and complex tables can be wrong; prefer structured sources or layout-aware extraction (Document Intelligence) when precision matters.
  CONTEXT & COST             Many images exhaust the window and the budget.
  LATENCY                    Image-heavy prompts increase prefill time (TTFT).
  PRIVACY                    Images may contain faces, IDs, medical or financial data — control where they go (see AI Security and Computer Vision with .NET).
  PROMPT INJECTION           Text inside an image can carry instructions ("visual prompt injection"); treat image content as untrusted data.
  VARIABILITY                Support for formats, sizes, number of images, and modalities differs by model; check limits.

PATTERNS
  HYBRID PIPELINES   Classical OCR/detection for precision and speed + the LLM for interpretation and normalization; send the LLM only the crops or fields that need judgment.
  DOCUMENTS          For PDFs, extract text where it exists and send page images only for scanned or visually complex pages.
  EVALUATE           Build a labeled set from real images, measure field-level accuracy, hallucination rate, and cost per image; route cheap/easy images to a smaller model.

STATIC / RULE-BASED      Route by feature or endpoint ("ticket classification -> small model; contract analysis -> strong model"). Simple, predictable, testable. The right starting point.

CAPABILITY-BASED         Route by what the request NEEDS: images -> a vision model; very long input -> a long-context model; tool use or structured output -> one that supports it; a specific language -> one strong in it.

SENSITIVITY-BASED        Route by data classification: requests containing PII/confidential data go to a self-hosted or private-network model; others may use a hosted frontier model.

COMPLEXITY / HEURISTIC   Use signals — input length, presence of code or math, number of constraints, conversation depth, tools requested — to guess difficulty.

CLASSIFIER / LEARNED ROUTER   A small model (an embedding + classifier, or a fine-tuned SLM) predicts whether the cheap model will suffice, trained on labeled examples of "strong model needed vs. not."
                         Research and open-source routers trained on preference data exist; evaluate before trusting.

LLM-AS-ROUTER            Ask a small LLM to triage the request. Flexible, but adds latency and cost and is itself non-deterministic — use sparingly.

CASCADE (TRY CHEAP, VERIFY, ESCALATE)   Send to the cheap model first; check the result with validators, confidence signals, or a judge; escalate to a stronger model only on failure. Most cost savings,
                         at the price of extra latency on escalated requests and wasted spend on failed cheap attempts.

CONFIDENCE-BASED         Use signals such as token log-probabilities (where available), agreement across multiple samples (self-consistency), schema/rule validation, or refusal/"unsure" markers to decide
                         whether to escalate.

COST- AND BUDGET-AWARE   Factor in tenant tier, remaining budget, and current load: premium tenants get the strong model; free tier gets the small one; degrade gracefully when a budget is nearly spent.

LATENCY-AWARE            Pick the faster model when the user is waiting and the task allows it.

ADAPTIVE (BANDIT / A-B)  Learn from outcomes (feedback, task success, cost) which model works best per request type, exploring a little while exploiting what works.

public sealed record RouteDecision(string Primary, IReadOnlyList<string> Fallbacks, string Reason);

public sealed record RouteContext(string Feature, string TenantTier, bool ContainsSensitiveData);

public interface IModelRouter
{
    RouteDecision Decide(IReadOnlyList<ChatMessage> messages, ChatOptions? options, RouteContext context);
}

public sealed class HeuristicRouter(Tokenizer tokenizer) : IModelRouter
{
    public RouteDecision Decide(IReadOnlyList<ChatMessage> messages, ChatOptions? options, RouteContext context)
    {
        // 1. Sensitivity first: data classification can't be traded for cost
        if (context.ContainsSensitiveData)
            return new("private-slm", ["private-large"], "sensitive data stays in-boundary");

        // 2. Capability requirements
        bool hasImage = messages.SelectMany(m => m.Contents).OfType<DataContent>().Any(d => d.MediaType?.StartsWith("image/") == true);
        if (hasImage)
            return new("hosted-vision", ["hosted-vision-alt"], "image input");

        int inputTokens = messages.Sum(m => tokenizer.CountTokens(m.Text ?? ""));
        if (inputTokens > 100_000)
            return new("hosted-long-context", ["hosted-strong"], "very long input");

        // 3. Difficulty and tier
        bool usesTools = options?.Tools is { Count: > 0 };
        if (usesTools || context.Feature is "contract-analysis" or "code-review")
            return new("hosted-strong", ["hosted-mid"], "tool use or hard task");

        if (context.TenantTier == "free")
            return new("private-slm", ["hosted-mid"], "free tier: cheapest adequate model");

        // 4. Default: a cheap model, with escalation available
        return new("hosted-small", ["hosted-mid", "hosted-strong"], "default");
    }
}

public sealed class RoutingChatClient(
    IReadOnlyDictionary<string, IChatClient> clients,
    IModelRouter router,
    ICircuitBreakers breakers,
    IRouteContextAccessor contextAccessor,
    ILogger<RoutingChatClient> logger) : IChatClient
{
    public async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        var list = messages.ToList();
        RouteDecision decision = router.Decide(list, options, contextAccessor.Current);
        Exception? lastError = null;

        foreach (string name in new[] { decision.Primary }.Concat(decision.Fallbacks))
        {
            if (breakers.IsOpen(name)) continue;                                     // skip models that are currently failing

            try
            {
                ChatResponse response = await clients[name].GetResponseAsync(list, options, cancellationToken);
                breakers.RecordSuccess(name);
                logger.LogInformation("Routed to {Model}: {Reason}", name, decision.Reason);   // always log the decision for cost and quality analysis
                return response;
            }
            catch (Exception ex) when (IsFailoverCandidate(ex) && !cancellationToken.IsCancellationRequested)
            {
                breakers.RecordFailure(name);
                lastError = ex;
                logger.LogWarning(ex, "Model {Model} failed; trying next fallback", name);
            }
        }

        throw new InvalidOperationException("All routes failed or are unavailable.", lastError);
    }

    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        var list = messages.ToList();
        RouteDecision decision = router.Decide(list, options, contextAccessor.Current);
        Exception? lastError = null;

        foreach (string name in new[] { decision.Primary }.Concat(decision.Fallbacks))
        {
            if (breakers.IsOpen(name)) continue;

            await using var enumerator = clients[name].GetStreamingResponseAsync(list, options, cancellationToken).GetAsyncEnumerator(cancellationToken);
            ChatResponseUpdate first;

            // Fail over ONLY BEFORE the first token reaches the caller. After output has started, switching models mid-answer would produce an incoherent response.
            try
            {
                if (!await enumerator.MoveNextAsync()) yield break;
                first = enumerator.Current;
                breakers.RecordSuccess(name);
            }
            catch (Exception ex) when (IsFailoverCandidate(ex) && !cancellationToken.IsCancellationRequested)
            {
                breakers.RecordFailure(name);
                lastError = ex;
                continue;
            }

            yield return first;
            while (await enumerator.MoveNextAsync())
                yield return enumerator.Current;                                      // errors after this point propagate to the caller
            yield break;
        }

        throw new InvalidOperationException("All routes failed or are unavailable.", lastError);
    }

    // Transient/provider-side failures justify failover; client bugs (bad request) do not — they'd fail on every model
    private static bool IsFailoverCandidate(Exception ex) => ex switch
    {
        HttpRequestException or TimeoutException => true,
        ClientResultException c => c.Status is 408 or 429 or >= 500,
        _ => false
    };

    public object? GetService(Type serviceType, object? serviceKey = null) => null;
    public void Dispose() { foreach (IChatClient c in clients.Values) c.Dispose(); }
}

// Try the cheap model; accept its answer only if it passes validation; otherwise escalate
public async Task<Classification> ClassifyWithCascadeAsync(string text, CancellationToken ct)
{
    ChatResponse<Classification> cheap = await _small.GetResponseAsync<Classification>(BuildMessages(text), new ChatOptions { Temperature = 0 }, cancellationToken: ct);

    if (_allowedLabels.Contains(cheap.Result.Category) && cheap.Result.Confidence >= 0.8)
    {
        _metrics.RecordRoute("small", escalated: false);
        return cheap.Result;
    }

    ChatResponse<Classification> strong = await _strong.GetResponseAsync<Classification>(BuildMessages(text), new ChatOptions { Temperature = 0 }, cancellationToken: ct);
    _metrics.RecordRoute("strong", escalated: true);
    return strong.Result;
}

ERROR-BASED FAILOVER     429 / 5xx / timeout on the primary -> the same model in another region/deployment, then an equivalent model from another provider, then a smaller model. Honor Retry-After; use
                         jittered backoff; trip a CIRCUIT BREAKER after repeated failures so you stop sending traffic to a failing backend and probe it periodically (half-open) until it recovers.

QUALITY-BASED ESCALATION Output fails schema/business validation, a citation check, or a judge threshold -> retry once with a stronger model (the cascade above).

CAPACITY-BASED SPILLOVER Reserved/provisioned capacity exhausted -> spill into pay-as-you-go capacity or a different deployment.

GRACEFUL DEGRADATION     When everything is down or over budget: serve a cached answer, a simpler non-LLM result (keyword search, a template), or an honest "temporarily unavailable" — rather than failing hard.

HEDGED REQUESTS          Send the same request to two backends and use whichever answers first, cancelling the other — cuts tail latency at roughly double cost. Reserve for latency-critical, low-volume paths.

MODEL ALIASES            Application code requests a LOGICAL name ("support-chat"); configuration maps it to concrete models and fallbacks, so changes need no code deploy.

PROMPT PORTABILITY       Prompts tuned for one model may behave differently on another (instruction following, formatting, refusals, tool-call syntax). Test each fallback with the SAME evaluation suite;
                         keep model-specific prompt variants when needed. A fallback that returns malformed output isn't a fallback.
FEATURE PARITY           Fallback models may lack tools, vision, structured outputs, or long context; the router must only choose models that satisfy the request's requirements.
CONSISTENCY WITHIN A CONVERSATION   Switching models mid-conversation can change tone or contradict earlier turns; consider pinning a conversation to a model unless it fails.
MID-STREAM FAILURES      Once tokens have streamed, you can't cleanly switch (as in the code above); decide whether to show a partial answer with an error, or retry fully behind a non-streaming path.
THE ROUTER'S OWN COST    Routing logic, classifiers, and judge calls add latency and spend; the savings must clearly exceed them.
EVALUATING ROUTING       Measure the COST-QUALITY CURVE: for a labeled set, compute quality and cost for "always small," "always strong," your router, and an oracle router (best model per request). A good router
                         sits near the oracle: close to strong-model quality at a fraction of the cost. Track the share routed to each model and the escalation rate in production.
DRIFT                    Request mix, model versions, and prices change — re-evaluate routing rules regularly.
SECURITY                 Never let the user (or injected text) choose the model or tier; derive it from server-side policy and entitlements (see AI Security, Section 12). Route sensitive data only to approved models.
OBSERVABILITY            Log every routing decision (chosen model, reason, fallbacks tried, escalation, cost, latency, outcome) and tag traces with it (see AI Observability).

public static class LlmMath
{
    private const double GiB = 1024.0 * 1024 * 1024;

    // Approximate parameter count of a dense decoder-only transformer: attention (4d²) + MLP (2 * m * d²) per layer, plus embeddings.
    // (Gated MLPs, grouped-query attention, and tied embeddings change the details; this is for sizing, not accounting.)
    public static long ApproxParameters(int layers, int dModel, int vocabSize, int mlpMultiplier = 4) =>
        (long)layers * (4L * dModel * dModel + 2L * mlpMultiplier * dModel * dModel) + (long)vocabSize * dModel;

    // Memory for the weights alone
    public static double WeightGiB(long parameters, double bytesPerParameter) => parameters * bytesPerParameter / GiB;

    // KV cache: 2 (K and V) x layers x kvHeads x headDim x tokens x batch x bytes per element
    public static double KvCacheGiB(int layers, int kvHeads, int headDim, int tokens, int batch, double bytesPerElement) =>
        2.0 * layers * kvHeads * headDim * tokens * batch * bytesPerElement / GiB;

    // Trainable parameters LoRA adds for one (d x k) matrix at rank r
    public static long LoraParameters(int d, int k, int rank) => (long)rank * (d + k);

    // Upper bound on single-stream decode speed: every generated token must read all weights from memory once
    public static double DecodeTokensPerSecondUpperBound(double weightBytes, double memoryBandwidthBytesPerSecond) =>
        memoryBandwidthBytesPerSecond / weightBytes;
}

EXAMPLES (illustrative)

 7B-parameter model weights:   FP16 ≈ 13 GiB   |   INT8 ≈ 6.5 GiB   |   4-bit ≈ 3.3 GiB

 KV cache, 7B-class with full multi-head attention (32 layers, 32 KV heads, head_dim 128, FP16):
     4,096 tokens x 1 sequence  = 2.0 GiB        x 16 concurrent sequences = 32 GiB
 Same with grouped-query attention (8 KV heads):  4,096 tokens x 1 sequence = 0.5 GiB     x 16 = 8 GiB
 Same with an FP8 KV cache: halve again.

 70B-class GQA model (80 layers, 8 KV heads, head_dim 128, FP16):  8,192 tokens x 1 sequence ≈ 2.5 GiB

 LoRA, one 4,096 x 4,096 matrix, rank 16:  131,072 trainable parameters vs 16,777,216 (≈ 0.78 %)

 DECODE SPEED BOUND: a 7B FP16 model reads ~14 GB per token; on a GPU with ~1 TB/s memory bandwidth, single-stream decoding is capped near 70 tokens/s.
   Quantize to 4-bit (≈3.5 GB per token) and the bound rises roughly 4x. Batching amortizes the weight reads across many sequences, which is why serving throughput scales far beyond the single-stream figure.

 TTFT is dominated by PREFILL (compute-bound, ~linear in prompt length plus attention's quadratic term); DECODE speed is bandwidth-bound — optimize them differently.

SIZING CHECKLIST FOR SELF-HOSTING
  GPU memory  ≥  weights (at chosen precision)  +  KV cache x concurrent sequences  +  activations/overhead (≈10-20%)
  Throughput  ≈  decode bound x effective batch size (until compute or cache memory saturates)
  Then LOAD TEST: the formulas find the neighborhood; measurement finds the answer (see AI Production / MLOps, Section 9).