Opening the Black Box of LLM Inference: Foundations of Generative AI

Anatomy of the Transformer from the Inside, from Input to Output Token

SGAEIA Research Series — Article 22

Aridio Silva

Independent Researcher, Brazil

Creator of SGAEIA — Secure Governed Autonomous Edge Intelligence Architecture

ORCID: 0009-0008-2411-6995

DOI: https://doi.org/10.5281/zenodo.23540392

Published: October 21, 2026

License: Creative Commons Attribution 4.0 International — CC BY 4.0

Table of Contents

1. Abstract
2. Introduction and scope
3. Overview of the inference flow
4. Input and tokenization
5. Embeddings and positional encoding
6. Self-attention
7. Multi-head attention and masks
8. Residual connection and normalization
9. Feed-forward network (FFN)
10. Multilayer stacking
11. Weights: what they are and how they are learned
12. Output: logits, softmax, and decoding
13. Case study: disambiguating banco and interpreting the mechanism
14. Emergent properties and scaling laws
15. Limitations, caveats, and open debates
16. From inference to governed execution: implications for SGAEIA
17. Conclusion
18. Glossary
19. Bibliography / References
20. Source code (Python)
21. About the Author
22. Research & Project Resources
23. Figures
24. License
25. Series continuity
26. Suggested citation

Abstract

This article provides a technical, formal, and didactic explanation of the inference process in Transformer-based language models, from input tokenization to next-token selection. It combines formulas, reproducible numerical examples, and six figures to explain embeddings, positional encoding, self-attention, masks, residual connections, normalization, feed-forward networks, logits, softmax, and decoding strategies. A small-scale case study of the ambiguous Portuguese word banco demonstrates how context changes representations without claiming to reproduce the behavior of a real trained model. Finally, the article distinguishes model inference from the governed execution of an agentic system and presents a public, non-normative interpretation of the implications for SGAEIA.

Keywords: AI; Transformer; inference; LLM; attention; tokenization; embeddings; softmax; emergent properties; SGAEIA.

1. Introduction and scope

Purpose of this section. It establishes the objective, conventions, and limits of the article before introducing any technique. This is necessary because the numbers and figures that follow are didactic; without this notice, they could be mistaken for measurements of a real model.

This document describes, step by step, what occurs inside a language model based on the Transformer architecture (Vaswani et al., 2017), from input text to output token. The recurring example is the ambiguous Portuguese sentence “O banco estava lotado” (“The bank was crowded/full”). The objective is to make the “black box” intelligible: each stage presents the concept, the mathematical mechanism, the adopted technique, a numerical example, and its supporting references.

Conventions. In-text references follow the (Author, year) format and are listed in Section 18. Technical terms appear in bold on first occurrence and are defined in the Glossary (Section 17). Matrices are written in uppercase (Q, K, V), and dimensions are written as n (sequence length), d_model (model dimension), and d_k (dimension of each head).

Notice on the numerical examples. The values in Sections 3–11 were calculated in Python/NumPy using reduced dimensions and hand-selected vectors for didactic purposes. They illustrate the arithmetic of the mechanism; they are not measurements of a real trained model. In real models, the weights are learned and the dimensions are much larger (for example, d_model = 512 in the original Transformer and 768 in BERT-base).

Scope. The article describes the Transformer in general terms, covering encoder-only, decoder-only, and encoder–decoder variants.

© 2026 Aridio Silva — Independent Researcher · ORCID 0009-0008-2411-6995

2. Overview of the inference flow

Why the flow contains these stages in this order. Each stage solves a problem left open by the previous one. The network operates only on numbers (tokenization); integer IDs have neither geometry nor order (embeddings and position); each token needs context (attention); the contextual mixture must be transformed, stabilized, and repeated in depth (FFN, Add & Norm, stacking); behavior depends on weights that must be learned (training); and the final result must become a token (logits, softmax, and decoding). The table summarizes the problem solved by each stage and what would be missing without it.

Inference in a language model is the execution of an already-trained model with fixed weights to produce an output from an input. In a generative model, execution is cyclical: the model calculates a probability distribution over the next token, a token is selected and appended to the input, and the cycle repeats (autoregressive generation). A complete computation through all layers is called a forward pass.

The original architecture combines an encoder and a decoder (Vaswani et al., 2017). Two derivative families dominate current use: encoder-only models, such as BERT (Devlin et al., 2019), which observe the sequence in both directions, and decoder-only models, such as the GPT series (Radford et al., 2018; Radford et al., 2019; Brown et al., 2020), which generate text from left to right. The mechanisms described in the following sections apply to all of them, with the primary difference being the attention mask (Section 6).

Training (Section 10) is the process that defines the weights. Inference only uses them.

3. Stage 1 — Input and tokenization

Why this stage is necessary. Computers apply arithmetic operations to numbers; without converting text into IDs, there is no input to the network. The vocabulary must be finite and fixed because the output layer has one dimension per token (Section 11). Subword tokenization addresses a real tradeoff: whole words leave new terms without a representation, whereas individual characters create long sequences and increase the cost of attention, which grows with the square of sequence length (Section 5; Sennrich et al., 2016).

Concept. Neural networks operate on numbers, not text. Tokenization converts a character string into a sequence of tokens (vocabulary units) and each token into an integer ID. The vocabulary is a finite set established before training, typically containing tens of thousands of entries.

Why subwords. Whole-word vocabularies do not represent rare or new words (the out-of-vocabulary, or OOV, problem), whereas character vocabularies produce overly long sequences. Subword units balance these extremes: frequent words become a single token, and rare words are decomposed into reusable pieces (Sennrich et al., 2016).

Techniques discussed.

• BPE (Byte Pair Encoding): starts from characters and iteratively merges the most frequent adjacent pair into a new symbol. It was proposed by Gage (1994) as a compression algorithm and adapted to machine translation by Sennrich et al. (2016). Byte-level variants are used in GPT-series models (Radford et al., 2019).
• WordPiece: selects merges by their increase in language-model likelihood rather than simple frequency (Schuster and Nakajima, 2012; Wu et al., 2016). It is used in BERT (Devlin et al., 2019).
• Unigram LM and SentencePiece: Unigram starts from a large vocabulary and prunes the entries that contribute least (Kudo, 2018). SentencePiece implements BPE and Unigram directly over raw text without whitespace pre-tokenization, making it language-independent (Kudo and Richardson, 2018).

Numerical example (BPE in a didactic corpus). Consider a corpus with the following word frequencies, where </w> marks the end of a word: low (5), lower (2), newest (6), widest (3). Initially, each word is a sequence of characters. At each step, the most frequent pair is merged, with ties resolved by order of occurrence. The code produces:

After eight merges, newest is a single token (newest</w>), low consists of the token low followed by </w>, and widest is decomposed into w, i, d, est</w>: a word never observed as a whole reuses the est</w> piece learned from newest.

Application to the recurring sentence. In a real vocabulary, “O banco estava lotado” usually produces only a few tokens per word because the words are common in Portuguese. For this article, we adopt the simplification 4 words = 4 tokens: [O] [banco] [estava] [lotado], with illustrative integer IDs (for example, 12, 4871, 905, 23019; actual values depend on the tokenizer). The ID for banco is the same for both senses of the word: tokenization does not disambiguate. The distinction emerges only in the attention layers (Section 12).

4. Stage 2 — Embeddings and positional encoding

Why this stage is necessary. An integer ID is an arbitrary label: ID 4871 is not more similar in meaning to ID 4872. An embedding converts each label into a vector in which training can organize similarity (Bengio et al., 2003; Mikolov et al., 2013), and subsequent operations such as sums and dot products are meaningful only over vectors. Positional encoding is necessary because attention does not inherently use order: “the cat eats the fish” and “the fish eats the cat” contain the same tokens, and without position information unmasked attention treats them as equivalent up to permutation (Vaswani et al., 2017). In causally masked decoders, some positional information can be inferred from the number of visible tokens, but explicit positional encoding remains standard practice.

Embedding. Each ID selects one row from a weight matrix W_E of dimension |V| × d_model, producing a dense vector that represents the token. The idea of representing words as continuous vectors in which geometric proximity reflects similarity of use comes from neural language models (Bengio et al., 2003) and word2vec (Mikolov et al., 2013). In the Transformer, the matrix is learned jointly with the rest of the model, the vectors are multiplied by √d_model, and the same matrix is often shared with the output projection (Vaswani et al., 2017; Press and Wolf, 2017). A central point is that the input embedding is static: banco receives the same vector in every sentence. Contextualization occurs later, through attention.

Positional encoding. Attention alone treats the input as a set: permuting the tokens permutes the output in the same way without distinguishing order. A position-dependent vector is therefore added to the embedding. The original proposal uses sinusoidal functions at geometrically varying frequencies (Vaswani et al., 2017):

Equation 1. Mathematical notation rendered as a high-resolution image for reliable display. Copyright 2026 Aridio Silva | CC BY 4.0.

Each pair of dimensions oscillates at a different frequency, and the difference between positions can be expressed as a linear transformation of the vectors, which facilitates learning relative-distance relationships (Vaswani et al., 2017).

Alternatives used in the literature.

• Learned positions: a trainable table used in BERT and GPT-2 (Devlin et al., 2019; Radford et al., 2019).
• Relative positions: attention becomes dependent on the distance between tokens (Shaw et al., 2018).
• RoPE (rotary position embeddings): rotates Q and K according to position, encoding relative position in the dot product itself (Su et al., 2021). It is used in models such as LLaMA (Touvron et al., 2023).
• ALiBi: adds a linear distance-dependent bias to attention scores, improving extrapolation to sequences longer than those seen in training (Press et al., 2022).

Numerical example. Let d_model = 4 (didactic; the original Transformer uses 512), with hand-selected embeddings for the four tokens and sinusoidal encoding computed by the formula above. For banco at position 1: PE = [sin(1), cos(1), sin(0.01), cos(0.01)] = [0.841; 0.540; 0.010; 1.000]. Adding the embedding [0; 1; 0.5; 0] yields x = [0.841; 1.540; 0.510; 1.000]. Figure 2 shows the three matrices; X = E + P, of dimension 4 × 4, is the input to the first layer.

5. Stage 3 — Self-attention

Why this stage is necessary. A word’s meaning depends on its neighbors, while the input embedding is static (Section 4): banco begins with the same vector in every sentence. A mechanism is needed in which each token consults the others using content-dependent rather than fixed-position weights. Recurrent models transmit information step by step, so distant dependencies cross many operations; with attention, any two positions can connect in one step, and the computation can be parallelized (Vaswani et al., 2017). Separating Query, Key, and Value is a design choice that decouples what a token seeks, what it offers for matching, and the content it passes onward. Division by √d_k prevents softmax saturation when d_k is large.

Concept. Self-attention allows each token to recompute its representation as a weighted average of the representations of every token in the sequence, including itself, with content-dependent weights. Attention first emerged in machine translation to align output words with input words (Bahdanau et al., 2015), and the Transformer generalized it to operate from a sequence onto itself (Vaswani et al., 2017).

Three roles per token. Three trainable weight matrices project each input vector:

• Query (Q = X·Wq): what this token is looking for.
• Key (K = X·Wk): what this token offers for matching.
• Value (V = X·Wv): the content this token passes on when selected.

Formula (scaled dot-product attention).

Equation 2. Mathematical notation rendered as a high-resolution image for reliable display. Copyright 2026 Aridio Silva | CC BY 4.0.

• QKᵀ measures compatibility between each Query and each Key (a high dot product means that the vectors point in similar directions).
• Division by √d_k: if the components of q and k have mean 0 and variance 1, the dot product has variance d_k. Without scaling, scores grow with dimensionality and push softmax into regions with near-zero gradients (Vaswani et al., 2017).
• Softmax: converts each row of scores into positive weights that sum to 1.
• Multiplication by V: the result is the weighted average of the Values, yielding the contextualized token.

Numerical example (d_model = 4, d_k = 2, values calculated by code). Using X from Section 4 and hand-selected Wq, Wk, and Wv matrices, the following values are obtained for the token banco at position 1:

The weights sum to 1.000. The output for banco is Z = 0.143·v_O + 0.459·v_banco + 0.265·v_estava + 0.133·v_lotado = [1.043; 1.360], where v_O = [1.000; 1.500], v_banco = [1.096; 2.040], v_estava = [1.419; 0.384], and v_lotado = [0.156; 0.810]. The vector for banco is no longer only banco: it incorporates information from the other tokens in proportion to the attention weights. Because the weights in this example were selected by hand, no linguistic meaning should be inferred from the resulting pattern; in a trained model, Wq, Wk, and Wv learn which relationships matter.

How to read the map (Figure 4). Rows are Queries (who asks); columns are Keys (who is attended to); each row sums to 1. On the left, every token can attend to every other token (encoder style). On the right, the causal mask sets future positions to zero (Section 6).

Cost. The score matrix contains n × n entries, so time and memory grow quadratically with sequence length. Implementations such as FlashAttention reduce memory traffic without changing the mathematical result (Dao et al., 2022).

6. Stage 4 — Multi-head attention and masks

Why this stage is necessary. A single attention operation produces only one weighted average per token, which, according to Vaswani et al. (2017), inhibits simultaneous attention to information from different representational subspaces. Multiple heads allow different relationships to be modeled in parallel without increasing total cost because each head operates in a smaller dimension. The causal mask is necessary for generation: during training, the entire text is available, and without the mask the model would “see” the token it must predict, learn to copy it, and then fail during inference, where the future does not exist. The padding mask prevents empty positions from influencing the result. Shared K/V variants and the KV cache exist because rereading and recomputing the entire past for every generated token would otherwise be prohibitive (Pope et al., 2023).

Concept. A single attention operation produces one weighted average per token. Multi-head attention executes h attention operations in parallel, each with its own Wq, Wk, and Wv matrices and operating in a subspace of dimension d_k = d_model / h. The head outputs are concatenated and projected through a Wo matrix (Vaswani et al., 2017):

Equation 3. Mathematical notation rendered as a high-resolution image for reliable display. Copyright 2026 Aridio Silva | CC BY 4.0.

Numerical example (original Transformer, base configuration). With d_model = 512 and h = 8, each head has d_k = d_v = 64, and total computational cost is similar to that of a single head with dimension 512 (Vaswani et al., 2017). Together, the four projection matrices in one attention layer contain 4 × 512 × 512 = 1,048,576 weights, excluding biases. For internal consistency, h must divide d_model; therefore, 12 heads do not apply to d_model = 512. BERT-base uses 12 heads with d_model = 768, yielding d_k = 64 (Devlin et al., 2019).

Why multiple heads. Each head can learn a different type of relationship. Attention analyses in BERT have identified heads that focus on delimiters, fixed positional offsets, and syntactic relationships such as verbs and direct objects (Clark et al., 2019). There is also evidence that many heads are redundant: a small subset accounts for most of the performance, and the remainder can be pruned with little loss (Voita et al., 2019; Michel et al., 2019).

Variants for efficient inference. In Multi-Query Attention, all heads share a single pair of K and V matrices (Shazeer, 2019). In Grouped-Query Attention, groups of heads share K and V, providing a compromise between quality and memory use (Ainslie et al., 2023). Both reduce the size of the KV cache, which stores previously computed K and V values so that the past need not be recomputed for each newly generated token (Pope et al., 2023).

Attention masks.

• Bidirectional (encoder): each position can attend to every other position. This is the BERT regime (Devlin et al., 2019).
• Causal (decoder): before softmax, scores for future positions receive −∞, making their weights exactly 0. This preserves the autoregressive property: the prediction at position t depends only on positions up to t (Vaswani et al., 2017; Radford et al., 2019).
• Padding: ignores empty positions used to equalize sequence lengths within a batch.

Numerical example of the causal mask. In the row for banco, the scaled scores were [3.165; 4.334; 3.785; 3.093]. With a causal mask, the final two positions (estava and lotado, which are in the future) become −∞, and softmax over [3.165; 4.334] yields [0.237; 0.763; 0; 0]. Therefore, in a generative model, banco cannot attend to *lotado*; disambiguation using later words occurs only after they have been read, at subsequent positions. In bidirectional models, such as the left-hand map in Figure 4, forward attention is permitted.

7. Stage 5 — Residual connection and normalization (Add & Norm)

Why this stage is necessary. Stacking many layers requires both the signal and the training gradient to pass through many transformations. Without a shortcut, deeper networks may perform worse during training, a degradation problem mitigated by residual connections (He et al., 2016). The residual path also preserves the original information and lets each sublayer learn only a correction. Normalization keeps value scales stable with depth (Ba et al., 2016), while its position (Pre-LN or Post-LN) affects training stability when careful warm-up is absent (Xiong et al., 2020). Without these mechanisms, the depth that gives the model capacity could not be trained reliably.

Residual connection. After each sublayer (attention or FFN), the sublayer input is added to its output: y = x + Sublayer(x). The idea comes from residual networks in computer vision (He et al., 2016). It creates a direct path for the signal and gradient, enabling deep stacks to be trained, and causes each sublayer to learn a correction to what already exists rather than reconstructing everything. In circuit interpretations, this path is the residual stream, a channel that every layer reads from and writes to (Elhage et al., 2021).

Normalization. Without scale control, values may grow or shrink across layers and destabilize training. LayerNorm normalizes each token separately across its d_model components, not across the batch, and then applies a learned scale γ and shift β (Ba et al., 2016):

Equation 4. Mathematical notation rendered as a high-resolution image for reliable display. Copyright 2026 Aridio Silva | CC BY 4.0.

RMSNorm omits mean subtraction and normalizes only by the root mean square, at lower cost and with comparable performance (Zhang and Sennrich, 2019); it is used in models such as LLaMA (Touvron et al., 2023).

Post-LN and Pre-LN. The original Transformer applies normalization after the residual addition (Post-LN). Many current models apply it before each sublayer (Pre-LN), a variant that trains more stably and reduces dependence on careful learning-rate warm-up (Xiong et al., 2020).

Numerical example (token banco, causal version, γ = 1, β = 0). The attention output projected by Wo is o = [1.074; 1.912; 0.537; 0.956]. The input was x = [0.841; 1.540; 0.510; 1.000]. Residual addition gives r = x + o = [1.915; 3.453; 1.047; 1.956]. The mean is μ = 2.093 and the variance is σ² = 0.748. LayerNorm with ε = 10⁻⁵ gives [(1.915 − 2.093)/0.865; …] = [−0.205; 1.572; −1.209; −0.158], with mean 0 and variance 1. By comparison, RMSNorm of the same vector, with root mean square 2.264, yields [0.846; 1.525; 0.462; 0.864]: it preserves the proportions among components without centering them at zero.

8. Stage 6 — Feed-forward network (FFN)

Why this stage is necessary. In attention, the weights depend on content, but the output is a weighted average of Values obtained through linear projections (V = X·Wv, followed by Wo); the only nonlinearity is the softmax that determines the weights. The FFN adds a nonlinear transformation applied to each token after contextual information has been assembled. There is also a capacity rationale: FFNs contain roughly two-thirds of a Transformer’s parameters and act as memories for patterns (Geva et al., 2021). Expanding to d_ff = 4 × d_model in the original model is a design choice that increases this capacity (Vaswani et al., 2017).

Concept. After attention, each position passes through a small neural network with two linear layers and a nonlinearity between them, applied independently to each token with the same weights at every position in the layer (Vaswani et al., 2017):

Equation 5. Mathematical notation rendered as a high-resolution image for reliable display. Copyright 2026 Aridio Silva | CC BY 4.0.

Attention mixes information between tokens. The FFN transforms information within each token, expanding it to a larger dimension d_ff (2048 in the original model, or 4 × d_model), applying the nonlinearity, and compressing it back to d_model.

Activation functions. ReLU is used in the original Transformer; GELU in BERT and GPT (Hendrycks and Gimpel, 2016; Devlin et al., 2019; Radford et al., 2019); and gated variants of GLU, such as SwiGLU, in more recent models (Shazeer, 2020; Touvron et al., 2023).

Interpretation. Geva et al. (2021) propose that the FFN functions as a key–value memory: columns of W1 detect patterns in the input vector (keys), while rows of W2 contribute associated information (values). This interpretation helps explain why much factual knowledge appears to reside in FFNs rather than attention. It is an interpretive hypothesis supported by empirical analysis, not a fact demonstrated for every model.

Numerical parameter example (base Transformer). W1 contains 512 × 2048 = 1,048,576 weights and W2 contains 2048 × 512 = 1,048,576, totaling 2,097,152 per layer without biases. Adding the 1,048,576 attention weights (Section 6), the layer contains approximately 3.15 million weights, roughly two-thirds of which belong to the FFN.

Numerical computation example (d_model = 4, d_ff = 8; W1 and W2 generated randomly with a fixed seed and no training). For the token estava at position 2, the FFN input is the normalized vector [1.389; −0.901; −0.999; 0.511]. After W1 and ReLU, the hidden layer is [0.907; 0.660; 0.330; 0; 0.623; 0; 1.541; 0.759]: six of the eight units are active and two are zeroed by ReLU. After W2, the output is [0.231; 0.954; 0.437; −0.291]. Adding it to the input through the residual path and normalizing yields [1.613; −0.351; −1.121; −0.141]. For banco at position 1, all eight hidden units are zero in this example, so the FFN output is zero and the vector crosses the sublayer unchanged through the residual path. In a trained model, the sparse activation pattern is learned, and each token activates a different subset of units.

9. Stage 7 — Multilayer stacking

Why this stage is necessary. One layer captures relationships in one step. Many tasks require composition: identify the token, relate it to context, infer its sense, and only then predict. Stacking enables successive refinement, and probing analyses reveal different levels of information across layers (Jawahar et al., 2019; Tenney et al., 2019; Belrose et al., 2023). Some documented mechanisms, such as induction heads, require at least two attention layers (Elhage et al., 2021; Olsson et al., 2022). More layers increase both capacity and cost, so depth is a design choice, as shown below.

Mechanism. A Transformer block follows the sequence attention → Add & Norm → FFN → Add & Norm. The model stacks N blocks: the output of layer l becomes the input to layer l + 1, and each layer has its own weights. The n × d_model matrix retains the same shape across the stack; what changes is the content of each vector as it is progressively refined.

What different depths tend to represent. Probing studies in BERT indicate an approximate progression: early layers capture surface and local-order patterns; intermediate layers capture syntactic structure; and higher layers capture semantic and long-range relationships (Jawahar et al., 2019). Tenney et al. (2019) observed that stages of the classical NLP pipeline—part-of-speech tagging, syntactic parsing, entities, semantic roles, and coreference—can be recovered in that order across the layers. Figure 5 summarizes this tendency.

Iterative refinement of the prediction. Techniques such as the logit lens and tuned lens project the intermediate state of each layer onto the vocabulary and show that the predicted next-token distribution becomes more precise layer by layer (Belrose et al., 2023).

Circuits that require depth. Induction heads are a documented example: one head in an earlier layer copies information from the previous token, and a later head uses it to locate past occurrences of a pattern and predict its continuation. This mechanism requires at least two attention layers and has been associated with the emergence of in-context learning during training (Elhage et al., 2021; Olsson et al., 2022).

Caveat. Layer boundaries vary with the model, task, and analytical method. Layer ranges used in didactic explanations—for example, “layer 6 resolves the sense of banco”—are illustrations, not results measured in this article.

10. Stage 8 — Weights: what they are and how they are learned

Why this stage is necessary. With random weights, the previous stages produce outputs unrelated to language, as in the numerical examples in this article. Training is what turns the architecture into a useful model. Next-token prediction is an effective objective because it provides a learning signal from any text without human labels. Backpropagation is necessary because millions to billions of parameters can only be adjusted using efficiently computed gradients (Rumelhart et al., 1986), while the optimizer, warm-up, and regularization stabilize those updates (Kingma and Ba, 2015; Loshchilov and Hutter, 2019; Vaswani et al., 2017). This stage also explains why inference uses fixed weights: what the model has learned is already encoded in them.

What weights are. Weights, or parameters, are the adjustable numbers in the model: the embedding matrix; Wq, Wk, Wv, Wo, W1, W2, biases, and the normalization parameters γ and β in each layer; and the final vocabulary projection. They remain fixed during inference. Everything the model has learned is encoded in these values.

Training objective. Generative models are trained to predict the next token. Given a sequence x₁…x_T, training minimizes the mean cross-entropy between the predicted distribution and the token that actually follows:

Equation 6. Mathematical notation rendered as a high-resolution image for reliable display. Copyright 2026 Aridio Silva | CC BY 4.0.

BERT uses a different objective, masked language modeling, in which some tokens are hidden and the model reconstructs them using context from both directions (Devlin et al., 2019). In both cases, there is no human label for meaning: the signal is the text itself. This supports the idea that disambiguation is an emergent strategy for reducing prediction error (Section 13).

Numerical example. In the output computed in Section 11 using random weights, the probability assigned to lotado is p = 0.022 and the probability assigned to cheio is 0.623. If the correct token were lotado, the loss would be −ln(0.022) = 3.806; if it were cheio, the loss would be −ln(0.623) = 0.473. The loss gradient with respect to the logits is simply p − y, where y is the one-hot vector for the correct token. For target lotado, the gradient is [−0.978; 0.114; 0.623; 0.094; 0.147], meaning that training pushes the lotado logit upward and the others downward in proportion to the error.

Backpropagation and optimization. The gradient is propagated backward through all layers by the chain rule, an algorithm known as backpropagation (Rumelhart et al., 1986), and an optimizer updates the weights. The original Transformer used Adam (Kingma and Ba, 2015) with a learning rate that grows linearly during warm-up and then decays, together with dropout and label smoothing (Vaswani et al., 2017). AdamW, which decouples weight-decay regularization from the adaptive update, is a common choice in current models (Loshchilov and Hutter, 2019).

Parallelism in training and sequence in inference. During training with a causal mask, all positions are computed at once because the correct text is already known. During inference, token t + 1 depends on the selected token t, so generation is sequential, and the KV cache (Section 6) prevents recomputation of the past.

Training phases. A common practice is pretraining on a large volume of raw text, followed by adaptation for specific use through supervised fine-tuning and alignment with human preferences, such as reinforcement learning from human feedback (Ouyang et al., 2022).

Scale. Language-model loss decreases regularly according to power laws as parameters, data, and compute increase (Kaplan et al., 2020). Hoffmann et al. (2022) showed that, for a fixed compute budget, parameters and data volume should grow proportionally and that many large models had been undertrained on data.

11. Stage 9 — Output: logits, softmax, and decoding

Why this stage is necessary. The final layer produces a vector of d_model values, while the user needs a token. Logits map that vector to one score per vocabulary token. Softmax converts the scores into a valid probability distribution—positive values summing to 1—which is required both for cross-entropy loss during training and for selecting or sampling a token during inference. Temperature and decoding rules are necessary because always choosing the most likely token tends to produce repetitive, degenerate text, while unrestricted sampling includes implausible tokens; top-k and top-p strategies balance these extremes (Holtzman et al., 2020).

Logits. The final-layer vector at the current position (h_final, of dimension d_model) is multiplied by an output matrix W_out of dimension d_model × |V|. The result is the logits: one unbounded real number for each vocabulary token. In many models, W_out is the transpose of the embedding matrix, a technique known as weight tying (Press and Wolf, 2017; Vaswani et al., 2017).

Softmax and temperature. Softmax converts logits into probabilities that sum to 1. Temperature T divides the logits before softmax: T < 1 concentrates probability on the most likely tokens, producing more deterministic output, whereas T > 1 spreads probability and produces greater variation. Temperature-scaled softmax was popularized in model distillation (Hinton et al., 2015).

Equation 7. Mathematical notation rendered as a high-resolution image for reliable display. Copyright 2026 Aridio Silva | CC BY 4.0.

Numerical example. Consider a didactic vocabulary of five tokens and logits calculated from the vector at the position estava, using random untrained output weights: lotado −0.818; vazio 0.815; cheio 2.516; aberto 0.626; verde 1.070. With T = 1, softmax gives lotado 0.022; vazio 0.114; cheio 0.623; aberto 0.094; verde 0.147, summing to 1.000. With T = 0.5, cheio rises to 0.899; with T = 2, it falls to 0.402 and the distribution becomes flatter (Figure 6). In a trained model, these weights would be adjusted so that plausible words after “O banco estava,” such as lotado, receive high probability; here the example demonstrates only the arithmetic.

Decoding strategies. Selecting a token from the distribution is a separate step from the model itself:

• Greedy: always chooses the highest-probability token; in this example, cheio.
• Sampling: draws a token according to the probabilities.
• Top-k: restricts sampling to the k most likely tokens and renormalizes (Fan et al., 2018). With k = 2: cheio 0.809 and verde 0.191.
• Top-p (nucleus): restricts sampling to the smallest set whose cumulative probability reaches p (Holtzman et al., 2020). With p = 0.9, the set is {cheio, verde, vazio, aberto}, with cumulative probability 0.978, renormalized to 0.637; 0.150; 0.117; 0.096.
• Beam search: maintains several candidate sequences in parallel and selects the one with the highest total probability.

Closing the cycle. The selected token is appended to the input, and the entire flow repeats while reusing the KV cache until an end-of-sequence token is generated or the length limit is reached.

12. Case study: disambiguating banco and interpreting the mechanism

Purpose of this section. It integrates Stages 3–9 into one minimal example. Without it, each mechanism would remain isolated and it would be unclear how they jointly resolve the ambiguity of banco.

The problem. The Portuguese word banco may denote either a financial institution or a bench. Tokenization assigns the same ID in both cases, and the input embedding is identical (Sections 3 and 4). The model does not consult a dictionary or apply explicit if … then rules. What distinguishes the senses is context: attention mixes information from neighboring tokens into the vector for banco, and in subsequent layers this mixed vector behaves differently.

Empirical evidence. In trained models, contextualized vectors for the same word tend to form separate clusters by sense (Reif et al., 2019), and representations become more context-specific in higher layers (Ethayarajh, 2019).

Numerical demonstration (didactic toy). Hand-selected four-dimensional embeddings use the axes [money, seat, people, wood]: banco = [0.7; 0.7; 0; 0] (ambiguous), lotado = [0; 0; 1; 0], clientes = [0.8; 0; 0.7; 0], praça = [0; 0.8; 0; 0.2], madeira = [0; 0.3; 0; 1], velho = [0; 0.2; 0; 0.5], estava = [0; 0; 0.1; 0]; the articles O, da, and de are null vectors. One-head attention uses Wq = Wk = 2·I, Wv = I, and d_k = 4. In each context, the contextualized vector for banco is computed and compared by cosine similarity with two prototypes: financial institution = [1; 0; 0.5; 0] and seat = [0; 1; 0; 0.5].

In isolation, the vector is equally distant from both senses (0.632 versus 0.632). In the context lotado de clientes, it moves closer to the financial-institution prototype (0.851 versus 0.468). In the context praça … madeira, it moves closer to the seat prototype (0.858 versus 0.452). The mechanism is the same in both cases; only the neighboring tokens change. In a real model, Wq, Wk, and Wv and the knowledge in the FFNs result from training, and the distinction is distributed across many heads and layers rather than one step in a toy example.

Why this emerges from training. To predict the next word in sentences such as “O banco estava lotado de ___,” the model must represent whether the sentence concerns a branch office or a seat. Statistically, lotado co-occurs with customers, queues, and branches. Sense representation is therefore a useful by-product of minimizing prediction error (Section 13).

Caution when interpreting attention maps. Visualizing attention weights—for example, with tools such as Vig (2019)—is a useful first analytical step, but attention weight is not equivalent to causal explanation: different weights can produce the same prediction, and the attention in one layer is only one part of the computation (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). Claims about what the model “uses” require intervention experiments or circuit analysis (Elhage et al., 2021; Olsson et al., 2022).

13. Emergent properties and scaling laws

Purpose of this section. It explains how a behavior such as disambiguation can exist without being explicitly programmed, linking the result of Section 12 to the training process in Section 10. It frames the debate on emergence without claiming more than the literature supports.

Concept. An emergent property is a characteristic of the system as a whole that is not programmed into any single component and arises from large-scale interaction among components (Anderson, 1972). Transformer code contains only matrix multiplications, additions, normalizations, and softmax; the training objective is only to predict the next token. Nothing explicitly names word senses, syntax, or reasoning, yet representations of this kind appear in the learned weights (Sections 9 and 12). A common analogy is an ant colony: no individual ant “knows” how to build a bridge, yet the colony builds one.

Capabilities that emerge with scale. Wei et al. (2022a) define an emergent capability as one that is absent in smaller models and present in larger models such that its appearance cannot be predicted by simple extrapolation from the smaller-model curves. Examples discussed in the literature include few-shot prompting in large models (Brown et al., 2020) and chain-of-thought reasoning (Wei et al., 2022b).

Scaling laws. While average prediction loss improves smoothly and predictably with parameters, data, and compute (Kaplan et al., 2020; Hoffmann et al., 2022), performance on specific tasks may appear to change abruptly. Ganguli et al. (2022) discuss this combination of predictability in loss and surprise in capabilities.

Debate: genuine emergence or a metric effect? Schaeffer et al. (2023) argue that some observed jumps result from discontinuous metrics, such as exact match, and that continuous metrics applied to the same models reveal gradual progress. The debate remains open, and the literature advises caution when claiming that a capability “suddenly appears.”

Do not confuse this with grokking. Grokking denotes a distinct phenomenon: in small algorithmic tasks, generalization appears long after the model has already memorized the training set (Power et al., 2022), and it has been analyzed using mechanistic interpretability techniques (Nanda et al., 2023). Grokking occurs over training time, whereas scale-based emergence occurs across model sizes.

Application to the banco case. Disambiguation of common senses is already observed in relatively small models such as BERT-base, with 110 million parameters (Reif et al., 2019). There is therefore no basis for asserting a fixed layer or parameter threshold at which the capability “appears.” The literature supports the conclusions that the capability results from training and that its quality and scope improve with scale.

Implications. If capabilities can arise without being explicitly designed, evaluation and interpretability become central to development: the model must be tested, not merely inspected at the code level, to determine what it does (Ganguli et al., 2022; Olsson et al., 2022).

14. Limitations, caveats, and open debates

Purpose of this section. It defines what the article does and does not demonstrate. This is necessary because the examples are simplified and because the mechanism itself has limitations and unresolved questions that readers should understand.

Limitations of this article.

• The numerical examples use very small dimensions, a single head, and hand-selected or random weights. They demonstrate the arithmetic, not the behavior of a trained model.
• For simplicity, the article omits biases in all projections, dropout, cross-attention in the decoder of the original model, and variants such as mixtures of experts and attention alternatives such as state-space models (Gu and Dao, 2023).
• Details such as the exact number of layers, activation function, normalization type, and positional encoding vary among models and are not always disclosed.

Debates and limitations of the mechanism itself.

• Attention and explanation. Attention weights are not, by themselves, a reliable explanation of model behavior (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019).
• Quadratic cost and long context. Attention cost grows with the square of sequence length, and long-context models may use information located in the middle of the context less effectively (Liu et al., 2024).
• Hallucination. The model computes likely tokens given the context; the architecture contains no step that verifies the truth of what it generates. Fluent but factually incorrect output is a documented problem (Ji et al., 2023).
• “Only statistics” or a “world model”? Positions diverge. One argues that models trained only on textual form cannot attain understanding of meaning (Bender and Koller, 2020). Another points to evidence that models trained for next-token prediction can construct internal representations of domain structure, such as the state of an Othello board (Li et al., 2023). The characterization of “understanding” remains contested, and this article is limited to the mechanisms and cited evidence.
• Emergence. See Section 13: the nature of capability jumps with scale remains debated (Wei et al., 2022a; Schaeffer et al., 2023).

15. From inference to governed execution: implications for SGAEIA

Purpose of this section. This section bridges the Transformer’s internal mechanics and the architectural problem investigated by SGAEIA. Its epistemic classification is SGAEIA Interpretation: it is a public conceptual reading of the mechanisms described in this article, not evidence of SGAEIA implementation, effectiveness, certification, or production validation.

Within the scope of this article, inference ends with a probability distribution and the selection of a token. An agentic system adds other layers: memory, planning, tools, credentials, delegation, policies, runtime, and the ability to modify external state. Model output should therefore be treated as a proposal or computational signal, not as authorization to produce an effect. This separation between model capability and governed action is central to the public SGAEIA architecture and its analysis of the transition from model to agent (Silva, 2026a; Silva, 2026b; Silva, 2026c).

This separation is especially important because partial interpretability does not eliminate the need for external controls. Attention weights, intermediate states, generated explanations, and chains of reasoning can provide useful signals, but none independently demonstrates the complete cause of a decision, the legitimacy of an action, or the future stability of behavior. The public SGAEIA formulation treats this limitation as a supervision problem: the less confidence there is in understanding the internal process, the less security can depend exclusively on that understanding and the more it must depend on bounded authority, mediated execution, revocation, and recovery (Silva, 2026d).

Observability must also be distinguished from evidence. Recording a prompt and response helps reconstruct part of an interaction, but it is insufficient to establish which identity initiated the operation, which policy was in force, what authority permitted a tool, what delegations occurred, and what effect was actually observed. In distributed or multi-agent systems, traces and spans help reconstruct trajectories, while evaluation and governed evidence answer different questions about outcome, legitimacy, and assurance. This relationship among tracing, evaluation, and evidence is examined in SGAEIA’s public research on agent execution (Silva, 2026e).

The connection to SGAEIA does not turn the article’s didactic examples into architectural controls. The table organizes high-level research questions and public properties while preserving the boundary between explanation and private mechanism. The operational conclusion is limited but important: understanding how an LLM produces a token is necessary to govern an LLM-based system, but it is not sufficient to authorize, observe, contain, or assure that system’s effects.

16. Conclusion

Opening the black box of inference makes it possible to follow the transformation of text into IDs, vectors, attention relationships, contextual representations, logits, and probabilities. The reduced examples show the arithmetic of each stage and make explicit where learned weights, masks, normalization, nonlinearity, and the decoding rule enter the process. They also clarify the limit of the demonstration: didactic numbers explain the mechanism, but they do not measure a real model and do not independently support claims about reasoning, safety, or reliability.

The banco case study shows why a token representation does not remain static after context passes through the layers. At the same time, the caveats concerning attention maps, emergence, and hallucination prevent a mathematical explanation from being confused with complete causal transparency. Inference is probabilistic, contextual, and dependent on weights, configuration, and decoding strategy; its analysis must preserve those conditions rather than convert fluency into certainty.

For SGAEIA, the principal implication is the separation between inference and effect. The model may propose a path, but authority determines whether that path is reachable, and policies and evidence determine whether it can become a legitimate action. Opening the black box of an agentic system therefore requires explaining not only how the token was produced, but also who authorized each action, under what limits, with what records, and with what capacity for interruption, revocation, and recovery.

17. Glossary of terms, concepts, and techniques

Purpose of this section. It fixes the terminology used in the article. This is necessary because the terms come from distinct fields, including computational linguistics, optimization, and interpretability.

Alphabetical order. The “Section” column indicates where each term is developed.

18. Bibliography / References

• Ainslie, J.; Lee-Thorp, J.; de Jong, M.; Zemlyanskiy, Y.; Lebrón, F.; Sanghai, S. (2023). GQA: Training generalized multi-query Transformer models from multi-head checkpoints. In: EMNLP 2023. arXiv:2305.13245. https://arxiv.org/abs/2305.13245
• Anderson, P. W. (1972). More is different. Science, 177(4047), 393–396. https://doi.org/10.1126/science.177.4047.393
• Ba, J. L.; Kiros, J. R.; Hinton, G. E. (2016). Layer normalization. arXiv:1607.06450. https://arxiv.org/abs/1607.06450
• Bahdanau, D.; Cho, K.; Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In: ICLR 2015. arXiv:1409.0473. https://arxiv.org/abs/1409.0473
• Belrose, N.; Furman, Z.; Smith, L.; Halawi, D.; Ostrovsky, I.; McKinney, L.; Biderman, S.; Steinhardt, J. (2023). Eliciting latent predictions from Transformers with the tuned lens. arXiv:2303.08112. https://arxiv.org/abs/2303.08112 [link auditado] [conferido]
• Bender, E. M.; Koller, A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. In: ACL 2020, p. 5185–5198. https://aclanthology.org/2020.acl-main.463/
• Bengio, Y.; Ducharme, R.; Vincent, P.; Jauvin, C. (2003). A neural probabilistic language model. Journal of Machine Learning Research, 3, 1137–1155. https://www.jmlr.org/papers/v3/bengio03a.html
• Brown, T. B. et al. (2020). Language models are few-shot learners. In: NeurIPS 33. arXiv:2005.14165. https://arxiv.org/abs/2005.14165
• Clark, K.; Khandelwal, U.; Levy, O.; Manning, C. D. (2019). What does BERT look at? An analysis of BERT's attention. In: BlackboxNLP Workshop at ACL 2019, p. 276–286. arXiv:1906.04341. https://arxiv.org/abs/1906.04341 [link auditado]
• Dao, T.; Fu, D. Y.; Ermon, S.; Rudra, A.; Ré, C. (2022). FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In: NeurIPS 35. arXiv:2205.14135. https://arxiv.org/abs/2205.14135
• Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. (2019). BERT: Pre-training of deep bidirectional Transformers for language understanding. In: NAACL-HLT 2019, p. 4171–4186. arXiv:1810.04805. https://arxiv.org/abs/1810.04805
• Elhage, N. et al. (2021). A mathematical framework for Transformer circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html
• Ethayarajh, K. (2019). How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In: EMNLP-IJCNLP 2019, p. 55–65. arXiv:1909.00512. https://aclanthology.org/D19-1006/ [link auditado] [conferido]
• Fan, A.; Lewis, M.; Dauphin, Y. (2018). Hierarchical neural story generation. In: ACL 2018, p. 889–898. arXiv:1805.04833. https://arxiv.org/abs/1805.04833
• Gage, P. (1994). A new algorithm for data compression. The C Users Journal, 12(2), 23–38. https://www.derczynski.com/papers/archive/BPE_Gage.pdf
• Ganguli, D. et al. (2022). Predictability and surprise in large generative models. In: ACM FAccT 2022, p. 1747–1764. arXiv:2202.07785. https://arxiv.org/abs/2202.07785
• Geva, M.; Schuster, R.; Berant, J.; Levy, O. (2021). Transformer feed-forward layers are key-value memories. In: EMNLP 2021, p. 5484–5495. arXiv:2012.14913. https://aclanthology.org/2021.emnlp-main.446 [link auditado] [conferido]
• Gu, A.; Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv:2312.00752. https://arxiv.org/abs/2312.00752
• He, K.; Zhang, X.; Ren, S.; Sun, J. (2016). Deep residual learning for image recognition. In: CVPR 2016, p. 770–778. arXiv:1512.03385. https://arxiv.org/abs/1512.03385
• Hendrycks, D.; Gimpel, K. (2016). Gaussian error linear units (GELUs). arXiv:1606.08415. https://arxiv.org/abs/1606.08415
• Hinton, G.; Vinyals, O.; Dean, J. (2015). Distilling the knowledge in a neural network. arXiv:1503.02531. https://arxiv.org/abs/1503.02531
• Hoffmann, J. et al. (2022). Training compute-optimal large language models. In: NeurIPS 35. arXiv:2203.15556. https://arxiv.org/abs/2203.15556
• Holtzman, A.; Buys, J.; Du, L.; Forbes, M.; Choi, Y. (2020). The curious case of neural text degeneration. In: ICLR 2020. arXiv:1904.09751. https://arxiv.org/abs/1904.09751
• Jain, S.; Wallace, B. C. (2019). Attention is not explanation. In: NAACL-HLT 2019, p. 3543–3556. arXiv:1902.10186. https://aclanthology.org/N19-1357/ [link auditado] [conferido]
• Jawahar, G.; Sagot, B.; Seddah, D. (2019). What does BERT learn about the structure of language? In: ACL 2019, p. 3651–3657. https://aclanthology.org/P19-1356/ [link auditado] [conferido]
• Ji, Z. et al. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), art. 248. arXiv:2202.03629. https://arxiv.org/abs/2202.03629 [link auditado]
• Kaplan, J. et al. (2020). Scaling laws for neural language models. arXiv:2001.08361. https://arxiv.org/abs/2001.08361
• Kingma, D. P.; Ba, J. (2015). Adam: A method for stochastic optimization. In: ICLR 2015. arXiv:1412.6980. https://arxiv.org/abs/1412.6980
• Kudo, T. (2018). Subword regularization: Improving neural network translation models with multiple subword candidates. In: ACL 2018, p. 66–75. arXiv:1804.10959. https://arxiv.org/abs/1804.10959
• Kudo, T.; Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In: EMNLP 2018 (System Demonstrations), p. 66–71. arXiv:1808.06226. https://arxiv.org/abs/1808.06226
• Li, K.; Hopkins, A. K.; Bau, D.; Viégas, F.; Pfister, H.; Wattenberg, M. (2023). Emergent world representations: Exploring a sequence model trained on a synthetic task. In: ICLR 2023. arXiv:2210.13382. https://arxiv.org/abs/2210.13382
• Liu, N. F. et al. (2024). Lost in the middle: How language models use long contexts. Transactions of the ACL, 12, 157–173. arXiv:2307.03172. https://aclanthology.org/2024.tacl-1.9/ [link auditado] [conferido]
• Loshchilov, I.; Hutter, F. (2019). Decoupled weight decay regularization. In: ICLR 2019. arXiv:1711.05101. https://arxiv.org/abs/1711.05101
• Michel, P.; Levy, O.; Neubig, G. (2019). Are sixteen heads really better than one? In: NeurIPS 32. arXiv:1905.10650. https://arxiv.org/abs/1905.10650
• Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781. https://arxiv.org/abs/1301.3781
• Nanda, N.; Chan, L.; Lieberum, T.; Smith, J.; Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. In: ICLR 2023. arXiv:2301.05217. https://arxiv.org/abs/2301.05217
• Olsson, C. et al. (2022). In-context learning and induction heads. Transformer Circuits Thread. arXiv:2209.11895. https://arxiv.org/abs/2209.11895
• Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. In: NeurIPS 35. arXiv:2203.02155. https://arxiv.org/abs/2203.02155
• Pope, R.; Douglas, S.; Chowdhery, A.; Devlin, J.; Bradbury, J.; Levskaya, A.; Heek, J.; Xiao, K.; Agrawal, S.; Dean, J. (2023). Efficiently scaling Transformer inference. Proceedings of Machine Learning and Systems, 5 (MLSys 2023). arXiv:2211.05102. https://arxiv.org/abs/2211.05102 [link auditado] [conferido]
• Power, A.; Burda, Y.; Edwards, H.; Babuschkin, I.; Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177. https://arxiv.org/abs/2201.02177
• Press, O.; Smith, N. A.; Lewis, M. (2022). Train short, test long: Attention with linear biases enables input length extrapolation. In: ICLR 2022. arXiv:2108.12409. https://arxiv.org/abs/2108.12409
• Press, O.; Wolf, L. (2017). Using the output embedding to improve language models. In: EACL 2017, p. 157–163. arXiv:1608.05859. https://arxiv.org/abs/1608.05859
• Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI, technical report. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
• Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI, technical report. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
• Reif, E.; Yuan, A.; Wattenberg, M.; Viegas, F. B.; Coenen, A.; Pearce, A.; Kim, B. (2019). Visualizing and measuring the geometry of BERT. In: NeurIPS 32, p. 8594–8603. arXiv:1906.02715. https://arxiv.org/abs/1906.02715 [link auditado] [conferido]
• Rumelhart, D. E.; Hinton, G. E.; Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. https://doi.org/10.1038/323533a0
• Schaeffer, R.; Miranda, B.; Koyejo, S. (2023). Are emergent abilities of large language models a mirage? In: NeurIPS 2023 (v. 36). arXiv:2304.15004. https://arxiv.org/abs/2304.15004 [link auditado] [conferido]
• Schuster, M.; Nakajima, K. (2012). Japanese and Korean voice search. In: ICASSP 2012, p. 5149–5152. https://doi.org/10.1109/ICASSP.2012.6289079
• Sennrich, R.; Haddow, B.; Birch, A. (2016). Neural machine translation of rare words with subword units. In: ACL 2016, p. 1715–1725. arXiv:1508.07909. https://arxiv.org/abs/1508.07909
• Shaw, P.; Uszkoreit, J.; Vaswani, A. (2018). Self-attention with relative position representations. In: NAACL-HLT 2018, p. 464–468. arXiv:1803.02155. https://arxiv.org/abs/1803.02155
• Shazeer, N. (2019). Fast Transformer decoding: One write-head is all you need. arXiv:1911.02150. https://arxiv.org/abs/1911.02150
• Shazeer, N. (2020). GLU variants improve Transformer. arXiv:2002.05202. https://arxiv.org/abs/2002.05202
• Silva, Aridio. (2026a). SGAEIA: Secure Governed Autonomous Edge Intelligence Architecture (Version 0.3.4). Zenodo. https://doi.org/10.5281/zenodo.22557796
• Silva, Aridio. (2026b). SGAEIA: A Reference Architecture for Secure Governed Autonomous Edge Intelligence. SGAEIA Research Series, Article 8. Zenodo. https://doi.org/10.5281/zenodo.22882020
• Silva, Aridio. (2026c). From Model Capability to Governed Action: An Architecture for Secure Agentic AI. SGAEIA Research Series, Article 11. Zenodo. https://doi.org/10.5281/zenodo.22837908
• Silva, Aridio. (2026d). AI's Alien Mind. SGAEIA Research Series, Article 14. Zenodo. https://doi.org/10.5281/zenodo.22964672
• Silva, Aridio. (2026e). Tracing AI Agent Execution: Foundations, State of the Art, and Implications for Distributed Multi-Agent Systems. SGAEIA Research Series, Article 18. Zenodo. https://doi.org/10.5281/zenodo.23284753
• Su, J.; Lu, Y.; Pan, S.; Murtadha, A.; Wen, B.; Liu, Y. (2021). RoFormer: Enhanced Transformer with rotary position embedding. arXiv:2104.09864. https://arxiv.org/abs/2104.09864
• Tenney, I.; Das, D.; Pavlick, E. (2019). BERT rediscovers the classical NLP pipeline. In: ACL 2019, p. 4593–4601. arXiv:1905.05950. https://aclanthology.org/P19-1452/ [link auditado] [conferido]
• Touvron, H. et al. (2023). LLaMA: Open and efficient foundation language models. arXiv:2302.13971. https://arxiv.org/abs/2302.13971
• Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; Polosukhin, I. (2017). Attention is all you need. In: Advances in Neural Information Processing Systems 30 (NIPS 2017), p. 5998–6008. arXiv:1706.03762. https://papers.neurips.cc/paper/7181-attention-is-all-you-need [link auditado] [conferido]
• Vig, J. (2019). A multiscale visualization of attention in the Transformer model. In: ACL 2019 (System Demonstrations), p. 37–42. arXiv:1906.05714. https://arxiv.org/abs/1906.05714
• Voita, E.; Talbot, D.; Moiseev, F.; Sennrich, R.; Titov, I. (2019). Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In: ACL 2019, p. 5797–5808. arXiv:1905.09418. https://arxiv.org/abs/1905.09418
• Wei, J.; Tay, Y.; Bommasani, R. et al. (2022a). Emergent abilities of large language models. Transactions on Machine Learning Research. arXiv:2206.07682. https://arxiv.org/abs/2206.07682 [link auditado] [conferido]
• Wei, J.; Wang, X.; Schuurmans, D. et al. (2022b). Chain-of-thought prompting elicits reasoning in large language models. In: NeurIPS 35. arXiv:2201.11903. https://arxiv.org/abs/2201.11903
• Wiegreffe, S.; Pinter, Y. (2019). Attention is not not explanation. In: EMNLP-IJCNLP 2019, p. 11–20. arXiv:1908.04626. https://arxiv.org/abs/1908.04626
• Wu, Y. et al. (2016). Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. https://arxiv.org/abs/1609.08144
• Xiong, R. et al. (2020). On layer normalization in the Transformer architecture. In: ICML 2020 (PMLR 119). arXiv:2002.04745. https://arxiv.org/abs/2002.04745
• Zhang, B.; Sennrich, R. (2019). Root mean square layer normalization. In: NeurIPS 32. arXiv:1910.07467. https://arxiv.org/abs/1910.07467

Reproducibility. The numerical examples in Sections 3–12 were computed in Python 3 with NumPy; W1, W2, and W_out were generated with a fixed random seed (0) and normal distributions with standard deviations 0.5 (W1, W2) and 0.8 (W_out). The programs are reproduced in Section 19; the parameter counts in Sections 6 and 8 are direct arithmetic and require no script.

19. Source code (Python)

Purpose of this section. It makes the article’s numbers, figures, and auxiliary files reproducible. The numerical examples have didactic value only if readers can rerun them and obtain the same results.

The programs below are reproduced as executed. Run 19.1 first, then 19.2, which reads the file written by 19.1; programs 19.3 and 19.4 can follow; 19.5 uses the files from 19.1 and 19.2. Dependencies are Python 3, NumPy, Matplotlib, and requests. The calculations in 19.1–19.4 were rerun from scratch before publication and reproduced the values cited in Sections 4–7, 11, and 12. Under this edition’s editorial policy, routines used exclusively to write PNG or PDF metadata are not included in the document.

Not reproduced are: (a) the parameter counts in Sections 6 and 8, which are direct arithmetic performed without a script; (b) editing tools used to insert text and links; and (c) conversion commands using Pandoc and XeLaTeX.

19.1 Numerical core of the didactic example

File: 01_nucleo_numerico_calc.py. Function. Defines the four tokens (“O”, “banco”, “estava”, “lotado”) with hand-selected four-dimensional embeddings; computes sinusoidal positional encoding and X = E + P; applies Q, K, and V projections (fixed matrices, d_k = 2), QKᵀ scores, division by √d_k, and bidirectional softmax; and repeats softmax with a causal mask. It then traverses the rest of the block—Wo projection, residual addition, LayerNorm, RMSNorm, an FFN with d_ff = 8 and random weights using seed 0, logits over five words, softmax with T = 0.5, 1, and 2, and sorting for top-p—and writes the results to toy.npz.

Supports. Values in Sections 4–6 and Figures 2 and 4. The FFN and output results produced here use the bidirectional chain and are not those cited in the text; Sections 7–11 use Program 19.2, which recomputes the chain with a causal mask.

import numpy as np, json
np.set_printoptions(precision=3, suppress=True)
tok=["O","banco","estava","lotado"]
# Didactic embeddings, d_model=4
E=np.array([[1.0,0.0,0.0,0.0],
            [0.0,1.0,0.5,0.0],
            [0.0,0.3,1.0,0.0],
            [0.0,0.8,0.0,1.0]])
d=4
def pe(n,d):
    P=np.zeros((n,d))
    for pos in range(n):
        for i in range(0,d,2):
            P[pos,i]=np.sin(pos/10000**(i/d)); P[pos,i+1]=np.cos(pos/10000**(i/d))
    return P
P=pe(4,d); print("PE\n",P)
X=E+P; print("X\n",X)
Wq=np.array([[1,0],[0,1],[1,0],[0,1]],float)*0.9
Wk=np.array([[0.5,0.2],[1,0],[0,1],[0.3,0.8]],float)
Wv=np.array([[1,0],[0,1],[0.5,0],[0,0.5]],float)
Q=X@Wq;K=X@Wk;V=X@Wv
print("Q\n",Q,"\nK\n",K,"\nV\n",V)
S=Q@K.T; print("S\n",S)
dk=2; Ss=S/np.sqrt(dk); print("S/sqrt\n",Ss)
def sm(z): z=z-z.max(-1,keepdims=True); e=np.exp(z); return e/e.sum(-1,keepdims=True)
A=sm(Ss); print("A\n",A, A.sum(1))
Z=A@V; print("Z\n",Z)
# causal
mask=np.triu(np.ones((4,4)),1).astype(bool)
Sc=Ss.copy(); Sc[mask]=-np.inf; Ac=sm(Sc); print("Acausal\n",Ac)
# project back with Wo (2->4)
Wo=np.array([[1,0,0.5,0],[0,1,0,0.5]],float)
O=Z@Wo; print("O\n",O)
R=X+O; print("R\n",R)
def ln(x,eps=1e-5): m=x.mean(-1,keepdims=True); v=x.var(-1,keepdims=True); return (x-m)/np.sqrt(v+eps)
Y=ln(R); print("LN\n",Y)
# rms
rms=R/np.sqrt((R**2).mean(-1,keepdims=True)+1e-5); print("RMS\n",rms)
# FFN d_ff=8 ReLU
rng=np.random.default_rng(0)
W1=rng.normal(0,0.5,(4,8)); b1=np.zeros(8); W2=rng.normal(0,0.5,(8,4)); b2=np.zeros(4)
H=np.maximum(0,Y@W1+b1); F=H@W2+b2
print("W1\n",W1,"\nW2\n",W2)
print("H\n",H,"\nF\n",F)
Y2=ln(Y+F); print("Y2\n",Y2)
# output: vocab
vocab=["lotado","vazio","cheio","aberto","verde"]
Wout=rng.normal(0,0.8,(4,5))
lg=Y2[2]@Wout  # posição de "estava" prevê próxima palavra
print("Wout\n",Wout,"\nlogits",lg)
p=sm(lg); print("p",p, p.sum())
for T in [0.5,1.0,2.0]:
    print(T, sm(lg/T))
tgt=0; print("CE",-np.log(p[tgt]))
# top-p
order=np.argsort(-p); cs=np.cumsum(p[order]); print([vocab[i] for i in order],cs)
np.savez("toy.npz",E=E,P=P,X=X,Q=Q,K=K,V=V,S=S,Ss=Ss,A=A,Ac=Ac,Z=Z,O=O,R=R,Y=Y,H=H,F=F,Y2=Y2,lg=lg,p=p)

19.2 Causal chain, output, and disambiguation of banco

File: 02_cadeia_causal_e_desambiguacao_calc2.py. Function. Reads toy.npz and recomputes the chain from causal attention: attention output (Zc, Oc), residual addition (Rc), LayerNorm (Yc), FFN, logits at the estava position, temperature-scaled softmax, cross-entropy, and ordering for top-k and top-p; it writes toy2.npz. It then executes the Section 12 demonstration using four-axis embeddings (money, seat, people, wood), one-head attention with Wq = Wk = 2·I, Wv = I, d_k = 4, and cosine similarity between the contextualized banco vector and two sense prototypes.

Supports. Sections 7, 8, and 10–12 and Figure 6. The p − y gradient (Section 10) and top-k and top-p renormalizations (Section 11) were calculated manually from the probabilities printed by this program rather than by code. Requires toy.npz from 19.1.

import numpy as np
np.set_printoptions(precision=3, suppress=True)
d=np.load("toy.npz"); 
def sm(z): z=z-z.max(-1,keepdims=True); e=np.exp(z); return e/e.sum(-1,keepdims=True)
def ln(x,eps=1e-5): m=x.mean(-1,keepdims=True); v=x.var(-1,keepdims=True); return (x-m)/np.sqrt(v+eps)
X,V,Ac=d["X"],d["V"],d["Ac"]
Wo=np.array([[1,0,0.5,0],[0,1,0,0.5]],float)
Zc=Ac@V; Oc=Zc@Wo; Rc=X+Oc; Yc=ln(Rc)
print("Zc\n",Zc,"\nOc\n",Oc,"\nRc\n",Rc,"\nYc\n",Yc)
rng=np.random.default_rng(0)
W1=rng.normal(0,0.5,(4,8)); W2=rng.normal(0,0.5,(8,4))
H=np.maximum(0,Yc@W1); F=H@W2; Y2=ln(Yc+F)
print("H\n",H,"\nF\n",F,"\nY2\n",Y2)
Wout=rng.normal(0,0.8,(4,5)); vocab=["lotado","vazio","cheio","aberto","verde"]
lg=Y2[2]@Wout; p=sm(lg)
print("Wout\n",Wout,"\nlg",lg,"\np",p)
for T in [0.5,1,2]: print(T,sm(lg/T))
print("CE lotado",-np.log(p[0]),"CE cheio",-np.log(p[2]))
o=np.argsort(-p); print([vocab[i] for i in o], np.cumsum(p[o]))
np.savez("toy2.npz",Zc=Zc,Oc=Oc,Rc=Rc,Yc=Yc,H=H,F=F,Y2=Y2,Wout=Wout,lg=lg,p=p,W1=W1,W2=W2)
# disambiguation
feat=["dinheiro","assento","pessoas","madeira"]
emb={"O":[0,0,0,0],"banco":[.7,.7,0,0],"estava":[0,0,.1,0],"lotado":[0,0,1,0],
"de":[0,0,0,0],"clientes":[.8,0,.7,0],"da":[0,0,0,0],"praça":[0,.8,0,.2],"madeira":[0,.3,0,1],"velho":[0,.2,0,.5],"gerente":[.9,0,.3,0],"fila":[.5,0,.9,0]}
def ctx(words, scale=2.0):
    Em=np.array([emb[w] for w in words],float)
    # Q=K=E*scale + vetor de sentinela p/ evitar nulos
    Q=K=Em*scale
    S=Q@K.T/np.sqrt(4)
    bi=words.index("banco")
    S[bi,:]+=0  # sem alteração
    A=sm(S)
    Z=A@Em
    return A[bi],Z[bi]
protoF=np.array([1,0,.5,0]);protoP=np.array([0,1,0,.5])
cos=lambda a,b:a@b/np.linalg.norm(a)/np.linalg.norm(b)
for name,w in [("fin",["O","banco","estava","lotado","de","clientes"]),("praca",["O","banco","da","praça","estava","velho","madeira"]),("isolado",["banco"])]:
    a,z=ctx(w); print(name,w,a,z,"cosFin",cos(z,protoF),"cosPraca",cos(z,protoP))

19.3 BPE merges in a didactic corpus

File: 03_bpe_didatico.py. Function. Executes eight BPE steps over the corpus low (5), lower (2), newest (6), and widest (3), using </w> as the end-of-word marker. At each step it prints the merged pair, its frequency, the three most frequent pairs, and the resulting vocabulary. Ties are resolved by order of occurrence.

Supports. The merge table in Section 3.

import re, collections
vocab={"l o w </w>":5,"l o w e r </w>":2,"n e w e s t </w>":6,"w i d e s t </w>":3}
def stats(v):
    p=collections.Counter()
    for w,f in v.items():
        s=w.split()
        for a,b in zip(s,s[1:]): p[(a,b)]+=f
    return p
def merge(pair,v):
    out={}
    bg=re.escape(" ".join(pair)); pat=re.compile(r'(?<!\S)'+bg+r'(?!\S)')
    for w,f in v.items(): out[pat.sub("".join(pair),w)]=f
    return out
for i in range(8):
    p=stats(vocab); best=max(p,key=lambda k:(p[k],))
    top=p.most_common(3)
    print(i+1,best,p[best],top[:3]); vocab=merge(best,vocab); print("  ",vocab)

19.4 LayerNorm and RMSNorm verification for banco

File: 04_verificar_ln_rms.py. Function. Recomputes the mean, variance, LayerNorm, and RMSNorm of the residual-sum vector for token banco at position 1 in the causal version, and verifies that LayerNorm yields mean 0 and variance 1.

Supports. The numerical example in Section 7. Requires toy.npz and toy2.npz from 19.1 and 19.2.

import numpy as np
d=np.load("toy2.npz"); R=d["Rc"]; r=R[1]
print("R",r,"mean",r.mean(),"var",r.var(),"std",np.sqrt(r.var()+1e-5))
print("LN",(r-r.mean())/np.sqrt(r.var()+1e-5))
print("RMS",np.sqrt((r**2).mean()+1e-5), r/np.sqrt((r**2).mean()+1e-5))
X=np.load("toy.npz")["X"]; print("X1",X[1],"O1",d["Oc"][1])
print(d["Yc"][1].mean(), d["Yc"][1].var())

19.5 Generation of Figures 1–6

File: 05_figuras.py. Function. Uses Matplotlib to draw the flow diagram (Figure 1), E/P/X heat maps (Figure 2), the Q/K/V mechanism diagram (Figure 3), bidirectional and causal attention maps (Figure 4), the layer-specialization scheme (Figure 5), and probability bars by temperature (Figure 6). It reads toy.npz and toy2.npz and writes to the fig/ directory, which must exist before execution.

Supports. Article figures. The program writes internal names (fig2_embedding.png, fig4_qkv.png, fig3_atencao.png, fig6_saida.png), which are renamed in the publication package; titles embedded in the images already include final numbering.

import numpy as np, matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.patches import FancyBboxPatch, FancyArrowPatch
plt.rcParams.update({"font.family":"DejaVu Sans","font.size":10})
CR="© 2026 Aridio Silva – Pesquisador Independente · ORCID 0009-0008-2411-6995"
d=np.load("toy.npz"); d2=np.load("toy2.npz")
tok=["O","banco","estava","lotado"]
C={"in":"#dbe9f6","tok":"#e3f1df","emb":"#fdf0d5","att":"#f9dcdc","ffn":"#e8e0f4","out":"#d9f0ee","gray":"#f0f0f0"}
def foot(fig): fig.text(0.5,0.01,CR,ha="center",fontsize=7.5,color="#666")
def box(ax,x,y,w,h,txt,fc,fs=9,bold=False):
    ax.add_patch(FancyBboxPatch((x,y),w,h,boxstyle="round,pad=0.02,rounding_size=0.08",fc=fc,ec="#444",lw=1))
    ax.text(x+w/2,y+h/2,txt,ha="center",va="center",fontsize=fs,fontweight="bold" if bold else "normal",wrap=True)
def arrow(ax,x1,y1,x2,y2):
    ax.add_patch(FancyArrowPatch((x1,y1),(x2,y2),arrowstyle="-|>",mutation_scale=12,color="#333",lw=1.2))

# FIG 1 flow
fig,ax=plt.subplots(figsize=(9,11)); ax.set_xlim(0,10); ax.set_ylim(0,14.2); ax.axis("off")
box(ax,2,12.9,6,0.9,"1. TEXTO DE ENTRADA\n\"O banco estava lotado\"",C["in"],bold=True)
box(ax,2,11.3,6,0.9,"2. TOKENIZAÇÃO (BPE / SentencePiece)\n[O] [banco] [estava] [lotado] → IDs inteiros",C["tok"],bold=True)
box(ax,2,9.8,6,0.9,"3. EMBEDDING + CODIFICAÇÃO POSICIONAL\nmatriz X ∈ ℝ^(n × d_model)",C["emb"],bold=True)
ax.add_patch(FancyBboxPatch((1.1,3.9),7.8,5.4,boxstyle="round,pad=0.02,rounding_size=0.1",fc="#fafafa",ec="#222",lw=1.6,ls="--"))
ax.text(5,8.95,"4. BLOCO TRANSFORMER  ×  N camadas (pesos próprios por camada)",ha="center",fontsize=9.5,fontweight="bold")
box(ax,2.3,7.65,5.6,0.8,"Self-Attention multi-cabeça\n(Q, K, V = X·Wq, X·Wk, X·Wv)",C["att"],fs=8.8)
box(ax,2.3,6.55,5.6,0.6,"Add & Norm (residual + normalização)",C["gray"],fs=8.8)
box(ax,2.3,5.3,5.6,0.8,"Rede Feed-Forward (FFN) por posição",C["ffn"],fs=8.8)
box(ax,2.3,4.2,5.6,0.6,"Add & Norm (residual + normalização)",C["gray"],fs=8.8)
arrow(ax,5,12.9,5,12.2);arrow(ax,5,11.3,5,10.7);arrow(ax,5,9.8,5,9.3)
arrow(ax,5,7.65,5,7.15);arrow(ax,5,6.55,5,6.1);arrow(ax,5,5.3,5,4.8)
for (ya,yb) in [(8.6,6.85),(6.35,4.5)]:
    ax.plot([1.6,1.6],[ya,yb],color="#c0392b",lw=1.4)
    ax.plot([1.6,2.3],[ya,ya],color="#c0392b",lw=1.4)
    ax.add_patch(FancyArrowPatch((1.6,yb),(2.3,yb),arrowstyle="-|>",mutation_scale=11,color="#c0392b",lw=1.4))
ax.text(1.45,7.3,"conexão\nresidual",rotation=90,fontsize=7.5,color="#c0392b",ha="right",va="center")
ax.text(1.45,5.4,"conexão\nresidual",rotation=90,fontsize=7.5,color="#c0392b",ha="right",va="center")
box(ax,2,2.5,6,0.9,"5. PROJEÇÃO PARA O VOCABULÁRIO → LOGITS\nh_final · W_out  (um valor por token do vocabulário)",C["out"],bold=True,fs=8.8)
box(ax,2,1.2,6,0.9,"6. SOFTMAX (com temperatura T) → PROBABILIDADES\nP(próximo token | contexto)",C["out"],bold=True,fs=8.8)
box(ax,2,-0.1,6,0.9,"7. DECODIFICAÇÃO (guloso, top-k, top-p, amostragem)\n→ token de saída (anexado à entrada; o ciclo se repete)",C["out"],bold=True,fs=8.8)
arrow(ax,5,3.9,5,3.4);arrow(ax,5,2.5,5,2.1);arrow(ax,5,1.2,5,0.8)
ax.set_ylim(-0.6,14.2)
ax.set_title("Figura 1 – Fluxo completo da inferência em um Transformer (visão geral)",fontsize=11,fontweight="bold",loc="left")
foot(fig); fig.savefig("fig/fig1_fluxo.png",dpi=170,bbox_inches="tight"); plt.close()

# FIG 2 embeddings + PE
fig,axs=plt.subplots(1,3,figsize=(11,3.6))
for a,M,t in zip(axs,[d["E"],d["P"],d["X"]],["E: embeddings didáticos (n=4, d=4)","P: codificação posicional senoidal","X = E + P (entrada da camada 1)"]):
    im=a.imshow(M,cmap="RdBu_r",vmin=-2,vmax=2)
    a.set_xticks(range(4));a.set_xticklabels([f"d{i+1}" for i in range(4)])
    a.set_yticks(range(4));a.set_yticklabels([f"{i}: {t_}" for i,t_ in enumerate(tok)])
    for i in range(4):
        for j in range(4): a.text(j,i,f"{M[i,j]:.2f}",ha="center",va="center",fontsize=8.5,color="white" if abs(M[i,j])>1.3 else "black")
    a.set_title(t,fontsize=9)
fig.suptitle("Figura 2 – Etapa 2: embeddings e codificação posicional (valores calculados; pesos didáticos)",fontsize=10.5,fontweight="bold",x=0.01,ha="left")
fig.tight_layout(rect=[0,0.08,1,0.93]); foot(fig); fig.savefig("fig/fig2_embedding.png",dpi=170); plt.close()

# FIG 3 attention
fig,axs=plt.subplots(1,2,figsize=(10.5,4.4))
for a,M,t in zip(axs,[d["A"],d["Ac"]],["Atenção bidirecional (estilo codificador)","Atenção causal (estilo decodificador)"]):
    im=a.imshow(M,cmap="YlOrBr",vmin=0,vmax=1)
    a.set_xticks(range(4));a.set_xticklabels(tok);a.set_yticks(range(4));a.set_yticklabels(tok)
    a.set_xlabel("Key (para quem olha)");a.set_ylabel("Query (quem pergunta)")
    for i in range(4):
        for j in range(4):
            a.text(j,i,f"{M[i,j]:.2f}",ha="center",va="center",fontsize=10,color="black" if M[i,j]<0.6 else "white")
    a.set_title(t,fontsize=10)
fig.suptitle("Figura 4 – Mapa de atenção softmax(QKᵀ/√d_k): cada linha soma 1,00",fontsize=11,fontweight="bold",x=0.01,ha="left")
fig.tight_layout(rect=[0,0.04,1,0.92]); foot(fig); fig.savefig("fig/fig3_atencao.png",dpi=170); plt.close()

# FIG 4 QKV mechanism
fig,ax=plt.subplots(figsize=(10,4.6)); ax.set_xlim(0,12); ax.set_ylim(0,6); ax.axis("off")
box(ax,0.1,2.4,1.5,1.2,"X\n(n × d_model)",C["emb"],bold=True)
for k,(lab,y) in enumerate([("Q = X·Wq",4.3),("K = X·Wk",2.5),("V = X·Wv",0.7)]):
    box(ax,2.4,y,1.8,0.9,lab,C["att"] if k<2 else C["ffn"],bold=True); arrow(ax,1.6,3.0,2.4,y+0.45)
box(ax,5.3,3.4,2.1,1.0,"Scores\nS = Q·Kᵀ / √d_k",C["gray"],fs=9); arrow(ax,4.2,4.75,5.3,4.0);arrow(ax,4.2,2.95,5.3,3.7)
box(ax,5.3,1.9,2.1,0.9,"(máscara causal\nopcional: −∞)",C["gray"],fs=8.5); arrow(ax,6.35,2.8,6.35,3.4)
box(ax,8.2,3.4,1.5,1.0,"A = softmax(S)\nlinhas somam 1",C["out"],fs=8.5); arrow(ax,7.4,3.9,8.2,3.9)
box(ax,8.2,1.0,1.5,1.0,"Z = A·V",C["out"],bold=True); arrow(ax,8.95,3.4,8.95,2.0); arrow(ax,4.2,1.15,8.2,1.4)
box(ax,10.2,1.0,1.7,1.0,"Z·Wo\n(saída da\ncamada)",C["emb"],fs=8.5); arrow(ax,9.7,1.5,10.3,1.5)
ax.set_title("Figura 3 – Mecanismo de atenção por produto escalar escalonado (Vaswani et al., 2017)",fontsize=11,fontweight="bold",loc="left")
foot(fig); fig.savefig("fig/fig4_qkv.png",dpi=170,bbox_inches="tight"); plt.close()

# FIG 5 layers (conceptual scheme)
fig,ax=plt.subplots(figsize=(9,5.6)); ax.set_xlim(0,10); ax.set_ylim(0,10); ax.axis("off")
L=[("Camadas finais (ex.: 9–12 em modelo de 12)","Representações próximas da tarefa de saída;\ncombinação de informação global; preparação do logit",C["out"],0.5),
   ("Camadas intermediárias (ex.: 4–8)","Semântica lexical e contextual; desambiguação de sentido;\nrelações entre palavras e entidades",C["att"],3.4),
   ("Camadas iniciais (ex.: 1–3)","Padrões superficiais e sintáticos locais;\nvizinhança, posição, classe gramatical",C["in"],6.3)]
for t,s,c,y in L:
    box(ax,1.2,y,7.6,2.4,"",c); ax.text(5,y+1.75,t,ha="center",fontsize=10.5,fontweight="bold"); ax.text(5,y+0.8,s,ha="center",fontsize=9.3)
arrow(ax,0.6,6.3,0.6,2.9); ax.text(0.45,4.6,"profundidade",rotation=90,va="center",ha="right",fontsize=8.5)
ax.text(5,9.3,"Esquema conceitual: tendência observada em estudos de sondagem em BERT\n(Tenney et al., 2019); as fronteiras entre camadas NÃO são exatas.",ha="center",fontsize=8.6,style="italic",color="#444")
ax.set_title("Figura 5 – Especialização aproximada por profundidade em Transformers empilhados",fontsize=11,fontweight="bold",loc="left")
foot(fig); fig.savefig("fig/fig5_camadas.png",dpi=170,bbox_inches="tight"); plt.close()

# FIG 6 output
vocab=["lotado","vazio","cheio","aberto","verde"]
lg=d2["lg"]
def sm(z): e=np.exp(z-z.max()); return e/e.sum()
fig,axs=plt.subplots(1,3,figsize=(11,3.6),sharey=True)
for a,T in zip(axs,[0.5,1.0,2.0]):
    p=sm(lg/T); a.bar(vocab,p,color="#4a90a4"); a.set_title(f"T = {T}",fontsize=10); a.set_ylim(0,1)
    for i,v in enumerate(p): a.text(i,v+0.02,f"{v:.3f}",ha="center",fontsize=8.5)
    a.tick_params(axis="x",labelrotation=30)
axs[0].set_ylabel("Probabilidade")
fig.suptitle("Figura 6 – Efeito da temperatura sobre softmax(logits/T) (logits calculados; pesos de saída aleatórios)",fontsize=10.5,fontweight="bold",x=0.01,ha="left")
fig.tight_layout(rect=[0,0.05,1,0.92]); foot(fig); fig.savefig("fig/fig6_saida.png",dpi=170); plt.close()

19.6 Extraction of bibliography links

File: 07_gerar_csv_links.py. Function. Reads Section 18 from the Markdown file and generates links_bibliografia.csv with author, year, expected title, URL, and audit status for each entry.

Supports. Input for Program 19.7.

import re,csv
s=open("abrindo-a-caixa-preta-da-inferencia-em-llms.md",encoding="utf-8").read()
# delimits Section 18 before the code appendix
ref=s.split("## 18. Bibliography / References")[1].split("## 19.")[0]
items=[l[2:] for l in ref.splitlines() if l.startswith("- ")]
rows=[]
for it in items:
    cite=it.split("(")[0].strip().rstrip(",;. ")
    yr=re.search(r"\((\d{4}[ab]?)\)\.\s+(.*)",it)
    rest=yr.group(2)
    m=re.match(r"(.*?[?!])\s|(.*?)\.\s",rest)
    titulo=(m.group(1) or m.group(2)) if m else rest[:80]
    urls=re.findall(r"<(https?://[^>]+)>",it)
    aud="link auditado" in it
    rows.append([len(rows)+1,cite.split(";")[0]+(" et al." if ";" in cite else ""),yr.group(1),titulo,urls[0],"auditado (busca)" if aud else "nao auditado"])
with open("links_bibliografia.csv","w",newline="",encoding="utf-8-sig") as f:
    w=csv.writer(f); w.writerow(["n","autor","ano","titulo_esperado","url","status_auditoria_20261020"]); w.writerows(rows)

19.7 HTTP audit of bibliography links

File: 08_auditar_links.py. Function. Opens each URL from the CSV, records the HTTP status and final URL, extracts the page title, and compares it with the expected title, classifying the entry as OK or REVIEW. It requires the requests library and Internet access. It was tested only against a local server, not against the live sites.

Supports. The auditoria_links_resultado.csv file to be generated by the reader (Section 18).

#!/usr/bin/env python3
"""Audita os links da bibliografia: abre cada URL, registra o código HTTP, a URL final
e compara o <title> da página com o título esperado. Uso: python3 auditar_links.py
Requer: pip install requests. Saída: auditoria_links_resultado.csv"""
import csv, re, sys, time, difflib, html
import requests

H = {"User-Agent": "Mozilla/5.0 (auditoria-bibliografia; contato: ORCID 0009-0008-2411-6995)"}

def norm(t):
    return re.sub(r"[^a-z0-9 ]", "", html.unescape(t).lower())

def titulo(txt):
    m = re.search(r"<title[^>]*>(.*?)</title>", txt, re.S | re.I)
    return re.sub(r"\s+", " ", m.group(1)).strip() if m else ""

rows = list(csv.DictReader(open("links_bibliografia.csv", encoding="utf-8-sig")))
out = []
for r in rows:
    url, esp = r["url"], r["titulo_esperado"]
    try:
        resp = requests.get(url, headers=H, timeout=25, allow_redirects=True)
        t = titulo(resp.text) if "html" in resp.headers.get("content-type", "") else ""
        sim = difflib.SequenceMatcher(None, norm(esp), norm(t)).ratio() if t else 0.0
        contem = norm(esp)[:40] in norm(t) if t else False
        # PDFs não têm <title>: apenas o status é verificável automaticamente
        ok = resp.status_code == 200 and (contem or sim >= 0.6 or not t)
        res = [r["n"], r["autor"], url, resp.status_code, resp.url, t[:90], round(sim, 2),
               "OK" if ok else "REVISAR (conferir título manualmente)"]
    except Exception as e:
        res = [r["n"], r["autor"], url, "erro", "", str(e)[:90], 0, "REVISAR (falha de acesso)"]
    out.append(res); print(res[0], res[3], res[7], url)
    time.sleep(1.0)
with open("auditoria_links_resultado.csv", "w", newline="", encoding="utf-8-sig") as f:
    w = csv.writer(f)
    w.writerow(["n", "autor", "url", "http", "url_final", "titulo_da_pagina", "similaridade", "resultado"])
    w.writerows(out)
print("\nConcluído:", sum(1 for o in out if o[7] == "OK"), "OK de", len(out))

About the Author

Aridio Silva is an independent researcher in Brazil working on the architecture, security, governance, and reliability of autonomous and distributed artificial intelligence systems.

His research focuses on Agentic AI, Multi-Agent Systems, Edge AI, AI Security, Zero Trust, Security-by-Design, AI Governance, Spec-Driven Development, and continuous security assurance.

He is the creator and lead researcher of SGAEIA — Secure Governed Autonomous Edge Intelligence Architecture, a research initiative investigating architectural foundations for secure, governed, auditable, and trustworthy autonomous AI systems across distributed edge and cloud environments.

Research & Project Resources

1. ORCID: https://orcid.org/0009-0008-2411-6995
2. Google Scholar: https://scholar.google.com/citations?user=rPn5O48AAAAJ
3. Zenodo — SGAEIA Community: https://zenodo.org/communities/sgaeia
4. OpenAIRE: https://explore.openaire.eu/search/find?fv0=Aridio%20Silva&f0=q
5. Medium: https://medium.com/@aridiosilva
6. DEV Community: https://dev.to/aridiosilva
7. GitHub: https://github.com/aridiosilva
8. LinkedIn: https://www.linkedin.com/in/aridio-silva-74997111/
9. SGAEIA — canonical project DOI: https://doi.org/10.5281/zenodo.22557796
10. Homepage: https://aridiosilva.com
11. SGAEIA Homepage: https://aridiosilva.com/sgaeia
12. SGAEIA LinkedIn: https://www.linkedin.com/company/sgaeia/

Figures

The cover is not numbered. Figures 1–6 are public technical and didactic illustrations. Their numbers and weights are explicitly identified as reduced examples; they are not measurements of a trained model and do not disclose private SGAEIA protocols, algorithms, state machines, policies, thresholds, or mechanisms. Final images include visible credit and should preserve creator, copyright, project, title, description, and license metadata.

License

Except where otherwise noted, the text, examples, and original illustrations in this article are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).

© 2026 Aridio Silva. This work may be shared and adapted for any purpose provided that appropriate attribution is given.

The SGAEIA software and research artifact remain subject to their own Apache License 2.0.

Autonomous AI. Governed by Design. Trusted by Evidence.

Series continuity

This work is SGAEIA Research Series — Article 22. Its version of record is published on Zenodo under DOI 10.5281/zenodo.23540392. The Portuguese and English editions share the same bibliographic record and are distinguished by the language stated in their filenames and metadata.

Suggested citation

Silva, Aridio. (2026). Opening the Black Box of LLM Inference: Foundations of Generative AI — Anatomy of the Transformer from the Inside, from Input to Output Token. SGAEIA Research Series, Article 22. Zenodo. https://doi.org/10.5281/zenodo.23540392

© 2026 Aridio Silva | Project SGAEIA | CC BY 4.0

ORCID 0009-0008-2411-6995 | DOI 10.5281/zenodo.23540392