Shadow EMA: model rationale and architecture

The idea

Shadow EMA is a causal character language model that represents history with compact recurrent states. Each character enters as a fixed vector. Several exponential moving averages (EMAs) describe how that vector stream has evolved. Each model layer then makes several soft reads from its own latent shelf and carries the results forward through another EMA. A final, independent output head maps the current hidden state to the next-character vocabulary.

The central design question is whether a small evolving state can do useful context work without retaining a key and value for every earlier character. The model still processes characters in order. It simply stores a fixed number of continuous vectors as its working memory.

“Shadow” refers to an internal shelf of latent values. Its slots are not decoded into words or characters. “Radix” was the working name for the parallel reads; in the implementation these are heads. Each head can ask a different question of its layer's shelf.

One character's path

flowchart LR
    C[Character ID] --> E[Fixed k-vector]
    E --> I[Input slow, fast, and change EMAs]
    E --> J[Concatenate four k-vectors]
    I --> J
    J --> P[Project to model width]
    P --> B[Shadow EMA blocks]
    B --> N[Final RMSNorm]
    N --> O[Untied output head]
    O --> L[Next-character logits]

Each shadow block also reads its own state from the previous character and writes an updated state for the next character. This temporal link is the model's learned recurrent memory.

For the current reference configuration:

The dimensions are choices, not identities imposed by the architecture. In particular, the 128-wide hidden state is projected to three 32-wide queries; it is not divided evenly into three pieces.

1. Fixed character identity

Let v be the vocabulary size and k the embedding width. Every character ID c has a fixed vector e(c) ∈ R^k. To construct the table, the code creates N evenly spaced scalar levels between -1 and 1. It samples k distinct levels for each character, with a deterministic random seed. The vectors are then frozen.

The bucket size is chosen by a heuristic that balances the number of levels shared across character vectors with use of the available levels:

N_overlap = round(k² / target_overlap)
N_coverage = floor(vk / coverage_margin)
N = max(k + 1, min(N_overlap, N_coverage))  when N_coverage > k

Otherwise the implementation uses max(k + 1, N_overlap). Shared scalar levels are only reuse of numbers. They do not imply that two characters have related meanings. What the model sees is the complete ordered vector for each character.

The fixed table provides stable character identities without a learned input embedding table. It does not itself encode word meaning or context. Later projections and recurrent states are responsible for interpreting each vector in its sequence.

2. Three input EMAs

The input stream has three k-wide states. All operations are coordinatewise, so each of the k components follows the same recurrence independently.

For character position t, with fixed vector x_t = e(c_t):

slow_t = slow_(t-1) + α_s (x_t - slow_(t-1))
fast_t = fast_(t-1) + α_f (x_t - fast_(t-1))
delta_t = x_t - x_(t-1)
change_t = change_(t-1) + α_r (delta_t - change_(t-1))
u_t = concat(x_t, slow_t, fast_t, change_t)
h_(0,t) = W_in u_t + b_in

The current defaults are α_s = 0.03, α_f = 0.5, and α_r = 0.5. A larger alpha follows the newest vector more closely. The slow state carries a smoothed trace, the fast state responds to recent characters, and the change state follows the difference between consecutive character vectors. Together they expose multiple time scales and local change to the first layer.

At the first character, both slow and fast are initialized to that character's fixed vector. The previous vector is set to that same vector and the change state starts at zero. Thus the first input is [x_0, x_0, x_0, 0], with no artificial change spike from an assumed preceding zero vector.

These states make the input order sensitive. They are proxies for recent context and progression, not explicit position numbers. Two different histories can lead to similar EMA states, and older details fade.

3. A layer's shadow shelf and three heads

Every layer ℓ has its own parameters. For each of its three heads j, it owns:

K_(ℓ,j) ∈ R^(v × d_q)      learned address keys
V_(ℓ,j) ∈ R^(v × k)        latent shadow values
S_(ℓ,j,t) ∈ R^k            recurrent shadow EMA state

d_q = 32 and k = 64 in the current reference configuration. The number of shelf rows is v, the same number as the input and output character vocabulary. This is a size choice. Shadow row i has no requirement to mean character i, and the three heads do not have to agree on what their row numbers mean. Each layer and head has its own keys and values.

Shadow values have two supported modes. In learned mode, the values are parameters. In fixed mode, they are randomly initialized once and kept as buffers. Address keys and query projections remain learned in both modes. A fixed shelf can still be read in different ways as the keys and queries change.

4. Query, soft read, and recurrent update

At position t, a head's query depends on the current hidden vector and the previous shadow state. The previous state is essential: it allows an earlier latent read to influence the next character's question.

For a layer, the three previous head states are concatenated and normalized. The hidden vector is normalized separately. A projection then produces all three queries:

q_input = concat(RMSNorm(h_(ℓ,t)), RMSNorm(concat_j S_(ℓ,j,t-1)))
[q_1, q_2, q_3] = reshape(W_q q_input + b_q)

Each head scores its v shelf keys, creates a dense probability distribution, and takes a weighted read:

score_(j,i) = (q_j · K_(ℓ,j,i)) / sqrt(d_q)
p_(j,i) = softmax_i(score_(j,i))
r_(ℓ,j,t) = Σ_i p_(j,i) V_(ℓ,j,i)
S_(ℓ,j,t) = S_(ℓ,j,t-1) + α_shadow (r_(ℓ,j,t) - S_(ℓ,j,t-1))

The current shadow alpha is 0.1. Each head makes one soft read over v slots. Three heads produce three reads and three persistent states. The model does not build a joint distribution over v³ addresses. Calling these reads “soft top-K” is an intuition for multiple possible selections; ordinary softmax remains dense, so every slot can contribute.

The updated three-head state is concatenated into a 3k-wide vector. The block normalizes it, projects it back to model width, and adds it to the residual stream. The code gives this feedback a learned scalar multiplier initialized to 0.1:

h'_(ℓ,t) = h_(ℓ,t) + g_ℓ W_out RMSNorm(concat_j S_(ℓ,j,t))
h_(ℓ+1,t) = h'_(ℓ,t) + MLP_ℓ(RMSNorm(h'_(ℓ,t)))

The MLP expands the hidden width fourfold, applies GELU, and projects back. Each block has its own normalization weights, query projection, keys, values, feedback projection, scalar feedback strength, and MLP.

5. Final output

After the last block, the model applies a final RMSNorm and an independent linear output head:

logits_t = W_output RMSNorm(h_(last,t)) + b_output

logits_t ∈ R^v scores the next character. W_output is not tied to the fixed input vectors or to any shadow value table. The vocabulary indexes determine which output logit corresponds to which character; the shadow shelves do not decode into characters at intermediate layers.

What is carried between characters?

The model's persistent state contains the previous fixed vector, the three input EMAs, and one 64-wide shadow EMA per head per layer. With k = 64, three heads, and two layers, that is 4 × 64 + 2 × 3 × 64 = 640 scalar state values per independent stream. The current hidden vector is computed from this state and the current character, then passed through the layers.

There is no stored sequence of previous token keys or values. There is also no attention matrix between character positions. A head's softmax addresses a fixed-size learned shelf; it does not compare the current character against earlier character positions. The model is therefore free of token-to-token self-attention, while still using an attention-like soft read over latent slots.

The state size does not grow with the number of characters processed. The price is compression: the model cannot directly revisit an arbitrary earlier character, and its EMA traces do not preserve an exact transcript. Its output can depend on history only through the states carried forward.

Why call the shadow state an internal workspace?

The shadow shelf lets a layer form a soft latent choice, feed its result into the current hidden stream, and carry a smoothed version into the next character's query. Repeating this across heads and layers makes the latent state an evolving workspace. Different layers can form different representations because their shelves and projections are independent.

“Internal chain of thought” is a motivating analogy, not a claim that individual shelf addresses or state vectors are readable reasoning steps. The architecture specifies a continuous recurrent computation. It does not force a particular human interpretation on any one slot.

Architectural character

Shadow EMA combines three kinds of information at each character: a stable symbol identity, coordinatewise traces of the input stream, and layer-specific latent traces from prior soft reads. It keeps the familiar residual block, normalization, MLP, and output-head structure of a small autoregressive language model, while replacing sequence self-attention with fixed-size recurrent state and parameterized shelf addressing.