GeoPair: How Cross-Layer Matching Compresses Transformers Without Fine-Tuning

Deploying a transformer under strict memory and compute limits often requires compressing large weight matrices without running another training cycle. Standard post-training factorization handles each layer independently, while shared-basis methods commonly pair adjacent layers and combine their activation statistics. Both strategies can miss useful cross-layer redundancy or distort the activation geometry that determines how approximation errors affect model outputs.

GeoPair addresses both problems. The primary paper, published September 22, 2026, presents a training-free pipeline that selects compatible layer pairs globally, factorizes each pair through a shared dictionary, and preserves the separate calibration geometry of every layer.

Compression as activation reconstruction

Approximating a weight matrix by raw weight distance alone treats every input direction equally. GeoPair instead optimizes activation-aware reconstruction.

For a weight matrix W and calibration activations X, the method forms a per-layer Gram matrix:

G = XᵀX

Tikhonov regularization makes this matrix nonsingular. A Cholesky factorization then gives:

Gη = LᵀL

The compressed weight matrix Ŵ is selected to minimize:

||L(W - Ŵ)||²F

The Cholesky factor L acts as a whitening transform. Errors in directions emphasized by the calibration activations receive greater weight than errors in directions that matter less for the observed data.

Crucially, every layer has its own Gram matrix and Cholesky factor. Layers at different depths can see different activation distributions, so their whitening geometries are not interchangeable. Simply averaging their covariance information can weaken reconstruction fidelity.

One dictionary, two coefficient matrices

For a selected pair of compatible weights, W1 and W2, GeoPair learns a shared dictionary D and separate coefficient matrices C1 and C2:

W1 ≈ DC1

W2 ≈ DC2

The objective preserves both whitening spaces:

||L1W1 - L1DC1||²F + ||L2W2 - L2DC2||²F

Sharing D captures structure common to the pair. Keeping C1 and C2 separate allows each layer to represent its own projection through that shared basis. Keeping L1 and L2 separate prevents the optimization from replacing two activation geometries with a heuristic global geometry.

Alternating optimization

GeoPair alternates between two updates:

1. Coefficient update: With D fixed, solve weighted least-squares problems for C1 and C2.
2. Dictionary update: With the coefficients fixed, solve a two-term generalized Sylvester equation for D.

The dictionary step is central to the method. Because the two reconstruction terms contain distinct whitening transforms, an ordinary shared least-squares update does not represent the full objective. GeoPair derives an exact generalized Sylvester update using simultaneous diagonalization of calibration matrix pencils.

This lets the pair share parameters while respecting each layer’s Gram matrix. It also replaces heuristic covariance aggregation with an optimization-driven update.

Global pairing instead of adjacency

A strong factorization still depends on selecting compatible layers. Pairing consecutive layers is simple, but adjacency does not guarantee structural alignment.

GeoPair computes pairwise structural distances between eligible weight matrices in whitened space. It then represents the grouping problem as a weighted graph:

• Weight matrices are vertices.
• Candidate pairings are edges.
• Edge weights encode pair compatibility or alignment.

The method chooses the global set of non-overlapping pairs through weighted maximum matching, solved with Edmonds’ Blossom algorithm. This is not a greedy sequence of local choices: selecting one edge changes which pairings remain available elsewhere in the model.

The paper’s grouping ablation reports that Frobenius-norm-based grouping performs better than cosine-similarity grouping and greedy consecutive pairing. The result supports global, data-aware selection over fixed adjacency within the evaluated setup.

Optional structured sparsity with HTP

Dense coefficient matrices can limit the compression achieved by dictionary sharing. GeoPair therefore optionally applies Hard Thresholding Pursuit, or HTP, during coefficient optimization.

HTP constrains each coefficient matrix to contain at most a specified number of nonzero entries. The dictionary remains shared, while the layer-specific codes become structured sparse representations. The paper also reports convergence guarantees for this sparse optimization and identifies a KS ratio of 2.5 as the best tested trade-off between dictionary rank and sparsity.

HTP is a pipeline option rather than a prerequisite for cross-layer matching. Its value depends on the target compression level and whether the eventual runtime can exploit the resulting structure.

What the reported benchmarks show

The clearest ablation uses Llama3.2 1B at CR = 0.2. The uncompressed baseline records 57.61 average accuracy and 5.73 perplexity. A Basis Sharing configuration using global whitening falls to 33.11 accuracy and 239.30 perplexity.

Replacing global whitening with the Sylvester solver raises accuracy to 41.26 and reduces perplexity to 57.95. Adding optimal grouping reaches 43.03 accuracy and 32.49 perplexity. The full configuration with HTP reports 54.09 accuracy and 6.74 perplexity.

A separate Llama2 7B comparison at a reported 20% compression ratio gives:

These are paper-reported results, not independent deployment measurements. The authors also report improvements over Basis Sharing across Qwen3, Gemma3, and Phi-4, plus evaluations spanning multiple modalities. The arXiv submission record identifies this release as version 1, while the Hugging Face Daily Papers listing includes GeoPair on September 24.

A high-level integration path

A practical integration can follow six stages:

1. Collect a representative calibration set.
2. Capture local activations for eligible transformer projections.
3. Build regularized Gram matrices and their Cholesky factors.
4. Compute whitened pair distances and run weighted maximum matching.
5. Optimize each pair by alternating coefficient and exact generalized Sylvester dictionary updates.
6. Optionally apply HTP, then replace the original weights with shared dictionaries and layer-specific coefficients.

Validation should compare task quality and perplexity against both the original model and independent per-layer compression. Memory use, preprocessing cost, numerical stability, and runtime behavior also need measurement in the intended environment.

Deployment implications and limits

GeoPair matters because it turns cross-layer sharing into a calibration-aware optimization problem rather than an adjacency rule. It requires no post-compression fine-tuning, which can simplify compression when retraining is unavailable or undesirable.

However, parameter reduction does not automatically establish lower latency. Actual speed depends on how shared dictionaries and sparse coefficients are executed by the target software and hardware. The supplied results do not justify assuming a specific throughput, energy, or latency gain, so those outcomes require direct benchmarking.

The main mathematical limitation is pairwise-only grouping. The current generalized Sylvester solver supports groups of two, and the paper identifies higher-order shared groups as a scalability challenge. GeoPair therefore demonstrates a principled route to cross-layer compression, but broader grouping and verified hardware-level gains remain open deployment questions.