VIDRAFT Open-Sources Aether-7B-5Attn: Full MoE Weights, Training Code, Data Recipe, and Logs Under Apache-2.0

TL;DR: VIDRAFT has released Aether-7B-5Attn, a ~6.59B-parameter Mixture-of-Experts foundation model on Hugging Face, with everything needed for full reproducibility — weights, training code, data recipe, hyperparameters, logs, and intermediate checkpoints — all under Apache-2.0. Its defining architectural feature is a 7×7 Latin-square layout spanning five distinct attention types across 49 layers, trained on ~144.2B tokens. Developers who want a fully auditable, commercially usable open foundation model have a rare end-to-end artifact to work with.

What it is

Aether-7B-5Attn is a fully open foundation model released by VIDRAFT, a Korean Pre-AGI AI startup. Here are the key facts from the release:

• Parameter count: ~6.59B active parameters in a Mixture-of-Experts (MoE) configuration
• Architecture: 49 transformer layers arranged in a 7×7 Latin-square pattern, each cell assigned one of five distinct attention types — giving the model its "5Attn" designation
• Training data: ~144.2B tokens
• License: Apache-2.0 — permissive for both research and commercial use
• Release artifacts: model weights, training source code, data recipe, full hyperparameter configs, training logs, evaluation code, and intermediate checkpoints at steps 110k, 115k, and 162k (~13 GB each)
• KV-cache: this build does not implement KV-cache
• Distribution: published on Hugging Face

The completeness of the release is notable. Most "open" model drops publish weights alone. VIDRAFT has published the full training lineage, making the build genuinely reproducible rather than just inspectable.

How it works

At a conceptual level, Aether-7B-5Attn combines two architectural ideas: Mixture-of-Experts routing and heterogeneous attention.

MoE routing means the model does not activate all parameters for every token. Instead, a routing mechanism selects a subset of "expert" sub-networks per forward pass. This allows a larger total parameter footprint while keeping per-token compute tractable — a well-established efficiency strategy used in models like Mixtral and others.

Heterogeneous attention via a Latin-square layout is the more distinctive design choice. Rather than stacking identical self-attention blocks, Aether-7B-5Attn assigns different attention mechanisms to different layers. The 7×7 Latin-square constraint ensures that each of the five attention types appears in a mathematically balanced, non-repeating pattern across the 49 layers. A Latin square is a combinatorial structure where every element appears exactly once in each row and column — applied here, it governs which attention variant occupies which layer slot, avoiding clustering of any single mechanism type. The goal, conceptually, is to give the model a diverse set of attention inductive biases distributed evenly throughout its depth.

This kind of structured heterogeneity in attention is an active research direction, and publishing the full training code alongside the weights means the research community can study whether and how the layout choice affects learned representations.

Benchmarks & results

The ReasonCore announcement explicitly notes that the release omits benchmark scores. No public evaluation numbers — MMLU, HellaSwag, ARC, or otherwise — are reported in this release announcement.

What is documented is the training scale: ~144.2B tokens processed to convergence, with checkpoints preserved at three intermediate steps. The evaluation code is included in the release, so developers can run standardized benchmarks against their own task distributions rather than relying on vendor-reported figures. For teams that distrust cherry-picked leaderboard results, this is arguably more useful.

How to try it

The model is publicly available on Hugging Face. You can download the weights and checkpoints using the standard Hugging Face CLI:

pip install huggingface_hub
huggingface-cli download VIDRAFT/Aether-7B-5Attn

The full release — weights, training code, data recipe, logs, and evaluation scripts — is available under Apache-2.0, meaning you can use, modify, and redistribute it freely, including for commercial purposes, with attribution.

• Weights & checkpoints: Hugging Face (VIDRAFT/Aether-7B-5Attn)
• Training code & evaluation scripts: included in the release artifacts
• License: Apache-2.0

Note: The release does not advertise an OpenAI-compatible hosted API endpoint. Inference is self-hosted from the published weights.

FAQ

Q: What makes the 7×7 Latin-square attention layout different from a standard transformer stack?
A: In a standard transformer, every layer uses the same attention mechanism. Aether-7B-5Attn instead rotates among five different attention types across its 49 layers, with the Latin-square constraint ensuring no type is over-represented in any positional slice of the network. This is a structural inductive bias — the hypothesis is that diverse attention mechanisms at different depths capture different relational patterns in the data.

Q: Can I fine-tune and redistribute this model commercially?
A: Yes. The Apache-2.0 license permits use, modification, and redistribution for commercial purposes, provided standard attribution requirements are met. The training code is also Apache-2.0, so derivative training pipelines are covered as well.

Q: The model has no KV-cache — does that limit deployment?
A: It does add inference overhead for autoregressive generation compared to KV-cache-enabled models, since past key-value states must be recomputed each step. This build appears oriented toward research reproducibility rather than optimized production serving. Teams deploying it at scale would likely want to add KV-cache support or use it as a base for further development.

Originally reported by ReasonCore (영문) (2026-07-20) — source article.