VIDRAFT Releases Aether-7B-5Attn: A Fully Open-Source MoE LLM with Five Heterogeneous Attention Mechanisms

TL;DR: Korean AI startup VIDRAFT has released Aether-7B-5Attn, a 6.59B-parameter Mixture-of-Experts foundation model on Hugging Face under Apache-2.0 — shipping not just weights but training data recipes, full training code, hyperparameters, training logs, intermediate checkpoints, and evaluation code. If you care about reproducibility, sovereign AI, or heterogeneous attention research, this one is worth a close look.

What it is

Aether-7B-5Attn is an open-source foundation LLM developed by VIDRAFT (비드래프트), a Seoul-based Pre-AGI AI startup. It was released on Hugging Face under the Apache-2.0 license and is positioned as a truly open model — not just "open weights" — following the philosophy of projects like Allen AI's OLMo, Apertus, and LLM-jp.

Key facts from the release:

• Total parameters: ~6.59 billion (MoE architecture)
• Active parameters per token: ~2.98 billion (sparse activation for inference efficiency)
• Attention layers: 49 transformer layers
• Architecture novelty: Five heterogeneous attention mechanisms integrated into a single model
• Training corpus: ~144.2 billion tokens
• Data mix highlights: Mathematics (37.8%), Korean (21.6%), English (21.6%) — explicitly bilingual by design, not an English-first model retrofitted for Korean
• License: Apache-2.0

Both a base model and an instruct-tuned model are available, along with a live demo, all through Hugging Face.

How it works

The central architectural idea is heterogeneous attention integration: rather than using a single attention mechanism throughout all layers, Aether-7B-5Attn combines five distinct attention types within one model:

1. Full Attention — standard global self-attention
2. Differential Attention — reduces attention noise by differencing two softmax attention maps
3. Sliding Window Attention — limits each token's receptive field to a local window for efficiency
4. NSA-family Sparse Attention — hardware-aware native sparse attention patterns
5. Hybrid Attention — a mixed strategy combining the above approaches

The key engineering question when mixing attention types is: where do you put each type in the layer stack? Random or greedy assignment risks clustering certain mechanisms at certain depths, which can create uneven capability distributions across the network. VIDRAFT addresses this with a 7×7 Latin Square layout across the 49 layers — a combinatorial design that ensures no single attention type is overrepresented at any particular depth. Think of it as an experiment design technique borrowed from statistics, applied to layer-wise architecture decisions.

On top of that, the MoE backbone means only a subset of parameters (~2.98B) are active per forward pass, keeping inference costs manageable relative to the total parameter count.

The "fully open" commitment means the repository is designed so an external team can trace the complete lineage: data composition and tokenization → training code and configuration → logs → intermediate checkpoints → evaluation procedures. The goal is independent reproducibility, not just deployability.

Benchmarks & results

VIDRAFT has previously disclosed the following public benchmark results for their Darwin model family (the production-grade line sitting above the Aether research architecture):

• K-AI Leaderboard: Overall #1 ranking in 2026
• GPQA Diamond: 90.9%
• Polaris Global Drug Discovery Leaderboard: 14 consecutive #1 rankings
• Metacognition Leaderboard: #1
• Hugging Face downloads: Darwin family has surpassed 1 million cumulative downloads

Specific benchmark numbers for Aether-7B-5Attn itself were not disclosed in the source article at time of publication. Given the full evaluation code is part of the open release, community-run evals on standard benchmarks (MMLU, HellaSwag, etc.) should emerge quickly.

How to try it

Both the base and instruct variants of Aether-7B-5Attn, plus a live interactive demo, are publicly available on Hugging Face. Search for VIDRAFT or Aether-7B-5Attn on huggingface.co to find the model cards and repository.

You can pull the model with the Hugging Face CLI:

pip install huggingface_hub
huggingface-cli download VIDRAFT/Aether-7B-5Attn

⚠️ Verify the exact repository slug on Hugging Face before running — use the official VIDRAFT organization page to confirm the published model ID.

Training code, hyperparameters, training logs, intermediate checkpoints, and evaluation scripts are all included in the public repository, consistent with the project's full-openness commitment.

FAQ

Q: How is this different from "open-weight" models like LLaMA or Mistral?
A: Open-weight models let you download and run weights, but typically don't publish training data composition, training code, or the configuration needed to reproduce training from scratch. Aether-7B-5Attn aims to publish all of that — data recipes, tokenization pipeline, training code, hyperparameters, logs, checkpoints, and eval code — so the entire manufacturing process is auditable and reproducible by anyone.

Q: Why does the Latin Square layout matter for the heterogeneous attention design?
A: When you mix multiple attention types across layers, naïve assignment can lead to certain mechanisms clustering at specific depths (e.g., all sparse attention in early layers, all full attention late). A Latin Square constraint distributes each type evenly across depth positions, preventing depth-wise bias and making it easier to isolate the contribution of each mechanism experimentally.

Q: Is this model production-ready, or more of a research release?
A: Based on the source article, Aether-7B-5Attn is framed as a research-grade foundation model release emphasizing transparency and reproducibility. VIDRAFT's Darwin family remains their production-tier line. That said, the Apache-2.0 license means you can use and build on it commercially.

Originally reported by 전자신문 (2026-07-20) — source article.