VIDRAFT Releases Aether-7B-5Attn: A Fully Transparent MoE Foundation Model on Hugging Face (Apache-2.0)

TL;DR: Korean deep-tech AI startup VIDRAFT has open-sourced Aether-7B-5Attn, a 6.59B-parameter Mixture-of-Experts foundation model on Hugging Face under the Apache-2.0 license. Unlike typical open-weight releases, this one ships training data composition details, training code, hyperparameters, logs, and evaluation code alongside the weights — making the full development pipeline reproducible. If you care about auditable, verifiable open-source AI, this is worth a look.

What it is

Aether-7B-5Attn is a Mixture-of-Experts (MoE) foundation model developed entirely in-house by VIDRAFT (비드래프트), a Korean Pre-AGI deep-tech startup whose research spans AI model development and physical-science R&D solutions (physics, chemistry, life sciences).

Key facts from the release:

• Total parameters: 6.59 billion
• Active parameters per token at inference: approximately 2.98 billion (sparse activation via MoE routing)
• Architecture depth: 49 transformer layers
• Attention heads: 5 distinct attention mechanism types (see How it works below)
• License: Apache-2.0 — commercially usable, modifiable, redistributable
• Release artifacts: model weights (base + instruct variants), training data composition breakdown, training code, hyperparameters, training logs, and evaluation code
• Training corpus size: approximately 144.2 billion tokens
• Training data composition: Mathematics 37.8%, Korean 21.6%, English 21.6%, with the remainder covering other domains
• Available on Hugging Face: base model, instruct model, and a live demo

VIDRAFT is also known for its Darwin model family, its inference acceleration engine VKAE, and its lightweight runtime engine VKUE. The company has surpassed 1 million cumulative downloads on Hugging Face across its model releases.

How it works

The two most architecturally interesting design decisions in Aether-7B-5Attn are its attention diversity strategy and its MoE sparse activation.

Five-Attention-Type Hybrid

Rather than stacking identical attention layers throughout the network, VIDRAFT combines five conceptually distinct attention mechanisms within a single model:

1. Full attention — standard dense self-attention over the full context window
2. Differential attention — a mechanism designed to reduce attention noise by computing differences between attention maps
3. Sliding Window attention — restricts each token's attention to a local neighborhood, reducing quadratic complexity for long sequences
4. NSA-family sparse attention — a Native Sparse Attention variant that selectively attends to a learned sparse subset of tokens
5. Hybrid attention — a combination layer that blends behaviors from the above types

The 49 layers are arranged in a 7×7 Latin Square pattern. A Latin Square ensures that each attention type appears exactly once in every row and column of the arrangement — meaning no single attention type dominates any particular depth band of the network. The result is a principled, balanced distribution of attention behaviors across the model's depth.

MoE Sparse Activation

The MoE design means that only ~2.98B of the 6.59B parameters are activated per forward pass, keeping inference compute closer to a 3B-class dense model while retaining the representational capacity of a larger one. This is the standard MoE efficiency trade-off: more parameters in storage, fewer active during inference.

Training Data Philosophy

The heavy weighting toward mathematics (37.8%) is a deliberate choice to build strong reasoning priors. The bilingual Korean/English split (21.6% each) makes this one of the few models explicitly co-optimized for both languages from pretraining, rather than treating Korean as a fine-tuning afterthought.

Benchmarks & results

The source article does not disclose specific benchmark scores or leaderboard rankings for Aether-7B-5Attn at this time. VIDRAFT has previously reported results on major AI performance benchmarks and drug-discovery-related evaluations through its Darwin model family, but numerical results for this specific model have not been published in the covered press release.

What is confirmed qualitatively:

• The model is designed for strong bilingual (Korean + English) performance
• The mathematics-heavy training corpus targets reasoning capability
• The sparse MoE design is intended to deliver favorable inference efficiency relative to parameter count

Check VIDRAFT's Hugging Face model cards for any benchmark tables published directly alongside the weights.

How to try it

The model is publicly available on Hugging Face. You can browse the model card, download weights, and access the live demo directly through VIDRAFT's Hugging Face profile. The base and instruct variants are both available.

# Install the Hugging Face CLI if you haven't already
pip install huggingface_hub

# Search for the model on Hugging Face (replace with the exact repo path on the model card)
huggingface-cli download vidraft/Aether-7B-5Attn

⚠️ Note: The exact Hugging Face repository path (org/model-name) is not specified in the source article. Navigate to huggingface.co and search for Aether-7B-5Attn or VIDRAFT to find the canonical repo URL before downloading.

The release also includes training code and evaluation code, so researchers can reproduce training runs or plug their own evaluation harnesses against the published checkpoints.

FAQ

Q: Is this a true open-source release or just an open-weight drop?
A: VIDRAFT explicitly targets the "true open-source" end of the spectrum. Beyond weights, they publish training data composition, training code, hyperparameters, training logs, and evaluation code — making the development pipeline auditable and reproducible by external researchers and organizations.

Q: Why does the MoE design matter for deployment?
A: With ~2.98B active parameters per token instead of the full 6.59B, inference FLOP cost is significantly lower than a dense 7B model. In practice, this means you may be able to serve it on hardware sized for a ~3B dense model while retaining stronger representational capacity — a meaningful cost/quality trade-off for production deployments.

Q: Is this model suitable for non-Korean use cases?
A: Yes. English accounts for 21.6% of the training data (equal to Korean), and the mathematics-heavy corpus (37.8%) is largely language-agnostic for reasoning tasks. The model is explicitly designed to serve both Korean and English-language environments.

Q: Can I use this commercially?
A: The Apache-2.0 license permits commercial use, modification, and redistribution. Always verify the specific license terms on the Hugging Face model card before production deployment.

Originally reported by 동아일보 (2026-07-20) — source article.