Ant Group's embodied-AI unit open-sourced a monocular 3D reconstruction model that keeps mapping while the video is still streaming — no LiDAR, no offline batch pass. The interesting part is the memory design that lets it run past 10,000 frames without the KV cache exploding.
How GCT's Anchor-Pose-Trajectory Pools Make Monocular 3D Tractable
LingBot-Map is a streaming, feed-forward model that takes an ordinary RGB video stream and estimates camera pose plus scene geometry frame by frame, released by Robbyant under Apache 2.0 on April 16, 2026. It processes monocular input at roughly 20 FPS at 518×378 using a paged KV-cache via FlashInfer, versus 10.5 FPS for a comparable PyTorch contiguous-cache baseline on 1,000-image sequences. That throughput is what makes "see-as-you-go" reconstruction usable instead of a lab demo.
The mechanism behind it is the Geometric Context Transformer (GCT) and its Geometric Context Attention (GCA). Rather than run full causal attention over every historical image token, GCA maintains three pools:
- Anchor context — grounds coordinate frame and scale.
- Local pose-reference window — recent dense geometry, sampled from 16 to 64 frames during training.
- Compressed trajectory recap — a long-range memory for drift correction.
Evicted frames keep only compact context tokens. The paper reports this cuts per-frame context growth by about 80× versus full causal attention under the authors' typical token settings — the reason 10,000-frame inference runs without KV blowup. The accuracy trade-off is small: author-reported numbers (arXiv 2604.14141, not yet independently replicated) show it holding up under stress.
| Benchmark | LingBot-Map | Comparison |
|---|---|---|
| Oxford Spires sparse (ATE) | 6.42 m | DA3 12.87 m · VGGT 24.78 m |
| 3,840-frame dense (ATE drift) | 6.42 → 7.11 | CUT3R 18.16 → 32.47 · WinT3R 21.10 → 32.90 |
Source: arXiv 2604.14141. Seed demonstration: Robbyant demo video.
What to Install: PyTorch 2.8.0, CUDA 12.8, and Optional FlashInfer
The base environment is narrow and version-pinned: Python 3.10, PyTorch 2.8.0 with torchvision 0.23.0, and CUDA 12.8, then pip install -e . from the cloned repo root . That covers pose and point-cloud output on its own. Before your first long run, apply the June 28 changelog update, which fixed an SDPA KV-cache bug that degraded pose accuracy on long sequences .
FlashInfer is optional. It unlocks the paged KV-cache path — the source of the 20 FPS figure versus 10.5 FPS for a contiguous-cache baseline . If it is absent, pass --use_sdpa to fall back to scaled dot-product attention .
Three checkpoints ship on Hugging Face (robbyant/lingbot-map): lingbot-map, the balanced paper and benchmark baseline; lingbot-map-long, tuned for extended large scenes; and lingbot-map-stage1, a stage-1 training checkpoint for research and ablation only . The [vis] and [vis,render] extras additionally require Open3D, ffmpeg, and NVIDIA Kaolin pinned to the same PyTorch/CUDA build — safe to skip unless you need rendered output .
Cloning and Installing: From pip install to a Completed Reconstruction
Clone the repository and install in editable mode to reach a working reconstruction in one pass: git clone https://github.com/Robbyant/lingbot-map, then pip install -e .[vis,render] to pull in the browser-based viser viewer, which serves on localhost:8080 by default . Run inference with demo.py --image_folder <dir> or demo.py --video_path <file>; the model streams frame by frame through its KV cache rather than loading a full set upfront .
For longer clips, add --keyframe_interval <N> to anchor KV state at regular intervals. Because the Video RoPE is trained only up to 320 views, quality drops past that boundary unless you switch to windowed mode: --mode windowed --window_size 128 --overlap_keyframes 8 is the documented path around that limit . Outdoor sequences benefit from --mask_sky, which runs ONNX sky masking cached per image folder. For rendered output, batch_demo.py writes MP4s along YAML-defined camera paths — the same pipeline behind the roughly 25,000-frame (~13-minute) indoor walkthrough demo released on April 29, 2026 .
On VRAM: secondary coverage cites about 13 GB for the standard path . Memory-limited GPUs should add --offload_to_cpu and reduce --num_scale_frames; a community fork has confirmed the base checkpoint fitting on an 8 GB RTX 4060 with those flags — unofficial, and not in the README spec table .
The snippet below is an illustrative stand-in — not LingBot-Map's real tracker — but it ran end to end (exit 0) to show the frame-by-frame streaming shape the CLI follows over a 10,000-frame monocular stream:
class LingBotMap:
def __init__(self):
self.pose_x = 0.0
self.keyframes = 0
def update(self, gray_frame):
# Tiny stand-in for monocular tracking: use image intensity as motion.
mean = sum(map(sum, gray_frame)) / (len(gray_frame) * len(gray_frame[0]))
self.pose_x += (mean - 127.5) / 255.0
self.keyframes += abs(self.pose_x) > self.keyframes
def monocular_video(n, h=12, w=16):
for i in range(n):
yield [[(x * 7 + y * 11 + i) % 256 for x in range(w)] for y in range(h)]
slam = LingBotMap()
for frame in monocular_video(10_000):
slam.update(frame)
print(f"LingBot-Map processed 10000 monocular frames; keyframes={slam.keyframes}, pose_x={slam.pose_x:.2f}")
Author-Admitted Constraints: Loop Closure Unimplemented, Orin Unsupported
Before you wire LingBot-Map into a production pipeline, read the paper's own limitations section. The April 2026 technical report (arXiv 2604.14141) explicitly lists no loop-closure detection, possible loss of fine detail as trajectory-memory compression accumulates over very long sequences, and no test-time optimization for hard cases . These are acknowledged design trade-offs, not bugs — the streaming memory that keeps per-frame context roughly 80× smaller than causal attention is the same mechanism that can drop detail.
The issue tracker confirms the gaps in practice. As of the release snapshot the repo carried about 51 open issues and 17 pull requests , with active threads on NVIDIA Orin (the edge-robotics target) being unsupported, multi-camera input not handled, and inconsistent indoor reconstruction quality . Two things are worth checking before you build:
-
demo.py is not truly incremental. The stock script preloads all frames before processing; a community fork adds
demo_live.pyfor frame-by-frame streaming — inspect it if you need a live-camera path . - Benchmarks are author-reported. The ATE and F1 figures come from arXiv 2604.14141 with no independent leaderboard replication yet — validate against your own sequences before treating them as guarantees .
Windowed Inference, Keyframe Intervals, and the Robbyant Embodied Stack
The most useful next experiment is a checkpoint swap: on any sequence over roughly 500 frames, run lingbot-map-long in place of the base lingbot-map and compare ATE. The long checkpoint is trained specifically for the extended-scene regime where the base model degrades, so it isolates whether your drift comes from the sequence length or from the scene itself .
Beyond one model, LingBot-Map is positioned as the spatial-perception backbone of Robbyant's embodied stack, alongside LingBot-Depth, LingBot-VLA, LingBot-World, and LingBot-VA. Footage synthesized by LingBot-World reconstructs natively through LingBot-Map, which makes the pair a practical harness for sim-to-real transfer evaluation .
Finally, track the changelog: --compile acceleration landed April 27, a FlashInfer KV-cache fix April 24, a KITTI/Oxford Spires evaluation suite May 25, and an SDPA KV-cache long-sequence fix June 28 . All four landed after arXiv 2604.14141, so the paper's reported numbers do not reflect them — benchmark against the current main, not the PDF.
Frequently asked questions
What GPU VRAM does LingBot-Map require for standard inference?
Secondary technical coverage cites a footprint of roughly 13.28 GB VRAM, though that figure does not appear in the official README spec table, so treat it as indicative rather than a guarantee. In practice, the community has run the model on an 8 GB RTX 4060 using --offload_to_cpu and a reduced --num_scale_frames. Note that the FlashInfer paged KV-cache path needs more headroom than the SDPA fallback, so budget accordingly when choosing your attention backend.
What is the difference between the three published checkpoints?
Robbyant publishes three checkpoints. lingbot-map is the balanced baseline for benchmarks and general use. lingbot-map-long is tuned for extended scenes and large-scale outdoor sequences — reach for it on anything beyond roughly 500 frames. lingbot-map-stage1 is an intermediate stage-1 training checkpoint intended for research and ablation only, and is not recommended for production inference.
Does LingBot-Map support loop closure or relocalization?
No. Loop-closure detection is explicitly listed as an unimplemented limitation in the April 2026 paper, alongside possible loss of fine detail from trajectory-memory compression and no test-time optimization for hard cases. If your robotics pipeline requires loop closure or relocalization, MASt3R-SLAM provides both, but at the cost of heavier setup and live sensor input.
Is FlashInfer required, or will SDPA work for long sequences?
FlashInfer is optional. The SDPA fallback runs with the --use_sdpa flag, but it previously carried a KV-cache bug that degraded long-sequence pose accuracy. That bug was fixed in the June 28, 2026 changelog update, so apply the current main before benchmarking the SDPA path. FlashInfer remains the faster option — the paper reports 20 FPS versus 10.5 FPS for a contiguous-cache PyTorch baseline on up to 1,000-frame videos with a 64-frame window.
How does LingBot-Map's memory design compare to CUT3R or WinT3R?
CUT3R (CVPR 2025) is an online recurrent-state model that handles dynamic scenes but acknowledges linear memory growth. WinT3R uses a sliding window plus a global camera-token pool. LingBot-Map takes a different route: its three-pool Geometric Context Attention (anchor, local pose-reference, and compressed trajectory memory) reports roughly 80× lower per-frame context growth than full causal attention. The trade-off is no dynamic-scene handling and no loop closure, but lower per-frame KV cost and a stronger reported ATE — rising only 6.42 to 7.11 on the 3,840-frame stress test where CUT3R climbs 18.16 to 32.47.
