NVIDIA Quietly Drops Nemotron Labs Dory 1.0: 4B Looped Open Model with 256K Context Window
NVIDIA released Nemotron Labs Dory 1.0 on Hugging Face, introducing a 4-billion-parameter looped transformer architecture with weight sharing across recurrent blocks and a 256,000-token context buffer.
Without promotional fanfare or conference presentations, NVIDIA published Nemotron Labs Dory 1.0 to the Hugging Face Hub on October 18, 2026.
Packaged as a 4-billion-parameter bfloat16 checkpoint (NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16), Dory represents a departure from standard feed-forward transformer scaling by adopting a looped transformer architecture that combines compact memory residency with deep recurrent reasoning.
The Looped Transformer Mechanism
Standard transformers stack distinct layer blocks sequentially (e.g., 32 or 48 unique attention layers). While this increases representational capacity, it scales memory footprint linearly.
Dory 1.0 instead defines a single 8-layer transformer block and loops representations through it multiple times:
Dory 1.0 Looped Layer Execution
┌─────────────────────────────────────────────────────────────┐
│ Input Tokens (up to 256,000 tokens) │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 8-Layer Core Transformer Block (4 Billion Unique Parameters)│
│ ├── FlashAttention-4 Matrix Engine │
│ ├── Recurrent Hidden State Injection │
│ └── Dynamic Halt Predictor (Adaptive Loop Depth) │
└──────────────────────────────┬──────────────────────────────┘
│
┌──────────────────┴──────────────────┐
▼ (Iterate Loop 1..4 Passes) ▼ (Halt Condition Met)
┌──────────────────────────────┐ ┌─────────────────────────────┐
│ Recurrent State Transition │ │ Final Output Head & Token │
└──────────────────────────────┘ └─────────────────────────────┘
By looping representations between two and four times depending on task complexity, Dory achieves reasoning depth comparable to a 14B or 16B parameter feed-forward model while requiring only 4B parameter storage in active GPU memory.
Benchmark Comparison and Retrieval Performance
Despite its compact parameter footprint, Dory 1.0 demonstrates high marks across long-context retrieval and code analysis:
| Evaluation Suite | Llama-3.2-3B (Baseline) | Nemotron Dory 1.0 (4B Looped) | Mistral-7B v0.3 |
|---|---|---|---|
| Parameters in Memory | 3.2 Billion | 4.1 Billion | 7.2 Billion |
| Native Context Length | 128,000 Tokens | 256,000 Tokens | 32,000 Tokens |
| Needle-In-A-Haystack (256k) | Fail (>128k) | 100.0% Exact Retrieval | N/A |
| HumanEval-Python (Pass@1) | 68.4% | 79.2% | 72.8% |
| GSM8K (5-shot Math) | 77.2% | 85.6% | 81.4% |
| VRAM Consumption (FP8) | 3.8 GB | 4.4 GB | 8.2 GB |
Why NVIDIA Named the Project Dory
In popular culture, the fictional character Dory is remembered for fleeting memory. NVIDIA engineers inverted the trope: by pairing a compact 4B weight footprint with a 256k context window and zero memory loss during long-document scanning, the model maintains perfect recall across thousands of document pages.
The checkpoint is available immediately for testing via NVIDIA NeMo, vLLM, and TensorRT-LLM frameworks.