EdgeRock-4B

EdgeRock-4B is the first open-source release of EdgeRock, the fixed-memory model family from Rock Matrix. It is a 4B-parameter text+image reasoning model (derived from Qwen3.5-4B) whose history memory does not grow with context length: at 256K tokens it needs only 72 MiB of attention KV per sequence—99.1% less than Qwen3.5-4B's 8 GiB.

Weights: huggingface.co/rockmatrix-ai/EdgeRock · Code: github.com/rockmatrix-ai/edgerock · Paper: arXiv 2610.10118 · Site: rockmatrix.ai/models

Key points

1. Architecture: 24 recurrent layers + 8 RMA attention layers. Recurrent state summarizes context, a 1,024-token local window handles recent tokens, and Rock Memory Anchors (RMA)—a fixed bank of 1,024 learned key-value records per KV group—pin important distant details.
2. O(1) memory: 72 MiB attention KV (97.76 MiB total persistent history) per sequence, regardless of whether it reasons for 4K or 256K tokens.
3. Reasoning: 82.9% mean pass@1 on AIME 2024–2026 vs at most 18.8% for 16 other bounded-state models (e.g. RWKV-7 class); 84.9% mean across six visual benchmarks.
4. Serving: 512 concurrent long-generation sequences on one Hopper 80GB GPU; 4.94x Qwen3.5-4B throughput on average, up to 6.42x; 233.7 tok/s decode at batch 1 with 128K in / 128K out.
5. Ablation: disabling RMA drops long-context query accuracy from 94.1% to 0.4%.

Why it matters

EdgeRock is a named, open alternative to ever-growing KV caches—relevant for long-running agents, on-device and high-concurrency serving.

How to try it

1. Download weights from Hugging Face and follow the inference instructions in the GitHub repo.
2. Use it for long reasoning, coding and agent workflows where context keeps growing; budget memory per sequence as a constant (~98 MiB history + weights).
3. Read the technical report for memory math, benchmarks and RMA ablations before comparing with full-attention 4B models.