Bottom Line: Mamba-2 is the better fit when long-context memory and per-token inference efficiency dominate; Transformers remain the safer production default when ecosystem maturity, broad checkpoint availability, and retrieval-sensitive quality consistency matter more; and Jamba or another hybrid SSD+attention model is the practical middle path when you need some of both.
How we compared Mamba-2, Transformers, and mixed SSD+attention models
The SSD paper (arXiv:2405.21060) establishes that Mamba-2 and masked attention are not competing designs from separate traditions — they are two contraction orders over the same underlying semiseparable matrix structure. That theoretical unification produced a new core layer reported to be 2–8× faster than Mamba's fused selective scan, while staying competitive with Transformers on language modeling. The comparison that matters for a production decision is not "did the paper claim a win" but rather: which architecture wins on which specific criteria, how you weight those criteria for your workload, and under what operational constraints.
The five criteria that govern this decision, ranked by production importance for most teams, are:
| Rank | Criterion | Why it matters | Mamba-2 signal | Transformer signal |
|---|---|---|---|---|
| 1 | Long-context memory | Determines whether inference fits on your target GPU budget | Constant recurrent state — no KV cache growth | KV cache scales with sequence length and batch size |
| 2 | Long-context quality | Decides whether the model can preserve exact retrieval behavior | Competitive; mixing in attention layers improves quality further | Full attention captures arbitrary token interactions |
| 3 | Core-layer throughput | Controls training cost and iteration speed | 2–8× faster vs. fused-scan baseline (paper-reported) | FlashAttention-2 narrows the gap at short-to-medium lengths |
| 4 | Training infrastructure | Determines which kernels, frameworks, and recipes are already proven at scale | Newer; fewer production references at 70B+ scale | Mature ecosystem: FSDP, DeepSpeed, vLLM, TGI all battle-tested |
| 5 | Ecosystem integration | Affects time-to-first-result for your team | Added to Hugging Face Transformers 2024-08-06; checkpoint support still limited | Broad: thousands of checkpoints, serving stacks, and fine-tuning recipes |
Before choosing Mamba-2 vs Transformers, compare sequence-length distribution, retrieval sensitivity, serving constraints, and how much integration risk you can absorb. Throughput and memory are where Mamba-2 leads. Ecosystem and quality consistency — particularly for instruction-following tasks that depend on precise in-context retrieval — are where Transformers remain stronger. No single criterion dominates every deployment scenario; you must weight them against your workload's actual sequence length distribution and operational constraints.
Evaluation criteria that matter for production decisions
The gap that top competitors miss: they explain the SSD theory but stop before ranking the criteria by production weight.
| Criterion | Mamba-2 | Transformer | Notes |
|---|---|---|---|
| Throughput (core layer) | Faster — 2–8× over fused-scan | Baseline | Gain is most pronounced at long sequence lengths |
| Inference memory | Lower at long context | Grows with context | SSM recurrent mode: O(1) per-token state |
| Training memory | Lower at long context | Quadratic attention memory | Attention-free SSM layers eliminate activation memory from attention |
| Integration friction | Higher — limited checkpoint support | Low | Only mistralai/Mamba-Codestral-7B-v0.1 confirmed working via HF docs |
| Ecosystem maturity | Lower | High | vLLM, TGI, PEFT, quantization all native to Transformers |
Throughput and memory are where Mamba-2 leads. Ecosystem and quality consistency — particularly for instruction-following tasks that depend on precise in-context retrieval — are where Transformers remain stronger. No single criterion dominates every deployment scenario; you must weight them against your workload's actual sequence length distribution and operational constraints.
What sources and evidence we used
Structured State Space Duality is the theoretical framework introduced in the paper "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" (Dao & Gu, arXiv:2405.21060). The framework's core claim is that structured state-space models and variants of attention are closely related through structured semiseparable matrices: "We show that these families of models are actually quite closely related, and develop a rich framework of theoretical understanding that connects structured state space models and variants of attention." The duality allows Mamba-2 to be implemented with a hardware-efficient two-pass algorithm that replaces the sequential fused scan in Mamba-1.
Pro Tip: Read the SSD paper directly (arXiv:2405.21060) and the Hugging Face Transformers Mamba2 docs as the canonical references for theory and integration details.
Pro Tip: For integration, the Hugging Face Transformers Mamba2 docs are the canonical practical reference — they include the only officially supported checkpoint path and note the left-padding and batching constraints that affect evaluation setup.
At-a-glance comparison of Mamba-2 vs Transformers
Mamba-2 is faster at the layer level and memory-efficient at long context; Transformers hold the ecosystem and quality edge. The paper's own framing is that Mamba-2 "continues to be competitive with Transformers on language modeling" — not that it surpasses them universally.
| Dimension | Mamba-2 | Transformer |
|---|---|---|
| Core-layer speed | 2–8× faster than Mamba fused scan | Optimized via FlashAttention-2 |
| Memory behavior | Fixed recurrent state; no KV cache growth | KV cache grows linearly with context |
| Long-context fit | Strong when sequence length dominates | Strong when exact retrieval and broad tooling matter |
| Integration status | Limited; HF Transformers support landed 2024-08-06 | Mature; broad checkpoint and serving support |
Where Mamba-2 wins on sequence length and kernel efficiency
Mamba-2's structural advantage comes directly from the SSD framework: by reframing selective SSMs as a semiseparable matrix contraction, the paper enables a blocked, hardware-efficient algorithm that avoids the sequential dependency bottleneck of Mamba-1's fused scan. The result is a core layer that the authors report as 2–8× faster than that fused-scan baseline.
That speedup is most consequential when sequence length is the binding constraint. Transformer attention complexity scales quadratically with sequence length — "Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity," as the Long Range Arena paper framed the motivation for this entire line of work. Mamba-2's recurrent inference mode maintains a fixed-size state regardless of how many tokens precede the current position, which eliminates both the KV cache memory pressure and the O(n²) attention compute at inference time.
Watch Out: The 2–8× speedup refers to the core SSD layer compared to Mamba-1's fused selective scan, and it matters most on long sequences where the blocked matmul can amortize overhead; on short sequences or with a less favorable kernel path, the advantage shrinks quickly. Do not treat this number as a universal latency SLA.
Where Transformers still hold practical advantages
Mamba-2 cannot currently replace Transformers in production for most teams. The paper explicitly frames Mamba-2 as "competitive with Transformers on language modeling" — not superior across the board. Three concrete constraints reinforce this.
Ecosystem depth: The Transformer ecosystem has years of optimization work behind it — FlashAttention-2, PagedAttention in vLLM, FSDP and DeepSpeed training recipes, PEFT/LoRA fine-tuning, and quantization pipelines all target the Transformer attention pattern natively. Mamba-2's integration into Hugging Face Transformers only landed on 2024-08-06, and supported checkpoints remain narrow.
Quality on in-context tasks: SSMs' recurrent compression of context can miss fine-grained token interactions that full attention captures. The paper itself notes better quality results when a small number of attention layers are mixed into the SSD stack — the pure Mamba-2 configuration is not always the highest-quality option.
Checkpoint availability: The official Hugging Face docs use mistralai/Mamba-Codestral-7B-v0.1 because the original Mamba-2 research checkpoints are not yet supported through that integration path. Teams evaluating Mamba-2 against a Llama 3.1 70B or similar Transformer baseline must account for this asymmetry.
Bottom Line: Use Transformers when quality consistency, broad tooling, and an established serving stack matter more than maximum long-context memory efficiency. Keep some attention layers if you need better retrieval fidelity, because the SSD paper's own hybrid framing shows that mixed SSD+attention can recover quality while preserving much of the memory benefit. The ecosystem advantage is real and currently unbridged by Mamba-2 for most production teams. Hybrid SSD+attention models (see Jamba below) may offer a workable middle path.
Mamba-2 and the SSD paper: what changed in practice
The SSD framework does not merely rename existing concepts — it changes what computations are tractable to implement efficiently on GPU. The central insight is that both selective SSMs and a class of masked attention operations correspond to different ways of contracting the same semiseparable matrix. Semiseparable matrices have bounded rank in off-diagonal blocks, and that structure enables blocked matrix algorithms that decompose the sequence computation into chunks processable in parallel, then recombine the per-chunk SSM states.
This gives Mamba-2 something Mamba-1 lacked: a natural mapping to tensor-core-friendly tiled matrix operations. Mamba-1's fused scan was sequential at the token level, forcing the kernel to process tokens in order. The SSD dual — expressed as a matrix multiplication over blocked semiseparable matrices — can be tiled across sequence chunks in the same way FlashAttention tiles the attention matrix across SRAM.
| Model / formulation | Contraction order | Structure being contracted | Practical effect |
|---|---|---|---|
| Selective SSMs (Mamba-1-style) | Sequential fused scan | Semiseparable state update | Limited token-level parallelism |
| Mamba-2 SSD layer | Blocked two-pass contraction | Same semiseparable matrix structure | Tensor-core-friendly tiled execution |
| Masked attention | Dense attention contraction | Same semiseparable matrix structure under SSD framing | Full token interaction, high parallelism |
| Aspect | Mamba-1 selective SSM | Mamba-2 SSD layer | Masked Attention |
|---|---|---|---|
| Computation structure | Sequential fused scan | Chunked semiseparable matmul | Dense QKV matmul |
| Parallelism | Limited by sequential dependency | Blocked: parallel across chunks | Fully parallel over tokens |
| State size | Recurrent hidden state | Recurrent hidden state (expanded) | KV cache (grows with n) |
| HW efficiency on H100 | Moderate (scan-limited) | High (tensor-core-mapped) | High (FlashAttention-2) |
| Theoretical connection | SSM | SSM = semiseparable contraction A | Attention = semiseparable contraction B |
Why the paper reports 2–8× faster core layers
The reported speedup baseline is Mamba-1's fused selective scan CUDA kernel, not a FlashAttention-2 Transformer. The SSD layer replaces that sequential scan with a chunked semiseparable matrix multiply. On hardware with high tensor-core throughput — such as the NVIDIA H100 — the blocked matmul maps efficiently to the hardware's tiled execution model in a way the sequential scan could not.
| Configuration | Baseline | Core-layer speed vs. Mamba-1 fused scan | Notes |
|---|---|---|---|
| Mamba-2 SSD (reported range, short sequences) | Mamba-1 fused selective scan | ~2× | Chunking overhead reduces relative gain |
| Mamba-2 SSD (reported range, long sequences) | Mamba-1 fused selective scan | ~8× | Blocked matmul amortizes overhead |
| Baseline: Mamba-1 fused scan | Mamba-1 fused selective scan | 1× (reference) | Sequential; limited HW parallelism |
The 2–8× range is not a confidence interval — it is a sequence-length-dependent operating range. As the SSD paper states: "a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling."
Watch Out: This benchmark measures the core layer in isolation. End-to-end training or serving throughput includes data loading, optimizer steps, sampling logic, and framework overhead — none of which benefit from the SSD kernel speedup. Expect a smaller wall-clock improvement in full-pipeline measurements.
Why mixed SSD + attention models can beat pure Mamba-2
Pure Mamba-2 compresses all context into a fixed recurrent state. That compression is the source of its memory efficiency — and the source of its quality ceiling. When a task requires precise retrieval of a token seen many steps earlier (e.g., multi-hop reasoning, exact copying, or long-document QA), the lossy compression of the SSM state can cause errors that attention — which retains exact token representations in the KV cache — does not make.
The SSD paper explicitly reports improved quality when a small number of attention layers are mixed into an otherwise SSD stack. The SSD framework makes this natural: because SSD and masked attention are dual formulations over the same semiseparable structure, mixing them in a single model is architecturally coherent rather than a hack.
| Choice | Throughput | Memory at long context | Quality on precise retrieval | Best use case |
|---|---|---|---|---|
| Pure Mamba-2 | Highest | Lowest | Good, not best | Long-context workloads where memory dominates |
| Pure Transformer | Moderate | Highest (KV cache) | Best | Retrieval-heavy workloads and broad production serving |
| Hybrid SSD + attention | High | Low-to-moderate | Better than pure Mamba-2 | Workloads needing both efficiency and stronger recall |
Decision rule: If your workload involves long-context retrieval or instruction-following tasks sensitive to precise token recall, pure Mamba-2 may underperform a Transformer of equivalent parameter count. Adding even a small fraction of attention layers (the SSD paper and Jamba both support this pattern) recovers most of that quality gap while preserving memory efficiency at long context.
Benchmarks and empirical signals to trust
The paper trains Mamba-2 models alongside Mamba-1 and Transformer baselines at matched parameter counts and dataset sizes — up to 8B parameters on up to 3.5T tokens — making the language-modeling comparison relatively controlled. The result reported is that Mamba-2 remains competitive with Transformers at these scales, with the core-layer speed advantage enabling faster training iteration.
| Signal | Reported outcome | Interpretation |
|---|---|---|
| Language modeling at matched scale | Competitive with Transformers | Mamba-2 is not a universal quality winner, but it is not merely a speed tradeoff either |
| Core-layer speed | 2–8× faster than Mamba fused scan | The performance gain comes from the SSD kernel, not from a better end-to-end serving stack |
| Mixed SSD + attention | Better quality than pure SSD in some settings | Retaining a small amount of attention is a practical quality lever |
For earlier SSM-vs-Transformer quality comparisons, the Mamba-1 paper provides a useful directional signal: "On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size". Mamba-2's SSD layer improves upon Mamba-1 architecturally, so this comparison establishes a lower bound on expected quality.
What the Long Range Arena and language-modeling results imply
Long Range Arena (LRA) is the standard controlled research suite for evaluating long-sequence models across synthetic and real-world tasks. Its motivation is direct: "Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity." LRA measures whether efficient architectures retain quality as sequence length grows — the regime where Mamba-2 theoretically benefits most.
| Evaluation context | Mamba-2 signal | Transformer signal | Caveat |
|---|---|---|---|
| LRA (long-range dependencies) | SSMs competitive or better at long range | Degrades at length without sparse attention | LRA is a research suite, not a production benchmark |
| Autoregressive LM (3B params) | Matches or exceeds same-size Transformer (Mamba-1 result; Mamba-2 competitive) | Strong baseline | Mamba-2 paper trained up to 8B; full benchmark table not publicly extracted |
| In-context retrieval | Weaker than full attention on precise recall | Strong | Gap narrows with hybrid SSD+attention |
Pro Tip: Treat LRA scores as directional, not deployment-predictive. LRA sequences are synthetically constructed; production tasks involve different noise distributions, vocabulary statistics, and retrieval patterns. Run your own evals on held-out data from your target domain before committing to architecture choice.
Why speed is not the same as end-to-end serving wins
A 2–8× faster core layer does not directly translate to a 2–8× end-to-end serving improvement. Several operational factors intervene when moving from isolated kernel benchmarks to a production inference stack.
The Hugging Face Transformers Mamba2 docs document two concrete issues that affect reproducibility of any benchmark you run through that integration path:
-
Batched vs. cached generation discrepancy: "This (and the addition of the reimplemented Mamba 2 kernels) results in a slight discrepancy between batched and cached generation." The docs also note that
cuda_kernels_forwarduses the original CUDA kernels if available. Whether you hit the original CUDA path or the reimplemented kernel path changes both speed and numerical output. -
Left-padding requirement: Mamba-2 batched generation requires left-padding, unlike the right-padding default common in Transformer batching pipelines. This changes how you construct evaluation batches and means naive comparison against a Transformer eval harness will produce incompatible results if padding convention is not matched.
Watch Out: If you benchmark Mamba-2 against a Transformer using a standard Hugging Face evaluation loop without explicitly setting
padding_side="left"for the tokenizer, controlling for kernel selection, and keeping precision mode fixed, your numbers will not be comparable. Small numerical discrepancies between batched and cached modes are expected and documented — do not over-interpret them as correctness failures.
How to decide between Mamba-2, Transformers, and Jamba
The choice between Mamba-2, Transformers, and Jamba reduces to three variables: sequence length, quality sensitivity, and how much operational risk your team can absorb from a less mature ecosystem.
| Use case | Best fit | Reasoning |
|---|---|---|
| Long-context generation (>16K tokens), memory-constrained | Mamba-2 | No KV cache growth; recurrent state is fixed |
| Streaming inference, low-latency per-token | Mamba-2 | O(1) recurrent step at generation time |
| Instruction-following, multi-hop QA, precise retrieval | Transformer or hybrid | Full attention preserves exact token representations |
| Large-scale production serving with existing tooling | Transformer | vLLM, TGI, quantization all native |
| Long-context with quality requirements above pure SSM | Jamba / hybrid | Mixes attention and SSM layers for both efficiency and quality |
| Research / ablation on SSM architectures | Mamba-2 | Best-documented SSM with SSD theory and HF integration |
Choose Mamba-2 when context length and memory dominate
Mamba-2 is the right choice when the sequence length exceeds what your serving infrastructure can handle efficiently with a KV cache, and when your task tolerates the mild quality gap on precise in-context retrieval.
Concretely: long-document summarization, streaming transcription, genomics sequence modeling, and retrieval-augmented generation pipelines where documents are very long but you do not need the model to copy specific tokens verbatim back to you. The fixed recurrent state means GPU memory usage does not scale with sequence length at inference — a meaningful operational constraint on multi-tenant serving hardware.
The Mamba-1 result — "outperforms Transformers of the same size and matches Transformers twice its size" on language modeling — suggests that in some regimes you can achieve Transformer-class quality at half the parameter count, which compounds the memory advantage.
Choose Transformers when quality, tooling, and broad support dominate
Transformers remain the safer default for most production deployments in 2026. FlashAttention-2 has substantially closed the efficiency gap at practical sequence lengths (up to 8K–32K tokens). Every major serving stack (vLLM, TGI, Triton Inference Server), training framework (DeepSpeed ZeRO, FSDP), and fine-tuning toolkit (PEFT, LoRA, QLoRA) is battle-tested against Transformer attention.
The ecosystem maturity gap is not abstract: when you run into an OOM, a numerical instability, or a generation quality regression in a Transformer serving stack, there is a documented playbook. For Mamba-2, that playbook is still being written, and the Hugging Face integration path currently restricts you to a single confirmed checkpoint (mistralai/Mamba-Codestral-7B-v0.1) — original Mamba-2 checkpoints are not yet supported. Teams that need to fine-tune at scale, deploy with quantization, or integrate with existing RAG pipelines should factor this constraint explicitly.
Where Jamba fits as a hybrid alternative
Jamba from AI21 Labs is the most visible production hybrid — it interleaves Transformer attention layers with SSM layers in a single model. The design directly implements the paper's caveat that mixing a small number of attention layers into an SSM stack recovers quality relative to pure Mamba-2.
| Architecture | Memory at long context | Quality on precise retrieval | Ecosystem maturity | Checkpoint availability |
|---|---|---|---|---|
| Pure Mamba-2 | Lowest | Good | Early-stage | Limited (HF: Codestral-7B) |
| Hybrid Jamba | Low-to-moderate | Better | Growing | AI21 managed |
| Pure Transformer | Highest | Best | Mature | Thousands of open checkpoints |
Jamba is the practical choice if you want SSM memory efficiency but cannot accept the quality reduction on retrieval-heavy tasks that pure Mamba-2 can produce. The trade-off: you take on a newer ecosystem with fewer community-contributed serving optimizations than the pure Transformer path, and you accept that the hybrid layer schedule is set by AI21 rather than tunable by your team.
Practical integration notes from the Hugging Face Mamba2 docs
Mamba-2 was added to Hugging Face Transformers on 2024-08-06. The integration exposes the model through the standard pipeline and AutoModel APIs, but with two operational constraints that differ from standard Transformer usage.
Watch Out: Mamba-2 batched generation requires left-padding. If you construct evaluation batches with the right-padding default common in Transformer pipelines, outputs will be incorrect. Set
tokenizer.padding_side = "left"before batched inference. Additionally, the Hugging Face docs note that batched and cached generation can produce slightly different outputs because the integration may use the original CUDA kernels when available and otherwise fall back to reimplemented Mamba 2 kernels; those kernel and precision differences are documented behavior, not a bug, but they mean you must control generation mode when comparing results.
Supported checkpoints and why the docs use Mamba-Codestral
Production Note: The official Hugging Face Transformers Mamba2 docs use
mistralai/Mamba-Codestral-7B-v0.1as the reference checkpoint because the original Mamba-2 research checkpoints are not yet supported through the documented integration path. If you attempt to load an original Mamba-2 checkpoint viaAutoModelForCausalLM.from_pretrained, it will either fail or produce unexpected behavior. Use the Codestral checkpoint for any initial integration testing. Monitor the Hugging Face Transformers model docs for Mamba2 for checkpoint support updates before designing an evaluation pipeline around a specific checkpoint.
What batched generation differences mean for evaluation
The Hugging Face docs explicitly warn: "This (and the addition of the reimplemented Mamba 2 kernels) results in a slight discrepancy between batched and cached generation." The source is kernel dispatch: the integration uses the original CUDA kernels when they are available in your environment (cuda_kernels_forward), and falls back to a reimplemented version otherwise. The two paths produce numerically close but not identical outputs.
Watch Out: When running evaluation benchmarks — perplexity, accuracy on classification tasks, or generation quality metrics — fix your generation mode (always batched or always cached), note which kernel path is active, and keep precision settings constant. Mixing batched and cached outputs in the same evaluation set will introduce non-deterministic variance. Small differences between modes are expected and do not indicate a model defect, but they will inflate variance in benchmark comparisons against Transformer baselines if not controlled.
FAQ on Mamba-2, Transformers, and Structured State Space Duality
Bottom Line: Mamba-2 is a strong candidate for long-context, memory-constrained workloads — its 2–8× layer-level speedup and constant inference state are real advantages. Transformers remain the safer production default due to ecosystem depth, broad checkpoint availability, and quality consistency on retrieval-sensitive tasks. For workloads that demand both, a hybrid SSD+attention model (Jamba or a custom mix) is the SSD paper's own recommended path. No single architecture dominates every regime; match architecture to sequence length, quality requirements, and the operational maturity your team can sustain.
Is Structured State Space Duality the same as attention?
No. Structured State Space Duality reveals that selective SSMs and a class of masked attention operations share the same underlying semiseparable matrix structure — and that each can be expressed as a different contraction order over that structure. The paper states: "a rich framework of theoretical understanding that connects structured state space models and variants of attention." Connection is not identity. SSMs maintain a fixed-size recurrent state and process tokens without explicit pairwise token interactions; attention computes exact pairwise scores over all tokens. The duality explains why both work and enables the Mamba-2 blocked algorithm, but Mamba-2 and a Transformer layer still behave differently on tasks that require exact token retrieval.
Pro Tip: The practical consequence of the duality is not interchangeability — it is the blocked matrix algorithm that maps Mamba-2's SSM computation onto tensor-core-friendly hardware operations. The quality difference between a pure SSM and full attention on retrieval tasks persists despite the theoretical relationship.
Is Mamba-2 faster than Transformers on every workload?
No. The paper's claim is a 2–8× speedup for the core SSD layer relative to Mamba-1's fused selective scan. That baseline is not FlashAttention-2, and the speedup range is sequence-length-dependent — the high end of the range applies at long sequences where the blocked matmul amortizes chunking overhead most effectively. At short sequence lengths, the advantage is closer to 2×. End-to-end serving latency includes sampling, tokenization, KV cache management (for Transformers), and framework dispatch — factors that can dominate the total latency budget at small batch sizes or short sequences.
Watch Out: Benchmarking Mamba-2 against a Transformer without controlling for batch size, sequence length, kernel availability, padding convention, and precision mode will produce numbers that are not comparable. The HF integration's documented batched-vs-cached discrepancy adds another confound. Design your benchmark to hold all variables constant except architecture.
Should I use mixed SSD and attention layers instead of pure Mamba-2?
Yes, if your workload includes any of these: multi-hop reasoning, document QA requiring precise token recall, long-context instruction following, or code generation where exact symbol copying matters. The SSD paper reports improved quality when a small number of attention layers are mixed into the SSD stack, and the theoretical duality makes such mixing architecturally clean. Pure Mamba-2 is the right choice only when memory and throughput at long context are the binding constraints and your task tolerates the quality trade-off — streaming summarization, online language model inference at very long context, and SSM-native sequence classification tasks are good fits. For anything where you cannot accept a quality reduction on precise retrieval, start with a hybrid.
Sources & References
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (arXiv:2405.21060) — Primary source for the SSD framework, Mamba-2 architecture, and 2–8× speedup claim
- Hugging Face Transformers Mamba2 Model Docs — Official integration documentation covering supported checkpoints, left-padding requirement, and batched-vs-cached generation behavior
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces (arXiv:2312.00752) — Mamba-1 paper providing the language-modeling baseline ("Mamba-3B outperforms Transformers of the same size")
- Long Range Arena: A Benchmark for Efficient Transformers (arXiv:2011.04006) — Standard research suite for long-sequence model evaluation; source for quadratic-attention complexity framing
- AI21 Jamba model family — Hybrid SSM+Transformer model family from AI21 Labs
- mistralai/Mamba-Codestral-7B-v0.1 on Hugging Face — Currently supported Mamba-2 checkpoint in the official HF Transformers integration path
Keywords
Keywords: Mamba-2, Transformers, Structured State Space Duality, state-spaces/mamba, mistralai/Mamba-Codestral-7B-v0.1, Hugging Face Transformers, selective state-space models (SSMs), masked attention, key-value cache, NVIDIA H100, Long Range Arena, Llama 3.1 70B, Jamba, FlashAttention-2
