Sutra-1.3B
Apache-2.0A Mixture-of-Experts language model pretrained from scratch in pure PyTorch.
1,611
downloads last month
Mixture of ExpertsMulti-head Latent AttentionPretrained from scratchEnglish + HindiPyTorch
1.32B parameters with Multi-head Latent Attention and DeepSeek-style auxiliary-loss-free expert routing — implemented from the papers, not adapted from an existing codebase. Own BPE tokenizer, own streaming data pipeline, own training loop. No pretrained weights, no transformers Trainer, no reference implementation anywhere in the stack.
Architecture, training & evaluationClick anywhere to expand
Architecture
- Parameters
- 1.32B total / 0.28B active (4.7× sparsity)
- Experts
- 48 routed + 1 shared, top-4
- Routing
- Sigmoid scoring, bias-based load balancing
- Attention
- MLA (Multi-head Latent Attention), kv_lora_rank 256
- Layers
- 16 — layer 0 dense, 1–15 MoE
- d_model / context
- 1024 / 4096
- Vocabulary
- 48,000 BPE (English + Devanagari)
Training
Pretraining4× RTX 6000 Ada · 4d 9h
18B tokens — English, Hindi, code, math
SFT18h
200K conversations
DPO6h
100K preference pairs
Evaluation
| Task | Random | Base | SFT | DPO |
|---|---|---|---|---|
| HellaSwag | 25.0 | 38.4 | 39.8 | 40.4 |
| ARC-easy | 25.0 | 45.0 | 44.8 | 45.0 |
| PIQA | 50.0 | 62.6 | 65.4 | 65.6 |
| WinoGrande | 50.0 | 50.6 | 49.0 | 49.0 |
Log-likelihood scoring, 500 examples per task, length-normalised accuracy. Pretraining held-out perplexity 15.00; SFT held-out perplexity 5.49.
Reading the numbers honestly
- ARC-easy and PIQA sit well above chance, so the model learned real commonsense — not just fluent grammar.
- WinoGrande sits at chance. That is the clearest signal of what 0.28B active parameters cannot buy: the pronoun-resolution reasoning that task measures never appeared.
- DPO's held-out preference accuracy came out at 47.5% against a 50% baseline, so the alignment stage did not generalise. The SFT and DPO checkpoints perform about equally.
- Trained on 18B tokens — roughly 500× less than comparable 1B models such as Llama 3.2 1B. It writes fluent English and follows formatting well, but does not reliably recall facts. Pair it with retrieval for anything knowledge-dependent.