The comprehensive, cutting-edge open-source framework, benchmark
and auto-research platform for next-generation Transformers

English · 简体中文

When the training budget is fixed, how should a language model spend its computation? NextTransformer trains recurrent depth, sparse and linear attention, residual-stream variants and hybrids on the same data, for the same 49.152B tokens, at the same theoretical training FLOPs per token, and evaluates them at the same checkpoints. Every result is an audited, file-backed record, and AI agents can research new architectures under the very same rules.

News

• 2026-11-05 · Scaling RFC: 4K-context Dense and MoE pretraining on ClimbMix at 200B and 400B tokens. Feedback on data, model configurations and evaluation is welcome.
• 2026-11-04 · NextTransformer-V by @Enderfga: a community multimodal track that trains every NextTransformer method on interleaved image and text tokens under the same compute-matched contract. Its first study, pre-registered, asks where hybrid backbones get the 2D layout of an image.
• 2026-10-31 · NextTransformer is now open source: 31 finished runs, 36 model cards, the full protocol, the documentation and the observatory.
• By 2026-11-14 · Arena v2, with full support for autonomous agent research.

Highlights

The observatory

Learning curves of every run, a replayable leaderboard, all 22 CORE tasks, architecture diagrams and model cards, in your browser.

Results

Full leaderboard · Training replay

Quick start

1. Install (Linux, NVIDIA GPUs, uv):

git clone https://github.com/Interactive-Active-AI/NextTransformer.git && cd NextTransformer
uv sync --frozen --extra gpu --extra upstream && source .venv/bin/activate
uv pip install --no-deps --require-hashes -r requirements/linear.txt   # linear-attention kernels

2. Prepare the data (ClimbMix at a pinned revision, about 600 GB, plus the tokenizer and CORE bundle):

cp site.native.example.json site.local.json        # set runtime_dir, resources and data_dir
hf download karpathy/climbmix-400b-shuffle --repo-type dataset \
  --revision 915333b4f8b8684f39aeaafea600fea6f43fb703 --local-dir /path/to/climbmix-400b-shuffle
source launch/env.sh --site-config site.local.json
python -m nexttransformer.prepare --site-config site.local.json

3. Train on one node with 8 GPUs:

launch/train.sh --config configs/baseline/dense_full.json --site-config site.local.json \
  --output runs/dense_full --micro-batch 8 --formal --resume auto

4. Multi-node training (2 nodes × 8 GPUs)

Install uv pip install accelerate==1.15.0 in the training environment. Submit one distributed job with the following per-node entry point. Map the platform's node rank, main node address and port to the three variables shown:

export FLA_DISABLE_BACKEND_DISPATCH=1   # registered FLA Triton path (linear and hybrid methods)
accelerate launch --config_file launch/accelerate.yaml \
  --machine_rank "$NODE_RANK" --main_process_ip "$MASTER_ADDR" --main_process_port "$MASTER_PORT" \
  -m nexttransformer.train --config configs/linear/gdn.json --site-config site.local.json \
  --output runs/gdn --micro-batch 16 --formal

On Slurm, submit sbatch launch/slurm.sbatch for the same example. The batch script gets the launch parameters from the allocation. Cluster setup and other node counts

Evaluation runs on schedule during training. The quick start guide covers smoke tests, evaluation, auditing and exporting a result.

Model zoo

Browse architectures by family

Family	Models
	Dense Baseline
	Standard Residual · AttnRes · HC · mHC
	UT-ACT · Fixed Recursive · Huginn · Ouro · DeepLoop · MoR · LoopFormer · Circle Loop
	DSA · QSA · NSA · SeerAttention-style · MoBA
	DeltaNet · GSA + QK-norm · GLA · KDA · RetNet · ELU-Norm · GDN
	GDN–MHA · GDN–SWA · DeltaNet–MHA · KDA–MLA-NoPE · GDN–QSA* · MoBA–MHA
	SSSL · MQA · GQA · MLA-QK · MLA-NoPE-QK

* Results coming soon. Papers, adaptations and configurations are in the model zoo.

The protocol at a glance

Data	ClimbMix at a pinned revision · one 32,768-token tokenizer · 2,048-token context · one global sample order
Budget	49.152B input tokens · 46,875 updates of 1,048,576 tokens · seed 42
Compute	4.672 GFLOPs per input token (forward, backward, auxiliary and optimizer work); ±1% at admission, ±2% over the run
Evaluation	Validation BPB at 189 points · full CORE-22 at six checkpoints · an independent final BPB
Baseline	Dense: full causal attention in every layer, trained under the same budget

Parameters, memory and speed are outcomes, not controls. The protocol explains every rule and the compute accounting what is counted.

Add your method

New methods and Arena entries follow the same submission process.

1. Open a proposal issue to discuss the method and run plan.
2. Implement the plugin on your fork, calibrate it to the shared budget, and complete a --formal run.
3. Once you have audited and exported the result, open one pull request to upstream main with the implementation and result. Link the proposal issue.

Start with the plugin guide and CONTRIBUTING.md.

Auto-research arena

Note

The arena is in active development. Arena v2, with full support for autonomous agent research (a research gym with MCP tools, metered compute, research traces and self-improving research loops), ships by November 14, 2026.

Any LLM, with or without a research framework, can already enter an architecture today: it is trained under the same protocol, ranked on the same leaderboard, and disclosed with how it was found. See the arena.

Built on NextTransformer

Community projects that extend NextTransformer.

NextTransformer-V by @Enderfga Every NextTransformer method on frozen SigLIP2 image tokens interleaved with text at an 8,192-token context, compute-matched within 1%. Its first study, pre-registered, asks where hybrid backbones read the 2D layout of an image.

Your project here Built something on NextTransformer's code or protocol? Share it in Show and tell or open a pull request that adds a card with a wide banner image. Listed projects are public and say in their README that they build on NextTransformer.

Roadmap

• By November 14, 2026: Arena v2 and agent-native research.
• November 2026: 4K-context pretraining on ClimbMix at 200B and 400B tokens, with Dense and MoE models.
• Open for feedback: data mixtures, model configurations and evaluation beyond CORE in the scaling RFC.
• Upcoming results: GDN + QSA and MoBA + MHA hybrids.
• Planned: checkpoints and the tokenizer on the Hugging Face Hub.

More on the roadmap.

Community

• Discord — chat with the community in #general (permanent invite)
• GitHub Discussions for questions and ideas
• WeChat group

Contributions are welcome; please read CONTRIBUTING.md and the code of conduct. Community projects are listed under Built on NextTransformer.

Citation

@software{nexttransformer2026,
  title   = {NextTransformer: An Open Framework and Fair Benchmark for Next-Generation Transformer Architectures},
  author  = {{NextTransformer contributors}},
  year    = {2026},
  version = {0.1.0},
  url     = {https://github.com/Interactive-Active-AI/NextTransformer}
}

Acknowledgments

NextTransformer builds on nanochat and ideas from modded-nanogpt, uses flash-linear-attention kernels, and trains on ClimbMix. We thank the authors of every method in the zoo. Released under the MIT License; see third-party notices.