Skip to content

feat(reference): add normalized Qwen3-Next primitives for Ascend groundwork - #709

Draft
erfgss wants to merge 1 commit into
RL-Align:mainfrom
erfgss:feat/qwen3-next-reference-primitives
Draft

erfgss wants to merge 1 commit into
RL-Align:mainfrom
erfgss:feat/qwen3-next-reference-primitives

Conversation

@erfgss

@erfgss erfgss commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Add normalized-input PyTorch reference primitives for Qwen3-Next Ascend groundwork: sigmoid gating, causal depthwise convolution with continuation cache, and sequential Gated DeltaNet with grouped heads and FP32 recurrent state.

Use an analytical sigmoid backward to prevent NaN gradients for extreme negative inputs, hoist Q/K normalization and grouped-head expansion out of the token loop, and return compact convolution caches. Include canonical registry dispatch, legacy import aliases, tests, benchmarks, operator documentation and CPU CI coverage. Fix UTF-8 source reading in the Windows layout compatibility test.

Relationship to existing work

Related to #428 and the references/contracts from #659 and #660. The new normalized GDN API takes beta logits, already-computed log decay and K-major FP32 state; the existing vllm_decode_v1 profile takes raw gate parameters and V-major paged state with its own rounding discipline. Tests explicitly convert those boundaries and compare within tolerance.

This is a draft to review the additional normalized interface and overlap with the existing sequence reference. It does not implement or qualify AscendC kernels, complete model assembly, MoE, TP/CP or provider-level bitwise equivalence.

Validation

  • 393 targeted tests passed, covering primitives, upstream linear-attention references/state contracts, registry/dispatch, legacy imports, Attention/LogP and tolerance contracts.
  • Black, isort, Ruff and Flake8 passed for changed Python files.
  • Strict MkDocs build passed.
  • CPU prefill/decode benchmark smoke run completed; extreme sigmoid gradients also tested on the available CUDA device.
  • NPU and full-model training/rollout were not run.

Before readiness

Agree on the normalized reference boundary and reuse strategy, then qualify Ascend kernels and their numerical profile on target hardware.

Replaces #708 after renaming the source branch; the commit and changes are identical.

Signed-off-by: weixin_51634168 <2082464740@qq.com>
@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant