Skip to content

About

Documentation for ghcr.io/infer-lab/sglang-runtime: SGLang runtime images for NVIDIA Blackwell (SM120) and Hopper (SM90), built from source and gated before publishing.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 

Repository files navigation

sglang-runtime

SGLang built from source for NVIDIA Blackwell (SM120) and Hopper (SM90), with FlashInfer and DeepGEMM from source and SM120 fixes not yet upstream.

Image

Tag Built Digest
ghcr.io/infer-lab/sglang-runtime:20261108-7ef2921ac4f8 2026-11-08 sha256:8a7b0895b0563260dd5230eabe08f83dc25177813a6a3802530da83d0cbccfa6
ghcr.io/infer-lab/sglang-runtime:stable 2026-11-08 sha256:8a7b0895b0563260dd5230eabe08f83dc25177813a6a3802530da83d0cbccfa6

Usage

The entrypoint activates the image's environment and runs the given command.

docker run --gpus all --ipc=host -p 30000:30000 -v /path/to/models:/models \
  ghcr.io/infer-lab/sglang-runtime:stable \
  python -m sglang.launch_server --model-path /models/<model> --tp 2 \
  --host 0.0.0.0 --port 30000

Contents

sglang

7ce26f621cb5: sgl-project/sglang at cbbc6b67f4

plus:

flashinfer

757780dcb9d6: flashinfer-ai/flashinfer at 5d2c80277

plus:

  • flashinfer#8740: feat(sm120_dsa_indexer): DeepSeek-V4.1 decode in the paged form: strided page caches and candidate blocks

DeepGEMM-sglang

9a464db6cdcb: sgl-project/DeepGEMM at 0f96f72

plus:

Serving

Each model was gated with the command below, run on the image's entrypoint with the model directory mounted; the parallelism and every setting shown are the ones that were measured.

python3 -m sglang.launch_server --model-path <model> --tensor-parallel-size 1 --context-length 16384 --mem-fraction-static 0.85 --trust-remote-code --enable-cache-report
python3 -m sglang.launch_server --model-path <model> --ple-offload-embedding --reasoning-parser qwen3 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-moe-runner-backend deep_gemm --tensor-parallel-size 2 --context-length 262144 --mem-fraction-static 0.85 --moe-runner-backend flashinfer_cutlass

Validation

Gated on RTX PRO 6000 Blackwell (SM120) GPUs before publishing; scores are chat GSM8K (lm-eval, 1,319 questions, flexible-extract).

Model Configuration Eval Result
Qwen/Qwen3-30B-A3B-Instruct-2507 TP1 smoke passed
nvidia/Qwen3.8-Flash-Next-NVFP4 TP2 smoke passed
nvidia/Qwen3.8-Flash-Next-NVFP4 TP2 gsm8k-chat 0.9757 (gate 0.96)

Built and gated with InferLab 0.23.3.

About

Documentation for ghcr.io/infer-lab/sglang-runtime: SGLang runtime images for NVIDIA Blackwell (SM120) and Hopper (SM90), built from source and gated before publishing.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors