RSL-RL-FlashSAC

RSL-RL Implementation of FlashSAC for IsaacLab. No forks: IsaacLab and rsl_rl are consumed as upstream dependencies; this repository only adds the FlashSAC extension layers.

FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control. Donghu Kim1*, Youngdo Lee23*, Minho Park2, Kinam Kim2, Takuma Seno4, I Made Aswin Nahrendra3, Sehee Min1, Daniel Palenicek56, Florian Vogt7, Danica Kragic7, Jan Peters568, Jaegul Choo2, and Hojoon Lee1

¹Holiday Robotics, ²KAIST, ³KRAFTON, ⁴Turing Inc, ⁵TU Darmstadt, ⁶hessian.AI, ⁷KTH Royal Institute of Technology, ⁸German Research Center for AI (DFKI), ^*Equal Contribution

🎬 Demo

Basic IsaacLab Tasks

Velocity Rough G1	Velocity Rough Anymal-D
Repose Cube Shadow	Open Drawer Franka

Sim-to-Real Locomotion (Unitree G1 Estimator)

Rough-terrain locomotion (Isaac-Velocity-Rough-G1-Estimator-v0): the policy sees proprioception only and recovers base linear velocity from an observation history encoder, trained with FlashSAC over a stair / box / wave terrain curriculum.

Walking	Turning	Push Perturbation	Stair (15cm)

Whole-Body Motion Tracking (Unitree G1 BeyondMimic)

BeyondMimic-style whole-body motion tracking (Isaac-Tracking-Flat-G1-v0, see Motion Tracking): every environment tracks a retargeted reference clip.

cr7_celebration	dance1_subject2

End-Effector Reaching (Franka FR3)

Pose reaching on a Franka Research 3 (Isaac-Reach-Franka-v0): the stock Isaac Lab reach task retargeted to the FR3, driven by relative joint-position deltas and held inside the arm's torque and velocity limits.

Simulation	Real

Benchmark

This implementation reproduces the official FlashSAC IsaacLab benchmark: 12 tasks (velocity locomotion, dexterous in-hand reposing, Franka manipulation) trained for 50M environment steps at 1024 parallel environments, using the single hyperparameter set shared across all tasks from the paper.

Official baseline curves (FlashSAC, PPO, FastTD3) are included under results/isaaclab/.

Repository Structure

rsl_rl_flashsac
  ├── rsl_rl_flashsac        # Algorithm layer on rsl-rl — usable without Isaac Lab
  │   ├── algorithms         # FlashSAC update (categorical double critic, weight norm, AMP, torch.compile)
  │   ├── models             # Actor and distributional critic
  │   ├── modules            # BatchNorm-embedded residual trunk and layers
  │   ├── runners            # OffPolicyRunner
  │   ├── storage            # Uniform / memory-efficient replay buffers (n-step)
  │   └── utils              # Reward normalization, LR schedule, exploration noise
  ├── isaaclab_flashsac      # Isaac Lab integration
  │   ├── rl_cfg.py          # Unified runner cfg + one-line per-task configs (task registry)
  │   ├── wrapper.py         # VecEnv wrapper: pre-reset observations for correct truncation bootstrapping
  │   └── scripts            # flashsac-train / flashsac-play / flashsac-eval
  ├── scripts                # install.sh (one-shot env setup), plot_benchmark.py
  ├── results/isaaclab       # Official benchmark curves (FlashSAC, PPO, FastTD3)
  ├── docs                   # Installation guide, benchmark figure
  └── licenses               # Upstream license copies

Setup

Robot assets are stored with Git LFS. Run git lfs install once per machine before cloning — without it the .usd assets check out as ~130-byte stubs and Isaac Sim fails to load the robot, with no error at clone time. Already cloned? See docs/installation.md to repair in place.

Training environment — one-shot install (uv venv + Isaac Sim 5.1 + Isaac Lab 2.3 + this repo):

bash scripts/install.sh              # see the header for version/path overrides
source .venv-isaaclab/bin/activate

Already have an Isaac Lab environment? Just install this repo into it:

pip install -e .

The rsl_rl_flashsac algorithm package installs without Isaac Lab (uv sync).

See docs/installation.md for details.

Quick Start

flashsac-train --task Isaac-Velocity-Rough-G1-v0 --num_envs 1024 --headless
flashsac-play  --task Isaac-Velocity-Rough-G1-Play-v0 --num_envs 32
flashsac-eval  --task Isaac-Velocity-Rough-G1-v0 --all_checkpoints

flashsac-play --export_policy writes the deterministic policy to <run_dir>/exported/ as TorchScript and ONNX (Estimator variants additionally export history_encoder.pt plus a config.yaml describing the joint order and observation scales, and self-verify that the two compose to the live policy's action).

Per-task configs in isaaclab_flashsac/rl_cfg.py override only the task name; everything else is the shared FlashSAC hyperparameter set. To log to Weights & Biases, set logger = {"class_name": "WandbLogWriter", "project_name": "flashsac"} in the task cfg (entity via WANDB_USERNAME) — flashsac-eval then appends its metrics to the same W&B run.

TODOs

• Deployment example
• More robot variants
• IsaacLab Newton backend support

Citation

If you use FlashSAC, please cite:

@article{kim2026flashsac,
  title={FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control},
  author={Kim, Donghu and Lee, Youngdo and Park, Minho and Kim, Kinam and Seno, Takuma and Nahrendra, I Made Aswin and Min, Sehee and Palenicek, Daniel and Vogt, Florian and Kragic, Danica and Peters, Jan and Choo, Jaegul and Lee, Hojoon},
  journal={arXiv preprint arXiv:2604.04539},
  year={2026},
}

Acknowledgements

This implementation builds on the following codebases:

• rsl_rl
• rsl_rl_sac
• UniLeg

License

BSD-3-Clause. The ported FlashSAC reference implementation is MIT-licensed; upstream license copies live in licenses/dependencies/.