diff --git a/EXTENSIONS.md b/EXTENSIONS.md new file mode 100644 index 0000000..6b236c8 --- /dev/null +++ b/EXTENSIONS.md @@ -0,0 +1,112 @@ +# Project extensions (16-831) + +Additions to FlashSAC for the course project. With the default config the training behaviour is unchanged. + +## Config keys + +All keys live in `configs/agent/flashSAC.yaml` and are set with `--overrides agent.=`. + +| Key | Values (default first) | Meaning | +|---|---|---| +| `sampler` | `uniform`, `swd`, `laber` | Replay sampling method | +| `swd_decay_steps` | `2000` | SWD decay horizon, in interaction steps | +| `swd_min_weight` | `0.1` | SWD weight floor | +| `laber_multiplier` | `4` | LaBER candidate batch = multiplier × `sample_batch_size` | +| `actor_delay` | `fixed`, `ema` | Actor delay rule | +| `actor_update_period` | `2` | Delay `d` for `actor_delay=fixed` | +| `actor_delay_ema_interval` | `100` | `k`: critic updates between delay recomputations | +| `actor_delay_ema_beta` | `0.9` | EMA decay, applied once per interval | +| `actor_delay_ema_scale` | `2.0` | Delay when the TD error equals its EMA | +| `actor_delay_ema_min` / `_max` | `1.0` / `8.0` | Clip range of the delay | +| `actor_lr_schedule` | `actor_step`, `global_step` | What drives the actor/temperature learning-rate schedule | + +`log_csv` (default `true`, in `configs/flashSAC_base.yaml`) turns the CSV log on or off. + +## Methods + +**SWD** (`flash_rl/buffers/torch_buffer.py`). WarpSAC's Sample Weight Decay: a transition added `age` interaction +steps ago is sampled with probability ∝ `max(swd_min_weight, 1 - age / swd_decay_steps)`. No importance correction, +as in WarpSAC: both the loss and the BatchNorm statistics follow the recency-biased batch. +This is FlashSAC + SWD, not full WarpSAC (which also changes the critic). + +**LaBER** (`FlashSACAgent._select_laber_batch`). Draw `laber_multiplier × B` transitions uniformly, score each with + + surprise_i = mean over the 2 critics of ‖softmax(logits(s_i, a_i)) − target_probs_i‖₂ + +and keep `B` of them with probability ∝ surprise (with replacement). The importance weights +`w_i = mean(surprise) / surprise_i` correct the critic update for this selection in two places: the loss terms, and the +BatchNorm batch statistics of the critic and target critic (weighted mean and variance). Both corrections are +self-normalised estimates of their uniform-sampling counterparts, so the update is approximately, not exactly, +unbiased. Scoring runs with BatchNorm in eval mode. The actor is updated on a uniform slice of the candidates, not on +the high-surprise selection. + +**EMA actor delay** (`flash_rl/agents/flashSAC/actor_delay.py`). Every `interval` critic updates, with `e` the mean +absolute TD error over those updates: + + ema = beta * ema + (1 - beta) * e + delay = clip(scale * e / ema, min, max) + +The delay is real-valued: each critic update earns `1/delay` credit and an actor update costs 1. + +**Actor LR schedule.** In FlashSAC the actor's cosine decay advances once per actor update, so its learning-rate +schedule depends on the delay. `actor_lr_schedule=global_step` ties it to the number of critic updates instead. Use it +for all runs of a delay comparison (it also changes the `d=2` baseline). + +## Running + + scripts/run_humanoid.sh [agent overrides...] + +runs the four IsaacLab humanoid tasks for each seed; see the examples at the top of the script. + +## Logs + +Each run writes `logs////seed_