OpenWebRL-4B, SelectionADM RPO (2 epochs)
OpenWebRL-4B-SFT fine-tuned with RPO (step-level DPO plus an SFT term) on preference pairs labelled by the SelectionARM action decision model (ADM), on states visited while actor + ADM executed tasks (search roll-in). This is the top model of our 2026-11-08 evaluation batch.
Contents
| Path | What |
|---|---|
model-0000{1,2}-of-00002.safetensors, configs, tokenizer |
Merged full weights (bf16), ready to serve |
merge-manifest.json |
Base model, adapter sha256, dataset sha256, update count of the merge |
adapter/ |
The LoRA adapter (final update 326) and its config, if you prefer base + adapter |
recipe/training-config.json |
Exact training config of this run |
recipe/metrics.jsonl |
Per-update training and validation metrics |
recipe/train_dpo_r1_ddp.py |
The exact trainer that ran (sha256 f468c20c...) |
recipe/train_dpo_lora_arms.sbatch, recipe/sbatch_merge_lora_arms.sbatch |
Training and merge launchers (Slurm, 2x H200) |
recipe/make_multiepoch.py |
Writes the 2-epoch config from the 1-epoch one |
Data
davidan03/openwebrl-arm-onpolicy-pairs:
5,206 training pairs (+442 validation), identical to this run's training file (sha256 09257e24...) except that image
paths are relative. Built from two collection passes (seeds 42 and 43) of OpenWebRL-4B-SFT + SelectionARM over 1,978
training tasks. At each state the actor proposed candidates, the ADM picked one and that pick was executed; chosen = the
ADM's pick, rejected = a sibling candidate. Only states from episodes judged successful are kept.
Recipe
- Base and reference model: OpenWebRL-4B-SFT (reference = the same model with adapters disabled).
- Objective: DPO with beta 0.1 on summed response log-probabilities, plus the per-token mean NLL of the chosen response (auxiliary SFT weight 1.0).
- LoRA rank 16, alpha 32, dropout 0.05; learning rate 5e-5 with 512 warmup examples, then a half cosine down to 0.5x at the final update; 32 pairs per update (2 GPUs x 16 gradient accumulation); 2 epochs = 326 updates in one fresh run; max 32,768 tokens; seed 42.
- Code:
recipe/train_dpo_r1_ddp.pyis a copy of Zixian Ma'sscripts/train_arm_joint_ddp.pywith the edits listed in its docstring. It importstrain_arm_joint_sft.pyandtrain_arm_c2.pyunchanged; the versions this run used are ondavidan03/OpenWebRLbranchdan_dpo_b05(sha256 02e0dd47... and 63a57d3a...). - Launch:
CFG=recipe/training-config.json sbatch train_dpo_lora_arms.sbatch, thensbatch_merge_lora_arms.sbatch(paths inside are our cluster's; adjust).
Evaluation (2026-11-08)
Zixian Ma's cross-benchmark protocol (openwebrl/docs/EVAL.md on the arm branch):
temperature 1.0, top-p 0.9, 1,024 new tokens per step, 30 steps, 32,768-token context, one screenshot,
1280x1000 viewport. Judges: GPT-5.6 Terra (Online-Mind2Web), GPT-4o (WebVoyager, DeepShop). All systems were evaluated in
the same batch. Overall success counts invalid episodes (aborted, judge errors) as failures; valid-only excludes them.
| System | WebVoyager | Online-Mind2Web | DeepShop | Avg (overall) | Avg (valid-only) |
|---|---|---|---|---|---|
| OpenWebRL-4B-SFT (base) | 41.5 | 30.7 | 46.0 | 39.4 | 42.4 |
| Teacher SFT (235B rollouts) | 43.7 | 33.3 | 41.3 | 39.4 | 42.1 |
| This model | 46.1 | 39.3 | 50.0 | 45.1 | 49.9 |
Pooled over all 1,045 tasks: 44.69% overall success, +5.6 points over the base model (95% CI [+2.7, +8.6]) and +4.3 over teacher SFT ([+1.2, +7.4]), paired task bootstrap.
Caveats: one evaluation run per benchmark; same-checkpoint runs vary by about 1-2 points pooled. On 2026-11-08 WebVoyager lost a share of tasks to a Wolfram Alpha outage and GitHub rate limiting (equal for every system). This model hits the 1,024-token step limit more often than the base model (about 12% of episodes vs 1%), mostly while deliberating before an action.
- Downloads last month
- 44
Model tree for davidan03/openwebrl-4b-selectionadm-rpo-2ep
Base model
OpenWebRL/OpenWebRL-4B-SFT