OpenWebRL-4B, SelectionADM RPO (2 epochs)

OpenWebRL-4B-SFT fine-tuned with RPO (step-level DPO plus an SFT term) on preference pairs labelled by the SelectionARM action decision model (ADM), on states visited while actor + ADM executed tasks (search roll-in). This is the top model of our 2026-11-08 evaluation batch.

Contents

Path What
model-0000{1,2}-of-00002.safetensors, configs, tokenizer Merged full weights (bf16), ready to serve
merge-manifest.json Base model, adapter sha256, dataset sha256, update count of the merge
adapter/ The LoRA adapter (final update 326) and its config, if you prefer base + adapter
recipe/training-config.json Exact training config of this run
recipe/metrics.jsonl Per-update training and validation metrics
recipe/train_dpo_r1_ddp.py The exact trainer that ran (sha256 f468c20c...)
recipe/train_dpo_lora_arms.sbatch, recipe/sbatch_merge_lora_arms.sbatch Training and merge launchers (Slurm, 2x H200)
recipe/make_multiepoch.py Writes the 2-epoch config from the 1-epoch one

Data

davidan03/openwebrl-arm-onpolicy-pairs: 5,206 training pairs (+442 validation), identical to this run's training file (sha256 09257e24...) except that image paths are relative. Built from two collection passes (seeds 42 and 43) of OpenWebRL-4B-SFT + SelectionARM over 1,978 training tasks. At each state the actor proposed candidates, the ADM picked one and that pick was executed; chosen = the ADM's pick, rejected = a sibling candidate. Only states from episodes judged successful are kept.

Recipe

  • Base and reference model: OpenWebRL-4B-SFT (reference = the same model with adapters disabled).
  • Objective: DPO with beta 0.1 on summed response log-probabilities, plus the per-token mean NLL of the chosen response (auxiliary SFT weight 1.0).
  • LoRA rank 16, alpha 32, dropout 0.05; learning rate 5e-5 with 512 warmup examples, then a half cosine down to 0.5x at the final update; 32 pairs per update (2 GPUs x 16 gradient accumulation); 2 epochs = 326 updates in one fresh run; max 32,768 tokens; seed 42.
  • Code: recipe/train_dpo_r1_ddp.py is a copy of Zixian Ma's scripts/train_arm_joint_ddp.py with the edits listed in its docstring. It imports train_arm_joint_sft.py and train_arm_c2.py unchanged; the versions this run used are on davidan03/OpenWebRL branch dan_dpo_b05 (sha256 02e0dd47... and 63a57d3a...).
  • Launch: CFG=recipe/training-config.json sbatch train_dpo_lora_arms.sbatch, then sbatch_merge_lora_arms.sbatch (paths inside are our cluster's; adjust).

Evaluation (2026-11-08)

Zixian Ma's cross-benchmark protocol (openwebrl/docs/EVAL.md on the arm branch): temperature 1.0, top-p 0.9, 1,024 new tokens per step, 30 steps, 32,768-token context, one screenshot, 1280x1000 viewport. Judges: GPT-5.6 Terra (Online-Mind2Web), GPT-4o (WebVoyager, DeepShop). All systems were evaluated in the same batch. Overall success counts invalid episodes (aborted, judge errors) as failures; valid-only excludes them.

System WebVoyager Online-Mind2Web DeepShop Avg (overall) Avg (valid-only)
OpenWebRL-4B-SFT (base) 41.5 30.7 46.0 39.4 42.4
Teacher SFT (235B rollouts) 43.7 33.3 41.3 39.4 42.1
This model 46.1 39.3 50.0 45.1 49.9

Pooled over all 1,045 tasks: 44.69% overall success, +5.6 points over the base model (95% CI [+2.7, +8.6]) and +4.3 over teacher SFT ([+1.2, +7.4]), paired task bootstrap.

Caveats: one evaluation run per benchmark; same-checkpoint runs vary by about 1-2 points pooled. On 2026-11-08 WebVoyager lost a share of tasks to a Wolfram Alpha outage and GitHub rate limiting (equal for every system). This model hits the 1,024-token step limit more often than the base model (about 12% of episodes vs 1%), mostly while deliberating before an action.

Downloads last month
44
Safetensors
Model size
4B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for davidan03/openwebrl-4b-selectionadm-rpo-2ep

Finetuned
(4)
this model

Dataset used to train davidan03/openwebrl-4b-selectionadm-rpo-2ep