2026.RA.P4-Pilot-DPO-Misc-Runs

Archive of the five partial, aborted or dry-run directories from the "rational agents" programme's p4_pilot_dpo/ training root, one subfolder per run, kept for provenance only. None of these is a finished model. Four hold only configs and logs; one holds the checkpoints of a preempted training job. The finished runs they precede live in their own repositories: 2026.RA.P4-Pilot-DPO-Qwen3-8B-v1b, 2026.RA.P4-Pilot-DPO-Qwen3-8B-v1b-ActionOnly, 2026.RA.P4-Pilot-SFT-Qwen3-8B-v1b, 2026.RA.SelfHarm-Polarity-DPO-Qwen3-8B and 2026.RA.SelfHarm-Polarity-DPO-Control-Qwen3-8B.

All runs use the same trainer, experiments/rational_agents/p4_train_dpo.py in the ii_mats repository: LoRA (r=32, alpha=64, dropout 0.05, all linear projections, 87.3M trainable parameters) DPO on Qwen/Qwen3-8B in bf16, micro-batch 1, max length 4096, seed 0. Each subfolder's train_config.json is the resolved configuration with code provenance, data_summary.json the per-split pair statistics after filtering, and train.log the log of the last invocation written there. Every subfolder's relative layout is identical to the local run directory it was archived from.

subfolder	status	what it was	slurm job	W&B
p4_pilot_dpo_qwen3_8b/	cancelled after 4 steps (timing probe); no weights	first P4 pilot launch on p4_dataset_v0, effective batch 8	16357916 (jagupard34, 2026-07-27)	17mk4c39
p4_pilot_dpo_qwen3_8b_ruled/	cancelled about two minutes after start; no weights	the "ruled pilot mix" on p4_dataset_v0, effective batch 4	16358022 (jagupard34)	s084es6n
p4_pilot_dpo_qwen3_8b_v1/	crashed, CUDA OOM in the first eval pass; no weights	first run on the corrected p4_dataset_v1	16358150 (jagupard34)	r9zag7ys
selfharm_dryrun/	dry run (--dry-run, no weights loaded)	pre-flight of the below-walkaway polarity arm	none (login node)	none (disabled)
selfharm_polarity_trained_v1_preempted_16600808/	preempted at step 61 of 76; checkpoints 12 to 60 with optimizer states	first launch of the polarity arm's trained condition	16600808 (iliad6, sc-loprio, 2026-08-09)	none (offline mode)

The three aborted P4 pilot predecessors

Context: research notes 0003 (the divergence-DPO dataset and its training mix) and 0004 (the pilot verdict). Each DPO pair is the seat's stored rendered view at a turn where the model diverged from an executable best-response oracle, chosen = the oracle's action rendered in the model's wire format with a rationale synthesized from oracle numbers, rejected = the model's own emission, weighted by oracle regret (--regret-weighting dataset).

• p4_pilot_dpo_qwen3_8b: the very first launch, before the training-mix ruling. Filters scaffold=canonical only, grad-accum 8, 941 train pairs / 145 episodes, about 117 planned steps. Job 16357916 was cancelled after 4 optimizer steps; its purpose became the wall-clock measurement recorded in p4_train.sbatch (50 to 100 s per step at effective batch 8). The train_config.json here is the dry-run stamp (dry_run: true, W&B disabled), the training invocation was p4_train_dpo.py --run-name p4_pilot_dpo_qwen3_8b --regret-weighting dataset.
• p4_pilot_dpo_qwen3_8b_ruled: the ruled pilot mix (own-model pairs, in-round turns, full information, canonical prompt), grad-accum 4, 329 train pairs / 62 episodes, about 82 planned steps, on p4_dataset_v0. Job 16358022 was terminated about two minutes after the model loaded, superseded once the v0 oracle-pricing bug was found (v0 pairs at turns where the acting seat proposed carry wrong targets; 24.6% of them get a different chosen action, so v0 must not be trained on). Invocation: p4_train_dpo.py --run-name p4_pilot_dpo_qwen3_8b_ruled --regret-weighting dataset --filter scaffold=canonical.
• p4_pilot_dpo_qwen3_8b_v1: the same mix on the corrected p4_dataset_v1 (326 train pairs / 62 episodes; 62 eval pairs). Job 16358150 trained to about step 25 and died with a CUDA out-of-memory error during the evaluation pass: TRL's entropy metric reshapes a non-contiguous float32 logits slice and at eval batch 2 that asked for 8.15 GiB on top of 32.6 GB allocated on a 48 GB A6000. The fix (eval batch 1, expandable_segments, checkpoints every 10 steps) is documented in p4_train.sbatch and produced p4_pilot_dpo_qwen3_8b_v1b, the run of record. Invocation: p4_train_dpo.py --run-name p4_pilot_dpo_qwen3_8b_v1 --regret-weighting dataset --filter scaffold=canonical.

p4_dataset_v0 was never published; the public 2026.RA.Divergence-DPO-Pairs is p4_dataset_v2 (bit-identical to v1 in every trained field) and ships the v0/v1 diff audits.

The two polarity-arm directories

Context: research notes 0040 (the arm's construction) and 0042 (its results). In this programme "self-harm" is a bargaining-game term: a negotiating seat signing or tabling a package that pays it less than its own private walk-away threshold, an error checkable from the seat's own prompt. The polarity arm is low-learning-rate LoRA DPO on 75 accept P versus reject P pairs drawn from Qwen3-8B's own rollouts (41 held out by game instance), meant to reduce that rate; its control is the same run with the label shuffled. Pair sets and all evaluation data: 2026.RA.SelfHarm-Polarity-Arm.

• selfharm_dryrun: the --dry-run pre-flight of the trained condition (no weights loaded): 75 train pairs / 49 episodes, 41 eval pairs / 21 episodes, none over 4,096 tokens, 72 planned optimizer steps at effective batch 4. Invocation: p4_train_dpo.py --run-name selfharm_dryrun --dry-run --wandb-mode disabled --dataset-dir /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_dpo_v1/datasets/selfharm_trained --pair-models Qwen3-8B --include-final-votes --include-private --regret-weighting none --min-eval-pairs 0 --epochs 4 --learning-rate 5e-6 --grad-accum 4.
• selfharm_polarity_trained_v1_preempted_16600808: the first launch of the trained condition, sbatch --export=ALL,ARM=trained sbatch_selfharm/train_polarity_dpo.sbatch (lr 5e-6 cosine, 4 epochs, grad-accum 4, eval and save every 12 steps, --regret-weighting none, --pair-contrast as_built). Job 16600808 on sc-loprio was preempted at step 61 of 76, 32 minutes in, with --no-requeue. It was restarted clean rather than resumed (job 16601191, which became 2026.RA.SelfHarm-Polarity-DPO-Qwen3-8B), because an arm that finished only by resuming would not be procedurally matched to a control that ran uninterrupted (note 0042, limitation 8). Kept as the record of that decision. Contents: checkpoints/checkpoint-{12,24,36,48,60}/ (LoRA + optimizer.pt + scheduler + RNG + trainer_state.json), checkpoints/README.md, train_config.json, data_summary.json; held-out DPO reward accuracy at step 60 was 0.829 with margin +0.39, on the same trajectory as the clean rerun. 78 files, 5.3 GB.

How to load a preempted checkpoint

from transformers import AutoModelForCausalLM
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "siddharthmb/2026.RA.P4-Pilot-DPO-Misc-Runs", subfolder="selfharm_polarity_trained_v1_preempted_16600808/checkpoints/checkpoint-60")

Whole tree: hf download siddharthmb/2026.RA.P4-Pilot-DPO-Misc-Runs --local-dir p4_pilot_dpo_misc.

Cluster paths

• Slurm logs for the P4 runs: /nlp/scr/siddharth/ii_mats/rational_agents/_slurm_logs/{16357916,16358022,16358150}.{out,err}; for the preempted polarity job: experiments/rational_agents/sbatch_selfharm/logs/selfharm_dpo_[rational_agents:rl-arm]selfharm_dpo_trained.16600808.out in the ii_mats repo.
• Code: experiments/rational_agents/p4_train_dpo.py, p4_train.sbatch, sbatch_selfharm/train_polarity_dpo.sbatch, selfharm_dpo_prepare.py.
• Research notes: experiments/rational_agents/research-notes/0003-*.md, 0004-p4-pilot-verdict.md, 0040-selfharm-polarity-arm.md, 0042-selfharm-polarity-arm-results.md.

Local provenance: Archived from /nlp/scr/siddharth/ii_mats/rational_agents/p4_pilot_dpo on 2026-10-11 by the MATS archive job 17819018 of the transcoder-adapters cleanup (report claude_updates/memory/reports/mats_archive.md in that repository). The run directory was uploaded unchanged (upload_large_folder), so every checkpoint, config and log keeps its original relative path. Every file name and every text file was scanned for credentials before upload (Hub, Anthropic, OpenAI, AWS and W&B key shapes, private-key headers, credential URLs, paths of the secrets store); the report lists what was excluded. The local copy was deleted after every uploaded file was verified on the Hub by name, size and hash, and a <name>.txt breadcrumb naming this repository was left beside it. Collection: https://huggingface.co/collections/siddharthmb/2026-mats-rational-agents-6acc1dd5623e8e00176b80a8; workspace index: https://huggingface.co/datasets/siddharthmb/2026.RA.Workspace-Index.