Super-squash branch 'main' using huggingface_hub

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/training_progress.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,210 @@
+---
+license: apache-2.0
+language:
+- en
+datasets:
+- openbmb/Ultra-FineWeb-L3
+- wikimedia/wikipedia
+- HuggingFaceTB/finemath
+- HuggingFaceTB/smollm-corpus
+- OpenCoder-LLM/opc-annealing-corpus
+tags:
+- diffusion-language-model
+- masked-diffusion
+- parallel-decoding
+- diloco
+- text-generation
+pipeline_tag: text-generation
+---
+
+<p align="center"><img src="assets/avatar.svg" width="200" alt="DiffuRefill-1B"></p>
+
+# DiffuRefill-1B
+
+**Status: pretraining, about half way.** Step ~100k of 200k, ~46B effective
+training tokens, merge round 290. The weights here are a live pretraining
+checkpoint: fluent English, facts still unreliable, not instruction-tuned. Use
+`global_ema.pt`; it is the best checkpoint at every round.
+
+A 1.08B masked-diffusion language model decoded with **confidence-targeted
+re-infilling**: draft the whole sequence in a handful of parallel denoising
+passes, then spend a few more passes rewriting only the tokens the model is
+least sure about. An autoregressive model needs one pass per token; this one
+needs ~20 passes for a whole sequence, whatever its length.
+
+| | |
+|---|---|
+| parameters | 1.08B (embedding tied to the output head) |
+| architecture | 18 blocks, dim 2048, 16 heads, SwiGLU 5632, RMSNorm, RoPE, **bidirectional** attention |
+| context | 2048 tokens |
+| tokenizer | `openbmb/MiniCPM4-0.5B` (73,440 tokens; vocab padded to 73,760 for MASK and PAD) |
+| objective | absorbing masked diffusion, loss on masked positions only |
+| training | DiLoCo across 3-5 single-GPU boxes, synchronised through this repository |
+| progress | step ~100k / 200k, ~46B effective tokens |
+
+## Training progress
+
+![training progress](assets/training_progress.png)
+
+Tracked every round on fixed, deterministic batches, so the change between two
+rounds carries no sampling noise:
+
+| measure | what it is | latest |
+|---|---|---|
+| held-out CE (EMA) | masked CE at t = 0.3 / 0.6 / 0.9 on a fixed Ultra-FineWeb slice no worker trains on | **2.939** (round 287) |
+| held-out CE (global) | the same for the latest merged global | 2.957 |
+| fact probe, mc | 1,050 PopQA facts written as plain sentences with the answer masked; the gold answer must beat three same-length distractors (chance 0.25) | **0.560** (was 0.540 before the data mix) |
+| fact probe, answer NLL | per-token NLL of the masked gold answer | **6.08** (was 6.55) |
+| fact probe, exact match (EMA) | the answer decoded token by token and compared with PopQA's accepted answers | 0.083 (was 0.075) |
+| decoding accuracy | confidence decoding of 50% masked held-out text in 8 passes: token / adjacent-pair accuracy | 0.653 / 0.453 (round 275) |
+
+The guard-probe curve at the top is measured on an Ultra-FineWeb slice. When
+the fact-dense sources were added at step ~96.8k, Ultra-FineWeb fell from 100%
+to 56% of the documents, and that probe rose by design while the fact probe
+improved quickly. From here the fact probe and the held-out CE are the numbers
+to watch. No public benchmark scores are reported yet: at this stage they would
+sit near chance and say little.
+
+## How it is trained
+
+**Fleet.** Ephemeral single-GPU boxes (RTX PRO 6000 Blackwell, 96 GB) with no
+interconnect, joined DiLoCo-style through this repository. Each worker trains 150
+local AdamW steps, pushes `ckpt_wNN.pt`, and adopts the merged `global_b.pt`.
+
+- **Merging.** The merger averages the round's checkpoints (a plain mean, folded
+  into a running fp32 sum as they arrive) and waits at most 150 s past the first
+  arrival.
+- **Phase.** Workers pick up a new global as soon as it appears and take its step
+  counter, so the fleet starts each interval together.
+- **EMA.** An exponential average of the globals (β 0.9) is published as
+  `global_ema.pt`.
+
+**Objective.**
+- `t ~ U(0.15, 1)` per row, with positions masked independently.
+- Half of the rows keep a random-length clean prefix, which is the shape of
+  prompted generation.
+- Attention stays inside each packed document.
+- On a quarter of the rows, names and numbers are masked as whole spans
+  (salient span masking). Masked token by token, a two-token name leaks through
+  its visible half; masked whole, it has to be recalled.
+
+**Optimiser.** AdamW (0.9, 0.95), weight decay 0.1, clip 1.0, peak LR 1.5e-4,
+cosine to 10% over 200k steps. Each worker step covers 64 × 2048 = 131,072
+tokens. The blocks are `torch.compile`d.
+
+**Data** (by document, from step ~96.8k):
+
+| source | share |
+|---|---|
+| [Ultra-FineWeb-L3](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3) Multi-Style + QA synthetic | 56% |
+| Wikipedia (en) | 15% |
+| FineMath 4+ | 8% |
+| Cosmopedia v2 | 8% |
+| OpenCoder annealing: algorithmic corpus + code snippets | 8% |
+| Wikipedia leads restated as textbook / lecture / museum-guide prose (own corpus, filtered for copying and for numbers not in the source) | 5% |
+
+Before step ~96.8k the model saw Ultra-FineWeb only.
+
+### What the run taught
+
+Measured on this model or on its 37.9M-parameter stand, with fixed probes:
+
+| change | result |
+|---|---|
+| peak LR 3e-4 → 1.5e-4 | Local drift over one interval fell from 0.076 to 0.013 CE. The merged model improved from 4.336 to 4.288. |
+| keeping local drift across a merge (`θ += global − pushed`) | Worse in 60 of 63 adoptions, +0.079 CE on average. The run uses plain replacement. |
+| EMA of the globals | Beats the latest global every round. Seeding training from it gave the gain back. |
+| outer Nesterov (μ 0.5) | Raised held-out CE for three rounds. Reverted. |
+| Muon switched on mid-run | Loss rose from 3.16 to 3.32. It was a 2× win from scratch on the stand, but not as a switch on a running model. |
+| span masking, PMI units, stratified t, complementary masks | Each lost to plain independent masking on the stand. Span masking was removed at step ~91.9k. |
+| fact-dense data mix + salient span masking | Fact-probe answer NLL 6.55 → 6.08 within about four hours. |
+| `torch.compile` per block | 17% faster steps, less memory. |
+| FP8 matmuls (torchao, rowwise or tensorwise) | No speed-up on this card. Not used. |
+
+## The method
+
+```
+1. K denoising steps           -> a full draft + a confidence value per position
+2. repeat R times:
+     take the p% least confident generated positions
+     re-mask them
+     one forward pass          -> new tokens, chosen with both sides in view
+     refresh their confidence
+```
+
+Cost is `K + R` forward passes. One refill pass fixes any number of positions at
+once, because the model processes the whole sequence in parallel. That is what
+makes targeted repair cheaper than more denoising.
+
+### Decoding configuration
+
+The draft and the refill deliberately use different sampling rules. The draft
+decides what the text is about and needs freedom. The refill touches a fifth of
+the tokens and needs discipline. Sharing one setting between them trades every
+metric straight against another.
+
+| stage | settings |
+|---|---|
+| draft | `K=12`, temperature `2.2 -> 0.9` annealed, Gumbel noise `2.5` on the unmask order, no truncation |
+| refill | `R=8`, `p=20%`, temperature `0.8`, `min_p=0.10` |
+| both | neighbour ban: subtract `6.0` from the logit of the token already standing to the left or right; never commit two positions closer than 2 in one pass |
+
+- **Gumbel unmask order.** Strict top-k confidence reveals the easy positions
+  first, such as articles and punctuation. Content words are left for last, when
+  the context is already frozen. Annealed Gumbel noise on the log confidences
+  breaks that order. It was the largest single effect in the pilot.
+- **Neighbour ban and commit gap.** Two adjacent positions sampled
+  independently both pick the locally likely token. That is how parallel
+  decoding produces repeats, and these two rules target exactly that.
+- **Split-stage sampling.** A hot, free draft followed by a cool, truncated
+  refill.
+
+## Pilot results (TinyStories, 37.9M parameters)
+
+Two models of identical architecture were trained from scratch on the same
+corpus: one autoregressive, one masked-diffusion. An **independent judge**
+(`roneneldan/TinyStories-33M`, related to neither) scored the NLL of each
+model's generated continuation.
+
+| | calls | NLL ↓ | doubled tokens ↓ | distinct-2 ↑ |
+|---|---|---|---|---|
+| real text | — | 1.15 | 1.38% | 0.478 |
+| autoregressive | 128 | 3.08 | 1.64% | 0.466 |
+| diffusion + refill, tuned | **20** | **2.87** | **0.30%** | 0.459 |
+
+The diffusion model matched autoregression in quality at a sixth of the passes.
+It produced five times fewer doubled tokens than the autoregressive baseline,
+below the rate in the real corpus. Diversity came out at parity, slightly under
+autoregression.
+
+**What did not work in the pilot:**
+- **Autoregressive repair of the draft.** It was worse than not repairing at all.
+  An AR model scoring position `i` cannot see `i+1`, so it duplicates the
+  neighbour it is not looking at.
+- **Self-repair training on the model's own drafts.** It was harmful at 60–100%
+  corruption and neutral at 10–35%.
+
+## Honest limits
+
+- **Unfinished.** Half of the planned steps remain, and facts are unreliable.
+  Expect confident, fluent mistakes.
+- **The pilot evidence is small.** 37.9M parameters on TinyStories does not show
+  that the decoding results transfer to 1B on natural text.
+- **The judge is an AR model.** A low NLL under it rewards predictable text as
+  well as good text.
+- **Diffusion costs more to train.** It learns only from masked positions, so it
+  reads more tokens than an AR model for the same quality. The saving is entirely
+  at inference.
+
+## Files
+
+| file | contents |
+|---|---|
+| `global_ema.pt` | EMA of the merged globals; the checkpoint to use |
+| `global_b.pt` | the latest merged global |
+| `ckpt_wNN.pt`, `log_wNN.txt` | per-worker checkpoints and logs, overwritten every interval |
+| `config.json` | the exact architecture, objective, data mix and training setup |
+| `pilots/` | held-out and fact-probe logs, stand pilot results |
+
+Checkpoints are `torch.save` dicts with a `model` state dict for the `Net` class
+described in `config.json`.

anneal/global_r760.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba83d63aa733cb18980b74df091af2236ca7cd1acda64ed97fe3d1b0af2a413d
+size 2454120580

anneal/global_r780.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0d7b407dfe6ead33c994a750237a6620d18768fd096c0425f34d4a4e02c7767
+size 2454120580

anneal/global_r800.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8c9744f854c867a570f9882a395bc8efd2aacb6c738926cb6a15867cbd06c0a
+size 2454120580

anneal/global_r820.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0c223a10325922c3b4279d378f9e2d75779ae35df9f1f48df8f8bf02590ffca
+size 2454120580

anneal/global_r840.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c30bf93abe59990fe462202f81315f0c1b0d7e54fec98034c1ca723590ce1fd
+size 2454120580

anneal/global_r860.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9708b087eed172de8e62d0292cb6a13ebfdb7257646609647a3fffc847b8cc68
+size 2454120580

anneal/global_r880.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80f072ce83c9a404cf77f594d175c7d46985c5a4712d7d0baf62c309a2959e83
+size 2454120580

ckpt_w0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc48c834e124a9bf4cc3bec73cb40a3c8cd85b52f09d128a4c563eac0579b468
+size 2152005695

ckpt_w1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c809ec752ae23731aa0609bbab96b6a33855607ce7e294d7faed40c38f2d02e5
+size 2152005695

ckpt_w10.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e59d70894c26651aba1ffe32ee12ec0aac14432c2d1c215a1c56475f20754c87
+size 2152006469

ckpt_w100.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d15f2c6c7893a4e6335dfc59f82baefb2a3995e09f255ac339c908c512119b4
+size 2152006603

ckpt_w101.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ccdb58193a6c600c2c93a28119b7909ebcc276aa6ee319537a81247e7af5174
+size 2152006603

ckpt_w102.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b7550fa3fc11cdfda6af123480e4d17210789d495e0143d648a56816e07b442
+size 2152006603

ckpt_w103.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a66cdfc8798387d530cb3d67a5e2460333e65b3f72afd999e7bb42bf1af85bb1
+size 2152006603

ckpt_w106.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bab673c873d29d747d950cba9c1d0c2edf4020086fb19e3005e733a88147b2e
+size 2152006603

ckpt_w107.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9c3a31047305c6c986bcf34b26dd05b9ab7f2a70add2a8fd3a64396d5cc6ac3
+size 2152006603

ckpt_w109.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b7062b2d1ba0ce7d0364f77c4c411d217dd243784ac52d05ce81e533ac48d4b
+size 2152006603

ckpt_w11.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9252ba8b8afd5685947825afb17da1fbcaad49b32b8e3403a663a8b2cf81b914
+size 2152006469

ckpt_w110.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7e9f3ee84d6bc1442d9edca928d00d01f485b4af72a25fa0e68a39c7d13696d
+size 2152006603

ckpt_w111.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f26570a83fe5cdf7a2bf40894e87a0807ffb6342974565210d2d666384ad9586
+size 2152006603

ckpt_w112.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61582fac4259bdaf5e9ebd6974a32640d056f67efa96c32cd0cd8dcfc71e6993
+size 2152006603

ckpt_w113.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1b12731d764da327e192f1f69d585a2d1e4b69b02bb0d59bb6a41cbaa5654e5
+size 2152006603

ckpt_w115.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:765c68e8bab49178a57d19075dd891493d8c3a7c27d4594c3cd8a3ec8f63a9d8
+size 2152006603

ckpt_w116.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9beacee12952287e8a7b0227945e490cb3e847ca6f686c2d070c39674bc782ac
+size 2152006603

ckpt_w117.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6e2929f9193258d937068138920023d8c8a208053e2c0b14825e0c65ae7adf0
+size 2152006603

ckpt_w118.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82f4bfc82f423d8cad5b74c1b72efd8cdb16b10f6f3f791c0e8700228deefd95
+size 2152006603

ckpt_w119.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5fecfd8bdc12f9f814eeac07bc4cf8eac9f04fae3f6b3885e0c850aba7976163
+size 2152006603

ckpt_w120.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bbcc39aecbbb6672b7e8103786d61e3cdcaec71db37e1074d44b6f32ec70898d
+size 2152006603

ckpt_w121.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e87ab48d00dedc65429fc69295da855558c666035df361930c1cdaa68f4f9be
+size 2152006603

ckpt_w122.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d44f7a9478248eb8fff728706b3715935473808a97de2bee23486076ff1b56b0
+size 2152006603

ckpt_w123.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eeb0f2d93b6016f9ebbce149b44f394a16c334c2611d915a8186d04a4841b3bc
+size 2152006603

ckpt_w124.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de7d8fdd1813ddb3b2cb1b13d1db17a575b9233cd05ae5741461bb010d194b56
+size 2152006603

ckpt_w125.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a58b3f3afa0a833aaf7ba054d207d7ddcbb2ef1b9409776b9590d6082f70d7d
+size 2152006603

ckpt_w127.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd8ef09d4ef09f6237d5150d0d67dc959cb6c8de64aa3a71d552d186a5a03ef0
+size 2152006603

ckpt_w128.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5cec99950bdba3ea7baaf79df9e4ba252542842952b6462411c24556333101a
+size 2152006603

ckpt_w129.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:444508900c9ab5c7eb77fe055db525b63f8c40e4813b4622facf7ec9124b8031
+size 2152006603

ckpt_w13.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb2c39cb7f4652f5c3b6d3ad5dfa639a297b98f742196bb9b94044eb3045118e
+size 2152006469

ckpt_w130.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a34bf6ef0e9f64dbc44142fe396ed0ff33fbe7c28e55ace962d7215de9fa281
+size 2152006603

ckpt_w131.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25148fc5d2d42fce0a0a7af51787b6bbff017b0c7cf57b1a7e06f74cc126ede8
+size 2152006603

ckpt_w132.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d31c9a0acf571e41920ad3a1a3869713c999c212ca46e06b70832e2dddba96f4
+size 2152006603

ckpt_w133.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:adca03cb6b01f40b38da58062061b3ac07299f95f644ac4c244533c34f3b09e1
+size 2152006603

ckpt_w134.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e97da28cf795c3781b9c2d3d402a8775327ca7115d1df629324b3a795de2cef
+size 2152006603

ckpt_w135.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e60974c45e2f5bed823a7bf66e20b7718ed7799102c920d48a63c938d4ee8962
+size 2152006603

ckpt_w136.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef12eb1910308641801d7cbfd0e80ee944e34f5deb13db5f287f2383975d5e18
+size 2152006603

ckpt_w137.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6bca39e573dde50987708eac46f48dfeff809a05ff963bb3b1388ba01af73970
+size 2152006603

ckpt_w138.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3870f437161cfa6dd79d0eaf56a4e0c8bdbe63e553f91fe9967aaf6ae7134d1b
+size 2152006603