Omega-S: A Functional Resilience Index for LLM Fine-Tuning

Both are the right questions, and I have measured answers to one of them and an honest "not tested" to the other.

Where the result does hold: Llama-3-8B, LoRA, code then prose, ten seeds. 84.1% retention against 62.9% for no regularizer, winning 9/10 seeds against none, 10/10 against weight decay and 8/10 against EWC, at under 4% added cost per step. That is the claim, and it is bounded by exactly the two things you are asking about.

Full fine-tuning: I ran it, and Omega-S does nothing there. A 0.3B model, full FT, one task pair: no regularizer gives a +5.01% code-loss increase, Omega gives +4.99% applied at every step. I swept the strength across four targets in case the schedule was starving it, and it never helps at any point. I re-ran with the published orientation of the M term in case that was the issue. Run-to-run noise on that harness is about 0.3 points, so the entire sweep sits inside or barely above it. L2-SP on the same runs roughly halves the forgetting without costing anything in prose loss. So the effect looks LoRA-specific, and I would not extrapolate the paper past that. One seed and one model size, so treat it as a boundary marker rather than a law.

That second point deserves more than a footnote, so here it is up front: L2-SP is not in the paper's baselines and it should have been. I ran it as a seventh arm on the published setup with the same ten seeds. It comes out at least on par with Omega-S: 6-3 with one tie in paired wins, sign test p=0.25, Wilcoxon p=0.049, so the two tests disagree and I am not claiming a separation in either direction. Per-seed data is in results/controls_20260808.json in the repo. The one axis where they do differ: Omega-S finishes task A with slightly more capability than no regularizer at all (0.3049 vs 0.2848, 7/10 seeds) while L2-SP is flat there. Neither contrast is significant alone, but the contrast between them is what you would expect from a method that restricts displacement versus one that constrains the shape of the solution without anchoring to any reference point. That, rather than raw retention, is the honest differentiator, and it is also why Omega-S needs no task boundary and no stored copy of the previous weights.

Close-to-base-distribution data: not tested. The measured mechanism is degree-variance reduction on the weight graph, which has no dependence on how far the fine-tuning data sits from the base distribution, so the naive prediction is that it behaves the same. But that is a prediction from a mechanism, not a measurement, and the paper reports that three of the four factors in the index are inert in this construction, so I am not going to lean on it. Chat formatting and tool-call syntax are the right test.

So I think your stopping rule is correct. One task pair, one architecture, LoRA only, and a baseline that belongs in the table and was published separately rather than in the paper. If you do run it on a close-domain pair, I would like to see the numbers whichever way they land, and a negative is genuinely more useful to me than a confirmation. The license carries an evaluation exception, so measuring costs you nothing there.