Upload folder using huggingface_hub

README.md

@@ -0,0 +1,45 @@
+# Structural FFN Decomposition Guides Cross-Model Compression and Quantization
+
+Artifacts for the paper by Yeonseong Cynn (River Lab, May 2026).
+
+## Summary
+
+Decomposes transformer FFN layers into structural (format-preserving) and classification-relevant components across BERT and GPT-2.
+
+Key findings:
+- Early-layer FFN is 90-200x more structural than classification-relevant; late layers approach 1:1
+- **Structural pruning**: head + FFN neuron removal with layer-wise retraining achieves 19.1% parameter reduction on BERT (SST-2) and 9.1% on GPT-2 with no accuracy loss
+- **Neuron pruning**: removing 8% rarely-active FFN neurons *improves* BERT accuracy by 0.3%
+- **Mixed-precision quantization**: INT4 on structurally-dominant layers (L1-L3) with STE retraining recovers to -2.1% loss
+
+## Files
+
+### Weights
+- `bert_sst2_int4_ste.pt` — BERT SST-2 with L1-L3 INT4 quantization + STE retraining. Standard BERT state_dict, loadable directly. Accuracy: 90.1% (original FP32: 92.4%).
+
+### Results — BERT (`results/bert/`)
+- `bert_structural_prune.json` — Per-layer structural pruning results (head/FFN reduction, accuracy)
+- `bert_sst2_all_prune.json` — All-layer simultaneous FFN pruning results
+- `bert_l8_prune_results.json` — L8 FFN correction + pruning (multi-seed)
+- `bert_quantize_results.json` — INT4/INT8 post-training quantization results
+- `bert_quantize_retrain.json` — INT4 STE retraining results
+
+### Results — GPT-2 (`results/gpt2/`)
+- `gpt2_structural_prune.json` — Per-layer structural pruning (head + FFN)
+- `gpt2_each_layer_prune.json` — Individual layer compression results
+- `gpt2_prune_validate.json` — Pruning validation (PPL, accuracy)
+
+### Figures
+- `figures/fig1_ratio.png` — FFN dual role ratio: BERT vs GPT-2 (log scale)
+- `figures/fig2_compression.png` — Per-layer compression rates comparison
+- `figures/fig3_pruning.png` — BERT SST-2 FFN neuron pruning curve
+- `figures/fig4_quantization.png` — INT4 quantization results (PTQ vs STE)
+
+## Base Models
+
+- BERT: [textattack/bert-base-uncased-SST-2](https://huggingface.co/textattack/bert-base-uncased-SST-2)
+- GPT-2: [gpt2](https://huggingface.co/gpt2) (124M, pre-trained)
+
+## License
+
+MIT

bert_sst2_int4_ste.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79966b6d38bca8420162c47e4b7417bc07b03bcc6770a4e743fd47f88cf246bd
+size 438002578

results/bert/bert_l8_prune_results.json

@@ -0,0 +1,42 @@
+[
+  {
+    "seed": 42,
+    "orig": 0.9243119266055045,
+    "fixed": 0.9185779816513762,
+    "pruned": 0.9139908256880734,
+    "keep": 706,
+    "total": 3072
+  },
+  {
+    "seed": 7,
+    "orig": 0.9243119266055045,
+    "fixed": 0.9231651376146789,
+    "pruned": 0.9139908256880734,
+    "keep": 707,
+    "total": 3072
+  },
+  {
+    "seed": 0,
+    "orig": 0.9243119266055045,
+    "fixed": 0.9197247706422018,
+    "pruned": 0.9208715596330275,
+    "keep": 713,
+    "total": 3072
+  },
+  {
+    "seed": 13,
+    "orig": 0.9243119266055045,
+    "fixed": 0.9174311926605505,
+    "pruned": 0.9162844036697247,
+    "keep": 712,
+    "total": 3072
+  },
+  {
+    "seed": 99,
+    "orig": 0.9243119266055045,
+    "fixed": 0.9208715596330275,
+    "pruned": 0.9151376146788991,
+    "keep": 703,
+    "total": 3072
+  }
+]

results/bert/bert_quantize_results.json

@@ -0,0 +1,37 @@
+{
+  "orig": 0.9243119266055045,
+  "results": {
+    "\uc804 \ub808\uc774\uc5b4 INT8": {
+      "accuracy": 0.9220183486238532,
+      "loss": 0.002293577981651307
+    },
+    "\uc804 \ub808\uc774\uc5b4 INT4": {
+      "accuracy": 0.5091743119266054,
+      "loss": 0.4151376146788991
+    },
+    "L1~L3 INT4 + \ub098\uba38\uc9c0 FP32": {
+      "accuracy": 0.8073394495412844,
+      "loss": 0.11697247706422009
+    },
+    "L1~L6 INT4 + \ub098\uba38\uc9c0 FP32": {
+      "accuracy": 0.5263761467889908,
+      "loss": 0.39793577981651373
+    },
+    "FFN\ub9cc \uc804\ubd80 INT4": {
+      "accuracy": 0.5091743119266054,
+      "loss": 0.4151376146788991
+    },
+    "L1~L6 FFN\ub9cc INT4": {
+      "accuracy": 0.5905963302752294,
+      "loss": 0.33371559633027514
+    },
+    "Mixed: L1~3 INT4, L4~8 INT8, L9~12 FP32": {
+      "accuracy": 0.8004587155963303,
+      "loss": 0.12385321100917424
+    },
+    "Mixed: L1~4 INT4, L5~10 INT8, L11~12 FP32": {
+      "accuracy": 0.7591743119266054,
+      "loss": 0.1651376146788991
+    }
+  }
+}

results/bert/bert_quantize_retrain.json

@@ -0,0 +1,12 @@
+{
+  "orig": 0.9243119266055045,
+  "ptq": 0.8073394495412844,
+  "retrained": 0.875,
+  "ste": 0.9013761467889908,
+  "layers": [
+    1,
+    2,
+    3
+  ],
+  "bits": 4
+}

results/bert/bert_sst2_all_prune.json

@@ -0,0 +1,41 @@
+{
+  "orig_accuracy": 0.9243119266055045,
+  "results": {
+    "0.001": {
+      "accuracy": 0.9231651376146789,
+      "removed": 439,
+      "pct": 1.190863715277778,
+      "params_saved": 674304
+    },
+    "0.005": {
+      "accuracy": 0.926605504587156,
+      "removed": 3092,
+      "pct": 8.387586805555555,
+      "params_saved": 4749312
+    },
+    "0.01": {
+      "accuracy": 0.9162844036697247,
+      "removed": 7205,
+      "pct": 19.544813368055554,
+      "params_saved": 11066880
+    },
+    "0.02": {
+      "accuracy": 0.6330275229357798,
+      "removed": 13933,
+      "pct": 37.79568142361111,
+      "params_saved": 21401088
+    },
+    "0.03": {
+      "accuracy": 0.5389908256880734,
+      "removed": 18383,
+      "pct": 49.86707899305556,
+      "params_saved": 28236288
+    },
+    "0.05": {
+      "accuracy": 0.5321100917431193,
+      "removed": 23803,
+      "pct": 64.56976996527779,
+      "params_saved": 36561408
+    }
+  }
+}

results/bert/bert_structural_prune.json

@@ -0,0 +1,246 @@
+{
+  "L1": {
+    "keep_heads": [
+      0,
+      2,
+      4,
+      5,
+      6,
+      8,
+      9,
+      11
+    ],
+    "ffn_keep": 1319,
+    "ffn_dead": 1753,
+    "1_pruned_acc": 0.9208715596330275,
+    "2_retrained_acc": 0.9197247706422018,
+    "orig_params": 7087872,
+    "final_params": 3606311,
+    "reduction_pct": 49.119975642901,
+    "saved": 3481561
+  },
+  "L2": {
+    "keep_heads": [
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11
+    ],
+    "ffn_keep": 1533,
+    "ffn_dead": 1539,
+    "1_pruned_acc": 0.9220183486238532,
+    "2_retrained_acc": 0.9208715596330275,
+    "orig_params": 7087872,
+    "final_params": 3935229,
+    "reduction_pct": 44.479400869541664,
+    "saved": 3152643
+  },
+  "L3": {
+    "keep_heads": [
+      0,
+      1,
+      3,
+      4,
+      5,
+      7,
+      8,
+      11
+    ],
+    "ffn_keep": 1932,
+    "ffn_dead": 1140,
+    "1_pruned_acc": 0.9231651376146789,
+    "2_retrained_acc": 0.9243119266055045,
+    "orig_params": 7087872,
+    "final_params": 4548492,
+    "reduction_pct": 35.827114259399714,
+    "saved": 2539380
+  },
+  "L4": {
+    "keep_heads": [
+      1,
+      2,
+      4,
+      6,
+      7,
+      8,
+      9,
+      11
+    ],
+    "ffn_keep": 2301,
+    "ffn_dead": 771,
+    "1_pruned_acc": 0.9197247706422018,
+    "2_retrained_acc": 0.9231651376146789,
+    "orig_params": 7087872,
+    "final_params": 5115645,
+    "reduction_pct": 27.825375514681983,
+    "saved": 1972227
+  },
+  "L5": {
+    "keep_heads": [
+      2,
+      3,
+      4,
+      7,
+      8,
+      9,
+      10,
+      11
+    ],
+    "ffn_keep": 2255,
+    "ffn_dead": 817,
+    "1_pruned_acc": 0.9254587155963303,
+    "2_retrained_acc": 0.9208715596330275,
+    "orig_params": 7087872,
+    "final_params": 5044943,
+    "reduction_pct": 28.822882241665766,
+    "saved": 2042929
+  },
+  "L6": {
+    "keep_heads": [
+      3,
+      4,
+      5,
+      6,
+      7,
+      9,
+      10,
+      11
+    ],
+    "ffn_keep": 2380,
+    "ffn_dead": 692,
+    "1_pruned_acc": 0.9220183486238532,
+    "2_retrained_acc": 0.9220183486238532,
+    "orig_params": 7087872,
+    "final_params": 5237068,
+    "reduction_pct": 26.11226613573157,
+    "saved": 1850804
+  },
+  "L7": {
+    "keep_heads": [
+      0,
+      2,
+      3,
+      4,
+      5,
+      6,
+      8,
+      9
+    ],
+    "ffn_keep": 2478,
+    "ffn_dead": 594,
+    "1_pruned_acc": 0.9220183486238532,
+    "2_retrained_acc": 0.9231651376146789,
+    "orig_params": 7087872,
+    "final_params": 5387694,
+    "reduction_pct": 23.987143108679163,
+    "saved": 1700178
+  },
+  "L8": {
+    "keep_heads": [
+      1,
+      2,
+      3,
+      4,
+      6,
+      8,
+      10,
+      11
+    ],
+    "ffn_keep": 2715,
+    "ffn_dead": 357,
+    "1_pruned_acc": 0.9220183486238532,
+    "2_retrained_acc": 0.9208715596330275,
+    "orig_params": 7087872,
+    "final_params": 5751963,
+    "reduction_pct": 18.847814971827933,
+    "saved": 1335909
+  },
+  "L9": {
+    "keep_heads": [
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      10,
+      11
+    ],
+    "ffn_keep": 2647,
+    "ffn_dead": 425,
+    "1_pruned_acc": 0.9220183486238532,
+    "2_retrained_acc": 0.9220183486238532,
+    "orig_params": 7087872,
+    "final_params": 5647447,
+    "reduction_pct": 20.322390133456132,
+    "saved": 1440425
+  },
+  "L10": {
+    "keep_heads": [
+      3,
+      4,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11
+    ],
+    "ffn_keep": 2627,
+    "ffn_dead": 445,
+    "1_pruned_acc": 0.9197247706422018,
+    "2_retrained_acc": 0.9277522935779816,
+    "orig_params": 7087872,
+    "final_params": 5616707,
+    "reduction_pct": 20.756088710405606,
+    "saved": 1471165
+  },
+  "L11": {
+    "keep_heads": [
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11
+    ],
+    "ffn_keep": 2594,
+    "ffn_dead": 478,
+    "1_pruned_acc": 0.9231651376146789,
+    "2_retrained_acc": 0.9243119266055045,
+    "orig_params": 7087872,
+    "final_params": 5565986,
+    "reduction_pct": 21.471691362372233,
+    "saved": 1521886
+  },
+  "L12": {
+    "keep_heads": [
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11
+    ],
+    "ffn_keep": 2818,
+    "ffn_dead": 254,
+    "1_pruned_acc": 0.9243119266055045,
+    "2_retrained_acc": 0.9231651376146789,
+    "orig_params": 7087872,
+    "final_params": 5910274,
+    "reduction_pct": 16.614267300538156,
+    "saved": 1177598
+  },
+  "summary": {
+    "orig_acc": 0.9243119266055045,
+    "total_saved": 23686705
+  }
+}

results/gpt2/gpt2_each_layer_prune.json

@@ -0,0 +1,350 @@
+{
+  "L1": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      11,
+      6,
+      4,
+      3
+    ],
+    "keep_heads": [
+      0,
+      1,
+      2,
+      5,
+      7,
+      8,
+      9,
+      10
+    ],
+    "1_head_pruned_ppl": 51.474700614883176,
+    "2_retrained_ppl": 42.78067766084466,
+    "ffn_dead": 18,
+    "ffn_keep": 3054,
+    "3_ffn_pruned_ppl": 42.800225355233856,
+    "4_final_ppl": 42.80013634231672,
+    "orig_params": 7087872,
+    "final_params": 6273006,
+    "reduction_pct": 11.496624092534402,
+    "response": "floor, and the cat was still asleep.\n\n\"I'm sorry,"
+  },
+  "L2": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      9,
+      8,
+      7,
+      0
+    ],
+    "keep_heads": [
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      10,
+      11
+    ],
+    "1_head_pruned_ppl": 43.49211919812955,
+    "2_retrained_ppl": 42.689468413731625,
+    "ffn_dead": 196,
+    "ffn_keep": 2876,
+    "3_ffn_pruned_ppl": 47.18787627109435,
+    "4_final_ppl": 42.73889757184751,
+    "orig_params": 7087872,
+    "final_params": 5999420,
+    "reduction_pct": 15.356541427384693,
+    "response": "floor, and the cat was still asleep.\n\n\"I'm sorry,"
+  },
+  "L3": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      1,
+      4,
+      2,
+      7
+    ],
+    "keep_heads": [
+      0,
+      3,
+      5,
+      6,
+      8,
+      9,
+      10,
+      11
+    ],
+    "1_head_pruned_ppl": 44.33594594422876,
+    "2_retrained_ppl": 42.45262338662148,
+    "ffn_dead": 333,
+    "ffn_keep": 2739,
+    "3_ffn_pruned_ppl": 43.83093806103229,
+    "4_final_ppl": 42.74821930283572,
+    "orig_params": 7087872,
+    "final_params": 5788851,
+    "reduction_pct": 18.32737667948857,
+    "response": "floor, and the cat was still asleep.\n\n\"I was like"
+  },
+  "L4": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      3,
+      1,
+      2,
+      7
+    ],
+    "keep_heads": [
+      0,
+      4,
+      5,
+      6,
+      8,
+      9,
+      10,
+      11
+    ],
+    "1_head_pruned_ppl": 43.35439449006305,
+    "2_retrained_ppl": 42.76731748730949,
+    "ffn_dead": 160,
+    "ffn_keep": 2912,
+    "3_ffn_pruned_ppl": 43.72942076446341,
+    "4_final_ppl": 42.85923799840107,
+    "orig_params": 7087872,
+    "final_params": 6054752,
+    "reduction_pct": 14.575883988875646,
+    "response": "floor, his head bowed, his eyes closed.\n\n\"I'm sor"
+  },
+  "L5": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      8,
+      4,
+      2,
+      3
+    ],
+    "keep_heads": [
+      0,
+      1,
+      5,
+      6,
+      7,
+      9,
+      10,
+      11
+    ],
+    "1_head_pruned_ppl": 43.05504096539171,
+    "2_retrained_ppl": 42.83700060587286,
+    "ffn_dead": 93,
+    "ffn_keep": 2979,
+    "3_ffn_pruned_ppl": 43.25582619860287,
+    "4_final_ppl": 42.84181086031711,
+    "orig_params": 7087872,
+    "final_params": 6157731,
+    "reduction_pct": 13.12299375609492,
+    "response": "bed, her head resting on the pillow.\n\n\"I'm sorry,"
+  },
+  "L6": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      3,
+      4,
+      6,
+      11
+    ],
+    "keep_heads": [
+      0,
+      1,
+      2,
+      5,
+      7,
+      8,
+      9,
+      10
+    ],
+    "1_head_pruned_ppl": 42.741236353443426,
+    "2_retrained_ppl": 42.86417964326868,
+    "ffn_dead": 89,
+    "ffn_keep": 2983,
+    "3_ffn_pruned_ppl": 43.53786287921236,
+    "4_final_ppl": 42.88954343116375,
+    "orig_params": 7087872,
+    "final_params": 6163879,
+    "reduction_pct": 13.036254040705025,
+    "response": "floor, and the cat was still asleep.\n\n\"I'm sorry,"
+  },
+  "L7": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      10,
+      4,
+      9,
+      8
+    ],
+    "keep_heads": [
+      0,
+      1,
+      2,
+      3,
+      5,
+      6,
+      7,
+      11
+    ],
+    "1_head_pruned_ppl": 42.82263481027395,
+    "2_retrained_ppl": 42.73473983330326,
+    "ffn_dead": 77,
+    "ffn_keep": 2995,
+    "3_ffn_pruned_ppl": 43.36903752224206,
+    "4_final_ppl": 42.77897999011683,
+    "orig_params": 7087872,
+    "final_params": 6182323,
+    "reduction_pct": 12.776034894535341,
+    "response": "floor, and the cat was still asleep.\n\n\"I'm not su"
+  },
+  "L8": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      2,
+      7,
+      10,
+      3
+    ],
+    "keep_heads": [
+      0,
+      1,
+      4,
+      5,
+      6,
+      8,
+      9,
+      11
+    ],
+    "1_head_pruned_ppl": 43.04766504591089,
+    "2_retrained_ppl": 42.95752446800668,
+    "ffn_dead": 61,
+    "ffn_keep": 3011,
+    "3_ffn_pruned_ppl": 43.518359113843076,
+    "4_final_ppl": 43.26192825871676,
+    "orig_params": 7087872,
+    "final_params": 6206915,
+    "reduction_pct": 12.429076032975765,
+    "response": "floor, and the cat was still asleep.\n\n\"I'm sorry,"
+  },
+  "L9": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      5,
+      1,
+      6,
+      11
+    ],
+    "keep_heads": [
+      0,
+      2,
+      3,
+      4,
+      7,
+      8,
+      9,
+      10
+    ],
+    "1_head_pruned_ppl": 42.9115192613442,
+    "2_retrained_ppl": 43.1090656041147,
+    "ffn_dead": 48,
+    "ffn_keep": 3024,
+    "3_ffn_pruned_ppl": 43.25361420060106,
+    "4_final_ppl": 43.12893368107996,
+    "orig_params": 7087872,
+    "final_params": 6226896,
+    "reduction_pct": 12.147171957958609,
+    "response": "floor, and the cat was still asleep.\n\n\"I'm not su"
+  },
+  "L10": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      6,
+      9,
+      0,
+      7
+    ],
+    "keep_heads": [
+      1,
+      2,
+      3,
+      4,
+      5,
+      8,
+      10,
+      11
+    ],
+    "1_head_pruned_ppl": 43.17806338090184,
+    "2_retrained_ppl": 42.96270073910099,
+    "ffn_dead": 40,
+    "ffn_keep": 3032,
+    "3_ffn_pruned_ppl": 43.25030736221565,
+    "4_final_ppl": 43.09593422928351,
+    "orig_params": 7087872,
+    "final_params": 6239192,
+    "reduction_pct": 11.97369252717882,
+    "response": "floor, his head bowed, his eyes closed.\n\n\"I'm sor"
+  },
+  "L11": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      3,
+      4,
+      11,
+      6
+    ],
+    "keep_heads": [
+      0,
+      1,
+      2,
+      5,
+      7,
+      8,
+      9,
+      10
+    ],
+    "1_head_pruned_ppl": 43.03652662965237,
+    "2_retrained_ppl": 43.043991276127464,
+    "ffn_dead": 45,
+    "ffn_keep": 3027,
+    "3_ffn_pruned_ppl": 43.32754885474287,
+    "4_final_ppl": 43.07131553183212,
+    "orig_params": 7087872,
+    "final_params": 6231507,
+    "reduction_pct": 12.082117171416188,
+    "response": "floor, and the cat was still asleep.\n\n\"I'm sorry,"
+  },
+  "L12": {
+    "orig_ppl": 42.69972909357083,
+    "remove_heads": [
+      11,
+      7,
+      10,
+      5
+    ],
+    "keep_heads": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      6,
+      8,
+      9
+    ],
+    "1_head_pruned_ppl": 42.88330372658153,
+    "2_retrained_ppl": 42.87923366407565,
+    "ffn_dead": 39,
+    "ffn_keep": 3033,
+    "3_ffn_pruned_ppl": 42.82775165013662,
+    "4_final_ppl": 43.02837596987722,
+    "orig_params": 7087872,
+    "final_params": 6240729,
+    "reduction_pct": 11.952007598331345,
+    "response": "floor, his head bowed, his eyes closed.\n\n\"I'm sor"
+  }
+}

results/gpt2/gpt2_prune_validate.json

@@ -0,0 +1,7 @@
+{
+  "orig_ppls": {
+    "val1": 42.69972909357083,
+    "val2": 60.21109614480912,
+    "test": 59.59504167758228
+  }
+}

results/gpt2/gpt2_structural_prune.json

@@ -0,0 +1,7 @@
+{
+  "0_orig": 42.69972909357083,
+  "1_head_pruned": 43.492672623598075,
+  "2_retrained": 42.68803812197702,
+  "3_ffn_pruned": 48.66275607347491,
+  "4_final": 42.789829607794296
+}