GautamKishore commited on
Commit
b76db7a
路
verified 路
1 Parent(s): 6091b21

card: hero, vs-table, concessions, centered figs

Browse files
Files changed (1) hide show
  1. README.md +69 -7
README.md CHANGED
@@ -54,7 +54,15 @@ model-index:
54
 
55
  **NIRNAY (啶ㄠた啶班啶`く, "decision") is a 450M open-weight classifier for banking intent and typed decisions.** One forward pass turns a state plus a question into calibrated probabilities. No text generation, no API key, no per-call bill. Apache-2.0, built by [Eulogik](https://eulogik.com).
56
 
57
- ![Banking77: NIRNAY vs Jev vs untrained, same 3,080 test cases](./assets/benchmark_banking77.png)
 
 
 
 
 
 
 
 
58
 
59
  Banking77 keywords for search: banking intent classification, 77-way intent classifier, intent detection model, small language model for classification, calibrated decision model, system one model, Jev alternative, Laya fine-tune, on-device text classifier, Apache 2.0 classifier.
60
 
@@ -68,7 +76,26 @@ Banking77 keywords for search: banking intent classification, 77-way intent clas
68
  | Julia-1 144M | 0.64 (their 72-label pilot, n=100, shortlist) | their card concedes grouped routing drops answers |
69
  | Untrained baseline | 0.143 (our template, our measurement) | fresh weights |
70
 
71
- Raw JSON: [`eval/banking77_phase_b.json`](./eval/banking77_phase_b.json), [`eval/banking77_phase_a.json`](./eval/banking77_phase_a.json). Same test split for every row above. The honest caveat: we fine-tuned on the train split, Jev answered zero-shot. That is exactly the Laya thesis (a model you fine-tune on your data), and this repo proves it works: +7.6 points over the API on identical cases.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
72
 
73
  Speed, batch-1, measured 2026-09-30: **209 ms on M4 MPS, 361 ms on CPU.** Faster than Jev API calls (310 to 478 ms in independent runs). Slower than Laya's 33 ms. Corpus latency work (ONNX) is open.
74
 
@@ -82,8 +109,8 @@ pip install git+https://github.com/eulogik/nirnay
82
  from nirnay.agent import NirnayAgent
83
 
84
  agent = NirnayAgent(device="mps", checkpoint_path="phase_b.pt", enable_byte_path=False)
85
- out = agent.act(
86
- state="My card was charged twice for the same order.",
87
  questions={"intent": {
88
  "type": "choice",
89
  "instructions": "Classify the banking intent.",
@@ -93,13 +120,36 @@ out = agent.act(
93
  print(out["answers"]["intent"]["probabilities"])
94
  ```
95
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
96
  The checkpoint loads with `NirnayAgent(checkpoint_path=...)`. Full eval harness: `scripts/eval_checkpoint.py`. Server: `nirnay.server` exposes `POST /v1/systemone` (Jev-compatible wire format).
97
 
98
  ## How it was trained
99
 
100
  Base is Laya 421M (Apache-2.0) plus ~30M of additions (concept bottleneck, deep supervision, coarse-to-fine pointer, byte path). 7,000 Phase A steps + 50 RLCD steps, all on one Mac. Three-group optimizer, seeded everything, hashes frozen before training.
101
 
102
- ![Probe accuracy and flat NCP across all 7,000 steps](./assets/training_stability.png)
 
 
103
 
104
  Two training deaths taught us the fixes (both landed, both gated):
105
 
@@ -112,13 +162,17 @@ Plus a safety net around training itself: heldout probes every 250 steps, best-c
112
 
113
  We report raw and fitted ECE on every eval, always. Phase_b on Banking77 test:
114
 
115
- ![Reliability diagram, measured on test set](./assets/reliability.png)
 
 
116
 
117
  Most mass sits above 0.9 confidence at 92% accuracy there. Per-bucket temperatures ship with the run.
118
 
119
  ## Architecture
120
 
121
- ![Forward pass](./assets/architecture.png)
 
 
122
 
123
  Bytes and token ids feed a frozen Laya encoder (LoRA adapters train). A concept bottleneck (layer-normed product-VQ, 4 chunks x 32 codes, mixture-of-slots) adds a learned residual. A 2-layer head scores dense markers; a coarse-to-fine pointer re-ranks the top 20 for 77-way decisions. Per-bucket temperatures calibrate the output. Deep supervision at layers 4/8/12 and RLCD exist only at training time.
124
 
@@ -159,3 +213,11 @@ NIRNAY is built by [Eulogik](https://eulogik.com) ([GitHub](https://github.com/e
159
  ## License
160
 
161
  Apache-2.0. Laya base (ConvAI Innovations, Apache-2.0). Banking77 data (PolyAI, CC-BY-4.0).
 
 
 
 
 
 
 
 
 
54
 
55
  **NIRNAY (啶ㄠた啶班啶`く, "decision") is a 450M open-weight classifier for banking intent and typed decisions.** One forward pass turns a state plus a question into calibrated probabilities. No text generation, no API key, no per-call bill. Apache-2.0, built by [Eulogik](https://eulogik.com).
56
 
57
+ <p align="center">
58
+ <img src="./assets/hero.png" alt="NIRNAY 450M: 87.9% on Banking77, fitted ECE 0.045, Apache-2.0" width="100%" />
59
+ </p>
60
+
61
+ [Model](https://huggingface.co/eulogik/nirnay-450m) 路 [Code](https://github.com/eulogik/nirnay) 路 [Paper](https://github.com/eulogik/nirnay/blob/main/papers/nirnay/main.pdf) 路 [Evals](./eval/banking77_phase_b.json)
62
+
63
+ <p align="center">
64
+ <img src="./assets/benchmark_banking77.png" alt="Banking77: NIRNAY vs Jev vs Julia-1 vs untrained, same 3,080 test cases" width="100%" />
65
+ </p>
66
 
67
  Banking77 keywords for search: banking intent classification, 77-way intent classifier, intent detection model, small language model for classification, calibrated decision model, system one model, Jev alternative, Laya fine-tune, on-device text classifier, Apache 2.0 classifier.
68
 
 
76
  | Julia-1 144M | 0.64 (their 72-label pilot, n=100, shortlist) | their card concedes grouped routing drops answers |
77
  | Untrained baseline | 0.143 (our template, our measurement) | fresh weights |
78
 
79
+ Raw JSON: [`eval/banking77_phase_b.json`](./eval/banking77_phase_b.json), [`eval/banking77_phase_a.json`](./eval/banking77_phase_a.json). Same test split for every row above.
80
+
81
+ > The honest caveat: we fine-tuned on the train split, Jev answered zero-shot. That is exactly the Laya thesis (a model you fine-tune on your data), and this repo proves it works: +7.6 points over the API on identical cases.
82
+
83
+ | Benchmark / Metric | Jev 1.13.0 | NIRNAY phase_b |
84
+ |---|---|---|
85
+ | Banking77 (same 3,080) | 0.803 (recorded run) | **0.8792** (fine-tuned) |
86
+ | ECE, raw | 0.197 gap (jevals, their setup) | **0.089** (our test) |
87
+ | Latency, 1 decision | 310 to 478 ms (API medians) | **209 ms** MPS / 361 ms CPU |
88
+ | Weights | closed API | **Apache-2.0** |
89
+ | Cost | per-call billing | **$0 self-hosted** |
90
+
91
+ Jev figures above are third-party published, never measured here (no API access); sample sizes and prompts differ. Ours are measured in this repo.
92
+
93
+ ## Where Jev leads
94
+
95
+ * **JevBench overall.** 72.1 vs our 0.55 public: intelligence, sealed decisions, calibration at scale. Different league, honestly stated.
96
+ * **Long documents.** Our context caps at 512 tokens. Their API takes more.
97
+ * **Zero setup.** Their API works with a key. Ours needs a local install plus a 466MB checkpoint download.
98
+ * **Languages.** We trained English banking. They serve many languages out of the box.
99
 
100
  Speed, batch-1, measured 2026-09-30: **209 ms on M4 MPS, 361 ms on CPU.** Faster than Jev API calls (310 to 478 ms in independent runs). Slower than Laya's 33 ms. Corpus latency work (ONNX) is open.
101
 
 
109
  from nirnay.agent import NirnayAgent
110
 
111
  agent = NirnayAgent(device="mps", checkpoint_path="phase_b.pt", enable_byte_path=False)
112
+ out = agent.system_one(
113
+ "My card was charged twice for the same order.",
114
  questions={"intent": {
115
  "type": "choice",
116
  "instructions": "Classify the banking intent.",
 
120
  print(out["answers"]["intent"]["probabilities"])
121
  ```
122
 
123
+ Choice, score, and yes/no ride the same call:
124
+
125
+ ```python
126
+ out = agent.system_one(
127
+ "Ticket T-4411: outage started 02:10, still down, 40 agents idle.",
128
+ questions={
129
+ "urgency": {
130
+ "type": "score",
131
+ "instructions": "How urgent is this?",
132
+ "criteria": ["not urgent", "soon", "blocking"],
133
+ },
134
+ "page_oncall": {
135
+ "type": "noul",
136
+ "instructions": "Should the on-call be paged?",
137
+ },
138
+ },
139
+ )
140
+ print(out["answers"]["urgency"]["score"]) # expected level over the rubric
141
+ print(out["answers"]["page_oncall"]["noul"]) # P(true)
142
+ ```
143
+
144
  The checkpoint loads with `NirnayAgent(checkpoint_path=...)`. Full eval harness: `scripts/eval_checkpoint.py`. Server: `nirnay.server` exposes `POST /v1/systemone` (Jev-compatible wire format).
145
 
146
  ## How it was trained
147
 
148
  Base is Laya 421M (Apache-2.0) plus ~30M of additions (concept bottleneck, deep supervision, coarse-to-fine pointer, byte path). 7,000 Phase A steps + 50 RLCD steps, all on one Mac. Three-group optimizer, seeded everything, hashes frozen before training.
149
 
150
+ <p align="center">
151
+ <img src="./assets/training_stability.png" alt="Probe accuracy and flat NCP across all 7,000 steps" width="100%" />
152
+ </p>
153
 
154
  Two training deaths taught us the fixes (both landed, both gated):
155
 
 
162
 
163
  We report raw and fitted ECE on every eval, always. Phase_b on Banking77 test:
164
 
165
+ <p align="center">
166
+ <img src="./assets/reliability.png" alt="Reliability diagram, measured on test set" width="100%" />
167
+ </p>
168
 
169
  Most mass sits above 0.9 confidence at 92% accuracy there. Per-bucket temperatures ship with the run.
170
 
171
  ## Architecture
172
 
173
+ <p align="center">
174
+ <img src="./assets/architecture.png" alt="Forward pass" width="100%" />
175
+ </p>
176
 
177
  Bytes and token ids feed a frozen Laya encoder (LoRA adapters train). A concept bottleneck (layer-normed product-VQ, 4 chunks x 32 codes, mixture-of-slots) adds a learned residual. A 2-layer head scores dense markers; a coarse-to-fine pointer re-ranks the top 20 for 77-way decisions. Per-bucket temperatures calibrate the output. Deep supervision at layers 4/8/12 and RLCD exist only at training time.
178
 
 
213
  ## License
214
 
215
  Apache-2.0. Laya base (ConvAI Innovations, Apache-2.0). Banking77 data (PolyAI, CC-BY-4.0).
216
+
217
+ ## Links
218
+
219
+ - **Model:** https://huggingface.co/eulogik/nirnay-450m
220
+ - **Code:** https://github.com/eulogik/nirnay
221
+ - **Paper:** https://github.com/eulogik/nirnay/blob/main/papers/nirnay/main.pdf
222
+ - **Evals:** ./eval/banking77_phase_b.json
223
+ - **Eulogik:** https://eulogik.com (contact: info@eulogik.com)