Text Classification
Laya
ONNX
Transformers
Safetensors
PyTorch
English
nirnay
banking77
intent-classification
intent-detection
calibrated-decisions
calibration
decision-model
system-one
small-language-model
edge-ai
on-device
guardrails
triage
lora
rlcd
Eval Results (legacy)
Instructions to use eulogik/nirnay-450m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Laya
How to use eulogik/nirnay-450m with Laya:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Transformers
How to use eulogik/nirnay-450m with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="eulogik/nirnay-450m")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("eulogik/nirnay-450m", device_map="auto") - Notebooks
- Google Colab
- Kaggle
card: hero, vs-table, concessions, centered figs
Browse files
README.md
CHANGED
|
@@ -54,7 +54,15 @@ model-index:
|
|
| 54 |
|
| 55 |
**NIRNAY (啶ㄠた啶班啶`く, "decision") is a 450M open-weight classifier for banking intent and typed decisions.** One forward pass turns a state plus a question into calibrated probabilities. No text generation, no API key, no per-call bill. Apache-2.0, built by [Eulogik](https://eulogik.com).
|
| 56 |
|
| 57 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
|
| 59 |
Banking77 keywords for search: banking intent classification, 77-way intent classifier, intent detection model, small language model for classification, calibrated decision model, system one model, Jev alternative, Laya fine-tune, on-device text classifier, Apache 2.0 classifier.
|
| 60 |
|
|
@@ -68,7 +76,26 @@ Banking77 keywords for search: banking intent classification, 77-way intent clas
|
|
| 68 |
| Julia-1 144M | 0.64 (their 72-label pilot, n=100, shortlist) | their card concedes grouped routing drops answers |
|
| 69 |
| Untrained baseline | 0.143 (our template, our measurement) | fresh weights |
|
| 70 |
|
| 71 |
-
Raw JSON: [`eval/banking77_phase_b.json`](./eval/banking77_phase_b.json), [`eval/banking77_phase_a.json`](./eval/banking77_phase_a.json). Same test split for every row above.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
|
| 73 |
Speed, batch-1, measured 2026-09-30: **209 ms on M4 MPS, 361 ms on CPU.** Faster than Jev API calls (310 to 478 ms in independent runs). Slower than Laya's 33 ms. Corpus latency work (ONNX) is open.
|
| 74 |
|
|
@@ -82,8 +109,8 @@ pip install git+https://github.com/eulogik/nirnay
|
|
| 82 |
from nirnay.agent import NirnayAgent
|
| 83 |
|
| 84 |
agent = NirnayAgent(device="mps", checkpoint_path="phase_b.pt", enable_byte_path=False)
|
| 85 |
-
out = agent.
|
| 86 |
-
|
| 87 |
questions={"intent": {
|
| 88 |
"type": "choice",
|
| 89 |
"instructions": "Classify the banking intent.",
|
|
@@ -93,13 +120,36 @@ out = agent.act(
|
|
| 93 |
print(out["answers"]["intent"]["probabilities"])
|
| 94 |
```
|
| 95 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 96 |
The checkpoint loads with `NirnayAgent(checkpoint_path=...)`. Full eval harness: `scripts/eval_checkpoint.py`. Server: `nirnay.server` exposes `POST /v1/systemone` (Jev-compatible wire format).
|
| 97 |
|
| 98 |
## How it was trained
|
| 99 |
|
| 100 |
Base is Laya 421M (Apache-2.0) plus ~30M of additions (concept bottleneck, deep supervision, coarse-to-fine pointer, byte path). 7,000 Phase A steps + 50 RLCD steps, all on one Mac. Three-group optimizer, seeded everything, hashes frozen before training.
|
| 101 |
|
| 102 |
-
|
|
|
|
|
|
|
| 103 |
|
| 104 |
Two training deaths taught us the fixes (both landed, both gated):
|
| 105 |
|
|
@@ -112,13 +162,17 @@ Plus a safety net around training itself: heldout probes every 250 steps, best-c
|
|
| 112 |
|
| 113 |
We report raw and fitted ECE on every eval, always. Phase_b on Banking77 test:
|
| 114 |
|
| 115 |
-
|
|
|
|
|
|
|
| 116 |
|
| 117 |
Most mass sits above 0.9 confidence at 92% accuracy there. Per-bucket temperatures ship with the run.
|
| 118 |
|
| 119 |
## Architecture
|
| 120 |
|
| 121 |
-
|
|
|
|
|
|
|
| 122 |
|
| 123 |
Bytes and token ids feed a frozen Laya encoder (LoRA adapters train). A concept bottleneck (layer-normed product-VQ, 4 chunks x 32 codes, mixture-of-slots) adds a learned residual. A 2-layer head scores dense markers; a coarse-to-fine pointer re-ranks the top 20 for 77-way decisions. Per-bucket temperatures calibrate the output. Deep supervision at layers 4/8/12 and RLCD exist only at training time.
|
| 124 |
|
|
@@ -159,3 +213,11 @@ NIRNAY is built by [Eulogik](https://eulogik.com) ([GitHub](https://github.com/e
|
|
| 159 |
## License
|
| 160 |
|
| 161 |
Apache-2.0. Laya base (ConvAI Innovations, Apache-2.0). Banking77 data (PolyAI, CC-BY-4.0).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
**NIRNAY (啶ㄠた啶班啶`く, "decision") is a 450M open-weight classifier for banking intent and typed decisions.** One forward pass turns a state plus a question into calibrated probabilities. No text generation, no API key, no per-call bill. Apache-2.0, built by [Eulogik](https://eulogik.com).
|
| 56 |
|
| 57 |
+
<p align="center">
|
| 58 |
+
<img src="./assets/hero.png" alt="NIRNAY 450M: 87.9% on Banking77, fitted ECE 0.045, Apache-2.0" width="100%" />
|
| 59 |
+
</p>
|
| 60 |
+
|
| 61 |
+
[Model](https://huggingface.co/eulogik/nirnay-450m) 路 [Code](https://github.com/eulogik/nirnay) 路 [Paper](https://github.com/eulogik/nirnay/blob/main/papers/nirnay/main.pdf) 路 [Evals](./eval/banking77_phase_b.json)
|
| 62 |
+
|
| 63 |
+
<p align="center">
|
| 64 |
+
<img src="./assets/benchmark_banking77.png" alt="Banking77: NIRNAY vs Jev vs Julia-1 vs untrained, same 3,080 test cases" width="100%" />
|
| 65 |
+
</p>
|
| 66 |
|
| 67 |
Banking77 keywords for search: banking intent classification, 77-way intent classifier, intent detection model, small language model for classification, calibrated decision model, system one model, Jev alternative, Laya fine-tune, on-device text classifier, Apache 2.0 classifier.
|
| 68 |
|
|
|
|
| 76 |
| Julia-1 144M | 0.64 (their 72-label pilot, n=100, shortlist) | their card concedes grouped routing drops answers |
|
| 77 |
| Untrained baseline | 0.143 (our template, our measurement) | fresh weights |
|
| 78 |
|
| 79 |
+
Raw JSON: [`eval/banking77_phase_b.json`](./eval/banking77_phase_b.json), [`eval/banking77_phase_a.json`](./eval/banking77_phase_a.json). Same test split for every row above.
|
| 80 |
+
|
| 81 |
+
> The honest caveat: we fine-tuned on the train split, Jev answered zero-shot. That is exactly the Laya thesis (a model you fine-tune on your data), and this repo proves it works: +7.6 points over the API on identical cases.
|
| 82 |
+
|
| 83 |
+
| Benchmark / Metric | Jev 1.13.0 | NIRNAY phase_b |
|
| 84 |
+
|---|---|---|
|
| 85 |
+
| Banking77 (same 3,080) | 0.803 (recorded run) | **0.8792** (fine-tuned) |
|
| 86 |
+
| ECE, raw | 0.197 gap (jevals, their setup) | **0.089** (our test) |
|
| 87 |
+
| Latency, 1 decision | 310 to 478 ms (API medians) | **209 ms** MPS / 361 ms CPU |
|
| 88 |
+
| Weights | closed API | **Apache-2.0** |
|
| 89 |
+
| Cost | per-call billing | **$0 self-hosted** |
|
| 90 |
+
|
| 91 |
+
Jev figures above are third-party published, never measured here (no API access); sample sizes and prompts differ. Ours are measured in this repo.
|
| 92 |
+
|
| 93 |
+
## Where Jev leads
|
| 94 |
+
|
| 95 |
+
* **JevBench overall.** 72.1 vs our 0.55 public: intelligence, sealed decisions, calibration at scale. Different league, honestly stated.
|
| 96 |
+
* **Long documents.** Our context caps at 512 tokens. Their API takes more.
|
| 97 |
+
* **Zero setup.** Their API works with a key. Ours needs a local install plus a 466MB checkpoint download.
|
| 98 |
+
* **Languages.** We trained English banking. They serve many languages out of the box.
|
| 99 |
|
| 100 |
Speed, batch-1, measured 2026-09-30: **209 ms on M4 MPS, 361 ms on CPU.** Faster than Jev API calls (310 to 478 ms in independent runs). Slower than Laya's 33 ms. Corpus latency work (ONNX) is open.
|
| 101 |
|
|
|
|
| 109 |
from nirnay.agent import NirnayAgent
|
| 110 |
|
| 111 |
agent = NirnayAgent(device="mps", checkpoint_path="phase_b.pt", enable_byte_path=False)
|
| 112 |
+
out = agent.system_one(
|
| 113 |
+
"My card was charged twice for the same order.",
|
| 114 |
questions={"intent": {
|
| 115 |
"type": "choice",
|
| 116 |
"instructions": "Classify the banking intent.",
|
|
|
|
| 120 |
print(out["answers"]["intent"]["probabilities"])
|
| 121 |
```
|
| 122 |
|
| 123 |
+
Choice, score, and yes/no ride the same call:
|
| 124 |
+
|
| 125 |
+
```python
|
| 126 |
+
out = agent.system_one(
|
| 127 |
+
"Ticket T-4411: outage started 02:10, still down, 40 agents idle.",
|
| 128 |
+
questions={
|
| 129 |
+
"urgency": {
|
| 130 |
+
"type": "score",
|
| 131 |
+
"instructions": "How urgent is this?",
|
| 132 |
+
"criteria": ["not urgent", "soon", "blocking"],
|
| 133 |
+
},
|
| 134 |
+
"page_oncall": {
|
| 135 |
+
"type": "noul",
|
| 136 |
+
"instructions": "Should the on-call be paged?",
|
| 137 |
+
},
|
| 138 |
+
},
|
| 139 |
+
)
|
| 140 |
+
print(out["answers"]["urgency"]["score"]) # expected level over the rubric
|
| 141 |
+
print(out["answers"]["page_oncall"]["noul"]) # P(true)
|
| 142 |
+
```
|
| 143 |
+
|
| 144 |
The checkpoint loads with `NirnayAgent(checkpoint_path=...)`. Full eval harness: `scripts/eval_checkpoint.py`. Server: `nirnay.server` exposes `POST /v1/systemone` (Jev-compatible wire format).
|
| 145 |
|
| 146 |
## How it was trained
|
| 147 |
|
| 148 |
Base is Laya 421M (Apache-2.0) plus ~30M of additions (concept bottleneck, deep supervision, coarse-to-fine pointer, byte path). 7,000 Phase A steps + 50 RLCD steps, all on one Mac. Three-group optimizer, seeded everything, hashes frozen before training.
|
| 149 |
|
| 150 |
+
<p align="center">
|
| 151 |
+
<img src="./assets/training_stability.png" alt="Probe accuracy and flat NCP across all 7,000 steps" width="100%" />
|
| 152 |
+
</p>
|
| 153 |
|
| 154 |
Two training deaths taught us the fixes (both landed, both gated):
|
| 155 |
|
|
|
|
| 162 |
|
| 163 |
We report raw and fitted ECE on every eval, always. Phase_b on Banking77 test:
|
| 164 |
|
| 165 |
+
<p align="center">
|
| 166 |
+
<img src="./assets/reliability.png" alt="Reliability diagram, measured on test set" width="100%" />
|
| 167 |
+
</p>
|
| 168 |
|
| 169 |
Most mass sits above 0.9 confidence at 92% accuracy there. Per-bucket temperatures ship with the run.
|
| 170 |
|
| 171 |
## Architecture
|
| 172 |
|
| 173 |
+
<p align="center">
|
| 174 |
+
<img src="./assets/architecture.png" alt="Forward pass" width="100%" />
|
| 175 |
+
</p>
|
| 176 |
|
| 177 |
Bytes and token ids feed a frozen Laya encoder (LoRA adapters train). A concept bottleneck (layer-normed product-VQ, 4 chunks x 32 codes, mixture-of-slots) adds a learned residual. A 2-layer head scores dense markers; a coarse-to-fine pointer re-ranks the top 20 for 77-way decisions. Per-bucket temperatures calibrate the output. Deep supervision at layers 4/8/12 and RLCD exist only at training time.
|
| 178 |
|
|
|
|
| 213 |
## License
|
| 214 |
|
| 215 |
Apache-2.0. Laya base (ConvAI Innovations, Apache-2.0). Banking77 data (PolyAI, CC-BY-4.0).
|
| 216 |
+
|
| 217 |
+
## Links
|
| 218 |
+
|
| 219 |
+
- **Model:** https://huggingface.co/eulogik/nirnay-450m
|
| 220 |
+
- **Code:** https://github.com/eulogik/nirnay
|
| 221 |
+
- **Paper:** https://github.com/eulogik/nirnay/blob/main/papers/nirnay/main.pdf
|
| 222 |
+
- **Evals:** ./eval/banking77_phase_b.json
|
| 223 |
+
- **Eulogik:** https://eulogik.com (contact: info@eulogik.com)
|