metadata
language:
- en
- zh
- de
- es
- ru
- ko
- fr
- ja
- pt
- tr
- pl
- ca
- nl
- ar
- sv
- it
- id
- hi
- fi
- vi
- he
- uk
- el
- ms
- cs
- ro
- da
- hu
- ta
- 'no'
- th
- ur
- hr
- bg
- lt
- la
- mi
- ml
- cy
- sk
- te
- fa
- lv
- bn
- sr
- az
- sl
- kn
- et
- mk
- br
- eu
- is
- hy
- ne
- mn
- bs
- kk
- sq
- sw
- gl
- mr
- pa
- si
- km
- sn
- yo
- so
- af
- oc
- ka
- be
- tg
- sd
- gu
- am
- yi
- lo
- uz
- fo
- ht
- ps
- tk
- nn
- mt
- sa
- lb
- my
- bo
- tl
- mg
- as
- tt
- haw
- ln
- ha
- ba
- jw
- su
- yue
library_name: transformers
tags:
- ternary-quant
- quantization
- ternary
- audio
- speech-to-text
- whisper
base_model: openai/whisper-large-v3
pipeline_tag: automatic-speech-recognition
license: apache-2.0
Whisper-large-v3 — Ternary Quantized
Ternary-quantized version of openai/whisper-large-v3, produced with ternary-quant.
This demonstrates ternary-quant's component-aware workflow for audio/speech models. The decoder is ternary-quantized while the audio encoder is preserved in FP16 for transcription quality. This is a HuggingFace-native PTQ artifact rather than a GGUF deployment artifact.
Quantization details
| Metric | Value |
|---|---|
| Scheme | tritplane3 (3-plane progressive ternary) |
| Components quantized | decoder (320 linear layers) |
| Audio encoder | Kept in FP16 (preserving audio understanding quality) |
| Stored size | 943.7 MB |
| FP16 size | 1677.7 MB |
| Compression ratio | 1.8x |
Usage
from ternary_quant.inference import load_ternary_model
import torch
model, processor = load_ternary_model(
"AsadIsmail/whisper-large-v3-ternary",
runtime_mode="cached",
device="cpu"
)
# Important: cast to float32 to match encoder conv1d dtype
model = model.float()
# Transcribe audio
import librosa
audio, sr = librosa.load("audio.mp3", sr=16000)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
inputs = {k: v.to("cpu").float() for k, v in inputs.items()}
with torch.no_grad():
predicted_ids = model.generate(**inputs)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription[0])
Reproduce
pip install ternary-quant
ternary-quant quantize-broad openai/whisper-large-v3 \
--output ./whisper-large-v3-ternary \
--components decoder \
--scheme tritplane3 --dtype float16 --eval