Image-to-Text
Transformers
Safetensors
Sinhala
vision-encoder-decoder
image-text-to-text
ocr
handwriting
sinhala
trocr
Instructions to use SriDoc/TrOCR-Sin-Handwritten with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use SriDoc/TrOCR-Sin-Handwritten with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "image-to-text" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("image-to-text", model="SriDoc/TrOCR-Sin-Handwritten")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForMultimodalLM tokenizer = AutoTokenizer.from_pretrained("SriDoc/TrOCR-Sin-Handwritten") model = AutoModelForMultimodalLM.from_pretrained("SriDoc/TrOCR-Sin-Handwritten", device_map="auto") - Notebooks
- Google Colab
- Kaggle
update readme file
Browse files
README.md
CHANGED
|
@@ -1,36 +1,86 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
|
|
|
|
|
|
| 3 |
base_model: kavg/TrOCR-SIN-DeiT
|
|
|
|
| 4 |
tags:
|
| 5 |
-
-
|
| 6 |
-
|
| 7 |
-
-
|
| 8 |
-
|
| 9 |
---
|
| 10 |
|
| 11 |
-
|
| 12 |
-
should probably proofread and complete it, then remove this comment. -->
|
| 13 |
|
| 14 |
-
|
| 15 |
|
| 16 |
-
|
| 17 |
-
It achieves the following results on the evaluation set:
|
| 18 |
-
- Loss: 2.9839
|
| 19 |
-
- Cer: 0.5253
|
| 20 |
|
| 21 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
-
|
| 24 |
|
| 25 |
-
|
|
|
|
|
|
|
| 26 |
|
| 27 |
-
|
| 28 |
|
| 29 |
-
|
| 30 |
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
### Training hyperparameters
|
| 36 |
|
|
@@ -46,6 +96,8 @@ The following hyperparameters were used during training:
|
|
| 46 |
|
| 47 |
### Training results
|
| 48 |
|
|
|
|
|
|
|
| 49 |
| Training Loss | Epoch | Step | Validation Loss | Cer |
|
| 50 |
|:-------------:|:-----:|:----:|:---------------:|:------:|
|
| 51 |
| 0.2915 | 3.45 | 100 | 1.8613 | 0.6450 |
|
|
@@ -78,7 +130,6 @@ The following hyperparameters were used during training:
|
|
| 78 |
| 0.0 | 96.55 | 2800 | 3.1309 | 0.5273 |
|
| 79 |
| 0.0 | 100.0 | 2900 | 2.9839 | 0.5253 |
|
| 80 |
|
| 81 |
-
|
| 82 |
### Framework versions
|
| 83 |
|
| 84 |
- Transformers 4.35.2
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
language:
|
| 4 |
+
- si
|
| 5 |
base_model: kavg/TrOCR-SIN-DeiT
|
| 6 |
+
pipeline_tag: image-to-text
|
| 7 |
tags:
|
| 8 |
+
- ocr
|
| 9 |
+
- handwriting
|
| 10 |
+
- sinhala
|
| 11 |
+
- trocr
|
| 12 |
---
|
| 13 |
|
| 14 |
+
# TrOCR-Sin-Handwritten
|
|
|
|
| 15 |
|
| 16 |
+
A Sinhala text-line recognizer for **handwritten** text, released with the SinFUND and SinOCR datasets ([SriDoc/datasets](https://github.com/SriDoc/datasets)).
|
| 17 |
|
| 18 |
+
## Model
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
+
- TrOCR-style vision encoder-decoder (`VisionEncoderDecoderModel`).
|
| 21 |
+
- The encoder was initialized from [`facebook/deit-base-distilled-patch16-224`](https://huggingface.co/facebook/deit-base-distilled-patch16-224) (ImageNet weights). The decoder was initialized from [`NLPC-UOM/SinBERT-large`](https://huggingface.co/NLPC-UOM/SinBERT-large), a RoBERTa model pre-trained on Sinhala text. The cross-attention layers started from random weights.
|
| 22 |
+
- No OCR-pretrained TrOCR weights were used.
|
| 23 |
+
- **Input:** one pre-cropped text-line image, resized to 224 x 224.
|
| 24 |
+
- **Decoding:** beam search with 4 beams and a maximum length of 64 tokens.
|
| 25 |
|
| 26 |
+
## Training data
|
| 27 |
|
| 28 |
+
This model was trained in two stages:
|
| 29 |
+
1. On the SinOCR printed training split. That model is [SriDoc/TrOCR-Sin-Printed](https://huggingface.co/SriDoc/TrOCR-Sin-Printed).
|
| 30 |
+
2. Then on the SinOCR handwritten training split (908 lines) for 100 epochs (2,900 steps), with a batch size of 32 and a learning rate of 5e-5.
|
| 31 |
|
| 32 |
+
## Results
|
| 33 |
|
| 34 |
+
Character error rate (CER; lower is better) on the SinOCR test splits:
|
| 35 |
|
| 36 |
+
| Test split | CER |
|
| 37 |
+
|---|---|
|
| 38 |
+
| SinOCR handwritten (227 lines) | 0.5253 |
|
| 39 |
+
| SinOCR printed (10,000 lines) | 0.2246 |
|
| 40 |
|
| 41 |
+
How the CER is computed:
|
| 42 |
+
- Characters are Unicode code points.
|
| 43 |
+
- The CER is pooled over the whole test split.
|
| 44 |
+
|
| 45 |
+
SinOCR has no separate validation split, so the test split was also the evaluation set during training. The lowest test CER occurred at the final checkpoint, so checkpoint selection did not change this result.
|
| 46 |
+
|
| 47 |
+
## Limitations
|
| 48 |
+
|
| 49 |
+
- **Errors remain common.** A CER of about 0.53 means roughly one character error for every two characters, so the output needs human checking.
|
| 50 |
+
- **Test writers were seen in training.** The handwritten test split is not writer-disjoint: 72% of its lines come from writers who also appear in training. The CER is therefore probably optimistic for new writers.
|
| 51 |
+
- **Do not use this model to measure OCR accuracy on the SinFUND test forms.** 132 of its training lines are crops of those forms. The dataset repository lists them in `sinocr_handwritten_provenance.csv`.
|
| 52 |
+
- The model reads single, pre-cropped lines. It does not find text on a page.
|
| 53 |
+
|
| 54 |
+
## Usage
|
| 55 |
+
|
| 56 |
+
```python
|
| 57 |
+
from transformers import VisionEncoderDecoderModel, TrOCRProcessor, AutoTokenizer, ViTImageProcessor
|
| 58 |
+
from PIL import Image
|
| 59 |
+
|
| 60 |
+
model = VisionEncoderDecoderModel.from_pretrained("SriDoc/TrOCR-Sin-Handwritten")
|
| 61 |
+
tokenizer = AutoTokenizer.from_pretrained("NLPC-UOM/SinBERT-large")
|
| 62 |
+
image_processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224")
|
| 63 |
+
processor = TrOCRProcessor(image_processor=image_processor, tokenizer=tokenizer)
|
| 64 |
+
|
| 65 |
+
image = Image.open("line.png").convert("RGB") # one cropped text line
|
| 66 |
+
pixel_values = processor(image, return_tensors="pt").pixel_values
|
| 67 |
+
generated_ids = model.generate(pixel_values)
|
| 68 |
+
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
## Citation
|
| 72 |
+
|
| 73 |
+
```bibtex
|
| 74 |
+
@article{gunathilaka2025sinfund,
|
| 75 |
+
title = {SinFUND and SinOCR: Benchmarks for Sinhala Handwritten OCR and Template-Free Form Understanding},
|
| 76 |
+
author = {Gunathilaka, Kavishka and Hewagama, Danusha and Pushpakumara, Supul and Ambegoda, Thanuja D.},
|
| 77 |
+
journal = {Research Square preprint},
|
| 78 |
+
year = {2025},
|
| 79 |
+
doi = {10.21203/rs.3.rs-6976719/v1}
|
| 80 |
+
}
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
## Training log
|
| 84 |
|
| 85 |
### Training hyperparameters
|
| 86 |
|
|
|
|
| 96 |
|
| 97 |
### Training results
|
| 98 |
|
| 99 |
+
The "Cer" and "Validation Loss" columns were measured on the SinOCR handwritten **test** split.
|
| 100 |
+
|
| 101 |
| Training Loss | Epoch | Step | Validation Loss | Cer |
|
| 102 |
|:-------------:|:-----:|:----:|:---------------:|:------:|
|
| 103 |
| 0.2915 | 3.45 | 100 | 1.8613 | 0.6450 |
|
|
|
|
| 130 |
| 0.0 | 96.55 | 2800 | 3.1309 | 0.5273 |
|
| 131 |
| 0.0 | 100.0 | 2900 | 2.9839 | 0.5253 |
|
| 132 |
|
|
|
|
| 133 |
### Framework versions
|
| 134 |
|
| 135 |
- Transformers 4.35.2
|