kaldan commited on
Commit
c2f29fa
·
verified ·
1 Parent(s): 03a7fec

Upload KenLM model

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. BoKenlm-syl-v0.1.arpa +3 -0
  3. README.md +66 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ BoKenlm-syl-v0.1.arpa filter=lfs diff=lfs merge=lfs -text
BoKenlm-syl-v0.1.arpa ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9543ec822400cae13c10a5ac67cf14ebae24e496950da76f32d40a979d111699
3
+ size 2666327690
README.md ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # BoKenlm-syl-v0.1 - Tibetan KenLM Language Model
2
+
3
+ A KenLM n-gram language model trained on Tibetan text, tokenized with syllable tokenizer.
4
+
5
+ ## Model Details
6
+
7
+ | Parameter | Value |
8
+ | --- | --- |
9
+ | **Model Type** | Modified Kneser-Ney 5-gram |
10
+ | **Tokenizer** | Tibetan syllable-based (botok-rs SimpleTokenizer) |
11
+ | **Training Corpus** | `bo_corpus.txt` |
12
+ | **Pruning** | 0 0 1 |
13
+ | **Tokens** | 153,703,256 |
14
+ | **Vocabulary Size** | 105,195 |
15
+
16
+ ## N-gram Statistics
17
+
18
+ | Order | Count | D1 | D2 | D3+ |
19
+ | --- | --- | --- | --- | --- |
20
+ | 1 | 105,195 | 0.7318 | 1.0392 | 1.3101 |
21
+ | 2 | 2,503,807 | 0.6749 | 1.0227 | 1.3576 |
22
+ | 3 | 6,956,090 | 0.7422 | 1.0862 | 1.3739 |
23
+ | 4 | 13,423,388 | 0.8156 | 1.1560 | 1.3755 |
24
+ | 5 | 16,307,695 | 0.7422 | 1.2835 | 1.5109 |
25
+
26
+ ## Memory Estimates
27
+
28
+ | Type | MB | Details |
29
+ | --- | --- | --- |
30
+ | probing | 806 | assuming -p 1.5 |
31
+ | probing | 937 | assuming -r models -p 1.5 |
32
+ | trie | 379 | without quantization |
33
+ | trie | 206 | assuming -q 8 -b 8 quantization |
34
+ | trie | 334 | assuming -a 22 array pointer compression |
35
+ | trie | 161 | assuming -a 22 -q 8 -b 8 array pointer compression and quantization |
36
+
37
+ ## Training Resources
38
+
39
+ | Metric | Value |
40
+ | --- | --- |
41
+ | **Peak Virtual Memory** | 12,333 MB |
42
+ | **Peak RSS** | 8,008 MB |
43
+ | **Wall Time** | 88.5s |
44
+ | **User Time** | 88.9s |
45
+ | **System Time** | 26.3s |
46
+
47
+ ## Usage
48
+
49
+ ```python
50
+ import kenlm
51
+
52
+ model = kenlm.Model("BoKenlm-syl-v0.1.arpa")
53
+
54
+ # Score a tokenized sentence
55
+ score = model.score("བོད་ སྐད་ ཀྱི་ ཚིག་ གྲུབ་ འདི་ ཡིན།")
56
+ print(score)
57
+ ```
58
+
59
+ ## Files
60
+
61
+ - `BoKenlm-syl-v0.1.arpa` — ARPA format language model
62
+ - `README.md` — This model card
63
+
64
+ ## License
65
+
66
+ Apache 2.0