Benchmarks
Every number, and how we got it.
Results are measured on fixed tests, and the reading results are shown with their 95% intervals. Where our lead isn't clear of the next system, we say so.
Printed Arabic · Kindi OCR v1.0
Printed-text test (MDP-90): 90 blocks from Arabic newspaper pages, 4,320 words, each block scored as a whole block, so errors in line and column order count too. Character error rate, lower is better.
Our interval overlaps Mistral OCR's, so we call it the lowest error in this comparison, not a clear win. The gap to PaddleOCR-VL and Tesseract is clear.
Kindi OCR in detail| System | Character error rate | 95% interval |
|---|---|---|
| Kindi OCR | 0.89% | 0.6–1.2 |
| Mistral OCR v4.1 | 1.29% | 1.0–1.6 |
| deep.sa OCR v2.0 | 2.63% | 1.9–3.7 |
| PaddleOCR-VL 1.6 | 5.4% | 4.2–6.5 |
| Tesseract | 6.53% | 4.5–9.3 |
0%2%4%6%8%10% |
Handwriting · Kindi HTR
Handwriting test: 2,000 handwritten lines, 24,453 words, in five historical Arabic scripts. The same lines for every system and every version. Character error rate, lower is better. In development.
Our upper bound, 9.26%, is below HATFormer's lower bound, 20.89%: the intervals don't overlap.
Kindi HTR in detail| System | Character error rate | 95% interval |
|---|---|---|
| Kindi v0.5 preview | 8.83% | 8.45–9.26 |
| HATFormer | 21.46% | 20.89–22.04 |
| sdkv2 | 37.36% | 36.81–37.93 |
| Mistral OCR v4.1 | 39.49% | 38.44–40.57 |
| Qari-OCR v0.3 | 47.93% | 47.01–48.88 |
| Tesseract | 62.83% | 62.22–63.49 |
0%10%20%30%40%50%60%70% |
| Item | HATFormer | Kindi v0.5 preview |
|---|---|---|
| Oriental | 32.0% | 3.7% |
| Ruq'ah | 17.9% | 5.8% |
| Nastaliq | 20.7% | 8.2% |
| Ta'liq | 13.4% | 8.8% |
| Naskh | 21.2% | 9.8% |
| Maghrebi | 33.9% | 10.4% |
| Modern | 41.4% | 15.3% |
Character error rate per script, lower is better, on a wider test of 8,136 lines. The 95% intervals don't overlap on any script; every bound is in the JSON. On modern handwriting, three other systems we tested read better than Kindi HTR.
Arabic retrieval · laya-ara-rag
For each question, up to 12 candidate passages; the score is how often the model's first pick is the right one. Trained on MIRACL-ar only; the other four sets it never saw.
| Set | Questions |
|---|---|
| MIRACL-ar | 2,896 |
| Mr.TyDi-ar | 2,000 |
| SadeemQuestion | 2,089 |
| MLQA-ar | 2,000 |
| XPQA-ar | 750 |
| Item | laya-multilingual (base) | laya-ara-rag |
|---|---|---|
| MIRACL-ar (seen in training) | 15.3% | 58.8% |
| Mr.TyDi-ar | 17.6% | 63.2% |
| SadeemQuestion | 16.8% | 87.1% |
| MLQA-ar | 13.3% | 58.2% |
| XPQA-ar | 21.3% | 66.5% |
Higher is better.
Speech bench
The same sentence goes to each Arabic voice service. We time the first sound, how fast audio is made, and the slow cases.
The speech bench in detail- 7
- services compared
- 66
- requests timed