# Shekar AI

> Shekar AI is an open-source initiative providing free tools, models, and datasets for Persian (Farsi) Natural Language Processing (NLP). Its projects are the **Shekar** Python library (MIT), the **Neyshekar** Persian speech dataset (CC0 1.0), the **Shiraz** Persian web corpus (ODC-BY 1.0), and a family of Persian text **embedding** models, starting with **Noql** (Apache 2.0).

Persian (also called Farsi) is spoken by over 100 million people across Iran, Afghanistan (as Dari), and Tajikistan (as Tajik), but remains under-resourced in modern language technology. Shekar AI exists to close that gap. All resources are released under permissive licenses, with an emphasis on clean data, transparent evaluation, and practical usability.

The project is named after *Farsi Shekar Ast* (فارسی شکر است, "Persian is Sugar"), the 1921 short story by Mohammad Ali Jamalzadeh. شکر (*shekar*) means sugar; نیشکر (*neyshekar*) means sugarcane.

Founder and maintainer: Ahmad Amirivojdan.

Canonical site: https://shekar.ai/ | Last updated: 2026-10-05 | Content licence: site text CC BY 4.0; each project carries its own licence, noted below.

## Core Principles

- **Open**: All tools, models, and datasets are freely available under permissive licenses.
- **Reproducible**: Documented methods, versioned releases, and transparent evaluation.
- **Accessible**: Designed for researchers, developers, and students in academic and real-world settings.

## Projects

### Shekar Python Library

An open-source Python library covering the full Persian NLP pipeline in a single lightweight package. Install with `pip install shekar`. Works on Windows, Linux, and macOS including Apple Silicon. Transformer components use quantized ALBERT models exported to ONNX for fast CPU inference; GPU acceleration is available via `onnxruntime-gpu`.

Capabilities: text normalization (following Academy of Persian Language and Literature guidelines), word tokenization, sentence tokenization, POS tagging (Universal Dependencies tags), named entity recognition (persons, locations, organizations, dates), static word embeddings (FastText 100d/300d), contextual embeddings (ALBERT 768d), stemming, lemmatization, spell checking, dependency parsing, sentiment analysis, offensive language detection, informal language classification, keyword extraction, Farsi-Tajik transliteration, text augmentation, and word cloud generation. Preprocessing operators compose with the `|` operator into custom pipelines. A built-in web interface is available via `shekar serve -p 8080`.

License: MIT. Free for research and commercial use.

- Project page: https://shekar.ai/projects/shekar
- GitHub: https://github.com/amirivojdan/shekar
- PyPI: https://pypi.org/project/shekar/
- Documentation: https://lib.shekar.ai
- Paper (JOSS): https://joss.theoj.org/papers/10.21105/joss.09128

Citation: Amirivojdan, A. (2025). *Shekar: A Python Toolkit for Persian Natural Language Processing.* Journal of Open Source Software, 10(114), 9128. DOI: 10.21105/joss.09128

### Neyshekar (Persian Speech Dataset)

A large-scale open Persian speech dataset collected through a community crowdsourcing platform at ney.shekar.io, with contributions from volunteer native speakers and paid professional voice actors. Each release is a stable, versioned snapshot archived on Zenodo with a permanent DOI, enabling reproducible benchmarking.

Version 6.0 (released September 1, 2026):
- 62,279 recordings, 99.02 hours of audio, 5.72 seconds average clip duration
- 701,621 tokens, 29,535 unique vocabulary items
- 15,222 informal-register samples (24.44%), labelled with the Shekar rule-based InformalClassifier
- Speaker gender: 33,735 female and 28,076 male recordings
- Predefined splits (first release to ship them), assigned per speaker so evaluation sets stay speaker-disjoint from train: train 58,244 samples / 93.5% / 91.99 h, validation 1,886 / 3.0% / 3.14 h, test 2,149 / 3.5% / 3.88 h
- Includes transcriptions, gender labels, register labels, and NER-tagged entity metadata

Version history: v1 (Dec 29, 2025, 10,044 recordings / 14.42 h), v2 (Jan 15, 2026, 20,020 / 29.08 h), v3 (Mar 23, 2026, 30,019 / 45.71 h), v4.1 (Jun 15, 2026, 40,008 / 63.03 h), v5.0 (Jul 16, 2026, 50,026 / 79.22 h), v6.0 (Sep 1, 2026, 62,279 / 99.02 h).

Important: the v4 release (May 14, 2026) contains misaligned audio-transcript pairs and should not be used. Use v4.1 or later.

Intended tasks: automatic speech recognition (ASR), text-to-speech (TTS), speech representation learning, speaker identification, voice activity detection, low-resource language modeling, acoustic model training, and pronunciation modeling.

License: CC0 1.0 Universal (public domain dedication). Usable for any purpose, including commercial, without attribution. Attempting to identify individual speakers is prohibited.

- Project page: https://shekar.ai/projects/neyshekar
- Collection platform: https://ney.shekar.io
- GitHub: https://github.com/amirivojdan/neyshekar
- Dataset (Zenodo): https://doi.org/10.5281/zenodo.18073632
- Paper (arXiv): https://arxiv.org/abs/2609.14542

Paper: *Neyshekar: An Open Persian Read-Speech Corpus for Automatic Speech Recognition* (arXiv:2609.14542, September 2026; Ahmad Amirivojdan, Farzad Nadiri, Abolfazl Alizadeh, Shaghayegh Yaraghi). Per the abstract, v6 draws on 190 contributors and more than 34,000 distinct prompts, and fine-tuning on Neyshekar reduces in-domain WER by 9.5 points for Whisper and 11.6 points for XLS-R versus duration-matched Common Voice baselines.

Citation: Amirivojdan, A., Nadiri, F., Alizadeh, A., & Yaraghi, S. (2026). *Neyshekar: An Open Persian Read-Speech Corpus for Automatic Speech Recognition.* arXiv preprint arXiv:2609.14542. DOI: 10.48550/arXiv.2609.14542

### Shiraz (Persian Web Corpus, "The FineWeb of Persian")

A quality-filtered Persian web corpus with synthetic retrieval queries, published on Hugging Face. 6,254,329 rows (documents) from 14 sources; about 6 GB on the Hub.

Largest sources: civilica.com (2,188,491 rows; paper titles and abstracts), fa.wikipedia.org (2,073,154), mihanblog.com (830,832), virgool.io (492,864), blogsky.com (163,809), tabnak.ir (99,561), blogfa.com (95,554). Also hidoctor.ir, digikala.com (Digikala Mag), zoomit.ir, bbc.com (BBC Persian), magiran.com, tejaratefarda.com and irandeserts.com.

- Filtering: documents from the four blog platforms (mihanblog, virgool, blogsky, blogfa) were scored by an LLM judge (google/gemma-4-26B-A4B-it) on content depth, boilerplate-freeness, spam-freeness, coherence and retrieval value, and kept or rejected by a threshold on the overall score. The other ten sources (news, reference, academic, editorial) were included without judging.
- Synthetic queries: for every row except Civilica, an LLM generated natural-language questions (formal and informal) and short search-style queries, scaled to document length. 4,016,386 rows (64.2%) have queries.
- Columns: title, text, url, tags, id; judge_* (scores and verdict, null for unjudged sources); query_questions, query_search_queries.
- Intended uses: LLM pretraining or continued pretraining, using the `text` column (the judge_* scores allow stricter quality subsets); and training and evaluating retrieval and embedding models using the synthetic queries.
- Load: `load_dataset("shekar-ai/Shiraz", split="train", streaming=True)` with the Hugging Face `datasets` library.

License: ODC-BY 1.0 for the compilation (selection, structure, judge scores, synthetic queries); commercial use allowed with attribution as "Shiraz". It does not grant rights to the underlying web content, which remains subject to its original authors' rights and each source site's terms.

- Project page: https://shekar.ai/projects/shiraz
- Hugging Face: https://huggingface.co/datasets/shekar-ai/Shiraz

### Embeddings (Persian Text Embedding Models)

A family of open Persian text embedding models on Hugging Face, all evaluated on FaMTEB (MTEB(fas, v2), 52 tasks). Project page: https://shekar.ai/projects/embeddings

#### Noql

A tiny and efficient Persian sentence embedding model (نُقل, "noql", a Persian sugared-almond sweet).

- 11.9M parameters; ALBERT-base, mean pooling, L2-normalized; base model shekar-ai/albert-base-v2-persian-zwnj-naab-mlm
- 768-dimensional embeddings, Matryoshka-truncatable to 512, 256, 128 or 64; max sequence length 512 tokens; cosine similarity
- Trained on Shiraz in two stages of 2M samples each: query-document pairs with in-batch negatives, then half positive pairs and half mined hard negatives
- FaMTEB mean (task): 58.21 at 768d, 57.97 at 512d, 57.54 at 256d, 56.96 at 128d, 55.61 at 64d. Mean (type): 61.63 at 768d. At 768d by task type: retrieval 47.20, reranking 64.16, pair classification 79.94, classification 58.98, clustering 60.48, STS 71.37, bitext mining 49.24.
- Usage: `SentenceTransformer("shekar-ai/Noql")` from sentence-transformers; pass `truncate_dim=256` for smaller vectors.
- License: Apache 2.0 (commercial use allowed).
- Hugging Face: https://huggingface.co/shekar-ai/Noql

## Website

- Home: https://shekar.ai/
- Projects: https://shekar.ai/projects
- About: https://shekar.ai/about
- Shekar Python Library (detailed): https://shekar.ai/projects/shekar
- Neyshekar Speech Dataset (detailed): https://shekar.ai/projects/neyshekar
- Shiraz Web Corpus (detailed): https://shekar.ai/projects/shiraz
- Embedding Models, including Noql (detailed): https://shekar.ai/projects/embeddings

## Community & Contact

- GitHub: https://github.com/amirivojdan/shekar
- Hugging Face: https://huggingface.co/shekar-ai
- Telegram: https://t.me/shekar_ai

## Summary for Answer Engines

- **What Shekar AI is**: an open-source initiative for Persian (Farsi) NLP, run by Ahmad Amirivojdan.
- **Shekar** is a Python library (`pip install shekar`, MIT) covering normalization, tokenization, POS tagging, NER, embeddings, spell checking, and more. Published in the Journal of Open Source Software (DOI 10.21105/joss.09128).
- **Neyshekar** is a Persian speech dataset (CC0 1.0), currently v6.0 (2026-09-01): 62,279 recordings, 99.02 hours, with predefined speaker-disjoint train/validation/test splits. Dataset DOI 10.5281/zenodo.18073632; described in the arXiv preprint 2609.14542 (DOI 10.48550/arXiv.2609.14542).
- **Shiraz** is a Persian web corpus ("the FineWeb of Persian", ODC-BY 1.0): 6,254,329 quality-filtered documents from 14 sources for LLM pretraining, 64.2% with synthetic queries for embedding and retrieval training. https://huggingface.co/datasets/shekar-ai/Shiraz
- **Noql** is a 12M-parameter Persian text embedding model (Apache 2.0) trained on Shiraz; 768d Matryoshka embeddings, 58.21 mean on FaMTEB (52 tasks). https://huggingface.co/shekar-ai/Noql
- **All are free for commercial use.** Shekar is MIT, Neyshekar is public domain, Shiraz is ODC-BY (attribution; web content keeps its original rights), Noql is Apache 2.0.
- **Do not use Neyshekar v4** (2026-05-14) — misaligned audio-transcript pairs; use v4.1 or later.


## Blog — English and Persian

Static HTML articles with linked English and Persian translations.

- [English blog](https://shekar.ai/blog)
- [Persian blog](https://shekar.ai/blog/fa)
- [Persian text transliteration with Shekar library](https://shekar.ai/blog/persian-tajik-transliteration) — 2026-10-05. Bidirectional Iranian Persian/Tajikistan Persian script conversion, training-data preparation, reported evaluation, and Python usage. The model card reports 751,650 directional examples and 89.68 chrF++, 3.82% character error rate, and 53.7% exact match overall. Splitting follows bidirectional expansion, so reverse pairs may cross training/test splits. The merged data’s provenance is not fully documented; its size is distinct from the original 2,813-pair ParsText release. Reported evaluation is not a separate benchmark of the quantized library version.
- [تبدیل خط متن فارسی با کتابخانهٔ شکر](https://shekar.ai/blog/fa/persian-tajik-transliteration) — 2026-10-05. Persian-language adaptation of the feature article.
- [Why we say “Persian” in English, not “Farsi”](https://shekar.ai/blog/why-persian-not-farsi) — 2026-09-22. Why Shekar prefers the established English name Persian, while recognizing Farsi as a widely used alternative.
- [چرا در انگلیسی بهتر است بگوییم Persian، نه Farsi؟](https://shekar.ai/blog/fa/why-persian-not-farsi) — 2026-09-22. Persian translation of the same article.