Skip to content

Pull requests: huggingface/tokenizers

New pull request New
29 Open 757 Closed
29 Open 757 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or 鈬� + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who鈥檚 assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
馃憤 馃憥 馃槃 馃帀 馃槙 鉂わ笍 馃殌 馃憖

Pull requests list

feat: add progress_format option for machine-readable JSON output
#1921 opened Dec 26, 2025 by podarok Loading鈥�
6 tasks done
1
Upgrade GitHub Actions for Node 24 compatibility
#1916 opened Dec 20, 2025 by salmanmkc Loading鈥�
Use unicode-normalization instead of unicode-normalization-alignments
#1912 opened Dec 14, 2025 by IvanIsCoding Loading鈥�
Add windows arm64 wheel build to python release
#1907 opened Dec 8, 2025 by finnagin Loading鈥�
1
2
Fix undefined names in docs/source/_ext/entities.py
#1895 opened Nov 28, 2025 by cclauss Loading鈥�
1
Python: Add ruff rules for asyncio and performance
#1894 opened Nov 28, 2025 by cclauss Loading鈥�
3
Implement Append normalizer
#1893 opened Nov 28, 2025 by ArthurZucker Loading鈥�
1
C and C++ bindings to Tokenizers Feature Request
#1888 opened Nov 21, 2025 by thammegowda Loading鈥�
8
Providing byte level offsets for effective alignment in Cross-Tokenizer On-Policy Distillation Feature Request
#1880 opened Oct 30, 2025 by JqzChandler Loading鈥�
5
Mark Python tests that need network access
#1872 opened Oct 2, 2025 by gordonmessmer Loading鈥�
2
Add a multithreaded tokenizer test and as well as 3.14 and 3.14t CI
#1864 opened Sep 12, 2025 by ngoldbaum Loading鈥�
4
feat: allow BPETrainer to be seeded with a set of initial tokens
#1862 opened Sep 6, 2025 by henrycharlesworth Loading鈥�
Fix unsigned integer underflow issue with truncation
#1859 opened Sep 1, 2025 by maxdebayser Loading鈥�
5
feat: add cli for tokenizer and training Feature Request
#1842 opened Aug 6, 2025 by b00f Loading鈥�
6
feat: whitespace optimize Feature Request
#1841 opened Aug 6, 2025 by b00f Loading鈥�
1
1
Unused Unicode Character Filter
#1832 opened Jul 23, 2025 by sanderland Loading鈥�
2
Add enforce_utf8_boundaries option to BpeTrainer
#1830 opened Jul 22, 2025 by sanderland Loading鈥�
10
Faster Whitespace PreTokenizer (Drop-in Replacement)
#1822 opened Jul 7, 2025 by 8ria Loading鈥�
1
Adding multiprocessing for sentencepiece_extractor
#1804 opened Jun 19, 2025 by AamodThakur Loading鈥�
Expose Encoding attributes via the buffer protocol interface
#1789 opened Jun 4, 2025 by mariosasko Loading鈥�
3
add group capture to replace Feature Request
#1788 opened Jun 3, 2025 by cboseak Loading鈥�
1
1
Add Truncate pre-tokenizer
#1783 opened May 27, 2025 by ArthurZucker • Draft
Make unigram cache optional
#1763 opened Apr 18, 2025 by wangrunji0408 Loading鈥�
3
Implement Append normalizer
#1755 opened Mar 24, 2025 by austinleedavis Loading鈥�
4
Pre-tokenizers that support multi-word/non-whitespace BPE in single pass
#1753 opened Mar 22, 2025 by mjbommar Loading鈥�
2
ProTip! Mix and match filters to narrow down what you鈥檙e looking for.