Skip to content

Issues: huggingface/tokenizers

ByteLevelBPETokenizer output seems weird
#203 opened Mar 24, 2020 by seyyaw
Open 2
Training a model from in-memory data
#198 by loicbarrault was closed Nov 28, 2020
Closed 1
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

224 Open 503 Closed
224 Open 503 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

hyper CVE-2022-31394
#1185 opened Mar 20, 2023 by crbean
CVE-2023-22466 vuln of the component tokio which is used to compiled in tokenizers.
#1184 opened Mar 20, 2023 by wqh17101
3
NodeJS: Can't initialize BertWordPiece using fromFile or init from vocabFile
#1180 opened Mar 15, 2023 by douglasqian
3
error LNK2038
#1178 opened Mar 11, 2023 by qdwang
1
Parallelism Implementation Question
#1177 opened Mar 8, 2023 by mert-kurttutan
2
Feature request: list-like behaviour for Sequence good first issue Good for newcomers
#1175 opened Mar 8, 2023 by davidgilbertson
1
add_prefix_space adds a space after added tokens
#1174 opened Mar 8, 2023 by davidgilbertson
1
GPT-2 tokeniser's decoder is incorrect and doesn't roundtrip
#1164 opened Feb 21, 2023 by hauntsaninja
1
Adding words to vocabulary and training for a new domain
#1160 opened Feb 9, 2023 by PeterM18
5
text is listed as optional in api but tokenizer throws error when missing
#1159 opened Feb 4, 2023 by epinnock
Support for incremental decoding
#1141 opened Jan 4, 2023 by njhill
4
How to porting subword-nmt to tokenizers?
#1138 opened Dec 29, 2022 by wannaphong
2
Can't import any modules
#1128 opened Dec 22, 2022 by kronkinatorix
1
How to decode with the existing tokenizer
#1118 opened Dec 6, 2022 by ZhiYuanZeng
4
C++ binding
#1117 opened Dec 6, 2022 by jiaqianjing
1
Is there any support for 'google/tapas-mini-finetuned-wtq' tokenizer?
#1114 opened Nov 26, 2022 by memetrusidovski
4
OpenSSL internal error when importing tokenizers module
#1113 opened Nov 21, 2022 by wai25
3
Adding treat_whitespace_as_suffix as a new feature to sentencepiece?
#1112 opened Nov 20, 2022 by Smu-Tan
1
NormalizedString append method failed after calling clear
#1110 opened Nov 17, 2022 by yanyongyu
6
How can I keep the initial input vocab and incremental add the new tokens during re-training a tokenizer?
#1109 opened Nov 17, 2022 by henryxiao1997
1
tokenizers 0.13.2 does not compile when default features are turned off
#1104 opened Nov 10, 2022 by jneuff
2
Difference between PreTrainedTokenizerFast Python and Node SentencePieceBPETokenizer
#1100 opened Nov 8, 2022 by loretoparisi
10
Module not found: Error: Can't resolve '../bin-package' in path
#1094 opened Nov 3, 2022 by zbloss
3
How to implement customized tokenizers in Rust.
#1093 opened Nov 2, 2022 by ucasqsl
3
Can not get package to build with Python 3.11 on a minimal linux environment
#1092 opened Nov 1, 2022 by ZetiMente
3
ProTip! Follow long discussions with comments:>50.