Skip to content

Pinned issues

ByteLevelBPETokenizer output seems weird
#203 opened Mar 24, 2020 by seyyaw
Open 2
Training a model from in-memory data
#198 by loicbarrault was closed Nov 28, 2020
Closed
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

169 Open 424 Closed
169 Open 424 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀
Can the lstrip argument of AddedToken be taken into account with an Unigram model?
#959 opened Mar 21, 2022 by SaulLu
2
Failing tests with python 3.10 due to changed error message
#957 opened Mar 21, 2022 by h-vetinari
1
3
Ruby bindings
#955 opened Mar 19, 2022 by ankane
1
4
char_to_token is broken when is_split_into_words is set to True
#948 opened Mar 9, 2022 by zorikg
3
0.11.5 and 0.11.6 packages not compatible with manylinux2010
#946 opened Mar 8, 2022 by vgod-dbx
5
PanicException: assertion failed: stride < max_len when using Question-Answering pipeline
#944 opened Mar 7, 2022 by IsCoelacanth
1
Support wasm
#935 opened Mar 1, 2022 by Narsil
4 tasks
0.13
Upgrade pyo3 version
#934 opened Mar 1, 2022 by Narsil 0.13
1
[Optional] Adding parallelization to Unigram trainer.
#933 opened Mar 1, 2022 by Narsil 0.12 (Bigscience enabled, clean modifications)
Prebuilding tokenizers for Mac M1.
#932 opened Mar 1, 2022 by Narsil 0.13
1
Enabling simpler flow of information from the Tokenizer to the trainer. bug
#930 opened Mar 1, 2022 by Narsil 0.13
@mishig25
Addition of CONTRIBUTING.md to Repository
#926 opened Feb 28, 2022 by beneyal
3
Tokenizers for Node 16?
#911 opened Feb 17, 2022 by etiennelunetta 0.13
1
tokenizers.models.BPE loses whitespace with GPT-2 pretrained vocab & merges
#905 opened Feb 13, 2022 by umbra-scientia
1
Some questions about building a tokenizer from scratch: vocab size can't decide actual vocab size and token order unstable.
#903 opened Feb 9, 2022 by catqaq
11
How to Suppress "Using bos_token, but it is not set yet..." in HuggingFace T5 Tokenizer
#902 opened Feb 8, 2022 by MRGLabs
3
get_vocab_size() is different from len(get_vocab())
#900 opened Feb 2, 2022 by anatoly-khomenko
1
Error: ThreadPoolBuildError
#899 opened Feb 1, 2022 by HarshkumarP
1
Missing Version info in build from source installation instruction
#898 opened Feb 1, 2022 by leo-du
5
Wrong alignments after calling NormalizedString.replace()
#892 opened Jan 26, 2022 by t-yamamura
10
PanicException For Result::unwarp()
#888 opened Jan 21, 2022 by Namco0816
3
vocab_size issue with Whitespace pre_tokenizer
#886 opened Jan 19, 2022 by ctoraman
9
[TBD] add a feature to continue training a tokenizer
#879 opened Jan 14, 2022 by SaulLu
2
Tokenizers | TypeError: not a string
#878 opened Jan 14, 2022 by danielbellhv
7
pyo3_runtime.PanicException: Missing additional token
#876 opened Jan 9, 2022 by chinoll
@Narsil
3
ProTip! Adding no:label will show everything without a label.