huggingface / tokenizers

Watch 59
Star 2.9k
Fork 179

Code
Issues 48
Pull requests 4
Actions
Projects 0
Security
Insights

Code Issues 48 Pull requests 4 Projects 0 Actions Security Pulse

Pinned issues

Feature Request: Support for sentencepiece unigram model

#53 opened Jan 12, 2020 by leslyarun

Open 6

Tokenizer saving/loading

#15 opened Dec 27, 2019 by n1t0

Open 2

ByteLevelBPETokenizer output seems weird

#203 opened Mar 24, 2020 by seyyaw

Open 2

Labels 12 Milestones 0

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Pick a username

Email Address

Password

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

48 Open 104 Closed

Author

Filter by author

Label

Filter by label

Use alt + click/return to exclude labels.

Projects

Filter by project

Milestones

Filter by milestone

Assignee

Filter by who’s assigned

Sort

Sort by

Newest Oldest Most commented Least commented Recently updated Least recently updated

Most reactions

Is ByteLevelBPETokenizer working with cyrillics?

#268 opened May 17, 2020 by purgenetik

BertWordPieceTokenizer is not reversible for quotation marks

#266 opened May 13, 2020 by tomhosking

Add the WordLevelTrainer

#265 opened May 12, 2020 by n1t0

Automatically Adding SOS/EOS Tokens

#263 opened May 11, 2020 by glennkroegel

Why doesn't this library share the same tokenizer api as the transformers library?

#259 opened May 6, 2020 by sabetAI

Tokenizer stalls / hangs when used in DataLoader (multiprocessing issue)

#258 opened May 5, 2020 by jiahuei

Advice for training BertWordPieceTokenizer multilingual tokenizer?

#257 opened May 5, 2020 by shenkev

Upgrade bin-package/index.node to node v12.16.3

#255 opened Apr 29, 2020 by micah5

XLM tokenizer?

#251 opened Apr 26, 2020 by mkumar10

Any plan to binding to java?

#242 opened Apr 20, 2020 by ericxsun

Difference to Roberta transformers tokenizer

#240 opened Apr 18, 2020 by psinger

how to install tokenizers from conda

#237 opened Apr 16, 2020 by liuwei1206

Whitespace tokenizer for training BERT from scratch

#232 opened Apr 13, 2020 by aqibsaeed

Convert saved pretrained tokenizers from transformers to tokenizers

#230 opened Apr 11, 2020 by NonaryR

NodeJS Build Error

#217 opened Apr 3, 2020 by ayir

tokenizer fails for large dataset

#214 opened Mar 31, 2020 by vr25

Is it possible to convert the tokenized tokens back to sentence?

#211 opened Mar 30, 2020 by Hhhhhhhhhhao

ByteLevelBPETokenizer output seems weird

#203 opened Mar 24, 2020 by seyyaw

BPE dropout not working as expected

#201 opened Mar 19, 2020 by rgdl

Training a model from in-memory data

#198 opened Mar 18, 2020 by loicbarrault

How to use tokenizers library for my own dataset with a mix of existing and new vocabulary

#194 opened Mar 11, 2020 by nikhilno1

python version hangs on encode_batch when run in subprocess

#187 opened Mar 4, 2020 by kretes

C/C++ binding interface

#185 opened Mar 4, 2020 by roman-kruglov

Format for .txt Files for Training BERT WordPiece Tokenizer?

#182 opened Mar 3, 2020 by siddk

ByteLevelBPETokenizer doesn't merge

#164 opened Feb 24, 2020 by apeguero1

Previous 1 2 Next

Previous Next

ProTip! Type g i on any issue or pull request to go back to the issue listing page.