google / sentencepiece

Watch 107
Star 4.3k
Fork 566

Code
Issues 36
Pull requests 2
Actions
Projects
Security
Insights

Code
Issues
Pull requests
Actions
Projects
Security
Insights

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Pick a username

Email Address

Password

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

36 Open 312 Closed

Author

Filter by author

Label

Filter by label

Use alt + click/return to exclude labels.

Projects

Filter by project

Milestones

Filter by milestone

Assignee

Filter by who’s assigned

Sort

Sort by

Newest Oldest Most commented Least commented Recently updated Least recently updated

Most reactions

Option to set log level. (identity normalization is really verbose)

#543 opened Sep 17, 2020 by villmow

reproducibility of subword regularization

#540 opened Sep 14, 2020 by Zenglinxiao

Cannot parse --input_sentence_size when the given value exceeds int32 maximum value

#538 opened Sep 5, 2020 by szhengac

[Feature request] Is there a way to input a file for the custom and control symbols?

#536 opened Sep 1, 2020 by alvations

How to use BPE-Dropout?

#535 opened Aug 28, 2020 by samin9796

Confusion with SP vocabularies

#532 opened Aug 21, 2020 by mjpost

Vocabulary generation no longer works?

#531 opened Aug 21, 2020 by mjpost

train tokenization model with or without punctuations?

#529 opened Aug 19, 2020 by DRosemei

--escape_whitespaces=false RuntimeError

#526 opened Jul 29, 2020 by Yunlongs

Could we don't consider the whitespace during the training phase.

#524 opened Jul 29, 2020 by Yunlongs

Possible to port underlying model to Java?

#523 opened Jul 27, 2020 by humzaiqbal

How to create new model file with restricted vocabulary?

#522 opened Jul 23, 2020 by sshleifer

/sentencepiece_1581963804834/work/src/trainer_interface.cc(336) [!sentences_.empty()]

#517 opened Jul 3, 2020 by puraminy

Leftover "▁" characters after spm_decode

#514 opened Jun 24, 2020 by lefterav

undefined symbol: FLAGS_minloglevel duplicate

#513 opened Jun 24, 2020 by lefterav

compilation issues

#512 opened Jun 24, 2020 by jwijffels

Is there any way to prevent the merge of specific tokens?

#509 opened Jun 20, 2020 by Saltychtao

`sentencepiece==0.1.92` seems breaking something

#505 opened Jun 9, 2020 by boy2000-007man

Mac whl has mix of Mach-O and ELF binaries

#504 opened Jun 8, 2020 by jhalexand

Using --minloglevel argument in v0.1.90+ causes an error

#492 opened May 22, 2020 by jasonnance

Batch Processing

#486 opened May 13, 2020 by kak-to-tak

Specify protobuf version when compiling from source protobuf

#481 opened Apr 27, 2020 by jchwenger

Word based algorithm in SentencePiece is not being able to preserve some character in Devanagiri

#442 opened Jan 15, 2020 by sanjibnarzary

issue using tf-sentencepiece with tf2 tf.data.Datasets execution environment

#430 opened Dec 9, 2019 by terrykong

Slow computing time using BPE+Seq2Seq model

#420 opened Nov 11, 2019 by Lvby

Previous 1 2 Next

Previous Next

ProTip! What’s not been updated in a month: updated:<2020-08-18.