google / sentencepiece

Watch 104
Star 4.3k
Fork 552

Code
Issues 32
Pull requests
Actions
Projects
Security
Insights

Code
Issues
Pull requests
Actions
Projects
Security
Insights

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Pick a username

Email Address

Password

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

32 Open 307 Closed

Author

Filter by author

Label

Filter by label

Use alt + click/return to exclude labels.

Projects

Filter by project

Milestones

Filter by milestone

Assignee

Filter by who’s assigned

Sort

Sort by

Newest Oldest Most commented Least commented Recently updated Least recently updated

Most reactions

train tokenization model with or without punctuations?

#529 opened Aug 19, 2020 by DRosemei

Syd:aloc keeps crashing

#528 opened Aug 12, 2020 by etetteh

--escape_whitespaces=false RuntimeError

#526 opened Jul 29, 2020 by Yunlongs

Could we don't consider the whitespace during the training phase.

#524 opened Jul 29, 2020 by Yunlongs

Possible to port underlying model to Java?

#523 opened Jul 27, 2020 by humzaiqbal

How to create new model file with restricted vocabulary?

#522 opened Jul 23, 2020 by sshleifer

/sentencepiece_1581963804834/work/src/trainer_interface.cc(336) [!sentences_.empty()]

#517 opened Jul 3, 2020 by puraminy

My training crashes with large corpus.

#516 opened Jun 27, 2020 by Srj

Leftover "▁" characters after spm_decode

#514 opened Jun 24, 2020 by lefterav

undefined symbol: FLAGS_minloglevel duplicate

#513 opened Jun 24, 2020 by lefterav

compilation issues

#512 opened Jun 24, 2020 by jwijffels

Is there any way to prevent the merge of specific tokens?

#509 opened Jun 20, 2020 by Saltychtao

`sentencepiece==0.1.92` seems breaking something

#505 opened Jun 9, 2020 by boy2000-007man

Mac whl has mix of Mach-O and ELF binaries

#504 opened Jun 8, 2020 by jhalexand

Deterministic sentencepiece model

#496 opened May 26, 2020 by alvations

Using --minloglevel argument in v0.1.90+ causes an error

#492 opened May 22, 2020 by jasonnance

Batch Processing

#486 opened May 13, 2020 by kak-to-tak

Specify protobuf version when compiling from source protobuf

#481 opened Apr 27, 2020 by jchwenger

Word based algorithm in SentencePiece is not being able to preserve some character in Devanagiri

#442 opened Jan 15, 2020 by sanjibnarzary

issue using tf-sentencepiece with tf2 tf.data.Datasets execution environment

#430 opened Dec 9, 2019 by terrykong

Slow computing time using BPE+Seq2Seq model

#420 opened Nov 11, 2019 by Lvby

Feature Request: Python Executables for training, encoding and decoding

#385 opened Aug 26, 2019 by bricksdont

sentencepiece and tensorflow protoc conflict protobuf

#383 opened Aug 23, 2019 by gobrewers14

How to create a static version of sentencepiece_processor_ops?

#369 opened Jul 27, 2019 by bsod90

can we train by Parallel Computing or Multithreading or multi-Progress

#366 opened Jul 12, 2019 by joytianya

Previous 1 2 Next

Previous Next

ProTip! Follow long discussions with comments:>50.