Highlights
- Arctic Code Vault Contributor
Create your own GitHub profile
Sign up for your own profile on GitHub, the best place to host code, manage projects, and build software alongside 50 million developers.
Sign up
Pinned
858 contributions in the last year
Contribution activity
October 2020
Created a pull request in huggingface/transformers that received 1 comment
[Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies
What does this PR do?
Both the SentencePiece and Tokenizers libraries can limit the users:
sentencepiece is not available on Conda on every platef…
- [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies
- Allow Custom Dataset in RAG Retriever
- Replaced torch.load for loading the pretrained vocab of TransformerXL tokenizer to pickle.load
- Fix tokenizer UnboundLocalError when padding is set to PaddingStrategy.MAX_LENGTH
- Allow soft dependencies in the namespace with ImportErrors at use
- Fix local_files_only for TF