An implementation of model parallel GPT-2 and GPT-3-style models using the mesh-tensorflow library.
-
Updated
Feb 25, 2022 - Python
Similar to other notes, need "tl;dr" notes for GPT-3 paper. Please use TEMPLATE.md format and follow instructions on README.md
Add a description, image, and links to the gpt-3 topic page so that developers can more easily learn about it.
To associate your repository with the gpt-3 topic, visit your repo's landing page and select "manage topics."
Describe the bug
When running
tools/preprocess_data.pyto tokenize my dataset, I was confused why the generated.binand.idxfiles were empty. It turns out thatlm_dataformat, the library which actually reads the dataset into the tokenization logic, was version 0.0.19 as specified in the requirements.txt file. This version of the library doesn't include support for uncompressed `.jso