botpress / node-sentencepiece

Quick Start

Prerequisites: Yarn, node and cmake
note : windows users also need Build Tools for Visual Studio package

1. clone the repo using the --recursive arg to fetch google/sentencepiece submodule

2. Run yarn to fetch node packages.

3. Run yarn build to build google/sentencepiece and the node binding

4. step outside the directory:
   >> cd ..

5. run node:
   >> node

6. require node-sentencepiece package
   (node) var sp = require('./node-sentencepiece')

7. instanciate a processor
   (node) var proc = new sp.Processor()

8. load a model
   (node) proc.loadModel('/path/to/model/m.model')

9. use the processor to get tokens (node) proc.encode('Never gonna give you up, Never gonna let you down')

   returns:
   [ '▁', 'N', 'ever', '▁gonna', '▁give', '▁you', '▁up', ',', '▁', 'N', 'ever', '▁gonna', '▁let', '▁you', '▁down' ]

10. you can get back the original input text from token by using the decode method

    (node) var inputText = 'Feel the rain on your skin No one else can feel it for you'

    (node) var proc = new sp.Processor()

    (node) proc.loadModel('/path/to/model/m.model')

    (node) var pieces = proc.encode(inputText)

    (node) var outputText = proc.decode(pieces, modelPath)

    (node) inputText === outpuText

    returns:
    true