Skip to content

Commit 53364d3

Browse files
authored
Add installation instructions to README (#148)
Added installation instructions for building from source.
1 parent 572e407 commit 53364d3

File tree

1 file changed

+7
-0
lines changed

1 file changed

+7
-0
lines changed

README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,13 @@
11
# tokenizers
22
C++ implementations for various tokenizers (sentencepiece, tiktoken etc). Useful for other PyTorch repos such as torchchat, ExecuTorch to build LLM runners using ExecuTorch stack or AOT Inductor stack.
33

4+
## Installation (from source)
5+
```
6+
git clone [email protected]:meta-pytorch/tokenizers.git
7+
cd ~/tokenizers
8+
git submodule update --init --recursive
9+
pip install -e .
10+
```
411

512
## SentencePiece tokenizer
613
Depend on https://github.com/google/sentencepiece from Google.

0 commit comments

Comments
 (0)