Where is block-level tokenizer and partial index algorithm

[Avv22]

Hello Dear Authors,

Can you please locate where partial index algorithm and sub-block level indexing is in your project? Also where I can find GTP to sort tokens?

Thank you.

[nhjclxc]

Hello Dear Authors,

Can you please locate where partial index algorithm and sub-block level indexing is in your project? Also where I can find GTP to sort tokens?

Thank you.

Hello, do you now know the use of block-level tokenizer?