Regarding Placeholder and other special tags ... is there any documentation on what these tags do so far?

#51
by jaey-p - opened

i remember seeing something similar with a couple hundred control tags on mistral large ... what're these for, any idea? I genuinely couldn't find an iota of info on this. does anyone have any clue about what these might be? Are there any of them suited for reasoning and thinking tasks?

{
            "id": 128795,
            "content": "<|place▁holder▁no▁795|>",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false,
            "special": true
        },
        {
            "id": 128796,
            "content": "<|place▁holder▁no▁796|>",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false,
            "special": true
        },
        {
            "id": 128797,
            "content": "<|place▁holder▁no▁797|>",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false,
            "special": true
        },
        {
            "id": 128798,
            "content": "<|place▁holder▁no▁798|>",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false,
            "special": true
        },
        {
            "id": 128799,
            "content": "<|place▁holder▁no▁799|>",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": false,
            "special": true
        },
        {
            "id": 128800,
            "content": "<|fim▁hole|>",
            "single_word": false,
            "lstrip": false,
            "rstrip": false,
            "normalized": true,
            "special": false
        },

Sign up or log in to comment