UltraLLaDA

We introduce UltraLLaDA , a scaled variant of LLaDA-8B-Base that extends the context length up to 128K tokens with light-weight post-training, enabling long-context comprehension and generation.

Eval

export PYTHONPATH=/path/to/eval/opencompass-0.4.2:$PYTHONPATH && \
export TIKTOKEN_CACHE_DIR=/path/to/eval/tiktoken/tiktoken_cache && \
export COMPASS_DATA_CACHE='/path/to/eval/data/'

• lonbench:

   python3 run.py /path/to/eval/opencompass-0.4.2/opencompass/eval/longbench/16K/eval_llada_long.py --dump-eval-details -r

• NIAH:

  python3 run.py /path/to/eval/opencompass-0.4.2/opencompass/eval/niah/128K/eval_llada_niah.py --dump-eval-details -r

• ruler:

  python3 run.py /path/to/eval/opencompass-0.4.2/opencompass/eval/ruler/32K/eval_llada_ruler.py --dump-eval-details -r