'None' values in the dataset preprocessing when reproducing the result on GSM8k

[kulinshah98]

Hi,

Thanks for open sourcing the code! I am trying to reproduce the results on Math (GSM8k) benchmark given in Table 1. I am using the yaml file given at the location "LLaMA-Factory/examples/train_full/gpt2_full_ddm-sft.yaml" and given instruction to run the code. However, I am facing the issue of having "None" values in the input data. To be specific, the batch['input_ids'] in the "LLaMA-Factory/src/llamafactory/train/ddm/workflow.py" seems to have None values between prompt and response (after question and before answer of the GSM8k). Because of this None value, I am getting the error "RuntimeError: Could not infer dtype of NoneType" during the start of the training. Can you help me fix this issue? Thank you in advance!

Best,
Kulin

[summmeer]

Which dataset do you use? If you use the one in DoT, there might be 1 or 2 data that contains an empty string, you can find and remove them.

[pengzhangzhi]

@summmeer Hi I'm curious about the evaluation code on Math (GSM8k) benchmark . It doesn't seems provided in the evaluation folder.

[summmeer]

@pengzhangzhi Hi, since we finetune the DLM to perform the GSM8k task instead of using zero/few-shot settings, please refer to LLaMA-Factory/examples/train_lora/llama2_lora_ddm-sft.yaml to do the training and inference. The score will be computed automatically after the inference. The metric computation code is:

DiffuLLaMA/LLaMA-Factory/src/llamafactory/train/ddm/metric.py

Line 16 in 51b86b4

def gsm_eval(preds, labels):

[pengzhangzhi]

Hi, do you have pretrained model?

[summmeer]

@pengzhangzhi Which model do you refer to? DiffuLLaMA or finetuned DiffuLLaMA in GSM8K data?

[pengzhangzhi]

finetuned! I'm testing my sampling algorithm and seeing if it works on finetuned DiffuLLaMA in GSM8K

[summmeer]

@pengzhangzhi Trying to check if I backup this checkpoint. Will ping you when I upload the weights.

[MhDang]

@pengzhangzhi Trying to check if I backup this checkpoint. Will ping you when I upload the weights.

Hi, thank you for the amazing work on DiffuLLaMA!

I saw the previous discussions regarding the finetuned DiffuLLaMA checkpoint for GSM8K. I just wanted to follow up and check if there are any updates on its availability!

[pengzhangzhi]

not really

[summmeer]

@pengzhangzhi @MhDang Hi, Thank you for waiting. I've uploaded the finetuned version (LoRA adapter) to huggingface. Please check it out. https://huggingface.co/diffusionfamily/diffullama-gsm
Be free to chat if you have any questions!

p.s. if everything goes well, the inference results on gsm test set should be: metric: {'acc': 0.6345716451857468}