Unable to reproduce the results

[MatthewCYM]

I tried to reproduce the results on GLUE benchmark. I got F1 score 66 on MPRC, which is much lower than the one reported in the paper(78). I also got accuracy 77 on SST-2. which is also abnormal.

Running script:

python ./run_glue_discrete_LM.py \
--task_name=mrpc \
--per_device_train_batch_size 16 \
--per_device_eval_batch_size 16 \
--weight_decay=0.1 --seed=42 \
--k_shot 16 --prompt_learning_rate 1e-4 \
--sample_size 20 --prompt_length 10 \
--prompt_search_space 200 \
--api_limit 8000 --ce_loss True

May I ask how to fix it?

[shizhediao]

Hi,

Thank you for your interest in our work!
It is caused by different hyper-parameters. Here is a set of hyper-parameters for MRPC in our experiments for your reference:

python ./run_glue_discrete_LM.py \
--task_name=mrpc \
--per_device_train_batch_size 16 \
--per_device_eval_batch_size 16 \
--weight_decay=0.1 --seed=42 \
--k_shot 16 --prompt_learning_rate 1e-3 \
--sample_size 30 --prompt_length 20 \
--prompt_search_space 100 \
--api_limit 8000 --ce_loss True

I obtain the results as follows:
Black_test_f1 = 79.52

Similarly, you could tune the hyper-params for SST-2 for a reasonable performance.

[MatthewCYM]

Thanks for your quick reply.

I try the provided hyperparameters. It will give below error:

Traceback (most recent call last):
  File "/mnt/disk1/Black-Box-Prompt-Learning/./run_glue_discrete_LM.py", line 823, in <module>
    main()
  File "/mnt/disk1/Black-Box-Prompt-Learning/./run_glue_discrete_LM.py", line 568, in main
    prompts_discrete_indices = prompts_dist.sample()
  File "/mnt/disk1/anaconda3/envs/bdpl/lib/python3.9/site-packages/torch/distributions/categorical.py", line 107, in sample
    samples_2d = torch.multinomial(probs_2d, sample_shape.numel(), True).T
RuntimeError: invalid multinomial distribution (encountering probability entry < 0)

I also find that the eval f1 becomes lower during the training process:

epoch 0: {'accuracy': 0.46875, 'f1': 0.6046511627906976}
epoch 1: {'accuracy': 0.46875, 'f1': 0.5853658536585366}
epoch 2: {'accuracy': 0.40625, 'f1': 0.4864864864864864}
epoch 3: {'accuracy': 0.34375, 'f1': 0.39999999999999997}

Since the provided pmi vocab download link is invalid. I generate the pmi vocab through

python pmi_ngram.py --dataset glue_data/MRPC/msr_paraphrase_train.txt --output_dir pmi/mrpc.txt

with glue data download from link.

May I ask if the pmi vocab is wrong?

[shizhediao]

Hi,
Sorry for my late reply. I was a bit swamped these days (ICLR attendance, NeurIPS...).
It is weird to see the error:
RuntimeError: invalid multinomial distribution (encountering probability entry < 0)
First of all, I think we need to solve this issue. I never see such a message.

I have updated the pmi vocab. Could you please download it again?

Thanks!

[MatthewCYM]

Hi @shizhediao ,
The pmi vocab at https://drive.google.com/file/d/1Yk6JSYKXrG356o8rs2dtMTOIBJgSJOGE/view?usp=share_link still doesn't exist.

[shizhediao]

Hi I uploaded it again to the Google Drive. Could you try this link?

Thanks!

[MatthewCYM]

Hi @shizhediao ,

I try the provided pmi vocab. It still gives the same error.

Thanks!

[shizhediao]

Could you provide more log information?
I checked with the error message (e.g., https://stackoverflow.com/questions/57627943/invalid-multinomial-distribution-encountering-probability-entry-0-at-pytorc) and doubted that it was caused by NaN value.
In our experiments, we sometimes observed NaN because of unstable training and inappropriate hyperparameters.

[MatthewCYM]

Hi, it's indeed caused by nan value. Do I need to tune the hyperparameters by myself? What's the range of each hyperparameter?

[shizhediao]

Hi
Although I did not know why the same hyperparameters do not work well on another machine, it would be good to tune the hyperparameters. At least, we should avoid training instability.
Here are some essential hyperparameters:
sample_size: >10 the larger the better
prompt_length: [0, 100]
prompt_learning_rate: [1e-6, 1e-4]

[ZiruiYan]

Hi @shizhediao,

Thanks for the well-documented codes. Currently, I'm experiencing the same issue as @MatthewCYM, where I'm encountering abnormal performance on several datasets. Specifically, I'm observing lower performance on MNLI (26.8 vs. 42.5), SST-2 (76.5 vs. 87.6), MPRC (68.8 vs. 78.1), and CoLA (-0.9 vs. 4.6) when using the hyperparameters mentioned in Table 6 in Appendix A.4. I'm curious if this discrepancy is due to differences in the environment or different hyperparameters being used. I noticed that you're using a batch size of 16 for the MRPC dataset, whereas Table 5 specifies a batch size of 32.

For the environment concern, when I install the environment using the command source install.sh, the version of the datasets package that gets installed is 2.6.1. However, running the code with this version throws an error:

File "./Black-Box-Prompt-Learning-main/run_glue_discrete_LM.py", line 260, in main
    raw_datasets = load_dataset("glue", args.task_name)
………………
TypeError: can only concatenate str (not "int") to str

After upgrade datasets package to version 2.12.0, I can run the code, but I encounter the same NaN error with the following command:

python ./run_glue_discrete_LM.py \
--task_name=mrpc \
--per_device_train_batch_size 16 \
--per_device_eval_batch_size 16 \
--weight_decay=0.1 --seed=42 \
--k_shot 16 --prompt_learning_rate 1e-3 \
--sample_size 30 --prompt_length 20 \
--prompt_search_space 100 \
--api_limit 8000 --ce_loss True

Could you provide more information about the environment you are using? In my case, the environment looks like this

# Name                    Version                   Build  Channel
datasets                  2.6.1                    pypi_0    pypi
huggingface-hub           0.15.1                   pypi_0    pypii
torch                     1.7.1                    pypi_0    pypi
torchaudio                0.7.2                    pypi_0    pypi
torchvision               0.8.2                    pypi_0    pypi

Additionally, if possible, could you provide a sample run for other datasets such as MNLI, SST-2, and CoLA?

Lastly, if @MatthewCYM happens to solve this problem, please let me know! Thanks in advance!

[shizhediao]

Hi,
We have uploaded some of the trained discrete prompts based on RoBERTa to Google Drive for your reference.
In addition, we provided the training hyper-parameters in the file name.
Please take a look.

Thanks!

[yanzhi0922]

Hello, I have encountered the same issue as them. After using the parameters you provided above, I also encountered the NaN issue.

[yanzhi0922]

Hi, after solving NaN, I tried the hyperparameters you gave above and got a result of only 0.577 for F1, which is much lower than the 0.792 you gave, which is unacceptable

[YerongLi]

Hi, with the original parameters, none of the four datasets has a declining average loss.

with the suggested hyperparamters suggested above #3 (comment), I encountered NaN

12/18/2025 18:18:18 - INFO - __main__ -   Total optimization steps = 60
----Optimizing black-box prompts----
loss_avg: 0.8742807408173879
  2%|███████▎                                                                                                                                                                                                                                                                                                                                                                                                                                               | 1/60 [00:06<05:54,  6.00s/it]loss_avg: 0.6392360587914785
  3%|██████████████▋                                                                                                                                                                                                                                                                                                                                                                                                                                        | 2/60 [00:12<06:18,  6.53s/it]12/18/2025 18:18:32 - INFO - __main__ - ** eval **
12/18/2025 18:18:32 - INFO - __main__ - epoch 0: {'accuracy': 0.34375, 'f1': 0.43243243243243246}
loss_avg: 0.7740660786628724
  5%|█████████████████████▉                                                                                                                                                                                                                                                                                                                                                                                                                                 | 3/60 [00:19<06:09,  6.47s/it]train_api_request has been used: 100x
loss_avg: 0.7285062491893768
  7%|█████████████████████████████▎                                                                                                                                                                                                                                                                                                                                                                                                                         | 4/60 [00:26<06:12,  6.64s/it]12/18/2025 18:18:45 - INFO - __main__ - ** eval **
12/18/2025 18:18:45 - INFO - __main__ - epoch 1: {'accuracy': 0.34375, 'f1': 0.43243243243243246}
loss_avg: 0.7618658542633057
  8%|████████████████████████████████████▌                                                                                                                                                                                                                                                                                                                                                                                                                  | 5/60 [00:33<06:18,  6.88s/it]loss_avg: 0.7429617325464885
 10%|███████████████████████████████████████████▉                                                                                                                                                                                                                                                                                                                                                                                                           | 6/60 [00:39<05:55,  6.58s/it]12/18/2025 18:18:58 - INFO - __main__ - ** eval **
12/18/2025 18:18:58 - INFO - __main__ - epoch 2: {'accuracy': 0.40625, 'f1': 0.5365853658536586}
train_api_request has been used: 200x
loss_avg: 0.8022278189659119
 12%|███████████████████████████████████████████████████▏                                                                                                                                                                                                                                                                                                                                                                                                   | 7/60 [00:45<05:31,  6.25s/it]loss_avg: 0.697675637404124
 13%|██████████████████████████████████████████████████████████▌                                                                                                                                                                                                                                                                                                                                                                                            | 8/60 [00:51<05:35,  6.45s/it]12/18/2025 18:19:11 - INFO - __main__ - ** eval **
12/18/2025 18:19:11 - INFO - __main__ - epoch 3: {'accuracy': 0.375, 'f1': 0.47368421052631576}
loss_avg: 0.7142187615235647
 15%|█████████████████████████████████████████████████████████████████▊                                                                                                                                                                                                                                                                                                                                                                                     | 9/60 [00:59<05:42,  6.71s/it]train_api_request has been used: 300x
loss_avg: 0.8045172293980917
 17%|█████████████████████████████████████████████████████████████████████████                                                                                                                                                                                                                                                                                                                                                                             | 10/60 [01:04<05:20,  6.41s/it]12/18/2025 18:19:24 - INFO - __main__ - ** eval **
12/18/2025 18:19:24 - INFO - __main__ - epoch 4: {'accuracy': 0.4375, 'f1': 0.5909090909090909}
loss_avg: 0.8116302450497945
 18%|████████████████████████████████████████████████████████████████████████████████▎                                                                                                                                                                                                                                                                                                                                                                     | 11/60 [01:12<05:27,  6.68s/it]loss_avg: 0.688676389058431
 20%|███████████████████████████████████████████████████████████████████████████████████████▌                                                                                                                                                                                                                                                                                                                                                              | 12/60 [01:18<05:07,  6.40s/it]12/18/2025 18:19:37 - INFO - __main__ - ** eval **
12/18/2025 18:19:37 - INFO - __main__ - epoch 5: {'accuracy': 0.4375, 'f1': 0.5714285714285714}
loss_avg: 0.7844806551933289
 22%|██████████████████████████████████████████████████████████████████████████████████████████████▉                                                                                                                                                                                                                                                                                                                                                       | 13/60 [01:25<05:13,  6.66s/it]train_api_request has been used: 400x
loss_avg: 0.7259239137172699
 23%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏                                                                                                                                                                                                                                                                                                                                               | 14/60 [01:31<04:57,  6.46s/it]12/18/2025 18:19:50 - INFO - __main__ - ** eval **
12/18/2025 18:19:50 - INFO - __main__ - epoch 6: {'accuracy': 0.4375, 'f1': 0.5714285714285714}
loss_avg: 0.6763239979743958
 25%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                                                                                                                                                                                                                                                                                        | 15/60 [01:37<04:43,  6.30s/it]loss_avg: 0.8287131647268932
 27%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                                                                                                                                                                                                                                                                                                                 | 16/60 [01:44<04:45,  6.48s/it]12/18/2025 18:20:03 - INFO - __main__ - ** eval **
12/18/2025 18:20:03 - INFO - __main__ - epoch 7: {'accuracy': 0.4375, 'f1': 0.55}
train_api_request has been used: 500x
loss_avg: 0.7573286533355713
 28%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                                                                                                                                                                                                                                                                                                                          | 17/60 [01:50<04:37,  6.46s/it]loss_avg: 0.7426965991655986
 30%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                                                                                                                                                                                                                                                                                                  | 18/60 [01:57<04:37,  6.60s/it]12/18/2025 18:20:16 - INFO - __main__ - ** eval **
12/18/2025 18:20:16 - INFO - __main__ - epoch 8: {'accuracy': 0.375, 'f1': 0.47368421052631576}
loss_avg: 0.6499031484127045
 32%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                                                                                                                                                                                                                                                                                           | 19/60 [02:03<04:28,  6.54s/it]Traceback (most recent call last):
  File "/data/coding/Black-Box-Prompt-Learning/./run_glue_discrete_LM.py", line 902, in <module>
    main()
  File "/data/coding/Black-Box-Prompt-Learning/./run_glue_discrete_LM.py", line 615, in main
    prompts_dist = torch.distributions.Categorical(prompts_probs)
  File "/data/miniconda/envs/torch/lib/python3.10/site-packages/torch/distributions/categorical.py", line 73, in __init__
    super().__init__(batch_shape, validate_args=validate_args)
  File "/data/miniconda/envs/torch/lib/python3.10/site-packages/torch/distributions/distribution.py", line 72, in __init__
    raise ValueError(
ValueError: Expected parameter probs (Tensor of shape (20, 100)) of distribution Categorical(probs: torch.Size([20, 100])) to satisfy the constraint Simplex(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<DivBackward0>)
Traceback (most recent call last):
  File "/data/coding/Black-Box-Prompt-Learning/./run_glue_discrete_LM.py", line 902, in <module>
    main()
  File "/data/coding/Black-Box-Prompt-Learning/./run_glue_discrete_LM.py", line 615, in main
    prompts_dist = torch.distributions.Categorical(prompts_probs)
  File "/data/miniconda/envs/torch/lib/python3.10/site-packages/torch/distributions/categorical.py", line 73, in __init__
    super().__init__(batch_shape, validate_args=validate_args)

Could you provide the hyperparameters for each one of the four datasets, with which your losses are actually declining