Error code: 400 - {'error': {'message': "Sorry! We've encountered an issue with repetitive patterns in your prompt. Please try again with a different prompt.", 'type': 'invalid_request_error', 'param': 'prompt', 'code': 'invalid_prompt'}}

[Tizzzzy]

Hi, currently I am running check_mplug.py on dataset coco val2014. I followed everything on README. At first, it run successful, however, it shows Error code: 400 - {'error': {'message': "Sorry! We've encountered an issue with repetitive patterns in your prompt. Please try again with a different prompt.", 'type': 'invalid_request_error', 'param': 'prompt', 'code': 'invalid_prompt'}} after few minutes. Can you please take a look?
Thank you

[Hyperwjf]

Hi! 👋 Thank you for reporting this issue! 🚀

Regarding the error you're encountering, this is actually the first time I've come across this specific issue. I found that this error typically occurs with GPT-3.5-turbo when a certain threshold is exceeded - specifically, when there are more than 200 repeated words in the prompt. https://community.openai.com/t/repetitive-prompt-error-400/716240

Here are a couple of suggestions that might help resolve the issue:

1. Consider trying a different version of GPT or other LLM API. In our experiment, we have verified that using Vicuna-v1.5-13b to replace GPT-3.5 can achieve comparable performance.
2. Review and modify the prompt to avoid excessive repetition.

Let me know if you need any further assistance!

[Tizzzzy]

Hi,
Thank you for your reply. Is it possible to apply LogicCheckGPT mitigation method to the LLaVA model on the POPE benchmark? The questions in the POPE benchmark are all binary (Yes/No), for example:

{"question_id": 1, "image": "COCO_val2014_000000016631.jpg", "text": "Is there a person in the image?", "label": "yes"}
{"question_id": 2, "image": "COCO_val2014_000000016631.jpg", "text": "Is there a refrigerator in the image?", "label": "no"}

Is it possible to use LogicCheckGPT to further improve LLaVA's performance on these types of binary questions?
If so, could you provide guidance on how to implement it?

Below is the current implementation I am using for LLaVA to generate answers from an image:

model_id = "llava-hf/llava-1.5-7b-hf"
model = LlavaForConditionalGeneration.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    low_cpu_mem_usage=True, 
).to(0)

processor = AutoProcessor.from_pretrained(model_id)

conversation = [
    {

      "role": "user",
      "content": [
          {"type": "text", "text": question_text},
          {"type": "image"},
        ],
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

inputs = processor(images=raw_image, text=prompt, return_tensors='pt').to(0, torch.float16)

output = model.generate(**inputs, max_new_tokens=20, do_sample=False)
answer_text = processor.decode(output[0][2:], skip_special_tokens=True)

[Hyperwjf]

Yes, LogicCheckGPT can definitely be integrated with LLaVA to enhance its performance on binary questions. To do this, you can replace the function get_owl_output.py and its associated variables with LLaVA's components. Additionally, I have just updated the file check_llava.py within the ./logiccheckgpt/ folder to facilitate this integration. Let me know if you need further guidance!

[Tizzzzy]

Hi,
Thank you for your response! Just to clarify, I should now be able to run the updated check_llava.py file with only a few path modifications, correct?
If that's not the case, could you let me know what additional changes are required?
If that is correct, I encountered an error when running the code:

Traceback (most recent call last):
  File "/LogicCheckGPT/./logiccheckgpt/check_llava.py", line 716, in <module>
    main(type)
  File "/LogicCheckGPT/./logiccheckgpt/check_llava.py", line 677, in main
    claim = get_llava_output(temp_image_path, text)
  File "/LogicCheckGPT/./logiccheckgpt/check_llava.py", line 621, in get_llava_output
    n_diff_input_output = (input_ids != output_ids[:, :input_token_len]).sum().item()
RuntimeError: The size of tensor a (51) must match the size of tensor b (20) at non-singleton dimension 1

My transformers version is 4.36.1, and I also tried using the latest version, and it gives me a different error. Let me know if you need any more information

[Hyperwjf]

Hi,
Thank you for your feedback and for reporting the issue!

We have revisited the code and made a few adjustments to the implementation details during the reorganization process. The error you encountered is likely due to recent updates in the LLaVA repository, which may have introduced changes in the model's behavior or input/output formats.

To address this, we have:

1. Verified the executability of the code. Updated the logiccheckgpt/check_llava.py file.
2. Updated the README.md to include usage guidelines

[Tizzzzy]

Hi,
Thank you so much for your help. I am now successfully replicated the experiment for LLaVa1.5 on POPE random benchmark. I got the result shown in below. I think it is high performance compare with other mitigation methods I have run.