I want to evaluate ShowUI on Multimodal-Mind2web, How can I get hf_test_full.json?

[ngiraffe]

First of all, thanks for sharing your awesome work!

I want to evaluate (not train) ShowUI on Multimodal-Mind2web via applying eval_only on train.py,

However, I think I need hf_test_full.json to do that.
As far as I know, there are only hf_train.json on https://huggingface.co/datasets/showlab/ShowUI-desktop/tree/main/metadata.

Could you please share file and script file required for us to execute inference/ evaluate ShowUI on Multimodal-Mind2web?

Best Regards,

[QinghongLin]

@ngiraffe Thanks for reminds! Yes, I forgot to upload this data, please give me some time to upload these metadata and images.

[syorami]

@QinghongLin hi also how can I get hf_test_full.json for dataset ScreenSpot? is there a way to generate it?

[QinghongLin]

@syorami the hf_test_full.json is available in here : https://huggingface.co/datasets/KevinQHLin/ScreenSpot/tree/main/metadata

[syorami]

@syorami the hf_test_full.json is available in here : https://huggingface.co/datasets/KevinQHLin/ScreenSpot/tree/main/metadata

@QinghongLin I donwloaded KevinQHLin/ScreenSpot hf data but found the images were broken and couldn't load. Could you have a check on the integrity?

>>> Image.open('mobile_ef01fd08-a7fd-42a3-b1aa-fe22b3f79730.png')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/lib/python3.10/site-packages/PIL/Image.py", line 3498, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file '/mnt/data/ScreenSpot/images/mobile_ef01fd08-a7fd-42a3-b1aa-fe22b3f79730.png'