Prepare the training dataset

[XavierLiu888]

Hello,

Can you provide a more detailed guidance for the training dataset preparation?

[sugarandgugu]

Hello, please follow this json: https://github.com/ZJU-REAL/GUI-G2/blob/main/example_training_json.json to construct your training dataset. The fields 'image_path', 'instruction', and 'abs_box' must be included in the JSON.

For example, you can use GTA1 Grounding dataset for training your model.

You need to transform their data to GUI-G2 data format.

image -> image_path

bbox -> abs_box

conversations[0]['value'].split('')[1] -> instruction

Hope it can help

[XavierLiu888]

Thanks for your answer! I notice that the paper leverage Widget Captioning, ShowUI-web and OmniAct as training dataset, while the downloaded files are not matched with provided json format. Is it required to write a custom script to convert these data into GUI-G2 data format? Could you provide a official data transform code? Hope to hear from you soon.

[sugarandgugu]

Due to company regulatory reasons, we currently have no plans to open source the code for processing datasets. You can follow the dataset examples provided by GUI-G2 and use LLM to write Python code for data construction.

[lihua8848]

I'm curious about how you obtain the bounding boxes (bbox) for Omniact. The project doesn't seem to provide a script for it, and it appears that simply checking if a point falls within a candidate bbox isn't sufficient to determine the target bbox.