Full fine-tune

[kopyl]

Could you please post a brief tutorial on how to fully fine-tune this model on, say, the pokemon dataset?

Right now it's very confusing for me :(
More questions than answers.

But this model is very cool, i was not that excited even about SDXL.

https://huggingface.co/datasets/lambdalabs/pokemon-blip-captions

[kopyl]

Some of the questions:

1. How do I specify LR? (UPD: answer found: in a config file: optimizer = dict(type='AdamW', lr=2e-5, weight_decay=3e-2, eps=1e-10))
2. How do I use HF dataset?

You said:

You ONLY need to change the config file in config and dataloader in dataset.

so

3. How do I need to change the config file?
4. How do I change the dataloader
5. What is PixArt_xl2_img512_internalms.py? Specifically what does internalms mean?
6. How do you decide which config to use? Only based on image size? Or are there any other criteria?

You said:

The directory structure for SAM dataset is:

cd ./data

SA1B
├──images/
│  ├──sa_xxxxx.jpg
│  ├──sa_xxxxx.jpg
│  ├──......
├──partition/
│  ├──part0.txt
│  ├──part1.txt
│  ├──......
├──caption_feature_wmask/
│  ├──sa_xxxxx.npz
│  ├──sa_xxxxx.npz
│  ├──......
├──img_vae_feature/
│  ├──train_vae_256/
│  │  ├──noflip/
│  │  │  ├──sa_xxxxx.npy
│  │  │  ├──sa_xxxxx.npy
│  │  │  ├──......

so

7. What is partition?
8. What is caption_feature_wmask and how do I get it?
9. What is img_vae_feature and how do I get it?
10. What is train_vae_256 and how do I get it?
11. What is noflip and how do I get it?
12. What's the SAM dataset and where can I find it and find instructions on how to build it?

In my dataset I only have 1. Images; 2. Captions. Why would I need anything else for a simple text to image training?

13. How do you specify to save a training checkpoint each, say, 1k steps?
14. How do you convert a training checkpoint to a Diffusers model which you can load like PixArtAlphaPipeline.from_pretrained("TUNED_MODEL_DIRECTORY", torch_dtype=torch.float16)

[kopyl]

you can look through the codebase in this repo

I did. Still nothing is clear :(

[lawrence-cj]

5.1: multi-scale input means the image size during training is not Square (i.e. 1024x1024), it may be 1024x512 which is according to the real image size. We don't use crop and resize in the multi-scale training. For more details: '2.3 Multi-Aspect Training' in https://arxiv.org/pdf/2307.01952.pdf
6.1 The config files talking about are in https://github.com/PixArt-alpha/PixArt-alpha/tree/master/configs/pixart_config, they are all examples for you to contruct your own config files changing the content (eg. data and so on) according to your dataset. I assume you are pretty good at python and Diffusion models. I can't write the exact config file and training code for you, sry for that.
10-11: This file is what you need to extract features in advance.
I encourage you to study about the GPU memory consumption by yourself, in that way you may familiar with the codebase more.

[kopyl]

6.2 Well, why didn't you follow the most common format for you initial training, which is Huggiface's image datasets format?

[kopyl]

6.1.1 I'm not asking to give me the config file, i just want to understand how should i need to format a dataset for Pixart full fine-tune...

I can't write ... training code

Well, isn't the code already on this repo?

[lawrence-cj]

The training code exists. The reason why we don't follow the HF's image dataset format is at the begining of the project, not everything is settled and changing the dataset format is not the most important work now. Maybe we will change it in the future, or if you are intersted in our project, we highly recomment you to change it and push a PR for our code base. Appreciate it.

[kopyl]

maybe using diffusers training pipeline instead is a better choice

Could you please tell how to do it?

[lawrence-cj]

Training pipeline in diffusers in not our duty, please refer to diffusers lib for more details. lol

[kopyl]

:(

[kopyl]

4.1

here have several dataset class for you to construct your own

Where? I can only see one dataset class InternalDataMS

[lawrence-cj]

https://github.com/PixArt-alpha/PixArt-alpha/tree/master/diffusion/data/datasets

[kopyl]

7.1

the information in the txt file is only the image path like 'dir/sa_xxxxx.jpg'

Could you please show how it looks? I would love to see how I need to format it

[kopyl]

Shouldn't images have extensions?

[kopyl]

Shouldn't part0.txt file have captions instead of file names?

[kopyl]

Also why do you have partitions for captions and don't have for images? So counterintuitive.

Do you have to name them following some format?

[lawrence-cj]

Here is why we do not need a extension: https://github.com/PixArt-alpha/PixArt-alpha/blob/master/diffusion/data/datasets/SA.py#L42C26-L42C26.
we store images and captions with the same name and different extensions, eg. imagedir/sa_3631.jpg and textdir/sa_3631.txt, that's why we don't need a extention in partition. the file in partition is convinient for us to use a subset of the SAM for usage.

[kopyl]

@lawrence-cj thank you very much

[kopyl]

15. I see this line in the config model = 'PixArt_XL_2'
    But what is it? On GitHub i only see these models:

• T5
• VAE
• PixArt-α-SAM-256
• PixArt-α-512
• PixArt-α-1024

[lawrence-cj]

You can clone the whole codebase and search for PixArt_XL_2, which is located at:

PixArt-alpha/diffusion/model/nets/PixArt.py

Line 282 in fe0cb78

def PixArt_XL_2(**kwargs):

[kopyl]

Are you talking about training from scratch?

[lawrence-cj]

Why say that? Maybe we can discuss more in Discord channel.

[kopyl]

@lawrence-cj just sent the message on the Discord channel :)

[kopyl]

16. Do I have to download the model or it's done automatically?

[lawrence-cj]

It should be downloaded automatically. refer: https://github.com/PixArt-alpha/PixArt-alpha/blob/master/tools/download.py
The codebase is kind of robust and growing, that's why I suggest you guys to look through the whole codebase for better future use.