Which dataset is being used for whitening and how many images?

[JosTheBossX]

In the code, I see that there is a block where it does the whitening if whitened decoder is not provided.
But it uses train_dir, is this the same dataset used to train the LDM? as in the Readme.md it states that only 500 images are used to fine-tune the LDM.

But in the paper when we see in A.1. it states in the Whitening subheading that 10k images are used to whiten (but not mentioned are these images, same as the train/val on which HiDDen was trained, or unseen data, or it doesn't matter what data we use to whiten?)

While in B.5. it states that :"as a reminder the whitening is performed on 1k vanilla images from COCO at 256 × 256"

So my main question is, how many images are being used to whiten the decoder, and what dataset are we using for it?

[pierrefdz]

Hi!
We used 10k images from the COCO dataset resized to 256x256. I haven't experimented much with the whitening dataset. I think it's important to have it at the same resolution as the images used when fine-tuning the LDM decoder, but other than that I don't know if it makes much difference.