Stable-Diffusion-Prompts Fidelity Evaluation

This project provides a script to evaluate the fidelity of the Stable-Diffusion (SD) model to prompts. The fidelity is assessed by comparing cosine similarity between two sets of prompts. The first set of prompts (Prompt A) is extracted from the metadata of PNG images generated by the Stable-Diffusion-WebUI project, while the second set of prompts (Prompt B) is generated by applying the WD 1.4 MOAT Tagger V2 model to the images.

Requirements

• Python 3.6 or above
• OpenAI API Key
• Libraries: os, json, concurrent.futures, typing, pathlib, png, requests, cv2, numpy, pandas, tqdm, onnxruntime, PIL, sklearn, huggingface_hub

You can install these requirements by running:

pip install -r requirements.txt

Usage

1. Clone this repository to your local machine.
2. Install the necessary Python packages.
3. Replace the API_KEY in the script with your own OpenAI API key.
4. Specify the directory containing the PNG images generated by the Stable-Diffusion-WebUI project in the folder_path variable.
5. Run the script using the command python text_extraction_and_similarity.py.

Output

The script will output a CSV file named 'results.csv' with the following columns:

• 'PNG File': the name of the PNG file
• 'Prompt A': the prompt extracted from the PNG file
• 'Prompt B': the prompt generated by the WD 1.4 MOAT Tagger V2 model
• 'Cosine Similarity': the cosine similarity between the embeddings of Prompt A and Prompt B

The script will also print the average, maximum, and minimum cosine similarity.

Notes

This script uses the cosine similarity to measure the similarity between the embeddings of the two sets of prompts. The embeddings are obtained using the 'text-embedding-ada-002' model from OpenAI.

The script also uses multi-threading to speed up the process of obtaining the embeddings. The number of threads can be adjusted using the max_workers parameter in the ThreadPoolExecutor function.

Please note that the script requires a stable internet connection to work properly, as it makes requests to the OpenAI API to obtain the embeddings.

Support

For any issues or questions, please open an issue on this repository.