Error Convert_Civitai_OpenVINO

[homevk15]

I tried Convert_Civitai_OpenVINO.py and got many warnings and an error.

D:\Program Files\Python310\lib\site-packages\diffusers\models\unet_2d_condition.py:915: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if dim % default_overall_up_factor != 0:
D:\Program Files\Python310\lib\site-packages\diffusers\models\downsampling.py:135: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert hidden_states.shape[1] == self.channels
D:\Program Files\Python310\lib\site-packages\diffusers\models\downsampling.py:144: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert hidden_states.shape[1] == self.channels
D:\Program Files\Python310\lib\site-packages\diffusers\models\upsampling.py:149: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert hidden_states.shape[1] == self.channels
D:\Program Files\Python310\lib\site-packages\diffusers\models\upsampling.py:165: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if hidden_states.shape[0] >= 64:
D:\Program Files\Python310\lib\site-packages\diffusers\models\unet_2d_condition.py:1206: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if not return_dict:
Unet successfully converted to ONNX
Traceback (most recent call last):
File "D:\astable-diffusion\openvino_notebooks\notebooks\225-stable-diffusion-text-to-image\t5\civitai.py", line 87, in
model = mo.convert_model(UNET_ONNX_PATH, compress_to_fp16=True)
File "D:\Program Files\Python310\lib\site-packages\openvino\tools\mo\convert.py", line 364, in convert_model
ov_model, _ = _convert(cli_parser, framework, params, True)
File "D:\Program Files\Python310\lib\site-packages\openvino\tools\mo\convert_impl.py", line 957, in _convert
raise e.with_traceback(None)
RuntimeError: Check 'false' failed at src\frontends\common\src\frontend.cpp:54:
Converting input model
bad allocation

torch: 2.1.2+cpu
onnx: 1.15.0
onnxruntime: 1.16.3
tensorflow-intel: 2.14.0
Intel Core i5-6500 CPU

I found openvinotoolkit/openvino#12747 that tensorflow-intel >=2.8.0 will cause unexpected error.
I tried to downgrade tensorflow-intel to a version tensorflow-intel==2.8.0, but it asked me to downgrade other packages like
Other possible workarounds are:

1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

Since you made it working, could you create virtualenv with needed packages?

[homevk15]

It would be good to have the next improvements:
1)virtualenv
2)negative prompt
3)possibility to run in Intel GPU (iGPU)
Let me know if you want to make any tests or get some donations. I'm very interested in making this project working.

[RedAndr]

@homevk15
Let's remove TensorFlow dependencies since the model was successfully converted to ONNX format. We can disregard the warnings for now. The issue seems to be with converting the ONNX model to OpenVINO format. As a workaround, you can comment out the conversion line in the code and instead use the Model Optimizer tool to do the conversion:
mo --input_model unet/unet.onnx --compress_to_fp16

About the negative prompt, it can be easily added. Something like that:

        bs_embed, seq_len, _ = uncond_embeddings.shape
        uncond_embeddings = uncond_embeddings.view(bs_embed, seq_len, -1)
        text_embeddings = torch.cat([uncond_embeddings, text_embeddings])

I don't have iGPU, so can't test the code on it.

[homevk15]

After updating tensorflow-intel to tensorflow-intel-2.14.1 and installing openvino-dev[tensorflow2] I got success.
I got unet, text_encoder, vae_decoder, vae_encoder (.xml and .bin files).
The program didn't generate an image
I see only
Loading pipeline components...: 100%
without errors.
Do you know what to change to get an image in an output?
Also, please, update Convert_Civitai_OpenVINO with the code of negative prompt you posted. I'm new to this stuff. Thank you.

[RedAndr]

Yes, this code is intended only to convert a model. It doesn't generate images. For that you should use https://github.com/bes-dev/stable_diffusion.openvino

[homevk15]

how opset calculated?
Is it opset = 16 for 512x512 images all the time?
After running demo.py (from https://github.com/bes-dev/stable_diffusion.openvino) with converted to OpenVINO model I got errors
Traceback (most recent call last):
File "D:\astable-diffusion\openvino_notebooks\notebooks\225-stable-diffusion-text-to-image\t6\demo.py", line 83, in
main(args)
File "D:\astable-diffusion\openvino_notebooks\notebooks\225-stable-diffusion-text-to-image\t6\demo.py", line 21, in main
scheduler = LMSDiscreteScheduler(
File "D:\Program Files\Python310\lib\site-packages\diffusers\configuration_utils.py", line 649, in inner_init
init(self, *args, **init_kwargs)
TypeError: LMSDiscreteScheduler.init() got an unexpected keyword argument 'tensor_format'
I don't know how to fix it.