How to Make Multiple Characters Interact in Storydiffusion Scenes?

[macnux]

I’m working with ComfyUI and Storydiffusion, and I’ve run into an issue. Whenever I try to generate scenes with two or more characters, I get an error that says StoryDiffusion_CLIPTextEncode.
list index out of range
This is my workflow:

Thew following get the mentioned error:

[Taylor] a girl img.
[Adam] a man img.

But if only one person it works okay.

[smthemex]

need [Adam] do something.... ,you just give 1 role prompts

[macnux]

After adding two roles, still same error:

got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
infer model is qwen_image
total_vram is 16302.5625,aggressive_offload is True,offload is True
The config attributes {'pooled_projection_dim': 768} were passed to QwenImageTransformer2DModel, but are not expected and will be ignored. Please verify your config.json configuration file.
load lora weights : D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\models\loras\Qwen-Image-Lightning-8steps-V1.1.safetensors
The config attributes {'text_encoder': ['transformers', 'Qwen2_5_VLForConditionalGeneration'], 'vae': ['diffusers', 'AutoencoderKLQwenImage']} were passed to QwenImagePipeline, but are not expected and will be ignored. Please verify your model_index.json configuration file.
Keyword arguments {'text_encoder': ['transformers', 'Qwen2_5_VLForConditionalGeneration'], 'vae': ['diffusers', 'AutoencoderKLQwenImage']} are not expected by QwenImagePipeline and will be ignored.
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 3/3 [00:00<00:00, 22.39it/s]
Expected types for transformer: (<class 'D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\custom_nodes\comfyui_storydiffusion.qwen_image.diffusers_wrapper.transformer_qwenimage.QwenImageTransformer2DModel'>,), got <class 'diffusers.models.transformers.transformer_qwenimage.QwenImageTransformer2DModel'>.
Loading transformer.
Using scaled fp8: fp8 matrix mult: False, scale input: False
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
!!! Exception during processing !!! list assignment index out of range
Traceback (most recent call last):
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\execution.py", line 496, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\execution.py", line 315, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\execution.py", line 289, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\execution.py", line 277, in process_inputs
result = f(**inputs)
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\custom_nodes\comfyui_storydiffusion\Storydiffusion_node.py", line 423, in encode
role_key_list=replicate_data_by_indices(role_list, index_char_1_list, index_char_2_list)
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\custom_nodes\comfyui_storydiffusion\model_loader_utils.py", line 942, in replicate_data_by_indices
new_list[idx] = data_list[0]
~~~~~~~~^^^^^
IndexError: list assignment index out of range

[smthemex]

NO ,you just fill one role。
need
[A] ...
[B] ...
[A] and [B]

you just
[A]
[A]
[A]

[macnux]

[Taylor] a girl img.
[Adam] a man img.

[Taylor] sleep on bed.
[Taylor] eat breakfast and she is sad.
[Taylor] talk to [Adam].

Error:
StoryDiffusion_CLIPTextEncode
list assignment index out of range

===========================================================
Another role:

[Taylor] a girl img.
[Adam] a man img.

[Taylor] sleep on bed.
[Adam] eat breakfast.
[Taylor] talk to [Adam].

Error:
StoryDiffusion_CLIPTextEncode
cannot access local variable 'ref_images' where it is not associated with a value

[smthemex]

[Taylor] a girl img.
[Adam] a man img.

[Taylor] sleep on bed.
[Adam] eat breakfast.
[Taylor] talk to [Adam]. it's right , but you dit path maybe has edit it in it ,so infer in qwen_edit mode, when use qwen_edit ,need link two image (two roles) to image

[macnux]

I'm sorry, but I didn't understand what you mean, can you share a working wokflow because I search the entire web for a working wokflow for storydiffusion and didn't find a one that allows me to make characters interact with each other.

[smthemex]

links or in your StoryDiffusion node directory should have this workflow file。

[macnux]

The workflow I shared above is from this file you shared and it dones't work when I add 2 characters and make them interact in a scene, you can check your self.

Workflow name: MS IMG2IMG (I didn't touched or changed anything I just run it)
text:
[Taylor] have breakfast by the window;
[Lecun] is working.
[Taylor] and [Lecun] dancing on the stage.

Error:
got prompt
!!! Exception during processing !!! exceptions must derive from BaseException
Traceback (most recent call last):
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\execution.py", line 496, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\execution.py", line 315, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\execution.py", line 289, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\execution.py", line 277, in process_inputs
result = f(**inputs)
File "D:\Source\Development_Sources\AI\ComfyUI\ComfyUI\custom_nodes\comfyui_storydiffusion\Storydiffusion_node.py", line 844, in encode
raise "when using msdiffusion ,(objectA) and (objectA) must be in the prompt."
TypeError: exceptions must derive from BaseException

Workflow name:msdiffusion+story+ 2rolesin1img ,support txt2img and img2img cn
Output is not correct in terms of the scene prompt.
Text:
[Taylor] have breakfast by the window;
[Lecun] is working.
[Taylor] and [Lecun] dancing on the stage.

The ouput is not ancing at all as you can see !!!

[smthemex]

yeah ,it's maybe a new bug, i will fix it sometioms.

[macnux]

Another workflow I tried and the scenes where 2 characters are interacting, it outputs all images except the images where 2 characters are interacting.

7 scenes and it output only 5 scenes and the 2 scenes where 2 characters are interacting are not generated.

Story_YammersAnime.json

[smthemex]

when use sdxl，only msdiffusion and story maker or story and maker mode can two role in one image or flux method

[macnux]

I need a working workflow, just one.
Can you share a working wokflow please for two role in one image?

[smthemex]

There will be a new update soon