The implementation of “MR2V: Customized Storytelling”

[lianqi1008]

Hi, thanks for releasing StoryMem.
I am trying to reproduce the MR2V customized storytelling examples shown on the project page, such as the clumsy man / young lovers / cafe examples. From the paper/project page, it seems that MR2V uses one or more reference images as memory and then generates a longer story video while preserving the referenced subject identity. I would like to understand how the long project-page videos were generated.

Could you please clarify the intended reproduction workflow?

1. Were the long MR2V project-page videos generated shot-by-shot using multiple text prompts, with memory updated after each generated shot? Or were they generated from a single long story-level prompt?
2. If they were generated shot-by-shot, what is the expected input format for the story script? For example, do you use a JSON file containing story_overview, video_prompts, first_frame_prompt, and scene-cut indicators?
3. How are the reference images provided as the initial memory for MR2V? Are they saved as initial keyframes before running the pipeline?
4. Are the exact prompts used for the project-page MR2V demos available, or could you share an example configuration?

I am currently able to generate a short single M2V clip, but I am not sure how to reproduce the longer MR2V story videos shown on the project page.

Any guidance or example command would be very helpful. Thanks!

[Kevin-thu]

Thank you for your interest in our work. Regarding your questions:

1. All videos are generated shot by shot using shot-level text prompts, either MR2V or M2V.
2. Yes. However, story_overview and first_frame_prompt are optional and mainly provided for human readability. Only the shot-level prompts in video_prompts are used for generation.
3. They are used as the initial keyframes, which serve as the initial memory for the M2V model.
4. Example configurations are provided in ST-Bench. In practice, you only need to save the reference frames as keyframes named 00_00_keyframe0.jpg, 00_00_keyframe1.jpg, etc., under the output path. Alternatively, you can slightly modify the code to directly accept reference images as the initial memory.

[lianqi1008]

@Kevin-thu Thanks for your reply. I am also trying to reproduce the quantitative metrics reported in the paper. I evaluated generated multi-shot stories using the following models:

• ViCLIP for Prompt Following and Cross-shot Consistency:

  • OpenGVLab/ViCLIP-L-14-hf
  • Download link: https://huggingface.co/OpenGVLab/ViCLIP-L-14-hf

• LAION aesthetic predictor for Aesthetic Quality:

  • sa_0_4_vit_l_14_linear.pth
  • Download link: https://raw.githubusercontent.com/LAION-AI/aesthetic-predictor/main/sa_0_4_vit_l_14_linear.pth

• OpenAI CLIP image encoder for aesthetic scoring:

  • ViT-L/14
  • Loaded through the official OpenAI CLIP package:
    https://github.com/openai/CLIP

My implementation computes the metrics as follows:

1. Aesthetic Quality

For each generated shot video, I read all frames and compute the LAION aesthetic predictor score on CLIP image features. The story-level aesthetic score is the mean over all shot-level
aesthetic scores.

2. Prompt Following

I use ViCLIP video/text cosine similarity.

• Global Prompt Following:

  • concatenate/read frames from all shots in a story;
  • uniformly sample global_frames=32;
  • encode them with ViCLIP vision encoder;
  • encode the story overview with ViCLIP text encoder;
  • compute cosine similarity.

• Single-shot Prompt Following:

  • for each shot, sample frames_per_shot=8 middle frames;
  • encode the shot video with ViCLIP vision encoder;
  • encode the corresponding shot prompt with ViCLIP text encoder;
  • compute cosine similarity;
  • average over all shots.

3. Cross-shot Consistency

For each shot, I encode the sampled video frames with ViCLIP vision encoder and compute cosine similarity between all shot pairs.

• Overall Consistency:

  • mean ViCLIP video-feature similarity across all shot pairs.

• Top-10 Consistency:

  • first compute ViCLIP text-feature similarity between all shot prompts;
  • select the top 10 most semantically related shot pairs by prompt similarity;
  • report the mean ViCLIP video-feature similarity of those selected pairs.

In other words, the Top-10 pairs are selected using prompt/text similarity, but the reported consistency score is still video/video feature similarity.

My current aggregate results are approximately:

{
  "Aesthetic Quality": 0.5995,
  "Prompt Following": {
    "Global": 0.2350,
    "Single-shot": 0.2316
  },
  "Cross-shot Consistency": {
    "Overall": 0.5806,
    "Top-10 Pairs": 0.6045
  }
}

Could you please clarify:

1. Is this the same evaluation protocol used in the paper?
2. For Aesthetic Quality, did you evaluate all frames or sampled frames?
3. For Prompt Following and Cross-shot Consistency, could you please provide the exact frame sampling strategy and number of frames used for each metric?

Any details would be very helpful for reproduction.

Thanks!