Workflow : I2V & T2V storytelling with multi-event temporal control with Prompt Relay
I2V & T2V Multi-Event Temporal Control with Prompt Relay
With the new ComfyUI node for Prompt Relay by Kijai, you can create multi-event or multi-scene videos with each segment having its own prompt and length.
A neat and easy way to set pace timing and length for intervals/scenes of your video or cue up a sequence of actions with temporal control.
And it "forces" you to be more structural and story based when you prompt, and with the timing segment part you can get some nice short stories or sequence of events....
The node seems to be work-in-progress going by the message on the repro, and so is this workflow as well ...
Updates and tweaks might come ;-) But seems to work quite nicely already... feel free to test it out ;-)
You need this node: https://github.com/kijai/ComfyUI-PromptRelay
(and you can read more about the concept here: https://gordonchen19.github.io/Prompt-Relay/ )
A workflow here (and there is a workflow at the repro of the node as well):
https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Movie-Maker
I2V - Multi-Ref Image & Multi-Event prompt relay workflow
Added a workflow variant of the multi-prompt workflow where you can also input a start frame reference image for each sequence, from idea by @APCOTech ;-)
( a little bit experimental, but seems to work ok from a couple of test runs )
To create different first frame images with consistent characters you can use Flux Klein (just prompt for a new scene with same characters), Qwen Image with Next-Scene or Multiple-Angles lora, or just ask Nano Banana, Chat GPT, or Qwen Chat to do it ;-) (I'll upload a "helper" workflow with Klein and Qwen to create first frame images and there is one natively already inside ComfyUI as well)
Can try it out here:
https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Movie-Maker
( strictly speaking the simpler workflow with one reference image, can pretty much do the same. Change camera angles, scene etc with consistent characters, if both/all are in the input ref. image. But having multiple ref.images gives you even more granular control)
as a video editor .. i think its not that useful to cut scene in the video itself. we are going to pull in to the editor and cut it anyway to add continuous music. what we need is consistent character like seedance 2.0
But thank you for your effort your workflows are amazing..
yes its more fun fun quick way to make a little story for those that want that, that are not video editors ;-) To create something quickly, share with friends, a short little social etc etc
For the more advanced users, you'd might want to use an external video editor, and generate lots of single clips and stitch together manually...
Consistent characters for LTX probably need lora. Either a trained one on a characters you use, or some "vace" like lora that takes any reference image as ID (and i bet the "eco system" around LTX will grow with loras, derivate models and more ... just like it did for Wan over time)
Timeline Editor for Prompt Relay
Added new workflow to use Kijai's new timeline node, that makes it really intuitive and easy to adjust and add the sequences you want in your video.
Each sequence takes its own prompt for your story telling ;-) Drag and resize each sequence in the editor, and add as many sequences as you want/need
Can try it out here:
https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Movie-Maker
And if you want to try out the timeline workflow, you need to update the Prompt Relay node:
https://github.com/kijai/ComfyUI-PromptRelay
Requires updated ComfyUI-PromptRelay for the timeline node*
https://github.com/kijai/ComfyUI-PromptRelay
Timeline Editor for Prompt Relay + Multi reference Image
A variant of the timeline workflow where you can set reference images per segment, also available to try..
(renamed the older files, to be a little shorter.... the workflow are same though)
Updated Timeline Workflows*
The NAG node fight for the same crossattention as the prompt relay.
So thats removed in the updated Timeline workflows.
The updated timeline workflows should be more accurate, and if using with multi-images, the NAG node would mess up the timing, and images would inject different to the prompt segment (and potentially give all sorts of weirdness ;-))
(The prompt relay node has also been updated, if you update to latest KJNodes. You can now use seconds at the timeline instead of frames, if you prefer that)
Prompt Relay with Custom Audio
Workflow added for prompt relay + custom audio. Single input image (but will add one for multi image as well).
And Kijai added an advanced option to the node to play around with for those who want to tinker and experiment ;-)
I tested your LTX-2.3_-_I2V_Short-Story_PromptRelay-Timeline_multi-image_multi-sequence.json workflow on a cloud L4 GPU, and it only use less than 15GB VRAM and less than 15GB RAM too 😯 pretty small memory footprints for a video model 👍
But it took about 11 minutes to generate 31 seconds video 😅 i guess L4 GPU wasn't fast enough. Then again this is the first inference, the 2nd one might be faster 🤔
seems like comfyu gotten quite a lot of VRAM updates lately, and probably still work in progress.. but on any given day it runs butter smooth (and on occasion where things might not, they are quick to fix)
15 GB VRAM is decent though .Should work ;-) and might be faster on 2nd run true. The first run is always a bit slower.. model loading, warming up and all
Prompt Relay + Custom Audio with multi reference image
Added a variant of the custom audio workflow that can take multi image references.
Although you can get quite the same with single reference image and prompting for "scene cut / new scene" etc with prompt describing the new scene, the multi ref image can give a bit more manual control.
A "music video" type of example, with song as custom audio.. but can be used for any type of storytelling :
Music Video - Prompt Relay combo
Uploaded a workflow variant of the "music video" workflows with Prompt Relay per group.
Makes it easier to variate a little bit, even within each group.
Its a lot of prompting though ;-)
(the video above is far from as much variation you can do, it was more a test run to check if all was working;-)_)
If you want to try out creating "music video" : https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Music-Video-Creator
at the end, in video combine node i am gettig the issue because of get_final_audio thing.
why is this the case?
Exception: An error occured in the ffmpeg subprocess:
[aac @ 0x6084d76e1880] Input contains (near) NaN/+-Inf
[aost#0:1/aac @ 0x6084d76e14c0] Error submitting audio frame to the encoder
Error while filtering: Invalid argument
[aac @ 0x6084d76e1880] Input contains (near) NaN/+-Inf
[aost#0:1/aac @ 0x6084d76e14c0] Error submitting audio frame to the encoder
Exception: An error occured in the ffmpeg subprocess:
[aac @ 0x6084d76e1880] Input contains (near) NaN/+-Inf
[aost#0:1/aac @ 0x6084d76e14c0] Error submitting audio frame to the encoder
Error while filtering: Invalid argument
[aac @ 0x6084d76e1880] Input contains (near) NaN/+-Inf
[aost#0:1/aac @ 0x6084d76e14c0] Error submitting audio frame to the encoder
File "/home/abhiraj/ComfyUI/ComfyUI/execution.py", line 534, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/abhiraj/ComfyUI/ComfyUI/execution.py", line 334, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/abhiraj/ComfyUI/ComfyUI/execution.py", line 308, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/home/abhiraj/ComfyUI/ComfyUI/execution.py", line 296, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "/home/abhiraj/ComfyUI/ComfyUI/custom_nodes/comfyui-videohelpersuite/videohelpersuite/nodes.py", line 600, in combine_video
raise Exception("An error occured in the ffmpeg subprocess:\n" \
Can someone help me ont his?
Looks like an ffmpeg error with the audio. Strange.
will check my end if the Video Combine node is broken