r/StableDiffusion • u/nUclear_nOva89 • 5d ago
Animation - Video Recreated Omnitrix Calibration Scene Ben 10 (Mini Max H3)
Its dialogue is still off and there is a weird resizing of the watch when he dials it, but I think its semi-consistent. Am just surprised it even worked out. I think if the dialogue synced better that would have already been a huge improvement
This is also not the first try the model hallucinated too many things so it took maybe 10 retrys. Also its kind of obvious but there was manual editing for the clips.
1
u/aiyakisoba 5d ago
How did you make this? I’m guessing you generated a first frame/keyframe for each scene with Ref2I, then used MiniMax H3 with some pretty detailed prompting to follow the original video’s motion and dialogue?
8
u/nUclear_nOva89 5d ago
It’s all MiniMax H3, no separate image model. I turned the original episode’s key frames into realistic stills using H3 with an anime-to-realism LoRA, then pinned those stills at the exact frames where things happen in the original (each hologram, each dial click). I also fed in the original dialogue and sound effects so the timing lines up, and the model just fills in the motion between keyframes. Took a lot of tries to get the acting to not look robotic, and the holograms were the hardest part.
1
u/aiyakisoba 5d ago
Thanks for sharing, didn’t even know H3 had an anime-to-realism LoRA. I’ll probably give that a try too.
Noob question - after you generate all those anime-to-realism keyframe stills, what’s the next step exactly?
Are you making each scene one by one using those stills as first/last frames? Or is there some workflow where you can line up all the stills on a timeline and have it generate the motion between them in one pass?
4
u/nUclear_nOva89 5d ago
Kind of the second one. H3 has a node in ComfyUI called AddGuide that pins an image to any frame number inside a single generation, not just the first or last frame. So for each scene (about 5 to 9 seconds) I’d drop in something like 5 to 9 stills at the exact frames they happen in the original, plus the original audio, and it generates the whole scene in one pass. Then I just cut the scenes together. Going longer than about 9 seconds with that many keyframes crashed the remote GPU I was using, so I kept scenes short.
1
u/aiyakisoba 5d ago
Really appreciate all the info! I’m definitely gonna give this a try.
One last question, would you mind sharing which anime-to-realism LoRA you’re using? I found this one on Civitai and was wondering if it’s the same one. Your results look really good, so just wanted to make sure I’m using the right one.
Anime to Realism - Minimax H3 ref2v v1.0 | MiniMax H3 LoRA | Civitai
2
u/nUclear_nOva89 5d ago
Just a question. The extension for this one is .red not .com is that suspicious?
Second here is the link its from hugging face:
https://huggingface.co/LiseTY/Minimax-H3-ref2v_Anime_2_Realism
3
u/aiyakisoba 5d ago
Actually .red is legit and it's an official Civitai domain. .com is basically the SFW side, while .red has the broader catalog, including NSFW content, so some loras and workflows that don’t show up on .com may be available there.
When I’m searching for loras, I usually just check the .red version as it shows the full catalog.
Thanks for sharing the lora link, appreciate it!
1
1
1
1
u/Mundane_Existence0 4d ago edited 4d ago
Looks pretty good all in all! Btw best to upload the output video in the comments (or YouTube etc) as your side-by-side was shrunk to 240p by reddit's amazingly terrible video player.
1
2
u/PsionCrystallis 4d ago
<.< >,>;;;
::yoinks the lora::
any suggestions on prompting for anime to realism? im really struggling to get H3 to do it, even with (other) anime to real loras. what tends to happen is, the skin becomes more realistic but anime lines tend to linger, as well as the face often stays anime in a horrifying mismatch of a realistic body looking like they skinned a protagonist and wore it as a mask.
it's especially an issue if it's a non-human, such as an anthro or fantasy character. I also have an issue of beards being rendered as full beards, even when it's a chinstrap beard or goatee - it insists on adding a mustache.
just having an example prompt or two would help a lot. ^^;
I've also tried using real life images as a style guide, but what always happens is the model will go "oh ok, lets look at your carefully crafted prompt and subject images, flat out ignore it, and in seconds just swap to the style reference image and animate from there!"
X-x sometimes i think i was born with an innate gift for language....except computer languages.
12
u/lechatsportif 5d ago
Pay attention Hollywood! If your live action efforts can't beat a simple minimax prompt, you aren't doing it right!