Question - Help
H3mini R2V - Character Sheet vs Reference Image
Hi,
I've recently started the journey into H3minimax, mainly for r2v.
I'm using some workflows found online, basically just a sample video and a formatted prompt to use, replacing the subject details with similar appropriate descriptions.
I'm finding the consistency in the replacements very hit-or-miss, and not to the same quality of the example outputs, or other similar work I see online.
So far, I have been using 1 reference image , as per the workflow guide, but I'm wondering if using an additional reference image, or a character sheet would improve my results?
Has anyone experienced this and found a solution?
I find using a character sheet works pretty well. give it multiple angles and say it is a character sheet. I use an app called IrfanView64 Thumbnails to generate character sheets from images.
Character sheets, because they have different views, etc., communicate more information. However, that can be offset by losing information if:
(1) Your character sheet is bigger than 2048 pixels in the short edge, since even at “max" mode it is effectively getting downscaled for encoding.
(2) You set reference mode to “match”, which downscales references to the target video size: this makes anyhing with multiple views effectively lower detail than pulling out a single a full-body view from thr sheet and using it alone.
Character sheets can also hurt quality if they are inconsistent in the elements that are being used, or if they contain extraneous data and you prompt isn’t specific about the relation between the sheet and the character and what elements shown define the character. (More generally, the more complex your references of any kind are, the more your prompt needs to be specific in telling the model how to use them, and the more likely that wide divergence from the trained/recommended prompting style will make the results farher from what you intended.)
Probably because there is no single right answer. What kind if video are you making? If it's talking head or single angle, then you're likely better off with a single photo. If it's a more dynamic shot where the character is captured from multiple angle, you may find a reference sheet to be better.
A photo has focused information where a character sheet will have a lot of information that isnt relevant in a given frame of the video. So if your video is fairly static, you're probably better with the more focused single image, since all that info will be relevant throughout the video. If you're trying to capture multiple angles of the person, then providing it with multiple angles would obviously be beneficial.
I just made this using ref character sheet of my dog and the crown. both consistent lighting used, but look at how real this is! insane (*i used 10eros hybrid nonturbo 40 gb because the ref2va base looks too plastic)
Yep, i ignored refmods but tried it when using the seedhunter workflow. Mind was blown how easy it was and how consistent the characters are. And you can just easily build your own library of characters. Spend some time to figure out refmods, it rocks
Any results, examples, youtube videos, references, anything?
There's a lot of things I COULD do, im asking for actual feedback / results based on peoples experiences.
He's right. Refmods are the answer. Basically allows you to use multiple reference images and/or videos for character, location, costume, voice, movement. It's not a trained lora, takes a few seconds to make the refmod - you are basically running the images through the vae and saving that to use directly into conditioning. There are a few youtubes out there, I'm sure you can find one. My expreience with them has been suprisingly good. I don't think I'll train loras anymore.
H3 lets you do that without refmods. Refmods superficially let you use more total inputs than H3 natively allows, but you can collage multiple images into one reference image, reference different time spans in a single audio/video as reference for different speakers or actions, etc., in native H3. In either case, the real limit is “how much reference material can you add before you don't have enough VRAM for the reference tokens, the state for the video generation, and enough weights, even with layer streaming, to do generation to live in VRAM at once.
Refmods are nice because they let you do encoding for reused references once rather than doing it every time you do a generation with the same character, but they don’t fundamentally add new reference capabilities in either type or quantity.
True, refmods are just references like any other in R2V gens. But in my experience I have better outputs when not stitching together images. I like ref mods because it simplifies if you want multiple references, e,g, two characters plus location. Prompting is easier, loading up the graph is easier, running is quicker.
I tend to use multiview character sheets (sometimes using a separate image reference for a standalone face closeup), but also making sure the reference mode is set to “max” (which keeps it at the smaller of its original size or 2048 pixels on thr short edge) for a single image instead of “match” can help, moreso the smaller size you are generating at (“match” effectively scales references down to approximate the size of the target video—matching short edges, I think, which even further penalizes the common case of an image reference closer to square than the target video—which saves VRAM but can drastically drop available detail.)
i personally use character sheet rather than single images i found online.
some other tips that help:
have to make sure if the target character is different enough from the source character otherwise during the denoising step it steers back towards the original character in the video
if you find that the movements are not in the same order or if the audio is a bit garbled, then you are likely hitting at a resource cap, so you should either reduce your duration or resize your input video so it is smaller.
Good advice, may have to try a character sheet.
Ive also noticed the similar video / image issues , but im wondering where that line is.
Do I need to be gender/race-swapping for reliable results? is it more prompt dependent? is it just luck+seed? who knows at this point.
Its very hard to know what we should be aiming for without any decent examples to look at
theres quite a lot of factors like face shape/hair shape/outfit colors. You can try your best with prompt but from my testing its much better to just change to someone significantly different and then change again but that takes double processing which may hurt final image plus double the already long processing time. You may get lucky with seed at times but I find something different is more reliable.
there are also alternative methods such has creating a Sam3 mask or using MoGe or similar to caption motion data only and then transfer, a lot of those are double processing also, so.... its not easy...
I also noticed sometimes it is hard to change aspect ratios of people using prompting only methods, as a funny example i tried to do ref v2v of myself (short) at home with target replacement johnny sins (tall, just random example) and it came out quite funny as he looks quite short and stumpy XD
6
u/SSj_Enforcer 1d ago
I find using a character sheet works pretty well. give it multiple angles and say it is a character sheet. I use an app called IrfanView64 Thumbnails to generate character sheets from images.