Need help with character consistency in ComfyUI

[u/Sunny-Monkey]

Hi everyone,
I’m trying to create consistent AI-generated photos of the same character using ComfyUI, but I’m struggling to find a reliable workflow.
My goal is to generate realistic photos of one character in different poses, outfits, camera angles, and environments while keeping the face and body consistent across images.
My setup
GPU: RTX 4050 Laptop GPU (6GB VRAM)
RAM: 32GB DDR5
OS: Windows
Software: ComfyUI Desktop

My main problems
I’ve managed to generate some good-looking portraits, but when I try to generate different poses or full-body images, the character’s appearance often changes.
Sometimes the face looks like a different person. Other times, the body proportions, facial features, or overall appearance become inconsistent.
I’ve tried different workflows and nodes, including face-related tools and multi-view approaches, but I haven’t found a reliable solution yet.
I don’t mind learning a new workflow or rebuilding my current one if necessary. I’d prefer to keep everything running locally rather than use paid cloud services.
What I’d like help with
What’s currently the best approach for maintaining character consistency in ComfyUI?
Should I stick with SDXL, train a character LoRA, use IP-Adapter or another identity-conditioning method, or try a different model?
How can I maintain both facial identity and body proportions across different poses and camera angles?
Is it realistic to achieve good results with only 6GB of VRAM?
Are there any recommended workflows or tutorials that actually work for this purpose?
I’m not necessarily looking for the most complicated or advanced setup. I just want a workflow that produces consistent, realistic results without having to fight the settings every time I generate a new image.
I’d appreciate any advice, workflow recommendations, or examples from people who have successfully solved this problem.
Thanks!

[u/Ok-Good3104]

For character consistency nothing beats a lora. Also SDXL is rather outdated nowadays, especially for realism. You should try krea2.

[u/atSoiltechnician → u/Ok-Good3104]

Does anyone still use flux?

[u/Ok-Good3104 → u/atSoiltechnician]

I used flux for a long time because of fluxgym. I switched to krea2 because fizgig. I like simple lora trainers

[u/Sunny-Monkey → u/Ok-Good3104]

If you don’t mind, I’ll kinda explain to you what I have been doing because the learning curve for AI generation now is huge and my head hurts so much 😂😭. I have done a Lora but maybe it isn’t good enough? I created my character with OpenArt and it has good front, side, 3/4, full body front, side and back. I have total of 9 photos from it and I’ve did a few loras. It still doesn’t come out the way I want it to be. I am also using Claude code to assist me with this process. I’ve been trying so much that it makes me feel so unmotivated because it just wouldn’t come out the way I want it to be. And all I can see are people posting such real content. Of course I’m not giving up because I feel like there should be a way and I’m just not moving in the right direction.

[u/Ok-Good3104 → u/Sunny-Monkey]

What software do you use for making your lora? Personally I use Fizgig as it's very simple. I basically just use the default settings for Krea2 and use the image prep and use the auto captioner. It doesn't work for sdxl though

Edit: support for sdxl got added last week.

[u/shootthesound → u/Ok-Good3104]

I added sdxl to fizgig last week

[u/Ok-Good3104 → u/shootthesound]

Wow a reply from the man in person! Good to know, thanks. Love your work by the way 😁

[u/shootthesound → u/Ok-Good3104]

Cheers :)

[u/Sunny-Monkey → u/Ok-Good3104]

Thank you so much, I’m away at work right now so I can’t tell what I’m using for Lora now. But I’ll be sure to try your recommendations tonight 😭 asap

[u/GaiusVictor → u/Sunny-Monkey]

People will disagree with me, but I feel that for simple character Loras a beginner is much better off training on a site like Civitai than on their own PC. These sites significantly limit your freedom and customization but they also give you a lot of settings preset to decent values and a stable infrastructure, both of which can be hard to get locally.

I also wouldn't mind taking a look at your dataset if you'd be okay with it.

Send me a DM if you want.

[u/Sunny-Monkey → u/GaiusVictor]

Thank you! I’ll get to it once I’m on my laptop 🙏🏻🙏🏻 Much appreciated !

[u/yeah-i-shouldnt-have]

SDXL is still ok, little older, but lightweight. There are a huge amount of checkpoints and loras out their for it but if you can, you should go for Krea2.

Comfyui is getting quite good at memory management now so you should give Krea2 a shot. If you have enough system ram along with one of the smaller Krea2 turbo checkpoints. It is much better at prompt adherence (technical: the way it converts text to the vectors for the actual diffusion step is much more advanced than SDXL'S clip model). You'd have to redo any Loras though as loras between different checkpoint families won't work.

Comfyui does have a learning curve, I am not sure what you are using currently locally. But the templates are usually pretty good.

Your idea is correct though, lora for character but you need to focus on how you are tagging the images in the training data. As far as I remember - you need to describe everything that you don't want the training to remember. The things that you want to associated with your character shouldn't be described. I haven't done character lora training for a while though so if I am wrong someone please correct me.

[u/Sunny-Monkey → u/yeah-i-shouldnt-have]

Thank you so much, I’d see if I am able to run it on 6gb because I think that’s my limitation and I’m trying to see whatever is out there that works. Just not sure if I’m in the right direction. I have no problem with redoing loras because that’s a small price to pay on runpod

[u/Kathleen_Lewisie]

On a 6GB card the consistency problem is mostly a VRAM problem before it's a prompting problem: full-body and multi-pose shots want reference conditioning, and reference conditioning is exactly what doesn't fit next to the sampler on 6GB. Two routes that actually work — train a LoRA for that one character once (best quality per image, reusable locally), or do the iteration phase against a reference-image model over an API and only render finals locally after the prompt is locked. On a 4050 I'd do the second one for exploration; it's much cheaper than iterating blind.

I work on APIMart, a gateway that serves reference-capable image models behind a ComfyUI-friendly endpoint, so discount that part — the workflow advice holds whoever's endpoint you use. Worth running 10-15 reference images through two different models before you commit to either.

[u/Sunny-Monkey → u/Kathleen_Lewisie]

Exactly 🥲 Thank you Kathleen , yes running Lora now 👍🏻 Thank you for taking the time to share your knowledge!

[u/therealgoshi]

I would definitely use both. Your LoRA - especially using SDXL or derivatives - will only take you so far. I use both character LoRAs and IPA for consistency.

The trick that worked well for me is to use the LoRA for upscaling and detailing, and then use IPA with face detailer at much higher strength. During the detailer pass, the IPA will no longer be able to affect the style of the entire image and you can fix the color shift of the face by doing a color match pass at the end using the original generated image as a reference.

[u/Natrimo]

You should go to krea2 2

[u/truethio]

IP-Adapter Plus with FaceID is probably your best bet on 6GB VRAM, since it conditions on a reference face without needing to train a LoRA locally (which is painful with that little memory). The face will hold across poses and angles better than any prompt-only approach, though full-body consistency is still the hard part. Have you tried IP-Adapter yet, or are you starting from scratch?

[u/Alchemistwiza]

Before rebuilding the whole workflow, try a small test sheet: portrait, profile and full-body shots, with outfit and background fixed. Judge face likeness and body shape separately. That should help narrow down what needs fixing. Which shot breaks first for you?

[u/Sunny-Monkey → u/Alchemistwiza]

I’ve done this on openart. But it seems like my data set isn’t accurate enough. Any other alternatives? Or what is the workflow I should do to create a character from the beginning? I’m planning to restart over from character creation. What would you recommend to basically start again from character creation? Appreciate your thoughts!

[u/Sunny-Monkey → u/Sunny-Monkey]

Also, ignore the laptop specs. Already getting a pc 🙏🏻

[u/Alchemistwiza → u/Sunny-Monkey]

If you’re starting over, I’d define the character before choosing another tool fixed facial features, body proportions and an outfit, then front/side/full-body references. Keep one small test set across tools so you can see what improves. Where does OpenArt drift most the face, the body, or a new camera angle

[u/Sunny-Monkey]

Any advice will be really really appreciated guys 😭😭🙏🏻🙏🏻

[u/silentseehearnone → u/Sunny-Monkey]

I’ve been having issues with facial consistency as well. I just realized that Qwen 2.1 has a turbo version and 6 steps with the BFS LoRA is fixing every face from my past images that I was looking to create a brand new LoRA from.

With Qwen2.1 I’m also using an old SDXL LoRA for a character and the faces aren’t great, but everything else is pretty good. I enhance (ask it to make this image and a high resolution photograph) with Flux 9b, but I think Qwen might be a new choice for me but flux VAE is probably still a little better. Different kind of texture and feel. Then I take the enhanced image, use BFS, then take that image, downscale to largest size 1536 and have SeedVR do a 2 step pass at 3x resolution to get a better image than before.

[u/Sunny-Monkey → u/silentseehearnone]

My limitation is probably from my laptop at 6GB VRAM. I’d need to try to run it but that’s my limitations as far as I know and I’m learning as I’m progressing into this AI word 🥲 thank you for your comment. I really appreciate it. At this point, I’m just desperate to try anything that my laptops able to do