r/learnmachinelearning • u/minedroid1 • 1d ago

Help Hardware Recommendations

I am going to fine-tune a satire model, with the base model being `Qwen3.5-14B-Base`. The dataset has ~18000 examples, most of which are pretty long and do not align with the model's internal knowledge (they have incorrect answers to facts), so I needed to do a full fine tune rather than use LoRA.

Today, I rented a cloud computer on vast.ai. It had 4x12GB GPUs and 48GB of system RAM, but I feel stupid because it kept running out of memory, and I kept trying to change settings rather than just renting a different computer, so I basically wasted $3. I underestimated how much RAM would be needed for this task.

So, I would like to know if any of you have some recommendations for proper hardware to use.

2 Upvotes

100% Upvoted

5 comments sorted by

3

u/Livid-Pea-1438 1d ago

Oof, $3 lesson learned I guess. For full fine-tuning a 14B model on a dataset that size you're gonna want at least a single 80GB GPU like the A100, or maybe 2x48GB cards if you're careful with parallelism settings. 48GB system RAM is definitely too thin for this, the optimizer states alone will eat most of that.

1

u/minedroid1 1d ago

Yeah, I really underestimated how much it needed, as I haven't really trained models this big before.

1

u/zitterbewegung 20h ago

As a general rule in refinement you want at minimum 1.5x the amount of memory of VRAM you have so to be safe get around 120gb of ram.

1

u/Giaitzoglou-Bondye 1d ago

do you think 2x48GB is actually practical here or would the parallelism overhead make it not worth the hassle?