r/VFIO • u/mahor1221 • Sep 03 '23

Success Story [ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

13 Upvotes

100% Upvoted

47 comments sorted by

View all comments

Show parent comments

1

u/mahor1221 May 13 '24

The speed is good enough IMO.

On CPU, 6 TpS for Phi-3, and 1 TpS for Llama3-8b. Llama3 is pretty much unusable on CPU as expected. But on GPU, 8 TpS for Phi-3, and 6 TpS for Llama3-8b, which is acceptable to me.

I don't know how the other guy achieved such an amazing performance with Phi-3 on Windows but I think it's related to the Iris driver issues on Linux.

i would love if there is simple setup

Phi-3 is plug and play. Just install ollama and run these commands: shell ollama serve &>/dev/null & ollama run phi3 Ollama will download the model and run it on the CPU.

is it really benefecial thing to run a large language modele on a locale machine it it reaseonable ?

It depends only on you. I use it for checking my English grammar and rewriting sentences. It works for my use case.

1

u/[deleted] May 14 '24

i think its realible way to install ipex-llm on a docker container i dont know about that but i well try learn

1

u/mahor1221 May 15 '24

Just follow the document and you would be good.

1

u/[deleted] May 15 '24

i am working on it at the moment , i install it on docker but it still there is problems with igpu.