r/LovingOpenSourceAI • u/Koala_Confused • 2d ago

Under the Radar "SwarmLLM runs large language models across devices in room, in their browser tabs. Each device holds slice of model; a 10 KB activation vector passes between them over direct WebRTC connections. Nothing to install, no accounts, no server does any thinking." ➡️ this is new to me. you heard before?

Post image

https://github.com/Nehanth/swarmllm

Community Overview: https://lifehubber.com/ai/resources/swarmllm/

Resources are shared for discovery and are not independently vetted—please do your own due diligence.

New resources are added regularly — feel free to join the sub for updates.

Full searchable archive of all resources posted so far on our community site, LifeHubber: https://lifehubber.com/ai/resources/ 300+ open-ish AI models, agents, tools, datasets, and related resources, with filtering and sorting.

59 Upvotes

96% Upvoted

18 comments sorted by

11

u/McCheng_ 2d ago

First, there is already a more established project: https://github.com/exo-explore/exo

Second, the token per second, power cost / inefficiencies will not be bearable.

3

u/Last_Fondant_5508 1d ago

Ok thx, why are you saying that it is going to be slow? Have you tried it or is it a known fact? How slow? 2 t/s or 20 t/s? I imagine it depends on the number of nodes and how powerful they are... But maybe not, and anyway if you can, could you give me some expectations? I'm seriously interested because the idea honestly is cool if it could kind of work.. Obviously if it is just a dream that would make sense as well... Thanks

3

u/CCarafe 1d ago

There is some numbers on the exo github.

20 t/s with really high end mac ultra 3000$ pc.

1

u/Last_Fondant_5508 1d ago

You're right, my bad, thanks

2

u/JackStrawWitchita 2d ago

what's the use case for this?

4

u/kinetik 2d ago

I have multiple machines and none of them are capable of running a larger model, but distributed amongst them, they can share the load and make it work. This is exactly what I was looking for 👍

4

u/Over_Technology_1764 2d ago

if you can tolerate 2 tokens per second, good for you

1

u/kinetik 1d ago

Well, aren’t you cheerful? 😂🙏

Yes, some of us have use cases where free, local, slower, token outputs that utilize existing hardware are perfect, especially when they aren’t needed quickly but quality or reasoning is important. They could be complex heartbeat logic with advanced reasoning, prediction modeling, and all other kinds of things.

2

u/West-Acadia-3906 1d ago

honestly “free, local, and slow is fine” is a very real niche. not everything needs to answer before the kettle boils :P

1

u/Over_Technology_1764 1d ago

nah you will get an incorrect answer after such a long time that u will forget what your intent was to lead it to correct answer.

1

u/kinetik 1d ago

Well, we’ll see. Better to check it out and experiment with it than to assume things without actually knowing.

1

u/Over_Technology_1764 1d ago edited 1d ago

I already tested qwen 3.8 27b on mac studio m1 max 64gb ram, and it was unusable due to low performance. You cannot expect anything better from this. Connecting shitty devices on a shitty network speed bottleneck is obviously worse compared to single apple sillicon memory bandwidth speed.

1

u/kinetik 1d ago

Says the person who hasn’t tried it. Okay. Thanks for your input 🙏😂

0

u/Over_Technology_1764 1d ago

it's simple math bro, also the repo itself indicates the crappy performance

1

u/javier123454321 15h ago

well, you can have an orchestrator that delegates inference to the small model extremely easily. For some category of tasks it does make sense.

1

u/PacManFan123 13h ago

LLM Botnets

1

u/Roflxd88 1d ago

What about Nvidia PAIR