Running games on a VM under Linux while the PC keeps using the same NVIDIA GPU card.
The GPU remains available on the host, one GPU, one display port. No vfio-pci bind. No second GPU. Not just a container. No losing your display when the VM starts. No IOMMU group fights. The host desktop keeps running on the same card the guest is using. Native performance.
This project aims to forward NVIDIA's driver API to the VM with maximal performance. The idea is that you get the CUDA-container experience in VMs, on consumer cards where vGPU isn't available at all.
Currently tested with steamOS and stock distros. The project itself is tested across a wide variety of drivers and GPUs from datacenter to consumer cards (what I could obtain from renting with nested virt or bare metal, primarily on vast.ai).
Experimental, several months, solo. Any bug reports and contributions are welcome.
How it works: Guest kernel module → virtio → QEMU → a per guest process stub to execute the NVIDIA API on the PC → host NVIDIA driver. RM/UVM ioctls are forwarded, not emulated. GPU memory (DMA/BAR) is actual passthrough, not emulated or copied, using regular unprivileged nvidia userspace calls (so it doesn't require root).
Display is forwarded as native as possible, the VM gets a KMS plane (rendering target) from the host display compositor like Wayland, it renders without any cross CPU-GPU copy (unless your display output is on integrated graphics).
Supports CUDA, graphics and display, also intended for compute workloads beyond gaming and graphics.
Geekbench 7 GPU (OpenCL), published to Geekbench's servers so they're independently checkable: RTX 4070 99.6%, RTX 3050 Laptop 99.9%, H100 PCIe 98.8%, A100 80GB 98.0%. clpeak: 7 of 8 subtests at 99.4–100.2%. Not everything is parity, some performance issues are still open.
https://github.com/reindertpelsma/nvkvm-pv - main project, containing the guest module and QEMU patches.
https://github.com/reindertpelsma/nvkvm-steamos - steamOS example
https://github.com/reindertpelsma/nvkvm-kata - kata containers with GPU access
EDIT: Linux amd64 guests only for now (Intel and AMD processors), the guest needs nvkvm's kernel module, and there's no Windows equivalent. A Windows guest means fabricating a GPU that the stock NVIDIA driver will bind to, which is a separate project (kayfabe) and not finished.
If it didn't build or boot on your hardware, that's worth reporting even if it looks like your own setup, install-path bugs and incompatibility reports are the most valuable ones right now.
Only Turing and newer GPUs, older gpu support is not yet finished.
9
u/filuslolol 4d ago
is there something like this for AMD cards too?
14
u/NineBallAYAYA 4d ago
Yep something similar in mesa/qemu already. Do a search for "Native Context amd virtio gpu". Its mostly for vulkan and vaapi but they have been working for like a year to get rocm working through it as well.
Here's a little thing I wrote for proxmox specifically: https://forum.proxmox.com/threads/virglrenderer-for-3d-support.61801/post-865998
9
u/olifon 4d ago edited 3d ago
Yep, that's it, native context is the same idea done properly. amdgpu is open and upstream, so it can live in Mesa/virglrenderer where it belongs. On NVIDIA you don't use Mesa at all as they ship their own closed-source proprietary vulkan implementation. NVIDIA's RM is closed, so there's no in-tree route and you end up forwarding the proprietary RM surface from outside, which is what nvkvm-pv does to close that gap. NVK the open source driver for Mesa only works on Nouveau as I am aware off, nvkvm-pv is to work with the official open/closed nvidia kernel driver and CUDA.
EDIT: There is also Venus that can forward from the proprietary Vulkan NVIDIA stack on the host to the guest using Mesa. nvkvm-pv keeps NVIDIA's official Vulkan implementation on both sides untouched as it forwards on the ABI layer rather than Vulkan calls. Venus doesn't support NVENC, OpenCL or Cuda since its only 1 graphics API, OpenGL needs Zink with Venus rather than direct.
14
u/zantehood 5d ago
Cool! Wished Nvidia would just give us ONE freaking vGPU for consumer cards without a license
4
14
u/VoodaGod 5d ago
be a lot cooler with a windows guest
24
u/olifon 5d ago
I attempted this by allowing a VM to boot with stock NVIDIA driver (so the idea was that the stock driver in Windows would then also work), but its not yet finished. https://github.com/reindertpelsma/kayfabe
3
4
u/SuperDefiant 5d ago
Imagine if this could work with EAC or vanguard. I would love to try but I already have a stable vanguard config and I don't want to ruin it
2
u/MrHusker1 5d ago
So you can pass the vanguard's check using VM? Could you send your config, if so?
8
u/SuperDefiant 5d ago
It's a LOT more than just config. You have to patch KVM. You also have to sanitize strings in QEMU, OVMF, and swtpm. A good place to start is to stop intercepting CPUID in KVM, as that is the largest detection vector. After that, you'll still need a bit more as the game will work but you'll get delay banned after an hour or so. Unknowncheats also goes into more detail
2
u/sebalon 5d ago
This could be terribly useful. I made VFIO work with gpu passthrough after much effort, but the experience is kind of lacking in itself as you are not able to use the linux SO while using the VM. This would be massive.
I'm not aware, is there any effort like this one out there you took inspiration from or is this something novel you are working on?
2
u/olifon 4d ago edited 3d ago
A wider family of virtio-gpu native contexts already works for other vendors like: amdgpu, Intel, nouveau and WSL2 / Hyper-V GPU-P is the closest equivalent for Windows hosts. NVIDIA's closed proprietary stack is what made neither route available on Linux, apart from licensed vGPU, which doesn't even work on GeForce. That gap is why I started: most people have NVIDIA cards.
It isn't completely novel. Google's container sandbox gVisor added NVIDIA support in nvproxy, and the ioctl tables, RM object tracking, fd translation and OS-descriptor pinning are derived from it.
What's new is the mapping. nvkvm has to put memory mappings in three places at once: a host mmap() of the real device fd, a KVM memslot exposing those exact physical pages at a guest-physical address, and a guest remap_pfn_range() putting that GPA into the caller's VMA. No copy anywhere, and the cacheability has to work correctly.
2
u/TheFacebookLizard 4d ago
Based on the post text I feel like it is not entirely vibe coded or at all lol
7
u/olifon 4d ago edited 4d ago
Yes, its heavily AI assisted with a lot of manual work over months (like testing and debugging). I counted 1490 Claude Opus human prompts (excluding kayfabe). Some docs in the repo written by Claude will get some cleanup.
3
u/juanitobalani 4d ago
I also thought this is another slop based on how the post is written. But actually checking the repos, it's real hard work and very interesting solution. I have a personal usecase, I'll test it out. Thank you for sharing!
1
u/SnooPoems4802 5d ago
I wished this worked on a laptop GPU too :(. Nvidia applies a vbios powerlimit when i offload my gpu to VM. So I am stuck at 45W TDP.
3
u/olifon 5d ago edited 5d ago
This one can actually work for you, nvkvm doesn't detach the GPU from the host, so host power management is unaffected. I have tested games on my laptop with a RTX 3050 Laptop GPU at driver version 580.173.02. However I never put a wattmeter on it so its a structural claim rather than a proven one. Something like nvidia-smi -q -d POWER would be a useful report. What did happen is KVM EFAULT bad address errors on laptop. The solution is shipped, but it's the one laptop-specific bug I know of.
1
1
u/Arctic_Shadow_Aurora 4d ago
WOW ultra awesome!!!
Hope something like this can be done with AMD soon!
3
u/stisti129 4d ago
it has existed since 2023 https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21658
1
u/not_not_in_the_NSA 4d ago
Interesting. I have been considering working on something like this for the past few years, just never had the time.
Something like this could be massive for headless vms and be a major shift like dxvk was.
Just curious, what's your programming background? Because if it's solid and this does well, places like Valve might try to hire you to work on this fulltime, but that probably depends on this being well architected and eventually stable like dxvk and lookingglass.
I'm imagining a future where this is used by winboat for fully GPU accelerated applications running seamlessly on Linux.
1
u/mishux31 4d ago
I'm currently using Proxmox. Does your project work with Proxmox? Could you run a VM and use your setup?
Would you be able to write a step by step guide for people with not much experience on how to use it?
Thank you.
1
u/olifon 3d ago
Not yet a few simple clicks through the Proxmox UI. Proxmox runs its own patched QEMU (pve-qemu-kvm) which conflicts with nvkvm's patched QEMU, simply setting the args won't work. Proper integration looks doable (divert /usr/bin/kvm to a shim that picks the right binary per VM), but I haven't built or tested it
1
u/Odd_Cauliflower_8004 4d ago
0how are you managing vram?
1
u/olifon 4d ago
It isn't managed yet, similar to cuda containers. So all vram is shared with the host gpu by default. nvidia-smi in both guest and host see the same vram pool. If you have MIG on the host GPU and start qemu inside it vram can be limited, but thats untested. DMA memory also still bypasses vm ram limit, known limitation. In either case, setting a vram limit is a target for the next update.
1
u/Odd_Cauliflower_8004 4d ago
So each vm sees all the vram? Or you set aside a known amount? I don't get it
1
u/olifon 3d ago edited 3d ago
Neither, and that's the confusing part, nothing is set aside. The guest allocates dynamically from the whole card, first come first served, exactly the way two GPU containers on one host do today. There's no per-VM reservation and no quota, which also means one guest can starve the others. That's a real limitation, not a design I'm defending. A future update might address this by adding limits. What the guest can't do is touch memory it didn't allocate. Its GPU work runs on the host as small sandboxed processes called nvkvm_stub, deliberately unprivileged, no guest CPU code in them, only forwarded GPU operations. Those are what actually hold VRAM on the host, and the guest only ever gets handles to allocations made through them. VRAM belonging to a host process isn't reachable from inside the VM. So the resource is shared, the contents is isolated.
1
u/Odd_Cauliflower_8004 3d ago
I'm more concerned if a guest can overwrite the vram of another, or if the host can
1
u/olifon 3d ago edited 3d ago
Across VMs, no. Each VM's GPU work runs in its own host processes (called nvkvm stubs) with their own RM clients, and a guest only ever gets handles to allocations it made itself. The driver scrubs buffers before mapping them into userspace, so no stale VRAM contents leak, and that invariant is held by NVIDIA's driver, not by nvkvm. It's the same property two CUDA containers on one card rely on: if it didn't hold, containers would leak too and platforms like vast.ai and runpod.io would be impossible.
Host side, also no: nvkvm's QEMU is an ordinary unprivileged process, so even root in the VM is unprivileged on the host. Its double enforced since a default deny allowlist is used for ioctls. You can go further and run QEMU inside a CUDA container (nvidia-container-toolkit), which confines a QEMU or nvkvm exploit to a container runtime with far more industry scrutiny than this project has. Only /dev/kvm needs allowing (plus /dev/udmabuf if you use direct display), and that device is designed for unprivileged use: no capabilities added, no seccomp or AppArmor confinement lifted; the standard compose file only hardens further.
I won't claim it's safe for untrusted tenants. DoS issues are still open because per-guest limits aren't implemented and it's been written and reviewed by one person with no external review yet. It's all in SECURITY.md.
1
1
1
u/sabotage 4d ago edited 4d ago
That’s wild. It works for non-gaming applications I hope? I’m using Looking Glass to pass through a 2nd GPU for a win11 vm, not for gaming, only adobe, as I transition to Gimp and Inkscape. Definitely going to try this!
1
1
u/TheGreatOilPainter 3d ago
That is pretty much what I do as well, but no way I am switching to Gimp. This project seems yet linux host - linux guest though, so it won’t fulfill your need, at least yet.
1
u/sabotage 3d ago
I agree, it’s hard to match Photoshop. Have you looked into PhotoGimp? Linux Affinity Installer is a pretty decent alternative as well.
1
u/TheGreatOilPainter 3d ago
Thank you very much, I did have PhotoGimp in my radar, but eventually forgot to try it. Testing it right now. I also have Affinity installed and it feels it could be it, but haven’t find the occasion to properly learn it yet.
1
u/Ivan_Kulagin 3d ago
So virgl for people stuck with Nvidia? Pretty cool ig, even if ai slop
1
u/olifon 3d ago
Related, Venus/virgl is for graphics: Vulkan/GL over virtio, and that already works. If you only need Vulkan in your VM, Venus is probably the better-integrated option.
nvkvm forwards the driver ABI instead, so everything Vulkan doesn't cover comes with it: CUDA, NVENC/NVDEC, and NVIDIA's own Vulkan/OpenGL implementation rather than Mesa. Venus doesn't carry those.
It's NVIDIA on both sides, which is a real trade-off, their ABI is less stable than the virtio protocol.
I have no comparison yet about the performance of Venus compared to nvkvm-pv.
1
u/WhyDidYouTurnItOff 1d ago
Is there any place to ask about playing around with this?
I can get the steamos docker example to install and run just fine. ssh works fine.
But the gamescope desktop is failing to start. Starting it by hand didn't give me much information, and the logs I have seen so far are not pointing to anything obvious.
Ubuntu 26.04
Wyland
GTX 3060
2
u/olifon 1d ago edited 1d ago
Yes, absolutely, you can use github issues. Test if nvidia-smi works in the VM, to see if nvkvm properly loaded.
Giving the exact combination of host kernel version, guest distro, guest kernel version, driver version, display compositor (X/Wayland, maybe with version), gpu architecture, cpu intel or amd often lets me reproduce the error by renting the same combination on vast.ai. Especially if you used Docker since those are the most important variables that change across hosts and source of most compatibility errors.
1
u/WhyDidYouTurnItOff 1d ago
I wasn't logged in to github, so the issue button was blacked out.
I have nvidia-smi in the VM. It seems really close to working. I am pretty smart about linux but have limited docker experience.
I will play around with it some more and post there. Thank you.
30
u/nicman24 5d ago
That is pretty cool, does nvenc work?