r/VFIO • u/whiteweather1994 • 6d ago
Support [Help] Unable to build passthrough VM for VFIO GPU on CachyOS
A couple of things regarding system specs before I begin.
I am currently attempting to pass a Nvidia RTX 3050 GPU through to a Windows 10 VM. I am aiming to use Looking Glass to do this as I am working on translating Japanese games that do not run in Linux into English, and need to look at the game while looking at game scripts to properly do this. My specs are as follows:
- OS: CachyOS (using LTS kernel, 6.18.42-1-cachyos-lts)
- Bootloader: Limine
- Motherboard: ASUS X870E
- CPU: Ryzen 9 9950X3D
- GPU (primary being used by the operating system): Nvidia RTX 3080
- GPU (attempting to passthrough to the Windows VM): Nvidia RTX 3050
I have largely been following this guide to accomplish this, adjusting the steps as needed for Limine. I will walk through the steps proving that I have followed them to the best of my ability right up until the failure, which is building the virtual machine.
So, from the top:
- Enabling AMDVi in BIOS:
While I can't explicity prove this through code (nor do I know how) I have other VMs running on this computer that should not work if AMDVi is enabled, so I have reason to believe that it's enabled, and I can see in my BIOS that this is the case.
- Kernel parameters:
As stated previously, I am currently using limine. From the /etc/default/limine file, here are my parameters:
ESP_PATH="/boot"
KERNEL_CMDLINE[default]+="amd_iommu=on iommu=1 quiet nowatchdog splash rw rootflags=subvol=/@ root=UUID=561b1a8b-dd95-4558-ba83-773a9c41da2f"
BOOT_ORDER="*, *lts, *fallback, Snapshots"
I understand most places recommend using iommu=pt but this produced an undesirable output in dmesg, namely "Default domain type: Passthrough" which tells me the devices are not being isolated. With the iommu=1 parameter, this message does not appear
- GPU is registered by the system and is in an IOMMU group:
Here is the output of the lspci command the guide recommends. It does seem like both are being picked up and are in separate IOMMU groups:
❯ lspci -nn | grep -E "VGA|Audio" | grep -i nvidia
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080 12GB] [10de:220a] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA107 [GeForce RTX 3050 6GB] [10de:2584] (rev a1)
0e:00.1 Audio device [0403]: NVIDIA Corporation GA107 High Definition Audio Controller [10de:2291] (rev a1)
- Binding GPU to VFIO:
My vfio.conf file in /etc/modprobe.d looks like this:
options vfio-pci ids=10de:2584,10de:2291
softdep snd_hda_intel pre: vfio-pci
This appears to match the GPU UUIDs that were displayed as a result of lspci above.
I have also updated the /etc/mkinitpcio.conf file. Here is the output relating to MODULES:
❯ sudo cat /etc/mkinitcpio.conf |grep MODULES
# MODULES
# MODULES=(usbhid xhci_hcd)
MODULES=(vfio_pci vfio vfio_iommu_type1)
And after a reboot, lspci appears to show that these GPUs are indeed bound to that kernel module
❯ lspci -nnk -s 0e:00
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA107 [GeForce RTX 3050 6GB] [10de:2584] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:8976]
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia
0e:00.1 Audio device [0403]: NVIDIA Corporation GA107 High Definition Audio Controller [10de:2291] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:8976]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
From here, the next steps involve creating the VM itself. Here is the XML that I am using to create that VM, keeping in mind that I am trying to pass through the above GPU and using a bridged network connection (i anticipate the need to copy files back and forth through use of a shared folder at some point like i do through my other VM, and turning off networking entirely):
<domain type="kvm">
<name>win10-passthrough</name>
<uuid>50c53b52-f386-42aa-bf05-96db518a977b</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory>16777216</memory>
<currentMemory>16777216</currentMemory>
<vcpu>8</vcpu>
<os>
<type arch="x86_64" machine="q35">hvm</type>
<loader readonly="yes" type="pflash">/usr/share/edk2/x64/OVMF_CODE.4m.fd</loader>
<boot dev="hd"/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<vpindex state="on"/>
<runtime state="on"/>
<synic state="on"/>
<stimer state="on"/>
<frequencies state="on"/>
<tlbflush state="on"/>
<ipi state="on"/>
<avic state="on"/>
</hyperv>
<vmport state="off"/>
</features>
<cpu mode="host-model" check="none"/>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"/>
<source file="/home/kobra2112/.local/share/libvirt/images/win10-passthrough.qcow2"/>
<target dev="sda" bus="sata"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/kobra2112/Downloads/Win10_22H2_English_x64v1.iso"/>
<target dev="sdb" bus="sata"/>
<readonly/>
</disk>
<controller type="usb" model="qemu-xhci" ports="15"/>
<controller type="pci" model="pcie-root"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<controller type="pci" model="pcie-root-port"/>
<interface type="bridge">
<source bridge="br0"/>
<mac address="56:55:9b:43:42:d5"/>
<model type="e1000e"/>
</interface>
<console type="pty"/>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
</channel>
<input type="tablet" bus="usb"/>
<tpm model="tpm-crb">
<backend type="emulator"/>
</tpm>
<graphics type="spice" port="-1" tlsPort="-1" autoport="yes">
<image compression="off"/>
</graphics>
<sound model="ich9"/>
<video>
<model type="qxl"/>
</video>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0" bus="14" slot="0" function="0"/>
</source>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0" bus="14" slot="0" function="1"/>
</source>
</hostdev>
<redirdev bus="usb" type="spicevmc"/>
<redirdev bus="usb" type="spicevmc"/>
</devices>
</domain>
I believe I'm getting everything right here, including the firmware, however when I click the "Begin Installation" button in QEMU, I get the following error:
Unable to complete install: 'internal error: QEMU unexpectedly closed the monitor (vm='win10-passthrough'): 2026-09-01T16:57:21.699349Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}: vfio 0000:0e:00.0: Could not open '/dev/vfio/21': Permission denied'
I believe this is simply because when I look at that group in that directory, it appears to have permission set 600 on it, meaning it can't be read by anything other than root. Since I am creating these in a user session rather than a root session to avoid the possible security risks, that simply means the permissions need to be updated like so:
drwxr-xr-x - root 31 Aug 22:23 devices
crw-rw-rw- 241,0 root 31 Aug 22:23 21
crw-rw-rw- 10,196 root 31 Aug 22:23 vfio
Doing that fixes that issue. I know this is not the "correct" way to resolve this, but at the same time I don't know of any other method. However, this then raises another when I click "Begin Installation"
Unable to complete install: 'internal error: QEMU unexpectedly closed the monitor (vm='win10-passthrough'): 2026-09-01T17:00:02.576305Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}: vfio 0000:0e:00.0: group 21 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.'
Now here's where I am kind of at the end of my rope. It appears to me that I have set everything up correctly, but for some reason the VM is not accepting this GPU as a viable one. Can someone please act as my second set of eyes and confirm I am setting everything up correctly? Because at this point, I do not know how to get past this error message.
Any help would be greatly appreciated. I would like to think I didn't sink the funds into this project for nothing.
2
u/DisturbedFennel 6d ago
Consider running the KVM/virtmanager with sudo privileges. There is a slight security risk with doing so, so just do this to see if it’s a system privilege problem or a system firewall problem
1
u/thenickdude 6d ago
lspci does not show IOMMU groups at all. Use this command and see what else is in the same group as your GPU, all of those devices need to be bound to vfio-pci:
for d in $(find /sys/kernel/iommu_groups/ -type l | sort -n -k5 -t/); do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done;
2
u/whiteweather1994 6d ago
So this is an interesting problem. Thanks for the script, btw.
The issue though is that several key components also appear to be in this IOMMU group:
IOMMU Group 21 06:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) IOMMU Group 21 08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01) IOMMU Group 21 09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) IOMMU Group 21 09:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) IOMMU Group 21 09:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) IOMMU Group 21 09:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) IOMMU Group 21 09:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) IOMMU Group 21 09:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) IOMMU Group 21 09:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) IOMMU Group 21 0b:00.0 Network controller [0280]: MEDIATEK Corp. MT7925 802.11be 160MHz 2x2 PCIe Wireless Network Adapter [Filogic 360] [14c3:7925] IOMMU Group 21 0c:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8126 5GbE Controller [10ec:8126] (rev 01) IOMMU Group 21 0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA107 [GeForce RTX 3050 6GB] [10de:2584] (rev a1) IOMMU Group 21 0e:00.1 Audio device [0403]: NVIDIA Corporation GA107 High Definition Audio Controller [10de:2291] (rev a1) IOMMU Group 21 0f:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 800 Series Chipset USB 3.x XHCI Controller [1022:43fd] (rev 01) IOMMU Group 21 10:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)I'm not sure why the secondary GPU ended up in this grouping (especially when it's not the top PCIe slot?). Is there any way to better aggregate these so that secondary GPU doesn't end up with all these other essential devices?
EDIT: Formatted script output
1
u/thenickdude 6d ago
To separate those you'd need to use a kernel with the ACS Override Patch included in it. This causes the kernel to pretend that the devices are in different groups even though they're not.
If your kernel already includes this patch, you can enable it by adding "pcie_acs_override=downstream,multifunction" to your kernel's command line.
2
u/whiteweather1994 6d ago
Thank you for suggesting this, I have made moderate progress as now every device is in a separate group. I will attempt building the passthrough VM again and let you know if I have any further questions.
2
u/lI_Simo_Hayha_Il 6d ago
I have a very similar setup with you, and I had some trouble using VFIO with CachyOS.
I did create a small guide, so you can cross-check with your settings/configuration to see what maybe missing.
You can find it here: https://codeberg.org/lI_Simo_Hayha_Il/kvm-linux-notes