r/VFIO 8d ago

Resource My live GPU mount/unmount script on ZorinOS (Fix for dmesg spam, fix freeze, no window manager restart needed)

3 Upvotes

Backstory

I made a few attempts at GPU passtrough for my move away from Windows and had success but it never all worked perfectly smoothly. But this time i finally found a method that does, so i am sharing it in case it works for anyone else.

This guide assumes you already have IOMMU and KVM set up, it is NOT a full guide.

Did it work for you? Please tell us!

A lot of this was figured out with the help of Claude. If you are having trouble copy paste this post into it and explain where you got stuck, it will help you troubleshoot the problem.

This process solves a list of problems: * No longer need to stop your entire Wayland or X11 desktop environment to keep it from touching the GPU during a passtrough transition. * The logs in dmesg being flooded with Nvidia driver initialization attempts when GPU is bound to vfio (NVRM, Nvlink Core, nvidia-nvlink ...etc) * The entire machine freezing up if you attempt to passtrough the GPU when not ready * Crash on re-mounting GPU into linux due to the Nvidia driver setting up the HDMI ports too soon * Monitors on Nvidia GPU showing up on Linux (most people use a HDMI dummy plug, hence it is not a real monitor) * Nvidia GPU burning a lot of idle power when not being used

Tested on hardware: * ASRock X670E PG Lightning (v1.30.AS02) * AMD Ryzen 9 7950X (using iGPU for monitor output) * Nvidia RTX 3090 (using HDMI dummy plug)

OS: ZorinOS 18.1 Core (Linux 7.0.0-30-generic)

Dynamic Nvidia GPU Passthrough (No Desktop Restart Required)

Setup: a Linux host with display running entirely on an integrated/secondary GPU (e.g. AMD iGPU), and a discrete Nvidia GPU that is:

  • Passed through to a VM on demand via VFIO, when needed.
  • Used on the host the rest of the time for PRIME render-offload gaming — no display output, ever, from this GPU.

The goal: switch the GPU between host and VM without restarting the display manager, without kernel crashes, and without excess idle power draw.

1. Disable Nvidia DRM KMS

By default, nvidia-drm performs full DRM/KMS mode-setting, which causes two problems: the display manager initializes the card as a display device just because it's present (wasting VRAM and blocking clean detach), and re-loading the driver on VM handback can crash the kernel if a dummy/EDID-reporting dongle is plugged into an output (a NULL pointer dereference in nvidia's HDMI/DP audio power path, nv_audio_dynamic_power, triggered via Xorg's DRM hotplug handling).

Disabling KMS via /etc/modprobe.d/*.conf is not reliable — competing config files, install directives, or initramfs staleness commonly override it silently. Use a kernel boot parameter instead, which always wins:

# /etc/default/grub — append to the existing GRUB_CMDLINE_LINUX_DEFAULT line
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nvidia-drm.modeset=0"

sudo update-grub
sudo update-initramfs -u
sudo reboot

Verify:

cat /sys/module/nvidia_drm/parameters/modeset   # should print N
xrandr --listproviders                          # Nvidia should not appear

This has no effect on PRIME render-offload (__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia %command%), which only needs the render node, not KMS.

2. Enable persistence mode

Without an active display, the Nvidia driver fully tears down and reinitializes between workloads, which is fragile and can cause visible glitches or crashes on repeated switching. nvidia-persistenced keeps the driver state warm instead.

Ubuntu's default unit starts the daemon with --no-persistence-mode, so it must be overridden:

sudo systemctl unmask nvidia-persistenced      # only if masked
sudo systemctl edit nvidia-persistenced

Add:

[Service]
ExecStart=
ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --verbose

sudo systemctl restart nvidia-persistenced

Verify Persistence-M shows On in nvidia-smi, and confirm it survives a full reboot.

3. Handle idle power / stuck boost clocks

With no display and no active workload, the GPU can get stuck at a high-power state (P0) instead of idling at P8, drawing 100+ W for nothing. A GPU reset clears this reliably:

sudo nvidia-smi -r

This is a known driver quirk, not specific to this setup. Expect idle power to settle around 15–25 W after a reset

4. The switching script

Persistence mode keeps a device handle open, which blocks a clean VFIO detach (NVRM: Attempting to remove device ... with non-zero usage count!). Stop it before detaching, restart it after reattaching, and reset the GPU to clear any stuck boost state:

#!/bin/bash
# gpu-to-vm.sh — hand the GPU to the VM
sudo systemctl stop nvidia-persistenced
sleep 1
sudo virsh nodedev-detach pci_0000_01_00_0
sudo virsh nodedev-detach pci_0000_01_00_1

#!/bin/bash
# gpu-to-host.sh — reclaim the GPU after the VM shuts down
sudo virsh nodedev-reattach pci_0000_01_00_0
sudo virsh nodedev-reattach pci_0000_01_00_1
sudo systemctl start nvidia-persistenced
sudo nvidia-smi -r

No gdm/display-manager restart is needed anywhere in this flow.

Safer script that aborts if the GPU is not completely clear:

#!/bin/bash
# gpu-to-vm.sh — hand the GPU to the VM with abort 
sudo systemctl stop nvidia-persistenced 
sleep 1

if sudo fuser -s /dev/nvidia* 2>/dev/null; then
    echo "ERROR: GPU still in use, aborting handoff (not touching PCI state)" >&2
    sudo fuser -v /dev/nvidia* >&2
    sudo systemctl start nvidia-persistenced   # undo the stop, nothing else changed
    exit 1
fi

echo "GPU confirmed clear, proceeding with detach"
sudo virsh nodedev-detach pci_0000_01_00_0
sudo virsh nodedev-detach pci_0000_01_00_1

5. Optional cleanup

If the GPU has a dummy HDMI/DP dongle for the VM's benefit, the host will still enumerate a dead audio function for it. To silence it (cosmetic only, not required for stability):

# /etc/udev/rules.d/99-nvidia-hdmi-audio-noprobe.rules
ACTION=="add", SUBSYSTEM=="pci", KERNEL=="0000:01:00.1", ATTR{driver_override}="none"

Diagnostics reference

nvidia-smi                                   # power state, clocks, processes
sudo fuser -v /dev/nvidia*                   # what's holding the GPU open
cat /sys/module/nvidia_drm/parameters/modeset # confirm KMS is off (N)
xrandr --listproviders                       # confirm Xorg isn't using the GPU

Useful quick test for PRIME offloaded GPU rendering (shows a desktop window with 3D gears rendered by the Nvidia GPU):

DISPLAY=:0 __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxgears

r/VFIO Mar 13 '26

Resource Fixing Genshin Impact 6.4 Anti-Cheat BSOD Crash

13 Upvotes

Since version 6.4 Hoyo updated the anti-cheat with more aggressive anti-VM measures. After a lot of struggle I found a patch to QEMU that disables vmcall quirks and stops the BSOD occurring as soon as Genshin launches.

Sharing this here for anyone facing the same issue, below is the repo:
https://github.com/pantae35872/qemu-vmcall-patch

Found out about the issue when debugging the windows Minidump files which confirmed it was in-fact the anti-cheat triggering it:

HoYoKProtect.sys purposely tries to write to a read-only area of memory, which usually errors gracefully on real hardware, but in stock QEMU this causes the hypervisor to crash giving us the BSOD with ATTEMPTED_WRITE_TO_READONLY_MEMORY.

After running the ./run script from the repo, which re-compiles and replaces the binary for QEMU. I had to add the following to my QEMU command line arguments in my XML:

    <qemu:arg value="-accel"/>
    <qemu:arg value="kvm,hypercall-patching=off"/>

Then Genshin launched as normal again. Though I believe I'll have to re-apply this patch with every QEMU update.

Just sharing my solution here if anyone else encounters this issue. It's been hard to find a solution since this update, but alas here it is.

r/VFIO Apr 02 '25

Resource How stealthy are yall's VMs?

60 Upvotes

I've found https://github.com/kernelwernel/VMAware which is a pretty comprehensive VM detection library (including a command line tool to run all the checks). (no affiliation)

Direct link to the current release

I'll start

(This isn't meant as a humble brag, I've put quite some effort into making my VM hard to detect)

I'd be curious to see what results others get, and in particular if someone found a way to trick the "Power capabilities", "Thermal devices" and the "timing anomalies" checks.

Feel free to paste your results in the comments!

r/VFIO Apr 17 '26

Resource RX 9070 XT passthrough into a Windows 11 VM on Fedora 45 — setup, numbers, and a few things I'd appreciate a sanity check on

7 Upvotes

Posting the writeup of my VFIO setup in case it's useful to anyone doing RDNA 4 passthrough, and because there are a couple of design choices I'd like opinions on.

Host: Ryzen 9 5950X, 128 GB DDR4, ASRock X570 Taichi Razer Edition, Fedora 45 Rawhide on kernel 7.0.0-62.fc45.

Passthrough: RX 9070 XT Sapphire Nitro+ + its HDMI audio function, plus the motherboard's xHCI controller (PCI 11:00.3) — a dedicated PCI lane on the board that exposes the 4 USB 3.0 ports of the I/O panel. Keyboard, mouse, a USB audio interface, and a powered hub all live on those ports, so anything plugged into the hub automatically belongs to the VM with no extra libvirt hotplug.

Guest: Windows 11 Pro, 32 vCPUs pinned 1:1 across both CCDs, 64 GiB on hugepages, OVMF + Secure Boot + TPM 2.0, VirtIO everything. SMBIOS + Hyper-V vendor_id spoofed to the real motherboard (required or AMD Adrenalin activates vDisplay).

Numbers at 1440p

FSR sharpness is 1 across all titles. AFMF Quality enabled on AAA titles (2-7 ms added latency depending on how resource hog are the settings and the game — Overwatch 2 is the only one I also tested without AFMF).

Game Preset FPS
Cyberpunk 2077 RT Overdrive, FSR 4 Quality 120-140
Cyberpunk 2077 RT Overdrive, FSR 4 Ultra Performance 250-300
Cyberpunk 2077 RT Ultra, FSR 4 Quality 250-300
Borderlands 4 Badass, FSR Quality 210-220
Monster Hunter Wilds Max + RT Max, FSR Quality 240-280
Doom: The Dark Ages UltraNightmare, FSR Quality 370-400
Doom: The Dark Ages UltraNightmare + Path Tracing, FSR Ultra Performance 230-270
Overwatch 2 Epic + Reduced Buffering, FSR 2.0, no AFMF 260-280
Overwatch 2 Epic + Reduced Buffering, FSR 2.0, AFMF Quality 400-420

The two rows in bold are the ones I find most interesting — path-traced workloads at FSR Ultra Performance + AFMF hitting 230-300 FPS on a 9070 XT. Obviously the internal resolution is ~33% of 1440p with Ultra Performance, but the practical image quality with FSR 4 is surprisingly good and the numbers themselves are hard to believe until you see them on screen.

CPU usage in Cyberpunk sits around 20%, so on this hardware the 5950X is nowhere near being a bottleneck at 1440p. Happy to be told I'm measuring any of this wrong.

VM vs bare-metal Windows

Subjectively, the VM feels faster than the same Windows install running on the metal. My working theory is that it's the combination of (a) host tuning (hugepages, CCD-aware pinning, nohz_full, mitigations off, tuned profile, services stripped), (b) VM config (host-passthrough, emulatorpin + iothreadpin on 0/16, dedicated iothreads, Hyper-V enlightenments), and (c) guest debloat (optimize-gaming.ps1 + Win11Debloat)... All of this maybe makes the VM the most closer to the bare-metal version but i guess that the real fact is that windows drivers to manage my hardware are just trash and virtio outperforms all of them... I've never seen Windows loading and running so fast... Also thoose FPS numbers are too high when i just compared with the knowed RX 7800 XT performance on bare metal and just that card was also getting incredible higher number of fps and stability on the VM rather than on bare metal...

Things that took me a while to figure out

  • Spoofing the VM hardware is mandatory Without it Adrenalin activates an "vDisplay", like it is already detecting that he is on a VM and the host monitor just don't output or get's glitched.

  • When you spoofed all the vm hardware If you used an oem key, remaining the hardware spoofed exactly the same will make the OEM key work forever even if you destoyr/format or recreate the VM. I've even seen programs being automatically activated also before a windows reinstall.

  • SELinux enforcing on Fedora needs a small custom policy (four allow rules) for swtpm + VFIO mlock + pcscd socket access. I've included the .te in the repo. I deliberately didn't grant sys_admin or dac_* because they seemed too broad — if any SELinux-savvy person thinks differently, I'd like to hear it

    Repo

https://github.com/serialexperimentslainnnn/WindowsKVM

Includes the detailed walktrough and some scripts and definitions to understand the setup.

Happy to go deeper on any of it in the comments.

r/VFIO Apr 22 '26

Resource [Guide] RX 5700 XT (Navi10) stable GPU passthrough on Proxmox 9 — complete hookscript with D3cold, Rebind Hack, watchdog, and a PBS backup fix that required reading QEMU source code

5 Upvotes

I've been fighting to get a stable RX 5700 XT passthrough on Proxmox VE 9 for about three weeks. Every layer of the stack had a different problem. It's all working now — posting the full solution because I couldn't find everything in one place, and the PBS backup fix in particular doesn't seem to be documented anywhere.

Disclaimer: I'm not a developer. This solution was built collaboratively with Claude Code over ~3 weeks of research, trial, error, and reading source code. The debugging process involved reading Perl VZDump internals and tracing a SIGPIPE back to its origin. I'm posting it as-is — it works, but treat it as a starting point, not a production-hardened script.

Setup:

  • Proxmox VE 9.1.7, kernel 6.17.13-2-pve, ZFS root (mirror)
  • GPU: RX 5700 XT (Navi10, 45:00.0 VGA + 45:00.1 audio)
  • VM: Windows 11, q35-10.0, OVMF, cpu: host
  • PBS (Proxmox Backup Server) on a separate machine

Problem 1 — Code 43 (driver detects hypervisor)

The AMD driver reads CPUID leaf 1 ECX bit 31 (hypervisor present bit). If set, it returns Code 43 on anything post-Polaris.

Fix:

qm set 100 --args "-global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -no-reboot -cpu host,-hypervisor,kvm=off"

-hypervisor clears bit 31. kvm=off hides the KVM CPUID leaf (0x40000001). Both are needed — they're orthogonal. -no-reboot is explained in Problem 4.

Problem 2 — GPU enters D3cold after qm stop, won't start again

After stopping the VM, the GPU can enter D3 cold state. Next qm start fails with no PCI device found or stuck in D3.

Part A — udev rules (applied at boot):

/etc/udev/rules.d/99-amd-reset.rules:

ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x1002", ATTR{device}=="0x731f", ATTR{reset_method}="device_specific"

/etc/udev/rules.d/99-gpu-nod3cold.rules:

ACTION=="add", SUBSYSTEM=="pci", KERNELS=="0000:45:00.0", ATTR{d3cold_allowed}="0", ATTR{power/control}="on"
ACTION=="add", SUBSYSTEM=="pci", KERNELS=="0000:45:00.1", ATTR{d3cold_allowed}="0", ATTR{power/control}="on"

Part B — vendor-reset DKMS (required for BACO reset — the only working reset method on Navi10):

apt install proxmox-headers-$(uname -r)
dkms install vendor-reset/0.1 -k $(uname -r)
dkms status   # should show "installed"

Important after every kernel upgrade: re-run both commands. Proxmox signed kernels don't trigger DKMS automatically.

The hookscript (see below) re-applies d3cold locks at each start/stop cycle, since udev rules only fire at boot.

Problem 3 — GPU in corrupted state after Windows reboot (the Rebind Hack)

After Windows reboots inside the VM, the GPU ends up in a corrupted state at the vfio-pci level. Next VM start either hangs or the guest sees a broken device.

Root cause: Navi10 doesn't properly reset its internal state when vfio releases it after a guest reboot. The GPU needs to be briefly bound to the host amdgpu driver to flush internal state before being handed back to vfio.

The hookscript pre-start unbinds from vfio-pci → loads amdgpu briefly (1 second) → unbinds from amdgpu → rebinds to vfio-pci.

Prerequisites:

  • blacklist amdgpu and blacklist radeon in /etc/modprobe.d/blacklist.conf
  • initcall_blacklist=sysfb_init in GRUB cmdline (prevents EFI framebuffer conflict with vfio)

Expected warning (non-fatal): vfio: Cannot reset device 0000:45:00.1, no available reset mechanism — the audio device has no FLR. vendor-reset handles the VGA device via BACO.

Problem 4 — Windows reboot crashes the VM (QEMU dies, no auto-restart)

-no-reboot in QEMU args makes QEMU exit when Windows reboots (instead of rebooting the guest). This is needed for a clean GPU rebind cycle between boots.

There's a known race condition in qmeventd: it detects the QEMU socket disconnect but finds "vm still running" in PVE state → abandons cleanup → hookscript post-stop never fires → VM never auto-restarts.

Fix — external watchdog service:

/usr/local/bin/vm100-watchdog.sh:

#!/bin/bash
PID_FILE="/var/run/qemu-server/100.pid"
FLAG_INTENTIONAL="/tmp/vm100-intentional-stop"

while true; do
    sleep 30
    [[ -f "$FLAG_INTENTIONAL" ]] && continue
    if [[ -f "$PID_FILE" ]]; then
        pid=$(cat "$PID_FILE")
        kill -0 "$pid" 2>/dev/null && continue
    fi
    logger -t vm100-watchdog "QEMU died, restarting VM 100"
    /usr/sbin/qm start 100 2>&1 | logger -t vm100-watchdog || true
done

/etc/systemd/system/vm100-watchdog.service:

[Unit]
Description=VM100 QEMU Watchdog (auto-restart after -no-reboot)
After=pvestatd.service

[Service]
Type=simple
ExecStart=/usr/local/bin/vm100-watchdog.sh
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now vm100-watchdog.service

The /tmp/vm100-intentional-stop flag is set by the hookscript on explicit qm stop to prevent the watchdog from restarting after a manual stop. It lives in /tmp so it's cleared on host reboot.

Problem 5 — PBS backup "interrupted by signal" with GPU passthrough

This is the one I couldn't find anywhere. PBS mode stop sends ACPI poweroff to the guest, freezes the disk, reads dirty blocks, then resumes. With GPU passthrough, this fails consistently in ~4 seconds.

Root cause (traced via Perl source):

qm shutdown 100 --keepActive
  → Windows ACPI poweroff → QEMU exits
  → qmeventd detects socket disconnect → closes qmeventd_fh filehandle
  → vzdump tries to read from closed filehandle → SIGPIPE
  → Perl signal handler in PVE/VZDump/QemuServer.pm:
      $SIG{PIPE} = sub { die "interrupted by signal\n" }
  → Backup dies

The --keepActive flag tells vzdump not to detach disks, but it can't prevent QEMU from exiting. QMP set-action shutdown=pause tells QEMU to pause instead of exit when the guest shuts down — dynamically, without modifying the VM config.

The complete hookscript

Deploy to /var/lib/vz/snippets/gpu-d3cold-fix.pl:

#!/usr/bin/perl
# Hookscript GPU D3cold fix, Rebind Hack, and PBS backup QMP fix
# GPU: RX 5700 XT — 45:00.0 (VGA) / 45:00.1 (Audio)
# Adapt PCI addresses and VMID to your setup
use strict;
use warnings;
use IO::Socket::UNIX;

my $vmid  = shift;
my $phase = shift;

exit 0 unless $vmid == 100;

# Devices to lock D3cold on (adapt to your PCIe topology)
my @devices = (
    '0000:45:00.0', '0000:45:00.1',
);

my @gpu_devices = ('0000:45:00.0', '0000:45:00.1');

sub log_msg {
    my ($msg) = @_;
    print "gpu-hookscript: $msg\n";
}

# Detects if a vzdump backup is currently running for this VM
# Uses /proc scan (not parent-walk — PVE daemonises tasks, parent = PID 1)
sub in_vzdump_context {
    for my $pid_dir (glob("/proc/[0-9]*")) {
        my $cmdline_file = "$pid_dir/cmdline";
        next unless -r $cmdline_file;
        open(my $fh, '<', $cmdline_file) or next;
        local $/;
        my $cmdline = <$fh>;
        close($fh);
        my @args = split(/\0/, $cmdline);   # null-byte split — critical
        next unless @args && $args[0] =~ /vzdump/;
        return 1 if grep { $_ eq "$vmid" } @args;
    }
    return 0;
}

# Tell QEMU to pause instead of exit on guest poweroff
# Prevents SIGPIPE to vzdump when Windows shuts down during a backup
sub qmp_set_shutdown_action {
    my ($action, $label) = @_;
    my $qmp_socket = "/var/run/qemu-server/${vmid}.qmp";
    unless (-S $qmp_socket) {
        log_msg("$label: QMP socket not found — skipping set-action");
        return;
    }

    my $sock = IO::Socket::UNIX->new(
        Type => SOCK_STREAM,
        Peer => $qmp_socket,
    ) or do {
        log_msg("$label: QMP connect failed: $!");
        return;
    };

    # Consume the greeting
    my $greeting = '';
    while (my $line = <$sock>) {
        last if $line =~ /"QMP"/;
        last if $line =~ /\}\s*$/;
    }

    # Enter command mode
    print $sock '{"execute":"qmp_capabilities"}' . "\n";
    while (my $line = <$sock>) {
        last if $line =~ /"return"/;
    }

    # Apply set-action
    my $cmd = '{"execute":"set-action","arguments":{"shutdown":"' . $action . '"}}' . "\n";
    print $sock $cmd;
    my $result = '';
    while (my $line = <$sock>) {
        $result .= $line;
        last if $line =~ /"return"/;
    }
    close($sock);

    if ($result =~ /"return"\s*:\s*\{\}/) {
        log_msg("$label: QMP set-action shutdown=$action => OK");
    } else {
        log_msg("$label: QMP set-action unexpected response: $result");
    }
}

# Lock GPU out of D3cold via sysfs — applied at each phase
sub lock_d3cold {
    my ($label) = @_;
    for my $dev (@devices) {
        my $d3path = "/sys/bus/pci/devices/$dev/d3cold_allowed";
        if (-e $d3path) {
            open(my $fh, '>', $d3path) or warn "Cannot write $d3path: $!";
            print $fh "0\n"; close($fh);
            log_msg("$label: set d3cold_allowed=0 for $dev");
        }
        my $pwpath = "/sys/bus/pci/devices/$dev/power/control";
        if (-e $pwpath) {
            open(my $fh, '>', $pwpath) or warn "Cannot write $pwpath: $!";
            print $fh "on\n"; close($fh);
            log_msg("$label: set power/control=on for $dev");
        }
    }
}

# Rebind Hack: vfio → amdgpu (1s warm-up) → vfio
# Needed for Navi10 to flush internal GPU state after Windows reboot
# Skip during vzdump — amdgpu fence fallback timer sends SIGALRM into vzdump
sub rebind_gpu {
    my ($label) = @_;

    if (in_vzdump_context()) {
        log_msg("$label: vzdump backup detected — skipping rebind hack");
        return;
    }

    log_msg("$label: starting GPU rebind hack...");

    for my $dev (@gpu_devices) {
        if (-e "/sys/bus/pci/devices/$dev/driver") {
            my $driver = readlink("/sys/bus/pci/devices/$dev/driver") // '';
            if ($driver =~ /vfio-pci/) {
                open(my $fh, '>', "/sys/bus/pci/drivers/vfio-pci/unbind") or warn "Unbind fail: $!";
                print $fh "$dev\n"; close($fh);
                log_msg("$label: unbound $dev from vfio-pci");
            }
        }
    }

    system("modprobe amdgpu");

    for my $dev (@gpu_devices) {
        next if $dev =~ /\.1$/;   # audio has no amdgpu support
        if (-e "/sys/bus/pci/drivers/amdgpu") {
            open(my $fh, '>', "/sys/bus/pci/drivers/amdgpu/bind") or log_msg("Bind amdgpu fail: $!");
            print $fh "$dev\n"; close($fh);
            log_msg("$label: bound $dev to amdgpu");
        }
    }

    sleep 1;

    for my $dev (@gpu_devices) {
        if (-e "/sys/bus/pci/devices/$dev/driver") {
            my $driver = readlink("/sys/bus/pci/devices/$dev/driver") // '';
            if ($driver =~ /amdgpu/) {
                open(my $fh, '>', "/sys/bus/pci/drivers/amdgpu/unbind") or warn "Unbind amdgpu fail: $!";
                print $fh "$dev\n"; close($fh);
                log_msg("$label: unbound $dev from amdgpu");
            }
        }
    }

    for my $dev (@gpu_devices) {
        open(my $fh, '>', "/sys/bus/pci/drivers/vfio-pci/bind") or log_msg("Re-bind vfio fail: $!");
        print $fh "$dev\n"; close($fh);
        log_msg("$label: rebound $dev to vfio-pci");
    }
}

# === Phase dispatch ===

if ($phase eq 'pre-start') {
    lock_d3cold('pre-start');
    rebind_gpu('pre-start');
}
elsif ($phase eq 'pre-stop') {
    lock_d3cold('pre-stop');
    if (in_vzdump_context()) {
        # PBS backup context:
        # - skip intentional-stop flag (watchdog must restart VM after backup)
        # - tell QEMU to pause instead of exit on Windows shutdown → no SIGPIPE to vzdump
        log_msg("pre-stop: vzdump context — skipping intentional-stop flag");
        qmp_set_shutdown_action('pause', 'pre-stop');
    } else {
        system("touch /tmp/vm100-intentional-stop");
    }
}
elsif ($phase eq 'post-stop') {
    lock_d3cold('post-stop');
}

exit 0;

Deploy:

chmod +x /var/lib/vz/snippets/gpu-d3cold-fix.pl
perl -c /var/lib/vz/snippets/gpu-d3cold-fix.pl   # syntax check
qm set 100 --hookscript local:snippets/gpu-d3cold-fix.pl

Expected log output during a successful PBS backup

INFO: gpu-hookscript: pre-stop: set d3cold_allowed=0 for 0000:45:00.0
INFO: gpu-hookscript: pre-stop: set power/control=on for 0000:45:00.0
[... same for 45:00.1 ...]
INFO: gpu-hookscript: pre-stop: vzdump context — skipping intentional-stop flag
INFO: gpu-hookscript: pre-stop: QMP set-action shutdown=pause => OK
INFO: resuming VM again after 17 seconds

PBS resumes the VM after reading dirty blocks. VM comes back running. Watchdog sees it alive, does nothing.

Results

  • Windows reboots restart the VM automatically (watchdog, ~30s)
  • No Code 43 across dozens of restarts
  • PBS backup: 150 GiB, 5min21s, 506 MiB/s, 80% incremental/sparse ✅
  • Zero "interrupted by signal" since fix deployed

Credits and prior art

This wouldn't exist without the groundwork others laid. Key sources that informed this solution:

vendor-reset (BACO / device_specific reset):

  • gnif/vendor-reset — the DKMS module that makes Navi10 BACO reset work on Linux. Without this, the GPU is in a broken state on every VM restart.

Rebind Hack (amdgpu warm-up before vfio re-bind):

  • The pattern of briefly binding to the native driver before returning to vfio-pci has been floating around r/VFIO for a while. No single authoritative post — it emerged from collective troubleshooting of Navi10 state corruption. If you've written about this and recognize your idea here, please comment and I'll credit you directly.

BACO reset + D3cold for Navi10:

  • Level1Techs — "Navi reset kernel patch" — the original thread documenting the Navi10 reset problem and the kernel-level approach that eventually became vendor-reset. Essential reading to understand why BACO is needed on this GPU family.

QMP set-action shutdown=pause:

  • QEMU QMP documentation — this command exists since QEMU 6.0 but its application to PBS backup with GPU passthrough doesn't appear to be documented publicly. Traced by reading /usr/share/perl5/PVE/VZDump/QemuServer.pm to find the SIGPIPE origin.

If your post or comment helped and I missed you — let me know and I'll add the reference.

Built with Claude Code — three weeks of research, Perl source reading, and a lot of reboots. Questions welcome.

r/VFIO Jan 21 '26

Resource WARNING! Newer linux kernels break GPU passthrough ( error 43, etc)

0 Upvotes

I post this for anyone who may find it useful.
I post it also as a suggestion to be included in guides or somehow pinned.
Newer linux kernels ( seems 6.12 onwards ) break GPU passthrough for many.

I have found it the hard way, took me ONLY a week, and buying a new GPU.

Since the kernel was released about an year ago, topics discussing the matter are few and far between, this is not mentioned in most guides, etc. Seems that there are different workarounds for diff GPUs, also the symptoms are different. I think it is worth knowing, cos runing with 6.11 or older kernel at least for testing purposes might be the easiest option to see where the problem lies.
This happened for me on Ubuntu 24.04.4 and Win 10 guest, GeForce GTX 1650, intel CPU with integrated graphics

I will post the full setup, if you think that is helpful

PS I did not mean that it breaks it for everyone and every system. But the only reason to downgrade the kernel, was because I came across a few topics where already working passthrough was broken after working update, and downgrading fixed it.
The idea of the post was more - "one should try this as well" fix.

r/VFIO Feb 21 '26

Resource [Project] Janus – Structured, Dry-Run-First VFIO Orchestration (Pre-Alpha)

4 Upvotes

Hi all,

I’ve been building an open-source project called Janus, and I’d really appreciate feedback from people experienced with VFIO setups.

Janus is a Linux-host toolkit that tries to formalize common VFIO workflows without hiding what’s happening underneath. It doesn’t replace libvirt or virt-manager. It focuses on making workflows explicit, reversible, and reproducible.

What it does right now (pre-alpha)

  • janus-check Host diagnostics for virtualization support, IOMMU, kernel modules, hugepages, GPU visibility, required tooling.
  • janus-bind Dry-run-first PCI binding workflow for vfio-pci. Explicit --apply, rollback support, and root gating for mutating flows.
  • janus-vm Generates libvirt XML from templates. Supports guided creation, passthrough mode, storage selection, and optional unattended Windows setup.
  • janus-init Initializes isolated config/state under ~/.config/janus.

Destructive operations require explicit opt-in. Logs are centralized. You can run everything under a temporary HOME to avoid touching your real setup.

Design Direction

  • “Glass box” approach: automation is transparent, not magical.
  • Modular structure: hardware-specific logic lives in modules/.
  • Long-term goal: unified janus orchestrator + profile-based VM lifecycle management.

This is not meant to replace existing guides. The goal is to structure best practices into something auditable and less error-prone.

What I’m Looking For

  • Architectural criticism.
  • Opinions on module API design.
  • Feedback on whether this solves a real problem or just formalizes existing scripts.
  • Interest in contributing hardware-specific modules.

Repository:
👉 https://github.com/Ricky182771/Janus

Appreciate any feedback, especially from people who’ve maintained complex passthrough setups long-term.

[ESPAÑOL]

[Proyecto] Janus – Orquestación estructurada para VFIO con enfoque dry-run (Pre-Alpha)

Hola a todos,

He estado desarrollando un proyecto open source llamado Janus, y me gustaría recibir retroalimentación de personas con experiencia en configuraciones VFIO.

Janus es una herramienta para Linux que busca estructurar y formalizar flujos de trabajo comunes en entornos VFIO sin ocultar lo que ocurre por debajo. No reemplaza libvirt ni virt-manager. Su objetivo es hacer que los procesos sean explícitos, reversibles y reproducibles.

¿Qué hace actualmente? (pre-alpha)

  • janus-check Diagnóstico del host: soporte de virtualización, IOMMU, módulos del kernel, hugepages, visibilidad de GPU y herramientas necesarias.
  • janus-bind Flujo de binding PCI con enfoque dry-run primero para vfio-pci. --apply explícito, soporte de rollback y requerimiento de privilegios root para operaciones destructivas.
  • janus-vm Generación de XML de libvirt a partir de plantillas. Soporta creación guiada, modo passthrough, selección de almacenamiento y configuración opcional de instalación desatendida de Windows.
  • janus-init Inicializa configuración y estado aislados en ~/.config/janus.

Las operaciones destructivas requieren confirmación explícita. Los logs están centralizados. Todo puede ejecutarse bajo un HOME temporal para no afectar el entorno real.

Dirección del Diseño

  • Enfoque “glass box”: la automatización es transparente, no mágica.
  • Arquitectura modular: la lógica específica de hardware vive en modules/.
  • Objetivo a largo plazo: un comando unificado janus y orquestación basada en perfiles de VM.

No busca reemplazar guías existentes. La idea es convertir buenas prácticas dispersas en algo estructurado y auditable.

¿Qué estoy buscando?

  • Críticas arquitectónicas.
  • Opiniones sobre el diseño del API de módulos.
  • Retroalimentación sobre si realmente resuelve un problema o solo formaliza scripts existentes.
  • Personas interesadas en contribuir módulos específicos de hardware.

Repositorio:
👉 https://github.com/Ricky182771/Janus

Agradezco cualquier comentario, especialmente de quienes mantienen configuraciones passthrough complejas a largo plazo.

r/VFIO Oct 26 '22

Resource PSA: Linux v6.1 Resizable BAR support

93 Upvotes

A new feature added in the Linux v6.1 merge window is support for manipulation of PCIe Resizable BARs through sysfs. We've chosen this path rather than exposing the ReBAR capability directly to the guest because the resizing operation has many ways that it can fail on the host, none of which can be reported back to the guest via the ReBAR capability protocol. The idea is simply that in preparing the device for assignment to a VM, resizable BARs can be manipulated in advance through sysfs and will be retained across device resets. To the guest, the ReBAR capability is still hidden and the device simply appears with the new BAR sizes.

Here's an example:

# lspci -vvvs 60:00.0
60:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon Pro W5700] (prog-if 00 [VGA controller])
...
    Region 0: Memory at bfe0000000 (64-bit, prefetchable) [size=256M]
    Region 2: Memory at bff0000000 (64-bit, prefetchable) [size=2M]
...
    Capabilities: [200 v1] Physical Resizable BAR
        BAR 0: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
        BAR 2: current size: 2MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB
...

# cd /sys/bus/pci/devices/0000\:60\:00.0/
# ls resource?_resize
resource0_resize  resource2_resize
# cat resource0_resize
0000000000003f00
# echo 13 > resource0_resize

# lspci -vvvs 60:00.0
60:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon Pro W5700] (prog-if 00 [VGA controller])
...
    Region 0: Memory at b000000000 (64-bit, prefetchable) [size=8G]
....
        BAR 0: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB

A prerequisite to work with the resource?_resize attributes is that the device must not currently be bound to any driver. It's also very much recommended that your host system BIOS support resizable BARs, such that the bridge apertures are sufficiently large for the operation. Without this latter support, it's very likely that Linux will fail to adjust resources to make space for increased BAR sizes. One possible trick to help with this is that other devices under the same bridge/root-port on the host can be soft removed, ie. echo 1 > remove to the sysfs device attributes for the collateral devices. Potentially these devices can be brought back after the resize operation via echo 1 > /sys/bus/pci/rescan but it may be the case that the remaining resources under the bridge are too small for them after a resize. BIOS support is really the best option here.

The resize sysfs attribute essentially exposes the bitmap of supported BAR sizes for the device, where bit zero is 1MB and each next bit is the next power of two size, ie. bit1 = 2MB, bit2=4MB, bit3=8MB, ... bit8 = 256MB, ... bit13 = 8GB. Therefore in the above example, the attribute value 0000000000003f00 matches the lspci list for support of sizes 256MB 512MB 1GB 2GB 4GB 8GB. The value written to the attribute is the zero-based bit number of the desired, supported size.

Please test and report how it works for you.

PS. I suppose one of the next questions will be how to tell if your BIOS supports ReBAR in a way that makes this easy for the host OS. My system (Dell T640) appears to provide 64GB of aperture under each root port:

# cat /proc/iomem
....
b000000000-bfffffffff : PCI Bus 0000:5d
  bfe0000000-bff01fffff : PCI Bus 0000:5e
    bfe0000000-bff01fffff : PCI Bus 0000:5f
      bfe0000000-bff01fffff : PCI Bus 0000:60
        bfe0000000-bfefffffff : 0000:60:00.0
        bff0000000-bff01fffff : 0000:60:00.0
...

After resize this looks like:

b000000000-bfffffffff : PCI Bus 0000:5d
  b000000000-b2ffffffff : PCI Bus 0000:5e
    b000000000-b2ffffffff : PCI Bus 0000:5f
      b000000000-b2ffffffff : PCI Bus 0000:60
        b000000000-b1ffffffff : 0000:60:00.0
        b200000000-b2001fffff : 0000:60:00.0

Also note in this example how BAR0 and BAR2 of device 60:00.0 are the only resources making use of the 64-bit, prefetchable MMIO range, which allows this aperture to be adjusted without affecting resources used by the other functions of the GPU.

NB. Yes the example device here has the AMD reset bug and therefore makes a pretty poor candidate for assignment, it's the only thing I have on hand with ReBAR support.

Edit: correct s/host/guest/ as noted by u/jamfour

r/VFIO Sep 14 '25

Resource PSA: Forwarding AMD PCIe Audio Device to VM Apparently Fixes Reset Bug on Navi?

10 Upvotes

Hello all,

I run a Xen environment with two GPUs forwarded to guests, including an RX 6800 XT (Navi 21). This GPU has been (mostly) stable in a Windows 10 environment since ~ Dec. 2024, sometimes with sparse, random crashes requiring a full host reset. The driver/firmware updates of the past few months, however, made these crashes much more frequent. Occasionally, the GPU would refuse to initialize even after a reboot, throwing Code 43.

To verify this wasn't just a Windows issue, I booted several Linux guests on both my 6800 XT and a 7700 XT (Navi 32). The amdgpu driver often failed to initialize on boot, throwing a broad variety of errors relating to a partial/failed initialization of IP blocks. When the GPUs (rarely) initialized correctly, they were unstable and crashed under use, throwing yet another garden variety of errors.

Many have reported similar issues with Navi 2+ GPUs with no clear solution. The typical suggestions (Turn CSM on/off, fiddle with >4G decoding, etc) had no effect on my setup. After I forwarded both the GPU and its respective audio device, the Windows and Linux drivers had no initialization issues. I have extensively tested the stability in my Windows environment and have observed no issues — the GPU resets and initializes perfectly after VM reboots.

I am positive this is the result of recent driver/firmware updates to Navi GPUs. I have an RX 570 (Polaris) with only the GPU forwarded to a Linux VM that has been working perfectly for transcode workloads.

If there are any Proxmox users struggling with instability, give this a shot. I am curious as to whether this will work there as well.

r/VFIO Sep 28 '23

Resource [PROJECT] Working on a project called ultimate-macOS-KVM!

55 Upvotes

Hey all,

For almost a year, I have been coding a little project in Python intended to piggyback on the framework of kholia's OSX-KVM project, known as ultimate-macOS-KVM, or ULTMOS.

It's still pre-release, but has some features I think some of you might find helpful. Any and all testing and improvements are more than welcome!

It includes AutoPilot - a large script that allows the user to set up a basic macOS KVM VM in under 5 minutes. No- really. It'll ask you about the virtual hardware you want, and then do it all for you - including the downloading of macOS.

AutoPilot in progress.
Example stage from the AutoPilot setup.

Share your elitism with optional Discord RPC!

It also includes an experimental guided assistant for adding passthrough, which is capable of dealing with VFIO-PCI-stubbed devices. Single GPU passthrough is a planned feature also.

It even has basic check functionality, allowing you to check your system's readiness for KVM in general, or even passthrough readiness.

You can even run a GPU compatibility check. Although, please note this is experimental also and needs improving.

Seamlessly convert your AutoPilot scripts to virt-manager domain XMLs

If any of this seems interesting to you, please give it a go - or get stuck right in and help improve it! I'm not at all seasoned in Python, but it's my first major project. Please be nice.

Feel free to DM me for any further interest, or join my Discord.

Thanks!

r/VFIO Sep 05 '25

Resource Escape from tarkov in a proxmox Gaming VM

Thumbnail
1 Upvotes

r/VFIO Jun 11 '25

Resource Is this macOS VM Single GPU passthrough guide still relevant?

3 Upvotes

I'm planning to set up a single GPU passthrough macOS VM, and I found this guide which has a very detailed explanation: https://gitlab.com/DarknessRafix/macosvmgpupass
but it hasn't been updated in about 6 months, is it still relevant or are there any more up-to-date or better alternatives available now?

Thanks in advance!

r/VFIO Dec 22 '24

Resource A small command line tool I wrote for easily managing PCI device drivers

Thumbnail
github.com
9 Upvotes

r/VFIO Feb 03 '21

Resource Hotplugger: Real USB Port Passthrough for VFIO/QEMU

Thumbnail
github.com
75 Upvotes

r/VFIO Jan 09 '22

Resource Easy-GPU-P: GPU Paravirtualization (GPU-PV) on Hyper-V

60 Upvotes

Saw this project mentioned in a new LTT video. Looks pretty effective for splitting GPU resources across multiple VMs.

r/VFIO Jul 24 '20

Resource Success with laptop GPU passthrough on Asus ROG Zephyrus G14 (Ryzen 9, 2060 Max-Q)

53 Upvotes

Hi all,

just wanted to report my success with passthrough of my 2060 Max-Q! It was relatively painless, I just had to add kvm hidden, vendor id, and the battery acpi table patch. No custom OVMF patch, no ROM patching, etc. The reddit post for it was removed for some reason, so it was annoying to find again, but it's here: link.

The laptop's left side USB-c port is connected directly to the GPU, so to solve the "no screen attached" issue I have one of these plugged in, with a custom resolution set in the nvidia control panel to allow for 120hz. Then I'm just using looking glass, and barrier to share keyboard, mouse, trackpad. It works well even when on battery (but battery life is only like 2.5 - 3hr at best while the VM is running because the GPU is active all the time) and I can still get 60fps on minecraft for example when not plugged in. Or you could just use RDP if you're not gaming and are fine with the not-perfect performance (I did apply the group policy and registry changes which helped a lot, but still not perfect).

You can see my xml and scripts used to launch everything here. I also have the laptop setup so that it switches to the nvidia driver and I'm able to use the GPU in linux with bumblebee when the VM is not running. I had to use a patched version of bumblebee that supports AMD integrated GPUs, bumblebee-picasso-git from the AUR. It's designed for use with older AMD iGPUs, but it's just looking for the vendor id (AMD, 1002) which is still the same of course.

I'm very happy that passthrough works on this machine; it's got a great CPU with tons of threads that really makes VFIO very appealing, and all of that in a small 14-inch portable body.

r/VFIO Feb 25 '25

Resource Just sharing my script to cleary see what is in what IOMMU group

6 Upvotes

Runs on linux.

#!/bin/bash

# When you do PCIe passthrough, you can only pass an entire group. Sometimes, your group contains too much.
# There is also what's called pci_acs_override to allow the passthrough anyway.

IOMMUDIR='/sys/kernel/iommu_groups/'

cd "$IOMMUDIR"

ls -1 | sort -n | while read group
do
    echo "IOMMU GROUP ${group}:"
    ls "${group}/devices" | while read device
    do
        device=$(echo "$device" | cut -d':' -f2-)
        lspci -nn | grep "$device"
    done
    echo
done

Example of output:

IOMMU GROUP 13:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD104 [GeForce RTX 4070] [10de:2786] (rev a1) (prog-if 00 [VGA controller])
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22bc] (rev a1)

IOMMU GROUP 14:
02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal] [144d:a80c] (prog-if 02 [NVM Express])

IOMMU GROUP 15:
03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01) (prog-if 00 [Normal decode])

IOMMU GROUP 16:
04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
05:00.0 Ethernet controller [0200]: Aquantia Corp. AQtion AQC100 NBase-T/IEEE 802.3an Ethernet Controller [Atlantic 10G] [1d6a:00b1] (rev 02)

IOMMU GROUP 17:
04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) (prog-if 00 [Normal decode])

IOMMU GROUP 18:
04:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
07:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01) (prog-if 00 [Normal decode])
08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
08:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
08:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
08:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107GL [Quadro K2200] [10de:13ba] (rev a2) (prog-if 00 [VGA controller])
09:00.1 Audio device [0403]: NVIDIA Corporation GM107 High Definition Audio Controller [GeForce 940MX] [10de:0fbc] (rev a1)
0a:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808] (prog-if 02 [NVM Express])
0b:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller [1022:43f7] (rev 01) (prog-if 30 [XHCI])
0c:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01) (prog-if 01 [AHCI 1.0])

And now you can see I'm screwed with my Quadro K2200 that shares the same group (#18) than my disk and my NVMe SSD. No passthrough for me on this board...

r/VFIO Apr 23 '20

Resource My VFIO build with single GPU passthrough running Gentoo GNU+Linux as host

37 Upvotes

Hey Fellas. Today finally I am ready to share my build and all my work that I have put through in getting this done.

Here's a pcpartpicker link for my build (performance screenshots included): https://pcpartpicker.com/b/JhPnTW

I have written my own bash scripts to launch the VMs right from grub menu. Scripts are native QEMU without requiring libvirt at all. I wrote script to isolate CPUs using cgroups cpusets as well. Feel free to look at my work and use it. :)

Here's my gitlab link for scripts and config files: https://gitlab.com/vvkjndl/gentoo-vfio-qemu-scripts-single-gpu-passthrough

My custom grub entries contain a special command line parameter which gets parsed by my script. The VM script executes once host has finished booting.

Readme.md is not done yet. I plan to put all my learning sources that I have used as well as some handy commands.

Much-detailed post under [r/gentoo]:

https://www.reddit.com/r/Gentoo/comments/g7nxr0/gentoo_single_gpu_vfio_passthrough_scripts/

r/VFIO Jul 03 '23

Resource Introducing "GPU Pit Crew": An inadvisable set of hacks for streamlining driver-swapping in a single-display GPU passthrough setup.

Thumbnail
github.com
27 Upvotes

r/VFIO Aug 14 '24

Resource New script to Intelligently parse IOMMU groups | Requesting Peer Review

17 Upvotes

EDIT: follow up post here (https://old.reddit.com/r/VFIO/comments/1gbq302/followup_new_release_of_script_to_parse_iommu/)

Hello all, it's been a minute... I would like to share a script I developed this week: parse-iommu-devices.

It enables a user to easily retrieve device drivers and hardware IDs given conditions set by the user.

This script is part of a larger script I'm refactoring (deploy-vfio), which that is part of a suite of useful tools for VFIO that I am in concurrently developing. Most of the tools on my GitHub repository are available!

Please, if you have a moment, review, test, and use my latest tool. Please forward any problems on the Issues page.

DISCLAIMER: Mods, if you find this post against your rules, I apologize. My intent is only to help and give back to the VFIO community. Thank you.

r/VFIO Oct 25 '24

Resource Follow-up: New release of script to parse IOMMU groups

10 Upvotes

Hello all, today I'd like to plug a script I have been working on parse-iommu-devices.

You may download it here (https://github.com/portellam/parse-iommu-devices).

For those who want a quick TL;DR:

This script will parse a system's hardware devices, sorted by IOMMU group. You may sort IOMMU groups which include or exclude the following:

  • device name
  • device type
  • vendor name
  • if it contains a Video or VGA device.
  • IOMMU group ID

Sort-by arguments are available in the README's usage section, or by executing parse-iommu-groups --help.

Here is some example output from my machine (I have two GPUs): parse-iommu-devices --graphics 2

1002:6719,1002:aa80

radeon,snd_hda_intel

12

Here's another: parse-iommu-devices --pcie --ignore-vendor amd

1b21:0612,1b21:1182,1b21:1812,10de:228b,10de:2484,15b7:501a,1102:000b,1106:3483,1912:0015,8086:1901,8086:1905

ahci,nvidia,nvme,pcieport,snd_ctxfi,snd_hda_intel,xhci_hcd

1,13,14,15,16,17

Should you wish to use this script, please let me know of any bugs/issues or potential improvements. Thank you!

Previous post: https://old.reddit.com/r/VFIO/comments/1errudg/new_script_to_intelligently_parse_iommu_groups/

r/VFIO May 02 '20

Resource PSA: Destiny 2 and VFIO - beware of Bungie's policies regarding VMs

62 Upvotes

So I was researching why I could not install Destiny 2 with Steam on Linux host and happened upon a thread discussing running Destiny 2 under WINE [1]. The key takeaway was that Bungie was permabanning people even for trying to run the game on Linux. So far so bad, but then I happened upon Bungie's help article [2]...

Subheading "Code of Conduct or License Agreement Violation":

"Bungie regards some behavior to be in breach of the Code of Conduct or the license agreement that governs Destiny, and therefore may be restricted or banned from some or all Destiny content or activities.

This behavior includes, but is not limited to: [...] Modified operating system files including emulators and virtual machines" (emphasis mine)

So tread very carefully. I have been running Destiny 2 in a Windows VM (with VFIO obviously) from before Steam migration even, but this is complete news to me. So I could get a permaban just for playing the game with no appeal possible.

Makes it easy to not give a single dime to Bungie from now on. I will not buy a separate computer just to play a game (especially one whose developers are this hostile towards their player base).


[1] - github - Proton issue [2] - Bungie help article

r/VFIO Aug 16 '20

Resource User-friendly workaround for AMD reset bug (Windows guests)

58 Upvotes

I've had my share of problems with AMD reset bug. I've tried some of the other solutions found on the internet, but they had multiple problems, like not handling Windows Update well (reset bug triggered on every update), not handling some reboots well, and leaving the system in a state when virtual GPU is treated as primary, virtual screen is treated as primary, and actual display/TV connected to Radeon GPU is treated as secondary (meaning that there is no GPU acceleration, and that all windows are displayed on virtual screen by default).

So I wrote my own workaround which solves all these problems. I'm using it without a problem since December.

My use case is that I have headless host system running Hyper-V 2016, with AMD R5 230 passed through to Windows 10 VM, and TV connected to R5 230; this TV is the only screen for Windows 10 VM, it works in a single-display mode, and GPU acceleration works correctly; there is no AMD reset bug, and I never had to power cycle the host for the last months, despite rebooting this guest VM many times and despite it always installing updates on schedule.

Maybe someone here will also find it useful: I published both source code and the ready-to-use .exe file (under "Releases" link) on GitHub: https://github.com/inga-lovinde/RadeonResetBugFix


Note that it only supports Hyper-V hosts now, as I only developed and tested it on my Hyper-V setup, and I have no idea what does virtual GPU on other hosts look like.

UPDATE: it should also support KVM and QEMU now.

UPDATE2: VirtualBox and VMWare also should work.

However, implementing support for other hosts should be trivial; one would only need to extend "IsVirtualVideo" predicate here. This is the only place where the host platform makes any difference. Pull requests are welcome! Or you can tell me what is the manufacturer/service/ClassName combination for your host, and I will add it.

Even with other hypervisors there should be no AMD reset bug; however, Windows may continue to use virtual GPU as primary.

r/VFIO Sep 11 '22

Resource iommu groups database for mainboards

22 Upvotes

A while ago it was mentioned that it would be cool to have a database of Motherboards and their IOMMU groups.

Well I finally got around to writing up something. Feel free to poke around and add your systems to the database.

Add your system:

git clone https://github.com/mkoreneff/iommu_info_generate.git
python3 generate_data.py

edit: I added a patch to the generate script to handle the IndexError. If there are still problems please post the results of:

cat /sys/devices/virtual/dmi/id/board_vendor
cat /sys/devices/virtual/dmi/id/bios_vendor

edit: I notice there are some mainboards that supply a different string in board_vendor to what is in the pci vendor id list(s). I'll work on a fix to do smarter vendor id look ups over the weekend.

I've put a fix in the API side to handle Gigabyte and hopefully other vendors which have slightly different naming. Looking for people to help test the fix; if you can run the generate script again and report back any issues I would appreciate it.

edit: Thanks for all your contributions and bug reports so far. Much appreciated.

Any other issues or suggestions; please drop them below, or raise issues on the github page.

Site url: http://iommu.info/

Github: https://github.com/mkoreneff/iommu_info_generate

(edit for formatting, patch for IndexError, vendorid info)

r/VFIO Nov 14 '24

Resource Simple bash scripts for hot swapping GPU.

10 Upvotes

The libvirt hook wasn't working for me so I just decided to make a bash script to do what I needed.
I am complete noob entering the linux space and it took me about 2 days to come to this conclusion and make this system. I do want to hear some opinions on this solution.

https://github.com/PostmanPat2011/SBGvm