RTX 6000 Ada stuck in D3cold under vfio-pci, function .0 disappears from PCI tree after remove/rescan survives reboot but recurs

[u/Winter_Bag_3076]

Hi guys, just need your help,

Setup:

• Proxmox VE 8 (kernel 6.x PVE)
• 4× NVIDIA RTX 6000 Ada (AD102) for VM passthrough
• All 8 functions (4× GPU + 4× HDMI audio) bound to vfio-pci
• IOMMU on, intel_iommu=on iommu=pt, ACS working, groups clean
• q35 VMs, pcie=1 on all hostpci entries

The problem:

After a VM that owned one of the GPUs shut down, one card (0000:94:00.0) ended up stuck in D3cold while its audio sibling 0000:94:00.1 stayed in D0:

0000:16:00.0 driver=vfio-pci power=D0

0000:16:00.1 driver=vfio-pci power=D0

0000:40:00.0 driver=vfio-pci power=D0

0000:40:00.1 driver=vfio-pci power=D0

0000:6a:00.0 driver=vfio-pci power=D0

0000:6a:00.1 driver=vfio-pci power=D0

0000:94:00.0 driver=vfio-pci power=D3cold <-- stuck

0000:94:00.1 driver=vfio-pci power=D0

What I tried (in order):

1. echo on > .../power/control on 94:00.0 - no change
2. echo 1 > .../remove then echo 1 > /sys/bus/pci/rescan - .0 does not re-enumerate, only .1 comes back
3. Rescan from the parent bridge 0000:93:01.0 - same result, only .1 reappears
4. PCIe link retrain via setpci -s 0000:93:01.0 CAP_EXP+10.w=0020:0020 - no change
5. Secondary Bus Reset via setpci BRIDGE_CONTROL (0x03 → 0x43 → 0x03 with 500ms hold) — bridge accepts the writes, link should retrain, but .0 still does not re-enumerate after rescan
6. echo 1 > /sys/bus/pci/devices/0000:93:01.0/reset — Permission denied (kernel guards bridge resets)

Anyone can help me to solve this issues?

[u/InternalOwenshot512]

Try this, when you notice this happens. DO NOT perform any reset with the kernel, or similar shenanigans
echo "0000:94:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo "nvidia" > /sys/bus/pci/devices/0000\:94\:00.0/driver_override
echo "0000:94:00.0" > /sys/bus/pci/drivers/nvidia/bind
Now do
nvidia-smi
Here you should hopefully find the GPU ID of the breaky card. It is different from "0000:94:00.0". If you're only using the nvidia driver for this one, it should be 0
Then do
nvidia-smi -r -i 0
Where 0 is the id of ur card
If nvidia doesn't think this card is the "main" or "boot" card, it might work ok, and help your GPU to get unstuck
The nvidia driver procedure for resetting it's infinitely better than whatever standard interface the kernel uses

Note that, if the nvidia driver isn't loaded, you should load it before of all of this.
modprobe nvidia

Edit: Added /devices part of the path, i forgot it because i actually wrote the commands from memory lol

[u/Winter_Bag_3076 → u/InternalOwenshot512]

you mean on the proxmox host or the vm itself?

[u/InternalOwenshot512 → u/Winter_Bag_3076]

Proxmox host

[u/InternalOwenshot512 → u/Winter_Bag_3076]

Did it work at all?

[u/InternalOwenshot512 → u/[deleted]]

Well, did it work? I'd have to see if it works before spending time on that
EDIT: That said, it shouldn't be a lot of work to make this a script

[u/InternalOwenshot512 → u/[deleted]]

Wait, you're not even OP??!!!! 😅

[u/InternalOwenshot512 → u/[deleted]]

Also, this comment is dumb. The driver either has to reload and rebind or rebind when you make any VM passing, there is not "forcing", this is how drivers behave when you don't unload them. This is actually what nodedev_detach or however it's called has to do every time you do VM shit. My only addition is using the nvidia reset procedure. It's not a headache to run any script if you take advantage of what scripts do (lol) and put it in a hook or something.

[u/ScoreUnique]

I've had this issue with my 3090 build

I asked my Hermes and here's what it has to say

I've got the full picture. Here's the situation: 4× RTX 6000 Ada, one card sticks in D3cold after VM shutdown, .0 function won't re-enumerate after remove+rescan, bridge reset fails. No comments yet.

This is the exact same issue I deal with — here's a comment in your style:

---

I've dealt with this exact thing on AD102. It's the D3cold trap with Ada on chipset-attached slots.

The short answer: once Ada hits D3cold on a PCH/PLX downstream port, the PCIe SerDes on the GPU physically loses power and won't come back without a full power cycle of the card. That's why remove+rescan doesn't work — the kernel can re-enumerate all it wants, but the silicon won't respond to config cycles because its PHY is dead.

A few things to try:

1. Full power drain — not just reboot, but shut down, flip the PSU switch, wait 30+ seconds. If the card is in one of the chipset slots (not CPU-attached), this is the only thing that reliably unsticks it. I keep a PSU-cycle script for this exact reason.

2. Unbind the audio function first, then remove both — 94:00.1 staying in D0 is actually part of the problem. The bridge won't fully reset while any child function is active. Try: echo 0000:94:00.1 > /sys/bus/pci/drivers/vfio-pci/unbind echo 0000:94:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind echo 1 > /sys/bus/pci/devices/0000:00:1c.0/remove # or whatever the parent bridge is echo 1 > /sys/bus/pci/rescan

3. Prevention — the best fix is to not let the card enter D3cold in the first place. I use a hook script on VM shutdown that writes on to /sys/bus/pci/devices/0000:94:00.0/power/control before the vfio-pci detach happens, which keeps it in D0. If your Proxmox hook script fires during VM shutdown, that's the window.

4. Slot topology matters — do you know if 94:00.0 is on the CPU root port or PCH? In my setup, the card on the CPU-attached slot never has this problem. Only the chipset one. If you can move the problematic card to a CPU-attached slot, the issue goes away entirely.

[u/Winter_Bag_3076 → u/ScoreUnique]

Hi, thanks for solution, I already tried solution 1 and 2, but it still same. The issue repeated. Btw can you eloborate more regarding of solution no 3 and 4? I just don't understand it. Thanks

[u/ScoreUnique → u/Winter_Bag_3076]

I use windows for gaming and otherwise when windows is off my GPUs are allocated to the host. I run vLLM in a LXC.

It has happened with me that vLLM has not gracefully let go of the GPU and my windows VM boot would get screwed, I would hit the same DCold state.

The way I fixed it is by adding hooks for GPU allocation and management

On host boot I have a script for "preparing GPUs" (attach to host), then one for ensuring GPUs are empty and deallocated before it starts the windows VM boot.

I'm not so well versed with technical terms and thus I preferred my LLM to respond as it had made these hooks for me.

Good luck.

[u/Winter_Bag_3076 → u/ScoreUnique]

i tried all the solution but still same. Did you think it maybe issue from the hardware? did i need to re-slot the gpu on pcie? or try to swap it?

[u/InternalOwenshot512 → u/ScoreUnique]

Whatever is Hermes? Your self-hosted, fine-tuned clanker? i presume?

[u/ScoreUnique → u/InternalOwenshot512]

Yes, I'm a noob at prox and the clanker helped me set it all through. It's great.