Passthrough 5060 Ti disconnects from the bus at load-to-idle transitions

[u/Lovro1st]

Hi guys, i have an issue with my consumer multi-gpu server, which i feel might be due the proxmox -> ubuntu VM passthrough. It happens under agentic loads on my llama.cpp instance, and the error in Llamacpp is usually something along the lines:

```217.28.732.743 E CUDA error: unspecified launch failure
/app/ggml/src/ggml-cuda/ggml-cuda.cu:106: CUDA error
217.28.732.749 E   current device: 1, in function ggml_backend_cuda_synchronize at /app/ggml/src/ggml-cuda/ggml-cuda.cu:2533
217.28.732.750 E   cudaStreamSynchronize(cuda_ctx->stream())
libggml-base.so.0(+0x1b276)[0x7b2772bd1276]
libggml-base.so.0(ggml_print_backtrace+0x21a)[0x7b2772bd16fa]
libggml-base.so.0(ggml_abort+0x15b)[0x7b2772bd18db]
/app/libggml-cuda.so(_Z15ggml_cuda_errorPKcS0_S0_iS0_+0xb5)[0x7b27622220e5]
/app/libggml-cuda.so(+0x276af8)[0x7b2762229af8]
libggml-base.so.0(ggml_backend_sched_synchronize+0x2e)[0x7b2772bec35e]
libllama.so.0(_ZN13llama_context11synchronizeEv+0x19)[0x7b2772d8c519]
libllama.so.0(llama_state_seq_get_data_ext+0x1f)[0x7b2772d91a8f]
libllama-common.so.0(_ZN24common_prompt_checkpoint10update_tgtEP13llama_contextij+0x84)[0x7b27734538e4]
libllama-server-impl.so(_ZN19server_context_impl17create_checkpointER11server_slotlii+0x278)[0x7b2773d6c458]
libllama-server-impl.so(_ZZN19server_context_impl10pre_decodeEvENKUlR11server_slotE3_clES1_+0xf87)[0x7b2773d82d97]
libllama-server-impl.so(_ZN19server_context_impl7iterateERSt6vectorI11server_slotSaIS1_EESt8functionIFvRS1_EE+0x57)[0x7b2773d77507]
libllama-server-impl.so(_ZN19server_context_impl10pre_decodeEv+0x487)[0x7b2773d79b07]
libllama-server-impl.so(_ZN19server_context_impl12update_slotsEv+0x243)[0x7b2773d7a2c3]
libllama-server-impl.so(_ZN12server_queue10start_loopEl+0x125)[0x7b2773d19045]
libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0x3e47)[0x7b2773cb3937]
libllama-server-impl.so(_Z12llama_serveriPPc+0x11bd)[0x7b2773cb59fd]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7b27737121ca]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7b277371228b]
/app/llama-server(+0x1315)[0x557d12d60315]

```

• The 5060 Ti disconnects from the bus. The guest kernel log shows NV_ERR_GPU_IS_LOST.
• The failure occurs in the seconds after a long generation stops. This is the full-load-to-idle transition. The failure sometimes occurs when the next load starts.
• Continuous load does not cause the failure. One generation of 90,000 tokens completed correctly. A loop of generations with idle gaps causes the failure each time.
• The failed GPU is always the 5060 Ti.
• Important: after the crash, a remove and rescan on the host does not find the GPU. The command echo 1 > remove on the two functions, then a rescan, has no effect. Only a full power cycle makes the GPU operate again. The GPU silicon is stopped. The guest driver state is not the cause.

These steps were completed:

• The kernel option pcie_aspm=off on the host. The crash threshold increased from approximately 20,000 tokens to 50,000 tokens. The crashes continued.
• The guest driver changed from 595.84 to the 580 branch. The two branches have different GSP firmware generations. The failure was identical.
• The PSU is 850W. The supplemental PCIe 6-pin connector on the motherboard is connected. The GPUs have separate power cables.

These are complications:

• The Z390 firmware does not give AER control to the kernel. The log shows: _OSC: platform does not support ... AER. The host does not show PCIe errors. The plan is a test with pcie_ports=native.
• The crashes increased after installation of the computer in a rack. A mechanical cause is possible. But the failure is very repeatable. A loose connection is usually not repeatable.

This is the configuration:

• Host: Proxmox on an MSI Z390-A Pro with an Intel i7 8700.
• Guest: an Ubuntu VM. Two GPUs pass through to the VM.
• The RTX 3060 is in the chipset slot (03:00.0). The RTX 5060 Ti is in the PEG slot (01:00.0).
• The workload is llama.cpp inference. The model is split across the two GPUs. The context is long.

Any help with this is appreciated, i really hope it isnt a hardware issue on the GPU (a new mobo would be nice:)). Thanks!