Hi,
Thank you for your reply and for the suggested troubleshooting steps.
I have now performed some additional debugging and have a few more findings that may help identify the root cause.
Unfortunately, the suggested amd_iommu ivmd_unity_map=1 parameter does not appear to be available on Proxmox VE 9.1, neither with kernel 6.17.13-6-pve nor 7.0.0-3-pve.
The amd_iommu driver is built into the kernel and exposes no runtime parameters:
ls /sys/module/amd_iommu/parameters
find /sys/module -name ‘unity’ -o -name ‘ivmd*’
Both commands return no results.
My current kernel command line is:
BOOT_IMAGE=/boot/vmlinuz-6.17.13-6-pve root=/dev/mapper/pve-root ro quiet nvme_core.default_ps_max_latency_us=0 amd_iommu=on iommu=pt
Additional findings
I removed the ACS override and repeated the test.
The GPU is now placed in a single IOMMU group together with its audio function, which is expected.
However, the VFIO error is now reported against the upstream PCIe bridge, not the NVIDIA GPU itself:
pcieport 0000:02:00.0:
Firmware has requested this device have a 1:1 IOMMU mapping,
rejecting configuring the device without a 1:1 mapping.
The PCIe topology is:
00:01.1 AMD Strix PCIe USB4 Bridge
└─ 01:00.0 ASMedia PCIe Bridge
└─ 02:00.0 ASMedia PCIe Bridge
├─ 03:00.0 NVIDIA GeForce RTX 3090
└─ 03:00.1 NVIDIA HD Audio
This suggests that the firmware requirement is associated with the USB4 PCIe tunnel / ASMedia bridge, rather than with the NVIDIA GPU endpoint itself.
ACPI IVRS
I also extracted and disassembled the ACPI IVRS table.
The firmware exposes several IVMD (Memory Definition) entries, and the Linux kernel reports matching reserved/direct regions for the GPU’s IOMMU group.
For example:
0×00000000771f1000 0×00000000771f2fff direct
These address ranges also appear in the IVRS table.
Host functionality
Everything works perfectly on the host:
NVIDIA proprietary driver loads correctly
nvidia-smi works
CUDA works
Docker works
Ollama successfully uses the RTX 3090
PCIe link trains correctly at PCIe Gen4 ×4 (16 GT/s)
The only operation that fails is when VFIO attempts to assign the device to a virtual machine.
Questions
Could your BIOS team please verify whether the USB4 firmware or IVRS table is advertising a mandatory 1:1 DMA mapping for devices behind the USB4 PCIe bridge?
Also:
Has Beelink successfully validated VFIO PCIe passthrough over USB4 on the SER10 MAX?
Is there a newer BIOS, AGESA version or beta BIOS that changes USB4 DMA protection, IVRS generation or Kernel DMA Protection behaviour?
I would be happy to test any beta BIOS or provide additional debugging information if needed.
Thank you very much for your assistance.
