Create Proxmox VM

Do not start the VM.

  1. Sanity Check – ensure VT-d is enabled in the BIOS
  2. Enable IOMMU on Proxmox host

nano /etc/default/grub

  • Add the following to GRUB_CMDLINE_LINUX_DEFAULT:
  • GRUB_CMDLINE_LINUX_DEFAULT=”quiet intel_iommu=on video=efifb:off pcie_acs_override=downstream”
  • Update Grub

update-grub

  • Blacklist both NVIDIA and Nouveau kernel modules in Proxmox host to make sure they do not load at boot

echo “blacklist nouveau” >> /etc/modprobe.d/blacklist.conf
echo “blacklist nvidia” >> /etc/modprobe.d/blacklist.conf

  • Update Initramfs

update-initramfs -k all -u
update-grub

  • Add vfio modules

nano /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

  • Find PCI address of GPU

lspci -v

  1. Find Vendor ID’s

lspci -n -s 15:00

  1. Force GPU to vfio
echo "options vfio-pci ids=10de:1c81,10de:0fb9 disable_vga=1" > /etc/modprobe.d/vfio.conf
  1. In Proxmox GUI change VM to use OMVF (Open Source UEFI) Bios and add an EFI HD option (change option to not start on boot)
  2. Reboot host

shutdown -r now

  1. Confirm IOMMU

dmesg | grep -e DMAR -e IOMMU

Output should look something like this:

  1. Check for interrupt remapping support level (all units must support) the DMAR-IR entry above does not indicate this

dmesg | grep ecap

On the IOMMU lines, the hexadecimal value after “ecap” indicates whether interrupt remapping is supported. If the last character of this value is an 8, 9, a, b, c, d, e, or an f, interrupt remapping is supported. For example, “ecap 1000” indicates there is no interrupt remapping support. “ecap 10207f” indicates interrupt remapping support, as the last character is an “f”.

Since I have “6” I must add an unsafe interrupt line to modprobe.d

echo “options vfio_iommu_type1 allow_unsafe_interrupts=1” > /etc/modprobe.d/iommu_unsafe_interrupts.conf

update-initramfs -k all -u
update-grub
shutdown -r now

  1. Verify IOMMU Isolation

find /sys/kernel/iommu_groups/ -type l

My IOMMU (Not so good but ok)

Good IOMMU (example)

To help with a bad IOMMU implementation add “pcie_acs_override=downstream” to grub options, which can help on some setup with bad ACS implementation.

(already done above)

  1. Edit the VM conf file

nano /etc/pve/qemu-server/<vmid>.conf

  1. Install OS and SSH into VM
  2. After you have verified that SSH works Passthrough the GPU (you will lose access to the console screen)

Modify /etc/pve/qemu-server/<vmid>.conf and add
hostpci0: <device address>,x-vga=on

Example:

hostpci0: 01:00,x-vga=on

update-initramfs -k all -u
update-grub
shutdown -r now