Utilizing PCI Passthrough (VT-d) on Proxmox VE

Overview

Proxmox Virtual Environment (Proxmox VE) is an open-source software server for virtualization management.

On Proxmox VE, hardware devices are seen as "virtual" hardware and can be used by all the Virtual Machines (VM) and the host in a system. Proxmox provides "PCI Passthrough" which enables direct access via the PCI Express bus to physical devices such as a network ports, WiFi cards, storage devices, GPU, and more. The advantage of PCI Passthrough is lower latency and higher performance. However, once a device is passed through to a VM, it can’t be used by other VMs or the host in the system.

PCI Passthrough will work with both AMI and coreboot BIOS, but certain devices like thermal sensors and the HD audio controller may not be able to be passed with coreboot BIOS.

Please see the chart at the bottom of this current article for VT-d/PCI Passthrough compatibility. (FW2B, FW4B, and FW4C are not supported)

*Important Note: IOMMU must be enabled for PCI Passthrough to work. In-depth instructions to configure PCI Passthrough/IOMMU can be found at: https://pve.proxmox.com/wiki/Pci_passthrough. A quick and simple run-down on enabling IOMMU can be found below.

*Additional Note: All of the instructions on this page regarding changing machine type and passing through PCI devices is under the assumption you have already properly installed a VM on Proxmox VE. If you need instructions on installing VMs on Proxmox VE, please check out our articles for installing a pfSense® CE VM or installing an OPNsense VM.

Enabling IOMMU

• These instructions are under the assumption you installed Proxmox with the ext filesystem
• Access the Proxmox VE console via an external monitor or through the Shell on the web management interface
• Type and enter: nano /etc/default/grub
• Add intel_iommu=on to GRUB_CMDLINE_LINUX_DEFAULT="quiet" (See the screenshot below, you can copy the line from the code box below)

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

• Write Out the settings and Exit
• Run the command update-grub to finalize changes
• Reboot your Vault

Follow the official Proxmox instructions for more in-depth information (https://pve.proxmox.com/wiki/Pci_passthrough)

Choosing q35 Machine Type for VMs

Selecting the q35 machine type allows for proper utilization of passed-through devices. Compared to the default machine type of i440fx, q35 is more closely related to how a real system's PCIe topology looks like, and has better support for native PCIe devices.

The q35 machine type can be selected via the Hardware tab of any given VM you have created on Proxmox VE.

 

OVMF (UEFI) vs SeaBIOS

Linux, Debian, and Windows

For Linux, Debian, and modern Windows Operating Systems, it is best to initially install the VM with the OVMF (UEFI) BIOS. Keep in mind there is not a single configuration that will work for absolutely every single PCI device, but selecting OVMF (UEFI) alongside the q35 machine type has the best success rate for PCI passthrough with Protectli Vaults.

Some OS's may not support UEFI, like Sophos XG.

FreeBSD (OPNsense and pfSense® CE)

For FreeBSD based OS's like OPNsense and pfSense® CE, using the default SeaBIOS with the q35 machine type will work with passing through network ports (NICs) and WiFi adapters.

Selecting and Passing Through PCI Devices

On the Hardware tab of any VM, click Add>PCI Device

From here, you will need to select Raw Device, and then choose a PCI device to passthrough. The following example showcases PCI devices on a FW6Br2 with AMI BIOS.

Simply choose the device you wish to add, and press the Add button. From here, the VM should be able to utilize the device the next time you start it.

Potential Uses

There is a plethora of possibilities with how you can utilize PCI devices with different OS's, so it's impossible to cover everything. However, we will go over a few interesting/useful options that can get you started.

Ethernet Ports/NICs

You can passthrough individual NICs to improve performance and decrease latency opposed to the default Linux Bridges. This can be done with both OVMF(UEFI) and SeaBIOS.

You should not passthrough the same NIC that is being used as Proxmox VE's management port, as this will cause a loss of connection to the management web interface, or will cause crashing.

Simply choose your desired interface connection as a PCI device, and it should immediately work for data throughput the next time you launch the VM.

If you passthrough a NIC to OPNsense or pfSense® CE, you will need to assign these interfaces on the OS's console.

In the following example you can see the VP2410's available interfaces.

 

USB Port Passthrough

By selecting the USB Controller as a PCI Device, you can connect USB storage devices, keyboards/mice, or whatever USB device you may want to use. In the following example we have a Protectli USB Drive plugged into the Vault, and it is accessible on an Ubuntu 22.04 VM.

 

Thermal Subsystem Passthrough

You can passthrough the thermal subsystem to a VM to detect internal running temperatures. This will only be possible if your Vault is flashed with the AMI BIOS. The VP2410 does not have the thermal subsystem available for passthrough.

In the following example, we have passed through the thermal subsystem to detect temperatures on a Fedora VM. We used the xsensors package to display the temperatures.

 

GPU Passthrough

You can passthrough the integrated graphics controller to a VM, allowing for higher video performance and the ability to view video output via a monitor connected to the Vault. Normally a monitor connected to a Proxmox server will display the Proxmox console/shell, but using GPU passthrough will display a VM's output instead.

This tends to work better with Linux distributions opposed to Windows. Windows may require additional drivers and additional troubleshooting to work.

Note regarding GPU passthrough on the VP6600 Series: The 12th Gen (and newer) Intel CPUs do not support the ability to passthrough the video display through to a physical HDMI/Display Port connection. It appears that some people have had success with passing through the iGPU on devices with CPUs from the same generation, though: See this link for a Proxmox Forum post that may send you down a rabbit hole.

When adding the graphics processor as a PCI device, you will need to check the Primary GPU and All Functions box.

Checking the Advanced box and choosing PCI-Express may help if you are experiencing any trouble with passing through the GPU.

Some of our Vault products may not have the integrated GPU labelled on the Proxmox WebGUI. For example, a VP4670 with AMI BIOS uses the follow PCI device (0000:00:02.0) to passthrough the GPU:

In the following example we have passed through the Intel UHD Graphics 620 on an FW6E, allowing for a monitor to display the Ubuntu VM's output via the HDMI port. Use this alongside USB passthrough so you can utilize a keyboard and mouse!

WiFi/Bluetooth Modules

You can passthrough WiFi Modules to use with your virtual machines.

If your Vault has an mPCIe or M.2 WiFi kit, you can choose them as PCI devices to passthrough. If you have the WAP01K WiFi kit, you need to passthrough your USB controller to utilize it.

Further information regarding OS compatibility is outlined in our WiFi on the Vault knowledge base article.

When adding the mPCIe or M.2 WiFi kit as a PCI device, expand the Advanced options and enable the ROM-Bar and PCI-Express options

The following example shows that the WAP01K WiFi kit can be added as a Wireless Interface on a pfSense® CE virtual machine. This can also be done on OPNsense.

VT-d Support

Conclusion

There are many different possibilities with how you can utilize PCI Passthrough, so we encourage you to experiment with it. There will occasionally be some troubleshooting required when getting PCI devices to work with different OS's. This typically revolves around changing the advanced settings when adding the PCI device, or sometimes installing drivers.

If you ever come across any issues, please reach out to us at support@protectli.com, or open up a support ticket.