Passthrough the GPU!
Explore IOMMU Grouping
Let's passthrough our second GPU now. Run the following command in your terminal:
#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;
This script should output information about your PCI device. If it does not, it may be because you haven't enabled the necessary settings in your BIOS.
This script provides a detailed view of IOMMU groups and their associated devices, helping you understand the connections within your system. This information is vital for configuring device passthrough in virtualization.
Identify IOMMU GPU IDs
The output from the above command will list your GPUs. Your second GPU, the one designated for passthrough, will be included in this list. Note its PCI (Peripheral Component Interconnect) numbers, also known as PCI IDs. These IDs are critical for the upcoming steps.
Here's an example of what you might see:
IOMMU Group 22:
09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] [1002:73ff] (rev c7)
IOMMU Group 23:
09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
Example IDs
Suppose your investigation reveals the following GPU IDs:
- 1002:73ff (GPU)
- 1002:ab28 (GPU Audio)
Keep these IDs handy, as they are crucial for the next steps.
Embed IDs in Your Bootloader Config
Configuring GPU passthrough involves specifying which GPU to allocate to the virtual machine by using its PCI IDs. These IDs pinpoint the GPU within the system, enabling direct hardware access.
Add the PCI IDs to your bootloader configuration. If you're using GRUB or another bootloader, include the following in the configuration line where you enabled virtualization:
vfio-pci.ids=1002:73ff,1002:ab28
Systemd-boot Example
Your updated configuration with systemd-boot should look like this:
title Arch Linux
linux /vmlinuz-linux
initrd /initramfs-linux.img
options root=<root> quiet rw intel_iommu=on iommu=pt vfio-pci.ids=1002:73ff,1002:ab28
For systemd-boot users, you do not need to regenerate the config.
GRUB Example
Your updated configuration with GRUB should look like this:
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt vfio-pci.ids=1002:73ff,1002:ab28 ..."
For GRUB users, you will need to regenerate the config file:
sudo grub-mkconfig -o /boot/grub/grub.cfg
Create a vfio.conf File
Next, create a new configuration file for vfio:
sudo touch /etc/modprobe.d/vfio.conf
Add the following content:
options vfio-pci ids=1002:73ff,1002:ab28
Update mkinitcpio.conf
Ensure that the necessary vfio modules are included in your initial RAM disk. Edit the /etc/mkinitcpio.conf file and update the MODULES() line:
MODULES=(vfio_pci vfio vfio_iommu_type1)
Regenerate the Image
With the configuration updated, rebuild the initial RAM disk and reboot your system:
sudo mkinitcpio -P # Regenerate all image
sudo mkinitcpio -p linux # Regenerate image only linux
Verify GPU Passthrough Status
Now reboot your system and check the status of your GPU. To check if your GPU is correctly configured for passthrough, run:
lspci -k
Look at your PCI setup. The Kernel driver in use entry should show vfio-pci. For example:
09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c7)
Subsystem: Tul Corporation / PowerColor Navi 23 [Radeon RX 6600/6600 XT/6600M]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
09:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
If you see vfio-pci, your GPU and GPU Audio are successfully set up for passthrough. If not, review your steps and try again. Success is within reach!