Dual NVIDIA GPU Passthrough on Linux: What Every Guide Gets Wrong

I spent two weeks getting a Windows 11 VM with GPU passthrough working on CachyOS. I started with the usual plan: dynamically detach my second NVIDIA GPU when the VM starts, stream the guest with Apollo and Moonlight, and keep both cards available to Linux when Windows is off.

Almost none of that survived contact with the actual machine.

Dynamic binding hard-froze the host. Apollo connected but never produced a frame. Adding Looking Glass shared memory made QEMU abort with a DMA mapping error. Once that was fixed, KDE Wayland introduced three more bugs.

It works now, reliably. This is the complete setup I ended up with, including the dead ends and the fixes that mattered.

The hardware and the goal

The split is deliberate. Both cards have roughly 16 GB VRAM, but the 5070 Ti has much higher memory bandwidth, so it stays with Linux for local LLM work. The 4060 Ti is dedicated to the Windows VM. Both can run at the same time with no contention.

Before copying anything below, substitute your own PCI addresses, device IDs, CPU topology, memory allocation, and IOMMU address width. The values here are specific to this machine.

1. Enable virtualization in the BIOS

Enable:

• Intel VT-x
• Intel VT-d
• Above 4G Decoding

Resizable BAR can usually stay enabled, but it isn't required for this setup.

2. Enable IOMMU in Linux

Add these parameters to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub:

intel_iommu=on iommu=pt

Then rebuild GRUB:

sudo grub-mkconfig -o /boot/grub/grub.cfg

intel_iommu=on enables Intel's IOMMU. iommu=pt identity-maps host devices so ordinary host DMA doesn't pay a translation penalty.

3. Identify the guest GPU and its IOMMU group

Find both GPU functions:

lspci -nn | grep -E 'VGA|3D|Audio'

On this machine, the guest card is:

07:00.0 VGA compatible controller [10de:2805]
07:00.1 Audio device             [10de:22bd]

Pass both the video and HDMI audio functions. They should also be isolated from unrelated devices in their IOMMU group:

for d in /sys/kernel/iommu_groups/*/devices/*; do
  group=${d#*/iommu_groups/}; group=${group%%/*}
  printf 'group %s: %s\n' "$group" "${d##*/}"
done | sort -V

My 4060 Ti's video and audio functions sit alone in one group, so I didn't need an ACS override. If your group contains storage or another device the host needs, stop here and fix the hardware topology first.

4. Bind the guest GPU to vfio-pci at boot

This is the most important part of the setup.

I initially tried dynamic binding: leave the 4060 Ti on the NVIDIA driver, then let libvirt detach it when Windows starts. That froze the host every time. The VM never started, the libvirt process entered uninterruptible D state, and shutdown hung until I forced the machine off.

The reason is subtle. NVIDIA's userspace GL and Vulkan stack opens every /dev/nvidiaN device on the machine, not only the card currently rendering the desktop. KWin, Xwayland, browsers, terminals, and other GPU applications all had the second card memory-mapped. Libvirt could not detach it from the driver.

Static binding makes that failure impossible. The 4060 Ti is claimed by vfio-pci during boot, before NVIDIA sees it. Linux never creates /dev/nvidia1, so no desktop process can hold the card.

Create /etc/modprobe.d/vfio.conf using your GPU's PCI IDs:

# Guest GPU video + HDMI audio
options vfio-pci ids=10de:2805,10de:22bd

# Ensure vfio-pci gets first chance at the devices
softdep nvidia pre: vfio-pci
softdep nvidia_drm pre: vfio-pci
softdep nvidia_modeset pre: vfio-pci
softdep nvidia_uvm pre: vfio-pci

In /etc/mkinitcpio.conf, add the VFIO modules:

MODULES=(vfio_pci vfio vfio_iommu_type1)

Then rebuild and reboot:

sudo mkinitcpio -P
sudo reboot

Do not add vfio_virqfd. It was merged into vfio in kernel 6.2 and no longer exists as a separate module.

Also remember that mkinitcpio's modconf hook copies /etc/modprobe.d into the initramfs. If you later change or remove vfio.conf, run mkinitcpio -P again. Deleting the file alone doesn't remove the old IDs from the boot image.

Verify after reboot:

# Both guest GPU functions should use vfio-pci
lspci -nnk -s 07:00 | grep -i 'driver in use'

# Host GPU should still use nvidia
lspci -nnk -s 01:00.0 | grep -i 'driver in use'

# Only the host GPU should appear
nvidia-smi --query-gpu=index,name,pci.bus_id --format=csv

# This machine should have only /dev/nvidia0
ls /dev/nvidia[0-9]*

The passthrough GPU disappearing from nvidia-smi is success.

5. Install the virtualization stack

sudo pacman -S qemu-full libvirt virt-manager edk2-ovmf swtpm dnsmasq virtio-win
sudo systemctl enable --now virtqemud.socket
sudo usermod -aG libvirt,kvm "$USER"

Log out and back in after changing group membership.

Current libvirt installations often use modular daemons. Check which one is active:

systemctl is-active virtqemud libvirtd

On my system, virtqemud is active and libvirtd is not. Restarting libvirtd therefore does nothing, despite what older guides say.

6. Create the Windows 11 VM

In virt-manager:

1. Create a VM from a Windows 11 ISO.
2. Allocate 32 GB RAM and 8 vCPUs.
3. Create a roughly 250 GB qcow2 disk on fast storage.
4. Use Q35 as the chipset.
5. Select an OVMF Secure Boot firmware image.
6. Add an emulated TPM 2.0 device using the CRB model.
7. Use VirtIO for the disk and network device.
8. Add the virtio-win ISO as a second CD-ROM.
9. Tick "Customize configuration before install".

Windows 11 Secure Boot inside OVMF is independent of the host's Secure Boot setting. My host Secure Boot is disabled, while the guest uses Secure Boot normally.

7. Fix the VM XML

Back it up first:

virsh -c qemu:///system dumpxml win11 > ~/win11.xml.backup
virsh -c qemu:///system edit win11

Hyper-V features and hidden KVM

<features>
  <acpi/>
  <apic/>
  <hyperv>
    <relaxed state='on'/>
    <vapic state='on'/>
    <spinlocks state='on' retries='8191'/>
    <vpindex state='on'/>
    <synic state='on'/>
    <stimer state='on'/>
    <reset state='on'/>
    <vendor_id state='on' value='kvm hyperv'/>
    <frequencies state='on'/>
  </hyperv>
  <kvm><hidden state='on'/></kvm>
  <vmport state='off'/>
  <ioapic driver='kvm'/>
  <smm state='on'/>
</features>

smm is required for UEFI Secure Boot. Hiding KVM also avoids older NVIDIA Code 43 behaviour and other software that needlessly rejects virtual machines.

CPU topology and pinning

The 13700K has 8 hyperthreaded P-cores (CPU 0-15) and 8 single-threaded E-cores (CPU 16-23). I give Windows P-cores 4-7, which are logical CPUs 8-15:

<vcpu placement='static'>8</vcpu>
<cpu mode='host-passthrough' check='none' migratable='on'>
  <topology sockets='1' dies='1' cores='4' threads='2'/>
  <cache mode='passthrough'/>
  <maxphysaddr mode='passthrough' limit='39'/>
</cpu>
<cputune>
  <vcpupin vcpu='0' cpuset='8'/>
  <vcpupin vcpu='1' cpuset='9'/>
  <vcpupin vcpu='2' cpuset='10'/>
  <vcpupin vcpu='3' cpuset='11'/>
  <vcpupin vcpu='4' cpuset='12'/>
  <vcpupin vcpu='5' cpuset='13'/>
  <vcpupin vcpu='6' cpuset='14'/>
  <vcpupin vcpu='7' cpuset='15'/>
  <emulatorpin cpuset='0-3'/>
  <iothreadpin iothread='1' cpuset='16-23'/>
</cputune>
<iothreads>1</iothreads>

The topology is 4 cores × 2 threads, not 8 cores × 1 thread. The vCPUs are pinned to four physical P-cores and their siblings. You can check the real topology with:

cat /sys/devices/system/cpu/cpu8/topology/thread_siblings_list

Hugepage-backed memory

<memory unit='KiB'>33554432</memory>
<currentMemory unit='KiB'>33554432</currentMemory>
<memoryBacking>
  <hugepages/>
  <locked/>
</memoryBacking>

This allocates 32 GB and prevents it from being swapped. I use libvirt lifecycle hooks to allocate 16,384 2 MB hugepages before startup and release them after shutdown. Static hugepages at boot also work, but they permanently reserve half the machine even when the VM is off.

The hooks only manage hugepages. They do not bind GPUs and they do not change the CPU governor. power-profiles-daemon already owns the governor, and writing to it behind the daemon's back creates misleading state.

8. Attach both GPU functions

Use virt-manager's "Add Hardware → PCI Host Device" and add both the GPU and its audio function. Or add this to the XML with your addresses:

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
  </source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
  </source>
</hostdev>

managed='yes' is safe with static binding. Libvirt sees that vfio-pci already owned the card before startup, so it doesn't reattach it to NVIDIA when the VM stops.

9. Add Looking Glass shared memory

Looking Glass copies the guest framebuffer through shared memory. There is no network stream, encoder, or decoder. For a VM running on the same machine, that's exactly what I wanted.

Add an IVSHMEM device:

<shmem name='looking-glass'>
  <model type='ivshmem-plain'/>
  <size unit='M'>128</size>
</shmem>

Size it using width × height × 4 × 2 + 10 MB, rounded up to a power of two. 64 MB is enough for 2560×1440. I use 128 MB so 4K also fits.

The DMA mapping failure

As soon as I added shared memory, QEMU died with:

qemu: hardware error: vfio: DMA mapping failed, unable to continue

The CPU exposes 46 physical address bits. In host-passthrough mode, OVMF may place the IVSHMEM BAR high in that address space. But my Intel IOMMU has a 39-bit Maximum Guest Address Width (MGAW). The GPU can't DMA to the high BAR, so VFIO rejects the mapping.

Read your own MGAW:

for d in /sys/class/iommu/dmar*/intel-iommu/cap; do
  cap=$(cat "$d")
  echo "$d: MGAW=$(( ( (0x$cap >> 16) & 0x3f ) + 1 )) bits"
done

Then cap the guest with the maxphysaddr line in the CPU XML:

<maxphysaddr mode='passthrough' limit='39'/>

Don't copy limit='40' from a forum. It fails on hardware whose IOMMU reports 39 bits. Use the value your machine reports.

Shared memory permissions

QEMU and your user both need access to the shared memory file. Create a tmpfiles rule:

sudo tee /etc/tmpfiles.d/10-looking-glass.conf >/dev/null <<'EOF'
f /dev/shm/looking-glass 0660 your-user kvm -
EOF
sudo systemd-tmpfiles --create /etc/tmpfiles.d/10-looking-glass.conf

Replace your-user. The file is recreated at boot because /dev/shm is tmpfs.

10. Install Windows and the guest drivers

Install Windows through the temporary SPICE/QXL console. At the disk selection screen, load viostor\w11\amd64 from the virtio-win ISO so the installer can see the VirtIO disk.

After installation:

1. Install virtio-win-gt-x64.msi.
2. Install the QEMU guest agent.
3. Install the normal NVIDIA GeForce driver.
4. Check Device Manager for the passed-through GPU and confirm there is no Code 43.
5. Install a maintained virtual display driver and set it to your monitor's resolution.
6. Install the IVSHMEM driver from the Looking Glass download page.
7. Install the Looking Glass Windows host application.

The virtio-win ISO does not include the IVSHMEM driver. It comes from Looking Glass. In Device Manager, the shared memory device initially appears as a yellow-flagged "PCI standard RAM Controller".

Use matching Looking Glass releases on both sides. A B7 client talking to a different host release may connect but never show a frame.

11. Remove the emulated display, but keep SPICE

Once the virtual display and Looking Glass host work, disable QXL:

<video>
  <model type='none'/>
</video>

Keep SPICE bound to localhost:

<graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
  <image compression='off'/>
</graphics>

Looking Glass has no input transport of its own. Keyboard and mouse go through SPICE. If QXL remains attached, Windows may make it the primary screen, and your pointer will move on an invisible display while Looking Glass shows the other one. That looks like a missing cursor, but it's really two displays disagreeing.

After QXL is removed, virt-manager's console becomes a black rectangle. That's expected. Looking Glass is now the console.

12. Install and configure the Looking Glass client

Use the package from your distribution or build a stable release that matches the Windows host. On a recent Arch/CachyOS toolchain, I had to use the B7 tag rather than master. Master built, but it did not communicate reliably with the B7 Windows host.

My ~/.looking-glass-client.ini:

[win]
fullScreen=yes
size=2560x1440
ignoreQuit=yes

[egl]
noBufferAge=yes

Don't put comments in this file. Looking Glass's ini parser treats comment lines as option names.

The three non-obvious settings are all there because of real KDE Wayland issues:

• size must match the monitor. The xdg backend ignores the first fullscreen configure size, so a mismatched initial buffer causes KWin to drop fullscreen.
• ignoreQuit=yes prevents KWin close events such as Alt+F4 from silently killing the window and leaving the SPICE thread stuck.
• noBufferAge=yes works around stale frames and the "mouse cursor paints the new image" artifact caused by incorrect EGL buffer age reporting on NVIDIA.

Run the client:

looking-glass-client

Looking Glass uses Scroll Lock as its escape key by default. Tap it to toggle input capture. Scroll Lock + F toggles fullscreen, and Scroll Lock + Q quits.

Why I didn't use Apollo and Moonlight

Apollo and Moonlight were my original plan. They connected, but I never received a frame. The Windows logs eventually showed the cause:

Microsoft Basic Render Driver ... 0 MiB
Failed to create encoder D3D11 device [0x887A0004]
libx264 [software]

Windows had no EDID-backed output on the NVIDIA GPU. Apollo selected the Basic Render Driver, NVENC failed to initialize, and it silently fell back to software x264. Moonlight sat there waiting for video that never became usable.

I tried firewall changes, different clients, auto-login, and a physical dummy plug. None fixed it. Looking Glass avoids the entire class of problem because it doesn't encode or stream anything.

The tradeoff is that Looking Glass is local to the host. It can't stream the VM to a TV or phone. For this machine, low-latency local access mattered more.

Optional: allocate hugepages only while the VM runs

You can reserve hugepages permanently at boot, or use libvirt hooks. I use hooks so Linux gets the memory back when Windows is off.

The start hook calculates the required count from the VM memory. For 32 GB using 2 MB pages:

32768 MB / 2 MB = 16384 hugepages

A minimal start hook can compact memory and allocate the pages:

#!/bin/bash
set -euo pipefail
PAGES=16384

echo 1 > /proc/sys/vm/compact_memory
echo "$PAGES" > /proc/sys/vm/nr_hugepages

actual=$(cat /proc/sys/vm/nr_hugepages)
if [ "$actual" -ne "$PAGES" ]; then
  echo "Could only allocate $actual/$PAGES hugepages" >&2
  exit 1
fi

The release hook returns them:

#!/bin/bash
set -euo pipefail
echo 0 > /proc/sys/vm/nr_hugepages

Place them under the lifecycle paths used by your libvirt hook dispatcher. Don't add GPU bind/unbind hooks; static binding means nothing should ever move between NVIDIA and vfio-pci.

Verification checklist

With the VM off:

# Guest GPU remains on vfio-pci
lspci -nnk -s 07:00 | grep -i 'driver in use'

# Host sees only the 5070 Ti
nvidia-smi --query-gpu=index,name --format=csv

# Shared memory permissions
stat -c '%U:%G %a %s' /dev/shm/looking-glass

With the VM running:

• Device Manager shows the RTX 4060 Ti without Code 43.
• The Looking Glass client reports the same version as the Windows host.
• The client reports a real BGRA frame format and your expected resolution.
• SPICE INPUTS and PLAYBACK channels connect.
• The host still sees only the 5070 Ti through nvidia-smi.
• The VM survives shutdown, force-off, and immediate restart.

What actually mattered

The working setup came down to four decisions:

1. Bind the guest NVIDIA GPU to vfio-pci at boot. Don't dynamically detach it from a live NVIDIA desktop.
2. Use Looking Glass for local display instead of fighting a headless NVENC setup.
3. Read the IOMMU's real MGAW and cap maxphysaddr before adding IVSHMEM.
4. Configure Looking Glass around the actual KDE Wayland and NVIDIA EGL behaviour.

Nothing moves when the VM starts. The 4060 Ti is always owned by vfio-pci, whether Windows is running or not. The 5070 Ti is always owned by NVIDIA. That sounds less flexible than dynamic binding, but it is predictable, and predictable is what you want when the alternative is holding down the power button.

The result is exactly what I wanted: CachyOS remains the real workstation, the faster GPU stays available for Linux gaming and local AI, and Windows is a VM I can open in a low-latency fullscreen window when I need it.