Setting up a virtual workstation in OpenShift with VFIO passthrough¶

Feb 27, 2023

25 min read

This article provides a detailed guide on how to configure OpenShift as a workstation with GPU PCI passthrough and Container Native Virtualization (CNV) on a single OpenShift node (SNO).

This setup allows you to leverage Kubernetes orchestration capabilities while still enjoying near-native performance for GPU-intensive applications.

Why this approach?

Run both containerized workloads and virtual machines on the same hardware
Use a single GPU for both Kubernetes pods and virtual machines by switching the driver binding
Achieve near-native performance for gaming and professional applications in VMs
Maintain the flexibility and power of Kubernetes/OpenShift for other workloads

In testing, this configuration successfully ran Microsoft Flight Simulator in a Windows VM with performance smiliar to a bare metal Windows installation.

The workstation used for this demo has the following hardware:

Component	Specification
CPU	AMD Ryzen 9 3950X 16-Core 32-Threads
Memory	64GB DDR4 3200MHz
GPU	Nvidia RTX 3080 FE 10GB
Storage	2x 2TB NVMe Disks (for virtual machine storage) 1x 500GB SSD Disk (for OpenShift root system)
Network	10Gbase-CX4 Mellanox Ethernet

Similar configurations with equivalent Intel CPUs should work with minor adjustments noted throughout the guide.

Installing OpenShift SNO¶

Before proceeding with the installation, ensure you’ve completed the backup steps for any existing partitions.

Backup of existing system partitions¶

To avoid boot order conflicts, the OpenShift assisted installer will format the first 512 bytes of any disks that contain a bootable partition. Therefore, it is important to backup and remove any existing partition table that you would like to preserve.

OpenShift Installation¶

Once any existing file system is backed up and there are no more bootable partitions, we can proceed with the OpenShift Single Node installation.

It is important to note that CoreOS, the underlying operating system, requires an entire disk for installation. For this workstation setup:

We’ll use the 500GB SSD disk for the OpenShift operating system
The two 2TB NVMe disks will be reserved for persistent volumes as LVM Physical volumes belonging to the same Volume Group
This configuration allows for flexible VM storage management while keeping the system installation separate

#!/bin/bash

OCP_VERSION=latest-4.10

curl -k https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux.tar.gz > oc.tar.gz
tar zxf oc.tar.gz
chmod +x oc && mv oc ~/.local/bin/

curl -k https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$OCP_VERSION/openshift-install-linux.tar.gz > openshift-install-linux.tar.gz
tar zxvf openshift-install-linux.tar.gz
chmod +x openshift-install && mv openshift-install ~/.local/bin/

curl $(openshift-install coreos print-stream-json | grep location | grep x86_64 | grep iso | cut -d\" -f4) > rhcos-live.x86_64.iso

install-config.yaml¶

# This file contains the configuration for an OpenShift cluster installation.

apiVersion: v1

# The base domain for the cluster.
baseDomain: epheo.eu

# Configuration for the compute nodes.
compute:
- name: worker
  replicas: 0 

# Configuration for the control plane nodes.
controlPlane:
  name: master
  replicas: 1 

# Metadata for the cluster.
metadata:
  name: da2

# Networking configuration for the cluster.
networking:
  networkType: OVNKubernetes
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16

# Platform configuration for the cluster.
platform:
  none: {}

# Configuration for bootstrapping the cluster.
bootstrapInPlace:
  installationDisk: /dev/sda

# Pull secret for accessing the OpenShift registry.
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"XXXXXXXX"}}}' 

# SSH key for accessing the cluster nodes.
sshKey: |
  ssh-rsa AAAAB3XXXXXXXXXXXXXXXXXXXXXXXXX

Generate OpenShift Container Platform assets¶

mkdir ocp && cp install-config.yaml ocp
openshift-install --dir=ocp create single-node-ignition-config

Embed the ignition data into the RHCOS ISO:¶

alias coreos-installer='podman run --privileged --rm \
      -v /dev:/dev -v /run/udev:/run/udev -v $PWD:/data \
      -w /data quay.io/coreos/coreos-installer:release'
cp ocp/bootstrap-in-place-for-live-iso.ign iso.ign
coreos-installer iso ignition embed -fi iso.ign rhcos-live.x86_64.iso
dd if=discovery_image_sno.iso of=/dev/usbkey status=progress

Once the ISO is copied to the USB drive, you can use the USB drive to boot your workstation node and install OpenShift Container Platform.

Install CNV Operator¶

Activate Intel VT or AMD-V hardware virtualization extensions in BIOS or UEFI.

cnv-resources.yaml¶

# This YAML file contains Kubernetes resources for installing the KubeVirt Hyperconverged Operator (HCO) on the OpenShift Container Platform.
# It creates a namespace named "openshift-cnv", an operator group named "kubevirt-hyperconverged-group" in the "openshift-cnv" namespace, and a subscription named "hco-operatorhub" in the "openshift-cnv" namespace.
# The subscription specifies the source, source namespace, name, starting CSV, and channel for the KubeVirt Hyperconverged Operator.

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-cnv
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: kubevirt-hyperconverged-group
  namespace: openshift-cnv
spec:
  targetNamespaces:
    - openshift-cnv
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.10.0
  channel: "stable"

oc apply -f cnv-resources.yaml

Installing the Virtctl client on your desktop.¶

subscription-manager repos --enable cnv-4.10-for-rhel-8-x86_64-rpms
dnf install kubevirt-virtctl

Configure OpenShift for single GPU passthrough¶

As our GPU is the only one attached to the node a few additional steps are required.

We will use MachineConfig to configure our node accordingly.

All MachineConfig are applied on the master machineset because we have a single node OpenShift. With a multi nodes cluster those would be applied to workers instead.

Passing kernel arguments at boot time¶

Multiple Kernel arguments have to be passed at boot time in order to configure our node for GPU passthrough. This can be done using the MachineConfigOperator.

amd_iommu=on: Enables IOMMU (Input/Output Memory Management Unit) support for AMD platforms, allowing for direct memory access (DMA) by PCI devices without going through the CPU. This improves performance and reduces overhead.
vga=off: Disables VGA (Video Graphics Array) console output during boot time.
rdblaclist=nouveau: Enables the blacklisting of the Nouveau open-source NVIDIA driver.
video=efifb:off: Disables the EFI (Extensible Firmware Interface) framebuffer console output during boot time.

https://docs.kernel.org/fb/fbcon.html

Setting Kernel Arguments at boot time.¶

variant: openshift
version: 4.10.0
metadata:
  name: 100-vfio
  labels:
    machineconfiguration.openshift.io/role: master
openshift:
  kernel_arguments:
    - amd_iommu=on
    - vga=off
    - rdblaclist=nouveau
    - 'video=efifb:off'

cd articles/openshift-workstation/machineconfig/build
butane -d . vfio-prepare.bu -o ../vfio-prepare.yaml
oc patch MachineConfig 100-vfio --type=merge -p ../vfio-prepare.yaml

Note

If you’re using an Intel CPU you’ll have to set intel_iommu=on instead.

Installing and configuring the NVIDIA GPU Operator¶

The NVIDIA GPU Operator automates the management of NVIDIA GPUs in Kubernetes environments.

Step 1: Install the GPU Operator¶

Navigate to the OpenShift web console
Go to Operators → OperatorHub
Search for “NVIDIA GPU Operator”
Select the operator and click Install
Keep the default installation settings and click Install again

Alternatively, you can install it through the CLI using the following commands:

oc create -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/master/deployments/git/operator-namespace.yaml
oc create -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/master/deployments/git/operator-source.yaml

Step 2: Configure the ClusterPolicy¶

When deploying the operator’s ClusterPolicy, we need to set sandboxWorkloads.enabled to true to enable the sandbox-device-plugin and vfio-manager components, which are essential for GPU passthrough.

sandboxWorkloadsEnabled.yaml¶

kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  sandboxWorkloads:
    defaultWorkload: container
    enabled: true

oc patch ClusterPolicy gpu-cluster-policy --type=merge -p sandboxWorkloadsEnabled.yaml

As the Nvidia GPU Operator does not officialy supports consumer grade GPUs it does not take the audio device into consideration and therefore doesn’t bind it to vfiopci driver. This has to be done manually but can be achieved once at boot time using the following machine config.

vfio-prepare.bu¶

variant: openshift
version: 4.10.0
metadata:
  name: 100-vfio
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  files:
  - path: /usr/local/bin/vfio-prepare
    mode: 0755
    overwrite: true
    contents:
      local: ./vfio-prepare.sh
  - path: /etc/modules-load.d/vfio-pci.conf
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci
systemd:
  units:
    - name: vfioprepare.service
      enabled: true
      contents: |
       [Unit]
       Description=Prepare vfio devices
       After=ignition-firstboot-complete.service
       Before=kubelet.service crio.service

       [Service]
       Type=oneshot
       ExecStart=/usr/local/bin/vfio-prepare

       [Install]
       WantedBy=kubelet.service

vfio-prepare.sh¶

#!/bin/bash

vfio_attach () {
  if [ -f "${path}/driver/unbind" ]; then
    echo $address > ${path}/driver/unbind
  fi
  echo vfio-pci > ${path}/driver_override
  echo $address > /sys/bus/pci/drivers/vfio-pci/bind || \
  echo $name > /sys/bus/pci/drivers/vfio-pci/new_id ||true
}

# 0a:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
address=0000:0a:00.1
path=/sys/bus/pci/devices/0000\:0a\:00.1
name="10de 1467"
vfio_attach

cd articles/openshift-workstation/machineconfig/build
butane -d . vfio-prepare.bu -o ../vfio-prepare.yaml
oc patch MachineConfig 100-vfio --type=merge -p ../vfio-prepare.yaml

Dynamically Switching GPU Drivers¶

One of the key advantages of this setup is the ability to use a single GPU for both container workloads and virtual machines without rebooting the system.

Use Case Scenario¶

Our workstation has a single NVIDIA GPU
Container workloads (such as AI/ML applications) require the NVIDIA kernel driver
Virtual machines with GPU passthrough require the VFIO-PCI driver
We need to switch between these modes without system reboots

Driver Switching Using Node Labels¶

The NVIDIA GPU Operator with sandbox workloads enabled provides a convenient way to switch driver bindings using node labels:

For container workloads (NVIDIA driver):

# Replace 'da2' with your node name
oc label node da2 --overwrite nvidia.com/gpu.workload.config=container

For VM passthrough (VFIO-PCI driver):

# Replace 'da2' with your node name
oc label node da2 --overwrite nvidia.com/gpu.workload.config=vm-passthrough

Notes on Driver Switching¶

The driver switching process takes a few minutes to complete
You can verify the current driver binding with lspci -nnk | grep -A3 NVIDIA
All GPU workloads must be stopped before switching drivers
No system reboot isually required for the switch to take effect
This have prouved to be a bit unreliable and may require a reboot

Add GPU as Hardware Device of your node¶

We indentify the Vendor and Product ID of the GPU

lspci -nnk |grep VGA

We indentify the device name provided by the gpu-feature-discovery.

oc get nodes da2 -ojson |jq .status.capacity |grep nvidia

kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  permittedHostDevices:
    pciHostDevices:
    - externalResourceProvider: true
      pciDeviceSelector: 10DE:2206
      resourceName: nvidia.com/GA102_GEFORCE_RTX_3080

oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv  --type=merge -d hyperconverged.yaml

The pciDeviceSelector field specifies the vendor ID and device ID of the PCI device, while the resourceName field specifies the name of the resource that will be created in Kubernetes/OpenShift.

Passthrough USB Host Controllers to the VM¶

For a complete desktop experience, you’ll want to connect input devices (mouse, keyboard) and audio devices directly to your virtual machine. Instead of passthrough individual USB devices, we’ll passthrough an entire USB controller to the VM for better performance and flexibility.

Step 1: Identify a Suitable USB Controller¶

First, we need to identify an appropriate USB controller that we can dedicate to the virtual machine:

List all PCI devices on your system:
lspci -nnk | grep -i usb
Example output: ` 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] `
Note the PCI address (e.g., 0b:00.3) and the device ID (1022:149c in the example).

Verify the IOMMU group of the controller to ensure it can be safely passed through:

find /sys/kernel/iommu_groups/ -iname "*0b:00.3*"
# Shows which IOMMU group contains this device

ls /sys/kernel/iommu_groups/27/devices/
# Lists all devices in the same IOMMU group

Important: For clean passthrough, the USB controller should ideally be alone in its IOMMU group. If other devices are in the same group, you’ll need to pass those through as well.

Add the USB Controller as Hardware Device of your node¶

Once identified we add its Vendor and product IDs to the list of permitted Host Devices.

Currently, Kubevirt does not allow providing a specific PCI address, therefore the pciDeviceSelector will match all similar USB Host Controller from the node. However, as we will only bind the one we are interested in to the VFIO-PCI driver the other ones will not be available for pci passthrough.

kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  permittedHostDevices:
    pciHostDevices:
      - pciDeviceSelector: 1022:149C
        resourceName: devices.kubevirt.io/USB3_Controller
      - pciDeviceSelector: 8086:2723
        resourceName: intel.com/WIFI_Controller

oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv  --type=merge -d hyperconverged.yaml

Binding the USB Controller to VFIO-PCI driver at boot time¶

vfio-prepare.bu¶

variant: openshift
version: 4.10.0
metadata:
  name: 100-vfio
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  files:
  - path: /usr/local/bin/vfio-prepare
    mode: 0755
    overwrite: true
    contents:
      local: ./vfio-prepare.sh
  - path: /etc/modules-load.d/vfio-pci.conf
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci
  - path: /etc/modprobe.d/vfio.conf
    mode: 0644
    overwrite: true
    contents:
      inline: |
        options vfio-pci ids=8086:2723,1022:149c
systemd:
  units:
    - name: vfioprepare.service
      enabled: true
      contents: |
       [Unit]
       Description=Prepare vfio devices
       After=ignition-firstboot-complete.service
       Before=kubelet.service crio.service

       [Service]
       Type=oneshot
       ExecStart=/usr/local/bin/vfio-prepare

       [Install]
       WantedBy=kubelet.service
openshift:
  kernel_arguments:
    - amd_iommu=on
    - vga=off
    - rdblaclist=nouveau
    - 'video=efifb:off'

Create a bash script to unbind specific PCI devices and bind them to the VFIO-PCI driver.

vfio-prepare.sh¶

#!/bin/bash

vfio_attach () {
  if [ -f "${path}/driver/unbind" ]; then
    echo $address > ${path}/driver/unbind
  fi
  echo vfio-pci > ${path}/driver_override
  echo $address > /sys/bus/pci/drivers/vfio-pci/bind || \
  echo $name > /sys/bus/pci/drivers/vfio-pci/new_id ||true
}

# 0a:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
address=0000:0a:00.1
path=/sys/bus/pci/devices/0000\:0a\:00.1
name="10de 1467"
vfio_attach

# Bind "useless" device to vfio-pci to satisfy IOMMU group
address=0000:07:00.0
path=/sys/bus/pci/devices/0000\:07\:00.0
name="1043 87c0"
vfio_attach

# Unbind USB switch and handle via vfio-pci kernel driver
address=0000:07:00.1
path=/sys/bus/pci/devices/0000\:07\:00.1
name="1043 87c0"
vfio_attach

# Unbind USB switch and handle via vfio-pci kernel driver
address=0000:07:00.3
path=/sys/bus/pci/devices/0000\:07\:00.3
name="1022 149c"
vfio_attach

# Unbind USB switch and handle via vfio-pci kernel driver
address=0000:0c:00.3
path=/sys/bus/pci/devices/0000\:0c\:00.3
name="1022 148c"
vfio_attach

cd articles/openshift-workstation/machineconfig/build
butane -d . vfio-prepare.bu -o ../vfio-prepare.yaml
oc patch MachineConfig 100-vfio --type=merge -p ../vfio-prepare.yaml

Creating a Virtual Machine with GPU Passthrough¶

This section guides you through creating virtual machines that can utilize the GPU via PCI passthrough. We’ll use existing LVM Logical Volumes where the operating system is already installed with UEFI boot.

Step 1: Create Persistent Volumes from LVM Disks¶

First, we need to make our LVM volumes available to OpenShift by creating a Persistent Volume Claims (PVCs). This assume you have the Local Storage Operator installed and running.

Create a YAML file for each VM disk. Here’s an example for a Fedora 35 VM:

fedora_pvc.yaml¶

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: fedora35
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 100Gi 
  storageClassName: lvms-vg1

Apply the YAML to create the PV and PVC:

oc apply -f fedora35.yaml

Verify the PV and PVC are created and bound:

oc get pv
oc get pvc -n <your-namespace>

Step 2: Defining the Virtual Machine with GPU Passthrough¶

When creating virtual machines for desktop use with GPU passthrough, several important configurations need to be applied:

Key Configuration Elements¶

GPU Passthrough: Pass the entire physical GPU to the VM

See also

https://kubevirt.io/user-guide/virtual_machines/host-devices/#pci-passthrough
Disable Virtual VGA: Remove the default emulated VGA device since we’re using the physical GPU

See also

https://kubevirt.io/api-reference/master/definitions.html#_v1_devices
USB Controller Passthrough: Include the USB controller for connecting peripherals directly
UEFI Boot: Use UEFI boot mode for compatibility with modern operating systems and GPU drivers

See also

https://docs.openshift.com/container-platform/4.12/virt/virtual_machines/advanced_vm_management/virt-efi-mode-for-vms.html
CPU/Memory Configuration: Allocate appropriate resources based on workload requirements

fedora.yaml¶

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: fedora
  namespace: epheo
spec:
  runStrategy: Halted
  template:
    metadata:
      labels:
        kubevirt.io/domain: fedora
    spec:
      architecture: amd64
      domain:
        cpu:
          cores: 8
          model: host-passthrough
          sockets: 2
          threads: 1
        features:
          acpi: {}
          smm:
            enabled: true 
        firmware:
          bootloader:
            efi:
              secureBoot: false # For Nvidia Driver...
        devices:
          disks:
            - bootOrder: 1
              disk:
                bus: virtio
              name: pvdisk
            - disk:
                bus: virtio
              name: cloudinitdisk
          autoattachGraphicsDevice: false
          gpus:
          - deviceName: nvidia.com/GA102_GEFORCE_RTX_3080
            name: gpuvideo
          hostDevices:
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller2
          - deviceName: intel.com/WIFI_Controller
            name: wificontroller
          interfaces:
          - masquerade: {}
            name: default
          - bridge: {}
            model: virtio
            name: nic-0
          networkInterfaceMultiqueue: true
          rng: {}
        machine:
          type: q35
        resources:
          requests:
            memory: 16G
      hostname: fedora
      networks:
      - name: default
        pod: {}
      - multus:
          networkName: br1
        name: nic-0
      terminationGracePeriodSeconds: 0
      volumes:
        - persistentVolumeClaim:
            claimName: 'fedora35'
          name: pvdisk
        - cloudInitNoCloud:
            userData: |-
              #cloud-config
              password: fedora
              chpasswd: { expire: False }
          name: cloudinitdisk

windows.yaml¶

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    vm.kubevirt.io/os: windows10
    vm.kubevirt.io/workload: desktop
  name: windows
spec:
  runStrategy: Manual
  template:
    metadata:
      labels:
        kubevirt.io/domain: windows
    spec:
      architecture: amd64
      domain:
        clock:
          timer:
            hpet:
              present: false
            hyperv: {}
            pit:
              tickPolicy: delay
            rtc:
              tickPolicy: catchup
          utc: {}
        cpu:
          cores: 8
          dedicatedCpuPlacement: true
          sockets: 2
          threads: 1
        devices:
          autoattachGraphicsDevice: false
          disks:
          - cdrom:
              bus: sata
            name: windows-guest-tools
          - bootOrder: 1
            disk:
              bus: virtio
            name: pvdisk
          - disk:
              bus: virtio
            name: pvdisk1
          gpus:
          - deviceName: nvidia.com/GA102_GEFORCE_RTX_3080
            name: gpuvideo
          hostDevices:
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller2
          - deviceName: intel.com/WIFI_Controller
            name: wificontroller
          interfaces:
          - bridge: {}
            model: virtio
            name: nic-0
          networkInterfaceMultiqueue: true
          rng: {}
          tpm: {}
        features:
          acpi: {}
          apic: {}
          hyperv:
            frequencies: {}
            ipi: {}
            reenlightenment: {}
            relaxed: {}
            reset: {}
            runtime: {}
            spinlocks:
              spinlocks: 8191
            synic: {}
            synictimer:
              direct: {}
            tlbflush: {}
            vapic: {}
            vpindex: {}
          smm: {}
        firmware:
          bootloader:
            efi:
              secureBoot: true
        machine:
          type: q35
        memory:
          hugepages:
            pageSize: 1Gi
        resources:
          requests:
            memory: 32Gi
      evictionStrategy: None
      hostname: windows
      networks:
      - multus:
          networkName: br1
        name: nic-0
      terminationGracePeriodSeconds: 3600
      volumes:
      - containerDisk:
          image: registry.redhat.io/container-native-virtualization/virtio-win-rhel9@sha256:0c536c7aba76eb9c1e75a8f2dc2bbfa017e90314d55b242599ea41f42ba4434f
        name: windows-guest-tools
      - name: pvdisk
        persistentVolumeClaim:
          claimName: windows
      - name: pvdisk1
        persistentVolumeClaim:
          claimName: windowsdata

Unused anymore, for reference only¶

Binding GPU to VFIO Driver at boot time¶

We first gather the PCI Vendor and product IDs from pciutils.

lspci -nn |grep VGA

100-sno-vfiopci.bu¶

variant: openshift
version: 4.10.0
metadata:
  name: 100-sno-vfiopci
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  files:
  - path: /etc/modprobe.d/vfio.conf
    mode: 0644
    overwrite: true
    contents:
      inline: |
        options vfio-pci ids=10de:2206,10de:1aef
  - path: /etc/modules-load.d/vfio-pci.conf 
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci

dnf install butane
butane 100-sno-vfiopci.bu -o 100-sno-vfiopci.yaml
oc apply -f 100-sno-vfiopci.yaml

98-sno-xhci-unbind.yaml¶

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 98-sno-xhci-unbind
spec:
  config:
    ignition:
      version: 3.1.0
    systemd:
      units:
      - contents: |
         [Unit]
         Description=Unbind USB Host Controller Driver
         After=ignition-firstboot-complete.service
         Before=kubelet.service crio.service

         [Service]
         Type=oneshot
         ExecStart=/bin/bash -c "/bin/echo 0000:0b:00.3 > /sys/bus/pci/devices/0000\\:0b\\:00.3/driver/unbind"
         ExecStart=/bin/bash -c "/bin/echo vfio-pci > /sys/bus/pci/devices/0000\\:0b\\:00.3/driver_override"
         ExecStart=/bin/bash -c "/bin/echo 1043 87c0 > /sys/bus/pci/drivers/vfio-pci/new_id"

         [Install]
         WantedBy=kubelet.service
        enabled: true
        name: unbindusbcontroller.service

Unbinding VTConsole at boot time¶

See also

https://docs.kernel.org/fb/fbcon.html

98-sno-vtconsole-unbind.yaml¶

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 98-sno-vtconsole-unbind
spec:
  config:
    ignition:
      version: 3.1.0
    systemd:
      units:
      - contents: |
         [Unit]
         Description=Dettach GPU VT Console 
         After=ignition-firstboot-complete.service
         Before=kubelet.service crio.service

         [Service]
         Type=oneshot
         ExecStart=/bin/bash -c "/bin/echo 0 > /sys/class/vtconsole/vtcon0/bind"

         [Install]
         WantedBy=kubelet.service
        enabled: true
        name: dettachvtconsole.service

What’s next¶

This chapter is kept as a reference for furture possible improvements.

Reducing the Control Plane footprint by relaying on microshift instead.
Using GPU from containers instead of virtual machines for Linux Desktop.

Replace node prep by qemu hooks¶

https://github.com/kubevirt/kubevirt/blob/main/examples/vmi-with-sidecar-hook.yaml

Enabling dedicated resources for virtual machines¶

Using MicroShift and RHEL for Edge¶

Troubleshooting¶

This section covers common issues you might encounter when setting up GPU passthrough with OpenShift and their solutions.

IOMMU Group Viability Issues¶

Problem: Virtual machine fails to start with an error similar to:

{"component":"virt-launcher","level":"error","msg":"Failed to start VirtualMachineInstance",
"reason":"virError... vfio 0000:07:00.1: group 19 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver."}

Diagnosis: This error occurs when not all devices in the same IOMMU group are bound to the vfio-pci driver. To check the IOMMU group:

# Check which devices are in the same IOMMU group
ls /sys/kernel/iommu_groups/19/devices/
# Output shows multiple devices in the group:
# 0000:03:08.0  0000:07:00.0  0000:07:00.1  0000:07:00.3

# Check what one of these devices is
lspci -nnks 07:00.0
# Output: AMD Starship/Matisse Reserved SPP [1022:1485]

Solution: All devices in the same IOMMU group need to be bound to the vfio-pci driver. Modify your vfio-prepare.sh script to include all devices in the IOMMU group:

# Add these lines to your vfio-prepare.sh script
echo "vfio-pci" > /sys/bus/pci/devices/0000:03:08.0/driver_override
echo "vfio-pci" > /sys/bus/pci/devices/0000:07:00.0/driver_override
echo "vfio-pci" > /sys/bus/pci/devices/0000:07:00.1/driver_override
echo "vfio-pci" > /sys/bus/pci/devices/0000:07:00.3/driver_override

# Make sure to unbind from current drivers first and then bind to vfio-pci
# as shown in the vfio-prepare.sh script example

Other Common Issues¶

No display output after GPU passthrough:

Ensure you’ve disabled the virtual VGA device in the VM specification
Check that you’ve passed through both the GPU and its audio device
Install the appropriate GPU drivers inside the virtual machine

Performance issues in Windows VM:

Ensure CPU pinning is configured correctly
Consider enabling huge pages for memory performance
Install the latest NVIDIA drivers from within the VM
Disable the Windows Game Bar and other overlay software

GPU driver switching fails:

Verify all GPU workloads are stopped before switching
Check the GPU operator pod logs: oc logs -n nvidia-gpu-operator <pod-name>
Verify IOMMU is properly enabled in BIOS/UEFI settings

For more troubleshooting help, check the logs of the following components:

virt-handler: oc logs -n openshift-cnv virt-handler-<hash>
virt-launcher: oc logs -n <namespace> virt-launcher-<vm-name>-<hash>
nvidia-driver-daemonset: oc logs -n nvidia-gpu-operator nvidia-driver-daemonset-<hash>