OpenShift Workstation with Single GPU passthrough#

Feb 27, 2023

25 min read

Introduction#

This article describes how to run OpenShift as a workstation with GPU PCI passthrough and Container Native Virtualization (CNV) to provide a virtualized desktop experience on a single OpenShift node. This is useful to provide a virtual desktop experience with a single GPU, and is used to run Microsoft FlightSimulator in a Windows VM with performances close from a Bare metal Windows installation.

Hardware description#

The workstation used for this demo has the following hardware:

AMD Ryzen 9 3950X 16-Core 32-Threads
64GB DDR4 3200MHz
Nvidia RTX 3080 FE 10GB
2x 2TB NVMe Disks (guests)
1x 500GB SSD Disk (root system)
10Gbase-CX4 Mellanox Ethernet

Backup of existing system partitions#

To avoid boot order conflicts, the OpenShift assisted installer will format the first 512 bytes of any disks that contain a bootable partition. Therefore, it is important to backup and remove any existing partition table that you would like to preserve.

Installing OpenShift SNO#

Once any existing file system is backed up and there is no more bootable partitions we can proceed with the OpenShift Single Node install.

It is important to note that CoreOS, the underlying operating system requires an entire disk for installation.

Here, we will keep the two NVMe disks for the persistant volumes as LVM Physical volumes belonging to a same Volume Group and we will use the SSD disk for the OpenShift operating system.

#!/bin/bash

OCP_VERSION=latest-4.10

curl -k https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux.tar.gz > oc.tar.gz
tar zxf oc.tar.gz
chmod +x oc && mv oc ~/.local/bin/

curl -k https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$OCP_VERSION/openshift-install-linux.tar.gz > openshift-install-linux.tar.gz
tar zxvf openshift-install-linux.tar.gz
chmod +x openshift-install && mv openshift-install ~/.local/bin/

curl $(openshift-install coreos print-stream-json | grep location | grep x86_64 | grep iso | cut -d\" -f4) > rhcos-live.x86_64.iso

install-config.yaml#

# This file contains the configuration for an OpenShift cluster installation.

apiVersion: v1

# The base domain for the cluster.
baseDomain: epheo.eu

# Configuration for the compute nodes.
compute:
- name: worker
  replicas: 0 

# Configuration for the control plane nodes.
controlPlane:
  name: master
  replicas: 1 

# Metadata for the cluster.
metadata:
  name: da2

# Networking configuration for the cluster.
networking:
  networkType: OVNKubernetes
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16

# Platform configuration for the cluster.
platform:
  none: {}

# Configuration for bootstrapping the cluster.
bootstrapInPlace:
  installationDisk: /dev/sda

# Pull secret for accessing the OpenShift registry.
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"XXXXXXXX"}}}' 

# SSH key for accessing the cluster nodes.
sshKey: |
  ssh-rsa AAAAB3XXXXXXXXXXXXXXXXXXXXXXXXX

Generate OpenShift Container Platform assets#

mkdir ocp && cp install-config.yaml ocp
openshift-install --dir=ocp create single-node-ignition-config

Embed the ignition data into the RHCOS ISO:#

alias coreos-installer='podman run --privileged --rm \
      -v /dev:/dev -v /run/udev:/run/udev -v $PWD:/data \
      -w /data quay.io/coreos/coreos-installer:release'
cp ocp/bootstrap-in-place-for-live-iso.ign iso.ign
coreos-installer iso ignition embed -fi iso.ign rhcos-live.x86_64.iso
dd if=discovery_image_sno.iso of=/dev/usbkey status=progress

Once the ISO is copied to the USB drive, you can use the USB drive to boot your workstation node and install OpenShift Container Platform.

Install CNV Operator#

Activate Intel VT or AMD-V hardware virtualization extensions in BIOS or UEFI.

cnv-resources.yaml#

# This YAML file contains Kubernetes resources for installing the KubeVirt Hyperconverged Operator (HCO) on the OpenShift Container Platform.
# It creates a namespace named "openshift-cnv", an operator group named "kubevirt-hyperconverged-group" in the "openshift-cnv" namespace, and a subscription named "hco-operatorhub" in the "openshift-cnv" namespace.
# The subscription specifies the source, source namespace, name, starting CSV, and channel for the KubeVirt Hyperconverged Operator.

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-cnv
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: kubevirt-hyperconverged-group
  namespace: openshift-cnv
spec:
  targetNamespaces:
    - openshift-cnv
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.10.0
  channel: "stable"

oc apply -f cnv-resources.yaml

Installing the Virtctl client on your desktop.#

subscription-manager repos --enable cnv-4.10-for-rhel-8-x86_64-rpms
dnf install kubevirt-virtctl

Remove Local Storage operator (if installed)#

As we do not need to manage LVM volumes automatically we would like to avoid automatically formating Logical Volumes once they are deleted from OpenShift.

While this could lead to data leak in a multi-tenant environment, removing the Local Storage Operator also avoid loosing your Virtual Machine partitions once you delete it.

Configure OpenShift for single GPU passthrough#

As our GPU is the only one attached to the node a few additional steps are required.

We will use MachineConfig to configure our node accordingly.

All MachineConfig are applied on the master machineset because we have a single node OpenShift. With a multi nodes cluster those would be applied to workers instead.

Passing kernel arguments at boot time#

Multiple Kernel arguments have to be passed at boot time in order to configure our node for GPU passthrough. This can be done using the MachineConfigOperator.

amd_iommu=on: Enables IOMMU (Input/Output Memory Management Unit) support for AMD platforms, allowing for direct memory access (DMA) by PCI devices without going through the CPU. This improves performance and reduces overhead.
vga=off: Disables VGA (Video Graphics Array) console output during boot time.
rdblaclist=nouveau: Enables the blacklisting of the Nouveau open-source NVIDIA driver.
video=efifb:off: Disables the EFI (Extensible Firmware Interface) framebuffer console output during boot time.

https://docs.kernel.org/fb/fbcon.html

Setting Kernel Arguments at boot time.#

variant: openshift
version: 4.10.0
metadata:
  name: 100-vfio
  labels:
    machineconfiguration.openshift.io/role: master
openshift:
  kernel_arguments:
    - amd_iommu=on
    - vga=off
    - rdblaclist=nouveau
    - 'video=efifb:off'

cd articles/openshift-workstation/machineconfig/build
butane -d . vfio-prepare.bu -o ../vfio-prepare.yaml
oc patch MachineConfig 100-vfio --type=merge -p ../vfio-prepare.yaml

Note

If you’re using an Intel CPU you’ll have to set intel_iommu=on instead.

Installing and configuring the NVidia GPU Operator#

Install the GPU Operator using OLM / OpenShift Marketplace.

When deploying the operator’s ClusterPolicy we have to set sandboxWorkloads.enabled to true to enable the sandbox-device-plugin and vfio-manager.

sandboxWorkloadsEnabled.yaml#

kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  sandboxWorkloads:
    defaultWorkload: container
    enabled: true

oc patch ClusterPolicy gpu-cluster-policy --type=merge -p sandboxWorkloadsEnabled.yaml

As the Nvidia GPU Operator does not supports consumer grade GPUs it does not take the audio device into consideration and therefore doesn’t bind it to vfiopci driver. This has to be done manually but can be achieved once at boot time using the following machine config.

vfio-prepare.bu#

variant: openshift
version: 4.10.0
metadata:
  name: 100-vfio
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  files:
  - path: /usr/local/bin/vfio-prepare
    mode: 0755
    overwrite: true
    contents:
      local: ./vfio-prepare.sh
  - path: /etc/modules-load.d/vfio-pci.conf
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci
systemd:
  units:
    - name: vfioprepare.service
      enabled: true
      contents: |
       [Unit]
       Description=Prepare vfio devices
       After=ignition-firstboot-complete.service
       Before=kubelet.service crio.service

       [Service]
       Type=oneshot
       ExecStart=/usr/local/bin/vfio-prepare

       [Install]
       WantedBy=kubelet.service

vfio-prepare.sh#

#!/bin/bash

vfio_attach () {
  if [ -f "${path}/driver/unbind" ]; then
    echo $address > ${path}/driver/unbind
  fi
  echo vfio-pci > ${path}/driver_override
  echo $address > /sys/bus/pci/drivers/vfio-pci/bind || \
  echo $name > /sys/bus/pci/drivers/vfio-pci/new_id ||true
}

# 0a:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
address=0000:0a:00.1
path=/sys/bus/pci/devices/0000\:0a\:00.1
name="10de 1467"
vfio_attach

cd articles/openshift-workstation/machineconfig/build
butane -d . vfio-prepare.bu -o ../vfio-prepare.yaml
oc patch MachineConfig 100-vfio --type=merge -p ../vfio-prepare.yaml

Changing the driver binded to the GPU#

This workstation only have a single GPU.
I’d like to use it for both Virtual Machines and AI/ML workload.
Containers requires the GPU device to bind to the Nvidia driver.
Virtual machines requires the GPU device to bind to the VFIO-PCI driver.
I’d like an efficient way to bind / unbind the GPU to a driver without reboot.

We can label the node in order to configure it with the GPU bound to Nvidia kernel driver in order to satisky container workloads.

oc label node da2 --overwrite nvidia.com/gpu.workload.config=container

Or to bind the GPU to the vfio-pci driver to satisfy Virtual Machines workloads with PCI passthrough.

oc label node da2 --overwrite nvidia.com/gpu.workload.config=vm-passthrough

The whole operation takes a few minutes.

Add GPU as Hardware Device of your node#

We indentify the Vendor and Product ID of the GPU

lspci -nnk |grep VGA

We indentify the device name provided by the gpu-feature-discovery.

oc get nodes da2 -ojson |jq .status.capacity |grep nvidia

kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  permittedHostDevices:
    pciHostDevices:
    - externalResourceProvider: true
      pciDeviceSelector: 10DE:2206
      resourceName: nvidia.com/GA102_GEFORCE_RTX_3080

oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv  --type=merge -d hyperconverged.yaml

The pciDeviceSelector field specifies the vendor ID and device ID of the PCI device, while the resourceName field specifies the name of the resource that will be created in Kubernetes/OpenShift.

Passthrough the USB Host Controllers to the VM#

In order to directly connect a mouse, keyboard, audio device etc directly to the VM we passthrough one if the USB controller directly to the VM.

Identify a USB Controller and its IOMMU group#

https://docs.openshift.com/container-platform/4.8/virt/virtual_machines/advanced_vm_management/virt-configuring-pci-passthrough.html

We first need to indentify it using pciutils.

lspci -nnk

After selecting the USB Controller we want to dedicate to the Virtual Machine we should verify that this is the only PCI device in its IOMMU group. We first look for the PCI address in the iommu_groups folder structure, the list the PCI addresses belonging to this IOMMU group.

find /sys/kernel/iommu_groups/ -iname "*0b:00.3*"
ls /sys/kernel/iommu_groups/27/devices/

Add the USB Controller as Hardware Device of your node#

Once identified we add its Vendor and product IDs to the list of permitted Host Devices.

Currently, Kubevirt does not allow providing a specific PCI address, therefore the pciDeviceSelector will match all similar USB Host Controller from the node. However, as we will only bind the one we are interested in to the VFIO-PCI driver the other ones will not be available for pci passthrough.

kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  permittedHostDevices:
    pciHostDevices:
      - pciDeviceSelector: 1022:149C
        resourceName: devices.kubevirt.io/USB3_Controller
      - pciDeviceSelector: 8086:2723
        resourceName: intel.com/WIFI_Controller

oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv  --type=merge -d hyperconverged.yaml

Binding the USB Controller to VFIO-PCI driver at boot time#

vfio-prepare.bu#

variant: openshift
version: 4.10.0
metadata:
  name: 100-vfio
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  files:
  - path: /usr/local/bin/vfio-prepare
    mode: 0755
    overwrite: true
    contents:
      local: ./vfio-prepare.sh
  - path: /etc/modules-load.d/vfio-pci.conf
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci
  - path: /etc/modprobe.d/vfio.conf
    mode: 0644
    overwrite: true
    contents:
      inline: |
        options vfio-pci ids=8086:2723,1022:149c
systemd:
  units:
    - name: vfioprepare.service
      enabled: true
      contents: |
       [Unit]
       Description=Prepare vfio devices
       After=ignition-firstboot-complete.service
       Before=kubelet.service crio.service

       [Service]
       Type=oneshot
       ExecStart=/usr/local/bin/vfio-prepare

       [Install]
       WantedBy=kubelet.service
openshift:
  kernel_arguments:
    - amd_iommu=on
    - vga=off
    - rdblaclist=nouveau
    - 'video=efifb:off'

Create a bash script to unbind specific PCI devices and bind them to the VFIO-PCI driver.

vfio-prepare.sh#

#!/bin/bash

vfio_attach () {
  if [ -f "${path}/driver/unbind" ]; then
    echo $address > ${path}/driver/unbind
  fi
  echo vfio-pci > ${path}/driver_override
  echo $address > /sys/bus/pci/drivers/vfio-pci/bind || \
  echo $name > /sys/bus/pci/drivers/vfio-pci/new_id ||true
}

# 0a:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
address=0000:0a:00.1
path=/sys/bus/pci/devices/0000\:0a\:00.1
name="10de 1467"
vfio_attach

# Bind "useless" device to vfio-pci to satisfy IOMMU group
address=0000:07:00.0
path=/sys/bus/pci/devices/0000\:07\:00.0
name="1043 87c0"
vfio_attach

# Unbind USB switch and handle via vfio-pci kernel driver
address=0000:07:00.1
path=/sys/bus/pci/devices/0000\:07\:00.1
name="1043 87c0"
vfio_attach

# Unbind USB switch and handle via vfio-pci kernel driver
address=0000:07:00.3
path=/sys/bus/pci/devices/0000\:07\:00.3
name="1022 149c"
vfio_attach

# Unbind USB switch and handle via vfio-pci kernel driver
address=0000:0c:00.3
path=/sys/bus/pci/devices/0000\:0c\:00.3
name="1022 148c"
vfio_attach

cd articles/openshift-workstation/machineconfig/build
butane -d . vfio-prepare.bu -o ../vfio-prepare.yaml
oc patch MachineConfig 100-vfio --type=merge -p ../vfio-prepare.yaml

Creating a Virtual Machine#

The virtual machine will use existing LVM Logical volumes, here we will assume we already have the Operating System installed on the LV with a UEFI boot.

Create PV and PV Claim out of local LVM disks#

Binding PV and PVC by label https://docs.openshift.com/container-platform/3.3/install_config/storage_examples/binding_pv_by_label.html

fedora35.yaml#

---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: fedora35
  labels:
    vol: fedora35
spec:
  capacity:
    storage: 100Gi
  local:
    path: /dev/fedora_da2/fedora35
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  volumeMode: Block
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - da2
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: fedora35
  labels:
    vol: fedora35
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 100Gi 
  storageClassName: local-storage

Defining the Virtual Machine#

The virtual machines we will use as Desktops comes with a few specities:

We will passthrough the entire GPU | Ref: https://kubevirt.io/2021/intel-vgpu-kubevirt.html
We will remove the existing default virtual VGA | Ref: https://kubevirt.io/api-reference/master/definitions.html#_v1_devices
We will passthrough an entire USB controller
We will use UEFI boot to be closer from typical BareMetal | Ref: https://docs.openshift.com/container-platform/4.10/virt/virtual_machines/advanced_vm_management/virt-efi-mode-for-vms.html

fedora.yaml#

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: fedora
  namespace: epheo
spec:
  runStrategy: Halted
  template:
    metadata:
      labels:
        kubevirt.io/domain: fedora
    spec:
      architecture: amd64
      domain:
        cpu:
          cores: 8
          model: host-passthrough
          sockets: 2
          threads: 1
        features:
          acpi: {}
          smm:
            enabled: true 
        firmware:
          bootloader:
            efi:
              secureBoot: false # For Nvidia Driver...
        devices:
          disks:
            - bootOrder: 1
              disk:
                bus: virtio
              name: pvdisk
            - disk:
                bus: virtio
              name: cloudinitdisk
          autoattachGraphicsDevice: false
          gpus:
          - deviceName: nvidia.com/GA102_GEFORCE_RTX_3080
            name: gpuvideo
          hostDevices:
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller2
          - deviceName: intel.com/WIFI_Controller
            name: wificontroller
          interfaces:
          - masquerade: {}
            name: default
          - bridge: {}
            model: virtio
            name: nic-0
          networkInterfaceMultiqueue: true
          rng: {}
        machine:
          type: q35
        resources:
          requests:
            memory: 16G
      hostname: fedora
      networks:
      - name: default
        pod: {}
      - multus:
          networkName: br1
        name: nic-0
      terminationGracePeriodSeconds: 0
      volumes:
        - persistentVolumeClaim:
            claimName: 'fedora35'
          name: pvdisk
        - cloudInitNoCloud:
            userData: |-
              #cloud-config
              password: fedora
              chpasswd: { expire: False }
          name: cloudinitdisk

windows.yaml#

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    vm.kubevirt.io/os: windows10
    vm.kubevirt.io/workload: desktop
  name: windows
spec:
  runStrategy: Manual
  template:
    metadata:
      labels:
        kubevirt.io/domain: windows
    spec:
      architecture: amd64
      domain:
        clock:
          timer:
            hpet:
              present: false
            hyperv: {}
            pit:
              tickPolicy: delay
            rtc:
              tickPolicy: catchup
          utc: {}
        cpu:
          cores: 8
          dedicatedCpuPlacement: true
          sockets: 2
          threads: 1
        devices:
          autoattachGraphicsDevice: false
          disks:
          - cdrom:
              bus: sata
            name: windows-guest-tools
          - bootOrder: 1
            disk:
              bus: virtio
            name: pvdisk
          - disk:
              bus: virtio
            name: pvdisk1
          gpus:
          - deviceName: nvidia.com/GA102_GEFORCE_RTX_3080
            name: gpuvideo
          hostDevices:
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller2
          - deviceName: intel.com/WIFI_Controller
            name: wificontroller
          interfaces:
          - bridge: {}
            model: virtio
            name: nic-0
          networkInterfaceMultiqueue: true
          rng: {}
          tpm: {}
        features:
          acpi: {}
          apic: {}
          hyperv:
            frequencies: {}
            ipi: {}
            reenlightenment: {}
            relaxed: {}
            reset: {}
            runtime: {}
            spinlocks:
              spinlocks: 8191
            synic: {}
            synictimer:
              direct: {}
            tlbflush: {}
            vapic: {}
            vpindex: {}
          smm: {}
        firmware:
          bootloader:
            efi:
              secureBoot: true
        machine:
          type: q35
        memory:
          hugepages:
            pageSize: 1Gi
        resources:
          requests:
            memory: 32Gi
      evictionStrategy: None
      hostname: windows
      networks:
      - multus:
          networkName: br1
        name: nic-0
      terminationGracePeriodSeconds: 3600
      volumes:
      - containerDisk:
          image: registry.redhat.io/container-native-virtualization/virtio-win-rhel9@sha256:0c536c7aba76eb9c1e75a8f2dc2bbfa017e90314d55b242599ea41f42ba4434f
        name: windows-guest-tools
      - name: pvdisk
        persistentVolumeClaim:
          claimName: windows
      - name: pvdisk1
        persistentVolumeClaim:
          claimName: windowsdata

Unused anymore, for reference only#

Binding GPU to VFIO Driver at boot time#

We first gather the PCI Vendor and product IDs from pciutils.

lspci -nn |grep VGA

100-sno-vfiopci.bu#

variant: openshift
version: 4.10.0
metadata:
  name: 100-sno-vfiopci
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  files:
  - path: /etc/modprobe.d/vfio.conf
    mode: 0644
    overwrite: true
    contents:
      inline: |
        options vfio-pci ids=10de:2206,10de:1aef
  - path: /etc/modules-load.d/vfio-pci.conf 
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci

dnf install butane
butane 100-sno-vfiopci.bu -o 100-sno-vfiopci.yaml
oc apply -f 100-sno-vfiopci.yaml

98-sno-xhci-unbind.yaml#

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 98-sno-xhci-unbind
spec:
  config:
    ignition:
      version: 3.1.0
    systemd:
      units:
      - contents: |
         [Unit]
         Description=Unbind USB Host Controller Driver
         After=ignition-firstboot-complete.service
         Before=kubelet.service crio.service

         [Service]
         Type=oneshot
         ExecStart=/bin/bash -c "/bin/echo 0000:0b:00.3 > /sys/bus/pci/devices/0000\\:0b\\:00.3/driver/unbind"
         ExecStart=/bin/bash -c "/bin/echo vfio-pci > /sys/bus/pci/devices/0000\\:0b\\:00.3/driver_override"
         ExecStart=/bin/bash -c "/bin/echo 1043 87c0 > /sys/bus/pci/drivers/vfio-pci/new_id"

         [Install]
         WantedBy=kubelet.service
        enabled: true
        name: unbindusbcontroller.service

Unbinding VTConsole at boot time#

See also

https://docs.kernel.org/fb/fbcon.html

98-sno-vtconsole-unbind.yaml#

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 98-sno-vtconsole-unbind
spec:
  config:
    ignition:
      version: 3.1.0
    systemd:
      units:
      - contents: |
         [Unit]
         Description=Dettach GPU VT Console 
         After=ignition-firstboot-complete.service
         Before=kubelet.service crio.service

         [Service]
         Type=oneshot
         ExecStart=/bin/bash -c "/bin/echo 0 > /sys/class/vtconsole/vtcon0/bind"

         [Install]
         WantedBy=kubelet.service
        enabled: true
        name: dettachvtconsole.service

What’s next#

This chapter is kept as a reference for furture possible improvements.

Reducing the Control Plane footprint by relaying on microshift instead.
Using GPU from containers instead of virtual machines for Linux Desktop.

Replace node prep by qemu hooks#

https://github.com/kubevirt/kubevirt/blob/main/examples/vmi-with-sidecar-hook.yaml

Enabling dedicated resources for virtual machines#

Using MicroShift and RHEL for Edge#

Troubleshooting#

{"component":"virt-launcher","kind":"","level":"error","msg":"Failed to start VirtualMachineInstance with flags 0.","name":"windows","namespace":"epheo","pos":"manager.go:1027","reason":"virError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2024-02-08T13:10:06.726594Z qemu-kvm: -device {\"driver\":\"vfio-pci\",\"host\":\"0000:07:00.1\",\"id\":\"ua-hostdevice-usbcontroller\",\"bus\":\"pci.9\",\"addr\":\"0x0\"}: vfio 0000:07:00.1: group 19 is not viable\nPlease ensure all devices within the iommu_group are bound to their vfio bus driver.')","timestamp":"2024-02-08T13:10:07.353704Z","uid":"cc6fa39c-db31-4f2e-bba1-42dfc4b6efad"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Failed to sync vmi","name":"windows","namespace":"epheo","pos":"server.go:202","reason":"virError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2024-02-08T13:10:06.726594Z qemu-kvm: -device {\"driver\":\"vfio-pci\",\"host\":\"0000:07:00.1\",\"id\":\"ua-hostdevice-usbcontroller\",\"bus\":\"pci.9\",\"addr\":\"0x0\"}: vfio 0000:07:00.1: group 19 is not viable\nPlease ensure all devices within the iommu_group are bound to their vfio bus driver.')","timestamp":"2024-02-08T13:10:07.353770Z","uid":"cc6fa39c-db31-4f2e-bba1-42dfc4b6efad"}

[core@da2 ~]$ ls /sys/kernel/iommu_groups/19/devices/
0000:03:08.0  0000:07:00.0  0000:07:00.1  0000:07:00.3

[core@da2 ~]$ lspci -nnks 07:00.0
07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
    Subsystem: ASUSTeK Computer Inc. Device [1043:87c0]