blog.epheo.eu

Debugging PostgreSQL Crash Loop in OpenShift

Sun, 15 Feb 2026 00:00:00

April 20, 2024

3 min read

Introduction

This article describes how to fix a common PostgreSQL issue in OpenShift when the database enters a crash loop with a “tuple concurrently updated” error. This problem typically occurs due to an unclean shutdown of the PostgreSQL server, leaving the database in an inconsistent state.

Understanding the Error

When starting a PostgreSQL pod in OpenShift, you might encounter the following error:

pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....LOG:  redirecting log output to logging
collector process
HINT:  Future log output will appear in directory "pg_log".
..... done
server started
=> sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
ERROR:  tuple concurrently updated

This error indicates that PostgreSQL has detected an issue with its internal data consistency. The “tuple concurrently updated” message suggests that a database tuple (row) was modified by multiple processes simultaneously, leaving the database in an inconsistent state.

Step-by-Step Solution

Follow these steps to resolve the issue:

Find the problematic PostgreSQL pod

First, locate the PostgreSQL pod that is stuck in the crash loop.
Start a debug session

Use the OpenShift command-line tool to start a debug session with the pod:
```
oc debug pod/<postgres-pod-name>
```
Scale down the deployment

In another terminal, scale the associated PostgreSQL deployment to zero pods:
```
oc scale deployment/<postgres-deployment-name> --replicas=0
```
Run the PostgreSQL startup script

From the debug session terminal, run the PostgreSQL startup script:
```
run-postgresql
```
This creates necessary configuration files that will allow you to manage the PostgreSQL server. You should see the same error output described above.

Stop PostgreSQL cleanly

Stop the PostgreSQL server with the following command:

pg_ctl stop -D /var/lib/pgsql/data/userdata

Expected output:

waiting for server to shut down.... done
server stopped

Start PostgreSQL manually

Start the PostgreSQL server manually to check if it initializes correctly:

pg_ctl start -D /var/lib/pgsql/data/userdata

Expected output:

server starting
LOG:  redirecting log output to logging collector process
HINT:  Future log output will appear in directory "pg_log".

The server should remain running without errors.

Stop PostgreSQL cleanly again

Ensure a clean shutdown by stopping PostgreSQL:

pg_ctl stop -D /var/lib/pgsql/data/userdata

Expected output:

waiting for server to shut down.... done
server stopped

Exit the debug session

Type exit to leave the debug session.
Scale up the deployment

Finally, scale the PostgreSQL deployment back up:
```
oc scale deployment/<postgres-deployment-name> --replicas=1
```
The PostgreSQL pod should now start normally without crashing.

Why This Works

This procedure works because it:

Allows PostgreSQL to perform a clean shutdown, ensuring all data is properly written
Clears any potentially corrupted transaction logs
Creates the necessary configuration files needed for proper operation
Eliminates race conditions that might occur during the container’s normal startup process

If you encounter this issue frequently with a particular PostgreSQL deployment, consider investigating:

Storage performance issues
Abrupt pod terminations
Resource constraints causing timeouts during shutdown
Improper backup procedures

Running vLLM on Strix Halo

Sun, 15 Feb 2026 00:00:00

Feb 15, 2026

5 min read

So I bought this machine, AMD RYZEN AI MAX+ 395 w/ Radeon 8060S and 128GB GTT RAM, and I knew that the ecosystem was still… ongoing. But it really felt like going back to the OpenStack early days, struggling with nightly builds, unreleased patches, vendor forks, etc. Maybe the issue is just Python in the end.

In short, AMD Strix Halo (gfx1151) isn’t in upstream vLLM’s supported GPU list yet, so this note walks through building vLLM against ROCm nightly (TheRock) on Fedora 43, including the patches needed to work around missing amdsmi support.

Download the full build script

Prerequisites

dnf install -y \
  python3.13 python3.13-devel \
  libatomic \
  gcc gcc-c++ make ffmpeg-free \
  aria2 \
  libdrm-devel zlib-devel openssl-devel procps-ng \
  numactl-devel gperftools-libs libibverbs-utils

Installing ROCm via TheRock nightly

AMD’s TheRock project publishes nightly tarballs to an S3 bucket, indexed by GFX target. We grab the latest one for gfx1151:

GFX=gfx1151
ROCM_PATH=/opt/rocm

S3="https://therock-nightly-tarball.s3.amazonaws.com"
PREFIX="therock-dist-linux-${GFX}-7"

KEY="$(curl -s "${S3}?list-type=2&prefix=${PREFIX}" \
  | grep -oP '(?<=<Key>)[^<]+' \
  | sort -V | tail -n1)"

aria2c -x 16 -s 16 -j 16 --file-allocation=none "${S3}/${KEY}" -o therock.tar.gz

mkdir -p "$ROCM_PATH"
tar xzf therock.tar.gz -C "$ROCM_PATH" --strip-components=1
rm therock.tar.gz

Environment setup

TheRock nightlies don’t ship a stable bitcode path, so we locate it dynamically:

BITCODE_PATH="$(find "$ROCM_PATH" -type d -name bitcode -print -quit)"

Then create a profile script so the environment is set for every shell:

cat > /etc/profile.d/rocm-sdk.sh <<EOF
export ROCM_PATH=$ROCM_PATH
export HIP_PLATFORM=amd
export HIP_PATH=$ROCM_PATH
export HIP_CLANG_PATH=$ROCM_PATH/llvm/bin
export HIP_DEVICE_LIB_PATH=$BITCODE_PATH
export PATH=$ROCM_PATH/bin:$ROCM_PATH/llvm/bin:\$PATH
export LD_LIBRARY_PATH=$ROCM_PATH/lib:$ROCM_PATH/lib64:$ROCM_PATH/llvm/lib:\$LD_LIBRARY_PATH
export ROCBLAS_USE_HIPBLASLT=1
export TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1
export VLLM_TARGET_DEVICE=rocm
export HIP_FORCE_DEV_KERNARG=1
export RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES=1
export LD_PRELOAD=/usr/lib64/libtcmalloc_minimal.so.4
EOF

source /etc/profile.d/rocm-sdk.sh

Notable variables:

TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL: enables experimental Triton kernels needed for gfx1151
HIP_FORCE_DEV_KERNARG: required for Strix Halo’s memory model
LD_PRELOAD=libtcmalloc_minimal: significant inference throughput gain

Installing PyTorch from ROCm nightly

AMD publishes per-target nightly wheels. The index URL must match the GFX target:

python -m pip install \
  --index-url "https://rocm.nightlies.amd.com/v2-staging/${GFX}/" \
  --pre torch torchaudio torchvision

Building flash-attention (ROCm fork)

ROCm’s fork of flash-attention includes AMD-specific Triton optimizations:

export FLASH_ATTENTION_TRITON_AMD_ENABLE="TRUE"
git clone https://github.com/ROCm/flash-attention.git /opt/flash-attention
cd /opt/flash-attention && git checkout main_perf
python setup.py install

Patching & building vLLM

vLLM probes amdsmi at import time to detect AMD GPUs, but amdsmi doesn’t know about Strix Halo yet. The patches:

Stub out amdsmi in vllm/platforms/__init__.py: disable the import, force is_rocm = True, and no-op the init/shutdown calls.
Mock amdsmi in vllm/platforms/rocm.py: inject a MagicMock module and hardcode device_name = "gfx1151" so vLLM sees a known ROCm device.
Set the GPU target in CMakeLists.txt: replace the default RDNA4 targets with gfx1151.

The build script applies these patches via an inline Python script, see vllm.sh for the full patch.

git clone https://github.com/vllm-project/vllm.git /opt/vllm
cd /opt/vllm

# Apply the amdsmi patches (see vllm.sh for details)
# ...

# Replace default GPU targets
sed -i "s/gfx1200;gfx1201/${GFX}/" CMakeLists.txt

# Fedora's GCC produces ABI-incompatible extensions — use ROCm's clang
export CC="$ROCM_PATH/llvm/bin/clang"
export CXX="$ROCM_PATH/llvm/bin/clang++"

export ROCM_HOME="$ROCM_PATH"
export PYTORCH_ROCM_ARCH="$GFX"
export HIP_ARCHITECTURES="$GFX"
export AMDGPU_TARGETS="$GFX"
export MAX_JOBS="4"

export CMAKE_ARGS="-DROCM_PATH=$ROCM_PATH -DHIP_PATH=$ROCM_PATH \
  -DAMDGPU_TARGETS=$GFX -DHIP_ARCHITECTURES=$GFX"

python -m pip wheel --no-build-isolation --no-deps -w /tmp/dist -v .
python -m pip install /tmp/dist/*.whl

bitsandbytes (ROCm fork)

For quantized model support, build the ROCm-enabled bitsandbytes fork:

export CMAKE_PREFIX_PATH="$ROCM_PATH"
git clone -b rocm_enabled_multi_backend \
  https://github.com/ROCm/bitsandbytes.git /opt/bitsandbytes
cd /opt/bitsandbytes

cmake -S . \
  -DGPU_TARGETS="$GFX" \
  -DBNB_ROCM_ARCH="$GFX" \
  -DCOMPUTE_BACKEND=hip

make -j"$(nproc)"
python -m pip install . --no-build-isolation --no-deps

How this blog is served to you

Sat, 16 Aug 2025 00:00:00

Aug 16, 2025

5 min read

This blog is just static files, served from a container, behind a Kubernetes Ingress Controller.

So a container image containing a whole userland and a fullfledged webserver with all bells and whistle behind yet another fullfledged proxy.

Now I thought, we’re in 2025 and disk space is expensive (Yes, I wrote that before it actually became true). An Nginx container image is at least 80MiB, If I would write my very own static web server and run it from a scratch container image I could go down to a few hundred KiB and save… Let’s do the maths: a 1TB NVMe is around 70Euros on Amazon (yes it was) so that’s around 0.00007€ per MiB, assuming my container image will be around 500KiB that’s a whooping 0.005 Euro difference, bargain !

So I decided to spend a few days of my summer holidays and go save those 0.005 Euro.

Who need logic when you have a goal.

Anyhow, I started working on it with my friend Claude, got a functionnal prototype, and finaly came back to reason: this is useless.

Unless… unless my 20 monthly visitors were too much for Nginx to handle and I needed a much more performant web server. Let’s set a 1 Million Request per seconds target so if those 20 monthly visitors suddenly decided to connect at the exact same time and all refresh my blog page 50 000 times every seconds, they’d be safe.

So I started working again, and got to a 454KiB container image, that serves around 700K requests per seconds from my laptop’s Intel Gen12 and 1M+ RPS on my server !

I now host my blog with it, and you, who read, are safe to refresh this page if you want. Unless… unless the ingress controler is now the bottleneck ! Let’s work on a new K8S Ingress Controller (or Gateway API) that implements io_uring and use eBPF.

Details

KISS (that’s its name) for “Kubernetes Instant Static Server” is taking some shortcuts to achieve its goals.

The first and more important one is no disk I/O, all content is pre-loaded in RAM and all requests are pre-generated at startup time.

Traditional Static Servers (nginx, Apache):

Read files from disk per request
Generate HTTP headers per request
Multiple system calls per response
Filesystem caching complexity

KISS Approach:

All files loaded into memory at startup
HTTP responses pre-generated once
Single write() system call per request
Zero runtime filesystem operations

If you are also mindfull of saving some disk space and serving a ridiculous amount of request per seconds without any guaranty, you can also try KISS.

Merging kubeconfig Files

Sun, 27 Apr 2025 00:00:00

Apr 25, 2025

3 min read

When working with multiple Kubernetes or OpenShift clusters, you’ll often have multiple kubeconfig files. Kubernetes provides tools to merge these configurations into a single file.

Prerequisites

Access to the Kubernetes or OpenShift clusters you want to manage
Existing kubeconfig files for each cluster
kubectl or oc command-line tools installed

Step-by-Step Merge Process

1. Backup Existing Configuration

Before merging, create a backup of your existing kubeconfig file to ensure you can revert to the original configuration if needed.

cp ~/.kube/config ~/.kube/config.backup

2. Set the KUBECONFIG Environment Variable

The KUBECONFIG environment variable allows you to specify multiple kubeconfig files. These files will be merged automatically by kubectl.

export KUBECONFIG=~/.kube/config:/path/to/second-kubeconfig

You can add as many kubeconfig files as needed, separating them with colons (on Linux/macOS) or semicolons (on Windows).

3. Merge the Files

Use the kubectl config view command with the --merge and --flatten options to merge the configurations into a single file.

kubectl config view --merge --flatten > ~/.kube/merged_kubeconfig

4. Replace the Original Configuration

Move the merged configuration to the default location.

mv ~/.kube/merged_kubeconfig ~/.kube/config

5. Verify the Merged Configuration

Ensure that the merged configuration works correctly by exploring and managing your contexts:

Listing Available Contexts

To view all available contexts in your merged configuration:

kubectl config get-contexts

This command displays a table with all contexts in your kubeconfig file, including: - The current active context (marked with an asterisk *) - Context names - Cluster names - Authentication users - Namespaces (if specified)

For a more detailed view of your entire kubeconfig:

kubectl config view

Checking the Current Context

To see which context is currently active:

kubectl config current-context

Switching Between Contexts

To switch to a different context:

kubectl config use-context <context-name>

OpenShift Virtualization Sidecar Implementation Guide

Sat, 26 Apr 2025 00:00:00

April 25, 2025

8 min read

Overview

The Sidecar feature in Kubevirt enables the attachment of additional containers to virtual machine pods. These sidecar containers run alongside the VM container, allowing for enhanced functionality such as monitoring, logging, debugging, and custom modifications to the VM environment without modifying the core VM itself.

This document provides instructions for enabling and implementing the Sidecar feature in an OpenShift Container Native Virtualization (CNV) environment.

Enabling the Sidecar Feature Gate

The Sidecar feature is controlled by a feature gate which must be explicitly enabled:

kubectl annotate --overwrite -n openshift-cnv hco kubevirt-hyperconverged \
  kubevirt.kubevirt.io/jsonpatch='[{"op": "add", "path": "/spec/configuration/developerConfiguration/featureGates/-", "value": "Sidecar"}]'

Verify Feature Gate is Enabled

To confirm the feature gate has been successfully added:

kubectl get kubevirt kubevirt-kubevirt-hyperconverged -n openshift-cnv \
  -o jsonpath='{.spec.configuration.developerConfiguration.featureGates}'

The output should include Sidecar in the list of enabled feature gates.

Implementing a Sidecar Container

KubeVirt Hook Types

KubeVirt sidecars support different hook types for various modification purposes:

onDefineDomain: Modifies the libvirt XML domain definition before VM creation
preCloudInitIso: Modifies cloud-init data before the ISO is generated

Sidecar Script Arguments

When writing sidecar scripts, it’s important to understand how arguments are passed to your script:

For onDefineDomain hook:

Argument 1: The hook name (onDefineDomain)
Argument 2: The --version parameter (e.g., v1alpha2)
Argument 3: The --vmi parameter with VMI information as JSON string
Argument 4: The --domain parameter with the current domain XML

For preCloudInitIso hook:

Argument 1: The hook name (preCloudInitIso)
Argument 2: The --version parameter (e.g., v1alpha2)
Argument 3: The --vmi parameter with VMI information as JSON string
Argument 4: The --cloud-init parameter with CloudInitData as JSON

Note

Your script should read the input XML/JSON from the appropriate argument (usually the 4th argument).
It must output the modified content to standard output (stdout).
The script must preserve any XML/JSON structure while making targeted modifications.
Any errors should be directed to standard error (stderr).

Example Hook ConfigMap

The following ConfigMap provides a script that modifies VM configurations by adding custom metadata to the VM’s libvirt XML definition using the onDefineDomain hook:

apiVersion: v1
kind: ConfigMap
metadata:
  name: xmleditscript
data:
  script.sh: |
    #!/bin/sh
    tempFile=`mktemp --dry-run`
    echo $4 > $tempFile
    sed -i "s|<baseBoard></baseBoard>|<baseBoard><entry name='manufacturer'>Radical Edward</entry></baseBoard>|" $tempFile
    cat $tempFile

Note

The libvirt domain XML is received as the 4th argument ($4), and the script’s standard output is used as the new domain definition to apply. This allows for dynamic modifications to the VM configuration.

Using Sidecars with VMs

To add a hook sidecar to a VM, modify the VM manifest to include the required annotations that specify the hook configuration. The ConfigMap containing your script must be referenced in the annotations:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
spec:
  template:
    metadata:
      annotations:
        hooks.kubevirt.io/hookSidecars: '[{"args": ["--version", "v1alpha2"],
          "configMap": {"name": "xmleditscript", "key": "script.sh", "hookPath": "/usr/bin/onDefineDomain"}}]'
    spec:
      domain:
        # VM configuration...
      volumes:
      - name: config-volume
        configMap:
          name: xmleditscript

Validating the Sidecar Modifications

After applying the sidecar configuration, you can verify that the changes have been successfully applied using several methods:

Method 1: Check VM XML from virt-launcher Pod

Get the virt-launcher pod for your VM:

oc get pods -n <namespace> | grep virt-launcher-<vm-name>

Examine the libvirt XML directly from the virt-launcher pod:

oc exec virt-launcher-<vm-name>-<random-id> -n <namespace> -- virsh dumpxml 1 | grep -A3 manufacturer

The output should include the modified XML with the custom manufacturer entry:

<entry name='manufacturer'>Radical Edward</entry>

Method 2: From Inside the VM

If your modification affects data visible to the guest OS (like SMBIOS data), you can also verify from inside the VM using tools like dmidecode:

Connect to the VM console or SSH into the VM
Run dmidecode to check the baseboard manufacturer:

sudo dmidecode -s baseboard-manufacturer

The output should show:

Radical Edward

References

OpenShift Console Banner

Sat, 26 Apr 2025 00:00:00

Apr 25, 2025

2 min read

Adding visual indicators to your OpenShift clusters can help users quickly identify which environment they’re working in, potentially preventing accidental changes in production environments.

This article demonstrates how to set a console banner in OpenShift to display important information like the cluster name or environment type.

Banner Customization

You can customize the following aspects of the banner:

backgroundColor: The background color of the banner (hex color code)
color: The text color (hex color code)
location: Where the banner appears. Options include: - BannerTop: Displays at the very top of the console (most common) - BannerBottom: Displays at the bottom of the console - BannerTopBottom: Displays at both top and bottom of the console - AlertBanner: Displays as a more prominent alert banner
text: The message to display to users

Example Colors

Production: Red (#880808)
Development: Blue (#0066CC)
Testing: Green (#2E8B57)
Staging: Orange (#FF8C00)

Common Use Cases

Identifying different clusters (prod, dev, test)
Warning users about scheduled maintenance
Highlighting important information about the environment

Note

Using distinct colors for different environments helps users visually identify which cluster they’re working with, reducing the risk of mistakes.

Applying the Configuration

Apply the configuration using the oc command:

oc apply -f banner.yaml

To remove the banner:

oc delete consolennotification banner

Simple Block Device Backup for OpenShift Lab Environments

Fri, 25 Apr 2025 00:00:00

April 25, 2025

8 min read

Introduction

Like the Borg collective from Star Trek assimilates technology, BorgBackup assimilates your data.

This article shows how to back up and restore LVM Logical Volumes from local storage to FreeNAS using BorgBackup. While not for enterprise production workloads, this method works well for lab environments.

Overview & Prerequisites

This backup solution backs up LVM Logical Volumes from Single Node OpenShift local storage to FreeNAS servers using:

BorgBackup: For efficient, deduplicated, encrypted backups
Container Image: Minimal image with BorgBackup and required tools
Kubernetes CronJobs: For scheduled backups of local LVM volumes
Kubernetes Pods: For restores to FreeNAS storage
Volume/Device Mounts: Direct access to LVM block devices on the SNO node

Note

All resources in this article are available in the GitHub repository.

Step 1: Prepare Your Environment

Container Image

Create a container image with BorgBackup:

Container image definition

FROM fedora:latest

RUN dnf install -y borgbackup

Build and push the image:

cd borg-container
podman build -t registry.example.com/borgbackup:latest .
podman push registry.example.com/borgbackup:latest

Resource Planning

Allocate resources for backup and restore pods:

Memory: 4Gi limit (1Gi request)
CPU: 2 cores limit (500m request)

These values balance performance with efficient resource usage.

Storage Architecture

The backup system uses two storage mechanisms:

Source LVM Storage: Direct access to LVM Logical Volumes on the SNO node
FreeNAS Repository Storage: iSCSI-based persistent volumes for BorgBackup repositories

This approach enables efficient backup of LVM volumes to FreeNAS while maintaining block-level operations.

Step 2: Configure and Run Backups

Create a CronJob for regular backups:

Example backup cronjob configuration

apiVersion: batch/v1
kind: CronJob
metadata:
  name: windows-backup-cronjob
spec:
  schedule: "0 0 31 2 *"  # A non-recurring time (e.g., Feb 31st)
  suspend: true  # Prevents the job from running automatically
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup-container
            image: registry.desku.be/epheo/borgbackup:latest
            env:
            - name: BORG_CONFIG_DIR
              value: /tmp/borg_config
            - name: BORG_CACHE_DIR
              value: /tmp/borg_cache
            - name: BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK
              value: "yes"
            command: ["/bin/bash", "-c"]
            args:
              - |
                echo "Starting Windows Backup pod with two mounted PVCs"
                mkdir -p $BORG_CONFIG_DIR $BORG_CACHE_DIR
                borg init --encryption=none /backup/
                borg create --read-special /backup::"backup-$(date +%Y-%m-%d_%H-%M-%S)" /dev/windows-block
            volumeDevices:
            - devicePath: /dev/windows-block
              name: windows-block
            volumeMounts:
            - mountPath: /backup
              name: windows-backup
            resources:
              requests:
                memory: "1Gi"
                cpu: "500m"
              limits:
                memory: "4Gi"
                cpu: "2"
          restartPolicy: OnFailure
          volumes:
          - name: windows-block
            persistentVolumeClaim:
              claimName: windows
          - name: windows-backup
            persistentVolumeClaim:
              claimName: windows-backup

Apply the configuration:

kubectl apply -f backup-cronjob.yaml

Run an immediate backup:

kubectl create job --from=cronjob/<backup-cronjob-name> manual-backup

Important

Ensure source block devices are accessible to the backup pod.

Step 3: Monitor and Manage Your Backups

Checking Backup Status

Monitor backup job status:

kubectl get jobs

Example output:

NAME                   COMPLETIONS   DURATION   AGE
block-backup-manual    1/1           2m15s      10m
vm-backup-1682456400   1/1           1m32s      2h

Examining Backup Logs

View logs to confirm success or troubleshoot:

kubectl logs <pod-name>

Tip

Use kubectl logs -f <pod-name> to follow logs in real-time.

Managing Backup Archives

List available backups:

# Create a temporary pod with the backup volume
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: borg-inspector
spec:
  containers:
  - name: borg-inspector
    image: registry.desku.be/epheo/borgbackup:latest
    command: ["sleep", "3600"]
    env:
    - name: BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK
      value: "yes"
    volumeMounts:
    - mountPath: /backup
      name: backup-volume
  volumes:
  - name: backup-volume
    persistentVolumeClaim:
      claimName: windowsdata-backup
EOF

# Wait for the pod to start
kubectl wait --for=condition=Ready pod/borg-inspector

# List backups
kubectl exec -it borg-inspector -- borg list /backup

# Break a lock if needed:
kubectl exec -it borg-inspector -- borg break-lock /backup

# Additional commands:
# Check repository
kubectl exec -it borg-inspector -- borg check /backup

# Show repository info
kubectl exec -it borg-inspector -- borg info /backup

# List archive contents
kubectl exec -it borg-inspector -- borg list /backup::archivename

# Extract files
kubectl exec -it borg-inspector -- borg extract /backup::archivename path/to/file

# Prune old archives
kubectl exec -it borg-inspector -- borg prune -v --list --keep-within=30d /backup

# Remove the temporary pod
kubectl delete pod borg-inspector

Step 4: Restore to a Local Volume

Restore a backup to a local volume:

Example restore pod configuration

apiVersion: v1
kind: Pod
metadata:
  name: windows-restore
  namespace: epheo
spec:
  containers:
  - name: restore-container
    image: registry.desku.be/epheo/borgbackup:latest
    env:
    - name: BORG_CONFIG_DIR
      value: /tmp/borg_config
    - name: BORG_CACHE_DIR
      value: /tmp/borg_cache
    - name: BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK
      value: "yes"
    command: ["/bin/bash", "-c"]
    args:
      - |
        echo "Starting Windows Backup pod with two mounted PVCs"
        mkdir -p $BORG_CONFIG_DIR $BORG_CACHE_DIR
        borg extract --progress --stdout /backup::$(borg list /backup --last 1 --format "{archive}") |dd of=/dev/windows-block
    volumeDevices:
    - devicePath: /dev/windows-block
      name: windows-block
    volumeMounts:
    - mountPath: /backup
      name: windows-backup
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
      limits:
        memory: "4Gi"
        cpu: "2"
  volumes:
  - name: windows-block
    persistentVolumeClaim:
      claimName: windows
  - name: windows-backup
    persistentVolumeClaim:
      claimName: windows-backup
  restartPolicy: OnFailure

Apply the configuration:

kubectl apply -f restore-pod.yaml

Important

Ensure target block devices are not in use during restore.

Step 5: Cross-Platform Restoration Process

This section explains how to restore backups from FreeNAS to LVM Logical Volumes on Single Node OpenShift.

Prerequisites

Before starting:

Install Democratic-CSI backend on your SNO cluster
Ensure access to FreeNAS storage with the backup repositories
Verify permissions to create storage resources in the target namespace

The process involves three steps:

Step 5.1: Create a Reference to the Existing Backup Volume

Create a PersistentVolume referencing your FreeNAS volume:

existing-backup-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: existing-backup-pv
spec:
  capacity:
    storage: 500Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: freenas-iscsi-csi
  csi:
    driver: org.democratic-csi.iscsi
    volumeHandle: existing-backup-volume-id  # ID of your existing volume on FreeNAS
    volumeAttributes:
      fs_type: ext4

Apply it:

kubectl apply -f existing-backup-pv.yaml

Step 5.2: Create a Claim for the Backup Volume

Create a PersistentVolumeClaim:

existing-backup-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: existing-backup-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Gi
  storageClassName: freenas-iscsi-csi
  volumeName: existing-backup-pv

Apply it:

kubectl apply -f existing-backup-pvc.yaml

Step 5.3: Create a Restore Pod

Create a restoration pod:

restore-from-backup-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: restore-from-existing-backup
spec:
  containers:
  - name: restore-container
    image: registry.desku.be/epheo/borgbackup:latest
    env:
    - name: BORG_CONFIG_DIR
      value: /tmp/borg_config
    - name: BORG_CACHE_DIR
      value: /tmp/borg_cache
    - name: BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK
      value: "yes"
    command: ["/bin/bash", "-c"]
    args:
      - |
        mkdir -p $BORG_CONFIG_DIR $BORG_CACHE_DIR
        borg extract --progress --stdout /existing-backup::$(borg list /existing-backup --last 1 --format "{archive}") | dd of=/dev/target-block bs=4M status=progress
    volumeDevices:
    - devicePath: /dev/target-block
      name: target-block-device
    volumeMounts:
    - mountPath: /existing-backup
      name: existing-backup-volume
  volumes:
  - name: existing-backup-volume
    persistentVolumeClaim:
      claimName: existing-backup-pvc
  - name: target-block-device
    persistentVolumeClaim:
      claimName: target-device-pvc

Apply it:

kubectl apply -f restore-from-backup-pod.yaml

Step 6: Verify the Restoration

Verify restored data:

Verification commands

 # Create verification directory
 kubectl exec -it restore-from-existing-backup -- mkdir -p /mnt/verify

 # Mount the restored device
 kubectl exec -it restore-from-existing-backup -- mount /dev/target-block /mnt/verify

 # Check contents
 kubectl exec -it restore-from-existing-backup -- ls -la /mnt/verify

Tip

For block devices with partition tables, use fdisk -l /dev/target-block before mounting.

Step 7: Directly Using Existing Block Devices with Kubernetes

Recent OpenShift versions can reinstall without wiping disks, preserving LVM Logical Volumes for recovery after reinstallation.

Identifying Available Block Devices

Identify block devices:

# From a debug pod
oc debug node/<node-name>
chroot /host

# List devices
lsblk

# Get device info
blkid /dev/sdX

Direct PersistentVolume Approach

Create a PersistentVolume referencing the block device:

local-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: existing-block-pv
spec:
  capacity:
    storage: 50Gi
  volumeMode: Block  # Use Block for raw device or Filesystem for formatted
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /dev/sdX  # Replace with your device path
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - sno-node-01  # Replace with your node name

Apply it:

kubectl apply -f local-pv.yaml

Creating a PersistentVolumeClaim

Create a PersistentVolumeClaim:

local-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: existing-block-pvc
  namespace: my-namespace
spec:
  volumeMode: Block  # Must match the PV's volumeMode
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  volumeName: existing-block-pv  # Reference the PV by name
  storageClassName: local-storage

Apply it:

kubectl apply -f local-pvc.yaml

Using the Block Device in a Pod

Use the block device in a pod:

block-device-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: block-device-pod
  namespace: my-namespace
spec:
  containers:
  - name: block-user
    image: registry.access.redhat.com/ubi8/ubi-minimal:latest
    command: ["sleep", "infinity"]
    volumeDevices:  # Use volumeDevices for Block mode
    - name: block-vol
      devicePath: /dev/xvda  # Device path in container
  volumes:
  - name: block-vol
    persistentVolumeClaim:
      claimName: existing-block-pvc

Apply it:

kubectl apply -f block-device-pod.yaml

Verification

Verify access to the block device:

# Check pod status
kubectl get pod -n my-namespace block-device-pod

# Verify block device
kubectl exec -it -n my-namespace block-device-pod -- lsblk

# Format and mount if needed
kubectl exec -it -n my-namespace block-device-pod -- mkfs.ext4 /dev/xvda
kubectl exec -it -n my-namespace block-device-pod -- mkdir -p /mnt/data
kubectl exec -it -n my-namespace block-device-pod -- mount /dev/xvda /mnt/data

Tip

Skip formatting for existing formatted devices.

This approach gives direct access to block devices while managing them through Kubernetes.

Resources and References

Additional resources:

OpenStack OVN Networking - Deep Dive and Debugging

Wed, 23 Apr 2025 00:00:00

Aug 22, 2025

12 min read

OVN (Open Virtual Network) provides advanced networking capabilities for OpenStack environments. This article explores how to debug, analyze and understand OVN’s networking components within an OpenStack deployment, helping you troubleshoot and gain deeper insights into the networking layer.

Locating OVN Services

OVN in OpenStack typically runs as a clustered service. First, we need to identify where these services are running in our environment.

# pcs status
Container bundle set: ovn-dbs-bundle [undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ovn-northd:pcmklatest]
   ovn-dbs-bundle-0     (ocf::ovn:ovndb-servers):       Master controller-3
   ovn-dbs-bundle-1     (ocf::ovn:ovndb-servers):       Slave controller-0
   ovn-dbs-bundle-2     (ocf::ovn:ovndb-servers):       Slave controller-2
 ip-172.17.1.200 (ocf::heartbeat:IPaddr2):       Started controller-1

We can also inspect the detailed properties of the OVN database service:

# pcs resource show ovn-dbs-bundle
Resource: ovndb_servers (class=ocf provider=ovn type=ovndb-servers)
 Attributes: inactive_probe_interval=180000 manage_northd=yes master_ip=172.17.1.179 nb_master_port=6641 sb_master_port=6642
 Meta Attrs: container-attribute-target=host notify=true
 Operations: demote interval=0s timeout=50s (ovndb_servers-demote-interval-0s)
             monitor interval=10s role=Master timeout=60s (ovndb_servers-monitor-interval-10s)
             monitor interval=30s role=Slave timeout=60s (ovndb_servers-monitor-interval-30s)
             notify interval=0s timeout=20s (ovndb_servers-notify-interval-0s)
             promote interval=0s timeout=50s (ovndb_servers-promote-interval-0s)
             start interval=0s timeout=200s (ovndb_servers-start-interval-0s)
             stop interval=0s timeout=200s (ovndb_servers-stop-interval-0s)

Executing OVN Commands in the Cluster

Before executing OVN commands, we need to identify which node is running the OVN databases as the master:

$ ansible -b -i /usr/bin/tripleo-ansible-inventory -m shell -a "pcs status | grep ovn-dbs-bundle | grep Master" controller-2
controller-3 | CHANGED | rc=0 >>
   ovn-dbs-bundle-0     (ocf::ovn:ovndb-servers):       Master controller-0

Now that we’ve identified the master node, we can execute read/write operations directly against the OVN Northbound (NB) and Southbound (SB) databases from the corresponding container:

# On the master controller node (controller-2)
[root@controller-0 ~]# podman exec -it ovn-dbs-bundle-podman-0 bash

# Creating a test logical switch
()[root@controller-1 /]# ovn-nbctl ls-add ovn_test_lswitch

# Verifying the switch was created
()[root@controller-3 /]# ovn-nbctl ls-list | grep ovn_test_lswitch
9905fd23-1ed2-4f57-b220-77b008b2116e (ovn_test_lswitch)

# Deleting the test switch
()[root@controller-1 /]# ovn-nbctl ls-del ovn_test_lswitch

# Confirming the switch was deleted
()[root@controller-2 /]# ovn-nbctl ls-list | grep -c ovn_test_lswitch
0

Remote OVN Database Access

We can also run commands from non-master nodes, but we need the Virtual IP (VIP) of the OVN database service:

[root@controller-2 ~]# ovs-vsctl get Open_Vswitch . external_ids:ovn-remote | cut -d':' -f 2
172.17.1.213

Using the IP address obtained above, we can execute commands against the OVN databases from any node:

# Creating a test logical switch in the NB database
()[root@controller-1 /]# ovn-nbctl --db="tcp:172.17.1.59:6641" ls-add ovn_test_lswitch

# Removing the test logical switch
()[root@controller-3 /]# ovn-nbctl --db="tcp:172.17.1.156:6641" ls-del ovn_test_lswitch

# Creating a test chassis in the SB database
()[root@controller-0 /]# ovn-sbctl --db="tcp:172.17.1.49:6642" chassis-add ovn_test_chassis geneve 127.0.0.1

# Removing the test chassis
()[root@controller-1 /]# ovn-sbctl --db="tcp:172.17.1.170:6642" chassis-del ovn_test_chassis

Note: By default, TCP port 6641 is used for the OVN Northbound database and 6642 for the OVN Southbound database.

Understanding OVN Database Structure

OVN uses two databases to manage the network: the Northbound (NB) and Southbound (SB) databases. Let’s explore their contents to understand how the logical and physical elements relate.

Northbound Database - Logical Network Elements

The Northbound database defines all the logical networking elements configured through Neutron. By examining this database, we can see the logical routers, ports, and NAT rules:

()[root@controller-1 /]# ovn-nbctl show
[...]
router a6b22fb8-6d75-4849-9650-5e255d023591 (neutron-a929f25e-e00f-4857-9e35-e0db72c396f2) (aka router1)
    port lrp-36ba33f2-31fc-4f41-a86d-7a8dc6a6bcb6  <- private1 subnet interface
        mac: "fa:16:3e:09:9a:bf"
        networks: ["192.168.30.1/24"]
    port lrp-5c3d686a-1918-4663-88c3-cdd9faa1d3b2  <- public subnet interface
        mac: "fa:16:3e:40:f8:46"
        networks: ["10.0.0.36/24"]
        gateway chassis: [21347b99-e853-4aa8-b7da-82aee8aa972a 50cf1414-5c00-4aa9-a1d7-8a45de1a72ae 956d66be-6c1a-437f-88ae-247045816147]
    port lrp-b139f4e9-01b9-4642-8835-5790ba8b1142  <- private2 subnet interface
        mac: "fa:16:3e:b4:08:73"
        networks: ["192.168.40.1/24"]
    nat 6401e181-0479-4a70-bd9e-06b8ecd83d21       <- Floating IP
        external ip: "10.0.0.130"
        logical ip: "192.168.40.66"
        type: "dnat_and_snat"
    nat 24adb45d-d48a-4c33-84e2-f9642d1a66ae       <- SNAT rule for private1 subnet
        external ip: "10.0.0.182"
        logical ip: "192.168.30.0/24"
        type: "snat"
    nat f1d6bb14-0f2e-4682-85a8-7b2a92b994bf       <- Floating IP
        external ip: "10.0.0.6"
        logical ip: "192.168.40.25"
        type: "dnat_and_snat"
    nat 182b12fd-e2aa-46c4-adcb-71e6d8276c04       <- Floating IP
        external ip: "10.0.0.146"
        logical ip: "192.168.30.108"
        type: "dnat_and_snat"
    nat 22a78989-5a46-4590-9532-9f31fa783b5a      <- SNAT rule for private2 subnet
        external ip: "10.0.0.194"
        logical ip: "192.168.40.0/24"
        type: "snat"

Southbound Database - Physical Mapping

The Southbound database maps the logical elements to physical locations in the infrastructure. By examining this database, we can see where instances and network elements are actually running:

()[root@controller-3 /]# ovn-sbctl show

Chassis "0140f271-2811-47f5-8174-5ca281aaeb41"
    hostname: "compute-2.redhat.local"
    Encap geneve
        ip: "172.17.2.86"
        options: {csum="true"}
    Port_Binding "a2ff1c60-c074-47c4-9c1e-c768ade269cb"  <- private2_vm1 in compute-1
    Port_Binding "b631ec31-d634-430c-b50e-1b7819ce7dbd"  <- private1_vm1 in compute-0
Chassis "554f96ab-2f35-40c6-a5eb-764067db3188"
    hostname: "compute-0.redhat.local"
    Encap geneve
        ip: "172.17.2.36"
        options: {csum="true"}
    Port_Binding "2d6249e2-d05c-48fd-81c6-7831cfe107b3"  <- private1_vm2 in compute-3
    Port_Binding "4b0b7e46-9217-4267-a090-418873a8b6f0"  <- private2_vm2 in compute-3
Chassis "21347b99-e853-4aa8-b7da-82aee8aa972a"
    hostname: "controller-2.redhat.local"
    Encap geneve
        ip: "172.17.2.83"
        options: {csum="true"}
    Port_Binding "cr-lrp-5c3d686a-1918-4663-88c3-cdd9faa1d3b2"      <- Gateway port in controller-2 (SNAT traffic)

[...]

Tracing Security Group Rules Through OVN

One of the most powerful aspects of OVN is being able to trace how a Neutron Security Group Rule (SGR) is translated through the networking stack. We can follow the path from:

Neutron Security Group Rule
OVN Access Control List (ACL)
Logical Flow in the Southbound DB
OpenFlow rules on compute nodes

Let’s start by examining a security group rule in Neutron:

(overcloud) [stack@undercloud-0 ~]$ openstack security group rule list
+--------------------------------------+-------------+-----------+-------------------+------------+--------------------------------------+--------------------------------------+
| ID                                   | IP Protocol | Ethertype | IP Range          | Port Range | Remote Security Group                | Security Group                       |
+--------------------------------------+-------------+-----------+-------------------+------------+--------------------------------------+--------------------------------------+
| 0a6cfa3d-9291-4331-8e52-82b7e3800200 | tcp         | IPv4      | 192.168.13.116/30 | 9999:9999  | None                                 | 3e484201-4fba-4d47-b18c-a515b6a1b8f3 |
+--------------------------------------+-------------+-----------+-------------------+------------+--------------------------------------+--------------------------------------+

Next, we can find the corresponding ACL entry in the OVN Northbound database by searching for the security group rule ID:

()[root@controller-2 /]# ovn-nbctl find ACL external_ids:"neutron\:security_group_rule_id"=0a6cfa3d-9291-4331-8e52-82b7e3800200
_uuid               : fc662a6a-1228-4c04-b1bf-3cb6f6e61c91
action              : allow-related
direction           : to-lport
external_ids        : {"neutron:security_group_rule_id"="0a6cfa3d-9291-4331-8e52-82b7e3800200"}
label               : 0
log                 : false
match               : "outport == @pg_3e484201_4fba_4d47_b18c_a515b6a1b8f3 && ip4 && ip4.src == 192.168.13.20/30 && tcp && tcp.dst == 9999"
priority            : 1002

Then, we can search for the corresponding logical flows in the Southbound database using the ACL’s UUID prefix:

()[root@controller-2 /]# ovn-sbctl find Logical_Flow external_ids:stage-hint=59ec7790
_uuid               : 0e465ba7-2a0d-45c8-9401-4446d8776c71
actions             : "next;"
external_ids        : {source="northd.c:6108", stage-hint="59ec7790", stage-name=ls_out_acl}
logical_datapath    : 334f6950-34fc-474d-8897-84a2d13846a0
match               : "reg0[8] == 1 && (outport == @pg_3e484201_4fba_4d47_b18c_a515b6a1b8f3 && ip4 && ip4.src == 192.168.13.186/30 && tcp && tcp.dst == 9999)"
pipeline            : egress
priority            : 2002
table_id            : 4
hash                : 0

_uuid               : 48213015-8c4e-4383-a564-83d5ca516695
actions             : "reg0[1] = 1; next;"
external_ids        : {source="northd.c:6084", stage-hint="59ec7790", stage-name=ls_out_acl}
logical_datapath    : 334f6950-34fc-474d-8897-84a2d13846a0
match               : "reg0[7] == 1 && (outport == @pg_3e484201_4fba_4d47_b18c_a515b6a1b8f3 && ip4 && ip4.src == 192.168.13.17/30 && tcp && tcp.dst == 9999)"
pipeline            : egress
priority            : 2002
table_id            : 4
hash                : 0

Finally, we can verify the actual OpenFlow rules installed on the compute nodes that implement these logical flows:

[root@compute-0 ~]# for i in 1c09a74f 4663ae1f; do ovs-ofctl dump-flows br-int | grep $i; done
 cookie=0x58a831c2, duration=134139.172s, table=44, n_packets=0, n_bytes=0, idle_age=18910, hard_age=4013, priority=2002,tcp,reg0=0x100/0x100,reg15=0x3,metadata=0x8,nw_src=192.168.13.144/30,tp_dst=9999 actions=resubmit(,45)
 cookie=0xf2e670b7, duration=128550.798s, table=44, n_packets=0, n_bytes=0, idle_age=9762, hard_age=8793, priority=2002,tcp,reg0=0x80/0x80,reg15=0x3,metadata=0x8,nw_src=192.168.13.31/30,tp_dst=9999 actions=load:0x1->NXM_NX_XXREG0[97],resubmit(,45)

# We can also check specific flows for specific ports
[root@compute-0 ~]# ovs-ofctl dump-flows br-int table=65 | grep 'reg15=0x3,metadata=0x8'

[root@compute-2 ~]# ovs-ofctl show br-int

The values reg15=0x3,metadata=0x8 identify a particular VM - in this case, csl-log-redhat-0 (192.168.13.92) on compute-0.

Packet Tracing with ovn-trace

One of the most powerful debugging tools in OVN is ovn-trace, which allows us to simulate packet flows through the logical network. This tool helps identify issues in packet processing before they reach the physical network.

How ovn-trace Works:

Reads the Logical_Flow and other tables from the OVN Southbound database
Simulates a packet’s path through logical networks by following the entire tree of possibilities
Shows how logical flows would process specific types of packets
Only simulates the OVN logical network (not the physical elements)
When used with the --ovs option, it will also show the OpenFlow rules installed in the bridge

This tool is invaluable for debugging connectivity issues, as it helps isolate whether the problem is in the logical configuration or the physical implementation.

Example: Tracing ICMP Traffic Between VMs

Let’s use ovn-trace to explore how an ICMP packet would travel between two guest instances:

()[root@compute-0 /]# ovn-trace --db="tcp:172.17.1.203:6642" --ovs --friendly-names --ct=new private2  'inport == "4b0b7e46-9217-4267-a090-418873a8b6f0" && eth.src == fa:16:3e:fe:90:9a && eth.dst == fa:16:3e:e3:e2:c8 && ip4.src == 192.168.40.158 && ip4.dst == 192.168.40.12 && ip.ttl == 64 && icmp4.type == 8'

#icmp,reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:ce:02:65,dl_dst=fa:16:3e:5b:01:ec,nw_src=192.168.30.229,nw_dst=192.168.30.48,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0

ingress(dp="private2", inport="private2_vm2")
---------------------------------------------
[...]
 4. ls_out_acl (ovn-northd.c:4549): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0 && (outport == @pg_1576d8a3_7db2_40e9_942f_90f7c37a355e && ip4 && ip4.src == 0.0.0.0/0 && icmp4), priority 2002, uuid eda37d0e
    cookie=0xeda37d0e, duration=677279.297s, table=44, n_packets=127, n_bytes=14905, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0/0x1,icmp,reg15=0x3,metadata=0x3 actions=resubmit(,45)
    next;

[...]
 9. ls_out_port_sec_l2 (ovn-northd.c:4081): outport == "private2_vm1" && eth.dst == {fa:16:3e:25:26:50}, priority 50, uuid 060cf739
    cookie=0x60cf739, duration=670224.080s, table=49, n_packets=4626, n_bytes=372616, priority=50,reg15=0x3,metadata=0x3,dl_dst=fa:16:3e:68:c6:ee actions=resubmit(,64)
    output;
    /* output to "private2_vm1", type "" */

In this trace example, we can see how an ICMP packet from private2_vm2 to private2_vm1 is processed through the logical pipeline. The trace shows several stages including ACL processing (step 4) and port security checks (step 9) before the packet is finally output to the destination VM.

Verifying Physical Network Flows

While ovn-trace is extremely helpful for debugging logical flows, sometimes we need to verify how packets are flowing through the physical network infrastructure:

Let’s explore an ICMP packet delivered to a VM on its compute node
We’ll ping between two guests and expect the packet to be delivered on the correct tap interface
By monitoring the flow’s n_packets field in table 65, we can confirm packet delivery

According to the ovn-architecture manual, table 65 performs the final translation between logical ports and physical ports, with the actual packet output happening in this table.

First, identify the OpenFlow port number for the VM’s tap interface:

[root@compute-2 /]# ovs-ofctl show br-int|grep tapf4ada4a0
 113(tap11d54329--3): addr:63:75:6e:e0:9f:a8

Then, monitor the OpenFlow rules to see if packets are being delivered:

[root@compute-0 ~]# watch -d -n1 "ovs-ofctl dump-flows br-int table=65 | grep 'output:113'"
cookie=0x0, duration=819057.906s, table=65, n_packets=4727, n_bytes=412655, idle_age=25642, hard_age=11420, priority=100,reg15=0x3,metadata=0x3 actions=output:113

When the n_packets counter increases, it confirms that packets are successfully reaching the destination VM’s interface.

Note: OpenFlow table numbers in OVN correspond to logical tables as follows: - Ingress pipeline tables: Logical table ID + 8 - Egress pipeline tables: Logical table ID + 40

For a detailed explanation of the OVN logical tables, refer to the ovn-northd manual page.

DHCP in OVN

Unlike in ML2/OVS, OVN serves DHCP locally in the compute nodes. The DHCP requests are sent to ovn-controller and replied according to the database contents. When a VM starts on a hypervisor, ovn-controller will install flows in table 20 with a controller action.

We can find the right DHCP_Options row in the NB database (Neutron inserts this every time a subnet is created):

()[root@controller-3 /]# ovn-nbctl find DHCP_Options external_ids:subnet_id=ce6193c4-80ef-448c-8153-4b8282eef0f3

_uuid               : c95ccc99-e50f-471c-978e-bf68dafef362
cidr                : "192.168.30.0/24"
external_ids        : {"neutron:revision_number"="0", subnet_id="ce6193c4-80ef-448c-8153-4b8282eef0f3"}
options             : {classless_static_route="{169.254.169.254/32,192.168.30.2, 0.0.0.0/0,192.168.30.1}", dns_server="{172.16.0.1, 10.0.0.1}", lease_time="43200", mtu="1442", router="192.168.30.1", server_id="192.168.30.1", server_mac="fa:16:3e:8c:0e:f5"}

The options include: * 169.254.169.254/32,192.168.30.2 -> Static route for the metadata service * 0.0.0.0/0,192.168.30.1 -> Default gateway route

We can also find the corresponding logical flow in the SB database:

()[root@controller-2 /]# ovn-sbctl find logical_flow external_ids:stage-name=ls_in_dhcp_options
_uuid               : 6b422d1e-0a41-4422-9390-e4cc67352e12
actions             : "reg0[3] = put_dhcp_opts(offerip = 192.168.30.182, classless_static_route = {169.254.169.254/32,192.168.30.2, 0.0.0.0/0,192.168.30.1}, dns_server = {172.16.0.1, 10.0.0.1}, lease_time = 43200, mtu = 1442, netmask = 255.255.255.0, router = 192.168.30.1, server_id = 192.168.30.1); next;"
external_ids        : {source="ovn-northd.c:5413", stage-name=ls_in_dhcp_options}
logical_datapath    : 3b7964f9-d315-4a93-85fb-c5d485ffeefd
match               : "inport == \"b631ec31-d634-430c-b50e-1b7819ce7dbd\" && eth.src == fa:16:3e:40:31:2c && ip4.src == 0.0.0.0 && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst == 67"
pipeline            : ingress
priority            : 100
table_id            : 12

In compute-1 we should be able to see the flow in table 20 (12+8):

()[root@compute-3 /]# ovs-ofctl dump-flows br-int |grep bbe8861b
 cookie=0xbbe8861b, duration=780275.467s, table=20, n_packets=0, n_bytes=0, idle_age=25685, hard_age=29126, priority=100,udp,reg14=0x3,metadata=0x2,dl_src=fa:16:3e:c8:4a:a8,nw_src=0.0.0.0,nw_dst=255.255.255.255,tp_src=68,tp_dst=67 actions=controller(userdata=00.00.00.20.00.00.00.240.00.01.de.10.00.00.98.63.c0.a8.1e.54.79.0e.20.a9.fe.a9.fe.c0.a8.1e.02.00.c0.a8.1e.01.06.08.ac.10.00.01.0a.00.00.01.83.04.00.00.a8.c0.1a.02.05.a2.01.04.ff.ff.ff.00.03.04.c0.a8.1e.01.36.04.c0.a8.1e.01,pause),resubmit(,21)

The controller action indicates that DHCP requests will be processed by the local ovn-controller process, which generates responses based on the configuration in the OVN databases.

DEFMS: Distributed, Encrypted and Free Mail System

Wed, 23 Apr 2025 00:00:00

Feb 24, 2018

2 min read

Let’s have a closer look at the current implementation of our global email delivery system.

It scaled out from the very early stage of our network while all mail addresses was summed up in a paper book to our current multi-billion addresses directory quite impressively, the workload and delivery effort is mostly shared across the few biggest current IT and telecom companies and the required bandwidth and data effort is duplicated between all sender, receivers and intermediates of the transmission. Content encryption or signature is still a marginal practice and while normally not accessible by a random tiers, the relevant intermediates and providers do not hesitate to monitor and analyze our day to day exchanges. Content is easily reversible, which so prevent (or at least should prevent) them to have any juridical value.

As of today, we used the available technologies and paradigms to scale from an inter-lab experiment to a multi-billion connection delivery network, but as new paradigms and technologies arrived, we now have all the tools to create a new email delivery network, compatible with the current protocols, that better suits our everyday needs.

Regarding how fast the importance of this media grew up those past two decades, one would now want to be able to rely on multiple sources for the storage and transport of his messages without having to necessarily trust those third-parties, one would also like to guaranty the integrity, the consistency and the immutability in time of his exchanges with someone. An entirely distributed, partly encrypted and free mail exchange system would allow us to communicate directly with the concerned receivers without restrictions of size, number, consistency or availability of the exchanged messages. Applications are numerous but more importantly this will help us reducing the total amount of storage and bandwith required by mail exchanges.

As encryption, peer to peer and data immutability are already widespread principles one remaining pain-point is in the compatibility with our existing systems and protocols. Such a compatibility with actual mail protocols will be achieved by non-ditributed gateway servers, this long-term temporary solution will encrypt and forward all email from standard MX systems to and from DEFMS.

Advanced File Permissions with ACL on Linux

Wed, 23 Apr 2025 00:00:00

Aug 06, 2010

10 min read

Traditional Unix file permissions have a significant limitation: they don’t allow us to grant different access levels to multiple users or groups simultaneously - such as read/write access for some groups while restricting others to read-only access.

This is where Access Control Lists (ACLs) come in. ACLs provide a more flexible permission system by allowing multiple access rules per file or directory. Although ACLs come pre-installed on many modern Linux distributions, knowing how to use them effectively remains essential, especially for scenarios like file sharing with Samba or managing multi-user environments.

Understanding the Difference: Traditional Permissions vs. ACLs

Before diving into ACLs, let’s compare them with traditional Unix permissions:

Feature	Traditional Unix Permissions	Access Control Lists (ACLs)
Permission Structure	Simple rwx for owner, group, others	Multiple permission entries for different users/groups
Permission Granularity	Limited to three entities	Unlimited entities with specific permissions
Inheritance Control	Limited (primarily through umask)	Explicit inheritance with default ACLs
Management Complexity	Simple to understand and manage	More complex but much more flexible
Command Tools	chmod, chown, chgrp	getfacl, setfacl
Windows Compatibility	Limited	Advanced support for Samba/CIFS sharing

By understanding these differences, you can make informed decisions about when to use traditional permissions versus ACLs for your file security needs.

Practical Use Case

Let’s consider a Samba file sharing setup in a small business with a directory called “ProjectDocs”. In this scenario:

The manager needs full control over ProjectDocs
The developers need read and execute permissions
The clients should only have read-only access

This is a perfect case where we need to manage the directory permissions with ACLs, as standard Unix permissions would not allow this level of granular control.

Implementing Our Use Case

Let’s implement the scenario described above with specific commands. Assuming you’ve already created a directory called “ProjectDocs”:

# Create the directory and test files
mkdir -p /data/ProjectDocs
touch /data/ProjectDocs/report.doc
touch /data/ProjectDocs/specs.pdf

# Set basic permissions first
sudo chown manager:managers /data/ProjectDocs
sudo chmod 750 /data/ProjectDocs

# Grant full access to the manager
sudo setfacl -m u:manager:rwx /data/ProjectDocs

# Set read and execute permissions for the developers group
sudo setfacl -m g:developers:rx /data/ProjectDocs

# Set read-only permissions for the clients group
sudo setfacl -m g:clients:r /data/ProjectDocs

# Set default ACLs so new files inherit these permissions
sudo setfacl -m d:u:manager:rwx /data/ProjectDocs
sudo setfacl -m d:g:developers:rx /data/ProjectDocs
sudo setfacl -m d:g:clients:r /data/ProjectDocs

# Apply ACLs recursively to existing files
sudo setfacl -R -m u:manager:rwx /data/ProjectDocs
sudo setfacl -R -m g:developers:rx /data/ProjectDocs
sudo setfacl -R -m g:clients:r /data/ProjectDocs

After executing these commands, the directory structure will have the exact permissions needed for our scenario: - Manager has full control (read, write, execute) - Developers can read and execute files, but not modify them - Clients can only read the files - All new files created in the directory will automatically inherit these permissions

You can verify the configuration with:

getfacl /data/ProjectDocs

Installation

On most modern Linux distributions, ACL support is pre-installed. If not, you can install it using your package manager:

For Red Hat-based systems:

sudo dnf install acl

For Debian-based systems:

sudo apt install acl

Enabling ACL Support

Most modern Linux filesystems (ext4, XFS, Btrfs) have ACL support enabled by default. However, if you need to explicitly enable it, you’ll need to modify the filesystem mount options.

First, identify the partition you want to modify. You can use the following command to list your mounted filesystems:

df -h

Let’s say we need to enable ACLs on /dev/nvme0n1p3 mounted at /data.

Edit the /etc/fstab file:

sudo nano /etc/fstab

Add the acl option to your partition’s mount options:

/dev/nvme0n1p3    /data    ext4    defaults,acl    0    2

To apply the changes without rebooting:

sudo mount -o remount,acl /data

Note: On most modern Linux systems with ext4, XFS, or Btrfs filesystems, ACL support is enabled by default, so this step may not be necessary.

Verifying ACL Support

To confirm that your filesystem supports ACLs, you can use one of these methods:

# Method 1: Check filesystem capabilities
sudo tune2fs -l /dev/nvme0n1p3 | grep "Default mount options"

# Method 2: Check if the current mount has ACL support
mount | grep "/data" | grep "acl"

# Method 3: Test if you can set an ACL (if this succeeds, ACLs are supported)
touch /data/test_acl
setfacl -m u:nobody:r /data/test_acl

Configuring ACLs for Files and Directories

There are two main commands for working with ACLs: getfacl and setfacl.

The getfacl command lists ACL permissions:

getfacl filename

It works similar to ls -l but provides more detailed information specific to ACLs. For example, to check permissions on a file named project-report.pdf:

getfacl project-report.pdf

The setfacl command configures ACL permissions for a file or directory. Its basic syntax is:

setfacl -[option] [specification] filename

The command can be broken down into these components:

Options: - -m to add or modify a rule - -b to remove all ACL entries - -x to remove specific ACL entries
Specification: [d:]type:name:permissions - d: (optional) - if present, applies to default ACLs (used for directories) - type - can be u (user), g (group), or o (other) - name - username or group name (not needed for “other”) - permissions - any combination of r (read), w (write), and x (execute)

Understanding ACL Masks

One of the most important concepts in Linux ACLs is the “mask”. The mask defines the maximum permissions that can be granted by any ACL entry for a file or directory. Think of it as an upper bound for permissions.

# Example ACL with mask
user::rw-        # Owner has read-write
user:alex:rwx    # User alex has read-write-execute, but...
group::r--       # Group owner has read-only
mask::r--        # The mask restricts to read-only!
other::---       # Others have no permissions

In this example, despite giving alex rwx permissions, the effective permissions will be r–, because the mask restricts the maximum permissions to read-only.

To explicitly set the mask value:

# Set the mask to allow read and execute
setfacl -m m::rx project-report.pdf

The mask is automatically recalculated when you add or modify ACL entries, unless you use the --no-mask option with setfacl.

Examples of ACL Usage

Give write permission to user “alex” on a file:

setfacl -m u:alex:w project-report.pdf

Give read and execute permissions to the “developers” group:

setfacl -m g:developers:rx project-report.pdf

Remove all ACL entries from a file:

setfacl -b project-report.pdf

Apply ACLs recursively to a directory and all its contents:

setfacl -R -m g:developers:rx ProjectDocs/

Set default ACLs on a directory (new files created in this directory will inherit these ACLs):

setfacl -m d:g:developers:rx ProjectDocs/

Give read permission to others (all users not specifically mentioned):

setfacl -m o::r project-report.pdf

View current ACL settings:

getfacl project-report.pdf

Remove a specific ACL entry (remove just alex’s permissions):

setfacl -x u:alex project-report.pdf

Copy ACLs from one file to another:

getfacl source_file.txt | setfacl --set-file=- destination_file.txt

The output might look like:

# file: project-report.pdf
# owner: manager
# group: admin
user::rw-
user:alex:rw-
group::r--
group:developers:r-x
mask::rwx
other::r--

Troubleshooting ACLs

When working with ACLs, you might encounter some common issues. Here are solutions for the most frequent problems:

ACL Commands Not Working

Verify your filesystem supports and has ACLs enabled:
```
tune2fs -l /dev/nvme0n1p3 | grep "Default mount options"
```
Check if the ACL package is installed:
```
which getfacl
```

Permission Denied Errors

Ensure you have sufficient permissions (usually requires root or ownership):

# Check your permissions on the file/directory
ls -la /path/to/file

# Use sudo if needed
sudo setfacl -m u:user:rw /path/to/file

ACLs Not Being Applied

Check if there’s a restrictive mask:

getfacl /path/to/file | grep mask

# Set a more permissive mask if needed
setfacl -m m::rwx /path/to/file

ACLs Lost After Copying Files

Use the correct copy command to preserve ACLs:

# Use cp with the -a (archive) option
cp -a source_file destination_file

# Or use rsync
rsync -av --acls source_file destination_file

Debugging Complex ACL Issues

For more complex issues, use verbose mode with ACL commands:
```
setfacl -v -m u:user:rw /path/to/file
```

Working with ACLs in Samba Shares

When using Samba to share files with Windows systems, you can manage ACLs through Windows File Explorer. This provides a familiar graphical interface for Windows users while maintaining the security controls you’ve established.

To enable this functionality in your Samba configuration, edit /etc/samba/smb.conf and ensure these options are set:

[global]
map acl inherit = yes
store dos attributes = yes

[ProjectDocs]
path = /data/ProjectDocs
read only = no
acl_xattr:ignore system acls = yes
inherit acls = yes

This configuration allows Windows clients to modify the ACLs through the familiar Windows security dialog, providing a seamless experience across platforms.

Conclusion

Access Control Lists provide a powerful way to manage complex permission requirements that go beyond the traditional Unix permission model. By understanding how to use getfacl and setfacl, you can create sophisticated permission structures that meet the needs of multi-user environments while maintaining proper security controls.

Useful Podman Commands

Wed, 23 Apr 2025 00:00:00

April 22, 2025

5 min read

Introduction

This article provides a collection of useful Podman commands and techniques for common container operations. Podman is a daemonless container engine that provides a Docker-compatible command line interface.

Squashing Container Images

Squashing container images reduces their size by combining multiple layers into one, making distribution more efficient.

podman build --layers --force-rm --squash-all --tag squashedimage - <<< "FROM regsitry/imagetosquash"

Container Run Commands

Here are several useful patterns for running containers with Podman:

Running a Gollum wiki server:

podman run --detach --name gollum --security-opt label=disable --userns=keep-id \
  -v /srv/wiki:/wiki -p 4567:4567 gollumwiki/gollum:master --default-keybind vim

Running a container with host networking:

podman run -d --network host -v /var/log/sshoney/:/root/logs/ --name sshoney 97f3876877e2

Running a MariaDB container in a pod:

podman run --detach --pod $POD_NAME \
  -e MYSQL_ROOT_PASSWORD=$DB_PASS \
  -e MYSQL_PASSWORD=$DB_PASS \
  -e MYSQL_DATABASE=$DB_NAME \
  -e MYSQL_USER=$DB_USER \
  --name $CONTAINER_NAME_DB \
  -v "$PWD/database":/var/lib/mysql docker.io/mariadb:latest

Running a WordPress container in a pod:

podman run --detach --pod $POD_NAME \
  -e WORDPRESS_DB_HOST=127.0.0.1:3306 \
  -e WORDPRESS_DB_NAME=$DB_NAME \
  -e WORDPRESS_DB_USER=$DB_USER \
  -e WORDPRESS_DB_PASSWORD=$DB_PASS \
  --name $CONTAINER_NAME_WP \
  -v "$PWD/html":/var/www/html docker.io/wordpress

Image Transfer Between Systems

When direct registry access is unavailable, you can save and load images for transfer:

podman image save 97f3876877e2 -o 97f3876877e2.tgz
podman load --input 97f3876877e2.tgz

Systemd Integration

Running containers as systemd services allows them to persist even when the user is logged out:

# Enable executing processes with logged out shell
loginctl enable-linger user

# Move to user systemd directory
cd ~/.config/systemd/user/

# Generate systemd files for container
podman generate systemd --restart-policy=always -t 1 --name container_name --files

# Enable and start the service
systemctl --user enable container-container_name.service

Layer 2 User Defined Networks for OpenShift Virtualization

Tue, 22 Apr 2025 00:00:00

April 22, 2025

20 min read

Learn how to set up Layer 2 User Defined Networks (UDN) for VMs in OpenShift Virtualization for isolated tenant networks with multi-node connectivity.

This guide shows how to implement UDNs in OpenShift Virtualization for network isolation between VMs within or across namespaces. UDNs create isolated tenant networks separate from the default pod network, enabling complex network layouts and better security.

See also

Learn more about OpenShift Virtualization networking in the official OpenShift Virtualization Documentation.

Note

This is compatible with OpenShift 4.18 and newer versions.

Implementation Steps

1. Understand User Defined Networks

OpenShift Virtualization offers two UDN types:

UserDefinedNetwork (UDN): Creates network isolation within a single namespace.
ClusterUserDefinedNetwork (Cluster UDN): Spans multiple namespaces, allowing VMs in different namespaces to communicate while staying isolated from other networks.

Layer 2 vs Layer 3 User Defined Networks

UDNs can use either Layer 2 or Layer 3 topology:

Layer 2 UDN

Creates a virtual switch across all cluster nodes
All VMs connect to the same subnet regardless of node location
Provides a single broadcast domain
Supports VM live migration between nodes
Best for apps needing Layer 2 adjacency
Simpler to configure

Layer 3 UDN

Creates unique Layer 2 segments per node with Layer 3 routing between them
Each node has its own subnet
Controls broadcast traffic within node-specific segments
Better scalability for large deployments
Supports NetworkPolicy for security control
Better for environments with many VMs

Primary vs Secondary Networks

Primary networks handle default traffic
Secondary networks add interfaces for specific traffic
A VM can have one primary but multiple secondary networks
UDNs work as either primary or secondary networks

Important

Compatibility Note:

Only Layer 2 UDNs work as primary networks for VMs
Layer 3 UDNs only work with regular pods, not VMs

This guide focuses on Layer 2 UDNs for VM workloads that need simplicity and migration support.

2. Create a Namespace for User Defined Network

Create a namespace with the UDN label:

udn-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: udn-example
  labels:
    k8s.ovn.org/primary-user-defined-network: ""

Important

Add this label when creating the namespace. You cannot add it to existing namespaces.

Apply the namespace:

oc create -f udn-namespace.yaml

3. Create a User Defined Network

Create a UDN in your namespace:

udn-example.yaml

apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  name: udn-example
  namespace: udn-example
spec:
  layer2:
    ipam:
      lifecycle: Persistent
    role: Primary
    subnets:
    - 10.200.0.0/16
  topology: Layer2

Apply the UDN:

oc create -f udn-example.yaml

Verify creation:

oc get userdefinednetwork -n udn-example

4. Create a VM on the User Defined Network

VMs in a namespace with a UDN automatically use the UDN as their primary network:

example-vm.yaml

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
  namespace: udn-example
spec:
  runStrategy: Always
  template:
    spec:
      domain:
        devices:
          disks:
          - name: rootdisk
            disk:
              bus: virtio
          interfaces:
          - name: default
            binding:
              name: l2bridge
        resources:
          requests:
            memory: 2Gi
      networks:
      - name: default
        pod: {}
      volumes:
      - name: rootdisk
        containerDisk:
          image: quay.io/containerdisks/fedora:latest

Note

Don’t modify the VM’s network config. The UDN system handles it automatically.

Important

Restart or live migrate the VM after network changes:

# Restart the VM:
virtctl restart example-vm

# Or live migrate (no downtime):
virtctl migrate example-vm

5. Create a Cluster User Defined Network

For cross-namespace communication, create a Cluster UDN:

Create namespaces with the proper labels:

udn-namespaces.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: udn-prod
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
    cluster-udn: prod
---
apiVersion: v1
kind: Namespace
metadata:
  name: udn-preprod
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
    cluster-udn: prod

Create the Cluster UDN:

cluster-udn-prod.yaml

apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  name: cluster-udn-prod
spec:
  namespaceSelector:
    matchLabels:
      cluster-udn: prod
  network:
    layer2:
      ipam:
        lifecycle: Persistent
      role: Primary
      subnets:
      - 10.100.0.0/16
    topology: Layer2

Apply the config:
```
oc create -f cluster-udn-prod.yaml
```
Verify creation:
```
oc get clusteruserdefinednetwork
```

Testing and Validation

1. Test VM Connectivity within UDN

Create multiple VMs in the same namespace with a UDN

Connect to the VM:

# Use console
virtctl console example-vm

# Or SSH if available
virtctl ssh example-vm

Check network config:
```
ip addr show
ip route
```
Test connectivity between VMs:
```
ping <other-vm-ip-address>
```
Verify VMs got IP addresses from the UDN subnet

2. Test VM Connectivity across Cluster UDN

Create VMs in different namespaces connected by a Cluster UDN
Check network config in each VM

Test connectivity:

# From VM in namespace 1
ping <vm-ip-in-namespace-2>

3. Test North/South Network Access

Test external network access:
```
# From inside the VM
ping 1.1.1.1
```

Important

Don’t manually configure IP addresses. The UDN controller manages IP allocation through IPAM.

See also

For direct north/south connectivity using the existing br-ex bridge, see OpenShift Virtualization with Localnet Configuration.

OpenShift Virtualization with Localnet Configuration

Tue, 22 Apr 2025 00:00:00

April 22, 2025

15 min read

This use case demonstrates how to configure OpenShift Virtualization to leverage the existing br-ex OVS bridge for direct north/south connectivity of virtual machines. Instead of using a dedicated interface, this approach reuses the main OpenShift interface to provide IP addresses from your baremetal network to your KubeVirt virtual machines.

See also

For more information about OpenShift Virtualization networking, check out the official OpenShift Virtualization Documentation.

Note

This is compatible with OpenShift 4.18 and newer versions.

Implementation Steps

1. Understanding Localnet and br-ex Bridge

The OpenShift br-ex bridge is typically used for external connectivity for the OpenShift cluster. By configuring a localnet mapping to this bridge, we can:

Allow virtual machines to obtain IP addresses directly from the baremetal network
Enable direct north/south connectivity without network translation
Simplify network architecture by reusing existing resources

Note

The br-ex bridge is part of OVN-Kubernetes networking in OpenShift. This approach eliminates the need for dedicated physical interfaces for VM connectivity.

2. Configure NodeNetworkConfigurationPolicy

Create a NodeNetworkConfigurationPolicy (NNCP) to add a localnet mapping to the br-ex bridge:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: br-ex-network
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: '' 
  desiredState:
    ovn:
      bridge-mappings:
      - localnet: br-ex-network
        bridge: br-ex 
        state: present

Apply the policy:

oc apply -f br-ex-nncp.yaml

3. Verify NodeNetworkConfigurationPolicy Status

Check that the policy has been applied correctly:

oc get nncp

Expected output:

NAME            STATUS      REASON
br-ex-network   Available   SuccessfullyConfigured

Verify the node network configuration enactments:

oc get nnce

Expected output:

NAME                         STATUS      STATUS AGE   REASON
<node-name>.br-ex-network   Available   <age>         SuccessfullyConfigured

4. Create NetworkAttachmentDefinition

Create a NetworkAttachmentDefinition (NAD) that will use the localnet:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: br-ex-network
  namespace: default
spec:
  config: '{
            "name":"br-ex-network",
            "type":"ovn-k8s-cni-overlay",
            "cniVersion":"0.4.0",
            "topology":"localnet",
            "netAttachDefName":"default/br-ex-network"
          }'

Apply the NetworkAttachmentDefinition:

oc apply -f br-ex-network-nad.yaml

VLAN Configuration Example

To configure a NetworkAttachmentDefinition with a specific VLAN ID, use the vlanID property in the config. This is particularly useful for environments where network segmentation is required:

Tip

Using VLANs with localnet can help maintain network isolation while still leveraging the existing physical infrastructure.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: vlan-network
  namespace: default
spec:
  config: |
    {
            "cniVersion": "0.3.1",
            "name": "vlan-network",
            "type": "ovn-k8s-cni-overlay",
            "topology": "localnet",
            "vlanID": 200,
            "netAttachDefName": "default/vlan-network"
    }

Apply the VLAN NetworkAttachmentDefinition:

oc apply -f vlan-network-nad.yaml

In this example, the virtual machines connected to this network will receive VLAN tagged traffic with VLAN ID 200.

5. Adding Network Interface to Virtual Machines

To attach a VM to the localnet bridge, modify your VM definition to include an additional network interface:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
spec:
  runStrategy: Always
  template:
    spec:
      domain:
        devices:
          disks:
          - name: rootdisk
            disk:
              bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          - name: br-ex-interface
            bridge: {}
        resources:
          requests:
            memory: 2Gi
      networks:
      - name: default
        pod: {}
      - name: br-ex-interface
        multus:
          networkName: default/br-ex-network
      volumes:
      - name: rootdisk
        containerDisk:
          image: quay.io/containerdisks/fedora:latest

Testing and Validation

1. Verify VM Network Configuration

Connect to the VM console or SSH:

virtctl console example-vm
# or
virtctl ssh example-vm

Check network interfaces:
```
ip addr show
```
Verify you have two interfaces:
- First interface connected to the pod network (default)
- Second interface connected to the br-ex network with an IP from your baremetal network

2. Test External Network Connectivity

Test network connectivity:
```
# From inside VM
ping 1.1.1.1
```
Test that external hosts can directly reach the VM on its baremetal IP:
```
# From an external machine
ping <vm-ip-address>
```

Important

VMs using localnet networking will be directly exposed to your physical network, so ensure proper security measures are in place.

See also

For isolated tenant networks without physical network exposure, see Layer 2 User Defined Networks for OpenShift Virtualization.

BGP Implementation in Red Hat OpenStack Services on OpenShift

Wed, 20 Nov 2024 00:00:00

Nov 20, 2024

30 min read

Warning

Deprecation Notice: The OVN BGP Agent was deprecated in RHOSO 18.0.10 (Feature Release 3). This component will be removed and replaced in a future release. An alternative BGP integration mechanism will be provided in a future release.

Key Components

RHOSO’s dynamic routing relies on four primary components working together in a distributed architecture that requires dedicated networking nodes:

OVN BGP Agent

The OVN BGP agent is a Python-based daemon running in the ovn_bgp_agent container on Compute and Networker nodes. Its primary functions include:

Database monitoring: Monitors the OVN northbound database for VM and floating IP events
Route management: Triggers FRR to advertise or withdraw routes based on workload lifecycle
Traffic redirection: Configures Linux kernel networking for proper traffic flow
Interface management: Manages the bgp-nic dummy interface for route advertisement

The agent operates by detecting changes in the OVN database and translating these into BGP routing decisions.

Note

In RHOSO 18.0, BGP is deployed through OpenStackControlPlane and OpenStackDataPlaneNodeSet custom resources managed by the RHOSO operators.

OVN BGP Agent Configuration (ovn_bgp_agent.conf)

[DEFAULT]
debug = False
reconcile_interval = 120
expose_tenant_network = False
driver = ovn_bgp_driver

FRR Container Suite

Free Range Routing runs as the frr container on all RHOSO nodes, providing enterprise-grade routing capabilities:

BGP Daemon (bgpd): Manages BGP peer relationships and route advertisements
BFD Daemon (bfdd): Provides sub-second failure detection
Zebra Daemon: Interfaces between FRR and the Linux kernel routing table
VTY Shell: Command-line interface for configuration and monitoring

Sample FRR Configuration for RHOSO

frr version 8.5
frr defaults traditional
hostname rhoso-compute-0
log syslog informational
service integrated-vtysh-config
!
router bgp 64999
 bgp router-id 172.30.1.10
 neighbor 172.30.1.254 remote-as 65000
 neighbor 172.30.1.254 bfd
 !
 address-family ipv4 unicast
  redistribute connected
  maximum-paths 8
 exit-address-family
!

Kernel Networking Integration

RHOSO leverages RHEL kernel networking features configured by the OVN BGP agent:

VRF (Virtual Routing and Forwarding): Network isolation using bgp_vrf
IP Rules: Direct traffic to appropriate routing tables
Dummy Interface: The bgp-nic interface for route advertisement
OVS Integration: Flow rules redirecting traffic to the OVN overlay

Dedicated Networking Nodes

RHOSO BGP deployments require dedicated networking nodes with specific architectural constraints:

Mandatory Architecture: BGP dynamic routing cannot function without dedicated networker nodes
DVR Integration: Must be deployed with Distributed Virtual Routing (DVR) enabled
Traffic Gateway Role: Networker nodes host neutron router gateways and CR-LRP (Chassis Redirect Logical Router Ports)
North-South Traffic: All external traffic to tenant networks flows through networker nodes
BGP Advertisement: Both compute and networker nodes run FRR and OVN BGP agent containers

Networker Node Configuration Requirements

# OpenShift node labels for dedicated networking
apiVersion: v1
kind: Node
metadata:
  name: rhoso-networker-0
  labels:
    node-role.kubernetes.io/rhoso-networker: ""
    feature.node.kubernetes.io/network-sriov.capable: "true"
spec:
  # Networker nodes require specific networking capabilities
  # and dedicated hardware for external connectivity

Architecture Constraints:

Control Plane OVN Gateways: Not supported with BGP (incompatible)
Octavia Load Balancer: Cannot be used with BGP dynamic routing
BFD Limitations: Bi-directional forwarding detection has known issues

Network Architecture

RHOSO BGP Network Topology

--- config: theme: neutral --- graph TB subgraph "External Network" TOR1[ToR Switch 1 AS 65000] TOR2[ToR Switch 2 AS 65000] EXT[External Networks 192.0.2.0/24] end subgraph "OpenShift Cluster" subgraph "RHOSO Control Plane" CP1[Controller Pod 1 172.30.1.1] CP2[Controller Pod 2 172.30.1.2] CP3[Controller Pod 3 172.30.1.3] end subgraph "RHOSO Compute Nodes" CN1[Compute Node 1 FRR + OVN BGP Agent] CN2[Compute Node 2 FRR + OVN BGP Agent] CN3[Compute Node 3 FRR + OVN BGP Agent] end subgraph "OVN Overlay" VM1[VM 10.0.0.10 Floating IP: 192.0.2.10] VM2[VM 10.0.0.20 Floating IP: 192.0.2.20] LB[Load Balancer 192.0.2.100] end end TOR1 -.->|eBGP AS 64999| CN1 TOR1 -.->|eBGP AS 64999| CN2 TOR2 -.->|eBGP AS 64999| CN2 TOR2 -.->|eBGP AS 64999| CN3 CN1 --> VM1 CN2 --> VM2 CN3 --> LB TOR1 <--> EXT TOR2 <--> EXT

BGP Component Interactions

The following diagram shows detailed interactions between all RHOSO BGP components as they operate within the OpenShift container environment:

--- config: theme: neutral --- graph TB subgraph "External Infrastructure" PEER1[BGP Peer 1 ToR Switch AS 65000] PEER2[BGP Peer 2 ToR Switch AS 65000] end subgraph "OpenShift Node (Compute/Networker)" subgraph "Container Runtime" subgraph "FRR Container" BGPD[BGP Daemon bgpd] ZEBRA[Zebra Daemon Route Manager] BFD[BFD Daemon bfdd] VTY[VTY Shell vtysh] end subgraph "OVN BGP Agent Container" AGENT[OVN BGP Agent Python Daemon] DRIVER[BGP Speaker Driver ovn_bgp_driver] end end subgraph "Host Kernel" KRTAB[Kernel Routing Table] BGPNIC[bgp-nic Interface Dummy Interface] BRVRF[bgp_vrf VRF Instance] IPRULES[IP Rules & Routes] end subgraph "OVS Integration" BREX[br-ex Provider Bridge] FLOWS[OVS Flow Rules] end subgraph "OVN Database" OVNNB[(OVN Northbound Database)] OVNSB[(OVN Southbound Database)] end end %% External BGP Sessions PEER1 <==>|TCP 179 BGP Session| BGPD PEER2 <==>|TCP 179 BGP Session| BGPD %% Internal Component Interactions BGPD <--> ZEBRA BGPD <--> BFD ZEBRA <--> KRTAB ZEBRA <--> BGPNIC %% OVN BGP Agent Monitoring AGENT --> OVNNB AGENT --> OVNSB AGENT --> BGPNIC AGENT --> IPRULES AGENT --> FLOWS %% Route Advertisement Flow BGPNIC --> KRTAB KRTAB --> ZEBRA ZEBRA --> BGPD BGPD --> PEER1 BGPD --> PEER2 %% Traffic Redirection IPRULES --> BREX FLOWS --> BREX BRVRF --> BREX %% Configuration Management VTY -.-> BGPD VTY -.-> ZEBRA VTY -.-> BFD DRIVER -.-> AGENT

Traffic Flow Process

When a VM is created or a floating IP is assigned, the following sequence occurs:

sequenceDiagram participant OVN as OVN Database participant Agent as OVN BGP Agent participant FRR as FRR BGP Daemon participant Kernel as Linux Kernel participant Peer as BGP Peer OVN->>Agent: VM created with IP 192.0.2.10 Agent->>Agent: Add IP to bgp-nic interface Agent->>Kernel: Configure IP rules and routes Agent->>Kernel: Configure OVS flows Kernel->>FRR: Route appears in kernel table FRR->>Peer: Advertise route 192.0.2.10/32 Peer->>Peer: Update routing table Note over Agent,Kernel: Traffic redirection configured Note over FRR,Peer: Route convergence complete

Private Network Advertising

RHOSO BGP supports advertising private tenant networks, though this feature is disabled by default due to security implications.

Tenant Network Exposure Configuration

Default Behavior: By default, only floating IPs and provider network IPs are advertised via BGP. Private tenant networks remain isolated within the OVN overlay.

Enabling Tenant Network Advertisement:

OVN BGP Agent Configuration with Tenant Network Exposure

[DEFAULT]
debug = False
reconcile_interval = 120
expose_tenant_network = True
driver = ovn_bgp_driver

Security Considerations:

Network Isolation: Enabling tenant network exposure breaks traditional OpenStack network isolation
Routing Policies: External routers must implement proper filtering to maintain security boundaries
Non-Overlapping CIDRs: Tenant networks must use unique, non-overlapping IP ranges
Access Control: External network infrastructure must enforce tenant access policies

Traffic Flow for Tenant Networks

When tenant network advertising is enabled, traffic follows a specific path through dedicated networking nodes:

--- config: theme: neutral --- graph TD CLIENT[External Client 203.0.113.100] TOR[ToR Switch BGP AS 65000 Routes: 10.1.0.0/24] subgraph NETWORKER ["Networker Node"] FRR1[FRR Container BGP: AS 64999 Advertises: 10.1.0.0/24] AGENT1[OVN BGP Agent Monitors OVN DB Configures CR-LRP] KERNEL1[Host Kernel IP Rules & Routes] BREX1[br-ex Bridge Provider Bridge] CRLRP1[CR-LRP Gateway 10.1.0.1] end subgraph COMPUTE ["Compute Node - OVN Overlay"] LROUTER[Logical Router tenant-router-1] LSWITCH[Logical Switch tenant-net-1 10.1.0.0/24] VM[Tenant VM 10.1.0.10] end %% Main Traffic Flow (Top to Bottom) CLIENT -->|1. Send packet dst: 10.1.0.10| TOR TOR -->|2. BGP route lookup forward to networker| KERNEL1 KERNEL1 -->|3. Apply IP rules route to br-ex| BREX1 BREX1 -->|4. Traffic injection enter OVN overlay| CRLRP1 CRLRP1 -->|5. L3 gateway route to logical switch| LROUTER LROUTER -->|6. ARP resolution L3 to L2 forwarding| LSWITCH LSWITCH -->|7. MAC lookup deliver packet| VM %% BGP Advertisement Flow (Dashed) AGENT1 -.->|Monitor tenant network events| LSWITCH AGENT1 -.->|Configure gateway on networker node| CRLRP1 AGENT1 -.->|Add route to bgp-nic interface| FRR1 FRR1 -.->|BGP UPDATE advertise 10.1.0.0/24| TOR

Key Technical Details:

CR-LRP Role: Chassis Redirect Logical Router Ports serve as the entry point for external traffic to tenant networks
Networker Node Gateway: All north-south traffic to tenant networks must traverse the networker node hosting the neutron router gateway
Route Advertisement: The OVN BGP agent on networker nodes advertises neutron router gateway ports when tenant network exposure is enabled

Implementation Requirements

Network Planning:

Tenant Network BGP Advertisement Example

# Tenant network configuration
router bgp 64999
  # Advertise tenant network ranges
  address-family ipv4 unicast
    network 10.0.0.0/24    # Tenant network 1
    network 10.1.0.0/24    # Tenant network 2
    network 10.2.0.0/24    # Tenant network 3
  exit-address-family

# Route filtering for security
ip prefix-list TENANT-NETWORKS permit 10.0.0.0/8 le 24
route-map TENANT-FILTER permit 10
  match ip address prefix-list TENANT-NETWORKS

Operational Considerations:

Network Overlap Detection: Implement monitoring to detect and prevent CIDR overlaps
Route Filtering: Configure external routers with appropriate filters to prevent route leaks
Multi-Tenancy: Consider impact on tenant isolation and implement additional security measures
Troubleshooting Complexity: Private network advertising increases troubleshooting complexity

Real-World Deployment Scenarios

Enterprise Multi-Zone Deployment

Use Case: Large enterprise with RHOSO deployed across multiple OpenShift clusters in different availability zones.

Multi-Zone Network Topology:

--- config: theme: neutral --- graph TB subgraph "Enterprise WAN" WAN[Enterprise WAN Core Network AS 65000] MPLS[MPLS/VPN Backbone] end subgraph "Availability Zone 1 (East)" subgraph "Zone 1 Network Infrastructure" TOR1A[ToR Switch 1A AS 65001] TOR1B[ToR Switch 1B AS 65001] SPINE1[Spine Switch 1 AS 65001] end subgraph "OpenShift Cluster 1" subgraph "RHOSO Control Plane 1" CP1A[Controller Pod 1A] CP1B[Controller Pod 1B] end subgraph "Networker Nodes Zone 1" NN1A[Networker 1A BGP AS 64999 Router-ID: 10.1.1.1] NN1B[Networker 1B BGP AS 64999 Router-ID: 10.1.1.2] end subgraph "Compute Nodes Zone 1" CN1A[Compute 1A] CN1B[Compute 1B] CN1C[Compute 1C] end end subgraph "Workloads Zone 1" VIP1[Control Plane VIP 203.0.113.10] TENANT1[Tenant Networks 10.1.0.0/16] FIP1[Floating IPs 203.0.113.100~200] end end subgraph "Availability Zone 2 (West)" subgraph "Zone 2 Network Infrastructure" TOR2A[ToR Switch 2A AS 65002] TOR2B[ToR Switch 2B AS 65002] SPINE2[Spine Switch 2 AS 65002] end subgraph "OpenShift Cluster 2" subgraph "RHOSO Control Plane 2" CP2A[Controller Pod 2A] CP2B[Controller Pod 2B] end subgraph "Networker Nodes Zone 2" NN2A[Networker 2A BGP AS 64998 Router-ID: 10.2.1.1] NN2B[Networker 2B BGP AS 64998 Router-ID: 10.2.1.2] end subgraph "Compute Nodes Zone 2" CN2A[Compute 2A] CN2B[Compute 2B] CN2C[Compute 2C] end end subgraph "Workloads Zone 2" VIP2[Control Plane VIP 203.0.113.11] TENANT2[Tenant Networks 10.2.0.0/16] FIP2[Floating IPs 203.0.113.201~255] end end %% WAN Connectivity WAN <--> MPLS MPLS <--> SPINE1 MPLS <--> SPINE2 %% Zone 1 Internal BGP SPINE1 <--> TOR1A SPINE1 <--> TOR1B TOR1A -.->|eBGP| NN1A TOR1A -.->|eBGP| CN1A TOR1B -.->|eBGP| NN1B TOR1B -.->|eBGP| CN1B %% Zone 2 Internal BGP SPINE2 <--> TOR2A SPINE2 <--> TOR2B TOR2A -.->|eBGP| NN2A TOR2A -.->|eBGP| CN2A TOR2B -.->|eBGP| NN2B TOR2B -.->|eBGP| CN2B %% Inter-Zone BGP Peering NN1A -.->|iBGP via WAN| NN2A NN1B -.->|iBGP via WAN| NN2B %% Workload Distribution NN1A --> VIP1 NN1A --> TENANT1 NN1A --> FIP1 NN2A --> VIP2 NN2A --> TENANT2 NN2A --> FIP2

Technical Implementation:

Multi-Zone BGP Configuration

# Zone 1 Configuration (AS 64999)
router bgp 64999
  bgp router-id 10.1.1.1

  # Local zone ToR peering (eBGP)
  neighbor 10.1.1.254 remote-as 65001
  neighbor 10.1.1.253 remote-as 65001

  # Inter-zone peering (iBGP confederation or eBGP)
  neighbor 10.2.1.1 remote-as 64998
  neighbor 10.2.1.1 ebgp-multihop 3

  address-family ipv4 unicast
    # Advertise zone-specific networks
    network 203.0.113.10/32    # Control plane VIP
    network 203.0.113.100/28   # Zone 1 floating IPs
    network 10.1.0.0/16        # Zone 1 tenant networks (if enabled)

    # ECMP for load balancing
    maximum-paths 4
    maximum-paths ibgp 4

    # Route filtering between zones
    neighbor 10.2.1.1 route-map ZONE2-IN in
    neighbor 10.2.1.1 route-map ZONE1-OUT out
  exit-address-family

# Zone 2 Configuration (AS 64998)
router bgp 64998
  bgp router-id 10.2.1.1

  # Local zone ToR peering (eBGP)
  neighbor 10.2.1.254 remote-as 65002
  neighbor 10.2.1.253 remote-as 65002

  # Inter-zone peering
  neighbor 10.1.1.1 remote-as 64999
  neighbor 10.1.1.1 ebgp-multihop 3

  address-family ipv4 unicast
    # Advertise zone-specific networks
    network 203.0.113.11/32    # Control plane VIP
    network 203.0.113.201/28   # Zone 2 floating IPs
    network 10.2.0.0/16        # Zone 2 tenant networks (if enabled)

    maximum-paths 4
    maximum-paths ibgp 4
  exit-address-family

Benefits: - Geographic Distribution: Workloads distributed across multiple data centers - Fault Isolation: Zone failures don’t impact other zones - Load Distribution: Traffic distributed based on BGP routing policies - Disaster Recovery: Automatic failover between zones via BGP route withdrawal - Scalability: Independent scaling of compute and network resources per zone

Control Plane High Availability

Use Case: RHOSO control plane services distributed across OpenShift nodes with BGP-advertised VIPs.

--- config: theme: neutral --- graph LR subgraph "OpenShift Cluster" CP1[Control Plane Pod 1 Node: ocp-master-1] CP2[Control Plane Pod 2 Node: ocp-master-2] CP3[Control Plane Pod 3 Node: ocp-master-3] VIP[Control Plane VIP 192.0.2.100] end subgraph "External Network" Client[API Clients] BGP[BGP Router] end CP1 -.-> VIP CP2 -.-> VIP CP3 -.-> VIP VIP --> BGP BGP --> Client

Implementation Details: - Kubernetes-native HA manages VIP assignment - OVN BGP agent advertises active VIP location - Sub-second failover with BFD - No single point of failure

Dedicated Networker Node Deployment

Use Case: Enterprise RHOSO deployment with dedicated networking infrastructure for BGP routing and tenant network isolation.

Architecture Requirements:

--- config: theme: neutral --- graph TB subgraph "External Infrastructure" TOR1[ToR Switch 1 AS 65000] TOR2[ToR Switch 2 AS 65000] FW[Enterprise Firewall] end subgraph "OpenShift Cluster" subgraph "Control Plane Nodes" CP1[Master 1] CP2[Master 2] CP3[Master 3] end subgraph "Dedicated Networker Nodes" NN1[Networker 1 FRR + OVN BGP Agent CR-LRP Host] NN2[Networker 2 FRR + OVN BGP Agent CR-LRP Host] end subgraph "Compute Nodes" CN1[Compute 1 FRR + OVN BGP Agent] CN2[Compute 2 FRR + OVN BGP Agent] CN3[Compute 3 FRR + OVN BGP Agent] end end subgraph "Tenant Networks" T1[Tenant A: 10.1.0.0/24] T2[Tenant B: 10.2.0.0/24] T3[Tenant C: 10.3.0.0/24] end TOR1 -.->|eBGP| NN1 TOR1 -.->|eBGP| CN1 TOR2 -.->|eBGP| NN2 TOR2 -.->|eBGP| CN2 NN1 --> T1 NN1 --> T2 NN2 --> T2 NN2 --> T3 FW <--> TOR1 FW <--> TOR2

Benefits: - Dedicated Traffic Path: All north-south traffic controlled through networker nodes - High Availability: Multiple networker nodes provide redundancy for tenant network access - Security Isolation: Clear separation between compute and networking functions - Scalability: Independent scaling of compute and network infrastructure

Deployment Considerations: - Hardware Requirements: Networker nodes need enhanced networking capabilities - Network Connectivity: Direct physical connections to external infrastructure - DVR Requirement: Must be deployed with Distributed Virtual Routing enabled - Monitoring: Enhanced monitoring required for CR-LRP and gateway functions

Configuration and Deployment

Prerequisites

RHOSO dynamic routing requires:

RHOSO 18.0 or later with ML2/OVN mechanism driver
OpenShift 4.18+ with appropriate node networking
BGP-capable network infrastructure (ToR switches, routers)
Dedicated networker nodes (mandatory for BGP deployments)
Distributed Virtual Routing (DVR) enabled
Proper network planning for ASN assignment and IP addressing

Critical Architecture Requirements:

No Control Plane OVN Gateways: BGP is incompatible with control plane OVN gateway deployments
No Octavia Load Balancer: Cannot be used simultaneously with BGP dynamic routing
No Distributed Control Plane: RHOSO dynamic routing does not support distributed control planes across datacenters
Non-overlapping CIDRs: When using tenant network advertising, all networks must use unique IP ranges
External BGP Peers: Network infrastructure must support BGP peering and route filtering

Note

Cross-Datacenter Deployment Design Considerations

RHOSO BGP deployments across datacenters require additional planning and design considerations:

Control Plane Design: Consider the implications of control plane service communication patterns across WAN links
Network Latency Planning: Evaluate network latency requirements for optimal OpenStack service performance
Database Architecture: Plan for database replication, backup, and disaster recovery strategies
Storage Design: Consider storage architecture options that balance performance, availability, and data locality

Alternative Options: Red Hat’s Distributed Compute Node (DCN) architecture provides a proven approach for multi-site deployments with centralized control planes and distributed compute resources.

OpenShift Integration

RHOSO BGP services integrate with OpenShift through:

Pod Deployment: BGP containers run as privileged pods with host networking ConfigMaps: Configuration stored as Kubernetes ConfigMaps Monitoring: Integration with OpenShift monitoring and alerting Networking: Uses OpenShift SDN or OVN-Kubernetes for pod networking

Production Operations

Monitoring and Observability

RHOSO BGP monitoring leverages OpenShift’s native observability:

Prometheus Metrics: FRR and OVN BGP agent export metrics Grafana Dashboards: Pre-built dashboards for BGP performance Alerting: Automated alerts for BGP session failures Logging: Centralized logging through OpenShift logging stack

BGP Status Monitoring Commands

# Check BGP session status
oc exec -n openstack ds/frr-bgp -- vtysh -c 'show bgp summary'

# View route advertisements
oc exec -n openstack ds/frr-bgp -- vtysh -c 'show ip bgp neighbors advertised-routes'

# Check OVN BGP agent status
oc logs -n openstack -l app=ovn-bgp-agent --tail=50

Scaling Operations

Adding new compute capacity with BGP requires:

OpenShift node addition: Standard OpenShift node scaling procedures
Automatic BGP configuration: DaemonSets ensure BGP services on new nodes
Network validation: Verify BGP peering and route advertisement
Workload validation: Test VM connectivity through new nodes

Network Failure and Recovery

RHOSO BGP deployments implement automated failure detection and recovery mechanisms:

--- config: theme: neutral --- flowchart TD START([Normal Operation BGP Sessions Active]) --> MONITOR{Monitoring Systems} MONITOR -->|BGP Session Failure| BGP_FAIL[BGP Session Down] MONITOR -->|Node Failure| NODE_FAIL[Networker Node Down] MONITOR -->|BFD Timeout| BFD_FAIL[BFD Session Timeout] BGP_FAIL --> BGP_DETECT{Failure Detection Method} BGP_DETECT -->|BFD Enabled| BFD_FAST[Sub-second Detection 100~300ms] BGP_DETECT -->|BGP Keepalive Only| BGP_SLOW[Standard Detection 30~90 seconds] BFD_FAST --> WITHDRAW_ROUTES[Withdraw BGP Routes from Failed Peer] BGP_SLOW --> WITHDRAW_ROUTES BFD_FAIL --> WITHDRAW_ROUTES NODE_FAIL --> CR_LRP_CHECK{CR-LRP Migration Available?} CR_LRP_CHECK -->|Yes| CR_LRP_MIGRATE[Migrate CR-LRP to Healthy Networker Node] CR_LRP_CHECK -->|No| TENANT_OUTAGE[Tenant Network Connectivity Lost] WITHDRAW_ROUTES --> REROUTE[Traffic Rerouted via Remaining Peers] CR_LRP_MIGRATE --> REROUTE REROUTE --> RECOVERY_CHECK{Service Recovery?} RECOVERY_CHECK -->|BGP Peer Recovery| PEER_RECOVERY[Re-establish BGP Session] RECOVERY_CHECK -->|Node Recovery| NODE_RECOVERY[Node Rejoins Cluster] PEER_RECOVERY --> READVERTISE[Re-advertise BGP Routes] NODE_RECOVERY --> POD_RESTART[Restart BGP Services] POD_RESTART --> READVERTISE READVERTISE --> CONVERGE[Network Convergence] CONVERGE --> START TENANT_OUTAGE --> MANUAL_INTERVENTION[Manual Intervention Required] MANUAL_INTERVENTION -->|Add Networker Node| NODE_RECOVERY

Failure Scenarios and Recovery Times:

RHOSO BGP Failure Recovery Matrix
Failure Type	Detection Time	Recovery Time	Impact Level
BGP Session Failure (BFD enabled)	100~300ms	1~3 seconds	Low (alternate paths)
BGP Session Failure (no BFD)	30~90 seconds	30~120 seconds	Medium (delayed reroute)
Single Networker Node	5~15 seconds	10~30 seconds	Medium (CR-LRP migration)
Multiple Networker Nodes	5~15 seconds	Manual intervention	High (tenant isolation)
Compute Node	30~60 seconds	60~180 seconds	Low (workload migration)

Automated Recovery Mechanisms:

BGP Route Withdrawal: Automatic route withdrawal upon session failure
BFD Fast Detection: Sub-second failure detection with proper BFD configuration
CR-LRP Migration: Automatic migration of Chassis Redirect Logical Router Ports
Traffic Rerouting: ECMP and alternate path utilization
Session Re-establishment: Automatic BGP session recovery

Troubleshooting

Common Issues and Solutions

BGP Sessions Not Establishing

Symptoms: BGP peers show “Idle” or “Connect” state

Diagnosis Commands

# Check BGP peer status
oc exec -n openstack ds/frr-bgp -- vtysh -c 'show bgp neighbors'

# Verify network connectivity
oc exec -n openstack ds/frr-bgp -- ping <peer-ip>

# Check firewall rules
oc exec -n openstack ds/frr-bgp -- ss -tulpn | grep 179

Solution: Verify network connectivity, ASN configuration, and firewall rules allowing TCP port 179.

Routes Not Being Advertised

Symptoms: External networks cannot reach RHOSO workloads

Diagnosis and Resolution

# Check if IPs are on bgp-nic interface
oc exec -n openstack ds/ovn-bgp-agent -- ip addr show bgp-nic

# Verify FRR is redistributing connected routes
oc exec -n openstack ds/frr-bgp -- vtysh -c 'show running-config'

# Check OVN BGP agent logs
oc logs -n openstack -l app=ovn-bgp-agent

Solution: Ensure OVN BGP agent is running and FRR has “redistribute connected” configured.

Tenant Networks Not Reachable

Symptoms: External clients cannot reach VMs on tenant networks despite expose_tenant_network = True

Tenant Network Troubleshooting

# Check if tenant network exposure is enabled
oc exec -n openstack ds/ovn-bgp-agent -- grep expose_tenant_network /etc/ovn_bgp_agent/ovn_bgp_agent.conf

# Verify CR-LRP (Chassis Redirect Logical Router Ports) are active on networker nodes
oc exec -n openstack ds/ovn-bgp-agent -- ovn-sbctl show | grep cr-lrp

# Check neutron router gateway port advertisement
oc exec -n openstack ds/frr-bgp -- vtysh -c 'show ip bgp | grep 10.0.0'

Solution: Verify networker nodes are hosting CR-LRP and neutron router gateways are properly advertised.

Networker Node Failures

Symptoms: Complete loss of external connectivity to tenant networks

Networker Node Failure Diagnosis

# Check networker node health
oc get nodes -l node-role.kubernetes.io/rhoso-networker

# Verify networker pods are running
oc get pods -n openstack -l app=rhoso-networker

# Check CR-LRP failover status
oc exec -n openstack ds/ovn-bgp-agent -- ovn-sbctl find Chassis_Redirect_Port

# Verify BGP session status on remaining networker nodes
oc exec -n openstack ds/frr-bgp -- vtysh -c 'show bgp summary'

Solution: Ensure multiple networker nodes are deployed for high availability and CR-LRP can migrate between nodes.

Slow Convergence

Symptoms: Long failover times during node or network failures

BFD Configuration for Fast Convergence

router bgp 64999
  neighbor 172.30.1.254 bfd
  neighbor 172.30.1.254 bfd profile fast-detect

bfd
  profile fast-detect
    detect-multiplier 3
    receive-interval 100
    transmit-interval 100

Known Issues

Floating IP port forwarding: Port forwarding with floating IPs fails when BGP dynamic routing is enabled (BZ 2160481)
Multicast routing: Multicast routing is not supported with BGP dynamic routing (BZ 2163477)

Performance Tuning

ECMP Configuration

Optimized ECMP Settings

router bgp 64999
  maximum-paths 8
  maximum-paths ibgp 8
  bestpath as-path multipath-relax

BGP Timers Optimization

Production BGP Timers

router bgp 64999
  neighbor 172.30.1.254 timers 10 30
  neighbor 172.30.1.254 capability extended-nexthop

Dual Interface Routing with NGINX Ingress in Kubernetes

Fri, 16 Aug 2024 00:00:00

Aug 16, 2024

15 min read

While I would consider this a basic requirement I never understood why Kubernetes Ingress Routers didn’t natively provides a way to expose private endpoints on one interface and public ones on a second interface (OpenShift does it quite well).

The following describes how one would implement true network separation in Kubernetes using dual interface routing with NGINX ingress controllers and externalIPs services and enable isolation between public and private services.

Architecture Overview

Our dual interface implementation consists of:

Public Interface (eth0): Routes external traffic to public services
Private Interface (eth1): Routes internal traffic to private services
ExternalIPs Services: Bind NGINX controllers to specific interface IPs
Ingress Class Selection: Applications choose routing via ingress class

External Clients
├── Public Interface (168.119.158.20:80/443)
│   └── nginx-public-external Service (externalIPs)
│       └── nginx-ingress-public Controller
│           ├── blog.epheo.eu (public blog)
│           └── epheo.eu (personal site)
│
└── Private Interface (172.16.0.11:80/443)
    └── nginx-private-external Service (externalIPs)
        └── nginx-ingress-private Controller
            ├── argocd (GitOps management)
            └── git (Git server)

Implementation Details

Network Interface Configuration

The implementation assumes two network interfaces on Kubernetes nodes:

eth0: 168.119.158.20    # Public interface (firewall: 80/443 only)
eth1: 172.16.0.11     # Private interface (no firewall restrictions)

ExternalIPs Services Setup

Create services that bind NGINX controllers to specific interface IPs:

nginx-public-external.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-public-external
  namespace: nginx-ingress-public
  labels:
    app.kubernetes.io/name: nginx-public-external
    app.kubernetes.io/component: networking
    app.kubernetes.io/instance: nginx-external-services
    app.kubernetes.io/part-of: platform
spec:
  type: ClusterIP
  externalIPs:
  - 168.119.158.20  # eth0 public interface
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: nginx-ingress-public
    app.kubernetes.io/component: controller

nginx-private-external.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-private-external
  namespace: nginx-ingress-private
  labels:
    app.kubernetes.io/name: nginx-private-external
    app.kubernetes.io/component: networking
    app.kubernetes.io/instance: nginx-external-services
    app.kubernetes.io/part-of: platform
spec:
  type: ClusterIP
  externalIPs:
  - 172.16.0.11  # eth1 private interface
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: nginx-ingress-private
    app.kubernetes.io/component: controller

NGINX Controller Configuration

Configure separate NGINX ingress controllers for each interface:

public-controller-values.yaml

controller:
  name: nginx-public
  ingressClass: nginx-public
  ingressClassResource:
    name: nginx-public
    enabled: true
    default: false
    controllerValue: "k8s.io/nginx-public"

  # Minimal configuration for reliable operation
  config:
    use-forwarded-headers: "true"
    compute-full-forwarded-for: "true"
    use-proxy-protocol: "false"

  # Resource configuration
  resources:
    requests:
      cpu: 100m
      memory: 90Mi
    limits:
      cpu: 500m
      memory: 256Mi

The private controller uses an identical configuration with nginx-private replacing nginx-public and k8s.io/nginx-private as the controller value.

Application Configuration

Public Service Example

Configure public-facing applications to use the public ingress class:

public-app-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: epheo-eu
  namespace: epheo-eu
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx-public
  tls:
    - hosts:
        - epheo.eu
      secretName: epheo-eu-tls
  rules:
    - host: epheo.eu
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: epheo-eu
                port:
                  number: 80

Private Service Example

Configure administrative and internal services to use the private ingress class:

argocd-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: self-signed-issuer
spec:
  ingressClassName: nginx-private
  tls:
    - hosts:
        - argocd.def.ms
      secretName: argocd-server-tls
  rules:
    - host: argocd.def.ms
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 80

Verification

Verify the ingress controllers are running and routing correctly:

# Check both controllers are running
kubectl get pods -n nginx-ingress-public
kubectl get pods -n nginx-ingress-private

# Verify services have correct externalIPs
kubectl get svc -n nginx-ingress-public nginx-public-external
kubectl get svc -n nginx-ingress-private nginx-private-external

# Test public routing
curl -H "Host: epheo.eu" http://168.119.158.20

# Test private routing
curl -H "Host: argocd.def.ms" http://172.16.0.11

Kubernetes Secret Management with SOPS + Age + Git

Fri, 16 Aug 2024 00:00:00

Aug 16, 2024

15 min read

I don’t like managing secrets in Kubernetes. Solutions like HashiCorp Vault are complicated and resource-intensive. Kubeseal was actually great but now that Bitnami got aquired by Broadcom, I’m concerned about the future of the apiVersion: bitnami.com/ API domain.

I wanted something fully transparent, so I can work with normal secrets in my repo, git commit, have the secrets provisioned on my k8s cluster and never have to care about encryption and secret management.

This repo boilerplate uses SOPS (Secrets OPerationS), AGE encryption and advanced gitattributes filters in order to provide a transparent workflow for developers to manage their secrets.

While this may be enough by itself we can also use sops-secrets-operator when using ArgoCD or Flux for GitOps.

All configurations and examples are available in the k8s-sops-secrets-boilerplate repository.

Why SOPS + Age for Secret Management?

Advantages of This Approach

Transparency: Secrets appear as plain text in your working directory
Simplicity: No external secret management infrastructure required
Git Integration: Automatic encryption when committing to Git
Developer Friendly: Works with existing tooling and workflows
Secure: Strong encryption with Age’s modern cryptographic design
Selective: Only encrypt what needs to be encrypted using annotations

Architecture Overview

The workflow is completely transparent:

Edit secrets in plain text locally
Git automatically encrypts them when staging changes
Secrets are stored encrypted in the upstream repository
Working directory always shows decrypted content

Setting Up the Environment

Installing Required Tools

First, install the required dependencies and tools:

# Install Python dependencies
pip3 install PyYAML

# Install SOPS
curl -LO https://github.com/getsops/sops/releases/download/v3.9.1/sops-v3.9.1.linux.amd64
sudo mv sops-v3.9.1.linux.amd64 /usr/local/bin/sops
sudo chmod +x /usr/local/bin/sops

# Install Age
curl -LO https://github.com/FiloSottile/age/releases/latest/download/age-v1.1.1-linux-amd64.tar.gz
tar xf age-v1.1.1-linux-amd64.tar.gz
sudo mv age/age* /usr/local/bin/
sudo chmod +x /usr/local/bin/age*

Note

Check the SOPS releases and Age releases pages for latest versions.

Generating Age Keys

Create encryption keys for your project:

# Create age directory and generate key pair
mkdir -p .age
age-keygen -o .age/age.key

# Extract the public key for SOPS configuration
grep "public key:" .age/age.key

# Add age directory to gitignore
echo ".age/" >> .gitignore

# Secure the private key
chmod 600 .age/age.key

Note

Store the private key securely and share only the public key with team members who need to encrypt secrets.

Project Configuration

SOPS Configuration

Create a .sops.yaml configuration file in your repository root:

.sops.yaml

creation_rules:
  - path_regex: \.secrets\.ya?ml$
    age: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    encrypted_regex: '^(data|stringData)$'
  - path_regex: secrets/.*\.ya?ml$
    age: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    encrypted_regex: '^(data|stringData)$'

Key configuration options:

path_regex: Files matching this pattern will be encrypted
age: Public key for encryption
encrypted_regex: Only encrypt specified YAML keys (data, stringData for secrets)

Git Filter Setup

The repository includes Python scripts for Git filter integration. First, make them executable:

# Make filter scripts executable
chmod +x sops-clean.py sops-smudge.py

Configure Git to automatically encrypt/decrypt files:

.gitattributes

*.secrets.yaml filter=sops diff=sops
*.secrets.yml filter=sops diff=sops
secrets/*.yaml filter=sops diff=sops
secrets/*.yml filter=sops diff=sops

Configure Git filters using the Python scripts:

# Configure SOPS filters with Python scripts
git config filter.sops.clean './sops-clean.py'
git config filter.sops.smudge './sops-smudge.py'
git config filter.sops.required true

Environment Variables

Set required environment variables to point to your age key:

.envrc (if using direnv)

export SOPS_AGE_KEY_FILE=.age/age.key

Or add to your shell profile:

echo 'export SOPS_AGE_KEY_FILE=.age/age.key' >> ~/.bashrc

Creating Encrypted Secrets

Basic Secret Creation

Create a Kubernetes secret with transparent encryption:

app.secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: production
  annotations:
    # This annotation ensures the secret is managed by our system
    secrets.k8s.io/managed-by: sops
type: Opaque
data:
  database-url: cG9zdGdyZXNxbDovL3VzZXI6cGFzc3dvcmRAZGIuZXhhbXBsZS5jb20vbXlkYg==
  api-key: bG9sLCB5b3UgcmVhbGx5IHRob3VnaHQgSSBkaWQsIHJpZ2h0ID8K
stringData:
  config.json: |
    {
      "database": {
        "host": "db.example.com",
        "port": 5432,
        "username": "myuser",
        "password": "mypassword"
      },
      "api": {
        "key": "super-secret-api-key",
        "endpoint": "https://api.example.com"
      }
    }

When you edit this file locally, you see plain text. When committed to Git, only the data and stringData sections are encrypted.

TLS Certificate Secrets

Manage TLS certificates securely:

tls.secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: app-tls
  namespace: production
type: kubernetes.io/tls
data:
  tls.crt: |
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    BAMMGnNlbGYtc2lnbmVkLWNlcnRpZmljYXRlLTAwHhcNMjEwMzEwMTYwNDAxWhcN
    ...
    LS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
  tls.key: |
    LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0t
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    wxPBDkdPPDGBX8YXbMR7cGVOcLd9qnkL4Zx7gV7lY1P5zt8jRB9XvV4qS4A1z8PF
    ...
    LS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLQ==

Working with Encrypted Secrets

Daily Workflow

The beauty of this system is its transparency:

# Edit secrets normally - they appear as plain text
vim app.secrets.yaml

# Add the file to Git - it gets encrypted automatically
git add app.secrets.yaml

# Commit - the encrypted version is stored
git commit -m "Update database credentials"

# Other team members can clone and immediately see decrypted content
git clone <repository>
cd <project>
cat app.secrets.yaml  # Shows decrypted content

Manual Encryption/Decryption

For advanced use cases, use SOPS directly:

# Encrypt a file manually
sops --encrypt --in-place secrets/app.secrets.yaml

# Decrypt a file manually
sops --decrypt secrets/app.secrets.yaml

# Edit encrypted file directly
sops secrets/app.secrets.yaml

# View decrypted content without modifying
sops --decrypt secrets/app.secrets.yaml | less

Key Rotation

Rotate encryption keys periodically:

# Generate new Age key
age-keygen -o new-key.txt

# Update .sops.yaml with new public key
# Re-encrypt all secrets with new key
find . -name "*.secrets.yaml" -exec sops updatekeys {} \;

# Update team members' key files
cp new-key.txt .age/age.key

Kubernetes Integration

SOPS Secrets Operator

For automated secret management, deploy the SOPS Secrets Operator:

sops-secrets-operator.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: sops-secrets-operator
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sops-secrets-operator
  namespace: sops-secrets-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sops-secrets-operator
  template:
    metadata:
      labels:
        app: sops-secrets-operator
    spec:
      serviceAccountName: sops-secrets-operator
      containers:
      - name: manager
        image: isindir/sops-secrets-operator:0.12.0
        env:
        - name: WATCH_NAMESPACE
          value: ""
        - name: SOPS_AGE_KEY
          valueFrom:
            secretKeyRef:
              name: sops-age-key
              key: key.txt
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi

The operator automatically syncs encrypted secrets from Git to Kubernetes secrets.

Advanced Patterns

Conditional Encryption

Use SOPS path-based rules for selective encryption:

.sops.yaml (advanced)

creation_rules:
  # Production secrets - always encrypted
  - path_regex: prod/.*\.ya?ml$
    age: agexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    encrypted_regex: '^(data|stringData|password|secret|key|token)$'

  # Development secrets - only sensitive fields
  - path_regex: dev/.*\.ya?ml$
    age: agexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    encrypted_regex: '^(password|secret|key|token)$'

  # Test secrets - minimal encryption
  - path_regex: test/.*\.ya?ml$
    age: agexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    encrypted_regex: '^(password)$'

Multi-Key Management

Support multiple teams with different keys:

.sops.yaml (multi-team)

creation_rules:
  # Platform team secrets
  - path_regex: platform/.*\.ya?ml$
    age: >-
      agexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,
      age1xyz...platform-team-key
    encrypted_regex: '^(data|stringData)$'

  # Development team secrets
  - path_regex: apps/.*\.ya?ml$
    age: >-
      age1abc...dev-team-key,
      agexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    encrypted_regex: '^(data|stringData)$'

Audit and Compliance

Track secret changes with Git history:

# View secret change history
git log --oneline -- secrets/

# See who changed secrets
git blame secrets/production.secrets.yaml

# Diff encrypted secrets
git diff HEAD~1 secrets/production.secrets.yaml

Testing and Validation

Repository Testing Framework

The repository includes comprehensive end-to-end testing scripts:

# Quick filter testing - tests Git filters work correctly
./run-e2e-tests.sh local

# Complete GitOps workflow test - full end-to-end validation
./run-e2e-tests.sh full

These tests validate:

Git filter encryption/decryption functionality
SOPS configuration correctness
Complete GitOps workflow with ArgoCD/Flux integration
Secret operator deployment and management

Manual Validation Scripts

Create additional validation scripts for your workflow:

validate-secrets.sh

#!/bin/bash

# Validate all secret files can be decrypted
find . -name "*.secrets.yaml" | while read file; do
    echo "Validating $file..."
    if sops --decrypt "$file" > /dev/null 2>&1; then
        echo "✓ $file is valid"
    else
        echo "✗ $file failed validation"
        exit 1
    fi
done

# Check for unencrypted sensitive data
if grep -r "password:\|secret:\|key:" . --include="*.yaml" --exclude="*.secrets.yaml"; then
    echo "⚠ Found potential unencrypted secrets"
    exit 1
fi

echo "All validations passed"

Debugging Commands

# Test SOPS configuration
sops --config .sops.yaml encrypt /dev/null

# Verify Age key
age-keygen -y .age/age.key

# Check Git filters
git config --list | grep sops

# Test encryption/decryption
echo "test: secret" | sops --encrypt /dev/stdin | sops --decrypt /dev/stdin

# Test Git filter scripts
./sops-clean.py < test.secrets.yaml
./sops-smudge.py < encrypted-test.yaml

Running Mistral:7b LLM on OpenShift

Mon, 05 Feb 2024 00:00:00

Feb 05, 2024

10 min read

Running your local LLM model with Ollama on your OpenShift cluster.

Ollama is an open-source tool that simplifies running and managing Large Language Models (LLMs) locally. It provides a simple API for managing model lifecycles and making inference requests.

Mistral is a powerful Large Language Model trained by a French start-up that currently outperforms other models of the same size. By combining Mistral with Ollama on OpenShift, you can run a production-grade LLM environment within your own infrastructure.

All the files and configurations referenced in this article are available in the openshift-ollama GitHub repository.

Running Mistral:7B on OpenShift

Architecture Overview

Our deployment consists of the following components:

An Ollama server pod with GPU access for model execution
Persistent storage for model weights and cache
Internal and external routes for API access
Optional integration with client applications (Telegram bot, VSCode plugins, iOS app)

Deploying Ollama

First, we’ll create the required OpenShift namespace:

Within an ollama namespace

openshift-ollama.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: ollama

We create the PVC and Deployment using the ollama official container image:

https://hub.docker.com/r/ollama/ollama

openshift-ollama.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-storage
  namespace: ollama
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 100Gi

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ollama
spec:
  selector:
    matchLabels:
      name: ollama
  template:
    metadata:
      labels:
        name: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - name: http
          containerPort: 11434
          protocol: TCP
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /.ollama
          name: ollama-storage
        resources:
          limits:
            nvidia.com/gpu: 1
      restartPolicy: Always
      imagePullPolicy: Always
      volumes:
      - name: ollama-storage
        persistentVolumeClaim:
          claimName: ollama-storage

We notice the nvidia.com/gpu: 1 resources parameter.

Exposing the Ollama API endpoint

We expose the Ollama API endpoint both internally (for other containers like our Telegram Bot) and externally (for services such as VSCode plugins or mobile apps).

Warning

Security note: The external endpoint in this setup is only exposed on a private network accessed via VPN. Since we haven’t configured authentication for this endpoint, exposing it to a public IP would create a significant security risk.

openshift-ollama.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: ollama
spec:
  type: ClusterIP
  selector:
    name: ollama
  ports:
  - port: 80
    name: http
    targetPort: http
    protocol: TCP

---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: ollama
  namespace: ollama
  labels: {}
spec:
  to:
    kind: Service
    name: ollama
  tls: null
  port:
    targetPort: http

Running Mistral LLM on Ollama

Now that our Ollama service is up and running, we can pull the Mistral:7b model using the exposed API endpoint:

$ curl -X POST http://$(oc get route ollama -n ollama -ojsonpath='{.spec.host}')/api/pull -d '{"name": "mistral:7b"}'

Note

Depending on your network speed and GPU capabilities, this may take several minutes as it downloads and prepares the model.

After the download is complete, we can validate that our model has been successfully loaded:

$ curl -s http://$(oc get route ollama -n ollama -ojsonpath='{.spec.host}')/api/tags |jq .
{
  "models": [
    {
      "name": "mistral:7b",
      "model": "mistral:7b",
      "modified_at": "2024-02-03T19:44:00.872177836Z",
      "size": 4109865159,
      "digest": "61e88e884507ba5e06c49b40e6226884b2a16e872382c2b44a42f2d119d804a5",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": [
          "llama"
        ],
        "parameter_size": "7B",
        "quantization_level": "Q4_0"
      }
    }
  ]
}

The response contains important information:

format: GGUF (GPT-Generated Unified Format) - optimized for inference
family: llama - the base architecture family
parameter_size: 7B - the model has 7 billion parameters
quantization_level: Q4_0 - the precision level of the model weights

Creating Custom Models with Specializations

One of Ollama’s powerful features is the ability to create custom models with specialized behaviors without retraining. Let’s create a model specialized in OpenShift documentation:

$ curl http://$(oc get route ollama -n ollama -ojsonpath='{.spec.host}')/api/create -d '{
  "name": "ocplibrarian",
  "modelfile": "FROM mistral:7b\nSYSTEM You are a Librarian, specialized in retrieving content from the OpenShift documentation."
}'

This creates a new model called “ocplibrarian” that inherits all capabilities from Mistral:7b but has been given a specific persona and purpose through the SYSTEM instruction.

Alternative Models: OpenHermes 2.5

Mistral is one of many models you can run on Ollama. Let’s also try OpenHermes 2.5, an instruction-tuned variant of Mistral that offers improved performance on many tasks:

$ curl -X POST http://$(oc get route ollama -n ollama -ojsonpath='{.spec.host}')/api/pull -d '{"name": "openhermes2.5-mistral:7b-q4_K_M"}'

After pulling the model, we can create a customized version with a specific personality:

$ curl http://$(oc get route ollama -n ollama -ojsonpath='{.spec.host}')/api/create -d '{
  "name": "hermes2",
  "modelfile": "FROM openhermes2.5-mistral:7b-q4_K_M\nSYSTEM You are \"Hermes 2\", a conversational AI assistant developed by Teknium. Your purpose is to assist users with helpful, accurate information and engaging conversation."
}'

The model is now available as “hermes2” and carries the specified system prompt that defines its behavior.

Interacting with the LLM

Building a Telegram Bot Interface

For a practical demonstration of the Ollama API, let’s create a simple Telegram bot that serves as an interface to our LLM. We’ll call it Tellama.

The core functionality retrieves user messages from Telegram, forwards them to the Ollama API, and returns the LLM’s responses:

tellama.py

async def chat(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    data = {
      "model": "mistral:7b",
      "messages": [
        {
          "role": "user",
          "content": update.message.text
        }
      ],
      "stream": False
    }
    response = requests.post(OLLAMA_ENDPOINT, json=data )

    if response.status_code == 200:
        ollama_response = response.json().get('message').get('content')
        await update.message.reply_text(ollama_response)
    else:
        await update.message.reply_text('Sorry, there was an error processing your request.')

This implementation is intentionally simple and has some limitations: - No conversation history or context is maintained between messages - No support for streaming responses - Limited error handling

These limitations could be addressed in a more robust implementation, but this serves as a good starting point for testing the API.

Deploying the Telegram Bot on OpenShift

Now let’s containerize our Python script using a UBI (Universal Base Image) container and deploy it to our OpenShift cluster:

# Build the container image
cd tellama
podman build .

# Get the image ID and OpenShift registry URL
export image_id=$(echo $(podman images --format json |jq .[0].Id) | cut -c 2-13)
export ocp_registry=$(oc get route default-route -n openshift-image-registry -ojsonpath='{.spec.host}')

# Push to the OpenShift internal registry
podman login -u <user> -p $(oc whoami -t) ${ocp_registry}
podman push $image_id docker://${ocp_registry}/ollama/tellama:latest

After creating a new Telegram bot messaging @botfather we create a new Secret containing the Telegram Token.

ollama-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: ollama-secret
stringData:
  TELEGRAM_TOKEN: <your_telegram_token>
  OLLAMA_ENDPOINT: "http://ollama/api/chat"

We can now deploy our created image within the OpenShift cluster. Replacing the image: with the value of $ocp_registry

tellama.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tellama
  namespace: ollama
spec:
  selector:
    matchLabels:
      name: ollama
  template:
    metadata:
      labels:
        name: ollama
    spec:
      containers:
      - name: tellama
        image: default-route-openshift-image-registry.apps.da2.epheo.eu/ollama/tellama:latest
        resources: {}
        envFrom:
        - secretRef:
            name: ollama-secret
      restartPolicy: Always
      imagePullPolicy: Always

Integrating with Development Tools

Developer Experience with VSCode Extensions

With our Ollama service running, we can integrate it with various development tools. Here are some VSCode extensions that work well with self-hosted LLMs:

Continue: A VSCode extension similar to GitHub Copilot that can leverage your self-hosted LLM for code completion, explanation, and generation.

See also

https://github.com/continuedev/continue

You can install Continue directly from the VSCode extension marketplace. After installation, you’ll need to configure it to use your self-hosted Ollama service:

Create or edit the ~/.continue/config.json configuration file
Add Ollama as your LLM provider with the appropriate endpoint

Note

Replace the example endpoint with your actual Ollama route: oc get route ollama -n ollama -ojsonpath='{.spec.host}'

{
  "models": [
    {
      "title": "Mistral",
      "provider": "ollama",
      "model": "mistral:7b",
      "apiBase": "http://ollama-ollama.apps.da2.epheo.eu"
    }
  ]
}

commitollama is a specialized VSCode extension that generates meaningful Git commit messages using your self-hosted LLM.

Configuration is straightforward through your VSCode settings:

{
  "commitollama.custom.endpoint": "http://ollama-ollama.apps.da2.epheo.eu",
  "commitollama.model": "custom",
  "commitollama.custom.model": "mistral:7b"
}

With this configuration, you can simply stage your changes and let commitollama analyze the diff to generate descriptive commit messages.

Mobile Integration: Enchanted for iOS

For on-the-go access to your self-hosted LLM, you can use Enchanted, an open-source iOS application specifically designed for Ollama compatibility.

See also

https://github.com/AugustDev/enchanted

Enchanted offers a ChatGPT-like interface for interacting with your self-hosted models, providing a clean, intuitive mobile experience with features like:

Conversation history
Different model selection
Conversation context management
Share and export options

Note

Since our Ollama endpoint is exposed on a private network, your iOS device can connect either through a local WiFi network or by configuring a VPN connection to your network.

Performance Considerations

GPU Selection

While any NVIDIA GPU can run these models, performance varies considerably:

RTX 3080/3090 or better: Excellent for running multiple 7B models
CPU-only: Functional but with much slower inference speeds (5-10x slower)

Model Quantization

Quantization reduces model precision to improve performance at a small cost to accuracy:

Q4_0: Fastest inference, lowest VRAM usage (4-bit quantization)
Q5_K_M: Good balance between quality and performance
Q8_0: Higher quality results but requires more VRAM and slower inference

Memory Requirements

Typical memory requirements for a 7B parameter model:

Full precision (FP16): ~14GB VRAM
Q8_0 quantization: ~7GB VRAM
Q4_0 quantization: ~4GB VRAM

Troubleshooting

For common issues and their solutions, refer to the troubleshooting guide.

Common challenges include:

GPU detection issues
Memory limitations
Network connectivity problems
Model loading failures

Security Considerations

When deploying LLM services like Ollama, consider these important security aspects:

Authentication

The default Ollama API has no built-in authentication. For production use, consider:

Implementing a reverse proxy with authentication
Using OpenShift network policies to restrict access
Creating application-specific API keys

Data Privacy

One of the main advantages of self-hosting LLMs is data privacy:

All prompts and completions remain within your infrastructure
No data is sent to third-party services
Sensitive information can be processed safely

However, remember that the model itself may contain biases or potentially generate problematic content. Implement appropriate guardrails for your specific use case.

Resource Isolation

Use OpenShift’s resource management features to ensure your Ollama deployment:

Has appropriate resource limits to prevent cluster resource exhaustion
Is isolated from other critical workloads
Has priority classes set according to your needs

Setting up a Virtual Workstation in OpenShift with VFIO Passthrough

Mon, 27 Feb 2023 00:00:00

Feb 27, 2023

25 min read

This guide explains how to configure OpenShift as a workstation with GPU PCI passthrough using Container Native Virtualization (CNV) on a single OpenShift node (SNO). This setup delivers near-native performance for GPU-intensive applications while leveraging Kubernetes orchestration capabilities.

Key Benefits:

Run containerized workloads and virtual machines on the same hardware
Use a single GPU for both Kubernetes pods and VMs by switching driver binding
Achieve near-native performance for gaming and professional applications in VMs
Maintain Kubernetes/OpenShift flexibility for other workloads

In testing, this configuration successfully ran Microsoft Flight Simulator in a Windows VM with performance comparable to a bare metal Windows installation.

Hardware Used:

Component	Specification
CPU	AMD Ryzen 9 3950X (16-Core, 32-Threads)
Memory	64GB DDR4 3200MHz
GPU	Nvidia RTX 3080 FE 10GB
Storage	2x 2TB NVMe Disks (VM storage) 1x 500GB SSD Disk (OpenShift root system)
Network	10Gbase-CX4 Mellanox Ethernet

Similar configurations with Intel CPUs should work with minor adjustments noted throughout this guide.

Installing OpenShift SNO

Warning

The OpenShift assisted installer formats the first 512 bytes of any disk with a bootable partition. Back up and remove any existing partition tables you want to preserve. See the assisted-service source code for details.

OpenShift Installation

Note

You can use the OpenShift web UI installer in Red Hat Hybrid Cloud Console. This guides you through installation with the Assisted Installer service:

https://console.redhat.com/openshift/assisted-installer/clusters

This also provides an automated way to install multiple Operators from Day 0.

Relevant Operators for this setup:

Logical Volume Manager Storage
NMState
Node Feature Discovery
NVIDIA GPU
OpenShift Virtualization

After backing up existing file systems and removing bootable partitions, proceed with the OpenShift Single Node installation.

CoreOS requires an entire disk for installation:

500GB SSD for the OpenShift operating system
Two 2TB NVMe disks for persistent volumes as LVM Physical volumes in the same Volume Group
This setup enables flexible VM storage management while keeping the system installation separate

#!/bin/bash

OCP_VERSION=latest-4.10

curl -k https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux.tar.gz > oc.tar.gz
tar zxf oc.tar.gz
chmod +x oc && mv oc ~/.local/bin/

curl -k https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$OCP_VERSION/openshift-install-linux.tar.gz > openshift-install-linux.tar.gz
tar zxvf openshift-install-linux.tar.gz
chmod +x openshift-install && mv openshift-install ~/.local/bin/

curl $(openshift-install coreos print-stream-json | grep location | grep x86_64 | grep iso | cut -d\" -f4) > rhcos-live.x86_64.iso

install-config.yaml

# This file contains the configuration for an OpenShift cluster installation.

apiVersion: v1

# The base domain for the cluster.
baseDomain: epheo.eu

# Configuration for the compute nodes.
compute:
- name: worker
  replicas: 0 

# Configuration for the control plane nodes.
controlPlane:
  name: master
  replicas: 1 

# Metadata for the cluster.
metadata:
  name: da2

# Networking configuration for the cluster.
networking:
  networkType: OVNKubernetes
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16

# Platform configuration for the cluster.
platform:
  none: {}

# Configuration for bootstrapping the cluster.
bootstrapInPlace:
  installationDisk: /dev/sda

# Pull secret for accessing the OpenShift registry.
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"XXXXXXXX"}}}' 

# SSH key for accessing the cluster nodes.
sshKey: |
  ssh-rsa AAAAB3XXXXXXXXXXXXXXXXXXXXXXXXX

Generate OpenShift Container Platform assets

mkdir ocp && cp install-config.yaml ocp
openshift-install --dir=ocp create single-node-ignition-config

Embed the ignition data into the RHCOS ISO:

alias coreos-installer='podman run --privileged --rm \
      -v /dev:/dev -v /run/udev:/run/udev -v $PWD:/data \
      -w /data quay.io/coreos/coreos-installer:release'
cp ocp/bootstrap-in-place-for-live-iso.ign iso.ign
coreos-installer iso ignition embed -fi iso.ign rhcos-live.x86_64.iso
dd if=discovery_image_sno.iso of=/dev/usbkey status=progress

After copying the ISO to a USB drive, boot your workstation from it to install OpenShift.

Installing CNV Operator

Enable Intel VT or AMD-V hardware virtualization extensions in your BIOS/UEFI settings.

cnv-resources.yaml

# This YAML file contains Kubernetes resources for installing the KubeVirt Hyperconverged Operator (HCO) on the OpenShift Container Platform.
# It creates a namespace named "openshift-cnv", an operator group named "kubevirt-hyperconverged-group" in the "openshift-cnv" namespace, and a subscription named "hco-operatorhub" in the "openshift-cnv" namespace.
# The subscription specifies the source, source namespace, name, starting CSV, and channel for the KubeVirt Hyperconverged Operator.

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-cnv
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: kubevirt-hyperconverged-group
  namespace: openshift-cnv
spec:
  targetNamespaces:
    - openshift-cnv
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.10.0
  channel: "stable"

oc apply -f cnv-resources.yaml

Installing Virtctl client on your desktop

subscription-manager repos --enable cnv-4.10-for-rhel-8-x86_64-rpms
dnf install kubevirt-virtctl

Configuring OpenShift for GPU Passthrough

Since we’re working with a single GPU, additional configuration is required.

We’ll use MachineConfig to configure our node. In a single-node OpenShift setup, all MachineConfig changes apply to the master machineset. In multi-node clusters, these would apply to workers instead.

Setting Kernel Boot Arguments

To enable GPU passthrough, we need to pass several kernel arguments at boot time via the MachineConfigOperator:

amd_iommu=on: Enables IOMMU support for AMD platforms (use intel_iommu=on for Intel CPUs)
vga=off: Disables VGA console output during boot
rdblaclist=nouveau: Blacklists the Nouveau open-source NVIDIA driver
video=efifb:off: Disables EFI framebuffer console output

Setting Kernel Arguments at boot time.

variant: openshift
version: 4.10.0
metadata:
  name: 100-vfio
  labels:
    machineconfiguration.openshift.io/role: master
openshift:
  kernel_arguments:
    - amd_iommu=on
    - vga=off
    - rdblaclist=nouveau
    - 'video=efifb:off'

cd articles/openshift-workstation/machineconfig/build
butane -d . vfio-prepare.bu -o ../vfio-prepare.yaml
oc patch MachineConfig 100-vfio --type=merge -p ../vfio-prepare.yaml

Note

Intel CPU users: use intel_iommu=on instead of amd_iommu=on.

Installing the NVIDIA GPU Operator

The NVIDIA GPU Operator simplifies GPU management in Kubernetes environments.

Step 1: Install the Operator

Via OpenShift web console: 1. Go to Operators → OperatorHub 2. Search for “NVIDIA GPU Operator” 3. Select the operator and click Install 4. Keep default settings and click Install

Or via CLI:

oc create -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/master/deployments/git/operator-namespace.yaml
oc create -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/master/deployments/git/operator-source.yaml

Step 2: Configure the ClusterPolicy

Set sandboxWorkloads.enabled to true to enable the components needed for GPU passthrough:

sandboxWorkloadsEnabled.yaml

kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  sandboxWorkloads:
    defaultWorkload: container
    enabled: true

oc patch ClusterPolicy gpu-cluster-policy --type=merge -p sandboxWorkloadsEnabled.yaml

The NVIDIA GPU Operator doesn’t officially support consumer-grade GPUs and won’t automatically bind the GPU audio device to the vfio-pci driver. We’ll handle this manually with the following machine config:

vfio-prepare.bu

variant: openshift
version: 4.10.0
metadata:
  name: 100-vfio
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  files:
  - path: /usr/local/bin/vfio-prepare
    mode: 0755
    overwrite: true
    contents:
      local: ./vfio-prepare.sh
  - path: /etc/modules-load.d/vfio-pci.conf
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci
systemd:
  units:
    - name: vfioprepare.service
      enabled: true
      contents: |
       [Unit]
       Description=Prepare vfio devices
       After=ignition-firstboot-complete.service
       Before=kubelet.service crio.service

       [Service]
       Type=oneshot
       ExecStart=/usr/local/bin/vfio-prepare

       [Install]
       WantedBy=kubelet.service

vfio-prepare.sh

#!/bin/bash

vfio_attach () {
  if [ -f "${path}/driver/unbind" ]; then
    echo $address > ${path}/driver/unbind
  fi
  echo vfio-pci > ${path}/driver_override
  echo $address > /sys/bus/pci/drivers/vfio-pci/bind || \
  echo $name > /sys/bus/pci/drivers/vfio-pci/new_id ||true
}

# 0a:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
address=0000:0a:00.1
path=/sys/bus/pci/devices/0000\:0a\:00.1
name="10de 1467"
vfio_attach

cd articles/openshift-workstation/machineconfig/build
butane -d . vfio-prepare.bu -o ../vfio-prepare.yaml
oc patch MachineConfig 100-vfio --type=merge -p ../vfio-prepare.yaml

Dynamically Switching GPU Drivers

A key advantage of this setup is using a single GPU for both container workloads and VMs without rebooting.

Use Case Scenario

Single NVIDIA GPU shared between container workloads and VMs
Container workloads require the NVIDIA kernel driver
VMs with GPU passthrough require the VFIO-PCI driver
Switching between modes without rebooting

Driver Switching Using Node Labels

The NVIDIA GPU Operator with sandbox workloads enabled lets you switch driver bindings using node labels:

For container workloads (NVIDIA driver):

# Replace 'da2' with your node name
oc label node da2 --overwrite nvidia.com/gpu.workload.config=container

For VM passthrough (VFIO-PCI driver):

# Replace 'da2' with your node name
oc label node da2 --overwrite nvidia.com/gpu.workload.config=vm-passthrough

Notes on Driver Switching

Driver switching takes a few minutes
Verify current driver with lspci -nnk | grep -A3 NVIDIA
Stop all GPU workloads before switching drivers
No reboot is usually required
Can be occasionally unreliable and may require a reboot

Adding GPU as a Hardware Device

First, identify the GPU’s Vendor and Product ID:

lspci -nnk |grep VGA

Then, identify the device name provided by gpu-feature-discovery:

oc get nodes da2 -ojson |jq .status.capacity |grep nvidia

Now, add the GPU to the permitted host devices:

kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  permittedHostDevices:
    pciHostDevices:
    - externalResourceProvider: true
      pciDeviceSelector: 10DE:2206
      resourceName: nvidia.com/GA102_GEFORCE_RTX_3080

oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type=merge -f hyperconverged.yaml

The pciDeviceSelector specifies the vendor:device ID, while resourceName specifies the resource name in Kubernetes/OpenShift.

Passthrough USB Controllers to VMs

For a complete desktop experience, you’ll want to pass through an entire USB controller to your VM for better performance and flexibility.

Identifying a Suitable USB Controller

List all USB controllers:

lspci -nnk | grep -i usb

Example output:

0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

Note the PCI address (e.g., 0b:00.3) and device ID (1022:149c).

Check the IOMMU group:

find /sys/kernel/iommu_groups/ -iname "*0b:00.3*"
# Shows which IOMMU group contains this device

ls /sys/kernel/iommu_groups/27/devices/
# Lists all devices in the same IOMMU group

Important: For clean passthrough, the USB controller should ideally be alone in its IOMMU group. If other devices share the group, you’ll need to pass those through as well.

Adding the USB Controller as a Permitted Device

Add the controller’s Vendor and Product IDs to permitted host devices:

kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  permittedHostDevices:
    pciHostDevices:
      - pciDeviceSelector: 1022:149C
        resourceName: devices.kubevirt.io/USB3_Controller
      - pciDeviceSelector: 8086:2723
        resourceName: intel.com/WIFI_Controller

oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type=merge -f hyperconverged.yaml

Binding the USB Controller to VFIO-PCI Driver

vfio-prepare.bu

variant: openshift
version: 4.10.0
metadata:
  name: 100-vfio
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  files:
  - path: /usr/local/bin/vfio-prepare
    mode: 0755
    overwrite: true
    contents:
      local: ./vfio-prepare.sh
  - path: /etc/modules-load.d/vfio-pci.conf
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci
  - path: /etc/modprobe.d/vfio.conf
    mode: 0644
    overwrite: true
    contents:
      inline: |
        options vfio-pci ids=8086:2723,1022:149c
systemd:
  units:
    - name: vfioprepare.service
      enabled: true
      contents: |
       [Unit]
       Description=Prepare vfio devices
       After=ignition-firstboot-complete.service
       Before=kubelet.service crio.service

       [Service]
       Type=oneshot
       ExecStart=/usr/local/bin/vfio-prepare

       [Install]
       WantedBy=kubelet.service
openshift:
  kernel_arguments:
    - amd_iommu=on
    - vga=off
    - rdblaclist=nouveau
    - 'video=efifb:off'

Create a script to unbind the USB controller from its current driver and bind it to vfio-pci:

vfio-prepare.sh

#!/bin/bash

vfio_attach () {
  if [ -f "${path}/driver/unbind" ]; then
    echo $address > ${path}/driver/unbind
  fi
  echo vfio-pci > ${path}/driver_override
  echo $address > /sys/bus/pci/drivers/vfio-pci/bind || \
  echo $name > /sys/bus/pci/drivers/vfio-pci/new_id ||true
}

# 0a:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
address=0000:0a:00.1
path=/sys/bus/pci/devices/0000\:0a\:00.1
name="10de 1467"
vfio_attach

# Bind "useless" device to vfio-pci to satisfy IOMMU group
address=0000:07:00.0
path=/sys/bus/pci/devices/0000\:07\:00.0
name="1043 87c0"
vfio_attach

# Unbind USB switch and handle via vfio-pci kernel driver
address=0000:07:00.1
path=/sys/bus/pci/devices/0000\:07\:00.1
name="1043 87c0"
vfio_attach

# Unbind USB switch and handle via vfio-pci kernel driver
address=0000:07:00.3
path=/sys/bus/pci/devices/0000\:07\:00.3
name="1022 149c"
vfio_attach

# Unbind USB switch and handle via vfio-pci kernel driver
address=0000:0c:00.3
path=/sys/bus/pci/devices/0000\:0c\:00.3
name="1022 148c"
vfio_attach

cd articles/openshift-workstation/machineconfig/build
butane -d . vfio-prepare.bu -o ../vfio-prepare.yaml
oc patch MachineConfig 100-vfio --type=merge -p ../vfio-prepare.yaml

Creating VMs with GPU Passthrough

This section explains how to create VMs that can use GPU passthrough, using existing LVM Logical Volumes with UEFI boot.

Creating Persistent Volumes from LVM Disks

First, make LVM volumes available to OpenShift via Persistent Volume Claims (PVCs). This assumes you have the Local Storage Operator installed.

Create a YAML file for each VM disk:

fedora_pvc.yaml

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: fedora35
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 100Gi 
  storageClassName: lvms-vg1

Apply the YAML:

oc apply -f fedora35.yaml

Verify the PV and PVC are bound:

oc get pv
oc get pvc -n <your-namespace>

Defining VMs with GPU Passthrough

Key configuration elements for desktop VMs with GPU passthrough:

GPU Passthrough: Pass the entire physical GPU to the VM

See also

https://kubevirt.io/user-guide/virtual_machines/host-devices/#pci-passthrough
Disable Virtual VGA: Remove the emulated VGA device

See also

https://kubevirt.io/api-reference/master/definitions.html#_v1_devices
USB Controller Passthrough: For connecting peripherals directly
UEFI Boot: For compatibility with modern OSes and GPU drivers
CPU/Memory Configuration: Based on workload requirements

fedora.yaml

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: fedora
  namespace: epheo
spec:
  runStrategy: Halted
  template:
    metadata:
      labels:
        kubevirt.io/domain: fedora
    spec:
      architecture: amd64
      domain:
        cpu:
          cores: 8
          model: host-passthrough
          sockets: 2
          threads: 1
        features:
          acpi: {}
          smm:
            enabled: true 
        firmware:
          bootloader:
            efi:
              secureBoot: false # For Nvidia Driver...
        devices:
          disks:
            - bootOrder: 1
              disk:
                bus: virtio
              name: pvdisk
            - disk:
                bus: virtio
              name: cloudinitdisk
          autoattachGraphicsDevice: false
          gpus:
          - deviceName: nvidia.com/GA102_GEFORCE_RTX_3080
            name: gpuvideo
          hostDevices:
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller2
          - deviceName: intel.com/WIFI_Controller
            name: wificontroller
          interfaces:
          - masquerade: {}
            name: default
          - bridge: {}
            model: virtio
            name: nic-0
          networkInterfaceMultiqueue: true
          rng: {}
        machine:
          type: q35
        resources:
          requests:
            memory: 16G
      hostname: fedora
      networks:
      - name: default
        pod: {}
      - multus:
          networkName: br1
        name: nic-0
      terminationGracePeriodSeconds: 0
      volumes:
        - persistentVolumeClaim:
            claimName: 'fedora35'
          name: pvdisk
        - cloudInitNoCloud:
            userData: |-
              #cloud-config
              password: fedora
              chpasswd: { expire: False }
          name: cloudinitdisk

windows.yaml

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    vm.kubevirt.io/os: windows10
    vm.kubevirt.io/workload: desktop
  name: windows
spec:
  runStrategy: Manual
  template:
    metadata:
      labels:
        kubevirt.io/domain: windows
    spec:
      architecture: amd64
      domain:
        clock:
          timer:
            hpet:
              present: false
            hyperv: {}
            pit:
              tickPolicy: delay
            rtc:
              tickPolicy: catchup
          utc: {}
        cpu:
          cores: 8
          dedicatedCpuPlacement: true
          sockets: 2
          threads: 1
        devices:
          autoattachGraphicsDevice: false
          disks:
          - cdrom:
              bus: sata
            name: windows-guest-tools
          - bootOrder: 1
            disk:
              bus: virtio
            name: pvdisk
          - disk:
              bus: virtio
            name: pvdisk1
          gpus:
          - deviceName: nvidia.com/GA102_GEFORCE_RTX_3080
            name: gpuvideo
          hostDevices:
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller
          - deviceName: devices.kubevirt.io/USB3_Controller
            name: usbcontroller2
          - deviceName: intel.com/WIFI_Controller
            name: wificontroller
          interfaces:
          - bridge: {}
            model: virtio
            name: nic-0
          networkInterfaceMultiqueue: true
          rng: {}
          tpm: {}
        features:
          acpi: {}
          apic: {}
          hyperv:
            frequencies: {}
            ipi: {}
            reenlightenment: {}
            relaxed: {}
            reset: {}
            runtime: {}
            spinlocks:
              spinlocks: 8191
            synic: {}
            synictimer:
              direct: {}
            tlbflush: {}
            vapic: {}
            vpindex: {}
          smm: {}
        firmware:
          bootloader:
            efi:
              secureBoot: true
        machine:
          type: q35
        memory:
          hugepages:
            pageSize: 1Gi
        resources:
          requests:
            memory: 32Gi
      evictionStrategy: None
      hostname: windows
      networks:
      - multus:
          networkName: br1
        name: nic-0
      terminationGracePeriodSeconds: 3600
      volumes:
      - containerDisk:
          image: registry.redhat.io/container-native-virtualization/virtio-win-rhel9@sha256:0c536c7aba76eb9c1e75a8f2dc2bbfa017e90314d55b242599ea41f42ba4434f
        name: windows-guest-tools
      - name: pvdisk
        persistentVolumeClaim:
          claimName: windows
      - name: pvdisk1
        persistentVolumeClaim:
          claimName: windowsdata

Future Improvements

Some potential future improvements to this setup:

Using MicroShift instead of OpenShift to reduce Control Plane footprint
Running Linux Desktop in containers instead of VMs
Implementing more efficient resource allocation with CPU pinning and huge pages

Troubleshooting

IOMMU Group Issues

Problem: VM fails to start with:

{"component":"virt-launcher","level":"error","msg":"Failed to start VirtualMachineInstance",
"reason":"virError... vfio 0000:07:00.1: group 19 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver."}

Diagnosis: Not all devices in the IOMMU group are bound to vfio-pci. Check:

# Check devices in the IOMMU group
ls /sys/kernel/iommu_groups/19/devices/

# Check what these devices are
lspci -nnks 07:00.0

Solution: Bind all devices in the IOMMU group to vfio-pci:

# Add to vfio-prepare.sh
echo "vfio-pci" > /sys/bus/pci/devices/0000:03:08.0/driver_override
echo "vfio-pci" > /sys/bus/pci/devices/0000:07:00.0/driver_override
echo "vfio-pci" > /sys/bus/pci/devices/0000:07:00.1/driver_override
echo "vfio-pci" > /sys/bus/pci/devices/0000:07:00.3/driver_override

# Unbind from current drivers, then bind to vfio-pci

Common Issues and Solutions

No display output after GPU passthrough:

Disable virtual VGA in VM spec
Pass through both GPU and audio device
Install proper GPU drivers inside VM

Performance issues in Windows VM:

Configure CPU pinning correctly
Enable huge pages for better memory performance
Install latest NVIDIA drivers in VM
Disable Windows Game Bar and overlay software

GPU driver switching fails:

Stop all GPU workloads before switching
Check GPU operator logs: oc logs -n nvidia-gpu-operator <pod-name>
Verify IOMMU is enabled in BIOS/UEFI

For further troubleshooting, check logs:

virt-handler: oc logs -n openshift-cnv virt-handler-<hash>
virt-launcher: oc logs -n <namespace> virt-launcher-<vm-name>-<hash>
nvidia-driver-daemonset: oc logs -n nvidia-gpu-operator nvidia-driver-daemonset-<hash>

blog.epheo.eu

Debugging PostgreSQL Crash Loop in OpenShift

Introduction

Understanding the Error

Step-by-Step Solution

Why This Works

Running vLLM on Strix Halo

Prerequisites

Installing ROCm via TheRock nightly

Environment setup

Installing PyTorch from ROCm nightly

Building flash-attention (ROCm fork)

Patching & building vLLM

bitsandbytes (ROCm fork)

How this blog is served to you

Details

Merging kubeconfig Files

Prerequisites

Step-by-Step Merge Process

1. Backup Existing Configuration

2. Set the KUBECONFIG Environment Variable

3. Merge the Files

4. Replace the Original Configuration

5. Verify the Merged Configuration

Listing Available Contexts

Checking the Current Context

Switching Between Contexts

OpenShift Virtualization Sidecar Implementation Guide

Overview

Enabling the Sidecar Feature Gate

Verify Feature Gate is Enabled

Implementing a Sidecar Container

KubeVirt Hook Types

Sidecar Script Arguments

Example Hook ConfigMap

Using Sidecars with VMs

Validating the Sidecar Modifications

Method 1: Check VM XML from virt-launcher Pod

Method 2: From Inside the VM

References

OpenShift Console Banner

Setting up a Console Banner

Banner Customization

Example Colors

Common Use Cases

Applying the Configuration

Simple Block Device Backup for OpenShift Lab Environments

Introduction

Overview & Prerequisites

Step 1: Prepare Your Environment

Container Image

Resource Planning

Storage Architecture

Step 2: Configure and Run Backups

Step 3: Monitor and Manage Your Backups

Checking Backup Status

Examining Backup Logs

Managing Backup Archives

Step 4: Restore to a Local Volume

Step 5: Cross-Platform Restoration Process

Prerequisites

Step 5.1: Create a Reference to the Existing Backup Volume

Step 5.2: Create a Claim for the Backup Volume

Step 5.3: Create a Restore Pod

Step 6: Verify the Restoration

Step 7: Directly Using Existing Block Devices with Kubernetes

Identifying Available Block Devices

Direct PersistentVolume Approach

Creating a PersistentVolumeClaim

Using the Block Device in a Pod

Verification

Resources and References

OpenStack OVN Networking - Deep Dive and Debugging

Locating OVN Services

Executing OVN Commands in the Cluster

Remote OVN Database Access

Understanding OVN Database Structure

Northbound Database - Logical Network Elements

Southbound Database - Physical Mapping

Tracing Security Group Rules Through OVN