Layer 2 User Defined Networks for OpenShift Virtualization¶

April 22, 2025

20 min read

Configure and implement Layer 2 User Defined Networks (UDN) for Virtual Machines in OpenShift Virtualization to create isolated tenant networks with multi-node connectivity.

This use case demonstrates how to implement User Defined Networks (UDN) in OpenShift Virtualization, allowing for network isolation between Virtual Machines within a namespace or across namespaces. User Defined Networks provide a way to create isolated tenant networks for VMs, separate from the default pod network, enabling more complex network topologies and improved security isolation.

Implementation Steps¶

1. Understand User Defined Networks¶

OpenShift Virtualization supports two types of User Defined Networks:

UserDefinedNetwork (UDN): Provides network isolation within a single namespace.
ClusterUserDefinedNetwork (Cluster UDN): Spans multiple namespaces, allowing VMs in different namespaces to communicate with each other while maintaining isolation from other networks.

Layer 2 vs Layer 3 User Defined Networks¶

User Defined Networks can be configured with either Layer 2 or Layer 3 topology:

Layer 2 UDN

Creates a virtual switch distributed across all nodes in the cluster
All VMs connect to the same subnet regardless of which node they’re on
Provides a single broadcast domain for all connected VMs
Supports live migration of VMs between nodes
Ideal for applications requiring Layer 2 adjacency or broadcast capabilities
Simpler to configure and manage for basic network isolation

Layer 3 UDN

Creates a unique Layer 2 segment per node with Layer 3 routing between segments
Each node has its own subnet, with routing providing connectivity between nodes
Better manages broadcast traffic by containing it within node-specific segments
Provides improved scalability for larger deployments
Supports NetworkPolicy for enhanced security control
Recommended for environments with many VMs where broadcast traffic might be problematic

Primary vs Secondary Networks

Primary networks handle all default traffic for a pod/VM unless otherwise specified
Secondary networks provide additional interfaces for specific traffic types
A pod/VM can have only one primary network but multiple secondary networks
UDNs can function as either primary or secondary networks

Important

Compatibility Note:

For Virtual Machines, only Layer 2 UDNs are supported as primary networks
Layer 3 UDNs are NOT supported for virtual machines and they can only be used with regular pods, not VMs

This guide focuses on Layer 2 UDNs which are commonly used for VM workloads requiring simplicity and migration capabilities.

2. Create a Namespace for User Defined Network¶

To use a User Defined Network, you must create a namespace with a specific label that enables UDN functionality:

udn-namespace.yaml¶

apiVersion: v1
kind: Namespace
metadata:
  name: udn-example
  labels:
    k8s.ovn.org/primary-user-defined-network: ""

Important

The label must be applied when the namespace is created. It cannot be added to an existing namespace.

Apply the namespace configuration:

oc create -f udn-namespace.yaml

3. Create a User Defined Network¶

Create a User Defined Network in the namespace:

udn-example.yaml¶

apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  name: udn-example
  namespace: udn-example
spec:
  layer2:
    ipam:
      lifecycle: Persistent
    role: Primary
    subnets:
    - 10.200.0.0/16
  topology: Layer2

Apply the UDN configuration:

oc create -f udn-example.yaml

Verify that the UDN was created successfully:

oc get userdefinednetwork -n udn-example

4. Create a VM on the User Defined Network¶

When creating a VM in a namespace with a User Defined Network, the VM will automatically use the UDN as its primary network. The VM should be created with the default network configuration.

Example VM manifest:

example-vm.yaml¶

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
  namespace: udn-example
spec:
  runStrategy: Always
  template:
    spec:
      domain:
        devices:
          disks:
          - name: rootdisk
            disk:
              bus: virtio
          interfaces:
          - name: default
            binding:
              name: l2bridge
        resources:
          requests:
            memory: 2Gi
      networks:
      - name: default
        pod: {}
      volumes:
      - name: rootdisk
        containerDisk:
          image: quay.io/containerdisks/fedora:latest

Note

Do not modify the network configuration for the VM. The network configuration is automatically handled by the UDN system.

Important

After attaching a VM to a network, you must restart or live migrate the VM for the network changes to take effect:

# To restart the VM:
virtctl restart example-vm

# To live migrate instead (avoids downtime):
virtctl migrate example-vm

5. Create a Cluster User Defined Network¶

For communications across multiple namespaces, you can create a Cluster User Defined Network:

Create namespaces with appropriate labels:

udn-namespaces.yaml¶

---
apiVersion: v1
kind: Namespace
metadata:
  name: udn-prod
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
    cluster-udn: prod
---
apiVersion: v1
kind: Namespace
metadata:
  name: udn-preprod
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
    cluster-udn: prod

Create the Cluster User Defined Network:

cluster-udn-prod.yaml¶

apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  name: cluster-udn-prod
spec:
  namespaceSelector:
    matchLabels:
      cluster-udn: prod
  network:
    layer2:
      ipam:
        lifecycle: Persistent
      role: Primary
      subnets:
      - 10.100.0.0/16
    topology: Layer2

Apply the Cluster UDN configuration:
```
oc create -f cluster-udn-prod.yaml
```
Verify the Cluster UDN creation:
```
oc get clusteruserdefinednetwork
```

Testing and Validation¶

1. Test VM Connectivity within UDN¶

Create multiple VMs in the same namespace with a UDN

Connect to the VM console or SSH:

# Connect to VM console
virtctl console example-vm

# Or use SSH if configured
virtctl ssh example-vm

Verify network configuration in each VM:
```
# From inside VM
ip addr show
ip route
```
Test connectivity between VMs in the same namespace:
```
# From inside VM
ping <other-vm-ip-address>
```
Verify that the VMs have received IP addresses from the UDN subnet

2. Test VM Connectivity across Cluster UDN¶

Create VMs in different namespaces connected by a Cluster UDN
Verify network configuration in each VM

Test connectivity between VMs in different namespaces:

# From inside VM in namespace 1
ping <vm-ip-in-namespace-2>

3. Test North/South Network Access¶

Verify that VMs can access external networks:
```
# From inside VM
ping 1.1.1.1
```

Important

Even though the UDN is a Layer 2 network, don’t manually configure IP addresses as UDN controller manages IP address allocation through its IPAM functionality.