Libvirt Infrastructure Module

This module deploys a libvirt-based infrastructure. It is designed to set up a complete networking and VM environment on a libvirt host, suitable for deploying a Kubernetes cluster.

Due to a limitation in Terraform's provider initialization, the setup of the libvirt host must be done in a separate step before deploying the actual libvirt resources.

Architecture

This module provisions the following resources on the libvirt host:

• Networking: Creates a public network (defaults: 172.16.0.0/24, fd02::/112) and a private network (defaults: 192.168.1.0/24, fd03::/112).

• VyOS Router: A VM that acts as a router between the public and private networks. It also provides DHCP and DNS services for the private network.

  • Gateway and NAT: Provides internet access for the VMs on the private network.
  • DHCP Server: Assigns static IPv4 addresses to Kubernetes nodes based on their MAC addresses. DHCPv6 is currently disabled.
  • DNS Server (CoreDNS): Provides DNS resolution for the cluster, including records for all nodes and Kubernetes API endpoints.
  • Load Balancer (HAProxy): Acts as a load balancer for the Kubernetes control plane and ingress traffic (ports 80, 443, 6443, and 22623).
  • BGP Peer: Establishes both IPv4 and IPv6 BGP sessions with Kubernetes nodes, enabling advanced networking features required by CNIs like Cilium.

• Testbox VM: A general-purpose Ubuntu-based VM connected to both networks that can be used for administrative or testing tasks.

  To facilitate BGP testing, the testbox VM is automatically configured with static routes that direct traffic for the 10.0.0.0/8 (IPv4) and fd01::/16 (IPv6) supernets to the VyOS router.

  This setup allows you to test end-to-end connectivity to services exposed via Cilium's BGP integration. When you configure Cilium to advertise its LoadBalancer IP ranges (e.g., 10.0.100.0/24) via BGP, the VyOS router learns these routes.

  You can then access your LoadBalancer IPs from the testbox. The traffic flow works as follows:

  1. The testbox sends traffic for the LoadBalancer IP to the VyOS router, following its broad static route.
  2. The VyOS router, having learned a more specific route from Cilium via BGP, forwards the traffic to the correct Kubernetes node.

  This allows you to verify that your BGP configuration is working correctly from an external client's perspective.

Usage

Follow these two steps to deploy the infrastructure:

Step 1: Prepare the Libvirt Host

This step runs a setup script on your libvirt host to install necessary packages and configure networking.

terraform init
terraform apply --target=module.libvirt-infra.null_resource.libvirt_host_setup

Step 2: Deploy the Libvirt Resources

After the host is prepared, you can deploy the full infrastructure, including the router and testbox VMs.

terraform apply

Requirements

• A host with libvirt installed and running.
• SSH access to the libvirt host with a private key.

Accessing the Environment

The setup script configures port forwarding on the libvirt host using iptables. This allows you to access the VMs and services running within the libvirt environment from the host machine's IP address.

SSH Access

• Router: Connect to the VyOS router via SSH. The default port is 8022. Use the password you configured (defaults to R0uter123!).

  ssh -p 8022 vyos@<libvirt_host_ip>

• Testbox: Connect to the testbox via SSH using the SSH key you provided. The default port is 8023 and the default username is testbox.

  ssh -p 8023 testbox@<libvirt_host_ip>

Kubernetes API

Port 6443 is forwarded for Kubernetes API access. Once your cluster is running, you can access its API server through the libvirt host's IP address: https://<libvirt_host_ip>:6443

Requirements

Name	Version
terraform	>=1.6.5
libvirt	0.7.6

Providers

Name	Version
libvirt	0.7.6
local	2.6.1
null	3.2.4

Modules

No modules.

Resources

Name	Type
libvirt_cloudinit_disk.router	resource
libvirt_cloudinit_disk.testbox	resource
libvirt_domain.router	resource
libvirt_domain.testbox	resource
libvirt_network.libvirt_private_network	resource
libvirt_network.libvirt_public_network	resource
libvirt_pool.main	resource
libvirt_volume.router	resource
libvirt_volume.router_base	resource
libvirt_volume.testbox	resource
libvirt_volume.testbox_base	resource
local_file.libvirt_host_setup_script	resource
null_resource.libvirt_host_setup	resource

Inputs

Name	Description	Type	Default	Required
dns_base_domain	base domain for the LAN network so the k8s nodes can get unique the FQDN name and resolved by the router	string	"local"	no
image_download_base_url	the base url of the images	string	n/a	yes
infra_name	The name of the equinix metal infrastructure.	string	n/a	yes
k8s_cluster_name	this will be used to create the FQDN name in the DNS record to follow the OCP guide(https://docs.openshift.com/container-platform/4.11/installing/installing_bare_metal/installing-bare-metal.html#installation-dns-user-infra-example_installing-bare-metal)	string	n/a	yes
k8s_master_count	number of master nodes, and we only support numbers 1 or 3 for now	number	"1"	no
k8s_master_hostname_prefix	prefix hostname of the master nodes	string	"masters"	no
k8s_master_mac_prefix	prefix mac address of the master nodes	string	"52:54:00:aa:bb:a"	no
k8s_worker_count	number of worker nodes, and we only support numbers less than 9 for now	number	"2"	no
k8s_worker_hostname_prefix	prefix hostname of the worker nodes	string	"workers"	no
k8s_worker_mac_prefix	prefix mac address of the worker nodes	string	"52:54:00:aa:bb:b"	no
libvirt_host_ip	The IP address of the libvirt host.	string	n/a	yes
libvirt_root_private_key_path	The root private key path of the libvirt host.	string	n/a	yes
private_network_ipv4_cidr	the private network IPv4 cidr block for VMs, only /24 is supported for now	string	"192.168.1.0/24"	no
private_network_ipv6_cidr	the private network IPv6 cidr block for VMs, and only /112 is supported for now	string	"fd03::/112"	no
public_network_ipv4_cidr	the public network IPv4 cidr block for VMs, only /24 is supported for now	string	"172.16.0.0/24"	no
public_network_ipv6_cidr	the public network IPv6 cidr block for VMs, and only /112 is supported for now	string	"fd02::/112"	no
router_image_name	the name of the router image	string	"vyos-1.4-rolling.qcow2"	no
router_password	The login password for vyos.	string	"R0uter123!"	no
router_public_ssh_port	The public SSH port of the router from external.	number	8022	no
testbox_image_name	the name of the testbox image	string	"jammy-server-cloudimg-amd64-disk-kvm.img"	no
testbox_public_ssh_port	The public SSH port of the testbox from external.	number	8023	no
testbox_ssh_private_key	Content of the SSH private key.	string	n/a	yes
testbox_ssh_public_key	Content of the SSH public key.	string	n/a	yes
testbox_username	the username of the testbox	string	"testbox"	no

Outputs

Name	Description
libvirt_info	Information about the libvirt environment.
masters_info_list	The IP addresses, mac addresses, hostnames, and ipv6 addresses of the master nodes.
router_info	The connection details for the router.
testbox_info	The connection details for the testbox.
workers_info_list	The IP addresses, mac addresses, hostnames, and ipv6 addresses of the worker nodes.