Terraform Setup to Test Cilium on Talos

A Terraform module to manage a Talos-based Kubernetes on AWS (EC2 instances). Is uses the official Talos Terraform provider in the background. We mainly followed the provided siderolabs/contrib example.

Supported Features

• Install Talos Linux OS EC2 VMs
  • Due to Talos' setup requirements, it's only supported to deploy the EC2 instances in public subnets with public IPs assigned (access can be restricted via security groups)
  • AMD64 and ARM64 are supported
• Support for single- and multi-node cluster architectures
• Bootstrap Talos Kubernetes cluster with some infrastructure components:
  • Talos' KubePrism to get an internal endpoint for the KAPI (used for Cilium Kube-Proxy replacement)
  • kubernetes-sigs/metrics-server
  • alex1989hu/kubelet-serving-cert-approver inspired by Talos' Deploying Metrics Server guide.
  • AWS Cloud Provider
• Cilium features:
  • Kube-Proxy replacement
  • IPAM modes: kubernetes, cluster-pool

Example Usage

// Create a Talos Linux cluster
module "talos" {
  source = "git::https://github.com/isovalent/terraform-aws-talos?ref=<RELEASE_TAG>"

  // Supported Talos versions (and therefore K8s versions) can be found here: https://github.com/siderolabs/talos/releases
  talos_version         = "v1.10.4"
  kubernetes_version    = "1.33.1"
  cluster_name          = "talos-cute"
  region                = "eu-west-1"
  tags                  = local.tags
  // VPC needs to be created in advance via https://github.com/isovalent/terraform-aws-vpc
  vpc_id                = module.vpc.id
  pod_cidr              = "100.64.0.0/14"
  service_cidr          = "100.68.0.0/16"
  # Configure the allowed source CIDR range:
  external_source_cidrs = ["A.B.C.D/E"]
}

Terraform Module Doc

Requirements

Name	Version
terraform	>= 1.4.0
aws	~> 5.100
random	~> 3.7
talos	0.9.0-alpha.0

Providers

Name	Version
aws	~> 5.100
local	n/a
null	n/a
random	~> 3.7
talos	0.9.0-alpha.0

Modules

Name	Source	Version
cluster_sg	terraform-aws-modules/security-group/aws	~> 5.3
nlb_sg	terraform-aws-modules/security-group/aws	~> 5.3
talos_control_plane_nodes	terraform-aws-modules/ec2-instance/aws	~> 5.8
talos_worker_group	terraform-aws-modules/ec2-instance/aws	~> 5.8

Resources

Name	Type
aws_iam_policy.control_plane_ccm_policy	resource
aws_iam_policy.worker_ccm_policy	resource
aws_lb.api	resource
aws_lb_listener.https	resource
aws_lb_listener.talos	resource
aws_lb_target_group.k8s	resource
aws_lb_target_group.talos	resource
aws_lb_target_group_attachment.cp_k8s	resource
aws_lb_target_group_attachment.cp_talos	resource
local_file.kubeconfig	resource
local_file.talosconfig	resource
null_resource.wait_for_subnets	resource
random_string.workspace_id	resource
talos_cluster_kubeconfig.this	resource
talos_machine_bootstrap.this	resource
talos_machine_configuration_apply.controlplane	resource
talos_machine_configuration_apply.worker_group	resource
talos_machine_secrets.this	resource
aws_ami.talos	data source
aws_region.current	data source
aws_subnets.private	data source
aws_subnets.public	data source
aws_vpc.vpc	data source
talos_client_configuration.this	data source
talos_machine_configuration.controlplane	data source
talos_machine_configuration.worker_group	data source

Inputs

Name	Description	Type	Default	Required
admission_plugins	List of admission plugins to enable	string	"MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ServiceAccount"	no
allocate_node_cidrs	Whether to assign PodCIDRs to Node resources or not. Only needed in case Cilium runs in 'kubernetes' IPAM mode.	bool	true	no
allow_workload_on_cp_nodes	Allow workloads on CP nodes or not. Allowing it means Talos Linux default taints are removed from CP nodes which is typically required for single-node clusters. More details here: https://www.talos.dev/v1.5/talos-guides/howto/workers-on-controlplane/	bool	false	no
cluster_architecture	Cluster architecture. Choose 'arm64' or 'amd64'. If you choose 'arm64', ensure to also override the control_plane.instance_type and worker_groups.instance_type with an ARM64-based instance type like 'm7g.large'.	string	"amd64"	no
cluster_id	The ID of the cluster.	number	"1"	no
cluster_name	Name of cluster	string	n/a	yes
config_patch_files	Path to talos config path files that applies to all nodes	list(string)	[]	no
control_plane	Info for control plane that will be created	object({ instance_type = optional(string, "m5.large") config_patch_files = optional(list(string), []) tags = optional(map(string), {}) })	{}	no
controlplane_count	Defines how many controlplane nodes are deployed in the cluster.	number	3	no
deploy_external_cloud_provider_iam_policies	Whether to auto-deploy the externalCloudProvider-required IAM policies. See https://cloud-provider-aws.sigs.k8s.io/prerequisites/.	bool	false	no
disable_containerd_nri_plugins	Whether to disable the Talos containerd NRI plugins or not. Talos disables it by default. See https://www.talos.dev/latest/talos-guides/configuration/containerd/#enabling-nri-plugins. Supported since Talos v1.9.2 (see siderolabs/talos#10068).	bool	true	no
disable_kube_proxy	Whether to deploy Kube-Proxy or not. By default, KP shouldn't be deployed.	bool	true	no
enable_external_cloud_provider	Whether to enable or disable externalCloudProvider support. See https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/.	bool	false	no
external_cloud_provider_manifest	externalCloudProvider manifest to be applied if var.enable_external_cloud_provider is enabled. If you want to deploy it manually (e.g., via Helm chart), enable var.enable_external_cloud_provider but set this value to an empty string (""). See https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/.	string	"https://raw.githubusercontent.com/isovalent/terraform-aws-talos/main/manifests/aws-cloud-controller.yaml"	no
external_source_cidrs	Specify the external source CIDRs (use /32 for specific IP addresses) allowed for inbound traffic.	list(string)	n/a	yes
iam_instance_profile_control_plane	IAM instance profile to attach to the control plane instances to give AWS CCM the sufficient rights to execute.	string	null	no
iam_instance_profile_worker	IAM instance profile to attach to the worker instances to give AWS CCM the sufficient rights to execute.	string	null	no
kubernetes_version	Kubernetes version to use for the Talos cluster, if not set, the K8s version shipped with the selected Talos version will be used. Check https://www.talos.dev/latest/introduction/support-matrix/. For example '1.29.3'.	string	""	no
metadata_options	Metadata to attach to the instances.	map(string)	{ "http_endpoint": "enabled", "http_put_response_hop_limit": 1, "http_tokens": "optional" }	no
pod_cidr	The CIDR to use for Pods. Only required in case allocate_node_cidrs is set to 'true'. Otherwise, simply configure it inside Cilium's Helm values.	string	"100.64.0.0/14"	no
region	The region in which to create the Talos Linux cluster.	string	n/a	yes
service_cidr	The CIDR to use for services.	string	"100.68.0.0/16"	no
tags	The set of tags to place on the cluster.	map(string)	n/a	yes
talos_version	Talos version to use for the cluster, if not set, the newest Talos version. Check https://github.com/siderolabs/talos/releases for available releases.	string	"v1.10.4"	no
vpc_cidr	The IPv4 CIDR block for the VPC.	string	"10.0.0.0/16"	no
vpc_id	ID of the VPC where to place the VMs.	string	n/a	yes
worker_groups	List of node worker node groups to create	list(object({ name = string instance_type = optional(string, "m5.large") config_patch_files = optional(list(string), []) tags = optional(map(string), {}) }))	[ { "name": "default" } ]	no
workers_count	Defines how many worker nodes are deployed in the cluster.	number	2	no

Outputs

Name	Description
cluster_name	Name of cluster
elb_dns_name	[DEPRECATED: Use lb_dns_name instead] Public load balancer DNS name.
kubeconfig	Kubeconfig content
lb_arn	The ARN of the Network Load Balancer.
lb_dns_name	Public NLB DNS name.
lb_zone_id	The zone_id of the NLB for Route53 alias records.
path_to_kubeconfig_file	The generated kubeconfig.
path_to_talosconfig_file	The generated talosconfig.