diff --git a/docs/user-guide/host-settings.md b/docs/user-guide/host-settings.md index 9b759cf1..0da003ac 100644 --- a/docs/user-guide/host-settings.md +++ b/docs/user-guide/host-settings.md @@ -216,3 +216,136 @@ kubectl fabric vpc attach --vpc-subnet vpc-2/default --connection server-1--leaf [bonding]: https://www.kernel.org/doc/html/latest/networking/bonding.html +## Layer 3 Multi-Homing + +Hosts can be multi-homed at the network layer by configuring a virtual IP address (VIP) on the host. After the +virtual IP address is configured, services will bind to the VIP and be +available via multiple routes. Vpc-2 is shown in this example as partner in ICMP +traffic. The diagram illustrates the topology used for this setup: + +```mermaid +graph TD + S1([Spine 1]) + S2([Spine 2]) + + L1([Leaf-1]) + L2([Leaf-2]) + L3([Leaf-3]) + H1("`server-1 + 10.0.11.11/32`") + H2("`server-2 + 10.0.20.20/32`") + + S1 & S2 --> L1 & L2 & L3 + + subgraph Vpc-2 + L3 <--> H2 + end + + subgraph Vpc-1 + L1 <--10.0.10.10--> H1 + L2 <--10.0.9.10--> H1 + end + +``` + +### VPC Settings +The VPC needs one subnet per link to the host and an additional subnet to hold +the VIP. The IP addresses used in the following YAML file are examples, and can +be adapted to any environment. The configuration for the VPC holding the VIP is: + +```{yaml annotate title='vip-vpc.yaml' linenums='1'} +apiVersion: vpc.githedgehog.com/v1beta1 +kind: VPC +metadata: + name: vpc-1 + namespace: default +spec: + ipv4Namespace: default + subnets: + link1: + dhcp: + enable: false # optinal, can be true + gateway: 10.0.10.1 + subnet: 10.0.10.0/24 + vlan: 1010 + link2: + dhcp: + enable: false # optinal, can be true + gateway: 10.0.9.1 + subnet: 10.0.9.0/24 + vlan: 1009 + vip: + dhcp: + enable: false # must be false + gateway: 10.0.11.1 + subnet: 10.0.11.0/24 + vlan: 1011 + vlanNamespace: default +``` + +#### Multiple Subnets inside of a VPC +The above listing has three subnets for a single VPC. The subnets are: link1, +link2, and vip. Each subnet has its own VLAN. + +#### Attach Subnets to Connections +The fabric needs to know which connections to attach to `vpc-1`. + +```bash +kubectl fabric vpc attach --vpc-subnet vpc-1/link1 --connection server-1--unbundled--leaf-01 +kubectl fabric vpc attach --vpc-subnet vpc-1/link2 --connection server-1--unbundled--leaf-02 +kubectl fabric vpc attach --vpc-subnet vpc-1/vip --connection server-1--unbundled--leaf-01 --nativeVLAN true +kubectl fabric vpc attach --vpc-subnet vpc-1/vip --connection server-1--unbundled--leaf-02 --nativeVLAN true +``` + +The attachments are done so that routes to the VIP will be distributed via the +fabric to any VPCs that peer with `vpc-1`. + +##### The nativeVLAN setting + +The VIP attachments should not use tagged traffic, ensure to pass `--nativeVLAN +true` to the kubectl command. Alternatively edit the object after its created +and set: `nativeVLAN: true` in the object. + +### Host Settings + +In this example the host has two interfaces to the network. The interfaces are +attached to switch ports that are emitting tagged traffic, and are the default +routes for the host. A dummy interface is used to hold the VIP, but the address +could also be applied to the `lo` device. + +#### Create the Interfaces + +```bash +sudo ip link add link enp2s2 name enp2s2.1010 type vlan id 1010 +sudo ip link add link enp2s1 name enp2s1.1009 type vlan id 1009 +sudo ip link add loop type dummy +``` + +#### IP addresses for interfaces + +```bash +sudo ip addr add 10.0.10.10/24 dev enp2s2.1010 +sudo ip addr add 10.0.9.10/24 dev enp2s2.1009 +sudo ip addr add 10.0.11.11/32 dev loop +``` + +#### Configure the ECMP Default Route + +This command creates two default routes of equal weight. + +```bash +sudo ip route replace default proto static scope global\ + nexthop dev enp2s1.1009 via 10.0.9.1 weight 1\ + nexthop dev enp2s2.1010 via 10.0.10.1 weight 1 +``` + +#### Configure the Sysctl + +Ensure that sysctls that begin with `fib_` are according to the needs of the +environment. The relevant sysctl settings are `fib_` in the [linux kernel +documentation](https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html). +Special attention should be given to `fib_multipath_use_neigh`. If this is set +`0` packets will be emitted out an interface even if down, resulting in lost +packets. +