FRI network

Ansible playbooks to configure the FRI network. Network configuration resides in NetBox; an overview of core switches and servers can be found in the topology view.

Setup

Install dependencies with pip install --user -r requirements.txt or with the package manager. Since querying the API is not very fast, it is helpful to setup Ansible cache, for example by adding the following to ~/.profile or similar:

export ANSIBLE_INVENTORY_CACHE=True
export ANSIBLE_INVENTORY_CACHE_PLUGIN=jsonfile
export ANSIBLE_CACHE_PLUGIN_CONNECTION=~/.ansible/cache

Devices are accessible on a separate network, reachable through a WireGuard tunnel. For device access an SSH key is required, with the public key authorized for root on each device.

Usage

Create a read-only token in NetBox. Set variables required to access NetBox:

# one for nb_inventory and one for nb_lookup
export NETBOX_API_KEY=<token>
export NETBOX_TOKEN="${NETBOX_API_KEY}"
# same for both
export NETBOX_API=<netbox API endpoint>

Run one-off tasks with (add --key-file or other options as necessary):

ansible -i inventory.yml -m ping 'spine-*'

Run a playbook with:

ansible-playbook setup.yml -i inventory.yml -l 'spine-*'

NetBox data

The following values are used throughout the network and should be defined in a site-wide config context:

• dhcp: DHCP server address
• dns: list of DNS server IPv4 addresses
• dns6: list of DNS server IPv6 addresses
• domain: site domain
• nat: list of IPv4 ranges used for SNAT and DNAT
• ntp: list of NTP server addresses
• wg_ip: public IPv4 address for wireguard connections, anycast between firewall nodes
• wg_net: client wireguard IPv4 addresses are assigned from this range
• wg_net6: client wireguard IPv6 addresses are assigned from this range

Device‐ and network‐specific configuration is described in the following sections. All JSON samples in this document are subsets of the inventory as returned by Ansible. Omitted values should be set to null or empty unless stated otherwise (except for foreign keys such as type and role, where some values are omitted for brevity).

Network

Each VLAN is EVPNed among all leaves, so it can be used with any port on any access switch. All non‐local traffic leaves and enters a VLAN through exit leaves and optionally the firewall. Each firewalled or “inside” VLAN is placed in a separate VRF on the exits, so inter-VLAN traffic is also filtered. For example:

{
    "vid": 3208, "name": "office-rc",
    "group": { "name": "new-net" }, "role": { "name": "inside" },
    "tenant": { "slug": "fri-it" }
},
{
    "vid": 3308, "name": "server-rc",
    "group": { "name": "new-net" }, "role": { "name": "outside" },
    "tenant": { "slug": "fri-it" }
}

The group property is used to filter out legacy VLANs and should match the value in group_vars/all/main.yml. Set the role to inside or outside to place the network behind a firewall or not.

Associate one or more IPv4 and IPv6 prefixes to each VLAN. Assign the role dhcp-pool to an IPv4 prefix to enable a DHCP relay on the exits for that subnet. For example:

{
    "prefix": "2001:1470:fffd:3208::/64", "family": { "value": 6 },
    "tenant": { "slug": "fri-it" },
    "vlan": { "name": "office-rc" }
},
{
    "prefix": "10.32.8.0/24", "family": { "value": 4 },
    "tenant": { "slug": "fri-it" },
    "vlan": { "name": "office-rc" }
    "role": { "slug": "dhcp-pool" },
}

Devices

For most devices a management interface must be defined to run Ansible scripts, with at least the IP address and default gateway set:

{
    "name": "eth0", "type": { "value": "1000base-t" },
    "mgmt_only": true,
    "mac_address": "98:03:9B:9C:2D:10",
    "ip_addresses": [ { "address": "10.20.30.40/24" } ],
    "custom_fields": { "gateway": { "address": "10.20.30.1/24" } }
}

The MAC address is only used in some playbooks to set the interface name.

L1 setup

To break out a port, create the appropriately named interfaces and disable the original interface:

{ "name": "swp14", "enabled": false },
{ "name": "swp14s0", "type": { "value": "25gbase-x-sfp28" } },
{ "name": "swp14s1", "type": { "value": "25gbase-x-sfp28" } },
{ "name": "swp14s2", "type": { "value": "25gbase-x-sfp28" } },
{ "name": "swp14s3", "type": { "value": "25gbase-x-sfp28" } }

Note that for SN2700 switches only odd‐numbered ports may be broken out; the next even‐numbered port must be disabled as well as the original port in this case. The new ports can be used normally in further configuration.

L3 setup

For L3 devices the asn custom field must be set. For the fabric and core servers we use private ASNs above 65000.

Each L3 node should define IPv4 and IPv6 addresses on the loopback interface. These are displayed e.g. by traceroute. The IPv4 loopback address is also used as the BGP router ID. For MLAG switches specify the same VXLAN anycast IP on both peers with the anycast role.

{
    "name": "lo", "type": { "value": "virtual" },
    "ip_addresses": [
        { "address": "10.34.0.8/32", "role": { "value": "loopback" } },
        { "address": "2001:1470:fffd:3400::8/128", "role": { "value: "loopback" } },
        { "address": "10.34.0.7/32", "role": { "value": "anycast" } }
    ],
}

Interfaces to L3 servers should have the tenant custom field defined:

{
    "name": "swp9", "type": { "value": "100gbase-x-qsfp28" },
    "custom_fields": { "tenant": { "slug": "lrk" } }
}

The tenant determines which prefixes can be received on this interface. It is important that all user‐facing ports either have a tenant defined or are disabled. Interfaces without a tenant are assumed to connect to fabric and allow all prefixes. TODO make previous sentence untrue and delete it

L2 setup

For leaf switches providing L2 access we must add a single bridge interface. If no VLANs are explicitly set, the bridge will allow any VLAN allowed on at least one of its ports. Otherwise it will only allow the specified VLANs.

{
    "name": "bridge", "type": { "value": "bridge" },
    "mode": { "value": "tagged" },
    "tagged_vlans": [
        { "name": "vlan-foo", "vid": 1234 },
        { "name": "vlan-bar", "vid": 1235 }
    ]
}

For dual-attached devices we form a MLAG between two leaf switches. Each leaf must have the peer context key set to the hostname of the other leaf. Create a bond named peerlink as one of the bridge ports, and assign it the interfaces for inter-switch links. For example exit-1 with two links to exit-2:

{
    "name": "peerlink", "type": { "value": "lag" },
    "bridge": { "name": "bridge" },
    "mode": { "value": "tagged" }
}
{
    "name": "swp29",
    "lag": { "name": "peerlink" },
    "connected_endpoints": [ { "device": { "name": "exit-2" }, "name": "swp29" } ]
},
{
    "name": "swp30",
    "lag": { "name": "peerlink" },
    "connected_endpoints": [ { "device": { "name": "exit-2" }, "name": "swp30" } ]
},

For each dual‐attached L2 device (server or switch) first create a bond on each leaf. Note that, on Cumulus Linux on Mellanox switches, a bond must be created even if a single interface is used on a particular switch. For example, the bond for sw-bdc-c3-1 on exit-1:

{
    "name": "sw-bdc-c3-1", "type": { "value": "lag" },
    "bridge": { "name": "bridge" }
}

Assign the new bond all interfaces connecting to the device (here the bond has the name of the attached L2 switch sw-bdc-c3-1):

{
    "name": "swp23s0",
    "lag": { "name": "sw-bdc-c3-1" },
    "connected_endpoints": [ { "device": { "name": "sw-bdc-c3-1" }, "name": "ethernet 1/0/49" } ]
}

If a bond with the same name (except peerlink) exists on both peer switches, a MLAG ID is assigned automatically. In this case the (same) VXLAN anycast IP should be set on each leaf’s loopback interface.

The bond interface can be set as an access or a tagged port by setting the mode attribute. Either untagged_vlan or tagged_vlans should be set as appropriate in this case. Otherwise the bond will allow all VLANs allowed by bridge.

The device on the other end of the bond should use the active‐active 802.3ad (LACP) mode.

Access switches

Current access switches are D-Link DGS-1510 and FS 5860. Connection parameters for each device type are set in config contexts and applied automatically by Ansible.

Configuration templates for access switches are stored in NetBox. The templates support setting up port channels and port tagging.

To set up a bonded interface to exit switches, configure these interfaces:

{
    "name": "port-channel 1", "type": { "value": "lag" },
    "mode": { "value": "tagged" },
    "tagged_vlans": [
        { "name": "vlan-foo", "vid": 1234 },
        { "name": "vlan-bar", "vid": 1235 }
    ]
},
{
    "name": "ethernet 1/0/49", "lag": "port-channel 1",
    "link_peers": { "device": { "name": "exit-1" } }
}
{
    "name": "ethernet 1/0/50", "lag": "port-channel 1",
    "link_peers": { "device": { "name": "exit-2" } }
}

To enable an access interface, tag it with the appropriate VLAN(s), for example:

{
    "name": "ethernet 1/0/10",
    "mode": { "value": "access" },
    "untagged_vlan": { "vid": 1234 }
},
{
    "name": "ethernet 1/0/11",
    "mode": { "value": "tagged" },
    "tagged_vlans": [{ "vid": 1234 }, { "vid": 1235 }]
}

Interfaces marked as disabled are shut down.

Firewall

The setup consists of two firewall nodes and a control node.

For the firewall nodes, configure mgmt0 and lo as usual for L3 devices. Additionally, the firewall nodes should define the following interfaces:

{ "name": "lan0" },
{ "name": "lan1" },
{ "name": "mgmt1", "ip_addresses": [{ "address": "fe80::1/64" }] },

The MAC address should be defined for each interface, as they are renamed by the OS. The mgmt1 interface is used for synchronizing connection-tracking information and should use the fe80::1/64 and fe80::2/64 addresses for the first and second firewall node, respectively.

Each firewall node should have a local config context with the keys master and iface_sync defining the names of the control node and the synchronization interface.