Configuration

The NetBox device and interfaces should define at least the following properties:

{
    "asn": { "asn": 65000 },
    "cluster": "pve",
    "cluster_type": "proxmox",
    "name": "pve-2",
    "interfaces": [
        {
            "comment": "Link to fabric for BGP."
            "name": "lan0",
            "mac_address": "AA:BB:CC:DD:FF:00",
        },
        {
            "comment": "Link to fabric for BGP."
            "name": "lan1",
            "mac_address": "AA:BB:CC:DD:FF:01",
        },
        {
            "comment": "Addresses on this interface are announced via BGP to fabric."
            "name": "lo",
            "type": { "value": "virtual" },
            "ip_addresses": [
                {
                    "address": "10.0.0.102/32",
                    "dns_name": "pve-2.example.net",
                    "role": { "value": "loopback" },
                },
                {
                    "address": "1000::102/128",
                    "dns_name": "pve-2.example.net",
                    "role": { "value": "loopback" },
                }
            ],
        },
        {
            "comment": "A separate SSH instance will run in mgmt VRF."
            "name": "mgmt0",
            "mac_address": "AA:BB:CC:DD:EE:00",
            "vrf": { "name": "mgmt" },
            "ip_addresses": [
                { "address": "10.0.1.102/24", "vrf": { "name": "mgmt" } },
                { "address": "1001::102/64", "vrf": { "name": "mgmt" } }
            ],
        },
    ],
}

EVPN

Each server needs a unique ASN. EVPN routes to individual VMs are announced to fabric via BGP over lan* interfaces. To allow receiving EVPN routes on the connected switch interfaces, the switch config context should define ifaces_evpn:

"ifaces_evpn": ["swp11", "swp12"]

Firewall

Firewall rules are common to the whole cluster and are represented as NetBox services. Services must be attached to a specific device, so a “fake” VM is created to hold all rules for the cluster. Each service should then be added to the services custom field for the cluster. All traffic between cluster nodes is allowed by default.

Certificates

The role configures ACME settings based on the dns_name of the loopback IP address. Certificate has to be ordered manually once after setup.

Ceph

In the config context for device or cluster two additional options may be defined. Specifying

ceph-version: squid

will enable the no-subscription Ceph repository and install the given version.

LDAP

To sync users and groups with LDAP, add the following to the config context:

sync-ldap: realm

This will configure a daily cron job to sync users. The cluster password file should contain ldap_user and ldap_pass keys. The servers are autodiscovered using the domain property from the config context.

Cluster

After setup, join the device into an existing cluster with

ip vrf exec default pvecm add pve-1.example.net --link0 10.0.0.101

where pve-1 is an existing cluster node and --link0 specifies the loopback address of the new device.

3.3 KiB Raw Blame History