Arista VXLAN 2 – Automating Spine and Leaf Configuration

Arista, Automation, Lab, Python /

This is going to follow on from the previous post to get a basic three switch topology up and running with VXLAN. This time however, instead of doing more VXLAN I will concentrate on a way to automate the adding of spine and leaf switches to the fabric.
The purpose of this automation piece is to turn on all the switches in a new topology or add additional spine or leaf switches, and they will configure themselves with a working VXLAN config using one VRF and two VLANs.

The scripts used for this post can be found here on my GitHub.

The automation piece will be using previous posts that cover DHCP Options, ZTP and AEM.
The phases of what will happen are;

1 DHCP Options

DHCP options direct a new and unconfigured switch to retrieve the file config-ztp-fix.cfg
I have discussed how I managed to do this in this post on a Checkpoint firewall. This file is giving IP connectivity on the management interface and adding an AEM event that will download, run a python script to understand if it is a spine or leaf switch and then reload itself to boot into the new config.
The config file will also download another python script that will be used later for the VXLAN configuration.

2 AEM Runs ZTP1 Python Script

This has been previously described in this post.
The AEM event will run the previously downloaded python script that determines if the switch is a spine or leaf based off the number of connections that are active. If there are four or more interfaces active, then it is a spine, if not it is a leaf. There may be some issues with this approach as the topology grows, but that is to be seen. For the lab topology of six switches, it is fine.
Now the script has determined the switch type, it will determine the number off LLDP connected port numbers and then create a management IP address of the switch type and number.

Hostnames & Management IPs
Spine 1: 172.17.3.101/24
Spine 2: 172.17.3.102/24
Leaf 1: 172.17.3.1/24
Leaf 2: 172.17.3.2/24
Leaf 1: 172.17.3.3/24
Leaf 2: 172.17.3.4/24

The python script will perform two more tasks, which are to remove the AEM config that started this process and to configure another AEM event that will run the VXLAN on reload. Now that the Python script has run, the switch is reloaded to boot into its new configuration.

There is a 60-second sleep timer on this script to allow all the devices to run the show lldp neoghbor command to be able to see each other. If the switch doesn’t see all f the switches that will be in the spine leaf topology, the hostname and IP will be different, causing a misconfiguration for the next stage.

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
 
#!/usr/bin/python
 
from collections import Counter
import subprocess
import sys
import time
 
# Delay for a 60 seconds
time.sleep(60)
output = subprocess.Popen(["/usr/bin/FastCli", "-c", "show lldp neigh"], stdout=subprocess.PIPE).communicate()[0]
out2 = list(iter(output.decode().split()))
 
# Count interfaces for spine
lookup = ["Ethernet1", "Ethernet2", "Ethernet3", "Ethernet4"]
all_counts = Counter(out2)
counts = {k: all_counts[k] for k in lookup}
print(f"all counts {all_counts}")
print(f"counts {counts}")
print(f"counts {counts.values()}")
 
# Determine if spine or leaf
for value in counts.values():
    print(value)
    if value >= 4:
        switch = "spine"
        break
    else:
        switch = "leaf"
 
# Determine spine number
for key, value in counts.items():
    if value >= 4:
        spine_num = key[8:]
        break
 
# Assing spine number hostname and IP
if switch == "spine":
    hostname = "spine{}".format(spine_num)
    ip_addr = "172.17.3.10{}".format(spine_num)
 
# Determine and assign leaf hostname and IP
if switch == "leaf":
    out3 = list(iter(output.decode().splitlines()))
    intnum = out3[8].split()[2][8:]
    hostname = "leaf{}".format(intnum)
    ip_addr = "172.17.3.{}".format(intnum)
 
 
# Add AEM for vxlan python script
# download python script from tftp
# create AEM
 
 
# Create startup-config file
f = open("/mnt/flash/startup-config", "w")
f.write("\nhostname {}".format(hostname))
f.write("\ninterface Management1\nip address {}/24".format(ip_addr))
f.write("\nip routing")
f.write("\nip route 0.0.0.0/0 172.17.3.254")
# Remove AEM ZTP1
f.write("\nno event-handler ZTP1")
# Add AEM for VXLAN
f.write("\nevent-handler VXLAN_CONFIG")
f.write("\ntrigger on-startup-config")
# Reload switch to run VXLAN AEM
f.write("\n action bash python3 /mnt/flash/AEM_VXLAN_DeviceReady.py; `FastCli -p 15 -c 'reload in 1 force reason VXLAN Configuration'`")
 
f.close()
sys.exit( 0 )
 

3 AEM Runs AEM_VXLAN_DeviceReady Python Script

I have previously written an AEM script that follows the same steps above, and then configures the XMPP feature using the hostname as the foundation for the XMPP username the switch will configure for itself. That can be found in this post.

This script will configure the spine and leaf switches with all the configuration to get VXLAN working. I have also added a VRF and 2 VLANs for a CUSTOMER1. This is used as a test tenant. All that is needed to be added is the VLANs on ports and some test PCs configuring to confirm network connectivity.

There is an important point to this script about the timing. The devices for a fresh topology must be booted around the same time. I have included a 60-second sleep timer in the script to allow all switches to boot and the show lldp neighbor command to run while they are all up. If this command runs when switches aren’t all in a fully operational mode, the configuration will not be correct.

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
 
import os
import subprocess
from collections import Counter
import time
 
 
def get_hostname():
 
    with open('/mnt/flash/startup-config') as f:
    # with open('startup-config') as f:
        startup = f.readlines()
 
    # Get Hostname
    for line in startup:
        if "hostname" in line:
            hostname = line.split("\n")[0][9:]
    return hostname
 
 
def get_lldp_interfaces():
    # run lldp bash command from python
    output = subprocess.Popen(["/usr/bin/FastCli", "-c", "show lldp neigh | i Ethernet"], stdout=subprocess.PIPE).communicate()[0]    
    out2 = list(iter(output.split()))
 
    leaf_count = 0
    spine_count = 0
    switch_list = []
    for switch in out2:
        # print(switch.decode())
        if "leaf" in switch.decode():
            leaf_count + 1
            switch_list.append(switch.decode())
        elif "spine" in switch.decode():
            spine_count + 1
            switch_list.append(switch.decode())
    
    return switch_list
 
 
# Configure IP Addresses
def interface_config(hostname, switch_list):
    interface = []
 
    if "spine" in hostname:
        for switch in switch_list:
            eth = f"\
interface ethernet{switch[-1]}\n\
description LINK to {switch.upper()}.LAB\n\
mtu 9200\n\
logging event link-status\n\
no switchport\n\
ip address 10.{hostname[-1]}.1{switch[-1]}.0/31\n\
arp aging timeout 1200\n\
"      
            interface.append(eth)
 
        lo0 = f"\
interface loopback 0\n\
description MANAGEMENT\n\
ip address 10.10.10.{hostname[-1]}/32\n\
"
        interface.extend([lo0])
 
    elif "leaf" in hostname:
        print(f"{hostname} Interface IP Addresses will be...")
        eth1 = f"10.{hostname[-1]}.1{hostname[-1]}.1/31"
        eth2 = f"10.{hostname[-1]}.1{hostname[-1]}.1/31"
 
        for switch in switch_list:
            eth = f"\
interface ethernet{switch[-1]}\n\
description LINK to {switch.upper()}.LAB\n\
mtu 9200\n\
logging event link-status\n\
no switchport\n\
ip address 10.{switch[-1]}.1{hostname[-1]}.1/31\n\
arp aging timeout 1200\n\
"                
            interface.append(eth)
 
        lo0 = f"\
interface loopback 0\n\
description MANAGEMENT\n\
ip address 10.10.10.1{hostname[-1]}/32\n\
"
 
        lo1 = f"\
interface loopback 1\n\
description LOGICAL VTEP\n\
ip address 172.20.{hostname[-1]}.{hostname[-1]}/32\n\
"
        interface.extend([eth1, eth2, lo0, lo1])
 
    interface_text = "\n".join(interface)
    print(interface_text)
 
    return interface
 
 
# BGP Underlay Config
def underlay_config(hostname, switch_list):
 
# ['leaf1', 'leaf2', 'leaf3', 'leaf4']
 
    if "spine" in hostname:
        # Create BGP Config
        bgp_config = []
        asn = 'router bgp 65000'
        router_id = f" router-id 10.10.10.{hostname[-1]}"
        distance = ' distance bgp 20 200 200'
        
        bgp_config.insert(0,asn)
        bgp_config.insert(1,router_id)
        bgp_config.insert(2,distance)
 
        for switch in switch_list:
            neighbour = f"\
neighbor LEAFS6511{switch[-1]} peer group\n\
neighbor LEAFS6511{switch[-1]} remote-as 6511{switch[-1]}\n\
neighbor 10.{hostname[-1]}.1{switch[-1]}.1 peer group LEAFS6511{switch[-1]}\n\
neighbor 10.{hostname[-1]}.1{switch[-1]}.1 description LEAF{switch[-1]}.LAB\n\
"
            bgp_config.append(neighbour)
 
    elif "leaf" in hostname:
        # Create BGP Config
        bgp_config = []
        asn = f'router bgp 6511{hostname[-1]}'
        router_id = f" router-id 10.10.10.1{hostname[-1]}"
        distance = ' distance bgp 20 200 200'
        bgp_group = ' neighbor SPINE peer group'
        bgp_group_asn = ' neighbor SPINE remote-as 65000'
        redistribute = ' redistribute connected\n'
 
        bgp_config.insert(0,asn)
        bgp_config.insert(1,router_id)
        bgp_config.insert(2,distance)
        bgp_config.insert(3,bgp_group)
        bgp_config.insert(4,bgp_group_asn)
 
        # ['spine1', 'spine2', 'spine3', 'spine4']
        for switch in switch_list:
            neighbour = f"\
neighbor 10.{switch[-1]}.1{hostname[-1]}.0 peer group SPINE\n\
neighbor 10.{switch[-1]}.1{hostname[-1]}.0 description SPINE{switch[-1]}.LAB\n\
"
            bgp_config.append(neighbour)
        bgp_config.append(redistribute)
 
    return bgp_config
 
 
# VTEP Config
def vtep_config(hostname):
    vtep_config = []
    asn = f'router bgp 6511{hostname[-1]}'
    router_id = f" router-id 10.10.10.1{hostname[-1]}"
    vtep_net = f" network 172.20.{hostname[-1]}.{hostname[-1]}/32\n"
 
    vtep_config.insert(0,asn)
    vtep_config.insert(1,router_id)
    vtep_config.insert(2,vtep_net)
 
    return vtep_config
 
 
# VXLAN EVPN
def vxlan_config(hostname, switch_list):
    
    if "spine" in hostname:
        vtep_bgp = [
                    "router bgp 65000",
                    " neighbor EVPN peer group",
                    " neighbor EVPN next-hop-unchanged",
                    " neighbor EVPN update-source Loopback0",
                    " neighbor EVPN ebgp-multihop 3",
                    " neighbor EVPN send-community extended",
                    " !"
                    ]
 
        for switch in switch_list:
            neighbour1 = f" neighbor 10.10.10.1{switch[-1]} peer group EVPN"
            neighbour2 = f" neighbor 10.10.10.1{switch[-1]} remote-as 6511{switch[-1]}"
            neighbour3 = f" neighbor 10.10.10.1{switch[-1]} description LEAF{switch[-1]}"
 
            vtep_bgp.extend([neighbour1, neighbour2, neighbour3])
 
    elif "leaf" in hostname:
        vtep_bgp = [
                    "interface Vxlan1",
                    " vxlan source-interface Loopback1",
                    " vxlan udp-port 4789\n!",
                    f"router bgp 6511{hostname[-1]}",
                    " neighbor EVPN peer group",
                    " neighbor EVPN remote-as 65000",
                    " neighbor EVPN update-source Loopback0",
                    " neighbor EVPN ebgp-multihop 3",
                    " neighbor EVPN send-community extended",
                    "!"
                    ]
 
        for switch in switch_list:
            neighbour1 = f" neighbor 10.10.10.{switch[-1]} peer group EVPN"
            neighbour2 = f" neighbor 10.10.10.{switch[-1]} description SPINE{switch[-1]}.LAB"
 
            vtep_bgp.extend([neighbour1, neighbour2])
 
    redistribute = ' redistribute connected'
    evpn_family = '!\n address-family evpn'
    evpn_active = '  neighbor EVPN activate\n'
    service = f'service routing protocols model multi-agent'
 
    vtep_bgp.extend([redistribute, evpn_family, evpn_active])
    vtep_bgp.insert(0,service)
 
    return vtep_bgp
 
 
# VXLAN EVPN VRF
def vxlan_vrf_config(hostname):
    
    vrf_config = [
                "vrf instance CUSTOMER1",
                "ip routing vrf CUSTOMER1",
                "vlan 20",
                "vlan 30",
                "interface Vlan20",
                "no autostate",
                "vrf CUSTOMER1",
                "ip address virtual 192.168.20.1/24",
                "interface Vlan30",
                "no autostate",
                "vrf CUSTOMER1",
                "ip address virtual 192.168.30.1/24",
                "interface Vxlan1",
                "vxlan vlan 20 vni 10020",
                "vxlan vlan 30 vni 10030",
                "vxlan vrf CUSTOMER1 vni 20120",
                f"router bgp 6511{hostname[-1]}",
                " vlan 20",
                f"rd 10.10.10.1{hostname[-1]}:20",
                "route-target both 20:20",
                "redistribute learned",
                "vlan 30",
                f"rd 10.10.10.1{hostname[-1]}:30",
                "route-target both 30:30",
                "redistribute learned",
                "vrf CUSTOMER1",
                f"rd 10.10.10.1{hostname[-1]}:20120",
                "route-target import evpn 20:120",
                "route-target export evpn 20:120",
                "route-target import evpn 30:130",
                "route-target export evpn 30:130",
                "redistribute connected",
                ]
 
    return vrf_config
 
 
 
# Save to the special location, see AEM post
 
 
if __name__ == '__main__':
    # introduce a wait period of 1 minute before executing so all switches are up and have basic hostnames
    time.sleep(60)
    hostname = get_hostname()
    switch_list = get_lldp_interfaces()
 
    # Standard config for all switches
    interfaces = interface_config(hostname, switch_list)
    underlay = underlay_config(hostname, switch_list)
 
    # Leaf specific config
    if "leaf" in hostname:
        vtep = vtep_config(hostname)
    
    # VXLAN config spine and leaf
    vxlan = vxlan_config(hostname, switch_list)
 
    if "leaf" in hostname:
        vrf = vxlan_vrf_config(hostname)
 
 
    # Save config to file
    with open('/mnt/flash/vxlan_config.cfg', 'w') as f:
        f.write('\n'.join(interfaces))
        f.write('\n'.join(underlay))
        if "leaf" in hostname:
            f.write('\n'.join(vtep))    
        f.write('\n'.join(vxlan))
        if "leaf" in hostname:
            f.write('\n'.join(vrf))
        # Remove VXLAN AEM
        f.write("\nno event-handler VXLAN_CONFIG")
 
 
    subprocess.Popen(["/usr/bin/FastCli", "-p 15", "-c", "copy flash:vxlan_config.cfg running-config"], stdout=subprocess.PIPE).communicate()[0]
    subprocess.Popen(["/usr/bin/FastCli", "-p 15", "-c", "wr mem"], stdout=subprocess.PIPE).communicate()[0]
    # Reload for multi-agent service
    subprocess.Popen(["/usr/bin/FastCli", "-p 15", "-c", "reload in 1 force reason VXLAN service multi-agent"], stdout=subprocess.PIPE).communicate()[0]
 

Testing

Test 1 – Three Switch Topology

For the first test, I have used the other half of my spine/leaf topology. This will be for three switches only. Spine2, Leaf3 and Leaf4 will be automatically configured from nothing. All configuration and scripts will be removed from them, and they will be booted up at the same time.

Below is a video of the process of the three switches booting up for the first time. There are a couple of reboots but once configured the only configuration left is to add the VLANs to the switchports and cable the Ubuntu PCs up.

Test 2 – Six Switch Topology Adding ECMP

I repeated the test buy this time used all six switches in the topology. The results were the same. Each spine and leaf got the correct hostname and IP. With this, they were able to configure all the elements correctly of my VXLAN topology, and I was then able to add in some PCs to ping between the leaf switches in the same network or different.

The next steps to this was to manually implement ECMP. This is only a single line in each switch’s BGP configuration. After testing, I can add it to the script.

ECMP allows for more than one route to be added to the routing table. This is useful for the VTEPs as they can use both spine switches, so have two links to utilise.

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
! Spines
router bgp 65000
maximum-paths 4 ecmp 64
 
! Leaf 1,2,3,4
router bgp 65111
maximum-paths 4 ecmp 64
router bgp 65112
maximum-paths 4 ecmp 64
router bgp 65113
maximum-paths 4 ecmp 64
router bgp 65114
maximum-paths 4 ecmp 64
 

The below output is taken from leaf1. This shows that BGP has two routes to the VTEPS via spine 1 and 2.

Next is the BGP routing table. This shows the VTEPs have the two routes installed into the routing table.

Running a ping from 192.168.20.101 on leaf 1 to 192.168.30.101 on leaf 3 yields the following EVPN BGP output.

Leave a Comment

Your email address will not be published. Required fields are marked *