I have been over the automatic creation of the VXLAN topology in the previous post. However, this stopped at the creation of the topology only. If in the future I wanted to add a new spine or leaf switch, this could be done, but the other switches wouldn’t be aware as they would also need to be rebuilt.
So in this post, I will address a method to boot up new switches and have the existing fabric become aware that there is an additional leaf or spine.
The scripts used for this post can be found here on my GitHub.
The process is quite simple and does require some manual intervention. It is not like the boot up and forget automation as the topology creation is.
– Process to boot up new switch is the same, ZTP and python scripts running from AEM to create VXLAN config.
– I have added XMPP configuration and all switches in the fabric will join a switch group (chat group)
– Before any boot up, the device username is added to the XMPP server
– Once boot up has finished, all switches will get issued with the command to run a python script that will recreate their VXLAN configuration and save it.
– All new spine and leaf switches are now known about over the fabric.
The base topology will be the switches as in the previous VXLAN posts.
Lab Prep
Ubuntu Server running; DHCP, TFTP and XMPP Servers
Creating the DHCP Server
Install the DHCP Server
|
0 1 2 3 |
sudo apt update sudo apt install isc-dhcp-server |
Configure the DHCP server with option 66 and 67
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 |
subnet 172.16.1.0 netmask 255.255.255.0 { default-lease-time 43200; max-lease-time 86400; range 172.16.1.220 172.16.1.250; option routers 172.16.1.1; option domain-name-servers 1.1.1.1, 8.8.8.8; #option domain-name "mydomain.example"; #option 66 option tftp-server-name "172.16.1.69"; #option 67 option bootfile-name "config-ztp-fix.cfg"; } |
Bind the DHCP server to an interface. In my case, it will be ens3. Use the command ip a to locate the interface to be used.
|
0 1 2 3 4 5 6 |
ip a ! locate interface to be used sudo nano /etc/default/isc-dhcp-server INTERFACESv4="ens3" |
Restart the DHCP Server and confirm it is running.
|
0 1 2 3 |
sudo systemctl restart isc-dhcp-server.service sudo systemctl status isc-dhcp-server.service |
Creating TFTP Server
Install the server
|
0 1 2 3 |
sudo apt update sudo apt install tftpd-hpa |
Check the server is running
|
0 1 2 |
sudo systemctl status tftpd-hpa |
Configure the TFTP Server. The options important to me are;
– Add in a user specified directory /tftp
– Setting the user to connect to the /tftp server when connecting, using the --secure option
– Adding the option to create files/directories on the TFTP server. --create option
|
0 1 2 |
sudo nano /etc/default/tftpd-hpa |
|
0 1 2 3 4 5 6 7 |
# /etc/default/tftpd-hpa TFTP_USERNAME="tftp" TFTP_DIRECTORY="/tftp" TFTP_ADDRESS=":69" TFTP_OPTIONS="--secure --create" |
Change the TFTP directory owner and group to the tftp user
|
0 1 2 |
sudo chown tftp:tftp /tftp |
Restart and check the server status
|
0 1 2 3 |
sudo systemctl restart tftpd-hpa sudo systemctl status tftpd-hpa |
Installing and configuring XMPP
This is a repeat of when I configure XMPP on Ubuntu in a previous post. All switches have been added to the XMPP server.
I have also used the same GAJIM client. The installation and configuration for the has been described in a previous post.
Not all switches are contacts in GAJIM client. This is because all switches are automatically joining the VXLAN switch-group (chat group) and so do not need to be individual contacts. I can reach them through the chat group without manually adding new switch contacts.
Automation Scripts
There are four files in total to get this running. Three are similar to the previous VXLAN 2 automation post. The new file is a python script that is very similar to the VXLAN config creation script. All the files can be found on GitHub.
In these scripts there is additional configuration for XMPP, this is pretty much lifted from the XMPP post
|
0 1 2 3 4 5 6 7 8 9 |
spine1(config-mgmt-xmpp)#sh run sec xmpp management xmpp no shutdown server 172.16.1.69 username spine1@securitydemo.lab password 7 03374F0E000E2F1E1E5B49 session privilege 15 switch-group vxlan@conference.securitydemo.lab domain securitydemo.lab |
- config-ztp-fix.cfg
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
! hostname provision-me ! ! interface Management1 description Provisioning-in-progress ip address dhcp ip route 0.0.0.0/0 172.16.1.1 ! ip routing event-handler ZTP1 trigger on-startup-config action bash `FastCli -p 15 -c 'copy tftp:172.16.1.69/ZTP1.py flash:'`; `FastCli -p 15 -c 'copy tftp:172.16.1.69/AEM_VXLAN_DeviceReady.py flash:'`; `FastCli -p 15 -c 'copy tftp:172.16.1.69/newSpineLeaf.py flash:'`; python3 /mnt/flash/ZTP1.py; `FastCli -p 15 -c 'reload in 1 force reason ZTP1 provisioning'` |
2. ZTP1.py
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
#!/usr/bin/python from collections import Counter import subprocess import sys import time # Delay for a 60 seconds time.sleep(60) output = subprocess.Popen(["/usr/bin/FastCli", "-c", "show lldp neigh"], stdout=subprocess.PIPE).communicate()[0] out2 = list(iter(output.decode().split())) # Count interfaces for spine lookup = ["Ethernet1", "Ethernet2", "Ethernet3", "Ethernet4"] all_counts = Counter(out2) counts = {k: all_counts[k] for k in lookup} print(f"all counts {all_counts}") print(f"counts {counts}") print(f"counts {counts.values()}") # Determine if spine or leaf for value in counts.values(): print(value) if value >= 4: switch = "spine" break else: switch = "leaf" # Determine spine number for key, value in counts.items(): if value >= 4: spine_num = key[8:] break # Assing spine number hostname and IP if switch == "spine": hostname = "spine{}".format(spine_num) ip_addr = "172.16.1.1{}".format(spine_num) # Determine and assign leaf hostname and IP if switch == "leaf": out3 = list(iter(output.decode().splitlines())) intnum = out3[8].split()[2][8:] hostname = "leaf{}".format(intnum) ip_addr = "172.16.1.1{}".format(intnum) # Add AEM for vxlan python script # download python script from tftp # create AEM # Create startup-config file f = open("/mnt/flash/startup-config", "w") f.write("\nhostname {}".format(hostname)) f.write("\ninterface Management1\nip address {}/24".format(ip_addr)) f.write("\nip routing") f.write("\nip route 0.0.0.0/0 172.16.1.1") # XMPP f.write("\nmanagement xmpp") f.write("\ndomain securitydemo.lab") f.write("\nusername {} password Stefan2020".format(hostname)) f.write("\nsession privilege 15") f.write("\nserver 172.16.1.69") f.write("\nswitch-group vxlan") f.write("\nno shutdown") # Remove AEM ZTP1 f.write("\nno event-handler ZTP1") # Add AEM for VXLAN f.write("\nevent-handler VXLAN_CONFIG") f.write("\ntrigger on-startup-config") # Reload switch to run VXLAN AEM f.write("\n action bash python3 /mnt/flash/AEM_VXLAN_DeviceReady.py; `FastCli -p 15 -c 'reload in 1 force reason VXLAN Configuration'`") f.close() sys.exit( 0 ) |
3. AEM_VXLAN_DeviceReady.py
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 |
import os import subprocess from collections import Counter import time def get_hostname(): with open('/mnt/flash/startup-config') as f: # with open('startup-config') as f: startup = f.readlines() # Get Hostname for line in startup: if "hostname" in line: hostname = line.split("\n")[0][9:] return hostname def get_lldp_interfaces(): # run lldp bash command from python output = subprocess.Popen(["/usr/bin/FastCli", "-c", "show lldp neigh | i Ethernet"], stdout=subprocess.PIPE).communicate()[0] out2 = list(iter(output.split())) leaf_count = 0 spine_count = 0 switch_list = [] for switch in out2: # print(switch.decode()) if "leaf" in switch.decode(): leaf_count + 1 switch_list.append(switch.decode()) elif "spine" in switch.decode(): spine_count + 1 switch_list.append(switch.decode()) return switch_list # Configure IP Addresses def interface_config(hostname, switch_list): interface = [] if "spine" in hostname: for switch in switch_list: eth = f"\ interface ethernet{switch[-1]}\n\ description LINK to {switch.upper()}.LAB\n\ mtu 9200\n\ logging event link-status\n\ no switchport\n\ ip address 10.{hostname[-1]}.1{switch[-1]}.0/31\n\ arp aging timeout 1200\n\ " interface.append(eth) lo0 = f"\ interface loopback 0\n\ description MANAGEMENT\n\ ip address 10.10.10.{hostname[-1]}/32\n\ " interface.extend([lo0]) elif "leaf" in hostname: print(f"{hostname} Interface IP Addresses will be...") eth1 = f"10.{hostname[-1]}.1{hostname[-1]}.1/31" eth2 = f"10.{hostname[-1]}.1{hostname[-1]}.1/31" for switch in switch_list: eth = f"\ interface ethernet{switch[-1]}\n\ description LINK to {switch.upper()}.LAB\n\ mtu 9200\n\ logging event link-status\n\ no switchport\n\ ip address 10.{switch[-1]}.1{hostname[-1]}.1/31\n\ arp aging timeout 1200\n\ " interface.append(eth) lo0 = f"\ interface loopback 0\n\ description MANAGEMENT\n\ ip address 10.10.10.1{hostname[-1]}/32\n\ " lo1 = f"\ interface loopback 1\n\ description LOGICAL VTEP\n\ ip address 172.20.{hostname[-1]}.{hostname[-1]}/32\n\ " interface.extend([eth1, eth2, lo0, lo1]) interface_text = "\n".join(interface) print(interface_text) return interface # BGP Underlay Config def underlay_config(hostname, switch_list): # ['leaf1', 'leaf2', 'leaf3', 'leaf4'] if "spine" in hostname: # Create BGP Config bgp_config = [] asn = 'router bgp 65000' router_id = f" router-id 10.10.10.{hostname[-1]}" ecmp = ' maximum-paths 4 ecmp 64' distance = ' distance bgp 20 200 200' bgp_config.insert(0,asn) bgp_config.insert(1,router_id) bgp_config.insert(2,distance) bgp_config.insert(3,ecmp) for switch in switch_list: neighbour = f"\ neighbor LEAFS6511{switch[-1]} peer group\n\ neighbor LEAFS6511{switch[-1]} remote-as 6511{switch[-1]}\n\ neighbor 10.{hostname[-1]}.1{switch[-1]}.1 peer group LEAFS6511{switch[-1]}\n\ neighbor 10.{hostname[-1]}.1{switch[-1]}.1 description LEAF{switch[-1]}.LAB\n\ " bgp_config.append(neighbour) elif "leaf" in hostname: # Create BGP Config bgp_config = [] asn = f'router bgp 6511{hostname[-1]}' router_id = f" router-id 10.10.10.1{hostname[-1]}" distance = ' distance bgp 20 200 200' ecmp = ' maximum-paths 4 ecmp 64' bgp_group = ' neighbor SPINE peer group' bgp_group_asn = ' neighbor SPINE remote-as 65000' redistribute = ' redistribute connected\n' bgp_config.insert(0,asn) bgp_config.insert(1,router_id) bgp_config.insert(2,distance) bgp_config.insert(3,bgp_group) bgp_config.insert(4,bgp_group_asn) bgp_config.insert(5,ecmp) # ['spine1', 'spine2', 'spine3', 'spine4'] for switch in switch_list: neighbour = f"\ neighbor 10.{switch[-1]}.1{hostname[-1]}.0 peer group SPINE\n\ neighbor 10.{switch[-1]}.1{hostname[-1]}.0 description SPINE{switch[-1]}.LAB\n\ " bgp_config.append(neighbour) bgp_config.append(redistribute) return bgp_config # VTEP Config def vtep_config(hostname): vtep_config = [] asn = f'router bgp 6511{hostname[-1]}' router_id = f" router-id 10.10.10.1{hostname[-1]}" vtep_net = f" network 172.20.{hostname[-1]}.{hostname[-1]}/32\n" vtep_config.insert(0,asn) vtep_config.insert(1,router_id) vtep_config.insert(2,vtep_net) return vtep_config # VXLAN EVPN def vxlan_config(hostname, switch_list): if "spine" in hostname: vtep_bgp = [ "router bgp 65000", " neighbor EVPN peer group", " neighbor EVPN next-hop-unchanged", " neighbor EVPN update-source Loopback0", " neighbor EVPN ebgp-multihop 3", " neighbor EVPN send-community extended", " !" ] for switch in switch_list: neighbour1 = f" neighbor 10.10.10.1{switch[-1]} peer group EVPN" neighbour2 = f" neighbor 10.10.10.1{switch[-1]} remote-as 6511{switch[-1]}" neighbour3 = f" neighbor 10.10.10.1{switch[-1]} description LEAF{switch[-1]}" vtep_bgp.extend([neighbour1, neighbour2, neighbour3]) elif "leaf" in hostname: vtep_bgp = [ "interface Vxlan1", " vxlan source-interface Loopback1", " vxlan udp-port 4789\n!", f"router bgp 6511{hostname[-1]}", " neighbor EVPN peer group", " neighbor EVPN remote-as 65000", " neighbor EVPN update-source Loopback0", " neighbor EVPN ebgp-multihop 3", " neighbor EVPN send-community extended", "!" ] for switch in switch_list: neighbour1 = f" neighbor 10.10.10.{switch[-1]} peer group EVPN" neighbour2 = f" neighbor 10.10.10.{switch[-1]} description SPINE{switch[-1]}.LAB" vtep_bgp.extend([neighbour1, neighbour2]) redistribute = ' redistribute connected' evpn_family = '!\n address-family evpn' evpn_active = ' neighbor EVPN activate\n' service = f'service routing protocols model multi-agent' vtep_bgp.extend([redistribute, evpn_family, evpn_active]) vtep_bgp.insert(0,service) return vtep_bgp # VXLAN EVPN VRF def vxlan_vrf_config(hostname): vrf_config = [ "vrf instance CUSTOMER1", "ip routing vrf CUSTOMER1", "vlan 20", "vlan 30", "interface Vlan20", "no autostate", "vrf CUSTOMER1", "ip address virtual 192.168.20.1/24", "interface Vlan30", "no autostate", "vrf CUSTOMER1", "ip address virtual 192.168.30.1/24", "interface Vxlan1", "vxlan vlan 20 vni 10020", "vxlan vlan 30 vni 10030", "vxlan vrf CUSTOMER1 vni 20120", f"router bgp 6511{hostname[-1]}", " vlan 20", f"rd 10.10.10.1{hostname[-1]}:20", "route-target both 20:20", "redistribute learned", "vlan 30", f"rd 10.10.10.1{hostname[-1]}:30", "route-target both 30:30", "redistribute learned", "vrf CUSTOMER1", f"rd 10.10.10.1{hostname[-1]}:20120", "route-target import evpn 20:120", "route-target export evpn 20:120", "route-target import evpn 30:130", "route-target export evpn 30:130", "redistribute connected", ] return vrf_config # Save to the special location, see AEM post if __name__ == '__main__': # introduce a wait period of 1 minute before executing so all switches are up and have basic hostnames time.sleep(60) hostname = get_hostname() switch_list = get_lldp_interfaces() # Standard config for all switches interfaces = interface_config(hostname, switch_list) underlay = underlay_config(hostname, switch_list) # Leaf specific config if "leaf" in hostname: vtep = vtep_config(hostname) # VXLAN config spine and leaf vxlan = vxlan_config(hostname, switch_list) if "leaf" in hostname: vrf = vxlan_vrf_config(hostname) # Save config to file with open('/mnt/flash/vxlan_config.cfg', 'w') as f: f.write('\n'.join(interfaces)) f.write('\n'.join(underlay)) if "leaf" in hostname: f.write('\n'.join(vtep)) f.write('\n'.join(vxlan)) if "leaf" in hostname: f.write('\n'.join(vrf)) # Remove VXLAN AEM f.write("\nno event-handler VXLAN_CONFIG") subprocess.Popen(["/usr/bin/FastCli", "-p 15", "-c", "copy flash:vxlan_config.cfg running-config"], stdout=subprocess.PIPE).communicate()[0] subprocess.Popen(["/usr/bin/FastCli", "-p 15", "-c", "wr mem"], stdout=subprocess.PIPE).communicate()[0] # Reload for multi-agent service subprocess.Popen(["/usr/bin/FastCli", "-p 15", "-c", "reload in 1 force reason VXLAN service multi-agent"], stdout=subprocess.PIPE).communicate()[0] |
4. newSpineLeaf.py
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 |
import os import subprocess from collections import Counter import time def get_hostname(): with open('/mnt/flash/startup-config') as f: # with open('startup-config') as f: startup = f.readlines() # Get Hostname for line in startup: if "hostname" in line: hostname = line.split("\n")[0][9:] return hostname def get_lldp_interfaces(): # run lldp bash command from python output = subprocess.Popen(["/usr/bin/FastCli", "-c", "show lldp neigh | i Ethernet"], stdout=subprocess.PIPE).communicate()[0] out2 = list(iter(output.split())) leaf_count = 0 spine_count = 0 switch_list = [] for switch in out2: # print(switch.decode()) if "leaf" in switch.decode(): leaf_count + 1 switch_list.append(switch.decode()) elif "spine" in switch.decode(): spine_count + 1 switch_list.append(switch.decode()) return switch_list # Configure IP Addresses def interface_config(hostname, switch_list): interface = [] if "spine" in hostname: for switch in switch_list: eth = f"\ interface ethernet{switch[-1]}\n\ description LINK to {switch.upper()}.LAB\n\ mtu 9200\n\ logging event link-status\n\ no switchport\n\ ip address 10.{hostname[-1]}.1{switch[-1]}.0/31\n\ arp aging timeout 1200\n\ " interface.append(eth) lo0 = f"\ interface loopback 0\n\ description MANAGEMENT\n\ ip address 10.10.10.{hostname[-1]}/32\n\ " interface.extend([lo0]) elif "leaf" in hostname: print(f"{hostname} Interface IP Addresses will be...") eth1 = f"10.{hostname[-1]}.1{hostname[-1]}.1/31" eth2 = f"10.{hostname[-1]}.1{hostname[-1]}.1/31" for switch in switch_list: eth = f"\ interface ethernet{switch[-1]}\n\ description LINK to {switch.upper()}.LAB\n\ mtu 9200\n\ logging event link-status\n\ no switchport\n\ ip address 10.{switch[-1]}.1{hostname[-1]}.1/31\n\ arp aging timeout 1200\n\ " interface.append(eth) lo0 = f"\ interface loopback 0\n\ description MANAGEMENT\n\ ip address 10.10.10.1{hostname[-1]}/32\n\ " lo1 = f"\ interface loopback 1\n\ description LOGICAL VTEP\n\ ip address 172.20.{hostname[-1]}.{hostname[-1]}/32\n\ " interface.extend([eth1, eth2, lo0, lo1]) interface_text = "\n".join(interface) print(interface_text) return interface # BGP Underlay Config def underlay_config(hostname, switch_list): # ['leaf1', 'leaf2', 'leaf3', 'leaf4'] if "spine" in hostname: # Create BGP Config bgp_config = [] asn = 'router bgp 65000' router_id = f" router-id 10.10.10.{hostname[-1]}" distance = ' distance bgp 20 200 200' bgp_config.insert(0,asn) bgp_config.insert(1,router_id) bgp_config.insert(2,distance) for switch in switch_list: neighbour = f"\ neighbor LEAFS6511{switch[-1]} peer group\n\ neighbor LEAFS6511{switch[-1]} remote-as 6511{switch[-1]}\n\ neighbor 10.{hostname[-1]}.1{switch[-1]}.1 peer group LEAFS6511{switch[-1]}\n\ neighbor 10.{hostname[-1]}.1{switch[-1]}.1 description LEAF{switch[-1]}.LAB\n\ " bgp_config.append(neighbour) elif "leaf" in hostname: # Create BGP Config bgp_config = [] asn = f'router bgp 6511{hostname[-1]}' router_id = f" router-id 10.10.10.1{hostname[-1]}" distance = ' distance bgp 20 200 200' bgp_group = ' neighbor SPINE peer group' bgp_group_asn = ' neighbor SPINE remote-as 65000' redistribute = ' redistribute connected\n' bgp_config.insert(0,asn) bgp_config.insert(1,router_id) bgp_config.insert(2,distance) bgp_config.insert(3,bgp_group) bgp_config.insert(4,bgp_group_asn) # ['spine1', 'spine2', 'spine3', 'spine4'] for switch in switch_list: neighbour = f"\ neighbor 10.{switch[-1]}.1{hostname[-1]}.0 peer group SPINE\n\ neighbor 10.{switch[-1]}.1{hostname[-1]}.0 description SPINE{switch[-1]}.LAB\n\ " bgp_config.append(neighbour) bgp_config.append(redistribute) return bgp_config # VTEP Config def vtep_config(hostname): vtep_config = [] asn = f'router bgp 6511{hostname[-1]}' router_id = f" router-id 10.10.10.1{hostname[-1]}" vtep_net = f" network 172.20.{hostname[-1]}.{hostname[-1]}/32\n" vtep_config.insert(0,asn) vtep_config.insert(1,router_id) vtep_config.insert(2,vtep_net) return vtep_config # VXLAN EVPN def vxlan_config(hostname, switch_list): if "spine" in hostname: vtep_bgp = [ "router bgp 65000", " neighbor EVPN peer group", " neighbor EVPN next-hop-unchanged", " neighbor EVPN update-source Loopback0", " neighbor EVPN ebgp-multihop 3", " neighbor EVPN send-community extended", " !" ] for switch in switch_list: neighbour1 = f" neighbor 10.10.10.1{switch[-1]} peer group EVPN" neighbour2 = f" neighbor 10.10.10.1{switch[-1]} remote-as 6511{switch[-1]}" neighbour3 = f" neighbor 10.10.10.1{switch[-1]} description LEAF{switch[-1]}" vtep_bgp.extend([neighbour1, neighbour2, neighbour3]) elif "leaf" in hostname: vtep_bgp = [ "interface Vxlan1", " vxlan source-interface Loopback1", " vxlan udp-port 4789\n!", f"router bgp 6511{hostname[-1]}", " neighbor EVPN peer group", " neighbor EVPN remote-as 65000", " neighbor EVPN update-source Loopback0", " neighbor EVPN ebgp-multihop 3", " neighbor EVPN send-community extended", "!" ] for switch in switch_list: neighbour1 = f" neighbor 10.10.10.{switch[-1]} peer group EVPN" neighbour2 = f" neighbor 10.10.10.{switch[-1]} description SPINE{switch[-1]}.LAB" vtep_bgp.extend([neighbour1, neighbour2]) redistribute = ' redistribute connected' evpn_family = '!\n address-family evpn' evpn_active = ' neighbor EVPN activate\n' vtep_bgp.extend([redistribute, evpn_family, evpn_active]) return vtep_bgp # VXLAN EVPN VRF def vxlan_vrf_config(hostname): vrf_config = [ "vrf instance CUSTOMER1", "ip routing vrf CUSTOMER1", "vlan 20", "vlan 30", "interface Vlan20", "no autostate", "vrf CUSTOMER1", "ip address virtual 192.168.20.1/24", "interface Vlan30", "no autostate", "vrf CUSTOMER1", "ip address virtual 192.168.30.1/24", "interface Vxlan1", "vxlan vlan 20 vni 10020", "vxlan vlan 30 vni 10030", "vxlan vrf CUSTOMER1 vni 20120", f"router bgp 6511{hostname[-1]}", " vlan 20", f"rd 10.10.10.1{hostname[-1]}:20", "route-target both 20:20", "redistribute learned", "vlan 30", f"rd 10.10.10.1{hostname[-1]}:30", "route-target both 30:30", "redistribute learned", "vrf CUSTOMER1", f"rd 10.10.10.1{hostname[-1]}:20120", "route-target import evpn 20:120", "route-target export evpn 20:120", "route-target import evpn 30:130", "route-target export evpn 30:130", "redistribute connected", ] return vrf_config if __name__ == '__main__': # introduce a wait period of 1 minute before executing so all switches are up and have basic hostnames hostname = get_hostname() switch_list = get_lldp_interfaces() # Standard config for all switches interfaces = interface_config(hostname, switch_list) underlay = underlay_config(hostname, switch_list) # Leaf specific config if "leaf" in hostname: vtep = vtep_config(hostname) # VXLAN config spine and leaf vxlan = vxlan_config(hostname, switch_list) if "leaf" in hostname: vrf = vxlan_vrf_config(hostname) # Save config to file with open('/mnt/flash/vxlan_config.cfg', 'w') as f: f.write('\n'.join(interfaces)) f.write('\n'.join(underlay)) if "leaf" in hostname: f.write('\n'.join(vtep)) f.write('\n'.join(vxlan)) if "leaf" in hostname: f.write('\n'.join(vrf)) subprocess.Popen(["/usr/bin/FastCli", "-p 15", "-c", "copy flash:vxlan_config.cfg running-config"], stdout=subprocess.PIPE).communicate()[0] subprocess.Popen(["/usr/bin/FastCli", "-p 15", "-c", "wr mem"], stdout=subprocess.PIPE).communicate()[0] |
Adding Leaf5
Before adding a new switch, the switch XMPP user must be added to the XMPP server. I have explained this in my XMPP configuration post, so a quick summary is that the commands are to be run as the root or ejabberd users.
|
0 1 2 3 |
osboxes@osboxes:~$ ejabberdctl register admin securitydemo.lab Stefan2020 ERROR: This command can only be run by root or the user ejabberd |
|
0 1 2 3 4 5 |
root@osboxes:~# ejabberdctl register admin securitydemo.lab Stefan2020 User admin@securitydemo.lab successfully registered root@osboxes:~# ejabberdctl register spine1 securitydemo.lab Stefan2020 User spine1@securitydemo.lab successfully registered |
The current state of the VXLAN network is correct. BGP is working, and the switches have learnt about each other’s VTEP. Below is an example output from Spine1 and Leaf3 to demonstrate.
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
spine1#sh ip bgp summary BGP summary information for VRF default Router identifier 10.10.10.1, local AS number 65000 Neighbor Status Codes: m - Under maintenance Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc LEAF1.LAB 10.1.11.1 4 65111 24 21 0 0 00:07:17 Estab 5 5 LEAF2.LAB 10.1.12.1 4 65112 25 18 0 0 00:07:29 Estab 5 5 LEAF3.LAB 10.1.13.1 4 65113 26 20 0 0 00:07:22 Estab 5 5 LEAF4.LAB 10.1.14.1 4 65114 22 21 0 0 00:07:17 Estab 5 5 LEAF1 10.10.10.11 4 65111 34 30 0 0 00:06:56 Estab 5 5 LEAF2 10.10.10.12 4 65112 43 27 0 0 00:07:25 Estab 5 5 LEAF3 10.10.10.13 4 65113 38 27 0 0 00:07:20 Estab 5 5 LEAF4 10.10.10.14 4 65114 30 29 0 0 00:07:07 Estab 5 5 |
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
leaf3#show ip bgp summary BGP summary information for VRF default Router identifier 10.10.10.13, local AS number 65113 Neighbor Status Codes: m - Under maintenance Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc SPINE1.LAB 10.1.13.0 4 65000 39 46 0 0 00:07:48 Estab 15 15 SPINE2.LAB 10.2.13.0 4 65000 38 59 0 0 00:07:47 Estab 15 15 SPINE1.LAB 10.10.10.1 4 65000 55 90 0 0 00:07:45 Estab 15 15 SPINE2.LAB 10.10.10.2 4 65000 57 89 0 0 00:07:40 Estab 15 15 leaf3#show vxlan vtep Remote VTEPS for Vxlan1: VTEP Tunnel Type(s) ---------------- -------------- 172.20.1.1 flood, unicast 172.20.2.2 flood, unicast 172.20.4.4 flood, unicast Total number of remote VTEPS: 3 |
Now that the users are added to the server, the new leaf switch can be turned on. This will go through the process of discovering what its hostname is and LLDP neighbours are From the hostname and LLDP neighbour details, the correct VXLAN configuration will be applied. At this point the new switch won’t have any BGP neighbours and be cut off from the rest of the fabric.
Leaf5 is cut off from the rest of the VXLAN fabric. The only functioning interface is the management.
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
leaf5#show ip bgp summary BGP summary information for VRF default Router identifier 10.10.10.15, local AS number 65115 Neighbor Status Codes: m - Under maintenance Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc SPINE1.LAB 10.1.15.0 4 65000 0 0 0 0 00:04:03 Active SPINE2.LAB 10.2.15.0 4 65000 0 0 0 0 00:04:03 Active SPINE1.LAB 10.10.10.1 4 65000 0 0 0 0 00:04:03 Active SPINE2.LAB 10.10.10.2 4 65000 0 0 0 0 00:04:03 Active leaf5#show vxlan vtep Remote VTEPS for Vxlan1: VTEP Tunnel Type(s) ---------- -------------- Total number of remote VTEPS: 0 |
To fix the fact that the new switch is not yet part of the fabric, a manual command will be run from XMPP to all switches in the VXLAN chat group, even the new leaf. This command is to run the python script newSpineLeaf.py This will take seconds to execute and each switch in the fabric will have the correct config, allowing the new leaf5 to function correctly.
|
0 1 2 |
bash python3 /mnt/flash/newSpineLeaf.py |
Now that the switches have reconfigured themselves, Leaf5 now has BGP neighbours, prefixes and knows about the remote VTEPs
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
leaf5#show ip bgp summary BGP summary information for VRF default Router identifier 10.10.10.15, local AS number 65115 Neighbor Status Codes: m - Under maintenance Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc SPINE1.LAB 10.1.15.0 4 65000 10 7 0 0 00:01:07 Estab 19 19 SPINE2.LAB 10.2.15.0 4 65000 10 12 0 0 00:01:07 Estab 19 19 SPINE1.LAB 10.10.10.1 4 65000 24 16 0 0 00:01:06 Estab 19 19 SPINE2.LAB 10.10.10.2 4 65000 24 28 0 0 00:00:59 Estab 19 19 leaf5#show vxlan vtep Remote VTEPS for Vxlan1: VTEP Tunnel Type(s) ---------------- -------------- 172.20.1.1 flood, unicast 172.20.2.2 flood, unicast 172.20.3.3 flood, unicast 172.20.4.4 flood, unicast Total number of remote VTEPS: 4 |
Adding Spine3
Adding a new spine switch will behave in the same way as adding a Leaf5 did.
Add the new switch user to the XMPP server
Cable and start the switch Spine3
Once that has booted up and is responding to commands sent via XMPP chat, all that is left is top run the newSpineLeaf.py script to add it to the fabric correctly.
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
spine3#show ip bgp summary BGP summary information for VRF default Router identifier 10.10.10.3, local AS number 65000 Neighbor Status Codes: m - Under maintenance Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc LEAF1.LAB 10.3.11.1 4 65111 0 0 0 0 00:02:20 Active LEAF2.LAB 10.3.12.1 4 65112 0 0 0 0 00:02:20 Active LEAF3.LAB 10.3.13.1 4 65113 0 0 0 0 00:02:20 Active LEAF4.LAB 10.3.14.1 4 65114 0 0 0 0 00:02:20 Active LEAF5.LAB 10.3.15.1 4 65115 0 0 0 0 00:02:20 Active LEAF1 10.10.10.11 4 65111 0 0 0 0 00:02:20 Active LEAF2 10.10.10.12 4 65112 0 0 0 0 00:02:20 Active LEAF3 10.10.10.13 4 65113 0 0 0 0 00:02:20 Active LEAF4 10.10.10.14 4 65114 0 0 0 0 00:02:20 Active LEAF5 10.10.10.15 4 65115 0 0 0 0 00:02:20 Active |
Now that all switches are responding, time to run the newSpineLeaf.py script.
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
spine3#show ip bgp summary BGP summary information for VRF default Router identifier 10.10.10.3, local AS number 65000 Neighbor Status Codes: m - Under maintenance Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc LEAF1.LAB 10.3.11.1 4 65111 15 10 0 0 00:01:51 Estab 6 6 LEAF2.LAB 10.3.12.1 4 65112 13 9 0 0 00:00:15 Estab 6 6 LEAF3.LAB 10.3.13.1 4 65113 15 10 0 0 00:01:51 Estab 6 6 LEAF4.LAB 10.3.14.1 4 65114 13 10 0 0 00:01:51 Estab 6 6 LEAF5.LAB 10.3.15.1 4 65115 14 11 0 0 00:01:51 Estab 6 6 LEAF1 10.10.10.11 4 65111 30 24 0 0 00:01:43 Estab 6 6 LEAF2 10.10.10.12 4 65112 28 23 0 0 00:00:10 Estab 6 6 LEAF3 10.10.10.13 4 65113 30 24 0 0 00:01:43 Estab 6 6 LEAF4 10.10.10.14 4 65114 30 24 0 0 00:01:50 Estab 6 6 LEAF5 10.10.10.15 4 65115 29 24 0 0 00:01:44 Estab 6 6 |
Below is BGP config taken from Leaf4 to show that ECMP is running and is going via the three spine switches for connectivity to the VTEPs.
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
leaf4#show ip route ! some output omitted B E 172.20.1.1/32 [20/0] via 10.1.14.0, Ethernet1 via 10.2.14.0, Ethernet2 via 10.3.14.0, Ethernet3 B E 172.20.2.2/32 [20/0] via 10.1.14.0, Ethernet1 via 10.2.14.0, Ethernet2 via 10.3.14.0, Ethernet3 B E 172.20.3.3/32 [20/0] via 10.1.14.0, Ethernet1 via 10.2.14.0, Ethernet2 via 10.3.14.0, Ethernet3 C 172.20.4.4/32 is directly connected, Loopback1 B E 172.20.5.5/32 [20/0] via 10.1.14.0, Ethernet1 via 10.2.14.0, Ethernet2 via 10.3.14.0, Ethernet3 |