Revision3 Scripts
Revision3 Lab
Revision3 Script
A link to the Revision3 scripts in this example may be found on my GitHub.
The overall changes to the function of the script has not changed. TACACS is still checked and reconfigured.
Main Changes are;
- Added NXOS v9
- Added Multi Threading for both check and config of the devices
Notable Changes
Adding NXOS v9 Switches
I had a couple of problems with adding the NXOS v9 switches. There were two problems that I needed to overcome.
The first was that the script was written in a way that in nothing was returned from “show run | i tacacs|aaa” for either IOS or NXOS type of devices it would return False. This was ok for the original script however in later revisions when I was pulling the removal config and adding it to a dictionary, False is not something that was being returned. Therefore causing an error as something was expected.
I have changed the behaviour of this. If the return output from the network device for the show command is blank, then a string of “#Nothing to do” is returned. This will be on the remove config. This was a problem that existed all along, it was just not ever discovered as there was always something returned to the show command.
$ python cisco_run.py
SSH connection established to 192.168.122.123:22
Interactive SSH session established
Hostname: NX3_v9
IP Address: 192.168.122.123
Getting Version
unknown
Getting AAA & TACACS
Traceback (most recent call last):
File "/home/centos/Scripts/2022/TACACS/revision3/cisco_run.py", line 45, in
show_tacacs, removal_conf = show_cmds.get_tacacs(connection, dev_details)
File "/home/centos/Scripts/2022/TACACS/revision3/deviceConnections.py", line 147, in get_tacacs
return show_tacacs, removal_conf
UnboundLocalError: local variable 'removal_conf' referenced before assignment
Fixed Output
$ python cisco_run.py SSH connection established to 192.168.122.123:22 Interactive SSH session established Hostname: NX3_v9 IP Address: 192.168.122.123 Getting Version unknown Getting AAA & TACACS #Nothing to do Closing connection ############################## SSH connection established to 192.168.122.124:22 Interactive SSH session established Hostname: NX4_v9 IP Address: 192.168.122.124 Getting Version unknown Getting AAA & TACACS #Nothing to do Closing connection ##############################
The second issue was that the returned parsed dictionary from TextFSM did not contain “nxos” in like the switch running v7.
- For NXOSv7 I was looking for: ‘descr’: ‘NX-OSv Chassis ‘
- The same key in v9 is: ‘descr’: ‘Nexus9000 9000v Chassis’
I have simply just added an elif that is looking for “Nexus” inside that value.
elif "Nexus" in inventory:
device_version = "nxos"
Multi Threading
For a basic overview of the multi threading used in this script please see my post on multi threading.
To get multi threading working required several changes. I needed to move the checking and configuring of the network devices into their own functions.
I also needed to modify how the configuration function knew which OS type it was talking to. This was the tricky part and I will explain below.
In the previous revisions the linear process of the script allowed for the list order to be the same for; opening the file, connecting to and checking TACACS on the devices and then configuring those devices .
As with multi threading some threads finish faster than others. So list returned was actually in a different order. For example R2 my own device that uses TACACS as authentication in the default lab state took the longest to finish. This means that instead of it being in second position in the list it was in last.
The problems occurred in the configuration section when I passed in the original device list order and then the out of order configuration list. This meant that my last NXOS device was receiving config meant for R2 an IOS 15 device.
To resolve this I created another dictionary (and added that to a list) that was the exact same as what Netmkio requires to make the connection. This was created when the multi threading started in the checking function. It means that I can pass this new device list that is in the same order as the check config to be my new order of connection.
Thereby making R2 always the last to be connected to and the remove/add config always correct.
Notice below that although the order is not the same, the configuration is applied correctly. This is because that some threads finish faster than others for the config. But the threading keeps track of what is passed in.
Check TACACS Output
I needed to change the print statements as with multi threading they were all out of order. There is a print lock for threads, but as the print statements come in after different stages that was no good to follow what was going on. Therefore I opted to use the output list called show_tacacs_list which is only populated when the connection and show commands have been successful.
As you can see below there are some errors, but these are handled in the same way as the previous scripts without any multi threading.
$ python cisco_run_mp.py SSH connection established to 192.168.122.104:22 Interactive SSH session established SSH connection established to 192.168.122.107:22 SSH connection established to 192.168.122.105:22 Interactive SSH session established Interactive SSH session established SSH connection established to 192.168.122.102:22 SSH connection established to 192.168.122.98:22 Interactive SSH session established Interactive SSH session established SSH connection established to 192.168.122.194:22 Interactive SSH session established SSH connection established to 192.168.122.123:22 Interactive SSH session established Entering the enable mode ... There is an error connecting to 192.168.122.124 Continuing... There is an error connecting to 192.168.122.107 Continuing... CSV written out ************************************************************ Error connecting to 192.168.122.124 Error connecting to 192.168.122.107 Successfully connected and gathered output from: NX1_v7 192.168.122.104 Successfully connected and gathered output from: R4 192.168.122.105 Successfully connected and gathered output from: R1 192.168.122.102 Successfully connected and gathered output from: R3 192.168.122.98 Successfully connected and gathered output from: R2 192.168.122.194 Successfully connected and gathered output from: NX3_v9 192.168.122.123 ************************************************************
| IP Address | Hostname | Username | Current Config | Removal Config | Device Version |
| 192.168.122.104 | NX1_v7 | admin | feature tacacs+ | #Nothing to do | nxos |
| 192.168.122.123 | NX3_v9 | admin | #Nothing to do | nxos | |
| 192.168.122.105 | R4 | admin | aaa new-model aaa session-id common | #Nothing to do | iosxe |
| 192.168.122.124 | NX4_v9 | admin | #Nothing to do | nxos | |
| 192.168.122.102 | R1 | admin | aaa new-model aaa session-id common | #Nothing to do | ios15 |
| 192.168.122.107 | R5 | admin | aaa new-model aaa session-id common | #Nothing to do | ios12 |
| 192.168.122.194 | R2 | skelly | aaa new-model aaa authentication login IPCISCOAUTH group tacacs+ local aaa authorization exec default group tacacs+ aaa accounting exec default start-stop group tacacs+ aaa session-id common tacacs-server host 192.168.122.97 key 123abc tacacs-server host 192.168.122.99 key 123abc tacacs-server host 192.168.122.98 key 123abc | no aaa authentication login IPCISCOAUTH group tacacs+ local no aaa authorization exec default group tacacs+ no aaa accounting exec default start-stop group tacacs+ no tacacs-server host 192.168.122.97 key 123abc no tacacs-server host 192.168.122.99 key 123abc no tacacs-server host 192.168.122.98 key 123abc | ios15 |
Config TACACS Output
Please enter the file name of the IOS 12 config file to apply to devices: iosnewtacacs.cfg Please enter the file name of the IOS 15 config file to apply to devices: iosnewtacacs.cfg Please enter the file name of the IOS XE config file to apply to devices: iosnewtacacs.cfg Please enter the file name of the NXOS config file to apply to devices: nxosnewtacacs.cfg SSH connection established to 192.168.122.98:22 Interactive SSH session established SSH connection established to 192.168.122.102:22 Interactive SSH session established SSH connection established to 192.168.122.194:22 Interactive SSH session established SSH connection established to 192.168.122.104:22 Interactive SSH session established SSH connection established to 192.168.122.105:22 Interactive SSH session established SSH connection established to 192.168.122.123:22 Interactive SSH session established Entering the enable mode ... CSV written out ************************************************************ No Errors to Report Successfully connected and modified config from: NX1_v7 192.168.122.104 Successfully connected and modified config from: R3 192.168.122.98 Successfully connected and modified config from: R1 192.168.122.102 Successfully connected and modified config from: NX3_v9 192.168.122.123 Successfully connected and modified config from: R2 192.168.122.194 Successfully connected and modified config from: R4 192.168.122.105 ************************************************************
| IP Address | Hostname | Username | Current Config |
| 192.168.122.104 | NX1_v7 | admin | feature tacacs+ ip tacacs source-interface Ethernet2/1 tacacs-server host 192.168.122.137 key 7 “123ftc” aaa group server tacacs+ NEWTACACS aaa authentication login default group NEWTACACS tacacs-server directed-request |
| 192.168.122.102 | R1 | admin | aaa new-model aaa authentication login IPCISCOAUTH group tacacs+ local aaa authorization exec default group tacacs+ aaa accounting exec default start-stop group tacacs+ aaa session-id common tacacs server NEWTACACS |
| 192.168.122.123 | NX3_v9 | admin | feature tacacs+ ip tacacs source-interface Ethernet2/1 tacacs-server host 192.168.122.137 key 7 “123ftc” aaa group server tacacs+ NEWTACACS aaa authentication login default group NEWTACACS tacacs-server directed-request |
| 192.168.122.124 | NX4_v9 | admin | feature tacacs+ ip tacacs source-interface Ethernet2/1 tacacs-server host 192.168.122.137 key 7 “123ftc” aaa group server tacacs+ NEWTACACS aaa authentication login default group NEWTACACS tacacs-server directed-request |
| 192.168.122.105 | R4 | admin | aaa new-model aaa authentication login IPCISCOAUTH group tacacs+ local aaa authorization exec default group tacacs+ aaa accounting exec default start-stop group tacacs+ aaa session-id common tacacs server NEWTACACS |
| 192.168.122.107 | R5 | admin | aaa new-model aaa authentication login IPCISCOAUTH group tacacs+ local aaa authorization exec default group tacacs+ aaa accounting exec default start-stop group tacacs+ aaa session-id common |
| 192.168.122.194 | R2 | skelly | aaa new-model aaa authentication login IPCISCOAUTH group tacacs+ local aaa authorization exec default group tacacs+ aaa accounting exec default start-stop group tacacs+ aaa session-id common tacacs-server host 192.168.122.97 key 123abc tacacs-server host 192.168.122.99 key 123abc tacacs-server host 192.168.122.98 key 123abc tacacs server NEWTACACS |
Function For Checking TACACS
def open_check_device_mp(q, thread_no, dev_ver_list, show_tacacs_list, dev_list_thread):
while True:
cisco_device = q.get()
print(cisco_device)
print("\n")
# 1. Create the connection
connection, hostname, host_ip, host_username, host_password = check_tacacs_cmds_iosxe.open_connection(cisco_device)
# 2. Get Version commands
print("Getting Version")
device_version = check_tacacs_cmds_iosxe.get_version(connection)
print(device_version)
dev_ver_dict = {"IP Address":host_ip,
"Hostname":hostname,
"Username":host_username,
"Device Version":device_version}
dev_ver_list.append(dev_ver_dict)
#3. Show TACACS
show_tacacs, removal_conf = show_cmds.get_tacacs(connection, dev_ver_dict)
show_tacacs_dict = {"IP Address":host_ip,
"Hostname":hostname,
"Username":host_username,
"Current Config":show_tacacs,
"Removal Config":removal_conf,
"Device Version":device_version}
show_tacacs_list.append(show_tacacs_dict)
#Create device list 2 for same config order as threading output
dev_list_dict = {"device_type":"cisco_ios",
"host":host_ip,
"username":host_username,
"password":host_password, # os.getenv("TACACS_DEVICE_PASSWORD")
"port":22,
"secret":"cisco", # os.getenv("ENABLE_DEVICE_PASSWORD")
"verbose":True}
dev_list_thread.append(dev_list_dict)
# 4. Close Connection
check_tacacs_cmds_iosxe.close_connection(connection)
q.task_done()
Check Function Multi Threading
check_error_ips = []
dev_ver_list = []
show_tacacs_list = []
dev_list_thread = []
q = queue.Queue()
# 1st CHECK!
try:
for thread_no in range(8):
worker = Thread(target=open_check_device_mp, args=(q, thread_no, dev_ver_list, show_tacacs_list, dev_list_thread, ), daemon=True)
worker.start()
for cisco_device in device_list:
q.put(cisco_device)
except:
error = cisco_device["host"]
print(f"There is an error connecting to {error}" )
print("Continuing...\n")
check_error_ips.append(cisco_device["host"])
q.join()
# Print IPs with errors
if check_error_ips != []:
for ip in check_error_ips:
print(f"Error connecting to {ip}")
with open (f"{basedir}/iosxe_TACACS_Check_Failure_IPs.txt", "a") as f:
f.write(ip + "\n")
# Write Output
write_file = f"{basedir}/iosxe_tacacs_check.csv"
check_tacacs_csv.write_csv(write_file,show_tacacs_list)