Nexus VPC Basic Configuration

This is to configure vPCs on Nexus 9000s in GNS3. For this lab I will repurpose my Nexus Multicast Lab.
I will need to add some extra links between mcast_nexus1 and 2 switches to accommodate the port-channel vPC link and the vPC keepalive link.

My topology is going to be a basic topology of four Nexus switches that will follow the Cisco configuration guide.

• vPC Configuration
  • Keepalive Configuration
  • Base vPC Configuration
  • Peer Link Configuration
  • Downstream Configuration
• Testing
  • Port Channel No LACP
  • Port Channel LACP Enabled
• Full Configuration

Configuring the vPC

For the configuration of the vPC there will be three links between Nexus1 and Nexus2.
1 is for the keepalive link
2 are a port channel for the peer link
The other links between Nexus1 and 2 down to Nexus3 and 4 will be the vPC downstream links.
I also have a generic GNS3 switch that is used as my management and will not be any part of the VPC. This is for SSH connectivity only.

Keepalive Configuration

Configured on both Nexus1 and 2 switches

Testing the new link from both Nexus1 and 2

Base vPC Configuration

To be applied on both Nexus1 and 2 switches

Next we can see the vPC role status. Currently nothing has been extablished as the peer links have not been configured. Below is output from Nexus1 to demonstrate the output.

Next apply the configuration to each switch, Nexus1 and 2 to create the vPC domain.

Now check the keepalive status. This brings up the first link that was configured. An example output is shown from Nexus1. Similar output will be on Nexus2

Peer Link Configuration

The peer link configuration is the port channel link between Nexus1 and 2. It creates the illusion of a single control plane between the two switches. This is interface is used for; vPC control traffic, orphaned port traffic and FHRP traffic. It is the single most important part of the vPC topology. There can be upto 32 active links and 8 in standby with LACP.

During my configuration in GNS3 I encountered an error when with either bringing up the vPC. If the interface speeds were left to auto, then the port channel wouldn’t fail to negotiate correctly and be in suspended mode.
If I hard set the interfaces to 100, 1000 or 10000 I would have an error such as the one below when configuring the port channel config on the physical ports.

To fix this I used the “force” command. This worked and allowed the configuration to be successfully applied. Seen below for my full configuration.

The output of show vpc now has has details!

Downstream Configuration

This next part is to configure the Nexus1 and 2 switches to allow individual network devices (routers/switches) to connect to the Nexus1 and 2 switches as part of a vPC. The downstream devices are unaware they are part of a vPC.

In this example the ports are configured for the downstream switch of mcast_nexus3.

Once setup we should see the vPC up on both Nexus1 and 2. Below is the output from Nexus1. Nexus2 is in the same state.

mcast_nexus3 now needs to be configured.

The port channel is now up for all three switches. There is no protocol on these port channels. The mode is just on.
Below is output from Nexus1. Nexus2 and 3 are similar.

Enable Peer Gateway

The peer gateway is configured on both peer switches. Nexus1 and 2.
The feature optimises the peer link and avoid the loss of FHRP traffic in a failure situation and forces the vPC peers to act as a gateway for packets destined to the peer’s MAC address.

Testing

For the testing I have moved mcast_server to be connected to Nexus3. There is also an SVI on VLAN 20 now that is configured on only Nexus1. No FHRPs are being used. The gateway is only on Nexus1 to begin with. I will run a series of tests that have mcast_server trying to reach the gateway in a working environment and in a failure environment.

No Port-channel Protocol – On

This example is not using LACP. The port channel is mode is “On”. There are no control message passing between the peer switches and Nexus3. Every switchport assumes that the port channel is up and functioning all of the time.

When all physical ports are in a working state there are no issues. mcast_server can reach the gateway all of the time via the link between Nexus3 and 1.

Shutting Down Nexus1 Eth1/2

This is the port that connect Nexus1 and Nexus3. It is the preferred path for the traffic going to the VLAN20 gateway.

With Eth1/2 on Nexus1 shutdown only, Nexus3 doesn’t see this from the port channel perspective as there is no negotiation protocol. There are no problems with the port channel.
This causes problem as I see it still sending frames down the link towards Nexus1.
What happens is;

• An ARP request goes out for 172.16.2.1 from the Windows mcast_server
• Nexus3 forwards this over to Nexus1, where nothing is heard back
  • The ARP request is sometimes sent 2/3 more times towards Nexus1
  • Nothing is returned as expected
  • The link is physically down on Nexus1, but logically up for port channel on Nexus3
• The last ARP request is sent via Nexus2 on the functioning port channel link
• An ARP response form 172.16.2.1, the gateway on Nexus1 is received
• ICMP traffic then works.
• But only for a few seconds until another ARP request is sent, repeating the process

Below are screenshots to explain the above. Note the timestamps that confirms the order.

I’m not sure why Windows is asking for the MAC address of the gateway every few seconds. I checked my own laptop and this is not sending ARP requests for the gateway every minute.
I have tried the same with a Cisco router just acting as a client. I found similar behaviour in the fact that traffic was sent straight to Nexus1 about half the time. I didn’t see the same ARP requests.
I’m not looking into this any further as I know the fix is to use LACP.

Port-channel Protocol – LACP

As show in the previous section, not using a port channel negotiation protocol can cause all sorts of issues that aren’t worth troubleshooting.
Using LACP will resolve the problems described in the previous section.

New Config to be applied to each switch vPC

Now all port channels are reconfigured and are using LACP

The same ping test to the gateway will be performed. Then shutting down Eth1/2 on Nexus1 switch to simulate a failure. As LACP is configured there should be a failure detected and the port channel between Nexus1 and 3 should be down.

As expected all traffic is fine when all interfaces are up.
When I shutdown Eth1/2 there is some traffic loss as the port channel on Nexus3 took around 30 seconds to go down. Then traffic was passed via Nexus2 with no issues.
Re-enabling the port causing no loss of traffic and caused the traffic to go the shortest path straight to Nexus1.

The reason that the port channel took 30 seconds to go down and caused traffic loss was due to the LACP rate which by default is 30 seconds.
I have changed this setting from “normal” to “fast” which reduces this to 1 second on both Nexus1 and Nexus3 interfaces

There are now a lot more LACP frames in the Wireshark captures. However I only lost one ICMP frame on the switch and the port channel on Nexus3 was down immediately.

Complete Configuration

As the configuration has been split up in a step by step guide I have added a full config. In this I have used trunk mode on the port channels instead of access mode.