DHCP snooping -Tracking MACs and relevant IPs from the switch

sh ip arp

This will help map the IPs to MACs, very helpful in vPC environments.

sh mac address table

VNTag- 802.1br and Instantiation of Virtual Interfaces

VN-Tag

VN-Tag refers to the format of the information tag inserted into each frame exchanged between the Cisco fabric extenders and the Nexus parent switch. It helps create a virtual link/path between different vif (virtual interfaces) residing within the single host and tied to single/multiple VMs.

Basically, it used by the NIV adapter to tag egressing packets which will be passed on to the FI through the FEX (VNTag is transparent to FEX).

For a non-NIV adapter, the switch applies the VNTag to egressing packets.

In short, VNTAG/VN-Link is MAC-in-MAC encapsulation to enable virtualization of network traffic

VIFs

Virtual Interfaces (VIFs) are used for virtualized Ethernet (Veths) and FC (Vfc) interfaces.

They can also help us identify the origin server to which these vifs belong to.

On SP association, the VIFs are created on the FI (samdb) and correspond to frame-level tags assigned to blade/rack physical adapters.

A 6-byte tag(VN-Tag) is inserted into the frame at egress on the server port by the adapter firmware and helps identify the interface. Every VIF (Veth or Vfc) will have a VN-Tag associated with it.

Each EthX/Y/Z interface (northbound ports) of a blade/rack server will have one or more VIFs associated with it.

VN-TAG FORMAT

EtherType [16-bits] = 0x8926
Destination Bit [1-bit] – Indicates which direction the frame is flowing.
Pointer bit [1-bit] – indicated that a vif_list_id is included in the tag.
Destination VIF – [14-bits] – Identifies the destination port.
Looped – Identifies the source vNIC, ment to identify multicast frames to ensure it is not forwarded back to where it originated.
Reserved [2-bits] – For future use.
Source VIF [12-bits] – vif_id of the downstream port.
Version [2-bits] – Set to 0

VNTag-802.1br - IEEE

A Typical UCS-N1K deployment with upstream N5K vPC pair.

Differentiating the Physical and Virtual Interfaces

Source :- http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/unified-computing-system/116277-maintain-mc-ucs-00.html

Diff b/w fault tolerance/load balancing Link Aggregation

The list that follows details the three options—a, b, and c—shown in Figure 2-1:

• Fault tolerance— One port (or NIC) is active (receives and transmits traffic) while the other ports are in standby (these ports don’t forward or receive traffic). When the active port fails, one of the ports that was previously in standby takes over with the same MAC address. This option is depicted as option a in Figure 2-1.

• Load balancing— Only one port receives traffic; all the ports transmit traffic. If the receive port fails, a new port is elected for the receive function. In Figure 2-1, this is option b.

• Link aggregation— A number of ports (NICs) form a bundle that logically looks like a single link with a bandwidth equivalent to the sum of the bandwidth of each single link. Cisco calls this configuration an EtherChannel. In Figure 2-1, this is option c.

  
  

VSM (Virtual Switching Module)

VSM is used to configure following attributes :-

1. VLANs
2. ACL, PVLANs etc
3. Netflow, ERSPAN
4. QoS
5. Physical NIC port configuration

What is Nexus 1000v

1. Nexus 1000V (N1kv) is a distributed edge virtual switch leveraging NX-OS which extends the switch to the ESX host and essentially sits b/w the host & the upstream switch. 
2. It can be installed as a OVM (VM) on any host or as a physical appliance (Nexus1010).
3. It has a sup (VSM) which controls multiple line cards in a sense (VEM) installed on a single host.
4. Each host can only support one VEM and VSM can manage a max of 64 VEMs.

Below we have a 2 VSM modules connected to 2 upstream switches which are then connected bunch of VEMs installed in individual ESX hosts.

VSM allows configuration of::-

1. VLANs
2. ACL, PVLANs etc
3. Netflow, ERSPAN
4. QoS
5. Physical NIC port configuration

A vethernet interface is defined for each VMNIC on the VM and an ethernet interface is defined for each physical NIC port on the ESX host.

1. Each ethernet interface is assigned an uplink port profile to provide uplink(external) connectivity.
2. The n/w admin can create port profiles on the VSM (with some/all of above attributes) and assign a vethernet port to any one of these port-profiles.

The design of N1Kv prevents loops by tagging/recognizing/switching that veth traffic to VEMs and ethernet traffic to uplink port of the physical NIC.

Why Nexus 1000v

Why N1kV :- With an ESX host and it's DVS allowing VMNICs on individual VMs to communicate with internal and external networks, we had following issues :-

A. Using single port or a port channel from DVS to upstream switch:-

1. VMs creating bridges and unintended loops.
2. Unintended VM to VM communication.
3. Even though VMNICs could be on separate internal networks, the uplink is common through the DVS and hence the all VM traffic would traverse through the upstream switch regardless of the requirement. 
4. VMs on wrong VLANs leading to no external/upstream connectivity.

B. Using Multiple ports with separate VLANs allowed on upstream switch per port on ESX host. 

1. This required high number of physical ports on the switches as well as each hosts (multiple NICs, multi-port NICs)
2. The n/w admin still had very little control over the DVS config and the VLANs on each of the VMNICs.

Setting/Backing up a Cisco UCS Fabric Interconnect - Logical/System configuration delete

Wiping out Logical and System Configuration from the switch

FI-A(local-mgmt)# erase configuration  - This erases all access configuration on the FI keeping the samdb in tact.

FI-A(local-mgmt)# erase samdb           - This will only remove the Logical Configuration on the FI (SP, VLAN, VSAN db, IP, MAC Pools etc.) while retaining the system configuration so that when the switches reboots after clearing the database, it will still be accessible using the pre-configured IPs


P.S. :- The 'erase configuration' command need to be executed on both switches separately. However, an 'erases samdb' command  executed on the primary should trigger an 'erase samdb' on the secondary switch automatically.

FI-A(local-mgmt)#

Backup Configuration

1. Logical :- An XML file that includes all logical configuration settings such as service profiles, VLANs, VSANs, pools, and policies. You can use the file generated from this backup to import these configuration settings to the original fabric interconnect or to a different fabric interconnect.
2. System :- An XML file that includes all system configuration settings such as usernames, roles, and locales. You can use the file generated from this backup to import these configuration settings to the original fabric interconnect or to a different fabric interconnect
3. All :- An XML file that includes all system and logical configuration settings. You can use the file generated from this backup to import these configuration settings to the original fabric interconnect or to a different fabric interconnect.
4. Full :- A binary file that includes a snapshot of the entire system. You can use the file generated from this backup to restore the system during disaster recovery. This file can restore or rebuild the configuration on the original fabric interconnect, or recreate the configuration on a different fabric interconnect. You cannot use this file for an import.

NPV and NPIV - Port modes

Configuring SAN port channel between MDS and UCS Fabric Interconnect 6xxx

With FIs in EHM connected to upstream MDS in npiv mode, all southbound hosts are logging into FI (npv) which is acting as a FCF, the FI port is NP as all N ports are proxied. The upstream MDS port is F port without any knowledge of npv or npiv. In a multi tenant environment utilizing different VSANs, a trunk is needed between the NP and F port which is accomplished by using the Fport trunking feature on both devices.

Configuration needed on MDS

feature npiv
feature fport-channel-trunk

interface port-channel X
switchport mode F
switchport rate-mode dedicated
switchport trunk mode on
switchport trunk allowed vsan 1,3-4,14
channel mode active

(*This way vsan 1 is native for pc 10)
vsan database
vsan 1 interface port-channel 10                                   

(*Adding ports to PC and reconfiguring ports to ensure completeness)
interface fc1/1-2                      
switchport mode F
switchport rate-mode dedicated
switchport trunk mode on
switchport trunk allowed vsan 1,3-4,14
channel-group 10 force

Shutdown the port channel on the MDS for now.

Configuration needed on MDS

1. Enable FC port Trunking on FIs.
2. Add the member interfaces on the FI and set the native VSAN to 1. 
3. Bring up the PC on FI followed by the PC on the MDS.

The PC should be up and the member ports should be trunking now. If not, check the Troubleshooting reference document below.


References :-
Configuring FC port channels
Configuring Trunking
Troubleshooting

Cisco UCS - vNIC/vHBA Allocation Scheme

vNIC/vHBA Allocation Scheme

Vnic to mezz card placement happens in two stages:

1. vCon to Adaptor Placement (Round Robin & Linear)
2. vNIC to vCon Placement

1. vCon to Adaptor Placement (Round Robin & Linear)

Round-Robin (the only option with UCSM version 1.4.1 and earlier)

No of Mezz Cards	VCon-1	VCon-2	VCon-3	VCon-4
1	1	1	1	1
2	1	2	1	2
3	1	2	3	2
4	1	2	3	4

• In a server with 1 pcie adaptor card, all the 4 VCons point to the same adaptor card.
• In a server with 2 pcie adaptor cards, VCon-1 and VCon-3 point to adaptor-1, VCon-2 and VCon-4 point to adaptor-2.
• With 3 pcie cards, VCon-1, VCon-2 and VCon-3 points to the corresponding adaptors 1, 2 and 3. VCon-4 points to adaptor-2.
• With 4 pcie cards, VCon-1, 2, 3, and 4 points to adaptors 1, 2, 3 and 4 in order.

In the above example, the adaptor ids are not the pcie slot numbers of the adaptors cards, but the id assigned to them during server discovery. VCon to adaptor placement is not dependent on the pcie slot number of the adaptor card.

Note that in case of 3 adaptors, 4^th VCon gets mapped to adaptor-2 instead of adaptor-1. This will change with future implementations of VCons.

Linear (available with UCSM 1.4.2 onwards)

Round Robin causes problems with vnic pcie orders as newer adapter are added to the server on a later stage. For solving this ordering issue, placement algorithm needs to get changed from round-robin to linear. Example,

No. of vnics placed per VCon

VCon-1 vnic-1, vnic-2

VCon-2 vnic-3, vnic-4

VCon-3 vnic-5, vnic-6

VCon-4 vnic-7, vnic-8

In order for the host to see the vnics in expected orders i.e. vnic-1, 2, 3, 4, 5, 6, 7, 8 on the pcie bus and for VCons to get properly distributed across adaptor cards, placement would happen like this.

VCon-1, VCon-2 goes to adaptor-1.

VCon-3, VCon-4 goes to adaptor-2.

This algorithmic change poses some backward compatibility issues. Upgrading from 1.4.1 to the release which will address this problem will cause host reboots because vnic/vhba placements will change. In order to minimize the impact the behavior would be made like the following:

Service profiles created before the release which will address this issue will keep using the old vcon-mezz placement algorithm (Round Robin) for their lifetime. This will preserve vnic placements on upgrade hence their servers will NOT reboot on upgrade to the newer release. New service profiles will use the new algorithm.

All the new service profiles created in the newer release will cause server reboots on downgrading to lower releases which have the old vcon-mezz placement algorithm.

On upgrade to the newer release as the service profiles created in lower releases will keep using the old algorithm, vnic placement would be preserved for them hence will NOT cause server reboots on downgrade.

2. vNIC to vCon Placement

1. Explicit assignment by the user. 
2. Implicit assignment i.e. Depending on system to do the assignment. 

Issues with using Explicit VCon Assignment

Explicit assignments of vnic/vhba to VCon-3 and VCon-4 results in pcie ordering issue. In UCSM 1.4.1 & earlier releases, VCon to adaptor placement happens in a round-robin order (See Below Table ).

Example: In a server with two adaptors VCon-1 and VCon-3 will point to adaptor-1 and VCon-2 and VCon-4 will point to adaptor-2.

	Round-Robin		Linear
vNic	Desired VCon	Actual VCon	Desired Order	Actual Order
vnic-1	1	1	1	1
vnic-2	2	2	2	1
vnic-3	3	1	3	2
vnic-4	4	2	4	2

Assuming host scans first adaptor-1 then adaptor-2, the vNics will get placed on the pcie bus in the order vnic-1, vnic-3, vnic-2, vnic-4. Hence pcie orders are different then what user requested. This case is rare though and can occur only in the following scenarios:


1. With 2 adaptors, User explicitly assigns VNics to 4 different VCons.
2. If a service profile is migrated from a blade with higher number of adaptors to a blade with lower number of adaptors.

The workaround is to let system do the placement. System has the intelligence to do the placement only on those VCons which have a corresponding adaptor with the same id. Hence in a server with 2 adaptors system will not place VNics on VCon-3 and VCon-4. In the same way on a server with 3 adaptors, system will not place VNics on VCon-4.


References:- VCon Specs and http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-1/b_UCSM_GUI_Configuration_Guide_2_1/b_UCSM_GUI_Configuration_Guide_2_1_chapter_011110.html#concept_FA2F616B0B9A47DB9FEEAE63D5995CDE

Table 1 vCon to Adapter Placement Using the Round Robin Mapping Scheme

Number of Adapters	vCon1 Assignment	vCon2 Assignment	vCon3 Assignment	vCon4 Assignment
1	Adapter1	Adapter1	Adapter1	Adapter1
2	Adapter1	Adapter2	Adapter1	Adapter2
3	Adapter1	Adapter2	Adapter3	Adapter2
4	Adapter1	Adapter2	Adapter3	Adapter4

Round Robin is the default mapping scheme.

Table 2 vCon to Adapter Placement Using the Linear Ordered Mapping Scheme

Number of Adapters	vCon1 Assignment	vCon2 Assignment	vCon3 Assignment	vCon4 Assignment
1	Adapter1	Adapter1	Adapter1	Adapter1
2	Adapter1	Adapter1	Adapter2	Adapter2
3	Adapter1	Adapter2	Adapter3	Adapter3
4	Adapter1	Adapter2	Adapter3	Adapter4

If using a vCon policy with two adapters in the B440, please be aware of the following mapping.

• vCon 2 to adapter 1 maps first
• vCon 1 to adapter 2 maps second 

More Details at 

https://techzone.cisco.com/t5/B-Series/Cruz-based-VIC-adapters-amp-PCI-addressing/ta-p/816526

https://supportforums.cisco.com/discussion/12720616/what-admin-host-port-ucs-manager

https://supportforums.cisco.com/discussion/12565226/vic-1340-pci-placement-issues

https://jeremywaldrop.wordpress.com/2016/02/15/ucs-vic-13401380-pci-placement/

Etherchannel

On a cat6k, following fields are used for etherchannel hash:-

Source XOR destination MAC

Source XOR destination IPv4

Source XOR destination IPv6

Show etherchannel load-balance hash-result interface po1 ip x.x.x.x y.y.y.y

(X.x.x.x - source IP/MAC

Y.y.y.y - destination IP/MAC)

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-virtual-switching-system-1440/109638-vss-pf-tshoot.html

Building a vPC domain

Configuration Steps:-
(Order does Matter !!)

1. Define domains.
2. Establish Peer Keep-alive connectivity.
3. Create Peer Link.
4. Reuse port-channels and Create vPCs.
5. Ensure the configurations are consistent.

vPC Domains:-
vPCs domain ID used to assign unique vPC system MAC address.
vPC Domain IDs should be unique within the layer 2 domain.

Connected hosts/switches to vPC domain see the vPC system MAC instead of the local MAC of the vPC member.

Perr Keep-Alive
Heartbeat between vPC peers.
Active-Active detection (in case vPC Peer-Link is down)
UDP message on port 3200, 96 bytes long (32 byte payload), includes version, time stamp, local & remote IPs and domain IDs.
Default timers: interval 1sec/ timeout 5 sec.

Ensure vPC PKL messages should NOT be routed over the vPC peer link !! (Peer-Link only comes up after we configure the Peer-Keep Alive)
The Peer Keep-Alive is best configured as a dedicated 1/10 GE port. or can be sent along with management traffic on mgmt0. As a last resort, it can be routed over L3 infrastructure.

Peer Link
Standard 802.1q trunk which carries CFS (Cisco Fabric Services) messages.
Carries flooded traffic from vPC peer, STP BPDUs, HSRP Hellos, IGMP updates etc.
Peer-Link must be 10/40/100 GE.
Peer-Link must be point to point link.

Use min 2 x 10GE ports
Use 2 separate cards for resiliency.
10GE ports in dedicated mode for oversubscribed modules.




Differences from STP:-
vPC has an independent Control Plane and Forwarding Plane

For Best Practices
References:- http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf 
Quick Config guide:- https://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/configuration_guide_c07-543563.html 

Difference - Ethanalyzer and SPAN

The Switched Port Analyzer (SPAN) feature—sometimes called port mirroring or port monitoring—selects network traffic for analysis by a network analyzer. The network analyzer can be a Cisco SwitchProbe, a Fibre Channel Analyzer, or other Remote Monitoring (RMON) probes.

Ethanalyzer is a Cisco NX-OS protocol analyzer tool based on the Wireshark open source code. This tool is a command-line version of Wireshark that captures and decodes packets. You can use Ethanalyzer to troubleshoot your network and analyze the control-plane traffic.

Ethanalyzer and SPAN

Ethanalyzer is a tool that collects frames that are destined to, or originate from, the Nexus 5000 control plane. Node to switch or switch to switch traffic can be seen with this tool.

SPAN is a feature whereby frames that are transient to the switch are copied to a second port for analysis. Node to switch or node to node traffic can be seen via this method.

The main difference between the Ethanalyzer and SPAN feature is that the Ethanalyzer captures control-plane traffic, while SPAN captures all traffic.Of course for remote span (across layer3), we use ERSPAN.

Netflow is another snipping protocol used to collect granular details between 2 vnics/hosts/ports or a vlan.

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/N5K_Troubleshooting_Guide/n5K_ts_oview.html 

More on this later....

Ok, got a chance to go over this topic with some more information on ERSPAN

http://packetpushers.net/erspan-new-favorite-packet-capturing-trick/

Uses Capture Filter -->ip proto 0x2f (in wireshark) - Any traffic with a GRE header (ERSPAN traffic)