FabricPath

Topology

clip_image002[4]

Hardware Review

Nexus 7K

 

First let’s review the hardware used for this lab :

N7K-1-1# show module

Mod  Ports  Module-Type                         Model              Status

  —–  ———————————– —————— ———-

1    32     1/10 Gbps Ethernet Module           N7K-F132XP-15      ok

3    48     10/100/1000 Mbps Ethernet Module    N7K-M148GT-11      ok

5    0      Supervisor module-1X                N7K-SUP1           active *

 

Xbar Ports  Module-Type                         Model              Status

  —–  ———————————– —————— ———-

1    0      Fabric Module 1                     N7K-C7010-FAB-1    ok

2    0      Fabric Module 1                     N7K-C7010-FAB-1    ok

3    0      Fabric Module 1                     N7K-C7010-FAB-1    ok

 

 

So we have F1 card and M1 card, this is a chassis in mixed mode. FabricPath is only supported on F1/F2 cards.

N5K

N5K-p1-1# sh mod

Mod Ports  Module-Type                      Model                  Status

— —–  ——————————– ———————- ————

1    32     O2 32X10GE/Modular Universal Pla N5K-C5548UP-SUP        active *

3    0      O2 Non L3 Daughter Card          N55-DL2                ok

 

The Nexus 5500 is a 32 ports 10GE.

LAB

VLAN Configuration

First let’s configure a very simple Layer 2 topology with the VLAN 100 : The VLAN will be allowed on trunks going between Nexus 5500 and Nexus 7000 :

N7K-1-1(config)# vlan 100

N7K-1-1(config-vlan)# exit

N7K-1-1(config)# int e1/1-8

N7K-1-1(config-if-range)# switchport mode trunk

N7K-1-1(config-if-range)# switchport trunk allowed vlan 100

N7K-1-1(config-if-range)# no sh

 

 

N5K-p1-1(config)# vlan 100

N5K-p1-1(config-vlan)# int e1/1-8

N5K-p1-1(config-if-range)# switchport mode trunk

N5K-p1-1(config-if-range)# switchport trunk allowed vlan 100

N5K-p1-1(config-if-range)# no sh

 

N7K-1-1# sh span vlan 100

 

VLAN0100

  Spanning tree enabled protocol rstp

  Root ID    Priority    32868

             Address     0024.98e8.01c2

             This bridge is the root

             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

 

  Bridge ID  Priority    32868  (priority 32768 sys-id-ext 100)

             Address     0024.98e8.01c2

             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

 

Interface        Role Sts Cost      Prio.Nbr Type

—————- —- — ——— ——– ——————————–

Eth1/1           Desg FWD 2         128.129  P2p

Eth1/2           Desg FWD 2         128.130  P2p

Eth1/3           Desg FWD 2         128.131  P2p

Eth1/4           Desg FWD 2         128.132  P2p

Eth1/5           Desg FWD 2         128.133  P2p

Eth1/6           Desg FWD 2         128.134  P2p

Eth1/7           Desg FWD 2         128.135  P2p

Eth1/8           Desg FWD 2         128.136  P2p

 

N5K-p1-1# sh span vlan 100

 

VLAN0100

  Spanning tree enabled protocol rstp

  Root ID    Priority    32868

             Address     0024.98e8.01c2

             Cost        2

             Port        129 (Ethernet1/1)

             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

 

  Bridge ID  Priority    32868  (priority 32768 sys-id-ext 100)

             Address     547f.ee22.81fc

             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

 

Interface        Role Sts Cost      Prio.Nbr Type

—————- —- — ——— ——– ——————————–

Eth1/1           Root FWD 2         128.129  P2p

Eth1/2           Altn BLK 2         128.130  P2p

Eth1/3           Altn BLK 2         128.131  P2p

Eth1/4           Altn BLK 2         128.132  P2p

Eth1/7           Desg FWD 2         128.135  P2p

 

Nothing complicated here !

What we see is that STP kicks in to avoid any layer 2 loops. This result as usual as blocking some ports. Know from a bandwidth and availability point of view it means we lose 3 10GE uplinks and we are prone to RSTP timers.

RSTP is the default mode on Nexus switches.

FabricPatch configuration

Now let’s configure FabricPath. The basic configuration is very simple.

First we need to check the licence, FabricPath doesn’t come for free and the ENHANCED_LAYER2 feature is needed :

N5K-p1-2# sh license usage

Feature                      Ins  Lic   Status Expiry Date Comments

                                 Count

——————————————————————————–

FCOE_NPV_PKG                  No       Unused            

FM_SERVER_PKG                 No       Unused            

ENTERPRISE_PKG                Yes      Unused Never      

FC_FEATURES_PKG               Yes      Unused Never      

VMFEX_FEATURE_PKG             No       Unused            

ENHANCED_LAYER2_PKG           No       Unused             Grace 99D 10H

LAN_BASE_SERVICES_PKG         No       Unused            

LAN_ENTERPRISE_SERVICES_PKG   Yes      Unused Never      

 

N7K-1-1# sh license usage

Feature                      Ins  Lic   Status Expiry Date Comments

                                 Count

——————————————————————————

MPLS_PKG                      No       Unused            

STORAGE-ENT                   No       Unused            

ENTERPRISE_PKG                No       Unused            

FCOE-N7K-F132XP               No    0   Unused            

ENHANCED_LAYER2_PKG           Yes      Unused Never      

SCALABLE_SERVICES_PKG         No       Unused            

TRANSPORT_SERVICES_PKG        Yes      Unused Never      

LAN_ADVANCED_SERVICES_PKG     Yes      Unused Never      

LAN_ENTERPRISE_SERVICES_PKG   Yes      Unused Never      

——————————————————————————

 

By default the features comes with a grace period, so if you do not have purchased the right licence you still can run the feature for a limited 120 day period.

On Nexus 7000 we need to activate the feature and then turn the VLAN into FabricPath mode. What this change compared to CE mode (Classical Ethernet) is that we switch on a new MAC Learning paradigm where the Source MAC Address of an incoming frame is learned only if the Destination MAC is already known. This is Conversational MAC Learning.

This is the major feature of FabricPath that enable the scalability of the protocol. From a practical point of view, it means that you only learn MACs from systems that have a bidirectionnal communication  and you will not learn the MAC addresses of remote systems that do not talk with system that you host (« you » being the device running FabricPath).

N7K-1-1(config)# feature-set fabricpath

N7K-1-1(config)# vlan 100

N7K-1-1(config-vlan)# mode fabricpath

 

FabricPath will randomly assign a Switch ID to the device. The Switch ID has the same purpose as the Router ID in OSPF, the goal is to allow ISIS to build and LSDB and uniquely identify the device.

 

The Switch ID can be statically assigned but in this case FabricPath will lose the capability to automagically resolve the potential conflicts.

 

N7K-1-1(config)# show fabricpath switch-id

                        FABRICPATH SWITCH-ID TABLE

Legend: ‘*’ – this system

=========================================================================

SWITCH-ID      SYSTEM-ID       FLAGS         STATE    STATIC  EMULATED

———-+—————-+————+———–+——————–

*1398        0024.98e8.01c2    Primary     Confirmed    No      No

Total Switch-ids: 1

 

Next step is to enable the interfaces for the FabricPath mode. This will identify which interfaces needs to encapsulates/decapsulates trafic.

FabricPath is NOT Ethernet, these interfaces will NOT be able to switch standard Ethernet Frame after the switch in FabricPath mode.

 

N7K-1-1(config)# int e1/1-8

N7K-1-1(config-if-range)# switchport mode fabricpath

 

2013 Mar  2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/8 up in fabricpath topology 0

2013 Mar  2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/8 is up in mode fabricpath

2013 Mar  2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/7 is up in mode fabricpath

2013 Mar  2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/7 up in fabricpath topology 0

2013 Mar  2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/6 is up in mode fabricpath

2013 Mar  2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/5 is up in mode fabricpath

2013 Mar  2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/4 is up in mode fabricpath

2013 Mar  2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/3 is up in mode fabricpath

2013 Mar  2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/6 up in fabricpath topology 0

2013 Mar  2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/2 is up in mode fabricpath

2013 Mar  2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/5 up in fabricpath topology 0

2013 Mar  2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/4 up in fabricpath topology 0

2013 Mar  2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/3 up in fabricpath topology 0

2013 Mar  2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/2 up in fabricpath topology 0

2013 Mar  2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode fabricpath

2013 Mar  2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/1 up in fabricpath topology 0

 

Now, STP has vanished for VLAN 100 because FabricPath is activated. This is another key point, unlike VPC STP does not run on the top of FabricPath.

 

The looped design is managed by the new mechanisms of FabricPath, the Conversational Learning, the fact that FabricPath knows the entire topology and the TTL value that is part of the FabricPath header.

 

N7K-1-1(config-if-range)# do sh span vlan 100

Spanning tree instance(s) for vlan does not exist.

Same operations on Nexus 5500, the only slight difference is that we need to install the feature before activating it :

 

N5K-p1-2# sh span vlan 100

 

VLAN0100

  Spanning tree enabled protocol rstp

  Root ID    Priority    32868

             Address     0005.73ba.637c

             This bridge is the root

             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

 

  Bridge ID  Priority    32868  (priority 32768 sys-id-ext 100)

             Address     0005.73ba.637c

             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

 

Interface        Role Sts Cost      Prio.Nbr Type

—————- —- — ——— ——– ——————————–

Eth1/1           Desg FWD 2         128.129  P2p

Eth1/3           Desg FWD 2         128.131  P2p

Eth1/4           Desg FWD 2         128.132  P2p

Eth1/5           Desg FWD 2         128.133  P2p

Eth1/6           Desg FWD 2         128.134  P2p

Eth1/7           Desg FWD 2         128.135  P2p

Eth1/8           Desg FWD 2         128.136  P2p

 

 

N5K-p1-1(config)# install feature-set fabricpath

N5K-p1-1(config)# feature-set fabricpath

N5K-p1-1(config)# vlan 100

N5K-p1-1(config-vlan)# mode fabricpath

N5K-p1-1(config-vlan)# int e1/1-8

N5K-p1-1(config-if-range)# switchport mode fabricpath

 

2013 Mar  2 12:04:35 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE:  isis_fabricpath-default [3736]  P2P adj L1 0024.98e8.01c2 over Ethernet1/3 – DOWN (New) on MT-0

2013 Mar  2 12:04:35 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE:  isis_fabricpath-default [3736]  P2P adj L1 0024.98e8.01c2 over Ethernet1/3 – UP on MT-0

2013 Mar  2 12:04:36 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE:  isis_fabricpath-default [3736]  P2P adj L1 0024.98e8.01c2 over Ethernet1/2 – DOWN (New) on MT-0

2013 Mar  2 12:04:36 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE:  isis_fabricpath-default [3736]  P2P adj L1 0024.98e8.01c2 over Ethernet1/2 – UP on MT-0

2013 Mar  2 12:04:36 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE:  isis_fabricpath-default [3736]  P2P adj L1 0024.98e8.01c2 over Ethernet1/1 – DOWN (New) on MT-0

2013 Mar  2 12:04:36 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE:  isis_fabricpath-default [3736]  P2P adj L1 0024.98e8.01c2 over Ethernet1/1 – UP on MT-0

2013 Mar  2 12:04:37 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE:  isis_fabricpath-default [3736]  P2P adj L1 N7K-1-1 over Ethernet1/4 – DOWN (New) on MT-0

2013 Mar  2 12:04:37 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE:  isis_fabricpath-default [3736]  P2P adj L1 N7K-1-1 over Ethernet1/4 – UP on MT-0

Now we can check the new type of the interfaces activated for FabricPath :

 

N5K-p1-1# sh int brief

 

——————————————————————————

Ethernet      VLAN    Type Mode   Status  Reason Speed     Port

Interface                                                                   

——————————————————————————

Eth1/1        1       eth  f-path up      none                        10G(D) –

Eth1/2        1       eth  f-path up      none                        10G(D) –

Eth1/3        1       eth  f-path up      none                        10G(D) –

Eth1/4        1       eth  f-path up      none                        10G(D) –

Eth1/5        1       eth  f-path up      none                        10G(D) –

Eth1/6        1       eth  f-path up      none                        10G(D) –

Eth1/7        1       eth  f-path up      none                        10G(D) –

Eth1/8        1       eth  f-path up      none                        10G(D) –

 

Like in OSPF we can check the status of the interface and the various values like the network type, the status, the circuit, the MTU and the Metric.

 

By default the reference metric is 400000M  which means that 10GE interfaces have a metric of 40 (400000M / 10000M). Of course this value can be tuned if necessary.

 

N7K-1-1# sh fabricpath isis interface brief

Fabricpath IS-IS domain: default

Interface    Type  Idx State        Circuit   MTU  Metric  Priority  Adjs/AdjsUp

——————————————————————————–

Ethernet1/1  P2P   4     Up/Ready   0x01/L1   1500 40      64          1/1

Ethernet1/2  P2P   1     Up/Ready   0x01/L1   1500 40      64          1/1

Ethernet1/3  P2P   2     Up/Ready   0x01/L1   1500 40      64          1/1

Ethernet1/4  P2P   3     Up/Ready   0x01/L1   1500 40      64          1/1

Ethernet1/5  P2P   5     Up/Ready   0x01/L1   1500 40      64          0/0

Ethernet1/6  P2P   6     Up/Ready   0x01/L1   1500 40      64          0/0

Ethernet1/7  P2P   7     Up/Ready   0x01/L1   1500 40      64          0/0

Ethernet1/8  P2P   8     Up/Ready   0x01/L1   1500 40      64          0/0

 

The detail of protocol can be obtained, this is very similar (again) to OSPF

 

N7K-1-1# show fabricpath isis

 

Fabricpath IS-IS domain : default

  System ID : 0024.98e8.01c2  IS-Type : L1

  SAP : 432  Queue Handle : 11

  Maximum LSP MTU: 1492

  Graceful Restart enabled. State: Inactive

  Last graceful restart status : none

  Metric-style : advertise(wide), accept(wide)

  Start-Mode: Complete [Start-type configuration]

  Area address(es) :

    00

  Process is up and running

  CIB ID: 4

  Interfaces supported by Fabricpath IS-IS :

    Ethernet1/1

    Ethernet1/2

    Ethernet1/3

    Ethernet1/4

    Ethernet1/5

    Ethernet1/6

    Ethernet1/7

    Ethernet1/8

  Level 1

  Authentication type and keychain not configured

  Authentication check specified

  MT-0 Ref-Bw: 400000

  Address family Swid unicast :

    Number of interface : 8

    Distance : 115

  L1 Next SPF: Inactive

 

The adjacencies can be check with the following command, note that FabricPath build L1 adjacencies only so from an ISIS perspective this is a flat L1 intra-area.

 

  N7K-1-1# show fabricpath isis adjacency

Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:

System ID       SNPA            Level  State  Hold Time  Interface

N5K-p1-1        N/A             1      UP     00:00:30   Ethernet1/1

N5K-p1-1        N/A             1      UP     00:00:22   Ethernet1/2

N5K-p1-1        N/A             1      UP     00:00:28   Ethernet1/3

N5K-p1-1        N/A             1      UP     00:00:31   Ethernet1/4

 

The switch IDs known to participate in the FabricPath domain can be checked. This is because the ISIS LSDB is able to identify all the switch IDs, the entire topology is known:

 

N7K-1-1# show fabricpath switch-id

                        FABRICPATH SWITCH-ID TABLE

Legend: ‘*’ – this system

=========================================================================

SWITCH-ID      SYSTEM-ID       FLAGS         STATE    STATIC  EMULATED

———-+—————-+————+———–+——————–

*1398        0024.98e8.01c2    Primary     Confirmed    No      No

2857        547f.ee22.81fc    Primary     Confirmed     No      No

Total Switch-ids: 2

 

Tips and Tricks, on Nexus there is a global command that can be used to turn all switchports into FabricPath interfaces.

 

N7K-1-2(config)# system default switchport fabricpath

 

FabricPath switch IDs can be manually assigned. There is no disruption of service when changing the switch ID because FabricPath runs some kind of interim switch ID will it ensure that the old one is flushed out the database.

 

N5K-p1-2(config)# fabricpath switch-id 52

 

The entire topology can be seen from any devices in the FabricPath domain. What we see is that FabricPath builds two trees.

 

The first one is built for unknown unicast, broadcast and multicast.

The second one is built for multicast.

Standard known unicast doesn’t use a tree, the frame will be moved based on the FabricPath route.

 

The first tree is built by electing a root, very much like STP the root is elected based on :

          Highest Root Priority

          Highest System ID

          Highest Switch ID

 

One the first tree is build, an Ftag is assigned to it.

 

Then the first elected root elects the second one which will be assigned a second Ftag.

 

N7K-1-1# show fabricpath isis topology summary

Fabricpath IS-IS domain: default FabricPath IS-IS Topology Summary

MT-0

  Configured interfaces:  Ethernet1/1  Ethernet1/2  Ethernet1/3  Ethernet1/4  Ethernet1/5  Ethernet1/6  Ethernet1/7  Ethernet1/8

  Number of trees: 2

    Tree id: 1, ftag: 1, root system: 547f.ee22.81fc, 51

    Tree id: 2, ftag: 2, root system: 0024.98e8.01c2, 71

 

The root election can be done on a predictive manner by changing the root-priority inside the fabricpath domain:

 

N7K-1-1(config)# fabricpath domain default

N7K-1-1(config-fabricpath-isis)# root-priority ?

  <1-255>  Root priority value per topology

           *Default value is 64

 

N7K-1-1(config-fabricpath-isis)# root-priority 255

 

N7K-1-2(config)# fabricpath domain default

N7K-1-2(config-fabricpath-isis)# root-priority ?

  <1-255>  Root priority value per topology

           *Default value is 64

 

N7K-1-2(config-fabricpath-isis)# root-priority 254

 

N7K-1-1# sh fabricpath isis topology summary

Fabricpath IS-IS domain: default FabricPath IS-IS Topology Summary

MT-0

  Configured interfaces:  Ethernet1/1  Ethernet1/2  Ethernet1/3  Ethernet1/4  Ethernet1/5  Ethernet1/6  Ethernet1/7  Ethernet1/8

  Number of trees: 2

    Tree id: 1, ftag: 1, root system: 0024.98e8.01c2, 71

    Tree id: 2, ftag: 2, root system: 001b.54c2.67c2, 72

 

The entire tree can be seen at any point of the FabricPath domain:

 

N7K-1-1# show fabricpath isis topology 0 trees

Fabricpath IS-IS domain: default

Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id

 

MT-0

Topology 0, Tree 1, Swid routing table

51, L1

 via Ethernet1/4, metric 40

52, L1

 via Ethernet1/8, metric 40

72, L1

 via Ethernet1/8, metric 80

 

Topology 0, Tree 2, Swid routing table

51, L1

 via Ethernet1/4, metric 40

52, L1

 via Ethernet1/4, metric 40

72, L1

 via Ethernet1/4, metric 0

 

 

 N5K-p1-1# show fabricpath isis topology 0 trees

Fabricpath IS-IS domain: default

Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id

 

MT-0

Topology 0, Tree 1, Swid routing table

52, L1

 via Ethernet1/4, metric 40

71, L1

 via Ethernet1/4, metric 0

72, L1

 via Ethernet1/4, metric 80

 

Topology 0, Tree 2, Swid routing table

52, L1

 via Ethernet1/8, metric 40

71, L1

 via Ethernet1/4, metric 80

72, L1

 via Ethernet1/8, metric 0

 

 N5K-p1-2# show fabricpath isis topology 0 trees

Fabricpath IS-IS domain: default

Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id

 

MT-0

Topology 0, Tree 1, Swid routing table

51, L1

 via Ethernet1/8, metric 40

71, L1

 via Ethernet1/8, metric 0

72, L1

 via Ethernet1/4, metric 80

 

Topology 0, Tree 2, Swid routing table

51, L1

 via Ethernet1/4, metric 40

71, L1

 via Ethernet1/4, metric 80

72, L1

 via Ethernet1/4, metric 0

 

 

 N7K-1-2# show fabricpath isis topology 0 trees

Fabricpath IS-IS domain: default

Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id

 

MT-0

Topology 0, Tree 1, Swid routing table

51, L1

 via Ethernet1/4, metric 40

52, L1

 via Ethernet1/4, metric 40

71, L1

 via Ethernet1/4, metric 0

 

Topology 0, Tree 2, Swid routing table

51, L1

 via Ethernet1/8, metric 40

52, L1

 via Ethernet1/4, metric 40

71, L1

 via Ethernet1/8, metric 80

 

FabricPath allows for ECMP load-balacing, up to 16 paths by default. The way ECMP is done can be modified, the default behavior is to mix L3/L4 information.

 

N7K-1-1(config)# fabricpath load-balance unicast ?

  <CR>

  destination         Include destination parameters

  include-vlan        Use vlan

  layer3              Only Layer-3 parameters considered

  layer4              Only Layer-4 parameters considered

  mixed               Mix of Layer-3 and Layer-4 paramaters (default)

  rotate-amount       Rotate amount for hash string

  source              Include source parameters

  source-destination  Include source and destination parameters

  symmetric           Symmetric (default)

  xor                 Include ex-or of source and destination parameters

 

  N7K-1-1# show  fabricpath load-balance

ECMP load-balancing configuration:

L3/L4 Preference: Mixed

Hash Control: Symmetric

Rotate amount: 0 bytes

Use VLAN: TRUE

 

 

Ftag load-balancing configuration:

Hash Control: Symmetric

Rotate amount: 0 bytes

Use VLAN: TRUE

 

The entire LSDB of the FabricPath domain can be looked at any point in the domain…

 

N7K-1-1# show fabricpath isis database

Fabricpath IS-IS domain: default LSP database

  LSPID                 Seq Number   Checksum  Lifetime   A/P/O/T

  N5K-p1-2.00-00        0x0000000E   0xA7C6    723        0/0/0/1

  N7K-1-2.00-00         0x00000021   0x07BC    1156       0/0/0/1

  N7K-1-1.00-00       * 0x00000021   0xA324    1158       0/0/0/1

  N5K-p1-1.00-00        0x00000012   0x8C91    1075       0/0/0/1

 

  N7K-1-1# show fabricpath isis database detail

Fabricpath IS-IS domain: default LSP database

  LSPID                 Seq Number   Checksum  Lifetime   A/P/O/T

  N5K-p1-2.00-00        0x0000000E   0xA7C6    702        0/0/0/1

    Instance      :  0x0000000C

    Area Address  :  00

    NLPID         :  0xC0

    Hostname      :  N5K-p1-2           Length : 8

    Extended IS   :  N7K-1-1.00         Metric : 40

    Extended IS   :  N7K-1-2.00         Metric : 40

    Extended IS   :  N7K-1-1.00         Metric : 40

    Extended IS   :  N7K-1-2.00         Metric : 40

    Extended IS   :  N7K-1-2.00         Metric : 40

    Extended IS   :  N7K-1-2.00         Metric : 40

    Extended IS   :  N7K-1-1.00         Metric : 40

    Extended IS   :  N7K-1-1.00         Metric : 40

    Capability    : Device Id: 52 Base Topology

      Base Topo Root Pri :

       Trees desired: 2  Trees computed: 2  Trees usable 2

      Nickname        :

       Priority: 0 Nickname: 52 BcastPriority: 64

      Nickname Migration :

       Swid: 52 Sec. Swid: 0

    Digest Offset :  0

  N7K-1-2.00-00         0x00000021   0x07BC    1135       0/0/0/1

    Instance      :  0x0000001C

    Area Address  :  00

    NLPID         :  0xC0

    Hostname      :  N7K-1-2            Length : 7

    Extended IS   :  N5K-p1-2.00        Metric : 40

    Extended IS   :  N5K-p1-2.00        Metric : 40

    Extended IS   :  N5K-p1-2.00        Metric : 40

    Extended IS   :  N5K-p1-2.00        Metric : 40

    Extended IS   :  N5K-p1-1.00        Metric : 40

    Extended IS   :  N5K-p1-1.00        Metric : 40

    Extended IS   :  N5K-p1-1.00        Metric : 40

    Extended IS   :  N5K-p1-1.00        Metric : 40

    Capability    : Device Id: 72 Base Topology

      Base Topo Root Pri :

       Trees desired: 2  Trees computed: 2  Trees usable 2

      Nickname        :

       Priority: 0 Nickname: 72 BcastPriority: 254

      Nickname Migration :

       Swid: 72 Sec. Swid: 0

    Digest Offset :  0

  N7K-1-1.00-00       * 0x00000021   0xA324    1137       0/0/0/1

    Instance      :  0x00000021

    Area Address  :  00

    NLPID         :  0xC0

    Hostname      :  N7K-1-1            Length : 7

    Extended IS   :  N5K-p1-2.00        Metric : 40

    Extended IS   :  N5K-p1-2.00        Metric : 40

    Extended IS   :  N5K-p1-2.00        Metric : 40

    Extended IS   :  N5K-p1-2.00        Metric : 40

    Extended IS   :  N5K-p1-1.00        Metric : 40

    Extended IS   :  N5K-p1-1.00        Metric : 40

    Extended IS   :  N5K-p1-1.00        Metric : 40

    Extended IS   :  N5K-p1-1.00        Metric : 40

    Capability    : Device Id: 71 Base Topology

      Base Topo Ftag  :

       Graph 1: Root: N7K-1-1 Primary: 1, Secondary: 0 Nickname 71

       Graph 2: Root: N7K-1-2 Primary: 2, Secondary: 0 Nickname 72

      Base Topo Roots :

       Graph 1: Root Nickname: 71

       Graph 2: Root Nickname: 72

      Base Topo Root Pri :

       Trees desired: 2  Trees computed: 2  Trees usable 2

      Nickname        :

       Priority: 0 Nickname: 71 BcastPriority: 255

      Nickname Migration :

       Swid: 71 Sec. Swid: 0

    Digest Offset :  0

  N5K-p1-1.00-00        0x00000012   0x8C91    1054       0/0/0/1

    Instance      :  0x00000010

    Area Address  :  00

    NLPID         :  0xC0

    Hostname      :  N5K-p1-1           Length : 8

    Extended IS   :  N7K-1-2.00         Metric : 40

    Extended IS   :  N7K-1-2.00         Metric : 40

    Extended IS   :  N7K-1-2.00         Metric : 40

    Extended IS   :  N7K-1-2.00         Metric : 40

    Extended IS   :  N7K-1-1.00         Metric : 40

    Extended IS   :  N7K-1-1.00         Metric : 40

    Extended IS   :  N7K-1-1.00         Metric : 40

    Extended IS   :  N7K-1-1.00         Metric : 40

    Capability    : Device Id: 51 Base Topology

      Base Topo Root Pri :

       Trees desired: 2  Trees computed: 2  Trees usable 2

      Nickname        :

       Priority: 0 Nickname: 51 BcastPriority: 64

      Nickname Migration :

       Swid: 51 Sec. Swid: 0

    Digest Offset :  0

 

 

 

vPC+

Let’s see now of vPC can be integrated with FabricPath.

 

The issue with the design is that the control plane of the vPC Peer  is not merge. So from a FabricPath point of view the two peers should have two separate Switch IDs… Which is wrong because vPC is designed to appear like a single switch.

 

This is the issue that vPC+ resolves by allowing two vPC peers to share a unique and common FabricPath switch ID.

 

First the definition of the vPC domain and the configuation of the FabricPath Switch ID under the vPC domain :

 

N5K-p1-1# sh run vpc

 

!Command: show running-config vpc

!Time: Sat Mar  2 13:51:58 2013

 

version 5.1(3)N1(1)

feature vpc

 

vpc domain 1

  role priority 1000

  fabricpath switch-id 1

 

 

  N5K-p1-2(config-vpc-domain)# sh run vpc

 

!Command: show running-config vpc

!Time: Sat Mar  2 13:24:27 2013

 

version 5.1(3)N1(1)

feature vpc

 

vpc domain 1

  role priority 2000

  fabricpath switch-id 1

 

The Peer Keepalive link can be configured as usual:

 

N5K-p1-1(config-vpc-domain)# peer-keepalive destination 10.0.8.202 source 10.0.8.201 vrf management

 

N5K-p1-2(config-vpc-domain)# peer-keepalive destination 10.0.8.201 vrf management source 10.0.8.202

 

N5K-p1-1(config-vpc-domain)# sh vpc

Legend:

                (*) – local vPC is down, forwarding via vPC peer-link

 

vPC domain id                   : 1

vPC+ switch id                  : 1

Peer status                     : peer link not configured

vPC keep-alive status           : peer is alive

vPC fabricpath status           : peer not found

Configuration consistency status: failed

Per-vlan consistency status     : failed

Configuration consistency reason: vPC peer-link does not exist

Type-2 consistency status       : failed

Type-2 consistency reason       : vPC peer-link does not exist

vPC role                        : none established

Number of vPCs configured       : 0

Peer Gateway                    : Disabled

Dual-active excluded VLANs      : –

Graceful Consistency Check      : Disabled (due to peer configuration)

 

The peer-link must be defined and must run as a FabricPath switchport:

 

N5K-p1-2(config-if)# sh run int po1

 

!Command: show running-config interface port-channel1

!Time: Sat Mar  2 13:32:33 2013

 

version 5.1(3)N1(1)

 

interface port-channel1

  switchport mode fabricpath

  speed 10000

  vpc peer-link

 

 

N5K-p1-1(config-if)# sh run int po1

 

!Command: show running-config interface port-channel1

!Time: Sat Mar  2 14:00:28 2013

 

version 5.1(3)N1(1)

 

interface port-channel1

  switchport mode fabricpath

  speed 10000

  vpc peer-link

 

Now vPC is aware that the peer is available through FabricPath:

 

  N5K-p1-1# sh vpc

Legend:

                (*) – local vPC is down, forwarding via vPC peer-link

 

vPC domain id                   : 1

vPC+ switch id                  : 1

Peer status                     : peer adjacency formed ok

vPC keep-alive status           : peer is alive

vPC fabricpath status           : peer is reachable through fabricpath

Configuration consistency status: success

Per-vlan consistency status     : success

Type-2 consistency status       : success

vPC role                        : primary

Number of vPCs configured       : 0

Peer Gateway                    : Disabled

Dual-active excluded VLANs      : –

Graceful Consistency Check      : Enabled

 

vPC Peer-link status

———————————————————————

id   Port   Status Active vlans

   —-   —— ————————————————–

1    Po1    up     100

 

N5K-p1-2(config-if)# sh vpc

Legend:

                (*) – local vPC is down, forwarding via vPC peer-link

 

vPC domain id                   : 1

vPC+ switch id                  : 1

Peer status                     : peer adjacency formed ok

vPC keep-alive status           : peer is alive

vPC fabricpath status           : peer is reachable through fabricpath

Configuration consistency status: success

Per-vlan consistency status     : success

Type-2 consistency status       : success

vPC role                        : secondary

Number of vPCs configured       : 0

Peer Gateway                    : Disabled

Dual-active excluded VLANs      : –

Graceful Consistency Check      : Enabled

 

vPC Peer-link status

———————————————————————

id   Port   Status Active vlans

   —-   —— ————————————————–

1    Po1    up     100

 

The ISIS FabricPath adjacency table has been modified to reflect the new design:

 

N5K-p1-2# show fabricpath isis adjacency

Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:

System ID       SNPA            Level  State  Hold Time  Interface

N5K-p1-1        N/A             1      UP     00:00:28   port-channel1

N7K-1-2         N/A             1      UP     00:00:29   Ethernet1/1

N7K-1-2         N/A             1      UP     00:00:27   Ethernet1/2

N7K-1-2         N/A             1      UP     00:00:24   Ethernet1/3

N7K-1-2         N/A             1      UP     00:00:27   Ethernet1/4

N7K-1-1         N/A             1      UP     00:00:33   Ethernet1/5

N7K-1-1         N/A             1      UP     00:00:24   Ethernet1/6

N7K-1-1         N/A             1      UP     00:00:27   Ethernet1/7

N7K-1-1         N/A             1      UP     00:00:22   Ethernet1/8

 

One very important design requirement of FabricPath is that devices doing the separation between CE and FP must be root of the STP domain and configured with the same priority. Cisco recommends 8192.

 

N5K-p1-1(config)# spanning-tree vlan 100 priority 8192

 

N5K-p1-2(config)# spanning-tree vlan 100 priority 8192

 

Now vPC member port can be configured towards end hosts (or FEXes).

 

N5K-p1-2# sh run int e1/11

 

!Command: show running-config interface Ethernet1/11

!Time: Sat Mar  2 13:40:30 2013

 

version 5.1(3)N1(1)

 

interface Ethernet1/11

  switchport access vlan 100

  speed 1000

  channel-group 6 mode active

 

  N5K-p1-2# sh run int po6

 

!Command: show running-config interface port-channel6

!Time: Sat Mar  2 13:40:41 2013

 

version 5.1(3)N1(1)

 

interface port-channel6

  switchport access vlan 100

  speed 1000

  vpc 6

 

  N5K-p1-1# sh run int e1/11

 

!Command: show running-config interface Ethernet1/11

!Time: Sat Mar  2 14:08:38 2013

 

version 5.1(3)N1(1)

 

interface Ethernet1/11

  switchport access vlan 100

  speed 1000

  channel-group 6 mode active

 

  N5K-p1-1# sh run int po6

 

!Command: show running-config interface port-channel6

!Time: Sat Mar  2 14:08:45 2013

 

version 5.1(3)N1(1)

 

interface port-channel6

  switchport access vlan 100

  speed 1000

  vpc 6

 

  N5K-p1-1# sh vpc

Legend:

                (*) – local vPC is down, forwarding via vPC peer-link

 

vPC domain id                   : 1

vPC+ switch id                  : 1

Peer status                     : peer adjacency formed ok

vPC keep-alive status           : peer is alive

vPC fabricpath status           : peer is reachable through fabricpath

Configuration consistency status: success

Per-vlan consistency status     : success

Type-2 consistency status       : success

vPC role                        : primary

Number of vPCs configured       : 1

Peer Gateway                    : Disabled

Dual-active excluded VLANs      : –

Graceful Consistency Check      : Enabled

 

vPC Peer-link status

———————————————————————

id   Port   Status Active vlans

   —-   —— ————————————————–

1    Po1    up     100

 

vPC status

—————————————————————————

id     Port        Status Consistency Reason       Active vlans vPC+ Attrib

     ———-  —— ———– ——       ———— ———–

6      Po6         up     success     success      100          DF: Partial

 

 

N5K-p1-2# sh vpc

Legend:

                (*) – local vPC is down, forwarding via vPC peer-link

 

vPC domain id                   : 1

vPC+ switch id                  : 1

Peer status                     : peer adjacency formed ok

vPC keep-alive status           : peer is alive

vPC fabricpath status           : peer is reachable through fabricpath

Configuration consistency status: success

Per-vlan consistency status     : success

Type-2 consistency status       : success

vPC role                        : secondary

Number of vPCs configured       : 1

Peer Gateway                    : Disabled

Dual-active excluded VLANs      : –

Graceful Consistency Check      : Enabled

 

vPC Peer-link status

———————————————————————

id   Port   Status Active vlans

   —-   —— ————————————————–

1    Po1    up     100

 

vPC status

—————————————————————————

id     Port        Status Consistency Reason       Active vlans vPC+ Attrib

     ———-  —— ———– ——       ———— ———–

6      Po6         up     success     success      100          DF: Partial

Note that the two peers are STP root of their STP domain and they share a common Bride ID, c84c.75fa.6000:

 

N5K-p1-1# sh span vlan 100

 

VLAN0100

  Spanning tree enabled protocol rstp

  Root ID    Priority    8292

             Address     c84c.75fa.6000

             This bridge is the root

             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

 

  Bridge ID  Priority    8292   (priority 8192 sys-id-ext 100)

             Address     c84c.75fa.6000

             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

 

Interface        Role Sts Cost      Prio.Nbr Type

—————- —- — ——— ——– ——————————–

Po6              Desg FWD 1         128.4101 (vPC) P2p

 

 

VLAN0100

  Spanning tree enabled protocol rstp

  Root ID    Priority    8292

             Address     c84c.75fa.6000

             This bridge is the root

             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

 

  Bridge ID  Priority    8292   (priority 8192 sys-id-ext 100)

             Address     c84c.75fa.6000

             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

 

Interface        Role Sts Cost      Prio.Nbr Type

—————- —- — ——— ——– ——————————–

Po6              Desg FWD 1         128.4101 (vPC) P2p

N5K-p1-1# show fabricpath switch-id

                        FABRICPATH SWITCH-ID TABLE

Legend: ‘*’ – this system

=========================================================================

SWITCH-ID      SYSTEM-ID       FLAGS         STATE    STATIC  EMULATED

———-+—————-+————+———–+——————–

1           547f.ee22.81fc    Primary     Confirmed     No      Yes

1           0005.73ba.637c    Primary     Confirmed     No      Yes

*51          547f.ee22.81fc    Primary     Confirmed    Yes     No

52          0005.73ba.637c    Primary     Confirmed     Yes     No

71          0024.98e8.01c2    Primary     Confirmed     Yes     No

72          001b.54c2.67c2    Primary     Confirmed     Yes     No

Total Switch-ids: 6

 

N7K-1-1# sh fabricpath route

FabricPath Unicast Route Table

‘a/b/c’ denotes ftag/switch-id/subswitch-id

‘[x/y]’ denotes [admin distance/metric]

ftag 0 is local ftag

subswitch-id 0 is default subswitch-id

 

 

FabricPath Unicast Route Table for Topology-Default

 

0/71/0, number of next-hops: 0

        via —- , [60/0], 0 day/s 01:47:48, local

1/1/0, number of next-hops: 8

        via Eth1/1, [115/40], 0 day/s 00:12:52, isis_fabricpath-default

        via Eth1/2, [115/40], 0 day/s 00:12:52, isis_fabricpath-default

        via Eth1/3, [115/40], 0 day/s 00:12:52, isis_fabricpath-default

        via Eth1/4, [115/40], 0 day/s 00:12:52, isis_fabricpath-default

        via Eth1/5, [115/40], 0 day/s 00:12:52, isis_fabricpath-default

        via Eth1/6, [115/40], 0 day/s 00:12:52, isis_fabricpath-default

        via Eth1/7, [115/40], 0 day/s 00:12:52, isis_fabricpath-default

        via Eth1/8, [115/40], 0 day/s 00:12:52, isis_fabricpath-default

1/51/0, number of next-hops: 4

        via Eth1/1, [115/40], 0 day/s 01:47:35, isis_fabricpath-default

        via Eth1/2, [115/40], 0 day/s 01:47:35, isis_fabricpath-default

        via Eth1/3, [115/40], 0 day/s 01:47:35, isis_fabricpath-default

        via Eth1/4, [115/40], 0 day/s 01:47:35, isis_fabricpath-default

1/52/0, number of next-hops: 4

        via Eth1/5, [115/40], 0 day/s 01:01:29, isis_fabricpath-default

        via Eth1/6, [115/40], 0 day/s 01:01:29, isis_fabricpath-default

        via Eth1/7, [115/40], 0 day/s 01:01:29, isis_fabricpath-default

        via Eth1/8, [115/40], 0 day/s 01:01:29, isis_fabricpath-default

1/72/0, number of next-hops: 8

        via Eth1/1, [115/80], 0 day/s 01:47:23, isis_fabricpath-default

        via Eth1/2, [115/80], 0 day/s 01:47:23, isis_fabricpath-default

        via Eth1/3, [115/80], 0 day/s 01:47:23, isis_fabricpath-default

        via Eth1/4, [115/80], 0 day/s 01:47:23, isis_fabricpath-default

        via Eth1/5, [115/80], 0 day/s 01:01:29, isis_fabricpath-default

        via Eth1/6, [115/80], 0 day/s 01:01:29, isis_fabricpath-default

        via Eth1/7, [115/80], 0 day/s 01:01:29, isis_fabricpath-default

        via Eth1/8, [115/80], 0 day/s 01:01:29, isis_fabricpath-default

 

Let’s see the result of the MAC address table now.

 

0050.568b.002d, 0014.1cad.fb0a and 0014.1cad.fb0a are end hosts. What we see is that the reachability of the remote hosts is not a port but is a FabricPath Switch ID.

 

For example from N5K-p1-1 perspective, 0014.1cad.fb0a is reacheable through a switch that has the FabricPath Switch ID 52.0.0.

 

The reachability of this FabricPath host can be resolved through the ISIS database and the unicast frame will be forwared directly to that device.

 

 

N5K-p1-1# sh mac address-table dynamic

Legend:

        * – primary entry, G – Gateway MAC, (R) – Routed MAC, O – Overlay MAC

        age – seconds since last seen,+ – primary entry using vPC Peer-Link

   VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID

———+—————–+——–+———+——+—-+——————

+ 100      0014.1cad.fb0a    dynamic   0          F    F  52.0.0

* 100      0015.1758.17dc    dynamic   10         F    F  Po6

* 100      0050.568b.002d    dynamic   20         F    F  Eth1/12

+ 100      0050.568b.002e    dynamic   0          F    F  52.0.0

* 100      58bc.27b7.1d96    dynamic   10         F    F  Eth1/12

 

 

N5K-p1-2# sh mac address-table dynamic

Legend:

        * – primary entry, G – Gateway MAC, (R) – Routed MAC, O – Overlay MAC

        age – seconds since last seen,+ – primary entry using vPC Peer-Link

   VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID

———+—————–+——–+———+——+—-+——————

* 100      0014.1cad.fb0a    dynamic   0          F    F  Eth1/12

* 100      0015.1758.17dc    dynamic   40         F    F  Po6

+ 100      0050.568b.002d    dynamic   0          F    F  51.0.0

* 100      0050.568b.002e    dynamic   20         F    F  Eth1/12

+ 100      58bc.27b7.1d96    dynamic   0          F    F  51.0.0

 

What we see here is that Nexus 7000 have learned the MAC addresses of end hosts too which is not what Conversational Learning should allow because the Nexus 7000 doesn’t host the end device and therefore do not know their MAC addresses.

 

The reason is that the 7000s are running in mixed chassis mode (with both F and M cards). M cards are capable of L3 routing and can do proxy routing for F cards. So when the broadcast is received in the F modules, M card will learn the MAC.

 

If the Nexus 7000 switches were only hosting F cards, this learning would not have happened.

 

 

N7K-1-1# sh mac address-table dynamic

Legend:

        * – primary entry, G – Gateway MAC, (R) – Routed MAC, O – Overlay MAC

        age – seconds since last seen,+ – primary entry using vPC Peer-Link

   VLAN     MAC Address      Type      age     Secure NTFY Ports/SWID.SSID.LID

———+—————–+——–+———+——+—-+——————

  100      0014.1cad.fb0a    dynamic   210        F    F  52.0.0

  100      0015.1758.17dc    dynamic   60         F    F  1.0.0

  100      0050.568b.002d    dynamic   90         F    F  51.0.0

  100      0050.568b.002e    dynamic   30         F    F  52.0.0

  100      58bc.27b7.1d96    dynamic   240        F    F  51.0.0

 

 

  N7K-1-2# sh mac address-table dynamic

Legend:

        * – primary entry, G – Gateway MAC, (R) – Routed MAC, O – Overlay MAC

        age – seconds since last seen,+ – primary entry using vPC Peer-Link

   VLAN     MAC Address      Type      age     Secure NTFY Ports/SWID.SSID.LID

———+—————–+——–+———+——+—-+——————

  100      0014.1cad.fb0a    dynamic   600        F    F  52.0.0

  100      0015.1758.17dc    dynamic   30         F    F  1.0.0

  100      0050.568b.002d    dynamic   450        F    F  51.0.0

  100      0050.568b.002e    dynamic   420        F    F  52.0.0

  100      58bc.27b7.1d96    dynamic   600        F    F  51.0.0

 

Next time the configuration will be complete with the addition of vPC+ on the 7Ks and L3 function like HSRP !

4 thoughts on “FabricPath

  1. Hi Romain! Thank you so much. This is very helpful in my future deployments. I was trying to find the vPC+ with HSRP in your blog but I can’t find it. Do you also have that information? Thanks again.

    • Hello Bill,

      Yes you can do dual sided vPC+.
      However I’m not sure to see what advantages it would have over running fabricpath between the 4 units.

      Thanks,

Leave a Reply

Your email address will not be published. Required fields are marked *