The Nexus FEX (Fabric Extender) is the 1 gbps fan-out device for the Nexus 5000. The notion of a fabric extender is a new concept to many folks; it is not a stand alone switch. It is managed and configured from the upstream Nexus 5000. To communicate between 2 ports on the Nexus 2148 you must be switched by the Nexus 5000 that feeds it. This means that to go from port 1 to port 2 (on a 2148) you will go to the Nexus 5000 which makes the switching decision and sends it back down to the 2148 to exit port 2.
Connecting the 2148 to a 5000 is surprisingly easy. In this article I will walk you though a very simple configuration. In this topology we have a Nexus 5020 with a Nexus 2148. We're using Ethernet 1/1 and 1/2 on the 5020 connecting to uplink 1 and 2 on the 2148T.
Once the 2148T is powered up, I ran cables between the 2 port pairs:
5020 Eth1/1 - 2148T uplink 1
5020 Eth1/2 - 2148T uplink 2
Now let's look at the simple Nexus 5020 configuration needed to use a 2148.
interface Ethernet1/1
switchport mode fex-fabric
fex associate 100
interface Ethernet1/2
switchport mode fex-fabric
fex associate 100
fex 100
pinning max-links 2
By default, a newly created fex group (100 in this instance) will only use 1 link. In this scenario, we have (2) 10 gbps link to be used, so the group was modified to account for both. That was done with the "pinning" command.
That's all there is to it—the 2148 is now online. At this point, you can start configuring the ports on your 2148. Since the FEX group number created above is 100, you would reference port 1 on the 2148 as "Ethernet 100/1/1".
switch# sh int status
--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth1/1 -- up 1 full 10G 1/10g
Eth1/2 -- up 1 full 10G 1/10g
Eth1/3 -- down 1 full 10G 1/10g
Eth1/4 -- down 1 full 10G 1/10g
Eth1/5 -- down 1 full 10G 1/10g
Eth1/6 -- down 1 full 10G 1/10g
Eth1/7 -- down 1 full 10G 1/10g
Eth1/8 -- down 1 full 10G 1/10g
Eth1/9 -- down 1 full 10G 1/10g
Eth1/10 -- down 1 full 10G 1/10g
Eth1/11 -- down 1 full 10G 1/10g
Eth1/12 -- down 1 full 10G 1/10g
Eth1/13 -- down 1 full 10G 1/10g
Eth1/14 -- down 1 full 10G 1/10g
Eth1/15 -- down 1 full 10G 1/10g
Eth1/16 -- down 1 full 10G 1/10g
Eth1/17 -- down 1 full 10G 10g
Eth1/18 -- down 1 full 10G 10g
Eth1/19 -- down 1 full 10G 10g
Eth1/20 -- down 1 full 10G 10g
Eth1/21 -- down 1 full 10G 10g
Eth1/22 -- down 1 full 10G 10g
Eth1/23 -- down 1 full 10G 10g
Eth1/24 -- down 1 full 10G 10g
Eth1/25 -- down 1 full 10G 10g
Eth1/26 -- down 1 full 10G 10g
Eth1/27 -- down 1 full 10G 10g
Eth1/28 -- down 1 full 10G 10g
Eth1/29 -- down 1 full 10G 10g
Eth1/30 -- down 1 full 10G 10g
Eth1/31 -- down 1 full 10G 10g
Eth1/32 -- down 1 full 10G 10g
Eth1/33 -- down 1 full 10G 10g
Eth1/34 -- down 1 full 10G 10g
Eth1/35 -- down 1 full 10G 10g
Eth1/36 -- down 1 full 10G 10g
Eth1/37 -- down 1 full 10G 10g
Eth1/38 -- down 1 full 10G 10g
Eth1/39 -- down 1 full 10G 10g
Eth1/40 -- down 1 full 10G 10g
mgmt0 -- down routed auto auto --
Eth100/1/1 -- down 1 full 1000 --
Eth100/1/2 -- down 1 full 1000 --
Eth100/1/3 -- down 1 full 1000 --
Eth100/1/4 -- down 1 full 1000 --
Eth100/1/5 -- down 1 full 1000 --
Eth100/1/6 -- down 1 full 1000 --
Eth100/1/7 -- down 1 full 1000 --
Eth100/1/8 -- down 1 full 1000 --
Eth100/1/9 -- down 1 full 1000 --
Eth100/1/10 -- down 1 full 1000 --
Eth100/1/11 -- down 1 full 1000 --
Eth100/1/12 -- down 1 full 1000 --
Eth100/1/13 -- down 1 full 1000 --
Eth100/1/14 -- down 1 full 1000 --
Eth100/1/15 -- down 1 full 1000 --
Eth100/1/16 -- down 1 full 1000 --
Eth100/1/17 -- down 1 full 1000 --
Eth100/1/18 -- down 1 full 1000 --
Eth100/1/19 -- down 1 full 1000 --
Eth100/1/20 -- down 1 full 1000 --
Eth100/1/21 -- down 1 full 1000 --
Eth100/1/22 -- down 1 full 1000 --
Eth100/1/23 -- down 1 full 1000 --
Eth100/1/24 -- down 1 full 1000 --
Eth100/1/25 -- down 1 full 1000 --
Eth100/1/26 -- down 1 full 1000 --
Eth100/1/27 -- down 1 full 1000 --
Eth100/1/28 -- down 1 full 1000 --
Eth100/1/29 -- down 1 full 1000 --
Eth100/1/30 -- down 1 full 1000 --
Eth100/1/31 -- down 1 full 1000 --
Eth100/1/32 -- down 1 full 1000 --
Eth100/1/33 -- down 1 full 1000 --
Eth100/1/34 -- down 1 full 1000 --
Eth100/1/35 -- down 1 full 1000 --
Eth100/1/36 -- down 1 full 1000 --
Eth100/1/37 -- down 1 full 1000 --
Eth100/1/38 -- down 1 full 1000 --
Eth100/1/39 -- down 1 full 1000 --
Eth100/1/40 -- down 1 full 1000 --
Eth100/1/41 -- down 1 full 1000 --
Eth100/1/42 -- down 1 full 1000 --
Eth100/1/43 -- down 1 full 1000 --
Eth100/1/44 -- down 1 full 1000 --
Eth100/1/45 -- down 1 full 1000 --
Eth100/1/46 -- down 1 full 1000 --
Eth100/1/47 -- down 1 full 1000 --
Eth100/1/48 -- down 1 full 1000 --
switch# show fex 100 detail
FEX: 100 Description: FEX0100 state: Online
FEX version: 4.1(3)N1(1a) [Switch version: 4.1(3)N1(1a)]
FEX Interim version: 4.1(3)N1(1a)
Switch Interim version: 4.1(3)N1(1a)
Extender Model: N2K-C2148T-1GE, Extender Serial: JEX1310EDPD
Part No: 73-12009-05
Card Id: 70, Mac Addr: 00:0d:ec:3b:35:42, Num Macs: 64
Module Sw Gen: 12338 [Switch Sw Gen: 20]
pinning-mode: static Max-links: 2
Fabric port for control traffic: Eth1/1
Fabric interface state:
Eth1/1 - Interface Up. State: Active
Eth1/2 - Interface Up. State: Active
Fex Port State Fabric Port Primary Fabric
Eth100/1/1 Down Eth1/1 Eth1/1
Eth100/1/2 Down Eth1/1 Eth1/1
Eth100/1/3 Down Eth1/1 Eth1/1
Eth100/1/4 Down Eth1/1 Eth1/1
Eth100/1/5 Down Eth1/1 Eth1/1
Eth100/1/6 Down Eth1/1 Eth1/1
Eth100/1/7 Down Eth1/1 Eth1/1
Eth100/1/8 Down Eth1/1 Eth1/1
Eth100/1/9 Down Eth1/1 Eth1/1
Eth100/1/10 Down Eth1/1 Eth1/1
Eth100/1/11 Down Eth1/1 Eth1/1
Eth100/1/12 Down Eth1/1 Eth1/1
Eth100/1/13 Down Eth1/1 Eth1/1
Eth100/1/14 Down Eth1/1 Eth1/1
Eth100/1/15 Down Eth1/1 Eth1/1
Eth100/1/16 Down Eth1/1 Eth1/1
Eth100/1/17 Down Eth1/1 Eth1/1
Eth100/1/18 Down Eth1/1 Eth1/1
Eth100/1/19 Down Eth1/1 Eth1/1
Eth100/1/20 Down Eth1/1 Eth1/1
Eth100/1/21 Down Eth1/1 Eth1/1
Eth100/1/22 Down Eth1/1 Eth1/1
Eth100/1/23 Down Eth1/1 Eth1/1
Eth100/1/24 Down Eth1/1 Eth1/1
Eth100/1/25 Down Eth1/2 Eth1/2
Eth100/1/26 Down Eth1/2 Eth1/2
Eth100/1/27 Down Eth1/2 Eth1/2
Eth100/1/28 Down Eth1/2 Eth1/2
Eth100/1/29 Down Eth1/2 Eth1/2
Eth100/1/30 Down Eth1/2 Eth1/2
Eth100/1/31 Down Eth1/2 Eth1/2
Eth100/1/32 Down Eth1/2 Eth1/2
Eth100/1/33 Down Eth1/2 Eth1/2
Eth100/1/34 Down Eth1/2 Eth1/2
Eth100/1/35 Down Eth1/2 Eth1/2
Eth100/1/36 Down Eth1/2 Eth1/2
Eth100/1/37 Down Eth1/2 Eth1/2
Eth100/1/38 Down Eth1/2 Eth1/2
Eth100/1/39 Down Eth1/2 Eth1/2
Eth100/1/40 Down Eth1/2 Eth1/2
Eth100/1/41 Down Eth1/2 Eth1/2
Eth100/1/42 Down Eth1/2 Eth1/2
Eth100/1/43 Down Eth1/2 Eth1/2
Eth100/1/44 Down Eth1/2 Eth1/2
Eth100/1/45 Down Eth1/2 Eth1/2
Eth100/1/46 Down Eth1/2 Eth1/2
Eth100/1/47 Down Eth1/2 Eth1/2
Eth100/1/48 Down Eth1/2 Eth1/2
Logs:
[11/30/2009 17:24:55.979316] Module register received
[11/30/2009 17:24:55.982666] Registration response sent
[11/30/2009 17:24:56.255137] Module Online Sequence
[11/30/2009 17:24:57.664735] Module Online
With 2 working uplinks, the fan-out ports on the 2148 are statically pinned to an uplink; in this case the first 24 ports use uplink #1 and the last 24 ports use uplink #2. That is easily viewed with the following commands:
switch# show int e1/1 fex-intf
Fabric FEX
Interface Interfaces
---------------------------------------------------
Eth1/1 Eth100/1/24 Eth100/1/23 Eth100/1/22 Eth100/1/21
Eth100/1/20 Eth100/1/19 Eth100/1/18 Eth100/1/17
Eth100/1/16 Eth100/1/15 Eth100/1/14 Eth100/1/13
Eth100/1/12 Eth100/1/11 Eth100/1/10 Eth100/1/9
Eth100/1/8 Eth100/1/7 Eth100/1/6 Eth100/1/5
Eth100/1/4 Eth100/1/3 Eth100/1/2 Eth100/1/1
switch# show int e1/2 fex-intf
Fabric FEX
Interface Interfaces
---------------------------------------------------
Eth1/2 Eth100/1/48 Eth100/1/47 Eth100/1/46 Eth100/1/45
Eth100/1/44 Eth100/1/43 Eth100/1/42 Eth100/1/41
Eth100/1/40 Eth100/1/39 Eth100/1/38 Eth100/1/37
Eth100/1/36 Eth100/1/35 Eth100/1/34 Eth100/1/33
Eth100/1/32 Eth100/1/31 Eth100/1/30 Eth100/1/29
Eth100/1/28 Eth100/1/27 Eth100/1/26 Eth100/1/25
Having ports statically pinned to a particular uplink port brings up an interesting question. What happens if you lose an uplink port? The user ports on the 2148 go down!
This is the Nexus 5020 port that leads to the 2148:
switch# sh int eth1/1 | inc Ethernet1/1
Ethernet1/1 is up
[Cable Unplugged]
switch# sh int eth1/1 | inc Ethernet1/1
Ethernet1/1 is down (Link not connected)
We now only have 1 operational FEX port leading to the 2148 which means 1/2 of the ports are dead. All of the ports that were pinned to Eth1/1 (as shown above) are down.
switch# show int eth100/1/1 | inc 100/1/1
Ethernet100/1/1 is down (fabricIfDown)
If you want to avoid this failure scenario, you can assign the physical ports on the Nexus 5000 to a Port Channel and apply the FEX commands to the channel. One slight difference with this approach is that your pinning command would be "pinning max-links 1", the 1 port included is actually the port channel which includes your 2 physical ports. This port channel approach can also be used with a vPC between (2) Nexus 5000s.
No comments:
Post a Comment