Ethernet Overhead - Catalyst

The Cisco IOS CLI is a fantastic tool for troubleshooting suspected network issues.  It is important to understand what you're looking at when parsing the output of the commands.  I'm specifically thinking about the importance of accurate counters in the 'show interface' command, as it relates to the Catalyst product line.

The 'show interface' output is somewhat deceiving when talking about utilization numbers.  On an Ethernet interface, you must account for 38 bytes of Ethernet overhead when you talk about utilization.  That is broken up as follows:

Inter Frame Gap          12 bytes
MAC Preamble (w/ SoF)     8 bytes
MAC Destination           6 bytes
MAC Source                6 bytes
Length                    2 bytes
CRC                       4 bytes
---------------------------------
Total Overhead           38 bytes

The Cisco Catalyst CLI doesn't account for all 38 bytes when it lists utilization figures, nor does it account for all of the overhead with the the SNMP counters.  Specifically, the IFG (Inter Frame Gap) as well as the MAC Preamble are not included.  What this means is they are only accounting for 18 bytes of overhead, the last 4 items in the above list, and not 38 bytes.  This is marginally better than the Cisco ASR 1000 router, which doesn't include the 4 byte CRC.

In environments where you have data flows that consist of very small packets (the worst case is a 46 byte IP packet), the Ethernet overhead is fairly significant, and omitting 20 bytes means that you're leaving off 24% of the total frame size!  A 46 byte IP packet takes up 84 bytes on the wire, 46 bytes for the IP packet (Layer 3) + 38 bytes for the Ethernet overhead (Layer 2).

The output from a 'show interface' on a port which is flooded with 100 mbps of 84 byte frames, looks as follows:

2950G#sh int fa0/1
FastEthernet0/1 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 0012.01b5.2e41 (bia 0012.01b5.2e41)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 194/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 100BaseTX
  input flow-control is unsupported output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:02:14
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 76191000 bits/sec, 148032 packets/sec
  30 second output rate 0 bits/sec, 0 packets/sec
     18576331 packets input, 1188885184 bytes, 0 no buffer
     Received 0 broadcasts (0 multicast)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     84 packets output, 6076 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

You'll notice that the 30 second average shows some bandwidth to spare, but if you run the numbers based on the packet rate, accounting for all 38 bytes of overhead, you'll notice that the link is saturated:

148032 packets/sec * 84 bytes = 12434688 bytes/second
12434688 bytes/second * 8 (conversion to bits) = 99,477,504 bits/second

At 99.4 mbps, out of 100 mbps, it's safe to say the link is saturated.

What this means is that output drops could start happening long before you see the load interval average approaching the link speed, simply because the CLI omits 20 bytes in their frame counters.

Network administrators shouldn't be so quick to discount user complaints of packet loss simply because the load interval shows bandwidth to spare.