ONS Code Upgrades & Expected Packet Loss

As a consultant, I frequently work in environments that don't have regularly scheduled maintenance windows. One of the internal BUs is always doing something critical or their customers are inflexible. For critical infrastructure, it's often difficult to coordinate maintenance windows with all of the parties, even in a redundant network where no outage is expected.

I've recently been tasked to upgrade a Cisco ONS 15454 DWDM solution from Release 9.0 to 9.2, which utilizes Xponders (GE-XP and OTU-2). In talking with some friends who are solely devoted to working on very large Cisco optical networks, and an optical Cisco SE, everyone agreed that most folks won't notice the upgrade. While that's comforting, I wanted quantifiable data on how much packet loss to expect. I also wanted to know about any potential link bounces that may occur on the client interfaces. That's not only helpful for me while I'm performing the upgrade, so I know if it's proceeding as expected, but it's also an important data point I can tell my customers.

I started by taking the lab hardware and setting up my worst case scenario. Often, in redundant networks, traffic can wrap to the protect path with very little impact to the production traffic. To eliminate this as a variable in the lab, I didn't redundantly connect the two nodes; I linearly connected them. Node 1 is a WSS/ROADM config while Node 2 is an AD4C (4 Channel Add/Drop). Each node has a GE-XP (Layer-2 Mode) with a SmartBits connected to a client side port. It's a very simple topology:

During the upgrade I sent a bi-directional 1 gbps stream between the two nodes. The test included upgrading both nodes, one at a time. The entire process took about an hour, and I'm happy to report that there was zero packet loss with no link bounces. If only my other devices could be upgraded with zero user impact.