Sometimes the hardest problems are the simple ones

It was some time in late 2004 when I received a call from a customer I had visited a few weeks earlier.  Back then, we were a small company with perhaps 80 employees, including a worldwide sales force of less than a dozen people.  As a startup company, I was a part-time systems engineer, rolling up my sleeves to help out with some customer deployments in addition to my Product Management responsibilities.

"Can you give me an RMA number?  I'm going to ship these Steelheads back to you.  I put them into the network, and nothing happened.  The WAN transfers are still slow."

I was disappointed.  This prospective customer originally seemed like the ideal customer.  They were using apps that we knew our Steelhead appliances would do well with.  Our Steelhead solution had worked marvelously with many other customers just like this one.

Desperately, I asked the customer to do some quick tests to check for a duplex mismatch.  An Ethernet duplex mismatch is an insidious problem that occurs when the interfaces of the two connected devices do not agree on the duplex mode (i.e., half or full duplex) to be used for the Ethernet connection between them.  A duplex mismatch causes some packets to be dropped, but it usually will not completely impair communication between the devices. This can give the illusion that everything is fine with the Ethernet connection.  But in fact, performance is degraded because packets are being dropped.

"No, it's definitely not a duplex mismatch.  We don't have those types of problems here. We make sure everything is set to autodiscover."

No matter what I said, the customer insisted nothing was wrong with his network, and he didn't see the need to do even basic troubleshooting.  The problem was clearly with the Steelhead appliances.  Since this was a relatively-large name-brand customer, I wasn't going to give up so easily.  But I didn't want to push any further and risk offending the customer.  Being from an unknown startup company, I was a nobody, and had no right to tell him what to do.  So instead, I offered to fly down to the customer site later in the week to diagnose the problem in-person. 

Upon arriving after the 90-minute flight, I did some quick ping flood tests and quickly identified the interface with the duplex mismatch.  Within 5 minutes, the customer was doing lightning-fast WAN transfers.  Files that previously took 7 minutes to download were now accessed within 10 seconds.  The customer was wide-eyed with disbelief at the results.

The above experience is not an isolated incident--it happens all the time.  Back then, we estimated that the root cause of fully one-third (1/3) of the technical assistance requests we received was a duplex mismatch.  That statistic probably hasn't improved by very much, even as of today.  Though it is a simple problem to fix, it remains the most common issue we encounter, and represents an unending challenge for our new customers and Riverbed TAC.

Josh Tseng