The multi-dimensional skill-based router for IPCS used a real-time database engine to calculate the best agent to take a waiting call or the best call to send to a newly available agent. I had taken the original code, adjusted it for test-driven development, and written unit tests that provided very complete coverage. This component communicated with the database engine. It published messages to other components and subscribed to receive asynchronous responses. With the tests, we could verify that every valid stimulus produced a valid response from the component. Looking at the logs when run under load and on multiprocessor machines, the sequence of behavior for every call showed that the component was behaving precisely according to spec, a perfect thread-safe state machine.

However, occasionally one of the other components would mess up and lose something. The router had given it the proper stimulus, it was supposed to give back the proper response, and then the router would do the next thing in the sequence. However, the other component had lost its way and wasn't giving back the proper response. While it had low-level timeouts and error handling in the communication layers, the router had no timeouts to cover for other components failing to meet contract. So, rather than failing the route, and would cheerfully let that call wait forever (hung) until the other component coughed up the event that was necessary to continue.

A few weeks before the entire team moved on to develop another product, I realized that this was not only a serious problem but that it was my serious problem, that making up for the failures of others was really a necessary part of writing a good robust component. I had the changes ready to code but never got the chance to apply them.

To be robust, code must do something sensible even when its peers are terminally confused.