Evaluating Software's Impact on System and System of Systems Reliability from SEI at CMU

Carnegie Mellon University Software Engineering Institute (SEI) has published a new White Paper: Evaluating Software's Impact on System and System of Systems Reliability by John B. Goodenough.

Alas I have to say the title is more impressive than the paper itself. I'd sum up the paper as saying "We need to define our terms to Contractors and Sub-Contractors when we talk about software to them".

There is one paragraph in the paper that I do think is particularly important:

"Hardware engineers typically think that software failures are deterministic because certain inputs or uses can reliably cause a failure. But although all software failures are deterministic in the sense that they occur every time certain conditions are met, the likelihood of the conditions being met becomes, eventually, a function of usage patterns and history, neither of which are deterministic. In effect, after egregious software faults have been removed, failure occurrences become non-deterministic. In fact, certain types of software failure are inherently non-deterministic because they depend on more knowledge of program state than is typically available. For example, failures due to race conditions and memory leaks typically depend on usage history and, for race conditions, subtle details of system state. Although these are removable design deficiencies, their occurrence appears to be random (although typically the frequency of such failures increases as the load on the software system increases). In short, it is not unreasonable to think of software failures as eventually mimicking hardware behavior in their seemingly non-deterministic occurrence."

The above describes why it can be so hard to write correct software, and even harder to test it. "Perfect Software" is something that only exists in theory.

Maybe a particular bug only manifests when the non-deterministic timing of the application has interrupts nested three or four levels deep (Some Atmel and Zilog parts let you do this), while turning on the radio, turning on the left turn signal and pressing on the brake simultaneously, and the unit has been running continuously for over 18 hours and 12 minutes, during a ESD event...

Software Bug or Electrostatic Discharge? Maybe the humidity was to high for the Cat for a good test?

Have you ever had a field report of a problem with a product that makes no sense, and is impossible to reproduce?

This type of problem falls into the class of problems known as Single Event Upset. SEU incidents can be caused by energetic Cosmic Rays, which I've seen people use as a excuse for their buggy software unfortunately. If you are designing equipment for operation *in* outer-space or near other ionizing radiation sources this is a real problems that needs dealt with, here on Earth not so much. However today's shrinking geometries are increasing the levels of susceptibility.

What is a real everyday problem that does not get addressed enough is "Static Electricity". The chain of custody from raw components to a finished product in the customers hands must consider the issue of Electrostatic Discharge (ESD) for the entire life cycle of a product.

If at all possible visit the contractor assembler or supplier of your hardware, with a critical eye to their handling and storage of boards during the entire assembly process.

What is important to understand is that ESD mitigation during building of a product is about culture and training, it is not about the parts and the assemblies.

What you should see:

• Any person that has contact with components or assemblies is wearing static dissipative smocks, and heal straps.
• Any person working at a bench working with an assembly is wearing a wrist strap. The exception being working with High Voltage or non-isolated line power supplies for personnel safety.
• Wrist-strap testers and a daily log of them being tested. Test at each shift change.
• All board assemblies are being held in appropriate static dissipative containers.
• Board assemblies are not touching each other. This can cause ESD as well as mechanical damage.
• Static dissipative chairs and floor mats.
• Static dissipative coatings on the floor, and logs of application of the coatings.
• In a design lab, a Electrostatic Gun, to create repeatable ESD events.

What you should not see:

• Inappropriate clothing like Wool Sweaters.
• Inappropriate furniture.
• Boards piled on top of each other.
• Boards being transported in cardboard trays.
• Boards being transported on Teflon coated cookie sheets, in the assumption that is an improvement over cardboard.
• Large number of random failures or reports from the floor of "bad parts".
• Large number of random failures in warranty repairs; indicative of latent ESD damage.
• Few wrist straps. No sign of wrist strap replacements.
• No wrist strap testers.
• Cats. Rubbing a Cat to make static is not a replacement for a Electrostatic Gun.

ESD really does effect the bottom line, so make a case from the bottom line point of view. If you'd like something that you can inflict on Management to try to get them to change the ESD culture, start with chapter seven from the US Air Force Directive General Shop Practice Requirements for the Repair, Maintenance, and Test of Electrical Equipment.

For more in-depth treatise of the subject start with the six part series from the Electrostatic Discharge Association:

ESD Fundamentals:

• Part One--An Introduction to ESD.
• Part Two--Principles of ESD Control.
• Part Three--Basic ESD Control Procedures and Materials.
• Part Four--Training and Auditing.
• Part Five--Device Sensitivity and Testing.
• Part Six--ESD Standards.

Keep in mind that a 25 Volt ESD event to our nano-sized components might be the equivalent of a large Oak Tree getting hit by Lightning at our Human level scale...