Genesis Spacecraft Crash: Bad testing

The spacecraft Genesis collected start dust samples from space and then sent its finding back to Earth. Only the delivery system failed (the picture shows the result of that failure):

Genesis spent 27 months in space, collecting solar wind ions thought to reflect the composition of the early solar system. But on 8 September 2004, a capsule containing the precious ions failed to release its parachutes and crashed into the Utah desert, destroying much of its contents.

How did this happen?

[I]n a detailed report released on Tuesday, the board has confirmed the backwards design as the main cause of the crash. The problem originally stemmed from the fact that the sensor design was copied from NASA's comet-dust collecting Stardust mission, which began development about a year ahead of work on Genesis, says board chair Michael Ryschkewitsch, director of NASA's Applied Engineering and Technology Directorate.

But Genesis used additional electronic components, so it was forced to use two electronics boxes rather than the single one used by Stardust. In the process of making that change, "the person doing the packaging lost track of the [sensor] orientation", Ryschkewitsch told New Scientist.

So, the engineer installed the sensors but did so backwards. Well, that happens, but what about testing the systems? Surely, tests would find this error.

The mistake was never caught because the sensors were never put into a centrifuge and tested, as originally planned. Instead, an electrical engineer – not trained in reviewing complex mechanical drawings – compared drawings of the Stardust and Genesis sensors, and incorrectly concluded the designs were the same.

That's the test. Some engineer, out of his field of expertise, simply compared drawings. Not exactly a test to believe, is it?

I spent years at the Navy's Operational Test and Evaluation Force and we had a mantra: End to end testing in the operational environment. Now, the truth is that that rarely happened. Frequently, testing was the last step in the devlopment of a system and by that time there was little money to pay for testing. So, like what happened here, the Navy did a similar thing. This episode with NASA shows what can happen when testing is short changed. Not a pretty site.

More information is here.