Climategate and Software Development

Two great pieces explaining why the software from the CRU leak is bad, why this is common and why it happens. [Link]

Effective peer review requires that the reviewers have a deep familiarity with the instruments, protocols and procedures used by the experimenters. A chemist familiar with the use of a gas-chromatograph can tell from a description whether the instrument was properly calibrated, configured and used in a particular circumstance. On the other hand, a particle physicist who never uses gas-chromatographs could not verify it was used properly.

Today, each instance of custom-written scientific software is like an unknown, novel piece of scientific hardware. Each piece of software might as well be an “amazing wozzlescope” for all that anyone has experience with its accuracy and precision. No one can even tell if it has subtly malfunctioned. As a result, the peer review of scientific software does not indicate even a whisper of the same level of external objective scrutiny that the peer review of scientific hardware indicates.

How did we let this problem develop? I think it was simply a matter of creeping normalcy. The importance of scientific software grew so slowly that we never developed the habit of questioning it.

Thirty years ago, most scientific software was no more complicated than your average household spreadsheet is today. Software was mostly just a numerical summation tool that merely accelerated the processing of data. If a scientist had a computer, great, but if not it didn’t change the actual conclusions of their experiments. As a result of software’s relatively trivial nature, peer reviewers, journal editorial boards and other scientist paid little attention to the software that experimenters used to produce their results.

Unfortunately, that attitude has persisted even as software has grown from a minor accessory into the tool that actually performs the experiment. Today many papers are nothing but reports of what a unique piece of software spit out after processing this or that great glob of data. These software programs are so huge, so complex and so unique that no one who wasn’t directly involved in their creation could hope to understand them without months of study.

And. [Link]

The design, production and maintenance of large pieces of software require project management skills greater than those required for large material construction projects. Computer programs are the most complicated pieces of technology ever created. By several orders of magnitude they have more “parts” and more interactions between those parts than any other technology.

Software engineers and software project managers have created procedures for managing that complexity. It begins with seemingly trivial things like style guides that regulate what names programmers can give to attributes of software and the associated datafiles. Then you have version control in which every change to the software is recorded in a database. Programmers have to document absolutely everything they do. Before they write code, there is extensive planning by many people. After the code is written comes the dreaded code review in which other programmers and managers go over the code line by line and look for faults. After the code reaches its semi-complete form, it is handed over to Quality Assurance which is staffed by drooling, befanged, malicious sociopaths who live for nothing more than to take a programmer’s greatest, most elegant code and rip it apart and possibly sexually violate it. (Yes, I’m still bitter.)

Institutions pay for all this oversight and double-checking and programmers tolerate it because it is impossible to create a large, reliable and accurate piece of software without such procedures firmly in place. Software is just too complex to wing it.

Clearly, nothing like these established procedures was used at CRU. Indeed, the code seems to have been written overwhelmingly by just two people (one at a time) over the past 30 years. Neither of these individuals was a formally trained programmer and there appears to have been no project planning or even formal documentation. Indeed, the comments of the second programmer, the hapless “Harry”, as he struggled to understand the work of his predecessor are now being read as a kind of programmer’s Icelandic saga describing a death march through an inexplicable maze of ineptitude and boobytraps.

CRU isn’t that unusual. Few scientific teams use any kind of formal software-project management. Why? Well, most people doing scientific programming are not educated as programmers. They’re usually just scientists who taught themselves programming. Moreover, most custom-written scientific software, no matter how large, doesn’t begin as a big, planned project. Instead the software evolves from some small, simple programs written to handle relatively trivial tasks. After a small program proves useful, the scientist finds another related processing task so instead of rewriting everything from scratch, he bolts that new task onto an existing program. Then he does it again and again and again…