Why do many talented scientists write horrible software? I am a software engineer and I have been working with people with academic backgrounds for several years. Many times, I've noticed that (even otherwise brilliant scientists) produce code of extremely low quality (unless their background was precisely Computer Science).

Since those people are very good in doing their research - and eventually obtain remarkable results - it seems they are clever enough to write decent code. Is it just that they don't think it's worth the effort? Plain arrogance? Lack of time?

Examples

It seems to me that in academia the most popular languages are C/C++ and Python (neglecting MATLAB and other vendor-specific languages). The language where I have seen the most amazing pieces of junk is actually C++. The main points are:


Really, really naive C++ code. They claim they chose C++ over Java/Python/whatever because "it's faster", but they new everything, even an array of 3 floats that is deallocated few lines later, where 3 is known at compile time.
They have learned pointers from C and they use only them.
Some of them (not most of them) have read some random blog posts about OOP and now put virtual everywhere, using abnormal levels of abstraction.
They are convinced of pointless optimization choices.
They lack proper memory management.
They copy/paste massive amounts of code from project to project and within the same project as well.


And in this list I am omitting the problems with the process, rather than the product. Scientists use:


no version control,
no automated builds,
no documentation,
no software process at all (neither agile, nor traditional waterfall).


The workflow is:


  I devise the algorithm, I write it as a massive 10k LOC piece of C++ and I click build somewhere.


As this assessment could be probably biased by my own experience, I have inspected some open source projects run by researchers (and maybe a few software engineers) and cited in many important papers. Virtually all of them were:


crashing on corner cases,
had ugly GUIs,
and the code - in my opinion - was ready for a complete rewrite.
I'll have a go at this. This is mainly my personal view based on my use and implementation of academic software. Like many of the comments already mentioned, I don't think bad software is specific to or even more frequent in  academia. That said, I think there are a few reasons why it occurs that are specific to the field.

Software is not a priority

In academics, the key performance indicators are all about paper publications. Software, while highly useful in my opinion, has very little value. More often than not, software implementations are side tracks or proof-of-concepts at best used to bump up the citation count (ofcourse exceptions exist).

As a software engineer, I am sure you are aware of the knowledge, time and effort required to produce quality software. Given the publish or perish mantra in academia, this time investment is often spent writing a new publication. It's a risk-reward consideration.

My personal point of view is that quality software is very important. For example, often, the first step in comparing a new algorithm with previous state-of-the-art is reimplementing the state-of-the-art due to a lack of implementation. Obviously this introduces a time delay and can cause a series of bugs. That said, I think in general little will change until software gets valued by performance indicators somehow.

Researchers are not programmers

Most researchers have little or no programming experience, though YMMV depending on the field. I think it's fair to say the majority of researchers learned to program by themselves when confronted with problems where they need it. Superficially learning a language is typically not the problem, but you need more than superficial knowledge to produce quality software (choosing data structures, design patterns, deep knowledge of the language, ...). This is a hurdle for self-taught programmers without a computer science background.

Another problem that less experienced programmers face is not realizing when refactoring is necessary. You can find countless pieces of poorly structured software. In research, this is a natural consequence of iteratively implementing while designing an algorithm which evolves into a monstrous piece of software full of hacks. That monster might still do what it was meant to, though, if you know exactly how to ask it nicely. For many researchers that's where the story ends: get the results and put the beast away forever. It often takes a serious time investment to wash the monster prior to talking it out for a walk in public. This is not always worth it.

Trends in machine learning

My field is machine learning, in which software is being valued increasingly. Examples of this trend include a growing software repository and a position paper by some big names in the field. I am very happy with this evolution, because quality software allows the field as a whole to progress faster and increases reproducibility.

A current patch to the problem is being able to publish papers about peer-reviewed software implementations. I know this is possible in machine learning and statistics. Such software is usually of higher quality.