<commit_before>---
layout: post
title:  Year of Hell - Surviving software rewrites
date:   2019-02-26
tag: technology
categories:
  - technology
keywords: "rewrite, architecture, platform, BDT, decision, tree"
author: klauss
---

![Silver]({{site.url}}/assets/images/silver_680x200_trans.png){: .center-image }

<!--excerpt.start-->
I've suffered a few "rewrites" over my career, both in professional projects
and in amateur ones. You know how it goes. You start your project coding
with a few reasonable assumptions, and you get a nice proof of concept working.
Then you realize your proof of concept became the actual thing, so you try
(in vain) to "tidy things up". But it works, and well, so you move on.

A few thousand kilometres down the road (because metric!), you look back
in horror at your code. It's awful, you've got tons of features nobody
uses, features that require a lot of code and support.

You also realize you've engineered the whole thing wrong. The assumptions that
seemed reasonable at the time turned out to be untrue after some real-world
experience. Or maybe they were true, but not anymore. Whatever the case, you
realize your whole software architecture no longer fits the problem.

Finally, you notice the quality of the code has considerably degraded. It's hard
to read. It's hard to think about it. Patches upon patches, some for optimization,
some for new features nobody wants, some to fix some obscure bug or worse,
some workaround to deal with someone else's bug.
A million paper cuts made your software hard to read, and hard to maintain.

And you think to yourself... I could do this much better if I rewrote it
today, with the knowledge I accumulated down the road. I know I can.

And if you've been in that situation and decided to go for it, [you know
how it ends as well](https://medium.com/@herbcaudill/lessons-from-6-software-rewrite-stories-635e4c8f7c22).
Almost always, in flames.
<!--excerpt.end-->

Rewrites are very difficult, very risky projects:

* **They're extremely long projects**
  
  It's not due to laziness
  it took years to get the software to that point. It takes time
  to discover how to do things, and more time to actually
  do those things. It's not like you'll be able to redo all that work in a week.
  Or a month. Or two.

* **They're risky**

  Since you've got to develop everything from scratch again,
  there's a very high chance you'll find something unexpected along the way.
  It's a statistical certainty. With so much work ahead, you'd have an easier
  time winning the lottery than finishing the rewrite on time and without a hitch.
  Add that the long time frames involved naturally imply huge
  error bars on any time estimation, and you've got eternal headaches for management.

* **They're giant scope creep magnets**

  Since we're rewriting it, I'll take the opportunity
  to fix this thing I did wrong. And since we're at it, we can
  improve on this... and that... famous last words.<commit_msg>Custom excerpt that does fit in the home page<commit_after>---
layout: post
title:  Year of Hell - Surviving software rewrites
date:   2019-02-26
tag: technology
categories:
  - technology
keywords: "rewrite, architecture, platform, BDT, decision, tree"
author: klauss
content: |
  I've suffered a few "rewrites" over my career.
  You know how it goes. You start your project coding
  with a few reasonable assumptions, and you get a nice proof of concept working.
  Then you realize your PoC became the actual thing, so you try
  (in vain) to "tidy things up". But it works, so you move on.

  A little later you look back in horror at your code, and you think to yourself...
  I could do this much better if I rewrote it today.

  And if you've been in that situation, you know how it ends.
  Almost always, in flames.
---

![Silver]({{site.url}}/assets/images/silver_680x200_trans.png){: .center-image }

I've suffered a few "rewrites" over my career, both in professional projects
and in amateur ones. You know how it goes. You start your project coding
with a few reasonable assumptions, and you get a nice proof of concept working.
Then you realize your proof of concept became the actual thing, so you try
(in vain) to "tidy things up". But it works, and well, so you move on.

A few thousand kilometres down the road (because metric!), you look back
in horror at your code. It's awful, you've got tons of features nobody
uses, features that require a lot of code and support.

You also realize you've engineered the whole thing wrong. The assumptions that
seemed reasonable at the time turned out to be untrue after some real-world
experience. Or maybe they were true, but not anymore. Whatever the case, you
realize your whole software architecture no longer fits the problem.

Finally, you notice the quality of the code has considerably degraded. It's hard
to read. It's hard to think about it. Patches upon patches, some for optimization,
some for new features nobody wants, some to fix some obscure bug or worse,
some workaround to deal with someone else's bug.
A million paper cuts made your software hard to read, and hard to maintain.

And you think to yourself... I could do this much better if I rewrote it
today, with the knowledge I accumulated down the road. I know I can.

And if you've been in that situation and decided to go for it, [you know
how it ends as well](https://medium.com/@herbcaudill/lessons-from-6-software-rewrite-stories-635e4c8f7c22).
Almost always, in flames.

Rewrites are very difficult, very risky projects:

* **They're extremely long projects**
  
  It's not due to laziness
  it took years to get the software to that point. It takes time
  to discover how to do things, and more time to actually
  do those things. It's not like you'll be able to redo all that work in a week.
  Or a month. Or two.

* **They're risky**

  Since you've got to develop everything from scratch again,
  there's a very high chance you'll find something unexpected along the way.
  It's a statistical certainty. With so much work ahead, you'd have an easier
  time winning the lottery than finishing the rewrite on time and without a hitch.
  Add that the long time frames involved naturally imply huge
  error bars on any time estimation, and you've got eternal headaches for management.

* **They're giant scope creep magnets**

  Since we're rewriting it, I'll take the opportunity
  to fix this thing I did wrong. And since we're at it, we can
  improve on this... and that... famous last words.
{'fraction_non_alphanumeric': 0.06129032258064516, 'fraction_numerical': 0.00706605222734255, 'mean_word_length': 3.895488721804511, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4813587', 'n_tokens_mistral': 1828, 'n_tokens_neox': 1681, 'n_words': 1038}