<story><title>The $440M software error at Knight Capital (2019)</title><url>https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html</url></story><parent_chain><item><author>fullsend</author><text>I appreciate your comment about pay. Recruiters will often tell me &amp;quot;it&amp;#x27;s finance so of course the pay will be substantial.&amp;quot; Then when we get to talking numbers they&amp;#x27;re like &amp;quot;300k a year&amp;quot;. Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it&amp;#x27;s 400 plus bonus or whatever, which is based on fund performance and yada yada. But it feels way off. I had heard so much about the staggering paydays at these places but it seems you need an ML PHD or some trading chops to be part of that.</text></item><item><author>kevstev</author><text>I worked in algo trading for years, eventually got out because quite frankly the level of risk I was carrying on my shoulders everyday for what I was being paid were just way out of whack, I at least personally never got the huge pay days that people talked about until after I left finance for more pure tech. Interestingly, I worked at Knight and my team pioneered trying to blow up the firm, but that was in 2004, and things were much friendlier- instead of front page news, it was a small blurb on page 3 of the markets section of the WSJ.&lt;p&gt;Anyway, I still have friends in that business. It hasn&amp;#x27;t really changed, they have too few people covering systems that are quite complex and while there are checks and such, no one really understands things entirely from end to end in detail that can prevent all problems.&lt;p&gt;I will never invest directly in an investment bank- either through carelessness or maliciousness I could have easily caused a 9 figure loss, if not more, and there were probably a thousand other people in the same position.&lt;p&gt;When I read the detailed writeup around this a few years back, I think by far the biggest issue was reusing a tag that had been previously used to denote which strategy to use.  I understand why they may have chosen to do so, at the Big Bank I was working at, getting a new fix tag to be passed through all the layers properly would involve at least two other teams and coordinating releases and probably several weeks worth of meetings. If you just reuse an old value you can avoid all that since everything is already set up.</text></item><item><author>bfm</author><text>The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012.&lt;p&gt;Some of the issues mentioned include:&lt;p&gt;&lt;pre&gt;&lt;code&gt;  - Keeping synthetic test data generation as part of a production build.
  - Keeping dead code for years.
  - Re-purposing a feature flag.
  - Refactoring without regression tests.
  - Manual deployments without peer reviews. They forgot to update one of their servers with the new code.
  - Automated alerts sent via email were ignored.
  - Rolled back to a version of the code running on the server they forgot to update, making things worse.
  - Rushing out a release without proper software engineering hygiene.
&lt;/code&gt;&lt;/pre&gt;
The article suggests improvements that could have prevented the chain of events.&lt;p&gt;For those here who are in HFT circles, have things improved after the Knight Capital Group debacle?&lt;p&gt;edit: formatting</text></item></parent_chain><comment><author>kevstev</author><text>Yeah, pay at the big banks is shit really, especially when you consider the utter lack of work&amp;#x2F;life balance. I left in 2013 making 150k, which was supposed to be supplemented by a ~40% bonus for the level I was at, but each year was &amp;quot;well its been a tough year...&amp;quot; and after getting a token amount one year, and then zeroes the next 2, after working 50-60 hour weeks, I was like I am not only done with this place, but this industry, and left for a 50% pay raise, my TC is now 4x where it was in those days. A neighbor of mine is more or less sitting in my exact seat there, and is somewhere in the 200-250k range.&lt;p&gt;That said, I went back to finance to work at one of the premier hedge funds out there, and they actually lived up to their expectations in terms of comp, that place was more like a tech firm though than any other firm I have ever worked at aside for maybe Knight. 8% annual increases were normal there. You can look in my post history back to 2018 if you want the name, I recently left after 5 years there and just want to stay out of their crosshairs- they monitor social media aggressively and there is deferred comp at stake.&lt;p&gt;At big banks, there are really only a very small number of people who are in tech that are getting paid- you have to know which questions to ask- where is the bonus pool coming from- are you &amp;quot;in the business&amp;quot; or the tech pool, which is a second class of citizen. I would have to be in a pretty bad place to ever consider going back to a bank, it was borderline abusive... always dangling the prospect of that big check that would make it worth it</text></comment>
<story><title>The $440M software error at Knight Capital (2019)</title><url>https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html</url></story><parent_chain><item><author>fullsend</author><text>I appreciate your comment about pay. Recruiters will often tell me &amp;quot;it&amp;#x27;s finance so of course the pay will be substantial.&amp;quot; Then when we get to talking numbers they&amp;#x27;re like &amp;quot;300k a year&amp;quot;. Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it&amp;#x27;s 400 plus bonus or whatever, which is based on fund performance and yada yada. But it feels way off. I had heard so much about the staggering paydays at these places but it seems you need an ML PHD or some trading chops to be part of that.</text></item><item><author>kevstev</author><text>I worked in algo trading for years, eventually got out because quite frankly the level of risk I was carrying on my shoulders everyday for what I was being paid were just way out of whack, I at least personally never got the huge pay days that people talked about until after I left finance for more pure tech. Interestingly, I worked at Knight and my team pioneered trying to blow up the firm, but that was in 2004, and things were much friendlier- instead of front page news, it was a small blurb on page 3 of the markets section of the WSJ.&lt;p&gt;Anyway, I still have friends in that business. It hasn&amp;#x27;t really changed, they have too few people covering systems that are quite complex and while there are checks and such, no one really understands things entirely from end to end in detail that can prevent all problems.&lt;p&gt;I will never invest directly in an investment bank- either through carelessness or maliciousness I could have easily caused a 9 figure loss, if not more, and there were probably a thousand other people in the same position.&lt;p&gt;When I read the detailed writeup around this a few years back, I think by far the biggest issue was reusing a tag that had been previously used to denote which strategy to use.  I understand why they may have chosen to do so, at the Big Bank I was working at, getting a new fix tag to be passed through all the layers properly would involve at least two other teams and coordinating releases and probably several weeks worth of meetings. If you just reuse an old value you can avoid all that since everything is already set up.</text></item><item><author>bfm</author><text>The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012.&lt;p&gt;Some of the issues mentioned include:&lt;p&gt;&lt;pre&gt;&lt;code&gt;  - Keeping synthetic test data generation as part of a production build.
  - Keeping dead code for years.
  - Re-purposing a feature flag.
  - Refactoring without regression tests.
  - Manual deployments without peer reviews. They forgot to update one of their servers with the new code.
  - Automated alerts sent via email were ignored.
  - Rolled back to a version of the code running on the server they forgot to update, making things worse.
  - Rushing out a release without proper software engineering hygiene.
&lt;/code&gt;&lt;/pre&gt;
The article suggests improvements that could have prevented the chain of events.&lt;p&gt;For those here who are in HFT circles, have things improved after the Knight Capital Group debacle?&lt;p&gt;edit: formatting</text></item></parent_chain><comment><author>caffeine</author><text>The attitude that finance pays more is a leftover from a previous era.  10-15 years ago it was true: the profits from HFT were so also way, way bigger and split up amongst a much smaller group of firms.&lt;p&gt;Now those firms are all in a completely competitive industry squeezing each other for basis points.&lt;p&gt;Meanwhile the definition of a FAANG is that it has an effective monopoly, and these companies are taking in way more money than the HFT industry. (Netflix is losing its monopoly but we can’t really drop N from the acronym without a replacement..)</text></comment>