<story><title>Notes on Theory of Distributed Systems [pdf]</title><url>http://www.cs.yale.edu/homes/aspnes/classes/465/notes.pdf</url></story><parent_chain><item><author>phtrivier</author><text>Is anyone teaching &amp;quot;Practice of boring Distributed Systems 101 for dummies on a budget with a tight schedule&amp;quot; ?&lt;p&gt;As in, &amp;quot;we have a PHP monolith used by all of 12 people in the accounting department, and for some reason we&amp;#x27;ve been tasked with making it run on multiple machines (&amp;quot;for redundancy&amp;quot; or something) by next month.&lt;p&gt;The original developpers left to start a Bitcoin scam.&lt;p&gt;Some exec read about the &amp;quot;cloud&amp;quot;, but we&amp;#x27;ll probably get just enough budget to buy a coffee to an AWS salesman.&lt;p&gt;Don&amp;#x27;t even dream of hiring a &amp;quot;DevOps&amp;quot; to deploy a kubernetes cluster to orchestrate anything. Don&amp;#x27;t dream of hiring anyone, actually. Or, paying anything, for that matter.&lt;p&gt;You had one machine ; here is a second machine. That&amp;#x27;s a 100% increase in your budget, now go get us some value with that !&lt;p&gt;And don&amp;#x27;t come back in three months to ask for another budget to &amp;#x27;upgrade&amp;#x27;.&amp;quot;&lt;p&gt;Where would someone start ?&lt;p&gt;(EDIT: To clarify, this is a tongue in cheek hyperbole scenario, not a cry for immediate help. Thanks to all who offered help ;)&lt;p&gt;Yet, I&amp;#x27;m curious about any resource on how to attack such problems, because I can only find material on how to handle large scale multi million users high availability stuff.)</text></item></parent_chain><comment><author>keule</author><text>&amp;gt; As in, &amp;quot;we have a PHP monolith used by all of 12 people in the accounting department, and for some reason we&amp;#x27;ve been tasked with making it run on multiple machines (&amp;quot;for redundancy&amp;quot; or something) by next month.&lt;p&gt;Usually, your monolith has these components: a web server (apache&amp;#x2F;nginx + php), a database, and other custom tooling.&lt;p&gt;&amp;gt; Where would someone start ?&lt;p&gt;I think a first step is to move the database to something managed, like AWS RDS or Azure Managed Databases. Herein lies the basis for scaling out your web tier later. And here you will find the most pain because there are likely: custom backup scripts, cron jobs, and other tools that access the DB in unforeseen ways.&lt;p&gt;If you get over that hump you have done your first big step towards a more robust model. Your DB will have automated backups, managed updates, rollover, read replicas etc. You may or may not see a performance increase, because you effectively split your workload across two machines.&lt;p&gt;_THEN_ you can front your web tier with a load balancer, i.e. you load balance to one machine. This gives you: better networking, custom error pages, support for sticky sessions (you likely need them later), and better&amp;#x2F;more monitoring.&lt;p&gt;From thereon you can start working on removing those custom scripts of the web tier machine and start splitting this into an _actual_ load-balanced infrastructure, going to two web-tier machines, where traffic is routed using sticky-sessions.&lt;p&gt;Depending on the application design you can start introducing containers.&lt;p&gt;Now, this approach will not give you a _cloud-native awesome microservice architecture_ with CI&amp;#x2F;CD and devops. But it will be enough to have higher availability and more robust handling of the (predictable) load in the near future. And on the way, you will remove bad patterns that eventually allow you to go to a better approach.&lt;p&gt;I would be interested in hearing if more people face this challenge. I don&amp;#x27;t know if guides exist around this on the webs.</text></comment>
<story><title>Notes on Theory of Distributed Systems [pdf]</title><url>http://www.cs.yale.edu/homes/aspnes/classes/465/notes.pdf</url></story><parent_chain><item><author>phtrivier</author><text>Is anyone teaching &amp;quot;Practice of boring Distributed Systems 101 for dummies on a budget with a tight schedule&amp;quot; ?&lt;p&gt;As in, &amp;quot;we have a PHP monolith used by all of 12 people in the accounting department, and for some reason we&amp;#x27;ve been tasked with making it run on multiple machines (&amp;quot;for redundancy&amp;quot; or something) by next month.&lt;p&gt;The original developpers left to start a Bitcoin scam.&lt;p&gt;Some exec read about the &amp;quot;cloud&amp;quot;, but we&amp;#x27;ll probably get just enough budget to buy a coffee to an AWS salesman.&lt;p&gt;Don&amp;#x27;t even dream of hiring a &amp;quot;DevOps&amp;quot; to deploy a kubernetes cluster to orchestrate anything. Don&amp;#x27;t dream of hiring anyone, actually. Or, paying anything, for that matter.&lt;p&gt;You had one machine ; here is a second machine. That&amp;#x27;s a 100% increase in your budget, now go get us some value with that !&lt;p&gt;And don&amp;#x27;t come back in three months to ask for another budget to &amp;#x27;upgrade&amp;#x27;.&amp;quot;&lt;p&gt;Where would someone start ?&lt;p&gt;(EDIT: To clarify, this is a tongue in cheek hyperbole scenario, not a cry for immediate help. Thanks to all who offered help ;)&lt;p&gt;Yet, I&amp;#x27;m curious about any resource on how to attack such problems, because I can only find material on how to handle large scale multi million users high availability stuff.)</text></item></parent_chain><comment><author>throwaway787544</author><text>If someone would pay for it I&amp;#x27;d  write that book. There are lots of different methods for different scenarios. There are some books on it but they&amp;#x27;re either very dry and technical or have very few examples.&lt;p&gt;Here&amp;#x27;s the cliffs notes version for your situation:&lt;p&gt;1. Build a server. Make an image&amp;#x2F;snapshot of it.&lt;p&gt;2. Build a second server from the snapshot.&lt;p&gt;3. Use rsync to copy files your PHP app writes from one machine (&amp;#x27;primary&amp;#x27;) to another (&amp;#x27;secondary&amp;#x27;).&lt;p&gt;4. To make a &amp;quot;safe&amp;quot; change, change the secondary server, test it.&lt;p&gt;5. To &amp;quot;deploy&amp;quot; the change, snapshot the secondary, build a new third server, stop writes on the primary, sync over the files to the third server one last time, point the primary hostname at the third server IP, test this new primary server, destroy the old primary server.&lt;p&gt;6. If you ever need to &amp;quot;roll back&amp;quot; a change, you can do that while there&amp;#x27;s still three servers up (blue&amp;#x2F;green), or deploy a new server with the last working snapshot.&lt;p&gt;7. Set up PagerDuty to wake you up if the primary dies. When it does, change the hostname of the first box to point to the IP of the second box.&lt;p&gt;That&amp;#x27;s just one way that is very simple. It is a redundant active&amp;#x2F;passive distributed system with redundant storage and immutable blue&amp;#x2F;green deployments. It can be considered high-availability although that term is somewhat loaded; ideally you&amp;#x27;d make as much of the system HA as possible, such as independent network connections to the backbone, independent power drops, UPC, etc (both for bare-metal and VMs).&lt;p&gt;You can get much more complicated but that&amp;#x27;s good enough for what they want (redundancy) and it buys you a lot of other benefits.</text></comment>