<story><title>Why I distrust Google Cloud more than than AWS or Azure</title><url>http://www.iasylum.net/writings/2021-04-21-why-I-distrust-google-cloud-more-than-AWS-or-Azure.html</url></story><parent_chain><item><author>goodcjw2</author><text>[DISCLAMER: I used to work at Google in general, but not at Google Cloud]&lt;p&gt;I&amp;#x27;m not sure whether this has been discussed here before, but I&amp;#x27;d love to take this forum to share an angle from the tech side of things:&lt;p&gt;IMO, Google is _cursed_ to keep deprecating its products and services. It&amp;#x27;s cursed by Google&amp;#x27;s famous choice of mono-repo tech stack.&lt;p&gt;It makes all the sense and has all the benefits. But at a cost: we had to keep every single line of code in active development mode. Whenever someone changed a line of code in a random file that&amp;#x27;s three steps away on your dependency chain, you will get a ticket to understand what has changed, make changes and fire up every tests (also fix them in 99% of the cases).&lt;p&gt;Yeah, the &amp;quot;Fuck You. Drop whatever you are doing because it’s not important. What is important is OUR time. It’s costing us time and money to support our shit, and we’re tired of it, so we’re not going to support it anymore.&amp;quot; is kind of true story for internal engineers.&lt;p&gt;We once had a shipped product (which took about 20-engineer-month to develop in the first place) in maintenance mode, but still requires a full time engineer to deal with those random things all the time. Would have save 90% of that person&amp;#x27;s time it it&amp;#x27;s on a sperate branch and we only need to focus on security patches. (NO, there is no such concept of branching in Google&amp;#x27;s dev system).&lt;p&gt;We kept doing this for a while and soon realized that there is no way we can sustain this, especially after the only guys who understand how everything works switched teams. Thus, it just became obvious that deprecation is the only &amp;quot;responsible&amp;quot; and &amp;quot;reasonable&amp;quot; choice.&lt;p&gt;Honestly, I think Google&amp;#x27;s engineering practice is somewhat flawed for the lack of a good solution to support shipped products in maintenance. As a result, there is either massively successful products being actively developed; or deprecated products.</text></item></parent_chain><comment><author>paxys</author><text>I have also worked at Google (in an unrelated department) and completely disagree with this. Maintaining old code &amp;amp; services is a problem everywhere. Monorepo vs multirepo, monolith service vs microservices etc. all have nothing to do with it. There will always be a broken dependency, a service&amp;#x2F;API&amp;#x2F;library you rely on about to deprecate, new urgent security patches, an outage somewhere upstream or downstream which you have to investigate, an important customer hitting an edge case which was hidden for years. You will &lt;i&gt;always&lt;/i&gt; need a dedicated team to support a live product regardless of how it is engineered.&lt;p&gt;The problem at Google was (and maybe still is) with lack of incentives at the product level to do any of this. You don&amp;#x27;t get a fat bonus and promotion for saying that you kept things working as they should, made an incremental update or fixed bugs. When your packet goes up to the committee (who don&amp;#x27;t know you and know nothing about your team or background), the only thing that works in your favor is a successful new product launch.&lt;p&gt;And as an engineer you still have multiple avenues to showcase your skills. That new product manager you just hired from Harvard Business School who is eager to climb the ladder does not. And due to the lack of a central cohesive product strategy, this PM also has complete control of your team&amp;#x27;s annual roadmap.</text></comment>
<story><title>Why I distrust Google Cloud more than than AWS or Azure</title><url>http://www.iasylum.net/writings/2021-04-21-why-I-distrust-google-cloud-more-than-AWS-or-Azure.html</url></story><parent_chain><item><author>goodcjw2</author><text>[DISCLAMER: I used to work at Google in general, but not at Google Cloud]&lt;p&gt;I&amp;#x27;m not sure whether this has been discussed here before, but I&amp;#x27;d love to take this forum to share an angle from the tech side of things:&lt;p&gt;IMO, Google is _cursed_ to keep deprecating its products and services. It&amp;#x27;s cursed by Google&amp;#x27;s famous choice of mono-repo tech stack.&lt;p&gt;It makes all the sense and has all the benefits. But at a cost: we had to keep every single line of code in active development mode. Whenever someone changed a line of code in a random file that&amp;#x27;s three steps away on your dependency chain, you will get a ticket to understand what has changed, make changes and fire up every tests (also fix them in 99% of the cases).&lt;p&gt;Yeah, the &amp;quot;Fuck You. Drop whatever you are doing because it’s not important. What is important is OUR time. It’s costing us time and money to support our shit, and we’re tired of it, so we’re not going to support it anymore.&amp;quot; is kind of true story for internal engineers.&lt;p&gt;We once had a shipped product (which took about 20-engineer-month to develop in the first place) in maintenance mode, but still requires a full time engineer to deal with those random things all the time. Would have save 90% of that person&amp;#x27;s time it it&amp;#x27;s on a sperate branch and we only need to focus on security patches. (NO, there is no such concept of branching in Google&amp;#x27;s dev system).&lt;p&gt;We kept doing this for a while and soon realized that there is no way we can sustain this, especially after the only guys who understand how everything works switched teams. Thus, it just became obvious that deprecation is the only &amp;quot;responsible&amp;quot; and &amp;quot;reasonable&amp;quot; choice.&lt;p&gt;Honestly, I think Google&amp;#x27;s engineering practice is somewhat flawed for the lack of a good solution to support shipped products in maintenance. As a result, there is either massively successful products being actively developed; or deprecated products.</text></item></parent_chain><comment><author>krasin</author><text>I work for Google for the 3rd time; 12+ years in total. It&amp;#x27;s first time when this issue is brought to my conscience. I think that the issue exists, but it&amp;#x27;s not due to the monorepo, it&amp;#x27;s due to the internal APIs changing.&lt;p&gt;I learned how to avoid the Google3 tax that you mentioned, when the old thing is deprecated, and the new one is not working yet.&lt;p&gt;Surprisingly, the answer for me was to embrace Google Cloud: its APIs have stability guarantees. My current project depends on Google Cloud Storage, Cloud Spanner, Cloud Firestore and very few internal technologies.&lt;p&gt;I believe that this is in general a trend at Google: increasing reliance on Google Cloud for new projects. In this sense, both internal and external developers are in the same boat.&lt;p&gt;As for the monorepo - it&amp;#x27;s a blessing, in my perspective. Less rotten code, much easier to contribute improvements across the stack.</text></comment>