<story><title>Moving The New York Times Games Platform to Google App Engine</title><url>https://open.nytimes.com/moving-the-new-york-times-games-platform-to-google-app-engine-e9337f2c9444</url></story><parent_chain><item><author>ciguy</author><text>As a DevOps consultant I&amp;#x27;ve actually worked with clients migrating stacks to and from GCE&amp;#x2F;AWS (Yeah, both ways, not the same client).&lt;p&gt;What I&amp;#x27;ve found in aggregate is that GCE is a bit easier to use at first as AWS has a LOT of features and terminology to learn. When it comes down to it though, many GCE services felt really immature, particularly their CloudSQL offering.&lt;p&gt;One client recently moved from GCE to AWS simply because their CloudSQL (Fully replicated with fail-over setup according to GCE recommendations) kept randomly dying for several minutes at a time. After a LOT of back and forth Google finally admitted that they had updated the replica and the master at the same time, so when it failed over the replica was also down.&lt;p&gt;There were other instances of unexplained downtime that were never adequately explained, but overall that experience was enough for me (And the client) to totally lose faith in the GCE teams competence. Even getting a serious investigation into intermittent downtime and an explanation took over a month. By that time our migration to AWS was in progress.&lt;p&gt;GCE never did explain why they would choose to apply updates to replica + master SQL at the same time and as far as I know they are still doing this. I asked if we could at least be notified of update events, was told that&amp;#x27;s not possible.&lt;p&gt;There were other issues as well that taken together just made GCE seem amateurish. I&amp;#x27;m sure as they mature a bit things will get better, and it is cheaper which is why I wouldn&amp;#x27;t necessarily recommend against them for startups just getting going today. By the time you are really scaling it&amp;#x27;s like they&amp;#x27;ll have more of the kinks worked out.</text></item></parent_chain><comment><author>funkjunky</author><text>(GCP support here)
This is a known bug, I&amp;#x27;ve worked at least a few cases where this happened.  There is a feature coming out soon that will allow different maintenance schedules to be set for masters&amp;#x2F;replicas, which will likely be automatically set for different times.  And, once the kinks get worked out, hopefully we&amp;#x27;ll be able to re-deploy the feature that shifted traffic to failovers while the master is being updated, and eliminating maintenance downtime altogether.</text></comment>
<story><title>Moving The New York Times Games Platform to Google App Engine</title><url>https://open.nytimes.com/moving-the-new-york-times-games-platform-to-google-app-engine-e9337f2c9444</url></story><parent_chain><item><author>ciguy</author><text>As a DevOps consultant I&amp;#x27;ve actually worked with clients migrating stacks to and from GCE&amp;#x2F;AWS (Yeah, both ways, not the same client).&lt;p&gt;What I&amp;#x27;ve found in aggregate is that GCE is a bit easier to use at first as AWS has a LOT of features and terminology to learn. When it comes down to it though, many GCE services felt really immature, particularly their CloudSQL offering.&lt;p&gt;One client recently moved from GCE to AWS simply because their CloudSQL (Fully replicated with fail-over setup according to GCE recommendations) kept randomly dying for several minutes at a time. After a LOT of back and forth Google finally admitted that they had updated the replica and the master at the same time, so when it failed over the replica was also down.&lt;p&gt;There were other instances of unexplained downtime that were never adequately explained, but overall that experience was enough for me (And the client) to totally lose faith in the GCE teams competence. Even getting a serious investigation into intermittent downtime and an explanation took over a month. By that time our migration to AWS was in progress.&lt;p&gt;GCE never did explain why they would choose to apply updates to replica + master SQL at the same time and as far as I know they are still doing this. I asked if we could at least be notified of update events, was told that&amp;#x27;s not possible.&lt;p&gt;There were other issues as well that taken together just made GCE seem amateurish. I&amp;#x27;m sure as they mature a bit things will get better, and it is cheaper which is why I wouldn&amp;#x27;t necessarily recommend against them for startups just getting going today. By the time you are really scaling it&amp;#x27;s like they&amp;#x27;ll have more of the kinks worked out.</text></item></parent_chain><comment><author>tscanausa</author><text>Hi, I am not sure you will read this as it&amp;#x27;s 10 hours after posting. The cloud sql updating both copies sounds like a bug. If you want to email me your case number I can look into it. I know you don&amp;#x27;t work with GCP anymore but I like to resolve the issue for other users.&lt;p&gt;Email: tsg@google.com&lt;p&gt;Disclosure: I work on gcp support. Not paid to be here.</text></comment>