<commit_before>---
layout: post
title: "How to turn screw-ups to your advantage"
date: 2015-01-23
comments: true
categories:
author: <NAME>
image_title: '/images/stock/screwup.jpg'
---

We at GitLab believe in providing a good service to all our users, be they customers, or non-paying users.
In our attempt to support our users, we sometimes make mistakes.

In this post, I'll be discussing two instances where we at GitLab dropped the ball.
I'll go through what went wrong and what we did to address our relationship with the affected party.
Then I'll indicate how that affected our standing with that party.
In the end, I'll mention the process we've followed to address our mistakes.

<!-- more -->

## The brownout

GitLab.com is our free SaaS service.
At close to 20k monthly active users, GitLab.com is used by many of our users for production.
We always notify the comunity of downtimes on its [twitter account](https://twitter.com/gitlabstatus) and offer [paid email support](https://gitlab.recurly.com/subscribe/gitlab-com-bronze-yearly-20) for it.

For years now, we'd been happy to report minimal unplanned downtime on GitLab.com, with planned downtime seldom exceeding one hour.
On the 31st of July 2014 though, GitLab.com went offline for a full 8 hours, affecting the workflow of a serious number of users.

The failure happened in the night between Sunday and Monday in Dutch time, so we only caught wind of the issue in the morning.
Neither the cause nor the solution were evident though.
So despite fervent investigative work, we found ourselves 30 minutes into the call (and 6 hours into the downtime) none the wiser about what had happened.

We decided to openly admit that we did not know why GitLab.com was down, but to be transparent about our actions to figure out what went wrong.
GitLab's CEO, <NAME>, decided to make a [public postmortem](https://docs.google.com/a/gitlab.com/document/d/1ScqXAdb6BjhsDzCo3qdPYbt1uULzgZqPO8zHeHHarS0/edit#heading=h.p95p4f6o0twk) which we've posted on [Hacker News](https://news.ycombinator.com/item?id=8003601).
Quite the number of anonymous users joined our public google doc to get a live feed of our exploration of GitLab, with some even giving feedback on possible actions.

Ultimately, we brought GitLab.com back up, apologized to the community, and updated the postmortem file with all the process improvements we've done to make sure a brownout of this magnitude never happens again.

The Hacker News thread reached the top and kept GitLab in the spotlight for quite the time.
And feedback from the community was [overly positive](https://twitter.com/search?q=gitlab%20postmortem&src=typd) regarding our transparent approach.
We believe GitLab has garnered a lot of trust in being both open about the mistake, and quick to take action.

## Waking up at 5 AM for nothing<commit_msg>Fix spelling, thanks Ben.<commit_after>---
layout: post
title: "How to turn screw-ups to your advantage"
date: 2015-01-23
comments: true
categories:
author: <NAME>
image_title: '/images/stock/screwup.jpg'
---

We at GitLab believe in providing a good service to all our users, be they customers, or non-paying users.
In our attempt to support our users, we sometimes make mistakes.

In this post, I'll be discussing two instances where we at GitLab dropped the ball.
I'll go through what went wrong and what we did to address our relationship with the affected party.
Then I'll indicate how that affected our standing with that party.
In the end, I'll mention the process we've followed to address our mistakes.

<!-- more -->

## The brownout

GitLab.com is our free SaaS service.
At close to 20k monthly active users, GitLab.com is used by many of our users for production.
We always notify the community of downtimes on its [twitter account](https://twitter.com/gitlabstatus) and offer [paid email support](https://gitlab.recurly.com/subscribe/gitlab-com-bronze-yearly-20) for it.

For years now, we'd been happy to report minimal unplanned downtime on GitLab.com, with planned downtime seldom exceeding one hour.
On the 31st of July 2014 though, GitLab.com went offline for a full 8 hours, affecting the workflow of a serious number of users.

The failure happened in the night between Sunday and Monday in Dutch time, so we only caught wind of the issue in the morning.
Neither the cause nor the solution were evident though.
So despite fervent investigative work, we found ourselves 30 minutes into the call (and 6 hours into the downtime) none the wiser about what had happened.

We decided to openly admit that we did not know why GitLab.com was down, but to be transparent about our actions to figure out what went wrong.
GitLab's CEO, <NAME>, decided to make a [public postmortem](https://docs.google.com/a/gitlab.com/document/d/1ScqXAdb6BjhsDzCo3qdPYbt1uULzgZqPO8zHeHHarS0/edit#heading=h.p95p4f6o0twk) which we've posted on [Hacker News](https://news.ycombinator.com/item?id=8003601).
Quite the number of anonymous users joined our public google doc to get a live feed of our exploration of GitLab, with some even giving feedback on possible actions.

Ultimately, we brought GitLab.com back up, apologized to the community, and updated the postmortem file with all the process improvements we've done to make sure a brownout of this magnitude never happens again.

The Hacker News thread reached the top and kept GitLab in the spotlight for quite the time.
And feedback from the community was [overly positive](https://twitter.com/search?q=gitlab%20postmortem&src=typd) regarding our transparent approach.
We believe GitLab has garnered a lot of trust in being both open about the mistake, and quick to take action.

## Waking up at 5 AM for nothing
{'fraction_non_alphanumeric': 0.05992640616786403, 'fraction_numerical': 0.015069213246889784, 'mean_word_length': 4.568780487804878, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '420551', 'n_tokens_mistral': 1659, 'n_tokens_neox': 1515, 'n_words': 832}