<commit_before># Backup restore

![backup banner](backup_hrz.png)

An application data backup creates an archive file that contains the database,
all repositories and all attachments.
This archive will be saved in `backup_path`, which is specified in the
`config/gitlab.yml` file.
The filename will be `[TIMESTAMP]_gitlab_backup.tar`, where `TIMESTAMP`
identifies the time at which each backup was created.

> In GitLab 8.15 we changed the timestamp format from `EPOCH` (`1393513186`)
> to `EPOCH_YYYY_MM_DD` (`1393513186_2014_02_27`)

You can only restore a backup to exactly the same version of GitLab on which it
was created.  The best way to migrate your repositories from one server to
another is through backup restore.

To restore a backup, you will also need to restore `/etc/gitlab/gitlab-secrets.json`
(for omnibus packages) or `/home/git/gitlab/.secret` (for installations
from source). This file contains the database encryption key and CI secret
variables used for two-factor authentication. If you fail to restore this
encryption key file along with the application data backup, users with two-factor
authentication enabled will lose access to your GitLab server.

## Create a backup of the GitLab system

Use this command if you've installed GitLab with the Omnibus package:
```
sudo gitlab-rake gitlab:backup:create
```
Use this if you've installed GitLab from source:
```
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
```
If you are running GitLab within a Docker container, you can run the backup from the host:
```
docker exec -t <container name> gitlab-rake gitlab:backup:create
```

You can specify that portions of the application data be skipped using the
environment variable `SKIP`. You can skip:

- `db` (database)
- `uploads` (attachments)
- `repositories` (Git repositories data)
- `builds` (CI build output logs)
- `artifacts` (CI build artifacts)
- `lfs` (LFS objects)
- `registry` (Container Registry images)

Separate multiple data types to skip using a comma. For example:

```
sudo gitlab-rake gitlab:backup:create SKIP=db,uploads
```

Example output:

```
Dumping database tables:
- Dumping table events... [DONE]
- Dumping table issues... [DONE]
- Dumping table keys... [DONE]
- Dumping table merge_requests... [DONE]
- Dumping table milestones... [DONE]
- Dumping table namespaces... [DONE]
- Dumping table notes... [DONE]
- Dumping table projects... [DONE]
- Dumping table protected_branches... [DONE]
- Dumping table schema_migrations... [DONE]
- Dumping table services... [DONE]
- Dumping table snippets... [DONE]
- Dumping table taggings... [DONE]
- Dumping table tags... [DONE]
- Dumping table users... [DONE]
- Dumping table users_projects... [DONE]
- Dumping table web_hooks... [DONE]
- Dumping table wikis... [DONE]
Dumping repositories:
- Dumping repository abcd... [DONE]
Creating backup archive: $TIMESTAMP_gitlab_backup.tar [DONE]
Deleting tmp directories...[DONE]
Deleting old backups... [SKIPPING]
```

## Upload backups to remote (cloud) storage

Starting with GitLab 7.4 you can let the backup script upload the '.tar' file it creates.
It uses the [Fog library](http://fog.io/) to perform the upload.
In the example below we use Amazon S3 for storage, but Fog also lets you use
[other storage providers](http://fog.io/storage/). GitLab
[imports cloud drivers](https://gitlab.com/gitlab-org/gitlab-ce/blob/30f5b9a5b711b46f1065baf755e413ceced5646b/Gemfile#L88)
for AWS, Google, OpenStack Swift and Rackspace as well. A local driver is
[also available](#uploading-to-locally-mounted-shares).

For omnibus packages, add the following to `/etc/gitlab/gitlab.rb`:

```ruby
gitlab_rails['backup_upload_connection'] = {
  'provider' => 'AWS',
  'region' => 'eu-west-1',
  'aws_access_key_id' => 'AKIAKIAKI',
  'aws_secret_access_key' => 'secret123'
  # If using an IAM Profile, leave aws_access_key_id & aws_secret_access_key empty
  # ie. 'aws_access_key_id' => '',
  # 'use_iam_profile' => 'true'
}
gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket'
```

Make sure to run `sudo gitlab-ctl reconfigure` after editing `/etc/gitlab/gitlab.rb` to reflect the changes.

For installations from source:
<commit_msg>Add pages to excluded directories in backup task [ci skip]<commit_after># Backup restore

![backup banner](backup_hrz.png)

An application data backup creates an archive file that contains the database,
all repositories and all attachments.
This archive will be saved in `backup_path`, which is specified in the
`config/gitlab.yml` file.
The filename will be `[TIMESTAMP]_gitlab_backup.tar`, where `TIMESTAMP`
identifies the time at which each backup was created.

> In GitLab 8.15 we changed the timestamp format from `EPOCH` (`1393513186`)
> to `EPOCH_YYYY_MM_DD` (`1393513186_2014_02_27`)

You can only restore a backup to exactly the same version of GitLab on which it
was created.  The best way to migrate your repositories from one server to
another is through backup restore.

To restore a backup, you will also need to restore `/etc/gitlab/gitlab-secrets.json`
(for omnibus packages) or `/home/git/gitlab/.secret` (for installations
from source). This file contains the database encryption key and CI secret
variables used for two-factor authentication. If you fail to restore this
encryption key file along with the application data backup, users with two-factor
authentication enabled will lose access to your GitLab server.

## Create a backup of the GitLab system

Use this command if you've installed GitLab with the Omnibus package:
```
sudo gitlab-rake gitlab:backup:create
```
Use this if you've installed GitLab from source:
```
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
```
If you are running GitLab within a Docker container, you can run the backup from the host:
```
docker exec -t <container name> gitlab-rake gitlab:backup:create
```

You can specify that portions of the application data be skipped using the
environment variable `SKIP`. You can skip:

- `db` (database)
- `uploads` (attachments)
- `repositories` (Git repositories data)
- `builds` (CI build output logs)
- `artifacts` (CI build artifacts)
- `lfs` (LFS objects)
- `registry` (Container Registry images)

Separate multiple data types to skip using a comma. For example:

```
sudo gitlab-rake gitlab:backup:create SKIP=db,uploads
```

Example output:

```
Dumping database tables:
- Dumping table events... [DONE]
- Dumping table issues... [DONE]
- Dumping table keys... [DONE]
- Dumping table merge_requests... [DONE]
- Dumping table milestones... [DONE]
- Dumping table namespaces... [DONE]
- Dumping table notes... [DONE]
- Dumping table projects... [DONE]
- Dumping table protected_branches... [DONE]
- Dumping table schema_migrations... [DONE]
- Dumping table services... [DONE]
- Dumping table snippets... [DONE]
- Dumping table taggings... [DONE]
- Dumping table tags... [DONE]
- Dumping table users... [DONE]
- Dumping table users_projects... [DONE]
- Dumping table web_hooks... [DONE]
- Dumping table wikis... [DONE]
Dumping repositories:
- Dumping repository abcd... [DONE]
Creating backup archive: $TIMESTAMP_gitlab_backup.tar [DONE]
Deleting tmp directories...[DONE]
Deleting old backups... [SKIPPING]
```

## Exclude specific directories from the backup

You can choose what should be backed up by adding the environment variable `SKIP`.
The available options are:

* `db`
* `uploads` (attachments)
* `repositories`
* `builds` (CI build output logs)
* `artifacts` (CI build artifacts)
* `lfs` (LFS objects)
* `pages` (pages content)

Use a comma to specify several options at the same time:

```
# use this command if you've installed GitLab with the Omnibus package
sudo gitlab-rake gitlab:backup:create SKIP=db,uploads

# if you've installed GitLab from source
sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production
```

## Upload backups to remote (cloud) storage

Starting with GitLab 7.4 you can let the backup script upload the '.tar' file it creates.
It uses the [Fog library](http://fog.io/) to perform the upload.
In the example below we use Amazon S3 for storage, but Fog also lets you use
[other storage providers](http://fog.io/storage/). GitLab
[imports cloud drivers](https://gitlab.com/gitlab-org/gitlab-ce/blob/30f5b9a5b711b46f1065baf755e413ceced5646b/Gemfile#L88)
for AWS, Google, OpenStack Swift and Rackspace as well. A local driver is
[also available](#uploading-to-locally-mounted-shares).

For omnibus packages, add the following to `/etc/gitlab/gitlab.rb`:

```ruby
gitlab_rails['backup_upload_connection'] = {
  'provider' => 'AWS',
  'region' => 'eu-west-1',
  'aws_access_key_id' => 'AKIAKIAKI',
  'aws_secret_access_key' => 'secret123'
  # If using an IAM Profile, leave aws_access_key_id & aws_secret_access_key empty
  # ie. 'aws_access_key_id' => '',
  # 'use_iam_profile' => 'true'
}
gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket'
```

Make sure to run `sudo gitlab-ctl reconfigure` after editing `/etc/gitlab/gitlab.rb` to reflect the changes.

For installations from source:

{'fraction_non_alphanumeric': 0.1058422268676422, 'fraction_numerical': 0.01430300363076246, 'mean_word_length': 4.586969883220651, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '339520', 'n_tokens_mistral': 3074, 'n_tokens_neox': 2800, 'n_words': 1088}