<commit_before># Redirector fetch process

## How to set up a new department

To be completed.

## How to run the fetch scripts

### Terms

* *Old url:* The url as it existed on the original site. eg `http://decc.gov.uk/path-to-something`
* *New url:* The new GOV.UK url we want to redirect to. eg `http://gov.uk/path-to-the-new-version`
* *Admin url:* The admin url equivalent for the new url. eg `http://whitehall-admin.production.alphagov.co.uk/government/admin/detailed-guides/12345`

### The process

#### Download a document mappings file

This file is generated each night at 2am from the latest production data, and serves two purposes:

* The providing of mappings from "old urls" to "new urls"
* The conversion of "admin urls" to publicly-consumable "new urls"

Download [this csv file (~15MB)](http://whitehall-admin.production.alphagov.co.uk/government/all_document_attachment_and_non_document_mappings.csv) and save as `document_mappings.csv` in the root of the redirect project.

**NOTE:** we've recently changed the mappings file we use, to ensure that we also can map things that aren't documents from admin urls to new urls.

#### Generate the redirects

Ensure that the source google doc is set up and the most recent version is published to CSV (ask Robin or Pete to do this.)

From the root of the project, run `./tools/generate-redirects.sh <department>`

eg `./tools/generate-redirects.sh decc`

This will regenerate files we keep checked in in `data/mappings/`. Any warnings or errors in the url formats are sent to $stderr in the following form:

    error url,error name,source file url,source file row number

#### View changes and check in

Use `git diff` to view the changes made to the file, and check through the changes with Robin or Pete before committing them.

It's useful to search through for any `301s` that are being removed as a sanity check, try `git diff` then type `/^-.*301$`.

#### Check the redirect build

If there are warnings about an unstable `redirector` build, there will be errors in the redirect files.

Look [here](http://ci.alphagov.co.uk/view/Transition/job/redirector/lastSuccessfulBuild/artifact/dist/) for any files named `<department>_incorrect.txt` (eg `og_decc_incorrect.txt`) - this will give you the list of links to fix in the source google doc.

#### Deploy

The redirector is deployed to production in a similar way to other projects.

You can still deploy the redirector on an unstable build as the offending links will not be generated by the redirector. Don't deploy if the main `redirector` or `redirector-deploy` fails, or if either of the `redirector-integration-` builds fail.<commit_msg>Small update to tools/README<commit_after># Redirector fetch process

## How to set up a new department

To be completed.

## How to run the fetch scripts

### Terms

* *Old url:* The url as it existed on the original site. eg `http://decc.gov.uk/path-to-something`
* *New url:* The new GOV.UK url we want to redirect to. eg `http://gov.uk/path-to-the-new-version`
* *Admin url:* The admin url equivalent for the new url. eg `http://whitehall-admin.production.alphagov.co.uk/government/admin/detailed-guides/12345`

### The process

#### Download a document mappings file

This file is generated each night at 2am from the latest production data, and serves two purposes:

* The providing of mappings from "old urls" to "new urls"
* The conversion of "admin urls" to publicly-consumable "new urls"

Download [this csv file (~15MB)](http://whitehall-admin.production.alphagov.co.uk/government/all_document_attachment_and_non_document_mappings.csv) and save as `document_mappings.csv` in the root of the redirect project.

**NOTE:** we've recently changed the mappings file we use, to ensure that we also can map things that aren't documents from admin urls to new urls.

#### Generate the redirects

Ensure that the source google doc is set up and the most recent version is published to CSV (ask Robin or Pete to do this.)

From the root of the project, run `./tools/generate-redirects.sh <department>`

eg `./tools/generate-redirects.sh decc`

This will regenerate files we keep checked in in `data/mappings/`. Any warnings or errors in the url formats are sent to $stderr in the following form:

    error url,error name,source file url,source file row number

#### View changes and check in

Use `git diff` to view the changes made to the file, and check through the changes with Robin or Pete before committing them.

It's useful to search through for any `301s` that are being removed as a sanity check, try `git diff` then type `/^-.*301$`.

#### Check the redirect build

If there are warnings about an unstable `redirector` build, there will be errors in the redirect files.

Look [here](http://ci.alphagov.co.uk/view/Transition/job/redirector/lastSuccessfulBuild/artifact/dist/) for any files named `<department>_incorrect.txt` (eg `og_decc_incorrect.txt`) - this will give you the list of links to fix in the source google doc.

#### Deploy

The redirector is deployed to production in a similar way to other projects.

You can still deploy the redirector on an unstable build as the offending links will not be generated by the redirector. Don't deploy if the main `redirector` or `redirector-deploy` fails, or if the `redirector-integration-subset` build fails.
{'fraction_non_alphanumeric': 0.08566073102155576, 'fraction_numerical': 0.005248359887535145, 'mean_word_length': 4.341341341341342, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3255597', 'n_tokens_mistral': 1549, 'n_tokens_neox': 1478, 'n_words': 730}