<commit_before># Moving a format to the new indexing process

These are the steps to move formats to the new indexing process described in [ADR 004 - Transition mainstream formats to a Publishing API derived search index
](./arch/adr-004-transition-mainstream-to-publishing-api-index.md)

Note: if you're adding a brand new format, see [make a new document type available to search](https://docs.publishing.service.gov.uk/manual/make-a-new-document-type-available-to-search.html) instead.

## Ensure the document type is migrated to publishing API
The publishing app should already notify the publishing API when documents are published and unpublished.

Example PRs for [adding a new document type](https://docs.publishing.service.gov.uk/manual/add-a-new-document-type.html):

- [Add external_content schema and document type](https://github.com/alphagov/govuk-content-schemas/pull/690)
- [Publish recommended links to publishing API](https://github.com/alphagov/search-admin/pull/97)
- [Add rake task for publishing all external links to the publishing API](https://github.com/alphagov/search-admin/pull/100/files)

Ensure that all fields the publishing app currently sends to Search API are included in the payload send to publishing API. If anything is missing, you'll need to update the publishing app and content schemas, and then re-publish existing content to the publishing API.

You may also want to clean up other inconsistencies before changing the indexing method.

Example PRs:

- [Prepare for moving to rummager](https://github.com/alphagov/calendars/pull/160/files)
- [Ensure we pass the description text to publishing API](https://github.com/alphagov/calendars/pull/162/files)

## Update the presenter to handle the new format
You'll need to update the elasticsearch presenter in Search API so that it handles any fields which are not yet used by other formats in the govuk index.<commit_msg>Add note about formats list to indexing docs<commit_after># Moving a format to the new indexing process

These are the steps to move formats to the new indexing process described in [ADR 004 - Transition mainstream formats to a Publishing API derived search index
](./arch/adr-004-transition-mainstream-to-publishing-api-index.md)

Note: if you're adding a brand new format, see [make a new document type available to search](https://docs.publishing.service.gov.uk/manual/make-a-new-document-type-available-to-search.html) instead.

## Ensure the document type is migrated to publishing API
The publishing app should already notify the publishing API when documents are published and unpublished.

Example PRs for [adding a new document type](https://docs.publishing.service.gov.uk/manual/add-a-new-document-type.html):

- [Add external_content schema and document type](https://github.com/alphagov/govuk-content-schemas/pull/690)
- [Publish recommended links to publishing API](https://github.com/alphagov/search-admin/pull/97)
- [Add rake task for publishing all external links to the publishing API](https://github.com/alphagov/search-admin/pull/100/files)

Ensure that all fields the publishing app currently sends to Search API are included in the payload send to publishing API. If anything is missing, you'll need to update the publishing app and content schemas, and then re-publish existing content to the publishing API.

You may also want to clean up other inconsistencies before changing the indexing method.

Example PRs:

- [Prepare for moving to rummager](https://github.com/alphagov/calendars/pull/160/files)
- [Ensure we pass the description text to publishing API](https://github.com/alphagov/calendars/pull/162/files)

## Add the format to the list in `lib/learn_to_rank/format_enums.rb`

We take format into account in our machine learning, which means we
need a mapping from formats to unique numbers.

## Update the presenter to handle the new format
You'll need to update the elasticsearch presenter in Search API so that it handles any fields which are not yet used by other formats in the govuk index.
{'fraction_non_alphanumeric': 0.07707558859975217, 'fraction_numerical': 0.009913258983890954, 'mean_word_length': 4.708628005657709, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2855329', 'n_tokens_mistral': 1104, 'n_tokens_neox': 1051, 'n_words': 474}