<commit_before>Solr
====

Media Cloud uses Solr as its text indexing and search engine. For information on running the Solr engine, see the [hosting docs](hosting/solr-hosting.markdown).

We use sharding within Solr to host the data across multiple server processes on each server and across
multiple servers.  Solr requires multiple shards in the same machine to take advantage of parallel processing
capabilities (including both multiple CPU cores and multiple disks) on the same machine.  The host docs referenced
above describe how we administer our Solr cluster, including scripts we use to create, start, stop, and reload
Solr shards.

The basic interaction between Solr and the rest of the platform is that we import any updated `story_sentences` into
Solr from the PostgreSQL server every hour by running an hourly Cron script on the Solr server.  The import script
knows which sentences to import by keep track of `db_row_last_updated` fields on the `stories`, `media`, and `story_sentences`
table.  The import script queries `story_sentences` for all distinct stories for which the `db_row_last_updated` value
is greater than the latest value in `solr_imports`.  Triggers in the PostgreSQL database update the
`story_sentences.db_row_last_updated` value on `story_sentences` whenever a related story, medium, story tag,
medium tag, or story sentence tag is updated.

The list of fields imported by Solr is configured in
[solr/collections/_base_collection/conf/schema.xml](../solr/collections/_base_collection/conf/schema.xml).  As of this doc, we index the
following fields:

* `story_sentences_id` (stored; must look up the specific returned sentence)
* `media_id`
* `stories_id` (stored; must return grouped stories_id to efficiently search for lists of stories)
* `sentence_number`
* `processed_stories_id` (stored; must query PostgreSQL for returned `stories_id`s ordered by specific `processed_stories_id`)
* `sentence` (stored; must get sentence back to efficiently count words without querying `story_sentences`)<commit_msg>Clarify docs<commit_after>Solr
====

Media Cloud uses Solr as its text indexing and search engine. For information on running the Solr engine, see the [hosting docs](hosting/solr-hosting.markdown).

We use sharding within Solr to host the data across multiple server processes on each server and across
multiple servers.  Solr requires multiple shards in the same machine to take advantage of parallel processing
capabilities (including both multiple CPU cores and multiple disks) on the same machine.  The host docs referenced
above describe how we administer our Solr cluster, including scripts we use to create, start, stop, and reload
Solr shards.

The basic interaction between Solr and the rest of the platform is that we import any updated `story_sentences` into
Solr from the PostgreSQL server every hour by running an hourly Cron script on the Solr server. IDs of the stories to be (re)imported into Solr are to be found in the `solr_import_stories` table.

The list of fields imported by Solr is configured in
[solr/collections/_base_collection/conf/schema.xml](../solr/collections/_base_collection/conf/schema.xml).  As of this doc, we index the
following fields:

* `story_sentences_id` (stored; must look up the specific returned sentence)
* `media_id`
* `stories_id` (stored; must return grouped stories_id to efficiently search for lists of stories)
* `sentence_number`
* `processed_stories_id` (stored; must query PostgreSQL for returned `stories_id`s ordered by specific `processed_stories_id`)
* `sentence` (stored; must get sentence back to efficiently count words without querying `story_sentences`)
{'fraction_non_alphanumeric': 0.05546448087431694, 'fraction_numerical': 0.0, 'mean_word_length': 5.445422535211268, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '869598', 'n_tokens_mistral': 997, 'n_tokens_neox': 955, 'n_words': 484}