<commit_before>% Media Cloud API Version 2
%

#API URLs

## Authentication

This document describes API calls for administrative users. These calls are intended for users running their own install of Media Cloud.
Users of the mediacloud.org API should refer instead to the Media Cloud API 2.0 Spec.

Please refer to the Media Cloud Api spec for general information on how requests should be constructed. 
Because the functionality of the admin api is largely a superset of the regular API, we do not include duplicative information in that document.

## Stories

A story represents a single published piece of content.  Each unique URL downloaded from any syndicated feed within 
a single media source is represented by a single story.  For example, a single New York Times newspaper story is a 
Media Cloud story, as is a single Instapundit blog post.  Only one story may exist for a given title for each 24 hours 
within a single media source.

The `story_text` of a story is either the content of the description field in the syndicated field or the extracted 
text of the content downloaded from the story's URL at the `collect_date`, depending on whether our full text RSS 
detection system has determined that the full text of each story can be found in the RSS of a given media source.

### Output description

The following table describes the meaning and origin of fields returned by both api/v2/stories/single and api/v2/stories/list in which we felt clarification was necessary.

| Field               | Description
| ------------------- | ----------------------------------------------------------------------
| `title`             | The story title as defined in the RSS feed. May contain HTML (depending on the source).
| `description`       | The story description as defined in the RSS feed. May contain HTML (depending on the source).
| `full_text_rss`     | If 1, the text of the story was obtained through the RSS feed.<br />If 0, the text of the story was obtained by extracting the article text from the HTML.
| `story_text`        | The text of the story.<br />If `full_text_rss` is non-zero, this is formed by stripping HTML from the title and description and concatenating them.<br />If `full_text_rss` is zero, this is formed by extracting the article text from the HTML.
| `story_sentences`   | A list of sentences in the story.<br />Generated from `story_text` by splitting it into sentences and removing any duplicate sentences occurring within the same source for the same week.
| `raw_1st_download`  | The contents of the first HTML page of the story.<br />Available regardless of the value of `full_text_rss`.<br />*Note:* only provided if the `raw_1st_download` parameter is non-zero.
| `publish_date`      | The publish date of the story as specified in the RSS feed.
| `tags` | A list of any tags associated with this story, including those written through the write-back api.
| `collect_date`      | The date the RSS feed was actually downloaded.
| `guid`              | The GUID field in the RSS feed. Defaults to the URL if no GUID is specified in the RSS feed.
| `corenlp`           | The raw json result from running the story text through the CoreNLP pipeline.



### api/v2/stories/list
  
| URL                             | Function
| ------------------------------- | ---------------------------------
| `api/v2/stories/list` | Return multiple processed stories

#### Query Parameters 

| Parameter                    | Default | Notes
| ---------------------------- | ------- | ------------------------------------------------------------------------------
| `last_processed_stories_id`  | 0       | Return stories in which the `processed_stories_id` is greater than this value.
| `rows`                       | 20      | Number of stories to return.
| `raw_1st_download`           | 0       | If non-zero, include the full HTML of the first page of the story.
| `corenlp`                    | 0       | If non-zero, include the corenlp json document with each story and each sentence
| `q`                          | null    | If specified, return only results that match the given Solr query.  Only one `q` parameter may be included.
| `fq`                         | null    | If specified, file results by the given Solr query.  More than one `fq` parameter may be included.<commit_msg>Get rid of information duplicated between the admin api and general api spec.<commit_after>% Media Cloud API Version 2
%

#API URLs

## Authentication

This document describes API calls for administrative users. These calls are intended for users running their own install of Media Cloud.
Users of the mediacloud.org API should refer instead to the Media Cloud API 2.0 Spec.

Please refer to the Media Cloud Api spec for general information on how requests should be constructed. 
Because the functionality of the admin api is largely a superset of the regular API, we do not include duplicative information in that document.

## Stories

A story represents a single published piece of content.  Each unique URL downloaded from any syndicated feed within 
a single media source is represented by a single story.  For example, a single New York Times newspaper story is a 
Media Cloud story, as is a single Instapundit blog post.  Only one story may exist for a given title for each 24 hours 
within a single media source.

The `story_text` of a story is either the content of the description field in the syndicated field or the extracted 
text of the content downloaded from the story's URL at the `collect_date`, depending on whether our full text RSS 
detection system has determined that the full text of each story can be found in the RSS of a given media source.

### Output description

The following table describes the meaning and origin of fields returned by the admin API for both api/v2/stories/single and api/v2/stories/list. (The admin API also returns all the fields available through the general API. Refer to the Media Cloud API 2.0 Spec for a list and description of these fields.)

| Field               | Description
| ------------------- | ----------------------------------------------------------------------
| `title`             | The story title as defined in the RSS feed. May contain HTML (depending on the source).
| `description`       | The story description as defined in the RSS feed. May contain HTML (depending on the source).
| `full_text_rss`     | If 1, the text of the story was obtained through the RSS feed.<br />If 0, the text of the story was obtained by extracting the article text from the HTML.
| `story_text`        | The text of the story.<br />If `full_text_rss` is non-zero, this is formed by stripping HTML from the title and description and concatenating them.<br />If `full_text_rss` is zero, this is formed by extracting the article text from the HTML.
| `story_sentences`   | A list of sentences in the story.<br />Generated from `story_text` by splitting it into sentences and removing any duplicate sentences occurring within the same source for the same week.
| `raw_1st_download`  | The contents of the first HTML page of the story.<br />Available regardless of the value of `full_text_rss`.<br />*Note:* only provided if the `raw_1st_download` parameter is non-zero.

### api/v2/stories/list
  
| URL                             | Function
| ------------------------------- | ---------------------------------
| `api/v2/stories/list` | Return multiple processed stories

#### Query Parameters 

| Parameter                    | Default | Notes
| ---------------------------- | ------- | ------------------------------------------------------------------------------
| `last_processed_stories_id`  | 0       | Return stories in which the `processed_stories_id` is greater than this value.
| `rows`                       | 20      | Number of stories to return.
| `raw_1st_download`           | 0       | If non-zero, include the full HTML of the first page of the story.
| `corenlp`                    | 0       | If non-zero, include the corenlp json document with each story and each sentence
| `q`                          | null    | If specified, return only results that match the given Solr query.  Only one `q` parameter may be included.
| `fq`                         | null    | If specified, file results by the given Solr query.  More than one `fq` parameter may be included.
{'fraction_non_alphanumeric': 0.1152744630071599, 'fraction_numerical': 0.00477326968973747, 'mean_word_length': 2.433428922572716, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3504171', 'n_tokens_mistral': 2016, 'n_tokens_neox': 1938, 'n_words': 1121}