<commit_before># Developer API v2.0

Aleph supports an extensive API for searching documents and entities. It
can also be used to retrieve raw metadata, source documents and other useful
details.

### General Overview

Aleph's API tries to follow a pragmatic approach based on the following
principles:

*   All API calls are prefixed with an API version; this version is `/api/2/`.
*   Responses and requests are both encoded as JSON. Requests should have the `Content-Type` and `Accept` headers set to `application/json`.
*   The application uses Representational State Transfer (REST) principles where convenient, but also has some procedural API calls.

#### Domain Model and REST Resources

Aleph exposes all significant aspects of it's domain model, which is explained in short below.

*   **Documents** (`/api/2/documents`) reflect a single text document or tabular document imported into Aleph. Every document is either imported as a text document with a number of text pages associated with it, or as a tabular document, which as a number of rows.
*   **Collections** (`/api/2/collections`) group documents and entities into sets, such as all the documents from a particular source, all the people relevant to a particular investigation, or all the companies on a country's sanctions list. They also control who can access information about an entity or document.
*   **Entities** (`/api/2/entities`) reflect structured data about persons or companies. Each entity has a name, a number of aliases and other details, such as their jurisdiction, a short summary, or contact and passport details. Entities are identified through a UUID.
*   **Roles** represent users and user groups, and they are connected via **Permissions** to sources and collections.

### Searching

The main search endpoint allows for a set of complex queries to be executed. Results will include documents _and_ entities, and their metadata, record snippets (for highlighting) and facet values that can be used to further summarize the result set.

<pre>GET /api/2/search</pre>

This accepts the following arguments:

*   `q`, a query string in ElasticSearch query syntax. Can include field searches, such as `title:penguin`.
*   `facet`, return facet values for the given metadata field, such as `languages`, `countries`, `mime_type` or `extension`. This can be specified multiple times for more than one facet to be added.
*   `filter:{field_name}`, filter the results by the given field. This is useful when used in conjunction with `facet` to create a drill-down mechanism. Useful fields are:
    *   `entities.id`, documents tagged with a particular entity.
    *   `collection_id`, documents belonging to a particular collection .
    *   `title`, of the document.
    *   `file_name`, of the source file.
    *   `source_url`, URL of the source file.
    *   `extension`, file extension of the source file.
    *   `languages`, in the document.
    *   `countries`, associated with the document.
    *   `keywords`, from the document.
    *   `emails`, email addresses mentioned in the document.
    *   `domains`, websites mentioned in the document.
    *   `phone_numbers,` mentioned in the document.
    *   `dates`, in any of the following formats: `yyyy-MM-dd`, `yyyy-MM`, `yyyy-MM-d`, `yyyy-M`, `yyyy`
    *   `mime_type`, of the source file.
    *   `author`, according to the source file's metadata.
    *   `summary`, of the document.
    *   `text`, entire text extracted from the document.
    *   `created_at`, when the document was added to aleph (`yyyy-mm -ddThh:ii:ss.uuuuuu`).
    *   `updated_at`, when the document was modified in aleph (`yyyy -mm-ddThh:ii:ss.uuuuuu`).
*   `limit`, the number of results to return, max. 10,000.
*   `offset`, the number of results to skip at the beginning of the result set.

By default, all queries will return a facet of the collections for which matching documents/entities have been found. A filter can be applied to show only results from a particular collection: `?filter:collection_id={collection_id}`.

If you know you only want to search documents (unstructured, ingested data) or entities (structured data which may have been extracted from a dataset, or entered by a human) you can use these arguments with the [/documents](#fetching-documents-and-metadata) or [/entities](#fetching-entities-and-metadata) endpoints.

#### Search Results as Excel

A second endpoint exists that accepts the same query parameters and will return an Excel 2007 (XML) file as it's result. The returned file contains a shortened representation of the results, but it is limited to 10,000 rows.

<pre>GET /api/2/search/export?q=pickles</pre>

### Reading and writing Collections

Collections can contain documents and/or entities. To get a (paged) list of all collections you have access to:

<pre>GET /api/2/collections</pre>

For the entities and documents in a particular collection:

<pre>GET /api/2/collections/{collection_id}</pre>

#### Updating collections

To update or delete a collection respectively, send a `POST` request with the new metadata to be added or a `DELETE` request to:

<pre>/api/2/collections/{collection_id}</pre>

To add new entities to a collection, you need to use the entities API. Deleting a collection also deletes all of the entities in the collection.

To create a new collection, `POST` the JSON metadata to:

<pre>/api/2/collections</pre>

#### Uploading documents to collections

To upload new documents to a collection, send a `POST` request with the files and metadata to:

<pre>/api/2/collections/{collection_id}/ingest</pre>

Metadata of a file can contain the following:

* `parent`: A dictionary containing `foreign_id` or `id` of parent Document.
* `foreign_id`: Foreign id of the document
* `title`: Title of the document
* `summary`: Summary of the document
* `author`: Author of the document
* `crawler`: Name of the crawler used to fetch the document
* `source_url`: URL of source file
* `file_name`: Name of the document
* `mime_type`: Mimetype of the document
* `headers`: Response headers while fetching the document
* `authored_at`: When the document was authored
* `modified_at`: Modification time of the document
* `published_at`: Publication time of the document
* `retrieved_at`: Retrieval time of the document
* `languages`: Languages in the document<commit_msg>Update docs<commit_after># Developer API v2.0

Aleph supports an extensive API for searching documents and entities. It
can also be used to retrieve raw metadata, source documents and other useful
details.

### General Overview

Aleph's API tries to follow a pragmatic approach based on the following
principles:

*   All API calls are prefixed with an API version; this version is `/api/2/`.
*   Responses and requests are both encoded as JSON. Requests should have the `Content-Type` and `Accept` headers set to `application/json`.
*   The application uses Representational State Transfer (REST) principles where convenient, but also has some procedural API calls.

#### Domain Model and REST Resources

Aleph exposes all significant aspects of it's domain model, which is explained in short below.

*   **Documents** (`/api/2/documents`) reflect a single text document or tabular document imported into Aleph. Every document is either imported as a text document with a number of text pages associated with it, or as a tabular document, which as a number of rows.
*   **Collections** (`/api/2/collections`) group documents and entities into sets, such as all the documents from a particular source, all the people relevant to a particular investigation, or all the companies on a country's sanctions list. They also control who can access information about an entity or document.
*   **Entities** (`/api/2/entities`) reflect structured data about persons or companies. Each entity has a name, a number of aliases and other details, such as their jurisdiction, a short summary, or contact and passport details. Entities are identified through a UUID.
*   **Roles** represent users and user groups, and they are connected via **Permissions** to sources and collections.

### Searching

The main search endpoint allows for a set of complex queries to be executed. Results will include documents _and_ entities, and their metadata, record snippets (for highlighting) and facet values that can be used to further summarize the result set.

<pre>GET /api/2/search</pre>

This accepts the following arguments:

*   `q`, a query string in ElasticSearch query syntax. Can include field searches, such as `title:penguin`.
*   `facet`, return facet values for the given metadata field, such as `languages`, `countries`, `mime_type` or `extension`. This can be specified multiple times for more than one facet to be added.
*   `filter:{field_name}`, filter the results by the given field. This is useful when used in conjunction with `facet` to create a drill-down mechanism. Useful fields are:
    *   `entities.id`, documents tagged with a particular entity.
    *   `collection_id`, documents belonging to a particular collection .
    *   `title`, of the document.
    *   `file_name`, of the source file.
    *   `source_url`, URL of the source file.
    *   `extension`, file extension of the source file.
    *   `languages`, in the document.
    *   `countries`, associated with the document.
    *   `keywords`, from the document.
    *   `emails`, email addresses mentioned in the document.
    *   `domains`, websites mentioned in the document.
    *   `phone_numbers,` mentioned in the document.
    *   `dates`, in any of the following formats: `yyyy-MM-dd`, `yyyy-MM`, `yyyy-MM-d`, `yyyy-M`, `yyyy`
    *   `mime_type`, of the source file.
    *   `author`, according to the source file's metadata.
    *   `summary`, of the document.
    *   `text`, entire text extracted from the document.
    *   `created_at`, when the document was added to aleph (`yyyy-mm -ddThh:ii:ss.uuuuuu`).
    *   `updated_at`, when the document was modified in aleph (`yyyy -mm-ddThh:ii:ss.uuuuuu`).
*   `limit`, the number of results to return, max. 10,000.
*   `offset`, the number of results to skip at the beginning of the result set.

By default, all queries will return a facet of the collections for which matching documents/entities have been found. A filter can be applied to show only results from a particular collection: `?filter:collection_id={collection_id}`.

If you know you only want to search documents (unstructured, ingested data) or entities (structured data which may have been extracted from a dataset, or entered by a human) you can use these arguments with the [/documents](#fetching-documents-and-metadata) or [/entities](#fetching-entities-and-metadata) endpoints.

#### Search Results as Excel

A second endpoint exists that accepts the same query parameters and will return an Excel 2007 (XML) file as it's result. The returned file contains a shortened representation of the results, but it is limited to 10,000 rows.

<pre>GET /api/2/search/export?q=pickles</pre>

### Reading and writing Collections

Collections can contain documents and/or entities. To get a (paged) list of all collections you have access to:

<pre>GET /api/2/collections</pre>

For the entities and documents in a particular collection:

<pre>GET /api/2/collections/{collection_id}</pre>

#### Updating collections

To update or delete a collection respectively, send a `POST` request with the new metadata to be added or a `DELETE` request to:

<pre>/api/2/collections/{collection_id}</pre>

To add new entities to a collection, you need to use the entities API. Deleting a collection also deletes all of the entities in the collection.

To create a new collection, `POST` the JSON metadata to:

<pre>/api/2/collections</pre>

#### Uploading documents to collections

To upload new documents to a collection, send a `POST` request with the files and metadata to:

<pre>/api/2/collections/{collection_id}/ingest</pre>

Metadata of a file is sent via a JSON-encoded POST variable, `meta`, which can contain nested values and lists for the following metadata fields:

* `parent`: A dictionary containing `foreign_id` or `id` of parent Document.
* `foreign_id`: Foreign id of the document
* `title`: Title of the document
* `summary`: Summary of the document
* `author`: Author of the document
* `crawler`: Name of the crawler used to fetch the document
* `source_url`: URL of source file
* `file_name`: Name of the document
* `mime_type`: MIME type of the document
* `headers`: Response headers while fetching the document
* `authored_at`: When the document was authored
* `modified_at`: Modification time of the document
* `published_at`: Publication time of the document
* `retrieved_at`: Retrieval time of the document
* `languages`: Languages in the document
{'fraction_non_alphanumeric': 0.08214146150840172, 'fraction_numerical': 0.004220398593200469, 'mean_word_length': 4.270181219110379, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2517040', 'n_tokens_mistral': 3477, 'n_tokens_neox': 3300, 'n_words': 1727}