<commit_before>---
title: Exploring Your Data
---

## Sample dataset

Now that we've gotten a glimpse of the basics, let's try to work on a more
realistic dataset. I've prepared a sample of fictitious JSON documents of
customer bank account information. Each document has the following schema:

```json
{
  "firstname": "Ashley",
  "lastname": "Clark",
  "age": 24,
  "gender": "female",
  "phone": "+1 (964) 525-3462",
  "company": "Pearlessa",
  "address": "477 Branton Street",
  "city": "Trexlertown",
  "state": "Indiana",
  "email": "<EMAIL>",
  "eyeColor": "green",
  "favoriteFruit": "strawberry",
  "account_number": 360070,
  "balance": "2550.21"
}
```

For the curious, this data was generated using [JSON Generator](www.json-generator.com),
so please ignore the actual values and semantics of the data as these are all
randomly generated.


## Loading the Sample Dataset

You can download the [sample dataset]({{ '/assets/accounts.json' | absolute_url }}){:target="_blank"}. Extract it to
our current directory and let's load it into our cluster as follows:

```sh
~ $ curl -H 'Content-Type: application/json' \
   -X POST --data-binary '@accounts.json' \
   'localhost:8880/bank/:restore?pretty'
```

And then you can use `:info` to get information about the new index:

```sh
~ $ curl 'localhost:8880/bank/:info?pretty'
{
    "#database_info": {
        "#uuid": "923a4470-7cdc-45ec-827c-fa85703fa8f6",
        "#doc_count": 1000,
        "#last_id": 1000,
        "#doc_del": 0,
        "#av_length": 22.432,
        "#doc_len_lower": 22,
        "#doc_len_upper": 25,
        "#has_positions": false
    }
}
```

Which means that we just successfully bulk indexed 1000 documents into the
bank index.


## The Search API

Now let's start with some simple searches. There are two basic ways to run
searches: one is by sending search parameters through the REST request URI and
the other by sending them through the REST request body. The request body
method allows you to be more expressive and also to define your searches in a
more readable JSON format. We'll try one example of the request URI method but
for the remainder of this guide, we will exclusively be using the request body
method.

The REST API for search is accessible from the `:search` endpoint. This example
returns all documents in the bank index:

{% capture json %}

```json
GET /bank/:search?q=*&sort=account_number&pretty
```
{% endcapture %}
{% include curl.html json=json %}

Let's first dissect the search call. We are searching (`:search` endpoint) in
the `bank` index, and the `q=*` parameter instructs Xapiand to match all
documents in the index. The `sort=account_number` parameter indicates to
sort the results using the `account_number` field of each document in an
ascending order. The `pretty` parameter just tells Xapiand to return
pretty-printed JSON results, the same effect can be achieved by using the
`Accept` header as in: `Accept: application/json; indent: 4`.

And the response (partially shown):

```json
{
  "#query": {
    "#total_count": 10,
    "#matches_estimated": 1000,
    "#hits": [
      {
          "city": "Fairview",
          "gender": "female",
          "balance": "1073.05",
          "firstname": "Hester",
          "lastname": "Blake",
          "company": "Affluex",
          "favoriteFruit": "strawberry",
          "eyeColor": "brown",
          "phone": "+1 (919) 400-3616",
          "state": "Virgin Islands",
          "account_number": 100123,
          "address": "756 Strauss Street",
          "age": 24,
          "email": "<EMAIL>",
          "_id": 233,
          "#docid": 233,
          "#rank": 0,
          "#weight": 0.0,
          "#percent": 100
      }, ...
    ]
  },
  "#took": 21.49
}
```

As for the response, we see the following parts:

* `#query.#total_count` - Total number of returned hits.
* `#query.#matches_estimated` - Number of estimated documents that match the query.
* `#query.#hits` - search results.
* `#took` - time in milliseconds for Xapiand to execute the search.

## Introducing the Query Language

Xapiand provides a JSON-style _domain-specific language_ that you can use to
execute queries. This is referred to as the Query DSL. The query language is
quite comprehensive and can be intimidating at first glance but the best way to
actually learn it is to start with a few basic examples.

{: .note}
The Query DSL method for searching is much more efficient.

Going back to our last example, we executed a query to retrieve all documents
using `q=*`. Here is the same exact search using the alternative request body
method:

{% capture json %}

```json
GET /bank/:search?pretty
{
  "_query": "*",
  "_sort": [
    { "account_number": "asc" }
  ]
}
```
{% endcapture %}
{% include curl.html json=json %}

The difference here is that instead of passing `q=*` in the URI, we POST a
JSON-style query request body to the `:search` API.

Dissecting the above, the query part tells us what our query definition is and
the `match_all` part is simply the type of query that we want to run. The
`match_all` query is simply a search for all documents in the specified index.

In addition to the query parameter, we also can pass other parameters to
influence the search results. In the example in the section above we passed in
sort, here we pass in `limit`:

{% capture json %}

```json
GET /bank/:search?pretty
{
  "_query": "*",
  "_limit": 1
}
```
{% endcapture %}
{% include curl.html json=json %}
<commit_msg>Update docs<commit_after>---
title: Exploring Your Data
---

## Sample dataset

Now that we've gotten a glimpse of the basics, let's try to work on a more
realistic dataset. I've prepared a sample of fictitious JSON documents of
customer bank account information. Each document has the following schema:

```json
{
  "firstname": "Ashley",
  "lastname": "Clark",
  "age": 24,
  "gender": "female",
  "phone": "+1 (964) 525-3462",
  "company": "Pearlessa",
  "address": "477 Branton Street",
  "city": "Trexlertown",
  "state": "Indiana",
  "email": "<EMAIL>",
  "eyeColor": "green",
  "favoriteFruit": "strawberry",
  "account_number": 360070,
  "balance": "2550.21"
}
```

For the curious, this data was generated using [JSON Generator](www.json-generator.com),
so please ignore the actual values and semantics of the data as these are all
randomly generated.


## Loading the Sample Dataset

You can download the [sample dataset]({{ '/assets/accounts.json' | absolute_url }}){:target="_blank"}. Extract it to
our current directory and let's load it into our cluster as follows:

```sh
~ $ curl -H 'Content-Type: application/json' \
   -X POST --data-binary '@accounts.json' \
   'localhost:8880/bank/:restore?pretty'
```

And then you can use `:info` to get information about the new index:

```sh
~ $ curl 'localhost:8880/bank/:info?pretty'
{
    "#database_info": {
        "#uuid": "923a4470-7cdc-45ec-827c-fa85703fa8f6",
        "#doc_count": 1000,
        "#last_id": 1000,
        "#doc_del": 0,
        "#av_length": 22.432,
        "#doc_len_lower": 22,
        "#doc_len_upper": 25,
        "#has_positions": false
    }
}
```

Which means that we just successfully bulk indexed 1000 documents into the
bank index.


## The Search API

Now let's start with some simple searches. There are two basic ways to run
searches: one is by sending search parameters through the REST request URI and
the other by sending them through the REST request body. The request body
method allows you to be more expressive and also to define your searches in a
more readable JSON format. We'll try one example of the request URI method but
for the remainder of this guide, we will exclusively be using the request body
method.

The REST API for search is accessible from the `:search` endpoint. This example
returns all documents in the bank index:

{% capture json %}

```json
GET /bank/:search?q=*&sort=account_number&pretty
```
{% endcapture %}
{% include curl.html json=json %}

Let's first dissect the search call. We are searching (`:search` endpoint) in
the `bank` index, and the `q=*` parameter instructs Xapiand to match all
documents in the index. The `sort=account_number` parameter indicates to
sort the results using the `account_number` field of each document in an
ascending order. The `pretty` parameter just tells Xapiand to return
pretty-printed JSON results, the same effect can be achieved by using the
`Accept` header as in: `Accept: application/json; indent: 4`.

And the response (partially shown):

```json
{
  "#query": {
    "#total_count": 10,
    "#matches_estimated": 1000,
    "#hits": [
      {
          "city": "Fairview",
          "gender": "female",
          "balance": "1073.05",
          "firstname": "Hester",
          "lastname": "Blake",
          "company": "Affluex",
          "favoriteFruit": "strawberry",
          "eyeColor": "brown",
          "phone": "+1 (919) 400-3616",
          "state": "Virgin Islands",
          "account_number": 100123,
          "address": "756 Strauss Street",
          "age": 24,
          "email": "<EMAIL>",
          "_id": 233,
          "#docid": 233,
          "#rank": 0,
          "#weight": 0.0,
          "#percent": 100
      }, ...
    ]
  },
  "#took": 21.49
}
```

As for the response, we see the following parts:

* `#query.#total_count` - Total number of returned hits.
* `#query.#matches_estimated` - Number of estimated documents that match the query.
* `#query.#hits` - search results.
* `#took` - time in milliseconds for Xapiand to execute the search.

## Introducing the Query Language

Xapiand provides a JSON-style _domain-specific language_ that you can use to
execute queries. This is referred to as the Query DSL. The query language is
quite comprehensive and can be intimidating at first glance but the best way to
actually learn it is to start with a few basic examples.

{: .note}
The Query DSL method for searching is much more efficient.

Going back to our last example, we executed a query to retrieve all documents
using `q=*`. Here is the same exact search using the alternative request body
method:

{% capture json %}

```json
GET /bank/:search?pretty
{
  "_query": "*",
  "_sort": "account_number"
}
```
{% endcapture %}
{% include curl.html json=json %}

The difference here is that instead of passing `q=*` in the URI, we POST a
JSON-style query request body to the `:search` API.

Dissecting the above, the query part tells us what our query definition is and
the `match_all` part is simply the type of query that we want to run. The
`match_all` query is simply a search for all documents in the specified index.

In addition to the query parameter, we also can pass other parameters to
influence the search results. In the example in the section above we passed in
sort, here we pass in `limit`:

{% capture json %}

```json
GET /bank/:search?pretty
{
  "_query": "*",
  "_limit": 1
}
```
{% endcapture %}
{% include curl.html json=json %}

{'fraction_non_alphanumeric': 0.11694248879948797, 'fraction_numerical': 0.023772515314985827, 'mean_word_length': 3.4736196319018404, 'pattern_counts': {'":': 103, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7623604', 'n_tokens_mistral': 3550, 'n_tokens_neox': 3228, 'n_words': 1380}