<commit_before>Sample Discovery Jobs
=====================
> This page was autogenerated by [JobSamples.java](../src/main/java/voyager/quickstart/discovery/JobSamples.java) on Tue Mar 18 22:19:26 PDT 2014


## Add a file
Add a file to the system
```json
{
  "action": "ADD",
  "path": "/Users/ryan/workspace/quickstart-java/docs/imgs/links.png",
  "entry": {"fields": {
    "format": "image/png",
    "bytes": 70154
  }}
}
```

## Add a URL
Add a URL to the system
```json
{
  "action": "ADD",
  "path": "http://public.voyagergis.com/kml/TheBestPlaces.kml",
  "entry": {"fields": {
    "name": "TheBestPlaces",
    "format": "application/vnd.google-earth.kml+xml"
  }}
}
```

## Add record without a resource reference
Fill in all the explicit properties
```json
{
  "id": "record1234",
  "action": "ADD",
  "entry": {
    "geo": {
      "lat": 37.78875904932722,
      "lon": -122.38787244901528
    },
    "meta": {"body": "<xml>raw XML metadata<\/xml>"},
    "fields": {
      "name": "Name (1395206366178)",
      "abstract": "some longer text about what we have",
      "copyright": "some copyright message"
    }
  }
}
```

## Entry with structure
children with <code>index: false</code> will appear in the tree, but not have their own record

```json
{
  "id": "tree_1234",
  "action": "ADD",
  "entry": {
    "fields": {
      "name": "Tree Root",
      "abstract": "some longer text about what we have"
    },
    "children": [{
      "index": false,
      "fields": {"name": "Sub 1"},
      "children": [
        {"fields": {
          "name": "Child 0",
          "abstract": "more info in an entry (0)"
        }},
        {"fields": {
          "name": "Child 1",
          "abstract": "more info in an entry (1)"
        }},
        {"fields": {
          "name": "Child 2",
          "abstract": "more info in an entry (2)"
        }}
      ]
    }]
  }
}
```
The tree structure is displayed in the ui as:

![structure](imgs/structure.png)


## Entry with links
Items with links to other data.

```json
{
  "id": "with_links_1234",
  "action": "ADD",
  "entry": {
    "geo": {
      "xmin": -88.59375,
      "ymin": 24.766785,
      "xmax": -78.222656,
      "ymax": 31.128199
    },
    "fields": {
      "name": "Item With Links",
      "abstract": "some longer text about what we have"
    },
    "links": [
      {
        "relation": "data",
        "name": "linked data",
        "path": "/Users/ryan/workspace/quickstart-java/docs/imgs/links.png",
        "format": "image/png"
      },
      {
        "relation": "data",
        "name": "tree data",
        "path": "/Users/ryan/workspace/quickstart-java/docs/imgs/structure.png"
      }
    ]
  }
}
```
Links are displayed in the ui as:

![structure](imgs/links.png)


## Entry with structure and child documents get indexed
When you add the '__to_extract', field, the child items are added to the extraction queue.

```json
{
  "id": "tree_and_extract",
  "action": "ADD",
  "entry": {
    "fields": {
      "name": "Tree Root",
      "abstract": "some longer text about what we have"
    },
    "children": [{
      "index": false,
      "fields": {"name": "Sub 1"},
      "children": [
        {"fields": {
          "name": "The Links PNG",
          "format": "image/png",
          "title": "Some Title From the Job",
          "path": "/Users/ryan/workspace/quickstart-java/docs/imgs/links.png",
          "__to_extract": true
        }},
        {"fields": {
          "name": "A Structure PNG",
          "abstract": "an abstract field",
          "path": "/Users/ryan/workspace/quickstart-java/docs/imgs/structure.png",
          "__to_extract": true<commit_msg>Update discoveryjob-examples.md<commit_after>Sample Discovery Jobs
=====================
> This page was autogenerated by [JobSamples.java](../src/main/java/voyager/quickstart/discovery/JobSamples.java) on Thu Mar 05 08:50:44 PST 2015


## Add a file
Add a file to the system
```json
{
  "action": "ADD",
  "path": "/Users/ryan/workspace/quickstart-java/docs/imgs/links.png",
  "entry": {"fields": {
    "format": "image/png",
    "bytes": 70154
  }}
}
```

## Add a URL
Add a URL to the system
```json
{
  "action": "ADD",
  "path": "http://public.voyagergis.com/kml/TheBestPlaces.kml",
  "entry": {"fields": {
    "name": "TheBestPlaces",
    "format": "application/vnd.google-earth.kml+xml"
  }}
}
```

## Add record without a resource reference
Fill in all the explicit properties
```json
{
  "id": "record1234",
  "action": "ADD",
  "entry": {
    "geo": {
      "lat": 37.78875904932722,
      "lon": -122.38787244901528
    },
    "meta": {"body": "<xml>raw XML metadata<\/xml>"},
    "fields": {
      "name": "Name (1425574244431)",
      "abstract": "some longer text about what we have",
      "copyright": "some copyright message"
    }
  }
}
```

## Entry with structure
children with <code>index: false</code> will appear in the tree, but not have their own record

```json
{
  "id": "tree_1234",
  "action": "ADD",
  "entry": {
    "fields": {
      "name": "Tree Root",
      "abstract": "some longer text about what we have"
    },
    "children": [{
      "index": false,
      "fields": {"name": "Sub 1"},
      "children": [
        {"fields": {
          "name": "Child 0",
          "abstract": "more info in an entry (0)"
        }},
        {"fields": {
          "name": "Child 1",
          "abstract": "more info in an entry (1)"
        }},
        {"fields": {
          "name": "Child 2",
          "abstract": "more info in an entry (2)"
        }}
      ]
    }]
  }
}
```
The tree structure is displayed in the ui as:

![structure](imgs/structure.png)


## Entry with links
Items with links to other data.

```json
{
  "id": "with_links_1234",
  "action": "ADD",
  "entry": {
    "geo": {
      "xmin": -88.59375,
      "ymin": 24.766785,
      "xmax": -78.222656,
      "ymax": 31.128199
    },
    "fields": {
      "name": "Item With Links",
      "abstract": "some longer text about what we have"
    },
    "links": [
      {
        "relation": "data",
        "name": "linked data",
        "path": "/Users/ryan/workspace/quickstart-java/docs/imgs/links.png",
        "format": "image/png"
      },
      {
        "relation": "data",
        "name": "tree data",
        "path": "/Users/ryan/workspace/quickstart-java/docs/imgs/structure.png"
      }
    ]
  }
}
```
Links are displayed in the ui as:

![structure](imgs/links.png)


## Entry with a preview/thumbnail image

```json
{
  "id": "with_image_1234",
  "action": "ADD",
  "entry": {"fields": {
    "name": "An Item with a thumbnail",
    "image_url": "https://raw.githubusercontent.com/voyagersearch/quickstart-java/master/docs/imgs/eclipse_2_set_variable.png"
  }}
}
```
NOTE: setting the field 'path_to_thumb' expects the thumbnail to be in the ${meta.dir} folder

## Entry with structure and child documents get indexed
When you add the '__to_extract', field, the child items are added to the extraction queue.

```json
{
  "id": "tree_and_extract",
  "action": "ADD",
  "entry": {
    "fields": {
      "name": "Tree Root",
      "abstract": "some longer text about what we have"
    },
    "children": [{
      "index": false,
      "fields": {"name": "Sub 1"},
      "children": [
        {"fields": {
          "name": "The Links PNG",
          "format": "image/png",
          "title": "Some Title From the Job",
          "path": "/Users/ryan/workspace/quickstart-java/docs/imgs/links.png",
          "__to_extract": true
        }},
        {"fields": {
          "name": "A Structure PNG",
          "abstract": "an abstract field",
          "path": "/Users/ryan/workspace/quickstart-java/docs/imgs/structure.png",
          "__to_extract": true
{'fraction_non_alphanumeric': 0.1845315338697119, 'fraction_numerical': 0.030236179600311445, 'mean_word_length': 2.4936536718041706, 'pattern_counts': {'":': 176, '<': 11, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1437695', 'n_tokens_mistral': 2794, 'n_tokens_neox': 2557, 'n_words': 690}