<commit_before>
# Datamodel

## Input

	/:dataset/:type(/:id)
	- source: id
	- target: id

- Dataset (`some-string`)
- Type (`PIT`)
- Identifiers (`abc123`)

### Types and structures

A **structure** in the graph is a node or edge.

They define the a central part of the graphmalizer core,
[see graph.yaml](../core/graph.yaml).

![node](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/node.svg)*node* 		![edge](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/edge.svg)*edge*

**Types** are the sorts of documents you feed the graphmalizer.

For example, here are some documents with types `A`, `B`, `C` and
identifiers `012`, `123`, `234`, `345`.

![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`A`,`012` ![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`A`,`123` ![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`B`,`234` ![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`C`,`234`

#### Configuration

To each type you assign a corresponding structure. Which types are
defined and to which structure they map is defined in [your
configuration](../config.json).

Again, an example:

> A ~ node, B ~ node, C ~ edge

![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`A` ~ ![node](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/node.svg)*node*

![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`B` ~ ![node](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/node.svg)*node*

![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`C` ~ ![node](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/edge.svg)*edge*

Given this association, updating a document of a certain type will in turn update the corresponding
structure in the graph.

For maximum confusion, when no configuration is specified,
the [default types](../core/config.js) are `node` and `edge`.
They *manipulate* nodes and edges in the graph and even
the names are the same, however
types are not the same thing as structures.

### Identifiers

All identifiers must be unique, across all datasets.

Identifiers which are lexicographically equal ("equal as a character string"),
are considered to point to the same structure, otherwise not.

The node structure requires `id`. The edge structure requires `source`,
`target` and will derive `id` from them, if `id` is not specified.

[see typeSpecific.js](../core/typeSpecific.js)

# Graph<commit_msg>stÿlüng<commit_after>
# Datamodel

## Input

	/:dataset/:type(/:id)
	- source: id
	- target: id

- Dataset (`some-string`)
- Type (`PIT`)
- Identifiers (`abc123`)

### Types and structures

A **structure** in the graph is a node or edge.

They define the a central part of the graphmalizer core,
[see graph.yaml](../core/graph.yaml).

![node](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/node.svg)*node* 		![edge](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/edge.svg)*edge*

**Types** are the sorts of documents you feed the graphmalizer.

Here are some documents with types `A`, `B`, `C` and
identifiers *012*, *123*, *234*, *345*.

![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`A`*012* ![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`A`*123* ![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`B`*234* ![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`C`*234*

#### Configuration

To each type you assign a corresponding structure. Which types are
defined and to which structure they map is defined in [your
configuration](../config.json).

For example:

![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`A` ~ ![node](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/node.svg)*node*

![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`B` ~ ![node](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/node.svg)*node*

![document](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/document.svg)`C` ~ ![node](https://cdn.rawgit.com/graphmalizer/prototype/bb05daee8af3dfb3b22552bb3958b5cdab4c9ff0/docs/edge.svg)*edge*

Given this association, updating a document of a certain type will in turn update the corresponding
structure in the graph.

For maximum confusion, when no configuration is specified,
the [default types](../core/config.js) are `node` and `edge`.
They *manipulate* nodes and edges in the graph and even
the names are the same, however
types are not the same thing as structures.

### Identifiers

All identifiers must be unique, across all datasets.

Identifiers which are lexicographically equal ("equal as a character string"),
are considered to point to the same structure, otherwise not.

The node structure requires `id`. The edge structure requires `source`,
`target` and will derive `id` from them, if `id` is not specified.

[see typeSpecific.js](../core/typeSpecific.js)

# Graph
{'fraction_non_alphanumeric': 0.13016354746248524, 'fraction_numerical': 0.0819423368740516, 'mean_word_length': 5.938011695906432, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 24, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6221357', 'n_tokens_mistral': 2386, 'n_tokens_neox': 2148, 'n_words': 438}