<commit_before>
# DDlog

DDlog is a higher-level language for writing DeepDive applications in succinct Datalog-like syntax.
We are gradually extending the language to allow expression of advanced SQL queries used by complex extractors as well as a variety of factors with Markov Logic-like syntax.
A reference for ddlog lanugage features can be found [here](https://github.com/HazyResearch/ddlog/wiki/DDlog-Language-Features).

## Writing a DDlog Program

A DDlog program consists of the following parts:

1. Schema declarations (eg. relational tables)
2. Data transformation rules
  a. User Defined Functions (UDFs)
  b. Function Call Rules


All DDlog code should be placed in a file named `app.ddlog` under the DeepDive application.
A complete example written in DDlog can be found from [examples/spouse_example/postgres/ddlog](https://github.com/HazyResearch/deepdive/blob/master/examples/spouse_example/postgres/ddlog)


### Basic Syntax

Each declaration and rule ends with a period (`.`).
The order of the rules have no meaning.
Comments in DDlog begin with a hash (`#`) character.

### Schema Declaration

First of all, we declare the relations we use throughout the program.  These relations refer to tables that will appear in your database.  See [deepdive sql TODO](fill in with real link) for details on how to interact with the database.
The order doesn't matter, but it's a good idea to place them at the beginning because that makes it easier to understand.

#### Relations Syntax
```
relation_name(
  column1_name  column1_type,
  column2_name  column2_type,
  ...).
```
Similar to a SQL table definition, a schema declaration is just the name of a relation followed by a comma separated list of the columns and their types.

Below is a realistic example.
```
article(
  id int,
  length int,
  author text,
  words text[]).
```
Here we are defining a relation named "article" with four columns, named "id", "length", "author" and "words" respectively. Each column has it's own type, here utilizing `int`, `text` and `text[]`.
[//]: # (TODO  Maybe link this to a real example in spouse example instead)

<!--
#### Variable Relations
We can declare a variable relation that we want DeepDive to predict the marginal probability for us.
The syntax is as follows.
```
relation_name?(column_name column_type)
```

An example would be.
```
has_spouse?(relation_id text)
```

Here we are defining a relation has_spouse which will be a table created in the database having the column "relation_id" but it will also have several other columns created by DeepDive for internal use.  Ultimately, the inferences that are generated by DeepDive will be stored in this relation.
-->
### User Defined Functions (UDFs)
DeepDive allows the user to define their own functions.  These functions will read a relation line by line and then output to another relation.  Therefore, when we define a function in DDlog, we must define it's input structure, output structure and how it is implemented.

If we are writing a UDF to classify the articles above into a set of classes, we might have another relation
```
classified_articles(
  article_id int,
  class text).
```

The implementation of our UDF will be in a file called "udf/classify.py".  (Note: this is saying that it is a python script that is in the 'udf' folder but DeepDive can execute any type of executable).  The syntax for defining the function is
```
function classify_articles over (id int, author text, length int, words text[])
  return (article_id int, class text)
  implementation "udf/classify.py" handles tsv lines.
```

Notice that we've replicated the fields in the classified_articles.  Since we've already defined a relation that this function is intended to fill, we can simplify this declaration by using `rows like`

```
function classify_articles over (id int, author text, length int, words text[])
  return rows like classified_articles
  implementation "udf/classify.py" handles tsv lines.
```

Also note that the input is similar to the articles relation, but the fields are in a different order.  This is on purpose to show that the input need not match the relation defintion exactly.  We will see how this comes into play in the Function Call Rules below.

### Function Call Rules
The functions defined above can be used to fill rows into a relation.  The syntax below will be used to call the classify function to fill the classified_articles relation using data from the articles relation.
```
classified_articles += classify_articles(id, author, length, words) :- 
  article(id, length, author words)
```

The classify_articles function is being called on the input from articles and the output is being fed into classified_articles.  Note that the articles relation fields are in a different order than the input for the classify_articles function.  This is to demonstrate that the columns are referenced by name so they can be reordered or omitted when calling the function as needed.

<commit_msg>Fixes trailing space<commit_after>
# DDlog

DDlog is a higher-level language for writing DeepDive applications in succinct Datalog-like syntax.
We are gradually extending the language to allow expression of advanced SQL queries used by complex extractors as well as a variety of factors with Markov Logic-like syntax.
A reference for ddlog lanugage features can be found [here](https://github.com/HazyResearch/ddlog/wiki/DDlog-Language-Features).

## Writing a DDlog Program

A DDlog program consists of the following parts:

1. Schema declarations (eg. relational tables)
2. Data transformation rules
  a. User Defined Functions (UDFs)
  b. Function Call Rules


All DDlog code should be placed in a file named `app.ddlog` under the DeepDive application.
A complete example written in DDlog can be found from [examples/spouse_example/postgres/ddlog](https://github.com/HazyResearch/deepdive/blob/master/examples/spouse_example/postgres/ddlog)


### Basic Syntax

Each declaration and rule ends with a period (`.`).
The order of the rules have no meaning.
Comments in DDlog begin with a hash (`#`) character.

### Schema Declaration

First of all, we declare the relations we use throughout the program.  These relations refer to tables that will appear in your database.  See [deepdive sql TODO](fill in with real link) for details on how to interact with the database.
The order doesn't matter, but it's a good idea to place them at the beginning because that makes it easier to understand.

#### Relations Syntax
```
relation_name(
  column1_name  column1_type,
  column2_name  column2_type,
  ...).
```
Similar to a SQL table definition, a schema declaration is just the name of a relation followed by a comma separated list of the columns and their types.

Below is a realistic example.
```
article(
  id int,
  length int,
  author text,
  words text[]).
```
Here we are defining a relation named "article" with four columns, named "id", "length", "author" and "words" respectively. Each column has it's own type, here utilizing `int`, `text` and `text[]`.
[//]: # (TODO  Maybe link this to a real example in spouse example instead)

<!--
#### Variable Relations
We can declare a variable relation that we want DeepDive to predict the marginal probability for us.
The syntax is as follows.
```
relation_name?(column_name column_type)
```

An example would be.
```
has_spouse?(relation_id text)
```

Here we are defining a relation has_spouse which will be a table created in the database having the column "relation_id" but it will also have several other columns created by DeepDive for internal use.  Ultimately, the inferences that are generated by DeepDive will be stored in this relation.
-->
### User Defined Functions (UDFs)
DeepDive allows the user to define their own functions.  These functions will read a relation line by line and then output to another relation.  Therefore, when we define a function in DDlog, we must define it's input structure, output structure and how it is implemented.

If we are writing a UDF to classify the articles above into a set of classes, we might have another relation
```
classified_articles(
  article_id int,
  class text).
```

The implementation of our UDF will be in a file called "udf/classify.py".  (Note: this is saying that it is a python script that is in the 'udf' folder but DeepDive can execute any type of executable).  The syntax for defining the function is
```
function classify_articles over (id int, author text, length int, words text[])
  return (article_id int, class text)
  implementation "udf/classify.py" handles tsv lines.
```

Notice that we've replicated the fields in the classified_articles.  Since we've already defined a relation that this function is intended to fill, we can simplify this declaration by using `rows like`

```
function classify_articles over (id int, author text, length int, words text[])
  return rows like classified_articles
  implementation "udf/classify.py" handles tsv lines.
```

Also note that the input is similar to the articles relation, but the fields are in a different order.  This is on purpose to show that the input need not match the relation defintion exactly.  We will see how this comes into play in the Function Call Rules below.

### Function Call Rules
The functions defined above can be used to fill rows into a relation.  The syntax below will be used to call the classify function to fill the classified_articles relation using data from the articles relation.
```
classified_articles += classify_articles(id, author, length, words) :-
  article(id, length, author words)
```

The classify_articles function is being called on the input from articles and the output is being fed into classified_articles.  Note that the articles relation fields are in a different order than the input for the classify_articles function.  This is to demonstrate that the columns are referenced by name so they can be reordered or omitted when calling the function as needed.


{'fraction_non_alphanumeric': 0.06133493686109441, 'fraction_numerical': 0.0012026458208057728, 'mean_word_length': 4.4649507119386636, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3368609', 'n_tokens_mistral': 2647, 'n_tokens_neox': 2486, 'n_words': 1438}