<commit_before># Policy Manager

## Quick Start

To start the policy manager simply run:

```
./neatpmd
```

in the `neat/policy` directory. See `./neatpmd -h` for more options.

## NEAT properties

NEAT properties are essentially `key|value` tuples describing attributes used by the components of the NEAT Policy Manager. In addition properties contain meta attributes such as a score denoting the importance or weight of the property. The syntax and behaviour of NEAT properties is documented [here](doc/NEAT%20Properties.md)

### NEAT Property Arrays

A NEAT *Property Array* is a simple object containing a set of NEAT properties. These objects are used to represent applications *requests* received through the NEAT API as well as the connection *candidates* generated by the PM as the request traverses the PM logic.
 
## NEAT Policies

Policies are based around NEAT properties. In addition to some metadata, each policy contains the following attributes:

+ `match`: object containing a set of properties which trigger the policy. A policy is triggered only if *all* of these properties are contained within (i.e., match) the input candidate. An empty or missing match object will match *any* candidate.
+ `properties`: object containing a set of properties which should be applied to the connection candidate (if feasible given the property precedences).

NEAT profiles are functionally identical to NEAT policies but are applied earlier in the Policy Manager workflow.

For more details refer to [__TODO__]().


 

## Policy Manager Workflow

The Policy Manager listens for *requests* generated by the NEAT logic and returns a list of *candidates* ranked according to the associated property scores. To achieve this the PM queries the Policy Information Base (PIB) and Characteristics Information Base (CIB) repositories.

Each NEAT request is processed in three steps:

1. **Profile Lookup**: the request properties are compared to all profile entries in the PIB. Whenever a profile entry is matched, the corresponding match property in the request is *replaced* with the associated profile properties. Profiles are specified using the same format as policies.
 
2. **CIB Lookup**: the request properties are compared against each entry in the CIB. The properties of a candidate are the union of the request and CIB entry property sets. Specifically, the properties are obtained by overlaying the request properties with the properties of a single CIB entry and updating the *intersection* of the two property sets with corresponding the values from the CIB entry properties.
The *N* entries with the largest aggregate score are appended to the candidate list.

3. **PIB Lookup**: For each candidate the PM iterates through all PIB policies and compares the match properties with the candidates properties. A policy is said to *match* a candidate whenever *all* of its match properties are found in the candidate properties. PIB entries are matched with a *shortest match first* strategy, i.e., policies with the smallest number of `match` properties are applied first. Subsequent, policies will *overwrite* any perviously applied policy properties. Conflicting policies must be identified by the NEAT logic.


## CIB format

[See CIB documentation](doc/CIB%20format.md)

# Example

## Running the PM


To start the policy manager run:

```
$ python3.5 ./neatpmd --cib ./example/cib/ --pib ./example/pib

```

in the `neat/policy` directory. The `--cib` and `--pib` options specify the respective locations of the CIB and the PIB. By default the PM will create a Unix domain socket located at `~/.neat/neat_pm_socket`, where it will listen for JSON strings containing application requests, and it will output the list of generated candidates. The directory for the domain socket may be overridden using the `--sock` option.

We can test `neatpmd` using the `socat` utility:

```
$ NEAT_PM_SOCKET=~/.neat/neat_pm_socket
$ JSON='{"transport": {"value": "TCP"}, "MTU": {"value": [1500, Infinity]}, "low_latency": {"precedence": 2, "value": true}, "remote_ip": {"precedence": 2, "value": "10:54:1.23"}}'
$ echo $JSON | socat -d -d STDIO UNIX-CONNECT:$NEAT_PM_SOCKET
``` 

or to read from a file:

``` 
$ NEAT_PM_SOCKET=~/.neat/neat_pm_socket
$ cat request.json | socat -d -d STDIO UNIX-CONNECT:$NEAT_PM_SOCKET
``` 


The PM will return a JSON string containing the list of connection candidates (two of them for the given example) into the Unix socket. 

```<commit_msg>PM fixed example in documentation<commit_after># Policy Manager

## Quick Start

To start the policy manager simply run:

```
./neatpmd
```

in the `neat/policy` directory. See `./neatpmd -h` for more options.

## NEAT properties

NEAT properties are essentially `key|value` tuples describing attributes used by the components of the NEAT Policy Manager. In addition properties contain meta attributes such as a score denoting the importance or weight of the property. The syntax and behaviour of NEAT properties is documented [here](doc/NEAT%20Properties.md)

### NEAT Property Arrays

A NEAT *Property Array* is a simple object containing a set of NEAT properties. These objects are used to represent applications *requests* received through the NEAT API as well as the connection *candidates* generated by the PM as the request traverses the PM logic.
 
## NEAT Policies

Policies are based around NEAT properties. In addition to some metadata, each policy contains the following attributes:

+ `match`: object containing a set of properties which trigger the policy. A policy is triggered only if *all* of these properties are contained within (i.e., match) the input candidate. An empty or missing match object will match *any* candidate.
+ `properties`: object containing a set of properties which should be applied to the connection candidate (if feasible given the property precedences).

NEAT profiles are functionally identical to NEAT policies but are applied earlier in the Policy Manager workflow.

For more details refer to [__TODO__]().


 

## Policy Manager Workflow

The Policy Manager listens for *requests* generated by the NEAT logic and returns a list of *candidates* ranked according to the associated property scores. To achieve this the PM queries the Policy Information Base (PIB) and Characteristics Information Base (CIB) repositories.

Each NEAT request is processed in three steps:

1. **Profile Lookup**: the request properties are compared to all profile entries in the PIB. Whenever a profile entry is matched, the corresponding match property in the request is *replaced* with the associated profile properties. Profiles are specified using the same format as policies.
 
2. **CIB Lookup**: the request properties are compared against each entry in the CIB. The properties of a candidate are the union of the request and CIB entry property sets. Specifically, the properties are obtained by overlaying the request properties with the properties of a single CIB entry and updating the *intersection* of the two property sets with corresponding the values from the CIB entry properties.
The *N* entries with the largest aggregate score are appended to the candidate list.

3. **PIB Lookup**: For each candidate the PM iterates through all PIB policies and compares the match properties with the candidates properties. A policy is said to *match* a candidate whenever *all* of its match properties are found in the candidate properties. PIB entries are matched with a *shortest match first* strategy, i.e., policies with the smallest number of `match` properties are applied first. Subsequent, policies will *overwrite* any perviously applied policy properties. Conflicting policies must be identified by the NEAT logic.


## CIB format

[See CIB documentation](doc/CIB%20format.md)

# Example

## Running the PM


To start the policy manager run:

```
$ python3.5 ./neatpmd --cib ./examples/cib/ --pib ./examples/pib

```

in the `neat/policy` directory. The `--cib` and `--pib` options specify the respective locations of the CIB and the PIB. By default the PM will create a Unix domain socket located at `~/.neat/neat_pm_socket`, where it will listen for JSON strings containing application requests, and it will output the list of generated candidates. The directory for the domain socket may be overridden using the `--sock` option.

We can test `neatpmd` using the `socat` utility:

```
$ NEAT_PM_SOCKET=~/.neat/neat_pm_socket
$ JSON='{"transport": {"value": "TCP"}, "MTU": {"value": [1500, Infinity]}, "low_latency": {"precedence": 2, "value": true}, "remote_ip": {"precedence": 2, "value": "10:54:1.23"}}'
$ echo $JSON | socat -d -d STDIO UNIX-CONNECT:$NEAT_PM_SOCKET
``` 

or to read from a file:

``` 
$ NEAT_PM_SOCKET=~/.neat/neat_pm_socket
$ cat request.json | socat -d -d STDIO UNIX-CONNECT:$NEAT_PM_SOCKET
``` 


The PM will return a JSON string containing the list of connection candidates (two of them for the given example) into the Unix socket. 

```
{'fraction_non_alphanumeric': 0.07174688057040998, 'fraction_numerical': 0.004901960784313725, 'mean_word_length': 4.6494650723725615, 'pattern_counts': {'":': 20, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1727281', 'n_tokens_mistral': 2373, 'n_tokens_neox': 2270, 'n_words': 1270}