<commit_before>Data Filters Brick
==================
This brick provides improved way to handle data permissions set up both through data permissions and variables.

## Important terms

### Data filters
Data filters are basically whatever comes after WHERE in MAQL. Imagine

    WHERE city IN ('San Francisco', 'Prague', 'Amsterdam')

The difference is that in both cases you can define it per user.

### Variable
Variable is a per user filter. You can apply a variable on per report basis so you can basically pick which report will be filtered and which not. This is different with data permissions.

### Data permissions
Data permission is again a filter that is defined per user. The difference that an intersection of all filters is applied to every computation a user performs automatically so it is well suited to implement security rules.

## Colum-based vs row-based sources of data
In many cases you cannot pick what kind of data you get. We are trying to help minimize glue code as much as possible. You can easily configure this brick to accept couple types of input with respect to its shape and name of the columns. Let's try to illustrate different shapes given the example with cities at the beginning.

Imagine there are two users in your system and you would like to set him with these values.

 login              | filter
--------------------|----------
 <EMAIL>   | WHERE city ('San Francisco', 'Prague', 'Amsterdam')
 <EMAIL>   | WHERE city ('San Francisco', 'Dublin')

### Column based

This is how the values would be expressed in a column based file.

 login              | city          | other_unused_value
--------------------|---------------|-------------
 <EMAIL>   | San Francisco | x
 <EMAIL>   | Prague        | x
 <EMAIL>   | Amsterdam     | y
 <EMAIL>   | Dublin        | y
 <EMAIL>   | San Francisco | y

Take note that multiple values are expressed by repeating the user. Brick also deduplicates the values before setting them as Data Filters. This might easily happen if you got the file for filters as an output of a join.

Notice that in our case the file has headers but this is not a requirement and brick can cope with both situations.

### Row based
While column based file is the most commonly used source but imagine that a different kind of file would arrive. The example above could be also expressed like this


  Note: Please use your imagination and fancy that the table does not have headers. Unfortunately Markdown does not seem to support that :-)

  .                 | .
--------------------|---------------------------------------
 <EMAIL>   | "San Francisco", "Prague", "Amsterdam"
 <EMAIL>   | "Dublin", "San Francisco"
 
Notice that this file cannot have headers since we do not know how many columns we will have  (They are d).

## Adding filters

Regardless of which type of source you use to define your filters the information you provide will be used as the blueprint for the end state of the project. It is very similar to how adding user works. You are providing how the end state should look like and the system determines what to do.

### Why declarative and not additive

Let me explain shortly why we decided to go the declarative ('give me how the project should look like and I will make it so') way as opposed to additive ('let me know each day who to add and remove'). The benefit of being declarative is the fact that it is stateless. If I give you the full file every day you know where you will end up regardless of how the project looks like right now. Even if you run things several times you end up in the same spot. If I give you an increment where you end up is dependent on the state of the project. If something goes wrong (loss of users) you probably have to get the full state anyway. Thanks to this the declarative way is also self healing. If you make a mistake one day it is going to fix itself automatically when you fix the data.

If the source system can provide only incremental data you have to do the extra work in your stateful ETL to provide a complete snapshot of the system (as you have to do for other parts as well). Taking data from an intermediary storage has also another benefit is that you own the data you were using to perform the tasks so if you need them later you can access them which is not necessarily guaranteed with data you do not own.


## Use cases

### Data permissions through value enumeration with column based file

This is what we need to set up.

    <EMAIL>: WHERE city IN ('San Francisco')
    <EMAIL>: WHERE city IN ('San Francisco', 'Prague')

This is how the file looks like

    <EMAIL>,'San Francisco'
    <EMAIL>,'Prague'
    <EMAIL>,'San Francisco'

This is how you have to set up your process

    {
      "input_source": { "type": "web", "url": "https://gist.githubusercontent.com/fluke777/7fdd8453c3c811ccc5a9/raw/155f6fd8414135e16994f30c2d6b16356872001c/gistfile1.txt" }
      "domain": "my_domain",
      "filters_config": {
        "user_column": "login",
        "labels": [{"label": "label.devs.dev_id.email", "column": "email"}]
      }
    }

Notes:

1) You can also use numbers to specify the "user_column" or "column" to be used in case you do not have headers.
2) If your file does not have a header you have to explicitly specify it using `"csv_headers": false` parameter.

### Data permissions through value enumeration with row based file

This is what we need to set up.

    <EMAIL>: WHERE city IN ('San Francisco')
    <EMAIL>: WHERE city IN ('San Francisco', 'Prague')

This is how the file looks like

    <EMAIL>,'San Francisco','Prague'
    <EMAIL>,'San Francisco'

This is how you have to set up your process<commit_msg>Fixing docs<commit_after>Data Filters Brick
==================
This brick provides improved way to handle data permissions set up both through data permissions and variables.

## Important terms

### Data filters
Data filters are basically whatever comes after WHERE in MAQL. Imagine

    WHERE city IN ('San Francisco', 'Prague', 'Amsterdam')

The difference is that in both cases you can define it per user.

### Variable
Variable is a per user filter. You can apply a variable on per report basis so you can basically pick which report will be filtered and which not. This is different with data permissions.

### Data permissions
Data permission is again a filter that is defined per user. The difference that an intersection of all filters is applied to every computation a user performs automatically so it is well suited to implement security rules.

## Colum-based vs row-based sources of data
In many cases you cannot pick what kind of data you get. We are trying to help minimize glue code as much as possible. You can easily configure this brick to accept couple types of input with respect to its shape and name of the columns. Let's try to illustrate different shapes given the example with cities at the beginning.

Imagine there are two users in your system and you would like to set him with these values.

 login              | filter
--------------------|----------
 <EMAIL>   | WHERE city ('San Francisco', 'Prague', 'Amsterdam')
 <EMAIL>   | WHERE city ('San Francisco', 'Dublin')

### Column based

This is how the values would be expressed in a column based file.

 login              | city          | other_unused_value
--------------------|---------------|-------------
 <EMAIL>   | San Francisco | x
 <EMAIL>   | Prague        | x
 <EMAIL>   | Amsterdam     | y
 <EMAIL>   | Dublin        | y
 <EMAIL>   | San Francisco | y

Take note that multiple values are expressed by repeating the user. Brick also deduplicates the values before setting them as Data Filters. This might easily happen if you got the file for filters as an output of a join.

Notice that in our case the file has headers but this is not a requirement and brick can cope with both situations.

### Row based
While column based file is the most commonly used source but imagine that a different kind of file would arrive. The example above could be also expressed like this


  Note: Please use your imagination and fancy that the table does not have headers. Unfortunately Markdown does not seem to support that :-)

  .                 | .
--------------------|---------------------------------------
 <EMAIL>   | "San Francisco", "Prague", "Amsterdam"
 <EMAIL>   | "Dublin", "San Francisco"
 
Notice that this file cannot have headers since we do not know how many columns we will have  (They are d).

## Adding filters

Regardless of which type of source you use to define your filters the information you provide will be used as the blueprint for the end state of the project. It is very similar to how adding user works. You are providing how the end state should look like and the system determines what to do.

### Why declarative and not additive

Let me explain shortly why we decided to go the declarative ('give me how the project should look like and I will make it so') way as opposed to additive ('let me know each day who to add and remove'). The benefit of being declarative is the fact that it is stateless. If I give you the full file every day you know where you will end up regardless of how the project looks like right now. Even if you run things several times you end up in the same spot. If I give you an increment where you end up is dependent on the state of the project. If something goes wrong (loss of users) you probably have to get the full state anyway. Thanks to this the declarative way is also self healing. If you make a mistake one day it is going to fix itself automatically when you fix the data.

If the source system can provide only incremental data you have to do the extra work in your stateful ETL to provide a complete snapshot of the system (as you have to do for other parts as well). Taking data from an intermediary storage has also another benefit is that you own the data you were using to perform the tasks so if you need them later you can access them which is not necessarily guaranteed with data you do not own.


## Use cases

### Data permissions through value enumeration with column based file

This is what we need to set up.

    <EMAIL>: WHERE city IN ('San Francisco')
    <EMAIL>: WHERE city IN ('San Francisco', 'Prague')

This is how the file looks like

    login,city
    <EMAIL>,'San Francisco'
    <EMAIL>,'Prague'
    <EMAIL>,'San Francisco'

This is how you have to set up your process

    {
      "input_source": { "type": "web", "url": "https://gist.githubusercontent.com/fluke777/7fdd8453c3c811ccc5a9/raw/155f6fd8414135e16994f30c2d6b16356872001c/gistfile1.txt" }
      "domain": "my_domain",
      "filters_config": {
        "user_column": "login",
        "labels": [{"label": "label.cities.city_id.name", "column": "city"}]
      }
    }

Notes:

1) You can also use numbers to specify the "user_column" or "column" to be used in case you do not have headers.
2) If your file does not have a header you have to explicitly specify it using `"csv_headers": false` parameter.

### Data permissions through value enumeration with row based file

This is what we need to set up.

    <EMAIL>: WHERE city IN ('San Francisco')
    <EMAIL>: WHERE city IN ('San Francisco', 'Prague')

This is how the file looks like

    <EMAIL>,'San Francisco','Prague'
    <EMAIL>,'San Francisco'

This is how you have to set up your process
{'fraction_non_alphanumeric': 0.08308422718480746, 'fraction_numerical': 0.008440302444170916, 'mean_word_length': 3.249159506910721, 'pattern_counts': {'":': 20, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1058727', 'n_tokens_mistral': 2893, 'n_tokens_neox': 2763, 'n_words': 1739}