<commit_before>## sflearn

**sflearn** is a small library implementing algorithms for inference of
deterministic transducers models. These models also include transducers with the
_bounded lookahead property_, i.e. transducers which consume more than one input
symbols with a single transition. These models are necessary in order to
represent programs such as string sanitizers. The library also contains a class
to convert the infered models into BEK programs (see the [BEK
tutorial](www.rise4fun.com/Bek/tutorial)).

The primary target of these algorithms are the inference of string manipulating
programs such as HTML encoders/decoders and filters used to sanitize untrusted
user input.

For more details, see the paper:

**Back in Black: Towards Formal, Black-Box Analysis of Sanitizers and Filters**  
_<NAME>, <NAME>, <NAME> and <NAME>_  
37th IEEE Symposium on Security and Privacy, 2016.


## Installation

To install the library run

    git clone https://github.com/GeorgeArgyros/sflearn  
    cd sflearn  
    python setup.py install

## Usage

In order to use this library one should inherit one of the learning algorithm
classes, either `MealyMachineLearner`, or `TransducerLearner`  and define the
methods `membership_query` and `equivalence_query`. For more details regarding
the logic of these methods consult the paper.

Conversion to BEK programs is performed by using the `BekProgram` class of the
library.

The `examples/` directory contains a number of practical examples on how to use
these functions to construct models of various kinds of string manipulating
programs.

**Note:** To avoid assumptions on the location of the library on the system,
the example code in the `examples/` directory utilizes an intermidiate
`context.py` file to import the library. If the library is installed in the
system then this call can be replaced with a normal import of the library.

## Authors

This library was designed and developed by <NAME> and is based on the algorithms developed by <NAME>, <NAME>, <NAME> and <NAME>.
<commit_msg>setup.cfg<commit_after>## sflearn

**sflearn** is a small library implementing algorithms for inference of
deterministic transducers models. These models also include transducers with the
_bounded lookahead property_, i.e. transducers which consume more than one input
symbols with a single transition. These models are necessary in order to
represent programs such as string sanitizers. The library also contains a class
to convert the infered models into BEK programs (see the [BEK
tutorial](www.rise4fun.com/Bek/tutorial)).

The primary target of these algorithms are the inference of string manipulating
programs such as HTML encoders used to sanitize untrusted user input.

For more details, see the paper:

**Back in Black: Towards Formal, Black-Box Analysis of Sanitizers and Filters**  
_<NAME>, <NAME>, <NAME> and <NAME>_  
37th IEEE Symposium on Security and Privacy, 2016.


## Installation

To install the library run

    git clone https://github.com/GeorgeArgyros/sflearn  
    cd sflearn  
    python setup.py install

## Usage

In order to use this library one should inherit one of the learning algorithm
classes, either `MealyMachineLearner`, or `TransducerLearner`  and define the
methods `membership_query` and `equivalence_query`. For more details regarding
the logic of these methods consult the paper.

Conversion to BEK programs is performed by using the `BekProgram` class of the
library.

The `examples/` directory contains a number of practical examples on how to use
these functions to construct models of various kinds of string manipulating
programs. The examples include:

* _html\_encoder.py_: Learns a model of the HTML encoder from the python cgi module.
* _basic\_lookahead.py_: Debugging program for lookahead transducers, learns a simple transducer defined in the file.
* _replacecomments.py_: Learns the ReplaceComments() de-obfuscation function used to remove SQL style `/* ... */` comments from strings. 
* _htmlspecialchars.py_: Learns 

To run an example

    cd examples/ 
    python replacecomments.py --bek

The `--bek` option will also generate a BEK program from the infered transducer. To see all options run with `-h`.


**Note:** To avoid assumptions on whether the library is installed when running the examples,
the code in the `examples/` directory utilizes an intermidiate
`context.py` file to import the library. If the library is installed in the
system then this call can be replaced with a normal import of the library.

## Authors

This library was designed and developed by <NAME> and is based on the algorithms developed by <NAME>, <NAME>, <NAME> and <NAME>.

{'fraction_non_alphanumeric': 0.055698371893744644, 'fraction_numerical': 0.0029991431019708655, 'mean_word_length': 4.591616766467066, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7071089', 'n_tokens_mistral': 1256, 'n_tokens_neox': 1168, 'n_words': 649}