<commit_before>

image:https://img.shields.io/npm/v/stream-line-parser.svg[npm,link=https://www.npmjs.com/package/stream-line-parser]
image:https://secure.travis-ci.org/darlenya/stream-line-parser.png[Build Status,link=http://travis-ci.org/darlenya/stream-line-parser]
image:https://david-dm.org/darlenya/stream-line-parser.svg[Dependency Status,link=https://david-dm.org/darlenya/stream-line-parser]
image:https://david-dm.org/darlenya/stream-line-parser/dev-status.svg[devDependency Status,link=https://david-dm.org/darlenya/stream-line-parser#info=devDependencies]



== Adapter Inbound File
This adapter open file streams and send them out as messages.
It could listen on directories for new files or it could be externally triggered.

=== Endpoints
This Adapter has two endpoints. One 'out' endpoint which send out the messages with
the red stream and one 'in' endpoint for external trigger events.

==== out
This endpoint send out file read messages.

.Endpoint description
[options="header", cols="2"]
|====
|Description|Value
|*Name*|out
|*Direction*|out
|*Active*|true
|*Passive*|false
|====

.Message Format
[source,js]
----
{
  "header": {
    "file_name": fileName,   <1>
    "directory": directory,  <2>
    "file_stat": stat        <3>
  },
  "payload": stream          <4>
}
----
<1> The Name of the file. Without the path.<commit_msg>doc: Update the documentation<commit_after>image:https://img.shields.io/npm/v/stream-line-parser.svg[npm,link=https://www.npmjs.com/package/stream-line-parser]
image:https://secure.travis-ci.org/darlenya/stream-line-parser.png[Build Status,link=http://travis-ci.org/darlenya/stream-line-parser]
image:https://david-dm.org/darlenya/stream-line-parser.svg[Dependency Status,link=https://david-dm.org/darlenya/stream-line-parser]
image:https://david-dm.org/darlenya/stream-line-parser/dev-status.svg[devDependency Status,link=https://david-dm.org/darlenya/stream-line-parser#info=devDependencies]

== Adapter Inbound File
This adapter open file streams and send them out as messages.
It could listen on directories for new files or it could be externally triggered.

=== Configuration
The configuration for this step is as follows.

.Message formats
[source,js]
----
{
  "regEx" : ".*\\.csv",      <1>
  "filter": filterFunction,  <2>
  "watchDir": directory      <3>
  "onlyReadNewFiles" : true  <4>
}
----
<1> (Optional)  A regular expression which will be matched against the file names changed. If the regEx did not
match no message will be generated.
<2> (Optional)  A custom function to filter the file names. The function looks like ''function (fileName){return true}''.
The function gets the file name as parameter and must return true or false.
<3> (Optional)  The directory which should be watched for changes.
<4> (Optional)  If set to true (default) then only new files added will be send. Else it will also
send the already existing files.

=== Endpoints
This Adapter has two endpoints. One 'out' endpoint which send out the messages with
the red stream and one 'in' endpoint for external trigger events.

==== out
This endpoint send out file read messages.

.Endpoint description
[options="header", cols="2"]
|====
|Description|Value
|*Name*|out
|*Direction*|out
|*Active*|true
|*Passive*|false
|====

.Message Format
[source,js]
----
{
  "header": {
    "file_name": fileName,   <1>
    "directory": directory,  <2>
    "file_stat": stat        <3>
  },
  "payload": stream          <4>
}
----
<1> The Name of the file. Without the path.
{'fraction_non_alphanumeric': 0.14175478708202344, 'fraction_numerical': 0.005715918833952558, 'mean_word_length': 4.2631578947368425, 'pattern_counts': {'":': 12, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 14, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1615759', 'n_tokens_mistral': 1139, 'n_tokens_neox': 1112, 'n_words': 323}