<commit_before># node-red-contrib-bigfile

"Big File" contrib for node-red. Original work by IBM doesn't fit my needs. With big files, multiple blocks are sent making parsing inconsistent. I also needed the file reader to use state of the art libraries
This node is able to manage big files by sending blocks or string  or lines

![alt tag](https://cloud.githubusercontent.com/assets/18165555/14589405/e3ae04e6-04e1-11e6-8591-5b8039a1e6b0.png)

![alt tag](https://cloud.githubusercontent.com/assets/18165555/14588392/dad963a6-04c8-11e6-8539-f9b4afc4cc32.png)

![alt tag](https://cloud.githubusercontent.com/assets/18165555/14588393/e0718708-04c8-11e6-888e-1489222be76e.png)

## Installation
```bash
npm install node-red-contrib-bigfile
cd node-red-contrib-bigfile
npm install
```

## Principles for Big Nodes

###1 can handle big data or block mode<commit_msg>not needed<commit_after># node-red-contrib-bigfile

"Big File" contrib for node-red. Original work by IBM doesn't fit my needs. With big files, multiple blocks are sent making parsing inconsistent. I also needed the file reader to use state of the art libraries
This node is able to manage big files by sending blocks or string  or lines

![alt tag](https://cloud.githubusercontent.com/assets/18165555/14589405/e3ae04e6-04e1-11e6-8591-5b8039a1e6b0.png)

![alt tag](https://cloud.githubusercontent.com/assets/18165555/14588392/dad963a6-04c8-11e6-8539-f9b4afc4cc32.png)

![alt tag](https://cloud.githubusercontent.com/assets/18165555/14588393/e0718708-04c8-11e6-888e-1489222be76e.png)

## Installation
```bash
npm install node-red-contrib-bigfile
```

## Principles for Big Nodes

###1 can handle big data or block mode
{'fraction_non_alphanumeric': 0.10673822301729279, 'fraction_numerical': 0.13714967203339296, 'mean_word_length': 5.014336917562724, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6754885', 'n_tokens_mistral': 716, 'n_tokens_neox': 580, 'n_words': 156}