<commit_before>h1. Introduction

Iterables support is a small collection of utility classes that I developed while processing a lot of content in files.

Processing content involves things like opening files, parsing them line by line, extracting blobs of json or xml that may span multiple line, dealing with character encoding problems, etc. 

Most of this stuff is pretty basic and yet you end up with a lot of boiler plate code that needs testing and attention. Java provides you with all the basic building blocks but you have to do a lot of work to put them together.

However, with these classes, you get some nice out of the box features and an easy to use framework with all of the building blocks you need for iterating over and processing blobs of information reliably and fast. This results in a lot less boiler plate code and makes things almost as easy as in python. Combine that with concurrency, decent memory management and a few other things Java coders take for granted and things are starting to look pretty OK.

Iterables support makes heavy use of several java language features:
* anything implementing Iterable<T> can be used with a for each type for loop.
* anything implementing Closeable cleans up after itself
* inner classes are the closest thing Java has to closures. 


h1. Features and design

Iterables support makes heavy use of several java language features:
* anything implementing Iterable<T> can be used with a for each type for loop.
* anything implementing Closeable cleans up after itself, this means no more finally blocks are needed
* inner classes are the closest thing Java has to closures. 

This framework represents inputs as iterables that one can iterate over using a foreach loop. Several iterables are provided that can be combined to do useful things.

* LineIterable. The most basic iterable is the LineIterable, which takes a stream and iterates over the lines as strings. It basically makes dealing with BufferedReaders a lot less tedious.

* CSVLineIterable. If you have a simple csv file, you can use the line iterable to construct a CSVLineIterable and iterate over the list of fields.

* MergingCSVIterable. Say you have two csv files that have a 1 to many relation and you want the join of those two files. Provided the files are sorted correctly, you can merge using an id. MergingCSVIterable does this. I've used it for processing millions of GeoNames and their translations (in a separate file).  

* If you have a big xml file with tags inside that you want to process that span multiple lines, use the BlobIterable and process each blob separately. Simply configure it with the begin and end tag and process each blob one by one.

* Often processing is expensive and you want to process things concurrently. This can be tricky to get right. So, use the ConcurrentProcessingIterable and configure it with the input iterable and a processor. It will do the rest for you and you simply iterate over the processed output.

* Sometimes you want to filter what you iterate over: FilteringIterable does that. Simply implement the Filter interface and pass an instance to the FilteringIterable and you've got filtering.

* Sometimes, you want to see what's next before it's coming. PeekableIterable allows a convenient peek() method that behaves just like next but without moving the iterator forward. 

* The Iterables class contains static methods to make usage of the above easy. You may want to use static imports and add this class to your eclipse Favorites.

h1. Installation

It's a maven project. So, checking it out and doing a mvn clean install should do the trick.

Alternatively, you can exercise your rights under the license and simply copy and adapt as needed. The "license":https://github.com/jillesvangurp/xmltools/blob/master/LICENSE allows you to do this and I have no problems with this although I do appreciate attribution.

Should anyone like this licensed differently, please contact me.

If anyone wants to fix stuff just send me a pull request.

h1. License

Like all my other projects, this project is licensed under the so-called MIT license. 

For more details see the LICENSE file

h1. Changelog

* 1.1<commit_msg>add link to blog post about my maven repo<commit_after>h1. Introduction

Iterables support is a small collection of utility classes that I developed while processing a lot of content in files.

Processing content involves things like opening files, parsing them line by line, extracting blobs of json or xml that may span multiple line, dealing with character encoding problems, etc. 

Most of this stuff is pretty basic and yet you end up with a lot of boiler plate code that needs testing and attention. Java provides you with all the basic building blocks but you have to do a lot of work to put them together.

However, with these classes, you get some nice out of the box features and an easy to use framework with all of the building blocks you need for iterating over and processing blobs of information reliably and fast. This results in a lot less boiler plate code and makes things almost as easy as in python. Combine that with concurrency, decent memory management and a few other things Java coders take for granted and things are starting to look pretty OK.

Iterables support makes heavy use of several java language features:
* anything implementing Iterable<T> can be used with a for each type for loop.
* anything implementing Closeable cleans up after itself
* inner classes are the closest thing Java has to closures. 


h1. Features and design

Iterables support makes heavy use of several java language features:
* anything implementing Iterable<T> can be used with a for each type for loop.
* anything implementing Closeable cleans up after itself, this means no more finally blocks are needed
* inner classes are the closest thing Java has to closures. 

This framework represents inputs as iterables that one can iterate over using a foreach loop. Several iterables are provided that can be combined to do useful things.

* LineIterable. The most basic iterable is the LineIterable, which takes a stream and iterates over the lines as strings. It basically makes dealing with BufferedReaders a lot less tedious.

* CSVLineIterable. If you have a simple csv file, you can use the line iterable to construct a CSVLineIterable and iterate over the list of fields.

* MergingCSVIterable. Say you have two csv files that have a 1 to many relation and you want the join of those two files. Provided the files are sorted correctly, you can merge using an id. MergingCSVIterable does this. I've used it for processing millions of GeoNames and their translations (in a separate file).  

* If you have a big xml file with tags inside that you want to process that span multiple lines, use the BlobIterable and process each blob separately. Simply configure it with the begin and end tag and process each blob one by one.

* Often processing is expensive and you want to process things concurrently. This can be tricky to get right. So, use the ConcurrentProcessingIterable and configure it with the input iterable and a processor. It will do the rest for you and you simply iterate over the processed output.

* Sometimes you want to filter what you iterate over: FilteringIterable does that. Simply implement the Filter interface and pass an instance to the FilteringIterable and you've got filtering.

* Sometimes, you want to see what's next before it's coming. PeekableIterable allows a convenient peek() method that behaves just like next but without moving the iterator forward. 

* The Iterables class contains static methods to make usage of the above easy. You may want to use static imports and add this class to your eclipse Favorites.

h1. Installation

It's a maven project. So, checking it out and doing a mvn clean install should do the trick.

Alternatively, you can exercise your rights under the license and simply copy and adapt as needed. The "license":https://github.com/jillesvangurp/xmltools/blob/master/LICENSE allows you to do this and I have no problems with this although I do appreciate attribution.

If you like to use pre-built jars, you can utilize my private maven repository as explained "here":http://www.jillesvangurp.com/2013/02/27/maven-and-my-github-projects/

Should anyone like this licensed differently, please contact me.

If anyone wants to fix stuff just send me a pull request.

h1. License

Like all my other projects, this project is licensed under the so-called MIT license. 

For more details see the LICENSE file

h1. Changelog

* 1.1
{'fraction_non_alphanumeric': 0.028481748430597534, 'fraction_numerical': 0.0027900488258544524, 'mean_word_length': 4.346799254195152, 'pattern_counts': {'":': 3, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1722069', 'n_tokens_mistral': 2047, 'n_tokens_neox': 1953, 'n_words': 1382}