<commit_before>CODE IMPROVEMENTS
=================

lmpampaletakis:
- Can we avoid passing DatabaseConfiguration dbConf in constructors? Is it possible to inject automatically the object or store it and request it from within the algorithms?

other:
- CSV reader and converter to Dataset. Eliminate Dataset.value2ColumnType() to detect the types?

- Serialization improvements: set serialVersionUID?
- create the required exceptions


DOCUMENTATION
=============

- Documenting the code
- How-to blog post on building a Text Classification model


NEW ALGORITHMS
==============

- include an anomaly detection algorithm
- develop the FunkSVD: Also PLSI as probabilistic version of SVD
- ability to search through the configuration space and find the best performing algorithmic configuration


CHECK OUT HUGE COLLECTION LIBS, DBS AND STORAGE:
================================================

Java StoredMap + BerkeleyDB:
http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/collections/StoredMap.html
http://www.oracle.com/technetwork/database/berkeleydb/overview/index-093405.html

Vanilla-java - HugeCollections:
https://code.google.com/p/vanilla-java/wiki/HugeCollections

Fastutil:
http://fastutil.di.unimi.it/#install

Joafip:
http://joafip.sourceforge.net/javadoc/net/sf/joafip/java/util/PHashMap.html
<commit_msg>Update todos.<commit_after>CODE IMPROVEMENTS
=================

lmpampaletakis:
- Be able to store and save the DatabaseConfiguration dbConf in properties file:
	public static DatabaseConfiguration load(String filepath);
	public boolean export(String filepath);

The config file should store also which Config object was selected by the user (InMemory vs MapDB). The object is then passed normally though constructors as is right now, UNLESS there is a better alternative.

other:
- CSV reader and converter to Dataset. Eliminate Dataset.value2ColumnType() to detect the types?

- Serialization improvements: set serialVersionUID?
- create the required exceptions


DOCUMENTATION
=============

- Documenting the code
- How-to blog post on building a Text Classification model


NEW ALGORITHMS
==============

- include an anomaly detection algorithm
- develop the FunkSVD: Also PLSI as probabilistic version of SVD
- ability to search through the configuration space and find the best performing algorithmic configuration


CHECK OUT HUGE COLLECTION LIBS, DBS AND STORAGE:
================================================

Java StoredMap + BerkeleyDB:
http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/collections/StoredMap.html
http://www.oracle.com/technetwork/database/berkeleydb/overview/index-093405.html

Vanilla-java - HugeCollections:
https://code.google.com/p/vanilla-java/wiki/HugeCollections

Fastutil:
http://fastutil.di.unimi.it/#install

Joafip:
http://joafip.sourceforge.net/javadoc/net/sf/joafip/java/util/PHashMap.html

{'fraction_non_alphanumeric': 0.12715008431703204, 'fraction_numerical': 0.009443507588532884, 'mean_word_length': 4.4322344322344325, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6655294', 'n_tokens_mistral': 924, 'n_tokens_neox': 820, 'n_words': 259}