<commit_before>6/3/13 - mike
For the time being, I'm going to store all of the classes I write to handle the data sources available to users of the browser tool in this folder.  Right now there is just the DerbyDB interface (and its dependency).  I'll refine the organization as I upload more pieces of the browser tool.

DerbyDB
This is a class written using the Derby API to handle basic operations with the large metadata table that was generated during an earlier phase of the project.  Currently, it is capable of:
- connecting to an existing, local database
- creating a new database and populating it using metatable.txt (or a similar structured table)
- filtering metatable.txt entries with ranges or fuzzy dates into separate tables within the Derby database so that queries can be made based on publication date and/or results ordered by date
- issuing SQL queries and returning the results as a String[][] matrix

Databases are connected to (or created) using the constructors.  In addition to specifying a local database or a seed metatable.txt file, a framework must be also be specified.  For now, this should always be "embedded".  Changes to the constructors to setup a networked connection shouldn't affect the rest of the methods.

To use an instance of DerbyDB to perform a search, the object will need to be passed into the object that deals with user input.  The user input object will then need to parse search fields into an SQL string for DerbyDB.  I left the query method open-ended, accepting just an SQL command as a String so that there wouldn't be any limitations on searches in DerbyDB.

This class will not work unless the 

TODO: 
- Proper exception handling
- A better way of getting from Derby's awkward ResultSet to a String[][] (needed by Java's table models)
- Consider indexing author/title columns in Derby.  Set HTid as primary key?<commit_msg>Added a line to readme.txt about dependencies for DerbyDB<commit_after>6/3/13 - mike
For the time being, I'm going to store all of the classes I write to handle the data sources available to users of the browser tool in this folder.  Right now there is just the DerbyDB interface (and its dependency).  I'll refine the organization as I upload more pieces of the browser tool.

DerbyDB
This is a class written using the Derby API to handle basic operations with the large metadata table that was generated during an earlier phase of the project.  Currently, it is capable of:
- connecting to an existing, local database
- creating a new database and populating it using metatable.txt (or a similar structured table)
- filtering metatable.txt entries with ranges or fuzzy dates into separate tables within the Derby database so that queries can be made based on publication date and/or results ordered by date
- issuing SQL queries and returning the results as a String[][] matrix

Databases are connected to (or created) using the constructors.  In addition to specifying a local database or a seed metatable.txt file, a framework must be also be specified.  For now, this should always be "embedded".  Changes to the constructors to setup a networked connection shouldn't affect the rest of the methods.

To use an instance of DerbyDB to perform a search, the object will need to be passed into the object that deals with user input.  The user input object will then need to parse search fields into an SQL string for DerbyDB.  I left the query method open-ended, accepting just an SQL command as a String so that there wouldn't be any limitations on searches in DerbyDB.

This class will not work unless Apache's derby.jar is included in the CLASSPATH!

TODO: 
- Proper exception handling
- A better way of getting from Derby's awkward ResultSet to a String[][] (needed by Java's table models)
- Consider indexing author/title columns in Derby.  Set HTid as primary key?
{'fraction_non_alphanumeric': 0.03475103734439834, 'fraction_numerical': 0.002074688796680498, 'mean_word_length': 4.26912568306011, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6923472', 'n_tokens_mistral': 933, 'n_tokens_neox': 880, 'n_words': 627}