<commit_before>### File Type Detection

This repository contains files for running Deep Neural Network that help generate NNModel files for Apache Tika.

## Dependecies

- Pandas
- Numpy
- Theano (for leveraging GPU and building deeper netwoks)


## Training Results

```python
Parameter `pieces` is unused for layer type `Sigmoid`.
Parameter `pieces` is unused for layer type `Sigmoid`.
Initializing neural network with 3 layers, 255 inputs and 2 outputs.
Training on dataset of 222,578 samples with 57,202,546 total size.
Terminating after specified 25 total iterations.
0.999588904464
```


| Mime type     		  | Test Accuracy     | Number of Hidden Layers      
| ------------------------|:------------------|:-----------------------
| application/x-grib      | 92.34%			  |  2
| application/x-grib   	  | 94.33%			  |  4
| application/xhtml  	  | 99.5%			  |  2


Current Run Statistics:
Reported accuracy on application/xhtml : 99.5%

## Running the project

Simply execute:

`python
neuralNetworkTrainer.py
`

This will cause the program to generate a nnmodel file that can be used with Tika.

## Understanding the input file

It is assumed that the input training files have the following format:<commit_msg>fix: adding doc fixes to reflect new folder restructuring.<commit_after>### File Type Detection

This repository contains files for running Deep Neural Network that help generate NNModel files for Apache Tika.

## Dependecies

- Pandas
- Numpy
- Theano (for leveraging GPU and building deeper netwoks)


## Training Results

```python
Parameter `pieces` is unused for layer type `Sigmoid`.
Parameter `pieces` is unused for layer type `Sigmoid`.
Initializing neural network with 3 layers, 255 inputs and 2 outputs.
Training on dataset of 222,578 samples with 57,202,546 total size.
Terminating after specified 25 total iterations.
0.999588904464
```


| Mime type     		  | Test Accuracy     | Number of Hidden Layers      
| ------------------------|:------------------|:-----------------------
| application/x-grib      | 92.34%			  |  2
| application/x-grib   	  | 94.33%			  |  4
| application/xhtml  	  | 99.5%			  |  2


Current Run Statistics:
Reported accuracy on application/xhtml : 99.5%

## Running the project

Simply execute:

`python
tester.py
`

This will cause the program to generate a nnmodel file that can be used with Tika.

## Understanding the input file

It is assumed that the input training files have the following format:
{'fraction_non_alphanumeric': 0.12099066179455949, 'fraction_numerical': 0.04141291108404385, 'mean_word_length': 2.838006230529595, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '586527', 'n_tokens_mistral': 790, 'n_tokens_neox': 668, 'n_words': 293}