<commit_before>hail-seizure
============

Submission for Kaggle's American Epilepsy Society Seizure Prediction Challenge

http://www.kaggle.com/c/seizure-detection

This README and repository modelled on https://www.kaggle.com/wiki/ModelSubmissionBestPractices

##Hardware / OS platform used
 * Various servers owned by Edinburgh University Informatics Department: 
      - 64 AMD Opteron cores, 256GB RAM, 4TB disk 
      - Scientific Linux
 * Various mid-high end desktops and laptops:
      - Intel processors (i3 and Xeons), 8-64GB RAM, 0.5-8TB disk
      - Arch Linux

##Dependencies

###Required

 * Python 3.4.1
 * scikit_learn-0.15.2
 * numpy-1.8.1
 * scipy
 * h5py

##Generate features

Place path to raw data organised by subject under the RAW_DATA_DIRS key of SETTINGS.json and check the values used in the SETTINGS.json
```
RAW_DATA_DIR/
  Dog_1/
    Dog_1_ictal_segment_1.mat
    Dog_1_ictal_segment_2.mat
    ...
    Dog_1_interictal_segment_1.mat
    Dog_1_interictal_segment_2.mat
    ...
    Dog_1_test_segment_1.mat
    Dog_1_test_segment_2.mat
    ...

  Dog_2/
  ...
```


Then run:
```
./preprocessing.m
```

This will calculate features used the feature functions specified in SETTINGS.json FEATURES field and output them to TRAIN_DATA_PATH directory as HDF5 files.

HDF5 structure: 

```
$feature_name.h5 = {$subject: {$type : {$segment_file_name : $feature_vector } } }

```
<commit_msg>Fix README to say how to run preprocessing.m<commit_after>hail-seizure
============

Submission for Kaggle's American Epilepsy Society Seizure Prediction Challenge

http://www.kaggle.com/c/seizure-detection

This README and repository modelled on https://www.kaggle.com/wiki/ModelSubmissionBestPractices

##Hardware / OS platform used
 * Various servers owned by Edinburgh University Informatics Department: 
      - 64 AMD Opteron cores, 256GB RAM, 4TB disk 
      - Scientific Linux
 * Various mid-high end desktops and laptops:
      - Intel processors (i3 and Xeons), 8-64GB RAM, 0.5-8TB disk
      - Arch Linux

##Dependencies

###Required

 * Python 3.4.1
 * scikit_learn-0.15.2
 * numpy-1.8.1
 * scipy
 * h5py

##Generate features

Place path to raw data organised by subject under the RAW_DATA_DIRS key of SETTINGS.json and check the values used in the SETTINGS.json
```
RAW_DATA_DIR/
  Dog_1/
    Dog_1_ictal_segment_1.mat
    Dog_1_ictal_segment_2.mat
    ...
    Dog_1_interictal_segment_1.mat
    Dog_1_interictal_segment_2.mat
    ...
    Dog_1_test_segment_1.mat
    Dog_1_test_segment_2.mat
    ...

  Dog_2/
  ...
```


Then run `./preprocessing.m` with:
```
matlab -nodisplay -nosplash -r "preprocessing"
```
or similar.

This will calculate features used the feature functions specified in SETTINGS.json FEATURES field and output them to TRAIN_DATA_PATH directory as HDF5 files.

HDF5 structure: 

```
$feature_name.h5 = {$subject: {$type : {$segment_file_name : $feature_vector } } }

```

{'fraction_non_alphanumeric': 0.09695101062007537, 'fraction_numerical': 0.028091812264474134, 'mean_word_length': 3.541213063763608, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3150815', 'n_tokens_mistral': 1065, 'n_tokens_neox': 984, 'n_words': 275}