<commit_before># deduper (**work in progress**)
Python file deduplication utility with an AngularJS front-end

## TODO
|feature|details|expected effort|
|-------|-------|-------|
|Messages|Create/Display messages in a uniform fashion|small|
|Thumbnails|Better UI representation for duplicates|medium|
|Unit Tests|Validate duplicate detection and stats|medium|
|e2e Tests|Validate UI functionality|medium|
|Dedup 1: delete|Support actually deleting all duplicates|medium|
|Dedup 2: reorg|Support copying all files to a new folder structure based on dates|medium|
|Dedup 3: quarantine|Support moving only duplicates to quarantine folder|medium|
|Minify|Compile/minify the javascript for efficient loading|small|

## Completed
|feature|details|date|
|-------|-------|-------|
|Custom paths|Add/Remove/Edit folder search paths|2/20/2016|

## dependencies
|library| usage | site | version
|-------|------|-------|---------|
|CherryPy | simple http server| http://cherrypy.org| 5.0.1 |
|angularjs|ux framework|https://angularjs.org|1.5.0|
|ui-grid | grid ui | http://ui-grid.info | 3.1.1 |
|bootstrap|CSS formatting done easy|http://getbootstrap.com|3.3.6|
|underscorejs|Useful JS utilities|http://underscorejs.org|1.8.3|
|async|Async pattern utilities for JS|https://github.com/caolan/async|1.5.2|<commit_msg>Added more TODO's and compatibility to readme.<commit_after># deduper (**work in progress**)
Python file deduplication utility with an AngularJS front-end

## compatibility
Currently only tested/enabled on Windows with Python 2.7.11+

## usage
1. clone from github repo
2. navigate to root of repo
3. npm start

## TODO
|feature|details|expected effort|
|-------|-------|-------|
|Setup|Determine all dependencies for clean-machine install|medium|
|Python 3.x|Verify compatibility with Python 3.x|small|
|Messages|Create/Display messages in a uniform fashion|small|
|Thumbnails|Better UI representation for duplicates|medium|
|Unit Tests|Validate duplicate detection and stats|medium|
|e2e Tests|Validate UI functionality|medium|
|Dedup 1: delete|Support actually deleting all duplicates|medium|
|Dedup 2: reorg|Support copying all files to a new folder structure based on dates|medium|
|Dedup 3: quarantine|Support moving only duplicates to quarantine folder|medium|
|Minify|Compile/minify the javascript for efficient loading|small|

## Completed
|feature|details|date|
|-------|-------|-------|
|Custom paths|Add/Remove/Edit folder search paths|2/20/2016|

## dependencies
|library| usage | site | version
|-------|------|-------|---------|
|CherryPy | simple http server| http://cherrypy.org| 5.0.1 |
|angularjs|ux framework|https://angularjs.org|1.5.0|
|ui-grid | grid ui | http://ui-grid.info | 3.1.1 |
|bootstrap|CSS formatting done easy|http://getbootstrap.com|3.3.6|
|underscorejs|Useful JS utilities|http://underscorejs.org|1.8.3|
|async|Async pattern utilities for JS|https://github.com/caolan/async|1.5.2|
{'fraction_non_alphanumeric': 0.16432728517630948, 'fraction_numerical': 0.02293735022252653, 'mean_word_length': 4.48217636022514, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4240021', 'n_tokens_mistral': 939, 'n_tokens_neox': 870, 'n_words': 261}