<commit_before># Lexos Roadmap (DRAFT)

The following roadmap incorporates most of the issues in GitHub and the projected developments in our grant proposals. As a rough timetable, it seems like we could get to v3.5 by midsummer 2017.

## v3.1. Minor Updates and Bug Fixes

### Already Done:
1.	Scrubber stop word radio button settings were made persistent ([#260](https://github.com/WheatonCS/Lexos/issues/260)).
2.	Labeling of duplicate original sources in Manage fixed ([#356](https://github.com/WheatonCS/Lexos/issues/356)).
3.	MUFI display tables were proofread ([#458](https://github.com/WheatonCS/Lexos/issues/458)) and the UI for reading the tables was enhanced with Ajax loading and sort, search, and paging capabilities.
4.	Possible errors caused by empty documents generated by Cutter were prevented by trimming documents of leading and ending white space and milestones ([#471](https://github.com/WheatonCS/Lexos/issues/471)).
5.	In Cutter, the “Cut documents on this term” field is now hidden on page load ([#486](https://github.com/WheatonCS/Lexos/issues/486)).
6.	Scalar’s internal links in “In the Margins” are now converted to full urls opening in a new window in Lexos ([#487](https://github.com/WheatonCS/Lexos/issues/487)).
7.	Multicloud improvements: (a) form submission converted to Ajax; (b) warning dialog added to tell user about browser freezes; (c) wordCounts array generated server-side for better performance; (d) coerce MALLET data to UTF-8 ([#498](https://github.com/WheatonCS/Lexos/issues/498)).
8.	A download link was added to the Manage tool.
9.	Scrubber and Cutter previews now display the name of the document label instead of the source file.
10. Improvements made to the `Delete Selected`, `Merge Selected Documents`, and `Apply Class to Selected Documents` functions in Maange.

### To Do:
1.	Finish the improved Windows installer ([#490](https://github.com/WheatonCS/Lexos/issues/490)) (almost done).

## v3.2. Minor Enhancements, Code Cleanup, and Workflow Improvements
1.  Migrate to python3 ([#434](https://github.com/WheatonCS/Lexos/issues/434)).
2.  Scrubber hyphen and consolidation enhancements ([#438](https://github.com/WheatonCS/Lexos/issues/438)).
3.	Handle digits more robustly in Scrubber. ([#460](https://github.com/WheatonCS/Lexos/issues/460))
4.	Migrate all remaining form submissions to Ajax ([#494](https://github.com/WheatonCS/Lexos/issues/494)).
5.	Enable the user to select how to replace markup elements in Scrubber. ([#457](https://github.com/WheatonCS/Lexos/issues/457))
6.	Adopt continuous integration for the workflow ([#497](https://github.com/WheatonCS/Lexos/issues/497)).
7.	Move dev docs and possibly installation materials to the Wiki or another repo ([#489](https://github.com/WheatonCS/Lexos/issues/489)).
8.	Add document label truncation where necessary ([#501](https://github.com/WheatonCS/Lexos/issues/501)).

## v3.3. Code Cleanup, and Workflow Improvements
1.	Implement webassets ([#485](https://github.com/WheatonCS/Lexos/issues/485)).
2.	Introduce error handling Ajax responses and other server errors ([#495](https://github.com/WheatonCS/Lexos/issues/495)).
3.	Refactor Javascript ([#496](https://github.com/WheatonCS/Lexos/issues/496)).

## v3.4. Re-Enable Lost Functionality
1.	Re-implement TF-IDF in Normalize options ([#444](https://github.com/WheatonCS/Lexos/issues/444)).
2.	Re-implement commented out silhouette score settings ([#478](https://github.com/WheatonCS/Lexos/issues/478)).

## v3.5. Migration to Python 3 and Preparations for New Functions
1.  Add more averages to statistics table ([#255](https://github.com/WheatonCS/Lexos/issues/255)).
2.	Move Python 3 branch to master (see below for Python 3 notes).<commit_msg>Fixed typo<commit_after># Lexos Roadmap (DRAFT)

The following roadmap incorporates most of the issues in GitHub and the projected developments in our grant proposals. As a rough timetable, it seems like we could get to v3.5 by midsummer 2017.

## v3.1. Minor Updates and Bug Fixes

### Already Done:
1.	Scrubber stop word radio button settings were made persistent ([#260](https://github.com/WheatonCS/Lexos/issues/260)).
2.	Labeling of duplicate original sources in Manage fixed ([#356](https://github.com/WheatonCS/Lexos/issues/356)).
3.	MUFI display tables were proofread ([#458](https://github.com/WheatonCS/Lexos/issues/458)) and the UI for reading the tables was enhanced with Ajax loading and sort, search, and paging capabilities.
4.	Possible errors caused by empty documents generated by Cutter were prevented by trimming documents of leading and ending white space and milestones ([#471](https://github.com/WheatonCS/Lexos/issues/471)).
5.	In Cutter, the “Cut documents on this term” field is now hidden on page load ([#486](https://github.com/WheatonCS/Lexos/issues/486)).
6.	Scalar’s internal links in “In the Margins” are now converted to full urls opening in a new window in Lexos ([#487](https://github.com/WheatonCS/Lexos/issues/487)).
7.	Multicloud improvements: (a) form submission converted to Ajax; (b) warning dialog added to tell user about browser freezes; (c) wordCounts array generated server-side for better performance; (d) coerce MALLET data to UTF-8 ([#498](https://github.com/WheatonCS/Lexos/issues/498)).
8.	A download link was added to the Manage tool.
9.	Scrubber and Cutter previews now display the name of the document label instead of the source file.
10. Improvements made to the `Delete Selected`, `Merge Selected Documents`, and `Apply Class to Selected Documents` functions in Manage.

### To Do:
1.	Finish the improved Windows installer ([#490](https://github.com/WheatonCS/Lexos/issues/490)) (almost done).

## v3.2. Minor Enhancements, Code Cleanup, and Workflow Improvements
1.  Migrate to python3 ([#434](https://github.com/WheatonCS/Lexos/issues/434)).
2.  Scrubber hyphen and consolidation enhancements ([#438](https://github.com/WheatonCS/Lexos/issues/438)).
3.	Handle digits more robustly in Scrubber. ([#460](https://github.com/WheatonCS/Lexos/issues/460))
4.	Migrate all remaining form submissions to Ajax ([#494](https://github.com/WheatonCS/Lexos/issues/494)).
5.	Enable the user to select how to replace markup elements in Scrubber. ([#457](https://github.com/WheatonCS/Lexos/issues/457))
6.	Adopt continuous integration for the workflow ([#497](https://github.com/WheatonCS/Lexos/issues/497)).
7.	Move dev docs and possibly installation materials to the Wiki or another repo ([#489](https://github.com/WheatonCS/Lexos/issues/489)).
8.	Add document label truncation where necessary ([#501](https://github.com/WheatonCS/Lexos/issues/501)).

## v3.3. Code Cleanup, and Workflow Improvements
1.	Implement webassets ([#485](https://github.com/WheatonCS/Lexos/issues/485)).
2.	Introduce error handling Ajax responses and other server errors ([#495](https://github.com/WheatonCS/Lexos/issues/495)).
3.	Refactor Javascript ([#496](https://github.com/WheatonCS/Lexos/issues/496)).

## v3.4. Re-Enable Lost Functionality
1.	Re-implement TF-IDF in Normalize options ([#444](https://github.com/WheatonCS/Lexos/issues/444)).
2.	Re-implement commented out silhouette score settings ([#478](https://github.com/WheatonCS/Lexos/issues/478)).

## v3.5. Migration to Python 3 and Preparations for New Functions
1.  Add more averages to statistics table ([#255](https://github.com/WheatonCS/Lexos/issues/255)).
2.	Move Python 3 branch to master (see below for Python 3 notes).
{'fraction_non_alphanumeric': 0.1241416453480544, 'fraction_numerical': 0.0484717921098694, 'mean_word_length': 4.74922600619195, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 44, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2162065', 'n_tokens_mistral': 2575, 'n_tokens_neox': 2216, 'n_words': 806}