<commit_before># MonkeyKing_crawler_recommender

Description
-----------

Inspired by BitTiger's tutorials on crawler and recommender, our goal is to build one to crawl the data from xiaomi appstore.

Plan
----

Based on our experiences on web development and descriptions metioned above, we take __Febrary, 2016__ as the __1st stage__ with the __primary__ goal of __prototyping__ our own crawler and recommender following the [Development Guildlines](https://github.com/BitTigerNY/AraChat/edit/master/README.md#Development Guildlines) metioned below. Here're some tentative schedules.

* __[2016/02/05 - 2016/02/07]__ Project Selection, Plan Discussion, and Proposal Draft Writing
* __[2016/02/08 - 2016/02/24]__ System Design, Resource Discovery, Project Implementation, Document Writing 
  * crawler
    * Follow the code of Bittiger tutorial
    * Modify the code for another appstore, run it locally
    * Modify the code for server (multiple workers)
    * Deploy the code on server
  * recommender
    * Follow the code of Bittiger tutorial
    * Modify the code for another appstore, run it locally
    * Modify the code for server (multiple workers)
    * Deploy the code on server
* __[2016/02/25 - 2016/02/29]__ User Manual Writing and Video Presentation Making

_Details of each schedule and task will be added later._

Resource
--------

1. __[BitTiger Project: AppStore - Crawler]__ https://slack-files.com/T0GUEMKEZ-F0J4G9QTT-274d3bc97e
1. __[BitTiger Project: AppStore - Recommender]__ https://slack-files.com/T0GUEMKEZ-F0J4G9QTT-274d3bc97e

Language & Framework
--------------------

+ Python

Development Guildlines
----------------------

- __Modularity.__ Following the principle _"loose coupling and high cohesion"_, each module should be standalone.

- __Minimalism.__ Each module should be kept short, simple, and concise. Every piece of code should be transparent upon first reading. 
- __Easy extensibility.__ New modules (as new classes and functions) are should be simply add, and existing modules should be extended easily.

Owner
-----

@team: MonkeyKing<commit_msg>updated Language, Framework & Methodology<commit_after># MonkeyKing_crawler_recommender

Description
-----------

Inspired by BitTiger's tutorials on crawler and recommender, our goal is to build one to crawl the data from xiaomi appstore.

Plan
----

Based on our experiences on web development and descriptions metioned above, we take __Febrary, 2016__ as the __1st stage__ with the __primary__ goal of __prototyping__ our own crawler and recommender following the [Development Guildlines](https://github.com/BitTigerNY/AraChat/edit/master/README.md#Development Guildlines) metioned below. Here're some tentative schedules.

* __[2016/02/05 - 2016/02/07]__ Project Selection, Plan Discussion, and Proposal Draft Writing
* __[2016/02/08 - 2016/02/24]__ System Design, Resource Discovery, Project Implementation, Document Writing 
  * crawler
    * Follow the code of Bittiger tutorial
    * Modify the code for another appstore, run it locally
    * Modify the code for server (multiple workers)
    * Deploy the code on server
  * recommender
    * Follow the code of Bittiger tutorial
    * Modify the code for another appstore, run it locally
    * Modify the code for server (multiple workers)
    * Deploy the code on server
* __[2016/02/25 - 2016/02/29]__ User Manual Writing and Video Presentation Making

_Details of each schedule and task will be added later._

Resource
--------

1. __[BitTiger Project: AppStore - Crawler]__ https://slack-files.com/T0GUEMKEZ-F0J4G9QTT-274d3bc97e
1. __[BitTiger Project: AppStore - Recommender]__ https://slack-files.com/T0GUEMKEZ-F0J4G9QTT-274d3bc97e

Language, Framework & Methodology
--------------------

+ Python 2.7.10, and 'pip install' following:
  + pylint
  + coverage
+ Test Driven Development (TDD)

Development Guildlines
----------------------

- __Modularity.__ Following the principle _"loose coupling and high cohesion"_, each module should be standalone.

- __Minimalism.__ Each module should be kept short, simple, and concise. Every piece of code should be transparent upon first reading. 
- __Easy extensibility.__ New modules (as new classes and functions) are should be simply add, and existing modules should be extended easily.

Owner
-----

@team: MonkeyKing
{'fraction_non_alphanumeric': 0.09158186864014801, 'fraction_numerical': 0.035615171137835334, 'mean_word_length': 3.7319474835886215, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2029162', 'n_tokens_mistral': 1369, 'n_tokens_neox': 1212, 'n_words': 488}