<commit_before>Definition
-----------
this module contains four kinds of files:
* crawler: it scrapes RSS/ATOM alike published content; legal web pages like
  WAP pages; APIs. Data are not differentiated as RSS, web pages or pictures,
videos and etc.

* database handler: mostly deals with permanently storing scraped data on to
  one or two dedicated database server.

* memory handler: data in memory could be provided by crawlers or automatically
  generated from database.

* feature interface: provides callers so that server routine program could
  regularly asked it to scrape data.

ToDos
------
1. extract image-processing methods
2. break down database storing method
3. integrate api scraping
4. extract alert-maintenance-related code from task
5. add newsbeuter-like mechanism to mitigate pressure on rss servers<commit_msg>added feed database stats collector to todo list<commit_after>Definition
-----------
this module contains four kinds of files:
* crawler: it scrapes RSS/ATOM alike published content; legal web pages like
  WAP pages; APIs. Data are not differentiated as RSS, web pages or pictures,
videos and etc.

* database handler: mostly deals with permanently storing scraped data on to
  one or two dedicated database server.

* memory handler: data in memory could be provided by crawlers or automatically
  generated from database.

* feature interface: provides callers so that server routine program could
  regularly asked it to scrape data.

ToDos
------
1. extract image-processing methods
2. break down database storing method
3. integrate api scraping
4. extract alert-maintenance-related code from task
5. add newsbeuter-like mechanism to mitigate pressure on rss servers
6. add feed table/database statistics collector
{'fraction_non_alphanumeric': 0.05584045584045584, 'fraction_numerical': 0.0062678062678062675, 'mean_word_length': 4.1345029239766085, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4847278', 'n_tokens_mistral': 434, 'n_tokens_neox': 413, 'n_words': 240}