<commit_before>scrapy-top
==========

simple tools like linux top command to view scrapy status

###Requests
1. telnetlib default in python distribution
2. prettytable https://code.google.com/p/prettytable/

###Run
For now, you can python scrapy-top -i 2 -h localhost -p 6023

1. -i: interval seconds, defaut 2 secs
2. -h: host, default localhost
3. -p: port scrapy default telnet port 6023<commit_msg>update description<commit_after>scrapy-top
==========

simple tools like linux top command to view scrapy status

###Requests
1. telnetlib default in python distribution
2. prettytable https://code.google.com/p/prettytable/

###Run
For now, you can python scrapy-top -i 2 -h localhost -p 6023

1. -i: interval seconds, defaut 2 secs
2. -h: host, default localhost
3. -p: port scrapy default telnet port 6023

###Output
+-------------------------------------+----------------------------+
|                 Key                 |           Value            |
+-------------------------------------+----------------------------+
| log_count/DEBUG                     | 27                         |
| request_depth_max                   | 4                          |
| start_time                          | 2014-04-01 02:12:42.619696 |
| log_count/INFO                      | 6                          |
| downloader/request_method_count/GET | 28                         |
| scheduler/dequeued/redis            | 28                         |
| downloader/request_bytes            | 12160                      |
+-------------------------------------+----------------------------+

{'fraction_non_alphanumeric': 0.2120253164556962, 'fraction_numerical': 0.039873417721518985, 'mean_word_length': 1.220505617977528, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6753669', 'n_tokens_mistral': 444, 'n_tokens_neox': 382, 'n_words': 126}