<commit_before>self-contained curriculum (test)
===============

ec2-based curriculum server test

Stack:

Nginx - listens to the world on port 80, load balances (but not much for us, since we only have one tornado instance running)
/etc/nginx/nginx.conf - this holds the mapping from "listen on" to "send traffic to X servers"

Upstart - systemd-like "make sure main apps are running and if not, restart them". Need to document more config stuff for this.

Redis - the local curriculum implementation uses Redis for storing chunks of language. Not sure how to port over timestamps (for date-range API queries - necessary?) or do authentication (separate DB seems easiest)
* redis "0" - language fragments, probably simple K:V or List. TTL should be 7 days or so
* redis "1" - list of URLS we have processed quite recently - TTL here should be ~1 hour. We check against this before re-fetching the page to do any NLP+chunking
* redis "2" - authentication DB? Populated at boot form a static groups.py-like file?

Beanstalk - handles the queue of URLS-to-analyze from SubmitHandler

/server/server.py - this is the main tornado instance.
* ApiHandler.py - services API queries. 
* BaseHandler.py - just a superclass contining the isAuth() function
* BrowserHandler.py - services web queries, currently only ZEN mode (this would be the prototype for other views)<commit_msg>docu cleanup<commit_after>self-contained curriculum (test)
===============

ec2-based curriculum server test

Stack:

Nginx - listens to the world on port 80, load balances (but not much for us, since we only have one tornado instance running)
/etc/nginx/nginx.conf - this holds the mapping from "listen on" to "send traffic to X servers"

Upstart - systemd-like "make sure main apps are running and if not, restart them". Need to document more config stuff for this.

Redis - the local curriculum implementation uses Redis for storing chunks of language. Not sure how to port over timestamps (for date-range API queries - necessary?) or do authentication (separate DB seems easiest)
* redis "0" - language fragments, probably simple K:V or List. TTL should be 7 days or so
* redis "1" - list of URLS we have processed quite recently - TTL here should be ~1 hour. We check against this before re-fetching the page to do any NLP+chunking
* redis "2" - authentication DB? Populated at boot form a static groups.py-like file?

Beanstalk - handles the queue of URLS-to-analyze from SubmitHandler

/worker.py - this takes jobs form Beanstalk, checks Redis.1 to see if it should be resolved, and resolves it into chunks that are loaded into Redis.0 (HEY, how are we going to keep track of groupIDs here?)

/server/server.py - this is the main tornado instance.
* ApiHandler.py - services API queries. 
* BaseHandler.py - just a superclass contining the isAuth() function
* BrowserHandler.py - services web queries, currently only ZEN mode (this would be the prototype for other views)
{'fraction_non_alphanumeric': 0.07152542372881356, 'fraction_numerical': 0.006101694915254237, 'mean_word_length': 4.150087260034904, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3761792', 'n_tokens_mistral': 805, 'n_tokens_neox': 780, 'n_words': 428}