<commit_before>serapis-web
===========

##How to run the project - (on Ubuntu)

1. Install the following:
                                             
        sudo apt-get install python-pip python-dev build-essential
        sudo pip install --upgrade pip
        sudo pip install cython amqp amqplib celery distribute django-celery mongoengine pymongo simplejson
        sudo apt-get install mongodb-server
        sudo apt-get install rabbitmq-server
<<<<<<< HEAD
	sudo apt-get install curl
        sudo pip install pycurl
        sudo pip install --upgrade pycurl
=======
>>>>>>> 7c4f154f19369c0d6c38511430433352b9d9a411

2. You need a settings.py file - maybe it would be better to use the one I have, for now, and modify the absolute paths from it, with paths on your machine (will fix this in the future).

3. Start a Celery - worker on your machine:

<<<<<<< HEAD
        python manage.py celery worker --loglevel=info

4. Start the Django server:

        python manage.py runserver --noreload
=======
	python manage.py celery worker --loglevel=info

4. Start the Django server:

	python manage.py runserver --noreload
>>>>>>> 7c4f154f19369c0d6c38511430433352b9d9a411

 // you can start it also without to --noreload, but from my experience, the automatic restart of the server didn't work so well. 

5. Open http://127.0.0.1:8000/serapis/upload/ in your browser.



##The functionality - roughly:


This is how things work in Django and Celery according to my understanding.

Django manages the frontend (view) and communicates with the database (the model), which has been given in settings.py. Celery is in charge of managing the execution of tasks that come from the Django (distribute the tasks to workers and gather the results from the workers back), using RabbitMQ as its broker.

Browser -> web server (which acts like the Celery-server client) -> saves data in the MongoDB and submits further tasks to be executed remotely (or not) by workers


- serapis/urls.py - contains url() functions called with 3 parameters:
		* regex to match the url from request
		* the name of the function (from views.py file or view_classes.py file) to be called when a request for that url comes in
		* (optional) name of that url chosen (to distinguish easier between different urls).

- views.py - contains functions that get as parameter a request and decide what to do with it further and what HttpResponse to return
- views_classes.py - has the same "logical" purpose as views.py, the only difference is that the functions to be called are encapsulated within classes (the framework offers both options - views as functions or as classes, the developer chooses which one he likes
		- each class has a 
				* template_name = field which tells which one is the corresponding html page for that view
				* form_class = field that tells which one is the corresponding form that gets initialized from that html-page-form
				* success_url = url to be used when/if the request is successful



- forms.py - contains forms as classes (that inherit forms.Form)
	 - the fields declared in this kind of classes correspond to fields from the  html page form

- tasks.py - file containing the tasks (defined as functions) to be queued and executed on other machines by the workers (workers = processes/threads started on one or more machines, listening to tasks 'coming' from RabbitMQ queues).

- models.py - Database model used in the application. Each python class defined here corresponds to a document or to an EmbeddedDocument ( a document embedded within another document). I tried to model the data in such a way, that the fields, most of them reside physically in the same document, but to organise them logically in separate entities: the actual document inherits from DynamicDocument (mongoengine class), while the nested types inherit from DynamicEmbeddedDocument. I thought this might ease the handling of the data from the html forms.

<<<<<<< HEAD
- serapis/templates/serapis - folder containing all the html pages. I have created a `base.html` class with the elements common for all the pages (e.g. header and footer) => all the `.html` files that I wish to contain these common elements should extend `base.html` page - by having on the first line of the file:
=======
- serapis/templates/serapis - folder containing all the html pages. I have created a base.html class with the elements common for all the pages (e.g. header and footer) => all the .html files that I wish to contain these common elements should extend **base.html** page - by having on the first line of the file:
>>>>>>> 7c4f154f19369c0d6c38511430433352b9d9a411

	{% extends "base.html" %}

In **base.html** one can also declare other blocks that are to be implemented in the html pages that extend the base.
	e.g.:
	{% block content %}
	<!-- Here comes some html...-->
    	{% endblock %}

###Flow:

Request -> url searched within urls.py file -> when an entry from urls.py matches, the corresponding view (function or class) is called, which eventually initializes a form with the data submitted from an html form.


##FAQ


1.    Do I need to have a web-server installed on the machine, where I want to run the project?

    **A:** The answer is: depends if you wish to run the project for development or production purposes. Django comes with a web-server, but that is purely for testing-development purposes. If you want to run the project in production, you should search for a different web-server.

2.    Why do I need class-based views? Why would I want to have a whole class for each view, when I could have only one function?<commit_msg>ReadMe fixed<commit_after>serapis-web
===========

##How to run the project - (on Ubuntu)

1. Install the following:
                                             
        sudo apt-get install python-pip python-dev build-essential
        sudo pip install --upgrade pip
        sudo pip install cython amqp amqplib celery distribute django-celery mongoengine pymongo simplejson
        sudo apt-get install mongodb-server
        sudo apt-get install rabbitmq-server
        sudo apt-get install curl
        sudo pip install pycurl
        sudo pip install --upgrade pycurl


2. You need a settings.py file - maybe it would be better to use the one I have, for now, and modify the absolute paths from it, with paths on your machine (will fix this in the future).

3. Start a Celery - worker on your machine:

        python manage.py celery worker --loglevel=info

4. Start the Django server:

        python manage.py runserver --noreload

 // you can start it also without to --noreload, but from my experience, the automatic restart of the server didn't work so well. 

5. Open http://127.0.0.1:8000/serapis/upload/ in your browser.



##The functionality - roughly:


This is how things work in Django and Celery according to my understanding.

Django manages the frontend (view) and communicates with the database (the model), which has been given in settings.py. Celery is in charge of managing the execution of tasks that come from the Django (distribute the tasks to workers and gather the results from the workers back), using RabbitMQ as its broker.

Browser -> web server (which acts like the Celery-server client) -> saves data in the MongoDB and submits further tasks to be executed remotely (or not) by workers


- serapis/urls.py - contains url() functions called with 3 parameters:
		* regex to match the url from request
		* the name of the function (from views.py file or view_classes.py file) to be called when a request for that url comes in
		* (optional) name of that url chosen (to distinguish easier between different urls).

- views.py - contains functions that get as parameter a request and decide what to do with it further and what HttpResponse to return
- views_classes.py - has the same "logical" purpose as views.py, the only difference is that the functions to be called are encapsulated within classes (the framework offers both options - views as functions or as classes, the developer chooses which one he likes
		- each class has a 
				* template_name = field which tells which one is the corresponding html page for that view
				* form_class = field that tells which one is the corresponding form that gets initialized from that html-page-form
				* success_url = url to be used when/if the request is successful



- forms.py - contains forms as classes (that inherit forms.Form)
	 - the fields declared in this kind of classes correspond to fields from the  html page form

- tasks.py - file containing the tasks (defined as functions) to be queued and executed on other machines by the workers (workers = processes/threads started on one or more machines, listening to tasks 'coming' from RabbitMQ queues).

- models.py - Database model used in the application. Each python class defined here corresponds to a document or to an EmbeddedDocument ( a document embedded within another document). I tried to model the data in such a way, that the fields, most of them reside physically in the same document, but to organise them logically in separate entities: the actual document inherits from DynamicDocument (mongoengine class), while the nested types inherit from DynamicEmbeddedDocument. I thought this might ease the handling of the data from the html forms.


- serapis/templates/serapis - folder containing all the html pages. I have created a base.html class with the elements common for all the pages (e.g. header and footer) => all the .html files that I wish to contain these common elements should extend **base.html** page - by having on the first line of the file:

	{% extends "base.html" %}

In **base.html** one can also declare other blocks that are to be implemented in the html pages that extend the base.
	e.g.:
	{% block content %}
	<!-- Here comes some html...-->
    	{% endblock %}

###Flow:

Request -> url searched within urls.py file -> when an entry from urls.py matches, the corresponding view (function or class) is called, which eventually initializes a form with the data submitted from an html form.


##FAQ


1.    Do I need to have a web-server installed on the machine, where I want to run the project?

    **A:** The answer is: depends if you wish to run the project for development or production purposes. Django comes with a web-server, but that is purely for testing-development purposes. If you want to run the project in production, you should search for a different web-server.

2.    Why do I need class-based views? Why would I want to have a whole class for each view, when I could have only one function?
{'fraction_non_alphanumeric': 0.060451977401129946, 'fraction_numerical': 0.01224105461393597, 'mean_word_length': 3.5388888888888888, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2154291', 'n_tokens_mistral': 2824, 'n_tokens_neox': 2586, 'n_words': 1550}