<commit_before># INSTALLATION
  
### Below is the guidelines to setup IAM user   
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html   
https://www.youtube.com/watch?v=Z4U5ymvEvKc   
  
  
(This procedure is to make a role for EC2)  
Now create new role  
roles --> Create new role --> Amazon EC2 -->select  
Attack policy --> AmazonEC2FullAccess, AmazonDynamoDBFullAccess  
select role name and save  
  
  
(This procedure is to make a role for Lambda)  
roles --> Create new role --> Amazon Lambda -->select  
Attack policy --> AmazonLambdaFullAccess  
select role name and save  
  
  
(This procedure is to make a role for DynamoDB)  
roles --> Create new role --> Amazon Config -->select  
Attack policy --> AWSLambdaDynamoDBExecutionRole  
select role name and save  
  
  
### Below is the guidelines about how to setup the EC2.  
setup EC2 instance  
https://www.youtube.com/watch?v=M2Wc8JIS-p8  
  
for the role use the role we created at the time of building IAM user  
  
IAM commandLine  
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html  
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html  
  
to download java  
http://stackoverflow.com/questions/14788345/how-to-install-jdk-on-ubuntu-linux  
  
get the dynamodb file for your local drive for local testing  
To start dynamodb use : java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar   
  
set up virtual environment : virtualenv --python=/usr/bin/python3 myenv  
upgrade pip : pip install --upgrade pip  
install all the requirements : pip install -r requirements.txt  
  
connect to EC2  
step 1)  
```aws ec2 get-console-output --instance-id instance_id```  
  
step 2)  
```chmod 400 <full path of the pem key>___.pem```  
  
step 3)  
```ssh -i <full path of the pem key>___.pem ec2-user@<aws ec2 instance public DNS>```  
  
  
To access your instance:  
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html  
  
  
Or else which I did, use Filezilla since it is a server model it can be worked with FTP client  
  
For downloading filezilla  
http://sourceforge.net/projects/filezilla/files/FileZilla_Client/3.25.2/FileZilla_3.25.2_x86_64-linux-gnu.tar.bz2/download  
  
To execute:  
  
extract --> bin/filezilla -->run  
  
If your filezilla is not executing.  
```  
sudo add-apt-repository ppa:ubuntu-toolchain-r/test   
sudo apt-get update  
sudo apt-get upgrade  
sudo apt-get dist-upgrade
```  
  
managing ec2 inside filezilla  
http://stackoverflow.com/questions/16744863/connect-to-amazon-ec2-file-directory-using-filezilla-and-sftp  
  
Having setup your filezilla upload all the files inside EC2 and database  
1) making a directory under ec2-user in ec2 instance and upload the EC2 folder and also the database folder   
2) upload the requirements.txt inside this folder in ec2  
  
now come back to command line:(I hope aws cli is configured, credentials are set, ec2 is running in command line as well in filezilla)   
```  
cd EC2

#returns the current version of the python
which python

#install python3.4 inside ec2  						  
sudo yum install python34

#make a virtual environment   			
virtualenv -p python3.4 myenv

#activate the virtual environment  		
source myenv/bin/activate	

pip install upgrade pip

#install the requirements  
pip install -r requirement.txt		

#to get the path of the current working directory  
pwd									
```  
  
having done all the requirement installed now lets set the cron job  
  
```  
#to check if any cronjob is set or not  
crontab -l

# this step is optional if you see the path is different then set the path accordingly and update the task.cron   								
nano task.cron

#add task to cron   							
crontab task.cron

#you can see the task is added to cron   						
crontab -l 								
```  
  
At last set whenever ec2 loads the virtual environment also loads  
```  
echo "source /home/ec2-user/EC2/myenv/bin/activate" >> /home/ec2-user/.bashrc  
```

____


# PROBLEM FACED  <commit_msg>updating EC2/README<commit_after># INSTALLATION
  
### Below is the guidelines to setup IAM user   
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html   
https://www.youtube.com/watch?v=Z4U5ymvEvKc   
  
  
(This procedure is to make a role for EC2)  
Now create new role  
roles --> Create new role --> Amazon EC2 -->select  
Attack policy --> AmazonEC2FullAccess, AmazonDynamoDBFullAccess  
select role name and save  
  
  
(This procedure is to make a role for Lambda)  
roles --> Create new role --> Amazon Lambda -->select  
Attack policy --> AmazonLambdaFullAccess  
select role name and save  
  
  
(This procedure is to make a role for DynamoDB)  
roles --> Create new role --> Amazon Config -->select  
Attack policy --> AWSLambdaDynamoDBExecutionRole  
select role name and save  
  
  
### Below is the guidelines about how to setup the EC2.  
setup EC2 instance  
https://www.youtube.com/watch?v=M2Wc8JIS-p8  
  
for the role use the role we created at the time of building IAM user  
  
IAM commandLine  
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html  
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html  
  
to download java  
http://stackoverflow.com/questions/14788345/how-to-install-jdk-on-ubuntu-linux  
  
get the dynamodb file for your local drive for local testing  
To start dynamodb use : java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar   
  
set up virtual environment : virtualenv --python=/usr/bin/python3 myenv  
upgrade pip : pip install --upgrade pip  
install all the requirements : pip install -r requirements.txt  
  
connect to EC2  
step 1)  
```aws ec2 get-console-output --instance-id instance_id```  
  
step 2)  
```chmod 400 <full path of the pem key>___.pem```  
  
step 3)  
```ssh -i <full path of the pem key>___.pem ec2-user@<aws ec2 instance public DNS>```  
  
  
To access your instance:  
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html  
  
  
Or else which I did, use Filezilla since it is a server model it can be worked with FTP client  
  
For downloading filezilla  
http://sourceforge.net/projects/filezilla/files/FileZilla_Client/3.25.2/FileZilla_3.25.2_x86_64-linux-gnu.tar.bz2/download  
  
To execute:  
  
extract --> bin/filezilla -->run  
  
If your filezilla is not executing.  
```  
sudo add-apt-repository ppa:ubuntu-toolchain-r/test   
sudo apt-get update  
sudo apt-get upgrade  
sudo apt-get dist-upgrade
```  
  
managing ec2 inside filezilla  
http://stackoverflow.com/questions/16744863/connect-to-amazon-ec2-file-directory-using-filezilla-and-sftp  
  
Having setup your filezilla upload all the files inside EC2 and database  
1) making a directory under ec2-user in ec2 instance and upload the EC2 folder and also the database folder   
2) upload the requirements.txt inside this folder in ec2  
  
now come back to command line:(I hope aws cli is configured, credentials are set, ec2 is running in command line as well in filezilla)   
```  
cd EC2

#returns the current version of the python
which python

#install python3.4 inside ec2  						  
sudo yum install python34

#make a virtual environment   			
virtualenv -p python3.4 myenv

#activate the virtual environment  		
source myenv/bin/activate	

pip install upgrade pip

#install the requirements  
pip install -r requirement.txt		

#to get the path of the current working directory  
pwd									
```  
  
having done all the requirement installed now lets set the cron job  
  
```  
#to check if any cronjob is set or not  
crontab -l

# this step is optional if you see the path is different then set the path accordingly and update the task.cron   								
nano task.cron

#add task to cron   							
crontab task.cron

#you can see the task is added to cron   						
crontab -l 								
```  
  
At last set whenever ec2 loads the virtual environment also loads  
```  
echo "source /home/ec2-user/EC2/myenv/bin/activate" >> /home/ec2-user/.bashrc  
```
 
**testing offine**  

* first start the local DynamoDb
* set DEPLOY = False, Second run the index.py.
This will trigger the local database and also it will print all the steps it is taking , so all the print statements are hidden inside the DEPLOY variable and is spread over the full project. So that only one variable can decide from the start whether its offline development or online.

____


# PROBLEM FACED  
{'fraction_non_alphanumeric': 0.0775303405207965, 'fraction_numerical': 0.017320607988688584, 'mean_word_length': 3.0945489628557645, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 41, 'https://': 6, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2701361', 'n_tokens_mistral': 2901, 'n_tokens_neox': 2643, 'n_words': 976}