<jupyter_start><jupyter_text>Web Scraper Execution Notebook**BLUF:** THis notebook imports the relevant python scraping scripts from github, builds the envirnoment, executes the scripts to scrape a selected website's files, saves the data locally, allows a user to export the data, then performs basic NLP on the content of the files to rank the files.* [x] The target websites are located here [Link to website list](https://drive.google.com/file/d/1y8FL1rBu8yb2HEn7-ok-ILy1Wq3UrDZU/view?usp=sharing).  Note: website list restricted, for access see the author.* [x] The desired search terms are stored in this file [Link to search terms list](https://drive.google.com/file/d/18MyoEiAYTG6w6JRejv5d4qVK4RxYyRu2/view?usp=sharing).  Note: file access restricted, for access see the author.**Problem Statement:**Builds a tool to scrape websites and then analyze the information to see if it contains key terms.  Provide the analyst with a rank-ordered list of documents that require human review. 1. [x] Construct a Python scripts that can be used to scrape the pdfs from a set of [websites](https://drive.google.com/file/d/1y8FL1rBu8yb2HEn7-ok-ILy1Wq3UrDZU/view?usp=sharing).  2. [x] Construct a python Notebook that executes the python script and then performs natural language processing (NLP) to the scrapped data.    - [x] 2.1. Build the initial section that executes the python scripts.  - [x] 2.2. Rank order the scrapped results by relevance to the analyist. The ranking will be based on the [list of key terms](https://drive.google.com/file/d/18MyoEiAYTG6w6JRejv5d4qVK4RxYyRu2/view?usp=sharing).  - [x] 2.3. include an ability to automate the translation of a selected file and perform sentiment analysisportions of the scraped content.  - [x] 2.4. Provide the user with the ability to save the scrapped content.- Other possible things:   - [x] The code should allow the list of keywords to change over time.    -  The system  should immediately flag certain keywords for human review regardless of other content in the document.    -  The system should be able to retrieve and analyze office documents (PDFs,spreadsheets, and word processor documents).   - [x] The system should perform sentiment analysis of selected content.  Verify a clean environment, and that external IP is different from your own. Check yours [here](https://www.google.com/search?q=whatmyip).<jupyter_code>import socket
import subprocess

hostname = socket.gethostname()
internal_ip = socket.gethostbyname(hostname)
external_ip = subprocess.check_output('curl ifconfig.me', shell=True)
print('~'*35)
print("Colab VM Hostname: {}".format(hostname))
print("Colab Internal IP: {}".format(internal_ip))
print('Colab External IP: {}'.format(external_ip.decode('utf-8')))
print('~'*35)<jupyter_output><empty_output><jupyter_text>Prepare the environment 0. git the repo and install all dependancies1. Install any dependancies2. Load the list of search sites and search terms<jupyter_code>import os
## 0. Clone the repo and then change the working directory to the cloned folder
root_folder = '/content' # Root folder for storing the files
git_repo =  '698S_BOG'  # Repo we want to clone
git_user = 'PurpleDin0' # User/org we want to clone the repo from
os.chdir(root_folder)
!git clone https://github.com/{git_user}/{git_repo}.git
repo_dir = os.path.join(os.getcwd(), git_repo)
os.chdir(repo_dir)

## 1. Install the required libraries
!pip install -r requirements.txt

# Caution: Colab may tell you that you need to restart, 
# But as of 22 June 2020 you don't have to, the code still works even with the different versions
## 2. Load the list of search sites and search terms

# Import PyDrive and associated libraries.
# This only needs to be done once per notebook.
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
import pandas as pd


# Authenticate and create the PyDrive client.
# This only needs to be done once per notebook.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

# Download a file based on its file ID.
# A file ID looks like: laggVyWshwcyP6kEI-y_W3P8D26sz
# 
# Get the website list file and load it to a dataframe
# Website list file is a csv with the columns [Website, Domain, FQD, notes]
file_id = '1y8FL1rBu8yb2HEn7-ok-ILy1Wq3UrDZU'  ## USER UPDATEABLE VALUE! ##
downloaded = drive.CreateFile({'id': file_id})
downloaded.FetchMetadata()
filename = downloaded.metadata['originalFilename']
downloaded.GetContentFile(filename)
website_list_df = pd.read_csv(filename)
# Get the search term file and load it to a dataframe
# Search term file is a csv with columns [Importance,Term_(foreign),Term_(english)]
file_id = '18MyoEiAYTG6w6JRejv5d4qVK4RxYyRu2' ## USER UPDATEABLE VALUE! ##
downloaded = drive.CreateFile({'id': file_id})
downloaded.FetchMetadata()
filename = downloaded.metadata['originalFilename'] 
downloaded.GetContentFile(filename)
search_terms_df = pd.read_csv(filename)


# Cool banner printer
def banner(t, s='~'):
    l = s * (len(t) + 4)
    print(l + '\n' + '{0} {1} {0}'.format(s, t) + '\n' + l)

# Formatted line
def line(t):
  print("\n", "~"*(26-(len(t)//2)), t, "~"*(26-(len(t)//2)))

banner('Colab environment configured')<jupyter_output><empty_output><jupyter_text>Run the spider web crawler script Menu for selecting the site to crawl.<jupyter_code>ALLOWED = ''
URL = ''
def menu():
  banner("Select a site to scrape")
  for i, site in enumerate(website_list_df['FQD']):
    print('['+str(i)+'] '+ site)
  opt = input('Select a site by #:\n ')
  try:
    opt = int(opt)
    if opt in range(0, website_list_df['FQD'].count()):
      global ALLOWED
      global URL
      ALLOWED = website_list_df['Domain'][opt]
      URL = website_list_df['FQD'][opt]
      line('Selected: {}'.format(URL))
    else:
      print('Input must be in range.')
      menu()
  except Exception as e:
    print('Input must be a number.')
    menu()   

menu()<jupyter_output><empty_output><jupyter_text>Execute the script that gets the links- run this to execute the webcrawler script that gets the links from the target webpage<jupyter_code>!python3 sixnineeight.py ALLOWED_DOMAINS={ALLOWED} URLS={URL} # This is a custom spider class, see script for custom configs<jupyter_output><empty_output><jupyter_text>Download the files and create a zip file for processing- run this to scrape the pdfs from the target webpage<jupyter_code>import os
import requests
import zipfile
import pandas as pd
from urllib.parse import urlparse, urljoin
from tqdm import tqdm

# Prints a banner
banner('Beginning file downloads')

# Sets custom user-agent string to masquerade 
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36'}

# Reads the links fro the json file and sorts for .pdf (or other) filetypes found
pdf_links = pd.read_json('/content/698S_BOG/links.json')
pdf_links = pdf_links[pdf_links['link'].str.contains(".pdf", na=False)] 

# Loops through the list of links
for url in pdf_links.link.values.tolist():
  # Gets the basename of the file and if a dynamic link sets a new name
  filename = os.path.basename(urlparse(url).path)
  if os.path.exists(os.path.join('content', filename)):
    filename = filename + urlparse(url).query + '.pdf'
  
  print('*'*50)
  print("Requesting: " + url)
  try:
    # Checks if content directory exists, if not, create
    if not os.path.exists('content'):
      os.makedirs('content')

    # Makes the request
    r = requests.get(url, stream=True, headers=headers)

    # Processes if link is valid, or raise generic exception
    if r.status_code == 200:
      # Chunks the file for progress bar
      total_size = int(r.headers.get('content-length', 0))
      block_size = 1024 

      # Progress bar parameters
      t=tqdm(total=total_size, unit='iB', unit_scale=True)
      
      # Downloads the files to the content folder
      with open(os.path.join('content', filename), 'wb') as f:
          for data in r.iter_content(block_size):
              t.update(len(data))
              f.write(data)
      
      # Error handling
      if total_size != 0 and t.n != total_size:
        raise
      
      # Close the progress bar object
      t.close()
    else:
      print('{} Response, Skipping {}.'.format(r.status_code, url))
    
  except Exception as e:
    print("Error: {}".format(e))

# Zip up the files
banner("Zipping up the files")

# Zip object walks directory and packs all files without directory structure
try:
  ziper = zipfile.ZipFile('content.zip', 'w', zipfile.ZIP_DEFLATED)
  for root, dirs, files in os.walk('content/'):
    for file in files:
      ziper.write(os.path.join(root, file), file)
  ziper.close()
except Exception as e:
  print("Could not zip the file. " + e)

# Cleans up the files and removes the content directory
try:
  if os.path.exists('content'):
    for root, dirs, files in os.walk('content/'):
      for file in tqdm(files):
        os.remove(os.path.join(root, file))
    os.removedirs('content')
  if os.path.exists('content.zip'):
    print('\nZip file saved')
  else:
    raise Exception("Could not write zip file.")
except Exception as e:
  print(e)<jupyter_output><empty_output><jupyter_text>Parse the documents for the desired text  Import parsing libraries<jupyter_code># Import packages needed form parsing
import os
import pandas as pd
import tika
tika.initVM()
from tika import parser<jupyter_output><empty_output><jupyter_text>Load the search files and search terms<jupyter_code># My take on unzipping the files again
import zipfile

working_folder = os.path.join('/content', '698S_BOG')  ## DEBUG ## hard_coded
filename = os.path.join(working_folder, 'content.zip')
scrape_folder = os.path.join(working_folder, 'scraped_files')

try:
  if not os.path.exists(scrape_folder):
    os.makedirs(scrape_folder)

  with zipfile.ZipFile(filename, 'r') as z:
    z.extractall(scrape_folder)
    
  os.chdir(scrape_folder)
  line("Files unzipped")
except Exception as e:
  print(e)


# Get the search term file and load it to a dataframe
os.chdir(os.path.join('/content', git_repo))
file_id = '18MyoEiAYTG6w6JRejv5d4qVK4RxYyRu2'
downloaded = drive.CreateFile({'id': file_id})
downloaded.FetchMetadata()
filename = downloaded.metadata['originalFilename']
downloaded.GetContentFile(filename)
search_terms_df = pd.read_csv(filename)

os.chdir(scrape_folder)<jupyter_output><empty_output><jupyter_text>Open the PDFs and parse the contents<jupyter_code>## open all the PDFs and parse their content into dataframes
pdf_df = pd.DataFrame()
for filename in os.listdir(scrape_folder):
    if filename.endswith(".pdf"):
        parsed = parser.from_file(filename)
        row = pd.DataFrame(
            [[filename, scrape_folder, parsed["metadata"], parsed["content"]]],
            index = [filename],
            columns = ['filename', 'folder', 'metadata', 'content'])
        pdf_df = pdf_df.append(row)

# Replace any NaN values from the parser with ''
pdf_df = pdf_df.fillna('') 

# Cound the number of occurances of each term in the content of each document
for term in search_terms_df['Term_(foreign)']:
    pdf_df[term] = pdf_df['content'].apply(lambda x : x.count(term))

## Calculate the relative importance of each search term based on the importance values stored in the search_terms.csv

# Function to compute the relative importance of a value in a list of importance values
# takes an importance val and list of importance values and returns a relative importance scaler
def compute_rel_importance(val, importance_vals):
  importance_vals.sort()
  importance_val_sum = sum(importance_vals)
  x = 0
  rel_importance = importance_val_sum
  while val != importance_vals[x]:
      x += 1
      rel_importance = rel_importance - importance_vals[x]
  return rel_importance

# Calculate and add a relative importance column to the search terms dataframe
importance_vals = list(dict.fromkeys(search_terms_df['Importance']))
search_terms_df['Relative_Importance'] = search_terms_df['Importance'].apply(lambda x : compute_rel_importance(x, importance_vals))

## Compute the relative importance of each file based on the (occurance*relative importance) of each search term 

for term in search_terms_df['Term_(foreign)']:
    term_score = term + '_score'
    pdf_df[term+'_score'] = pdf_df[term].apply(lambda x : x * search_terms_df.loc[search_terms_df['Term_(foreign)'] == term, 'Relative_Importance'])
col_to_sum = [item for item in list(pdf_df.columns) if any(sub in item for sub in ['score'])] 
pdf_df['importance_rank_score'] = pdf_df[col_to_sum].sum(axis=1)
pdf_df = pdf_df.drop(col_to_sum, axis=1)
pdf_df = pdf_df.sort_values(by='importance_rank_score', ascending=False)<jupyter_output><empty_output><jupyter_text>Show the dataframe<jupyter_code>## Display the whole dataframe
pdf_df

## Optionally, you can show just search term results by uncommenting the below two lines 
# view = list(search_terms_df['Term_(foreign)']) + ['importance_rank_score']
# pdf_df[view]<jupyter_output><empty_output><jupyter_text>Translate the text content of a selected document<jupyter_code># Import the unofficial google translate package
# This package allows us to call the google translate website for small ~1500 char translations
import googletrans
from googletrans import Translator

# Filename that you want to translate into english
# If you don't change it, it defaults to the highest ranked file
filename_to_translate = pdf_df.iloc[0].filename ## Replace this with the filename you want to translate

# Load the content of the desired file from the pandas dataframe
content_to_translate = pdf_df.loc[filename_to_translate].content

# clean any excessive line breaks (i.e. double) from the document
content_to_translate = content_to_translate.replace('\n\n', ' ')

# instantiate the translator object
translator = Translator()


# Check the length of the content, if it is more then 1500 batch the requests to translate
if len(content_to_translate) < 1500: 
  # Detect the language of the text
  detected_language = translator.detect(content_to_translate)

  # Translate the text and store the text output to 'translated_content'
  translated_batch = translator.translate(content_to_translate, dest = 'en', src = detected_language.lang)
  translated_content = translated_batch.text
else:
  # Initialize our variables, batch size, and the begining/end of the batch locs
  batch_size = 1500 # number of characters to translate in each iteration
  char_index_start = 0  # The start of the index
  char_index_end = char_index_start + batch_size # the end of the index
  translated_content = ''

  # Detect the language of the text
  detected_language = translator.detect(content_to_translate[char_index_start:char_index_end])
  
  # Translate the text in batches and store the text output to 'translated_content'
  while char_index_start < len(content_to_translate):
      translated_batch = translator.translate(content_to_translate[char_index_start:char_index_end], dest = 'en', src = detected_language.lang)
      translated_content = translated_content + translated_batch.text
      char_index_start = char_index_end
      if char_index_end + batch_size < len(content_to_translate):
        char_index_end += batch_size
      else: 
        char_index_end = len(content_to_translate)

banner('Translating ' + filename_to_translate)
print(translated_content)<jupyter_output><empty_output><jupyter_text>Perform basic sentiment analysis of the translated_contentUse TextBlob to find the sentiment of the translated text.  The sentiment property returns a namedtuple of the form Sentiment(polarity, subjectivity). The polarity score is a float within the range [-1.0, 1.0]. The subjectivity is a float within the range [0.0, 1.0] where 0.0 is very objective and 1.0 is very subjective.<jupyter_code># Import the TextBlob package
from textblob import TextBlob

translated_textblob = TextBlob(translated_content)
translated_textblob.sentiment
banner('You are a great American!!')<jupyter_output><empty_output><jupyter_text>Export the data and results for later useThe documents collected by this code will not persist after the virtual machine shuts down.  However, There are multiple ways to export the zipfile containing all the documents.  Below are two different ways.  I reccomend only running one of these, but if you want to save the file in multiple locations that is fine.1. Zip the files and dataframe with info on the files.2. Export the contents to google drive or download to your computer.    2.1 Save the content to a user selected folder on googledrive.    2.2 OR Download the file to the user's local computer.<jupyter_code>## 1. zip the files and the dataframe with the results
pdf_df.to_excel(os.path.join(scrape_folder, 'file_info.xlsx'))
%cd {working_folder}
export_filename = 'data_export.zip'
!zip -r -j {export_filename} {scrape_folder}/*.*
## 2.1 Mount the users google drive folder to '/gdrive/
from google.colab import drive
mount_folder = drive.mount('/gdrive')

# Select the output folder 
output_folder = 'scraper_output'

output_path = os.path.join(mount_folder, output_folder)
os.chdir(working_folder)

# Copy the output folder to that location
!cp {export_filename} {output_path}

## 2.2 Download the file to the user's local computer
from google.colab import files

files.download(export_filename)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09442795158712035, 'fraction_numerical': 0.011189769353733729, 'mean_word_length': 4.754599211563732, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 49, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '200938', 'license': 'MIT', 'n_tokens_mistral': 5569, 'n_tokens_neox': 5333, 'n_words': 1838, 'path': '698S_Project_BOG.ipynb', 'repo_name': 'PurpleDin0/698S_BOG', 'size': '17516'}