<jupyter_start><jupyter_text>--- Scrape Google Forever  ---                             Or till you go bankrupt, which ever is earlier.  This is the first part of the series and I honestly don't know when it's going to be over. But these parts will work independent from the rest of the series. Let's hope.  You start writing a code to scrape google, the first thing you need is a piece of code that can scrape a google. That's pretty basic and that's what you do El Scraper<jupyter_code># Add basic repos
import requests
import urllib
#import pandas as pd
from requests_html import HTML
from requests_html import HTMLSession
# Get the source code given a url
def get_source(url):
    # Given a url it's gonna give you the source code
    
    try:
        session = HTMLSession()
        response = session.get(url)
        return response
    
    except requests.exceptions.RequestException as e:
        print(e)
# Parse the query. We are gonna create the queries to be parsed later
def get_results(query):
    # When you give a query as the input it returns the sourcecode as response
    query = urllib.parse.quote_plus(query)
    response = get_source("https://www.google.com/search?q=" + query)
    return response
# There is gonna be a lot of noice in it. We are gonna need only link, title and text.
def parse_results(response):
    if not response:
        return {}
    
    css_identifier_result = ".tF2Cxc"
    css_identifier_title = "h3"
    css_identifier_link = ".yuRUbf a"
    css_identifier_text = ".IsZvec"
    
    results = response.html.find(css_identifier_result)

    output = []
    
    for result in results:
        title = result.find(css_identifier_title, first=True)
        title  = title.text if title is not None else ""
        link = result.find(css_identifier_link, first=True)
        link = link.attrs['href'] if link is not None else ""
        text = result.find(css_identifier_text, first=True)
        text = text.text if text is not None else ""

        item = {
            "title": title,
            "link": link,
            "text": text
        }
        
        output.append(item)
        
    return output
# Gonna wrap this all nicely
def google_search(query):
    response = get_results(query)
    return parse_results(response)

# And we can test it now
# Most of it has empty text. We are gonna change that hopefully
results = google_search("Python web scraping tutorial")

json.dumps(results)<jupyter_output><empty_output><jupyter_text>Now that's done you are gonna start scraping the google. Or so you would think ....  What about search keyword? Are you gonna use the same keyword over an over again? We are gonna search google forever. You are not gonna manually write keywords forever. So you randomly generate keywords. Let's see if we have got any libraries in python for that. You wanna say hi to Google Dot Com.  El Essential Generator Luckily there is a in-built generator that you can use. (Actually there are many generators out there, I just picked the first one I found)<jupyter_code># You install it first cause probably you don't have it
!pip install essential-generators

from essential_generators import DocumentGenerator
gen = DocumentGenerator()
print(gen.sentence())<jupyter_output>Is required; whether physical, mental, and social updates. This has.<jupyter_text>Now you can give this result to the google scraper<jupyter_code># Now you can give this result to the google scraper
google_search("A casinò. chemical substitute")<jupyter_output><empty_output><jupyter_text>Google has a say about everything. Almost. Let's wrap this baby real nice.<jupyter_code>def fetch_google_results():
    gen = DocumentGenerator()
    key = gen.sentence()
    return google_search(key)
fetch_google_results()<jupyter_output><empty_output><jupyter_text>I think we can wrap more things in it. Let's see. We can save the results to a file.  Google has a say about everything. Almost. Now we are not just gonna print the result. We need to save it somewhere in some format. We can do anything here. Got to a sql or nosql database. Just store as a csv or a json. Let's do json. The result is a dict anyway.<jupyter_code>import json
def fetch_google_results():
    gen = DocumentGenerator()
    key = gen.sentence()
    result_dict = google_search(key)
    json_object = json.dumps(result_dict) 
    return json_object
s = fetch_google_results()
s<jupyter_output><empty_output><jupyter_text>Where you wanna store it? Let's save it as a json file<jupyter_code>import json
def save_google_results_as_json():
    gen = DocumentGenerator()
    key = gen.sentence()
    result_dict = google_search(key)
    filename = key + ".json"
    with open(filename, "w") as json_saver:
        json.dump(result_dict, json_saver, indent=4, sort_keys=True)
        # we are gonna save the pretty print version of json using the indent =4 param even though no human will 
        # read it eventually<jupyter_output><empty_output><jupyter_text>Let's test it<jupyter_code>save_google_results_as_json()<jupyter_output><empty_output><jupyter_text>I have a beautiful json in the current folder named "Hosted UEFA however, coincided with many of the Isthmus of Suez and.json". We are gonna royally ignore the keywords Essential generator generates for now. It's doing what we asked of it.  Now we just need to keep doing it. Easy peasy Japaneasy!   "Let's scrape google forever"                                                                            he said.<jupyter_code>while True:
    # That's an infinite loop right there in it's raw form. Don't do this unless you know for sure your code won't run
    save_google_results_as_json()
# That is a stupid file name. Let's correct those
# Special mention : https://github.com/django/django/blob/main/django/utils/text.py
import re
def get_valid_filename(s):
    s = str(s).strip().replace(' ', '_')
    return re.sub(r'(?u)[^-\w.]', '', s)
import json
def save_google_results_as_json():
    gen = DocumentGenerator()
    key = gen.sentence()
    result_dict = google_search(key)
    filename = "Scraped/" + get_valid_filename(key[:40]) + ".json"
    with open(filename, "w") as json_saver:
        json.dump(result_dict, json_saver, indent=4, sort_keys=True)
        # we are gonna save the pretty print version of json using the indent =4 param even though no human will 
        # read it eventually<jupyter_output><empty_output><jupyter_text>"Let's scrape google forever"                                                                            he said again.<jupyter_code>while True:
    # That's an infinite loop right there in it's raw form. Don't do this unless you know for sure your code won't run
    save_google_results_as_json()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0749667307407955, 'fraction_numerical': 0.0011829069939376016, 'mean_word_length': 3.3526383526383525, 'pattern_counts': {'":': 3, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '457293', 'license': 'Apache-2.0', 'n_tokens_mistral': 1936, 'n_tokens_neox': 1879, 'n_words': 791, 'path': '1. Scrape Google.ipynb', 'repo_name': 'jo07/AWS-projects', 'size': '6763'}