<jupyter_start><jupyter_text>Part 1. Authentication in a service  1.1. What do you store in your Google Drive?Sometimes it can be quite troublesome to crawl web data - for example, when you can't just collect data from web-pages because the authentification to a website is required. Today's tutorial is about a dataset of special type - namely, Google Drive data. You will need to get access to the system using OAuth protocol, download and parse files of different types.Plan. 1. Download [this little archive](https://drive.google.com/open?id=1Xji4A_dEAm_ycnO0Eq6vxj7ThcqZyJZR), **unzip** it and place the folder anywhere inside your Google Drive. You should get a subtree of 6 folders with files of different types: presentations, pdf-files, texts, and even code.2. Go to [Google Drive API](https://developers.google.com/drive/api/v3/quickstart/python) documentation, read [intro](https://developers.google.com/drive/api/v3/about-sdk) and learn how to [search for files](https://developers.google.com/drive/api/v3/reference/files/list) and [download](https://developers.google.com/drive/api/v3/manage-downloads) them. Pay attention, that  working at `localhost` (jupyter) and at `google colab` can be slighty different. We expect you to run from localhost.3. Learn how to open from python such files as [pptx](https://python-pptx.readthedocs.io/en/latest/user/quickstart.html), pdf, docx or even use generalized libraries like [textract](https://textract.readthedocs.io/en/stable/index.html), save internal text in a file near.4. Write a code with returns names (with paths) of files for a given substring. Test on these queries.```segmentationalgorithmclassiferprintfpredecessorШеннонHuffmanfunctionconstructormachine learningdatasetПротасовProtasov```  1.1.1. Access GDrive Below is the example of how you can oranize your code - it's fine if you change it.Let's extract the list of all files that are contained (recursively) in the folder of interest. In my case, I called it `air_oauth_folder`.<jupyter_code># install some dependencies
!pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib
def gdrive_get_all_files_in_folder(folder_name):
    #TODO retrieve all files from a given folder   
    return []

def gdrive_download_file(file, path_to_save): 
    #TODO download file and save it under the path
    pass
folder_of_interest = 'air_oauth_folder'
files = gdrive_get_all_files_in_folder(folder_of_interest)

test_dir = "test_files"
for item in files:
    gdrive_download_file(item, test_dir)<jupyter_output><empty_output><jupyter_text>1.1.2. Tests Please fill free to change function signatures and behaviour.<jupyter_code>assert len(files) == 34, 'Number of files is incorrect'
print('n_files:', len(files))

print("file here means id and name, e.g.: ", files[0])

gdrive_download_file(files[0], '.')

import os.path
assert os.path.isfile(os.path.join('.', files[0][1])), "File is not downloaded correctly"<jupyter_output><empty_output><jupyter_text>1.2. Read files content 1.2.1. Read<jupyter_code># install dependencies
!pip install textract<jupyter_output><empty_output><jupyter_text>For windows please refer to - https://textract.readthedocs.io/en/latest/installation.htmldon-t-see-your-operating-system-installation-instructions-here- https://www.xpdfreader.com/download.htmlALSO BE CAREFUL WITH SPACES IN NAMES. Better save without spaces!!!!<jupyter_code>import textract 

def get_file_strings(path):
    #TODO change this function to handle different data types properly 
    # - textract is not able to parse everything
    # Take care of non-text data too
    texts = str(textract.process(path)).replace('\\n', '\n').replace('\\r', '').split('\n')
    return texts
# creating dictionary of parsed files
files_data = dict()
for file in os.scandir(test_dir):    
    strings = get_file_strings(file.path)
    if strings:
        files_data[file.name] = strings<jupyter_output><empty_output><jupyter_text>1.2.2. Tests for read<jupyter_code>assert len(files_data) == 31 
print(len(files_data))

assert "Protasov" in get_file_strings(os.path.join(test_dir, 'at least this file.txt')), "TXT File parsed incorrectly"
assert "A. Image classification" in get_file_strings(os.path.join(test_dir, 'deep-features-scene (1).pdf')), "PDF File parsed incorrectly"<jupyter_output><empty_output><jupyter_text>1.3. Tests<jupyter_code>def find(query):
    #TODO implement search procedure
    return []
queries = ["segmentation", "algorithm", "printf", "predecessor", "Huffman",
           "function", "constructor", "machine learning", "dataset", "Protasov"]

for query in queries:
    r = find(query)
#     print("Results for: ", query)
#     print("\t", r)
    assert len(r) > 0, "Query should return at least 1 document"
    assert len(r) > 1, "Query should return at least 2 documents"
    assert "at least this file.txt" in r, "This file has all the queries. It should be in a result"<jupyter_output><empty_output><jupyter_text>2. Parse me if you can Sometimes when crawling we have to parse websites that turn out to be SaaS - i.e., there is a special JS application which renders documents and which is downloaded first. Therefore, data that is to be rendered initially comes in a proprietary format. One of the examples is Google Drive. Last time we downladed and parsed some files from GDrive, however, we didn't parse GDrive-specific file formats, such as google sheets or google slides.Today we will learn to obtain and parse such data using Selenium - a special framework for testing web-applications. 2.1. Getting startedLet's try to load and parse the page the way we did before:<jupyter_code>import requests
from bs4 import BeautifulSoup
resp = requests.get("https://docs.google.com/presentation/d/1LuZvz3axBD8UuHLagdv0EbhsGEWJmpd7gN5KjwYCp9Y/edit?usp=sharing")
soup = BeautifulSoup(resp.text, 'lxml')
print(soup.body.text[:1000])<jupyter_output><empty_output><jupyter_text>As we see, the output is not what we expect. So, what can we do when a page is not being loaded right away, but is rather rendered by a script? Browser engines can help us get data. Let's try to load the same web-page, but do it in a different way: let's give some time to a browser to load the scripts and run them; and then will work with DOM (Document Object Model), but will get it from browser engine itself, not from BeautifulSoup.Where do we get browser engine from? Simply installing a browser will do the thing. How do we send commands to it from code and retrieve DOM? Service applications called drivers will interpret out commands and translate them into browser actions.For each browser engine suport you will need to:1. install browser itself;2. download 'driver' - binary executable, which passed commands from selenium to browser. E.g. [Gecko == Firefox](https://github.com/mozilla/geckodriver/releases), [ChromeDriver](http://chromedriver.storage.googleapis.com/index.html);3. unpack driver into a folder under PATH environment variable. Or specify exact binary location. 2.1.1. Download driverAnd place it in any folder or under PATH env. variable. 2.1.2. Install selenium<jupyter_code>!pip install -U selenium
from selenium import webdriver<jupyter_output><empty_output><jupyter_text>2.1.3. Launch browserThis will open browser window<jupyter_code>browser = webdriver.Firefox()
# or explicitly
# browser = webdriver.Firefox(
#     executable_path='C:/bin/geckodriver.exe', 
#     firefox_binary='C:/Program Files/Mozilla Firefox/firefox.exe'
# )<jupyter_output><empty_output><jupyter_text>2.1.4. Download the page<jupyter_code># navigate to page
browser.get('http://tiny.cc/00dhkz')
browser.implicitly_wait(5)  # wait 5 seconds

# select all text parts from document
elements = browser.find_elements_by_css_selector("g.sketchy-text-content-text")
# note that if number differs from launch to launch this means better extend wait time
print("Elements found:", len(elements))

# oh no! It glues all the words!
print("What if just a silly approach:", elements[0].text)

# GDrive stores all text blocks word-by-word
subnodes = elements[0].find_elements_by_css_selector("text")
text = " ".join(n.text for n in subnodes)
print("What if a smart approach:", text)
browser.quit()<jupyter_output><empty_output><jupyter_text>- Too slow, wait for browser to open, browser to render 2.2. HeadlessBrowsers (at least [FF](https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Headless_mode), [Chrome](https://intoli.com/blog/running-selenium-with-headless-chrome/), IE) have headless mode - no window rendering and so on. Means it should work much faster!<jupyter_code>options = webdriver.FirefoxOptions()

options.add_argument('headless')
options.add_argument('window-size=1200x600')
browser = webdriver.Firefox(options=options)
## SAME CODE

# navigate to page
browser.get('http://tiny.cc/00dhkz')
browser.implicitly_wait(5)  # wait 5 seconds

# select all text parts from document
elements = browser.find_elements_by_css_selector("g.sketchy-text-content-text")
# note that if number differs from launch to launch this means better extend wait time
print("Elements found:", len(elements))

# oh no! It adds NEW LINE. Behavior differs!!!!
print("What if just a silly approach:", elements[0].text)

# GDrive stores all text blocks word-by-word
subnodes = elements[0].find_elements_by_css_selector("text")
text = " ".join(n.text for n in subnodes)
print("What if a smart approach:", text)
browser.quit()<jupyter_output><empty_output><jupyter_text>2.2.1. NB Note, that browser behavior differs for the same code! 2.3. Task Our lectures usually have lot's of links. Here are the links to original (spring 2020) versions of the documents.[4. Vector space](https://docs.google.com/presentation/d/1UxjGZPPrPTM_3lCa_gWTk8yZI_qNmTKwtMxr8JZQCIc/edit?usp=sharing)[6. search trees](https://docs.google.com/presentation/d/1LuZvz3axBD8UuHLagdv0EbhsGEWJmpd7gN5KjwYCp9Y/edit?usp=sharing)[7-8. Web basics](https://docs.google.com/presentation/d/1bgsCgpjMcQmrFpblRI6oH9SnG4bjyo5SzSSdKxxHNlg/edit?usp=sharing)Please complete the following tasks: 2.3.1. Search for slides with numbersI want to type a word, and it should say which slides of which lecture has this word.<jupyter_code>def getTextAndImgsFromSlides(url):    
    slides_text = dict() # dictionary slide_num : slide_text
    img_list = [] # list of image urls 
    #TODO: parse google slides and save all text and image urls in slides_text and img_list
    # you should get the contents from ALL slides - however, you will see that at one moment 
    # of time only single slide + few slide previews on the left are visible. To be able to    
    # reach all slides you will need to scroll to and click these previews. While slide contents 
    # can be obtained from previews themselves, speaker notes (which you also have to extract)
    # can be viewed only if a particular slide is open.
    # to scroll the element of interest into view, use can this: 
    # browser.execute_script("arguments[0].scrollIntoView();", el)
    # to click the element, use can use ActionChains library   
    
    
    return slides_text, img_list<jupyter_output><empty_output><jupyter_text>Parsing three presentations<jupyter_code>links = ["https://docs.google.com/presentation/d/1UxjGZPPrPTM_3lCa_gWTk8yZI_qNmTKwtMxr8JZQCIc/edit?usp=sharing", 
         "https://docs.google.com/presentation/d/1LuZvz3axBD8UuHLagdv0EbhsGEWJmpd7gN5KjwYCp9Y/edit?usp=sharing",
         "https://docs.google.com/presentation/d/1bgsCgpjMcQmrFpblRI6oH9SnG4bjyo5SzSSdKxxHNlg/edit?usp=sharing"]

all_imgs = []
all_texts = dict()

for i, link in enumerate(links):
    texts, imgs = getTextAndImgsFromSlides(link)<jupyter_output><empty_output><jupyter_text>2.3.2. Tests<jupyter_code>texts, imgs = getTextAndImgsFromSlides('http://tiny.cc/00dhkz')

assert len(texts) == 35 # equal to the total number of slides in the presentation 
print(len(texts))

assert len(imgs) > 26 # can be more than that due to visitor icons
print(len(imgs))

assert any("Navigable" in value for value in texts.values()) # word is on a slide
assert any("MINUS" in value for value in texts.values()) # word is in speaker notes
queries = ["architecture", "algorithm", "function", "dataset", 
           "Protasov", "cosine", "модель", "например"]

for query in queries:
    r = find(query)
    print("Results for: ", query)
    print("\t", r)
    assert len(r) > 0, "Query should return at least 1 document"
    assert len(r) > 1, "Query should return at least 2 documents"<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09939277724512624, 'fraction_numerical': 0.01278363694471077, 'mean_word_length': 5.0497825036249395, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 62, 'https://': 19, 'lorem ipsum': 0, 'www.': 1, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '515292', 'license': 'MIT', 'n_tokens_mistral': 4052, 'n_tokens_neox': 3881, 'n_words': 1299, 'path': '[Spring 2021] Advanced Information retrieval/Week 2/HW/2021S-0304 GDrive magic.ipynb', 'repo_name': 'RomulusGwelt/Innopolis', 'size': '12516'}