<jupyter_start><jupyter_text>Download images (photographs) from the Web **Topics**1. Football Players2. Actors**Sources**1. Wikipedia2. Google Photos  Get Top football players images from Wikipedia  **Documentation**https://wikipedia.readthedocs.io/en/latest/quickstart.html<jupyter_code>import wikipedia
import requests
from tqdm import tqdm
# Documentation
# wikipedia.page?
top_football_players = ["<NAME>", "<NAME>", "<NAME>", "<NAME>",
                        "<NAME>", "<NAME>", "<NAME>",
                        "<NAME>.", "<NAME>", "<NAME>", "<NAME>",
                        "<NAME>", "<NAME>", "<NAME>", "<NAME>",
                        "<NAME>", "<NAME>", "<NAME>",
                        "<NAME>", "<NAME>", "Jorginho", "<NAME>", "<NAME>",
                        "<NAME>", "<NAME>", "<NAME>", 
                        "<NAME>", "<NAME>", "<NAME>", "<NAME>",
                        "<NAME> <NAME>", "<NAME>", "<NAME>",
                        "<NAME>",
                        "<NAME>", "<NAME>", "<NAME>", "<NAME>",
                        "<NAME>", "<NAME>", "<NAME>", "<NAME>",
                        "<NAME>", "<NAME>", "<NAME>", "<NAME>", "<NAME>",
                        "<NAME>", "<NAME>", "<NAME>",
                        "<NAME>", "Z<NAME>"]

print(f"Top {len(top_football_players)} football players.")<jupyter_output>Top 54 football players.<jupyter_text>Download images<jupyter_code>## Downlaod images

error_players = []

for player in top_football_players[33:]:

    wikipage = wikipedia.page(player)

    print("Page Title: %s" % wikipage.title)
    print("Page URL: %s" % wikipage.url)
    print("Nr. of images on page: %d" % len(wikipage.images))

    player_path = f"../data/{player}"
    
    try:
        os.mkdir(player_path)
    except FileExistsError:
        pass


    for i, image_url in tqdm(enumerate(wikipage.images)):

        response = requests.get(image_url)
        file = open(player_path + "/" + player+"_"+str(i)+"."+image_url.split(".")[-1], "wb")
        file.write(response. content)
        file.close()
import google_images_download
google_images_download.main<jupyter_output><empty_output><jupyter_text>https://levelup.gitconnected.com/how-to-download-google-images-using-python-2021-82e69c637d59 https://www.geeksforgeeks.org/how-to-download-google-images-using-python/  Google images<jupyter_code>from google_images_download import google_images_download
response = google_images_download.googleimagesdownload() 
def downloadimages(query):
    """
    - keywords is the search query
    - format is the image file format
    - limit is the number of images to be downloaded
    - print urs is to print the image file url
    - size is the image size which can be specified manually ("large, medium, icon")
    - aspect ratio denotes the height width ratio of images to download. ("tall, square, wide, panoramic")
    """
    arguments = {"keywords": query,
                 "format": "jpg",
                 "limit":100,
                 "print_urls":True}
                 # "size": "medium"
                 # "aspect_ratio":"panoramic"
    try:
        response.download(arguments)
      
    # Handling File NotFound Error    
    except FileNotFoundError: 
        pass<jupyter_output><empty_output><jupyter_text>https://github.com/ultralytics/google-images-download<jupyter_code># Driver Code
for query in ["<NAME>"]:
    downloadimages(query) 
    print()<jupyter_output>Item no.: 1 --> Item name = <NAME>
Evaluating...
Starting Download...


Unfortunately all 100 could not be downloaded because some images were not downloadable. 0 is all we got for this search filter!

Errors: 0<jupyter_text>DEV<jupyter_code>## // DEV //
player = "<NAME>"
wikipage = wikipedia.page(player)

print("Page Title: %s" % wikipage.title)
print("Page URL: %s" % wikipage.url)
print("Nr. of images on page: %d" % len(wikipage.images))
player_path = f"../data/{player}"
os.mkdir(player_path)
for i, image_url in tqdm(enumerate(wikipage.images)):
    response = requests.get(image_url)
    file = open(player_path + "/" + player+"_"+str(i)+"."+image_url.split(".")[-1], "wb")
    file.write(response. content)
    file.close()
# Downdload images
# ! wget https://upload.wikimedia.org/wikipedia/commons/e/e8/Robert_Lewandowski_2011_%283%29.jpg<jupyter_output><empty_output><jupyter_text>Check Final dataset<jupyter_code>players = os.listdir("../data/")
print(set(players).difference(set(top_football_players)))
print(set(top_football_players).difference(set(players)))<jupyter_output>set()
{'<NAME>', '<NAME>'}<jupyter_text>How to download Google images **Tutorial**:https://levelup.gitconnected.com/how-to-download-google-images-using-python-2021-82e69c637d59 **Step 1:** Setup Selenium**Selenium:** the great thing about Selenium is that you’re able to interact with web pages based on the chrome driver* Install chrome driver: https://chromedriver.chromium.org/downloads<jupyter_code># Imports Packages
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time 
# Opens up web driver and goes to Google Images
driver = webdriver.Chrome("/home/user7/chromedriver_linux64/chromedriver")
driver.get('https://www.google.ca/imghp?hl=en&tab=ri&authuser=0&ogbl')<jupyter_output><empty_output><jupyter_text>**Step 2:** Interacting With Google Home Page**How to write input text into the Google Images search bar.**To do this you need to find the location of the search bar on the page. This is done by finding the search bar’s `Xpath.` 1. To find the `Xpath` you need to open up the HTML of the web page which is done by clicking right anywhere on the page, and then pressing inspect.2. Find the HTML code that corresponds to that Google search bar. To do this click the cursor icon on the top left of the box that just popped up.3. Then click on the Google Images search box. When this is done the HTML on the right will update and a piece of HTML code will be highlighted in blue.4. To get the Xpath of the search bar, you just need to right-click on the highlighted blue text and press “Copy Xpath”.<jupyter_code># Setup the search box
box = driver.find_element_by_xpath('//*[@id="sbtc"]/div/div[2]/input')
# Define Keyword
keyword = '<NAME>'
# Write keyword in the google search box and press enter
box.send_keys(keyword)
box.send_keys(Keys.ENTER)<jupyter_output><empty_output><jupyter_text>**Step 3: Scrolling Down the Web Page** Google Images is an **infinite scrolling web page**. This means as you keep scrolling down the web page new images will load in until there are no more images left.Therefore, if we want to download hundreds of images we must first load them into the web page by scrolling down.<jupyter_code># Will keep scrolling down the webpage until it cannot scroll no more
last_height = driver.execute_script('return document.body.scrollHeight')

while True:
    driver.execute_script('window.scrollTo(0,document.body.scrollHeight)')
    time.sleep(2)
    new_height = driver.execute_script('return document.body.scrollHeight')
    try:
        driver.find_element_by_xpath('//*[@id="sbtc"]/div/div[2]/input').click()
        time.sleep(2)
    except:
        pass
    if new_height == last_height:
        break
    last_height = new_height<jupyter_output><empty_output><jupyter_text>**Step 4: Downloading (Screenshot) The Images**<jupyter_code># First image: //*[@id="islrg"]/div[1]/div[1]/a[1]/div[1]/img
# Second image: //*[@id="islrg"]/div[1]/div[2]/a[1]/div[1]/img
dir_path = f'../data/actors/{keyword}'
print(dir_path)
if not os.path.isdir(dir_path):
    os.mkdir(dir_path)
# Screenshot
for i in range(1,10):
    try:
        driver\
        .find_element_by_xpath('//*[@id="islrg"]/div[1]/div['+str(i)+']/a[1]/div[1]/img')\
        .screenshot(f'{dir_path}/{keyword}_{str(i)}.png')

    except Exception as e:
        print(e)
import urllib
# https://stackoverflow.com/questions/17361742/download-image-with-selenium-python
## Download
for i in range(1,10):
    try:
        driver\
        .find_element_by_xpath('//*[@id="islrg"]/div[1]/div['+str(i)+']/a[1]/div[1]/img')\
        .screenshot(f'{dir_path}/{keyword}_{str(i)}.png')

    except Exception as e:
        print(e)
src = driver.get_attribute('src')

# download the image
urllib.urlretrieve(src, "captcha.png")

driver.close()<jupyter_output><empty_output><jupyter_text>Download (not Screenshot) images<jupyter_code>from selenium import webdriver
import time, requests

def search_google(search_query):
    browser = webdriver.Chrome("/home/user7/chromedriver_linux64/chromedriver")
    search_url = f"https://www.google.com/search?site=&tbm=isch&source=hp&biw=1873&bih=990&q={search_query}"
    images_url = []

    # open browser and begin search
    browser.get(search_url)
    elements = browser.find_elements_by_class_name('rg_i')

    count = 0
    
    for e in elements:
        
        try:
            # get images source url
            e.click()
            time.sleep(1)
            element = browser.find_elements_by_class_name('v4dQwb')

            # Google image web site logic
            if count == 0:
                big_img = element[0].find_element_by_class_name('n3VNCb')
            else:
                big_img = element[1].find_element_by_class_name('n3VNCb')

            images_url.append(big_img.get_attribute("src"))


            # write image to file
            reponse = requests.get(images_url[count])
            if reponse.status_code == 200:
                with open(f"../data/actors/{search_query}/{search_query}_{count+1}.jpg","wb") as file:
                    file.write(reponse.content)
        except:
            pass
        
        count += 1
        print("Count:", count, end="\n")
        # Stop get and save after N
        if count >= 50:
            break

    return images_url<jupyter_output><empty_output><jupyter_text>Run<jupyter_code>actors =  [
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>-Lewis",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME> Jr.",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>",
"<NAME>"]
# actors[42:]
for search_query in actors[43:]:
    
    # Create directory to store photos
    dir_path = f'../data/actors/{search_query}'
    print(dir_path)
    if not os.path.isdir(dir_path):
        os.mkdir(dir_path)
    
    # Download photos
    items = search_google(search_query)<jupyter_output>../data/actors/<NAME>
Count: 1
Count: 2
Count: 3
Count: 4
Count: 5
Count: 6
Count: 7
Count: 8
Count: 9
Count: 10
Count: 11
Count: 12
Count: 13
Count: 14
Count: 15
Count: 16
Count: 17
Count: 18
Count: 19
Count: 20
Count: 21
Count: 22
Count: 23
Count: 24
Count: 25
Count: 26
Count: 27
Count: 28
Count: 29
Count: 30
Count: 31
Count: 32
Count: 33
Count: 34
Count: 35
Count: 36
Count: 37
Count: 38
Count: 39
Count: 40
Count: 41
Count: 42
Count: 43
Count: 44
Count: 45
Count: 46
Count: 47
Count: 48
../data/actors/<NAME>
Count: 1
Count: 2
Count: 3
Count: 4
Count: 5
Count: 6
Count: 7
Count: 8
Count: 9
Count: 10
Count: 11
Count: 12
Count: 13
Count: 14
Count: 15
Count: 16
Count: 17
Count: 18
Count: 19
Count: 20
Count: 21
Count: 22
Count: 23
Count: 24
Count: 25
Count: 26
Count: 27
Count: 28
Count: 29
Count: 30
Count: 31
Count: 32
Count: 33
Count: 34
Count: 35
Count: 36
Count: 37
Count: 38
Count: 39
Count: 40
Count: 41
Count: 42
Count: 43
Count: 44
Count: 45
Count: 46
Count: 47
Count:[...]
{'fraction_non_alphanumeric': 0.16345674912834135, 'fraction_numerical': 0.02498754773368753, 'mean_word_length': 3.554631379962193, 'pattern_counts': {'":': 6, '<': 192, '<?xml version=': 0, '>': 194, 'https://': 10, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '215333', 'license': 'MIT', 'n_tokens_mistral': 4333, 'n_tokens_neox': 4005, 'n_words': 1077, 'path': 'notebooks/Download Images.ipynb', 'repo_name': 'papaemman/multimedia_db', 'size': '13158'}