<jupyter_start><jupyter_text>Image Extraction The "unadjusted" centers will now be used to determine the location of the images to be extracted. This script extracts the necceary images for each ward using Google Static Maps API. The output is an image-mask pair for each ward. Due to the sheer number of images required for each ward, the script only extracts images for one municipality.  Overview of MethodsFor each ward:1. Read in the centers for each image.2. Adjust the centers down.3. Extract three images of size 400x425 from each adjusted center: one plain image and two with ward overlayed in pure black and green.4. Write each image to the directory data/images/MUNICIPALITY/ward_k where k is the ward number. The filenames for the images are i_j.png, i_j_with_ward.png and i_j_with_ward_green.png where (i,j) are the row and column positions, respectively, of the image in the grid of images for the specific ward (e.g. top left = (0,0)).5. Remove the Google watermarks (i.e. by cropping out the bottom 25 pixels).6. Use the images with the ward overlayed to isolate the pixels that are outside of the ward. Set those pixels to black in the image with the ward in black. Isolate the pixels that are inside the ward and set them to white in the image with the ward in black. The result is a mask for the plain image. There were some issues encountered when isolating the pixels of the image that contained the ward and the pixels that did not. This is because some of the images contained pure black pixels and some of the images contained green pixels outside of the overlayed ward. This was resolved as follows:    * Method for isolating pixels in ward: Let A be the pixels that are black in the image with the ward overlayed in black. Let B be the pixels that are green in the image with the ward overlayed in green. The pixels inside the ward are then A $\cap$ B.    * Method for isolating pixels out ward: Let A be the pixels that are not black in the image with the ward overlayed in black. Let B be the pixels that are not green in the image with the ward overlayed in green. The pixels outside the ward are then A $\cup$ B.7. Write the edited plain image to the directory data/images/MUNICIPALITY/ward_k/edited/i_j.png and the mask to data/images/MUNICIPALITY/ward_k/masks/i_j.png.  Libraries Used<jupyter_code>import cv2
import matplotlib.pyplot as plt
import numpy as np
import json
import requests
import pandas as pd
import os
import ast
import multiprocessing as mp
import time<jupyter_output><empty_output><jupyter_text>Constants  Conversion of lat/lng to pixelsIn order to sample without overlap from the grid of images within the ward, we need to obtain the length of the sampled images in the y- and the x-direction.<jupyter_code>## Radius of squares in grid, figured out by mixture of observation and calculation
Y_RAD = 0.00385
X_RAD = 0.0043<jupyter_output><empty_output><jupyter_text>Adjustment to crop out watermarksAlthough we need images of dimension 400x400, we are sampling images of size 400x425 (more pixels in the y-direction) and cropping the bottom 25px in order to remove the Google watermarks - resulting in a 400x400 image. However, to do this, we need to shift the center of our sampled images slighly lower, as we are removing pixels from only the bottom of the image, even though there are $\approx 12.5$ pixels at the top of the image that are also being added. Thus, we need to shift the center down "12.5 pixels". Converting this to lat/lng values, we get that:$200$px $\approx 0.00385^{\circ}$. So it follows that $12.5$px $\approx 0.00385^{\circ}/16$<jupyter_code>DELTA_Y = Y_RAD/16  # adjustment to crop out watermarks
# specific municipality to extract images for
MUNICIPALITY = 'Westonaria'
BASE_DIR = '../..'
DATA_DIR = os.path.join(BASE_DIR, 'data')
IMAGE_CENTER_DIR = os.path.join(DATA_DIR, 'image_centers')
SHAPE_DIR = os.path.join(DATA_DIR, 'shapefiles', MUNICIPALITY)
IMAGE_DIR = os.path.join(DATA_DIR, 'images', MUNICIPALITY)
# get the Google Static Maps API key
with open(os.path.join(BASE_DIR, "google_api_key.txt"), "r") as file:
    KEY = file.readline()

# KEY = "YOUR_API_KEY"<jupyter_output><empty_output><jupyter_text>Extract Images for each Ward  1. Read in the centers for each image<jupyter_code># read in ward image centers
df_ward_centers = pd.read_csv(os.path.join(IMAGE_CENTER_DIR, "{}_ward_image_centers.csv".format(MUNICIPALITY))) 
print(df_ward_centers.dtypes)
df_ward_centers.head()
N_WARDS = df_ward_centers.shape[0]
N_WARDS<jupyter_output><empty_output><jupyter_text>2., 3., 4.  Extracting three images for each adjusted center<jupyter_code>def download_img(center_str, path, col=1):
    if col == 1: # black ward and plain image
        url = "https://maps.googleapis.com/maps/api/staticmap?center={}&zoom=16&size=400x425&maptype=satellite&format=png&path=fillcolor:0x000000FF|color:0x000000FF|weight:1{}&key={}".format(center_str, path, KEY)
    elif col == 2: # green ward
        url = "https://maps.googleapis.com/maps/api/staticmap?center={}&zoom=16&size=400x425&maptype=satellite&format=png&path=fillcolor:0x00FF00FF|color:0x00FF00FF|weight:1{}&key={}".format(center_str, path, KEY)

    # HTTP requests
    response = requests.get(url)
    response.raise_for_status()
    return(response)
def get_imgs(center_str, ward_num, pos, path):
    
    # extrecting two images for each center
    response_with_ward = download_img(center_str, path)              # image with ward overlayed in black
    response_with_ward_green = download_img(center_str, path, 2)     # image with ward overlayed in green
    response_no_ward = download_img(center_str, "")                  # plain image
    
    if not os.path.exists(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num))):   # creating ward directory if it doesn't exist
        os.makedirs(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num)))
    
    # storing the responses in a file (images)
    
    with open(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), str(pos[0]) + "_" + str(pos[1]) + "_with_ward.png"), 'wb') as file:
       # writing data into the file
       file.write(response_with_ward.content)
        
    with open(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), str(pos[0]) + "_" + str(pos[1]) + "_with_ward_green.png"), 'wb') as file:
       # writing data into the file
       file.write(response_with_ward_green.content)

    with open(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), str(pos[0]) + "_" + str(pos[1]) + ".png"), 'wb') as file:
       # writing data into the file
       file.write(response_no_ward.content)
# function for converting coordinates to format for Google Static Maps API
def center_to_string(center):
    return str(center[0]) + "," + str(center[1])
# Opening JSON file of ward shapes
def get_all_ward_shapes(file_name):
    f = open(file_name,) 
    shape_dict = json.load(f)                     # Returns JSON object as a dictionary 
    f.close()
    return shape_dict
# Extracting coordinates for the ward
def get_ward_shape(shape_dict, ward_num):
    coords = shape_dict['features'][ward_num]["geometry"]["coordinates"][0]
    return coords
# Constructing path for ward outline in the Google Static Maps Api image.
def coords_to_path(coords):
    path = ""
    for lat, lng in coords:
        path = path + "|" + str(lng) + "," + str(lat)
    return path
# dictionary of coordinates for all the wards
shape_file_dict = get_all_ward_shapes(os.path.join(SHAPE_DIR, "{}_shp.json".format(MUNICIPALITY)))
# Checking Google Static Maps API path length retrictions. If this fails, look to 'simplify' the shapefile using mapshaper.
for ward_num in range(1, N_WARDS+1):
    coords = get_ward_shape(shape_file_dict, ward_num-1)
    path = coords_to_path(coords)
    print(len(path))
    assert len(path) < 8192, "path is too long: error for ward {}".format(ward_num)
def get_images_for_ward(ward_num):
    try:
        print('Starting extraction for ward {}'.format(ward_num))
        coords = get_ward_shape(shape_file_dict, ward_num-1)                                   # coordinates for ward outline
        path = coords_to_path(coords)                                                          # create path for ward outline
        assert len(path) < 8192, "path is too long: error for ward" + ward_num
        for i, row in enumerate(ast.literal_eval(df_ward_centers["img_centers"][ward_num-1])):
            for j, row_center in enumerate(row):
                center_shifted = (row_center[0] - DELTA_Y, row_center[1])                # shifting the center down by DELTA_Y
                get_imgs(center_to_string(center_shifted), ward_num, (i,j), path)      # uncomment to download images
        print('Images for ward {} extracted'.format(ward_num))
    except Exception as e:
        print(f'EXCEPTION: {e}')
        print(f'EXCEPTION: occured in ward {ward_num} in position ({i},{j})')
        print(f'EXCEPTION: thread shutting down')
        raise e<jupyter_output><empty_output><jupyter_text>Can either extract the images in parallel or sequentially.**Note:** Downloading the images can take a **very long time**.  Parallel<jupyter_code>print(mp.cpu_count())
## Parallel execution
pool = mp.Pool(mp.cpu_count())
for ward_num in range(1, N_WARDS+1):
    pass
#     pool.apply_async(get_images_for_ward, args=(ward_num,)) # uncomment to download images
pool.close()
pool.join()<jupyter_output>Starting extraction for ward 1Starting extraction for ward 2Starting extraction for ward 3Starting extraction for ward 4Starting extraction for ward 5Starting extraction for ward 6Starting extraction for ward 7Starting extraction for ward 8







Images for ward 4 extracted
Starting extraction for ward 9
Images for ward 7 extracted
Starting extraction for ward 10
Images for ward 10 extracted
Starting extraction for ward 11
Images for ward 3 extracted
Starting extraction for ward 12
Images for ward 11 extracted
Starting extraction for ward 13
Images for ward 13 extracted
Starting extraction for ward 14
Images for ward 14 extracted
Starting extraction for ward 15
Images for ward 8 extracted
Starting extraction for ward 16
Images for ward 12 extracted
Images for ward 9 extracted
Images for ward 15 extracted
Images for ward 6 extracted
Images for ward 2 extracted
Images for ward 16 extracted
Images for ward 5 extracted
Images for ward 1 extracted<jupyter_text>Sequentially<jupyter_code># Dowload images from Google Static Maps API for each ward
# Save in directory /data/images/MUNCIPALTIY/ward_i/
# Three images from same location: 
## row_col.png - plain img
## row_col_with_ward.png - img with ward overlayed in black
## row_col_with_ward_green.png - img with ward overlayed in green
# (row,col) are matrix coordinates
# use ast.literal_eval: '[1,2]' -> [1,2]

for ward_num in range(1, N_WARDS+1):
    print('Starting extraction for ward {}'.format(ward_num))
    coords = get_ward_shape(shape_file_dict, ward_num-1)                                   # coordinates for ward outline
    path = coords_to_path(coords)                                                          # create path for ward outline
    assert len(path) < 8192, "path is too long: error for ward" + ward_num
    for i, row in enumerate(ast.literal_eval(df_ward_centers["img_centers"][ward_num-1])):
        for j, row_center in enumerate(row):
            center_shifted = (row_center[0] - DELTA_Y, row_center[1])                # shifting the center down by DELTA_Y
            get_imgs(center_to_string(center_shifted), ward_num, (i,j), path)      # uncomment to download images
    print('Images for ward {} extracted'.format(ward_num))<jupyter_output>Starting extraction for ward 1
Images for ward 1 extracted
Starting extraction for ward 2
Images for ward 2 extracted
Starting extraction for ward 3
Images for ward 3 extracted
Starting extraction for ward 4
Images for ward 4 extracted
Starting extraction for ward 5
Images for ward 5 extracted
Starting extraction for ward 6
Images for ward 6 extracted
Starting extraction for ward 7
Images for ward 7 extracted
Starting extraction for ward 8
Images for ward 8 extracted
Starting extraction for ward 9
Images for ward 9 extracted
Starting extraction for ward 10
Images for ward 10 extracted
Starting extraction for ward 11
Images for ward 11 extracted
Starting extraction for ward 12
Images for ward 12 extracted
Starting extraction for ward 13
Images for ward 13 extracted
Starting extraction for ward 14
Images for ward 14 extracted
Starting extraction for ward 15
Images for ward 15 extracted
Starting extraction for ward 16
Images for ward 16 extracted<jupyter_text>Image Cropping and Mask Creation  5., 6., 7. Edit images, create mask and save<jupyter_code># For saving the edited image to ./images/ward_i/edited
def save_edited_img(ward_num, f_no_ward, img, mask):
    
    # create directories if they don't exist
    if not os.path.exists(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), 'edited')):
        os.makedirs(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), 'edited'))
    if not os.path.exists(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), 'masks')):
        os.makedirs(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), 'masks'))
    
    # writing the images to directories
    cv2.imwrite(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), 'edited', f_no_ward), img)
    cv2.imwrite(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), 'masks', f_no_ward), mask)
def transform_to_arr(inds):
    x_list, y_list = inds
    inds_transformed = [(i,j) for i,j in zip(x_list, y_list)]
    return(inds_transformed)
def get_intersecting_points(inds1_transformed, inds2_transformed):
    inter = set(inds1_transformed).intersection(set(inds2_transformed))
    return inter
def get_union_points(inds1_transformed, inds2_transformed):
    union = list(set().union(inds1_transformed, inds2_transformed))
    return union
def recreate_pix_object(inter):
    x_arr = []
    y_arr = []
    for xy in inter:
        x, y = xy
        x_arr.append(x)
        y_arr.append(y)
    return((np.array(x_arr).astype(int), np.array(y_arr).astype(int)))<jupyter_output><empty_output><jupyter_text>Note on get_pix_in_ward and get_pix_out_wardThe result of np.where() is a two-dimensional tuple of two numpy arrays: the first array contains the x-values for the pixels and the second array contains the y-values for the pixels. For example, pixels [1,2] and [3,4] would be represented as ([1,3],[2,4]). Finding the intersection and union of objects of this form proved to be challenging. Thus, we first tranformed the indices into a form that was easier to work with: the above pixels were transformed to be represented as [(1,2), (3,4)]. We could then apply built-in union and intersection functions. However, we then needed to transform the pixels back to their original representation.<jupyter_code>def get_pix_in_ward(inds1, inds2):
    inds_transformed_1 = transform_to_arr(inds1)
    inds_transformed_2 = transform_to_arr(inds2)
    inter = get_intersecting_points(inds_transformed_1, inds_transformed_2)
    return recreate_pix_object(inter)
def get_pix_out_ward(inds1, inds2):
    inds_transformed_1 = transform_to_arr(inds1)
    inds_transformed_2 = transform_to_arr(inds2)
    union = get_union_points(inds_transformed_1, inds_transformed_2)
    return recreate_pix_object(union)
# For setting the area of the img outside of the ward to black
# Edited image saved to data/images/MUNICIPALITY/ward_i/edited
def isolate_ward_in_img(ward_num, f_with_ward, f_with_ward_green, f_no_ward):
    
    # reading in images
    img_with_ward = cv2.imread(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), f_with_ward))
    img_with_ward_green = cv2.imread(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), f_with_ward_green))
    img_no_ward = cv2.imread(os.path.join(IMAGE_DIR, 'ward_' + str(ward_num), f_no_ward))

    # cropping out the watermarks
    img_with_ward_cropped = img_with_ward[0:400,:,:]                  
    img_with_ward_green_cropped = img_with_ward_green[0:400,:,:]  
    img_no_ward_cropped = img_no_ward[0:400,:,:] 

    # selecting all black pixels and all green pixels
    inds1 = np.where(np.all(img_with_ward_cropped == [0,0,0], axis=-1))
    inds2 = np.where(np.all(img_with_ward_green_cropped == [0,254,0], axis=-1))

    # selecting all non-black pixels and all non-green pixels
    inds1_not = np.where(~np.all(img_with_ward_cropped == [0,0,0], axis=-1))
    inds2_not = np.where(~np.all(img_with_ward_green_cropped == [0,254,0], axis=-1))

    pix_in_ward = get_pix_in_ward(inds1, inds2)             # getting the pixels in the ward
    pix_out_ward = get_pix_out_ward(inds1_not, inds2_not)   # getting the pixels out of the ward

    # don't want to save image if covers none of the ward area
    if len(pix_in_ward[0]) == 0 and len(pix_in_ward[1]) == 0:
#         print("Ward {}: no ward in image {}, not saving.".format(ward_num, f_no_ward))
        return 0, 0

    # creating mask
    img_with_ward_cropped[pix_in_ward] = [255,255,255]  # setting pixels out ward to pure black for the mask
    img_with_ward_cropped[pix_out_ward] = [0,0,0]       # setting pixels in ward to pure white for the mask

    # return edited plain image and mask
    return img_no_ward_cropped, img_with_ward_cropped<jupyter_output><empty_output><jupyter_text>Note on code chunk belowWe encountered an issue with looping through the files in the directory because the order of the images sometimes would change. Thus, we needed to define a standardized order for walking through the files. This was done by walking through all the columns of the first row, then moving onto the second row and doing the same, and continuing this until there were no more rows.Thus, the outer while loop iterates over the rows and the inner while loop iterates over the columns within the row. Once again, since this chunk can take **very long** (because it has to go through thousands of images) a sequential and parallel option have been provided.  Sequential<jupyter_code># Looping through the directories for each ward, editing the images, creating the masks, then saving
# the results in two directories ./images/ward_i/edited and ./images/ward_i/masks
# which contain the editted plain images and masks for each image, respectively.
for ward_num in range(1, N_WARDS+1):
    print(f'starting ward {ward_num}')
    directory = os.fsencode(os.path.join(IMAGE_DIR, "ward_" + str(ward_num)))
    i = 0        # row value in grid
    j = 0        # column value in grid
#     f_with_ward = f_no_ward = f_with_ward_green = ''
    file_name = os.fsencode("{}_{}.png".format(i,j))                                                            # getting filename at the row and col position
    files_in_dir = os.listdir(directory)
    while (file_name in files_in_dir):                                                                          # checking if filename exists in directory
        while(file_name in files_in_dir):                                                                       # checking if filename exists in directory
            f_no_ward = os.fsdecode(file_name)                                                                  # decoding filename to string
            f_with_ward = "{}_{}_with_ward.png".format(i,j)                                                     # getting image with ward in black
            f_with_ward_green = "{}_{}_with_ward_green.png".format(i,j)                                         # getting image with ward in green
            img_plain_cropped, mask = isolate_ward_in_img(ward_num, f_with_ward, f_with_ward_green, f_no_ward)  # cropping, editing and getting mask
            if type(img_plain_cropped) != int and type(mask) != int:                                            # check if image contains portion of ward
                pass
                save_edited_img(ward_num, f_no_ward, img_plain_cropped, mask)                                   # saving edited image and mask

            j += 1
            file_name = os.fsencode("{}_{}.png".format(i,j)) 
        j = 0
        i += 1
        file_name = os.fsencode("{}_{}.png".format(i,j)) 
    print(f'finished ward {ward_num}')<jupyter_output>starting ward 1
finished ward 1
starting ward 2
finished ward 2
starting ward 3
finished ward 3
starting ward 4
finished ward 4
starting ward 5
finished ward 5
starting ward 6
finished ward 6
starting ward 7
finished ward 7
starting ward 8
finished ward 8
starting ward 9
finished ward 9
starting ward 10
finished ward 10
starting ward 11
finished ward 11
starting ward 12
finished ward 12
starting ward 13
finished ward 13
starting ward 14
finished ward 14
starting ward 15
finished ward 15
starting ward 16
finished ward 16<jupyter_text>Parallel<jupyter_code># Looping through the directories for each ward, editing the images, creating the masks, then saving
# the results in two directories ./images/ward_i/edited and ./images/ward_i/masks
# which contain the editted plain images and masks for each image, respectively.
def crop_mask_save(ward_num):
    try:
        print(f'starting ward {ward_num}')
        directory = os.fsencode(os.path.join(IMAGE_DIR, "ward_" + str(ward_num)))
        i = 0        # row value in grid
        j = 0        # column value in grid
    #     f_with_ward = f_no_ward = f_with_ward_green = ''
        file_name = os.fsencode("{}_{}.png".format(i,j))                                                            # getting filename at the row and col position
        files_in_dir = os.listdir(directory)
        while (file_name in files_in_dir):                                                                          # checking if filename exists in directory
            while(file_name in files_in_dir):                                                                       # checking if filename exists in directory
                f_no_ward = os.fsdecode(file_name)                                                                  # decoding filename to string
                f_with_ward = "{}_{}_with_ward.png".format(i,j)                                                     # getting image with ward in black
                f_with_ward_green = "{}_{}_with_ward_green.png".format(i,j)                                         # getting image with ward in green
                img_plain_cropped, mask = isolate_ward_in_img(ward_num, f_with_ward, f_with_ward_green, f_no_ward)  # cropping, editing and getting mask
                if type(img_plain_cropped) != int and type(mask) != int:                                            # check if image contains portion of ward
    #                 pass
                    save_edited_img(ward_num, f_no_ward, img_plain_cropped, mask)                                   # saving edited image and mask

                j += 1
                file_name = os.fsencode("{}_{}.png".format(i,j)) 
            j = 0
            i += 1
            file_name = os.fsencode("{}_{}.png".format(i,j)) 
        print(f'finished ward {ward_num}')
    except Exception as e:
        print(f'EXCEPTION: {e}')
        print(f'EXCEPTION: occured in ward {ward_num} in position ({i},{j})')
        print(f'EXCEPTION: thread shutting down')
## Parallel execution
pool = mp.Pool(mp.cpu_count())
for ward_num in range(1, N_WARDS+1):
    pass
#     pool.apply_async(crop_mask_save, args=(ward_num,))   
pool.close()
pool.join()<jupyter_output>starting ward 2starting ward 1starting ward 3starting ward 5starting ward 6starting ward 4starting ward 7starting ward 8







finished ward 4
starting ward 9
finished ward 7
starting ward 10
finished ward 10
starting ward 11
finished ward 3
starting ward 12
finished ward 11
starting ward 13
finished ward 13
starting ward 14
finished ward 14
starting ward 15
finished ward 8
starting ward 16
finished ward 12
finished ward 9
finished ward 15
finished ward 6
finished ward 2
finished ward 16
finished ward 5
finished ward 1
{'fraction_non_alphanumeric': 0.08044005688948382, 'fraction_numerical': 0.019576675311637246, 'mean_word_length': 3.295185052102048, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 42, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 48, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '567574', 'license': 'MIT', 'n_tokens_mistral': 7532, 'n_tokens_neox': 6948, 'n_words': 2521, 'path': 'scripts/feature_extraction/get_images.ipynb', 'repo_name': 'ANDRYA005/Clustering-of-Ward-Level-Deprivation-through-the-use-of-Satellite-Imagery', 'size': '23906'}