<jupyter_start><jupyter_text>Code to prepare data for use in an OWG  Resize images and write to a seperate folder<jupyter_code>import os
import shutil
from PIL import Image
import numpy as np
from distutils.dir_util import copy_tree
import matplotlib.pyplot as plt
import time
def estimate_sharpness(img): 
    """
    Estimate image sharpness 
    Input:
        img - np.ndarray representing image
              img read as skimage.io.imread( 'imagefilename.jpg' )
    Returns:
        s - sharpness estimates

    adapted from https://stackoverflow.com/questions/6646371/detect-which-image-is-sharper
    """
    gy, gx = np.gradient(img)
    gnorm = np.sqrt(gx**2 + gy**2)
    sharpness = np.average(gnorm)
    return sharpness
# edited to remove the part where they convert to grayscale as my images already are grayscale. Removed contrast because I do not need it
# create new folder
newpath = "D:/Webcam Images/MI City/2017_imgprep"
#os.mkdir(newpath)
fromDirectory = "D:/Webcam Images/MI City/2017"
toDirectory = "D:/Webcam Images/MI City/2017_imgprep"
shutil.copy_tree(fromDirectory, toDirectory)
# create seperate folders for different views
view1path = "D:/Webcam Images/MI City/2017_imgprep/view1"
view2path = "D:/Webcam Images/MI City/2017_imgprep/view2"
os.mkdir(view1path)
os.mkdir(view2path)
# move images to correct folders and remove the view identifier
for filename in os.listdir(newpath):
    if filename.endswith(".01.jpg"):
        os.rename("D:/Webcam Images/MI City/2017_imgprep/{}".format(filename),"D:/Webcam Images/MI City/2017_imgprep/view1/{}".format(filename.replace(".01","")))
    elif filename.endswith(".02.jpg"):
        os.rename("D:/Webcam Images/MI City/2017_imgprep/{}".format(filename),"D:/Webcam Images/MI City/2017_imgprep/view2/{}".format(filename.replace(".02","")))
# crop images to square, convert to grayscale, and reduce resolution to 128x128
import matplotlib.pyplot as plt                       
import numpy as np
from PIL import Image

image = Image.open("C:/njc/src/mcyimgs/2019/20191011_1233.02.jpg")
bw = image.convert("L")
bwarray = np.array(bw)
print(len(np.shape(bwarray)))

arrayshape = np.index_exp[270:720, 830:1280]
small = bwarray[arrayshape]
print(np.shape(small))

plt.imshow(small, cmap = "gray")
print(np.shape(small))<jupyter_output><empty_output><jupyter_text>Make practice code into functions<jupyter_code>year = 2018
directory = "D:/Webcam Images/MI City"
newdir = directory+"/{}_imgprep".format(str(year))
print (newdir+"/view1")<jupyter_output><empty_output><jupyter_text>Note:  FunctionalityThe function below is specifically for the analysis of view \2 in MI City. This works with imagery from the old webcam location. It is designed for a proof of concept application of the OWG. This function will need to be adapted or rewritten to deal with contemporary datsets. Also, a record of the inputs to each version should be kept for research/repeatability purposes. Next StepsThe most important additon that is needed is a way to automate the creation of view folders within the new year_imgprep folder. This could be done by including a list of numbers as strings representing the camera view as an input, or using string parsing and appending the target string-numbers into a list within the function. The latter option is likely best as it requires no additonal inputs be added to the funciton<jupyter_code>m = [1,2,3,4]
np.mean(m)
"OWG" == "OWG"
y =[10]
for y in y:
    x =5

three = [3]

for three in three:
    print (x)<jupyter_output><empty_output><jupyter_text>Function version 1.5<jupyter_code>def prepowgimgs(year, directory, arrayshape, resolution, version):
    '''Create a new directory of images and prepare them for further data filtering. version 1.5
    Year is the target year of images, 
    directory is where the year files are stored,
    resolution is the resolutiont the images will be reduced to, 
    version is the version that these inputs are associate with. A record of this should be kept by hand. 
    '''
    
    #set counters 
    counter = 0
    nightcounter = 0
    blurcounter = 0
    failcounter = 0
    
    # create a new directory with the called year_imgprep 
    print("Creating folder")
    foldername = "/{}_imgprep".format(str(year))
    newdir = directory+foldername+"{}".format(version)
    try:
        shutil.rmtree(newdir)
        print("Folder found: deleting contents")
        os.mkdir(newdir)
    except:
        os.mkdir(newdir)

    # copy files into new directory, this can take a while
    print("Moving Files")
    copy_tree(directory+"/{}".format(str(year)), newdir)
    
    # seperate the views into different folders within the directory
    print("Seperating Views")
    view1path = newdir+"/view1"
    view2path = newdir+"/view2"
    os.mkdir(view1path)
    os.mkdir(view2path)
    for filename in os.listdir(newdir):
        if filename.endswith(".01.jpg"):
            os.rename(newdir+"/{}".format(filename), view1path+"/{}".format(filename.replace(".01","")))
        elif filename.endswith(".02.jpg"):
            os.rename(newdir+"/{}".format(filename), view2path+"/{}".format(filename.replace(".02","")))
    
    # edit images from desired view into cropped lower resolution grayscale and store in new folder
    print ("Editing images for analysis")
    for filename in os.listdir(view2path):
        if filename.endswith(".jpg"):
            image = Image.open(view2path+"/{}".format(filename))
            
            #convert image to grayscale
            bw = image.convert("L")
            
            # select target portion of image to make it square
            bwarray = np.array(bw)
            small = bwarray[arrayshape]
            small_img = Image.fromarray(small)
            
            # change image resoltuion
            resize = (resolution, resolution)
            smaller = small_img.resize(resize)
            
            # Average image intensity
            avgintensity = np.mean(plt.imread(smaller))
                                   
            # Image Sharpness
            sharpness, contrast = estimate_sharpness(plt.imread(smaller))
        
            # if avg image intensity is below 40 (meaning it was taken at night) toss out image
            if avgintensity < 40:
                nightcounter = nightcounter + 1
        
            # if image sharpness is less than 3.5 (the mean of sharpness for all images in 2017) toss out imaage
            elif sharpness < 3.5:
                blurcounter = blurcounter + 1
            
            else:
                # save images ready for interpolation and filtering as the same name sans underscore seperating date and time
                try:
                    smaller = smaller.save(view2path+"/"+filename)
                except IOError:
                    print("cannot create image for", filename)
                    failcounter = failcounter + 1
            
            # remove the underscore in the filename
            os.rename(view2path+"/"+filename, view2path+"/"+filename.replace("_",""))
                
                
    print (failcounter, "files in folder could not be processed: IOError")
    print(blurcounter, "images removed from training dataset due to insufficient sharpness score")
    print(nightcounter, "images removed from training dataset due to inusfficeint intensity score")
    print (nightcounter+blurcounter+failcounter, "total images removed from training dataset")
    print (counter, "images moved to OWG training folder")
        
    return print ("Completed.")<jupyter_output><empty_output><jupyter_text>Function version 2.0<jupyter_code>def prepowgimgs(year, directory, arrayshape, resolution, trial, viewlist):
    '''Create a new directory for images in each view and run QC and primary augmentation before interpolation and then model 
    training, validation, or testing. version 2.0
    
    Year is the target year of images, 
    directory is where the year files are stored,
    resolution is the resolutiont the images will be reduced to, 
    trial is the name of the trial that these inputs are associate with. A record of trial details should be kept by hand and stored
    in C:/njc/src/SSF/OWG/OWG_records/trial. 
    viewlist is a list of the view numbers found in the target year folder
    '''
    start = time.time()
    #set counters 
    counter = 0
    nightcounter = 0
    blurcounter = 0
    failcounter = 0
    
    # create a new directory with the called year_imgprep 
    foldername = "/{}_imgprep".format(str(year))
    newdir = directory+foldername+"{}".format(trial)
    try:
        shutil.rmtree(newdir)
        print("Folder found: deleting contents")
        print("Creating folder")
        os.mkdir(newdir)
    except:
        print("Creating folder")
        os.mkdir(newdir)

    # copy files into new directory, this can take a while
    print("Moving Files")
    copy_tree(directory+"/{}".format(str(year)), newdir)
    
    # seperate the views into different folders within the directory
    viewdirs = []
    for view in viewlist:
        print("Seperating View {}".format(view))
        viewpath = newdir+"/view{}".format(view)
        viewdirs.append(viewpath)
        os.mkdir(viewpath) 
    
    
    for viewpath in viewdirs:
        print ("Moving images to {}".format(viewpath))
        # move images into views
        for filename in os.listdir(newdir):
            if filename[-5] == viewpath[-1]:
                os.rename(newdir+"/{}".format(filename), viewpath+"/{}".format(filename.replace(filename[13:16],"")))   
        
        # edit images from desired view into cropped lower resolution grayscale and store in new folder
        print ("Augmenting and filtering imgaes in {}".format(viewpath))
        for filename in os.listdir(viewpath):
            if filename.endswith(".jpg"):
                image = Image.open(viewpath+"/{}".format(filename)).convert("L")
            
                # select target portion of image to make it square
                bwarray = np.array(image) 
                small = bwarray[arrayshape]
                
                # change image resoltuion
                img = Image.fromarray(small)
                resize = (resolution, resolution)
                smaller = img.resize(resize)
               
                #Get QC metrics
                sharpness = estimate_sharpness(np.asarray(smaller))
                avgintensity = np.mean(np.asarray(smaller))
                
                # perform QC
                if sharpness < 3.5 or avgintensity < 85:
                    os.remove(viewpath+"/"+filename)
                    failcounter = failcounter + 1
                else:
                    # save images to OWG directory
                    try:
                        counter = counter + 1
                        owgimage = smaller.save(viewpath+"/{}".format(filename))
                    except IOError:
                        print("cannot create image for", filename)
                        failcounter = failcounter + 1
        
                    # remove the underscore in the filename
                    os.rename(viewpath+"/"+filename, viewpath+"/"+filename.replace("_",""))
                    
    print (failcounter, "images removed for quality control")
    print (counter, "images processed")
    end = time.time()   
    return print ("Completed in {} minutes.".format((end - start)/60))<jupyter_output><empty_output><jupyter_text>Test function version 2.0<jupyter_code>year = 2017
directory = "C:/njc/src/mcyimgs"
arrayshape = np.index_exp[100:700, 680:1280]
resolution = 512
trial = 'test'
viewlist = [1,2]
prepowgimgs(year, directory, arrayshape, resolution, trial, viewlist)
photo = "C:/njc/src/mcyimgs/2017_imgpreptest/view2/201712271234.jpg"
pic = plt.imread(photo)
print (estimate_sharpness(pic))
print(np.mean(pic))
x = plt.imread("C:/njc/src/mcyimgs/2017_imgpreptest/view2/201712270033.jpg")
y = np.asarray(x)
z = np.mean(y)
print (z)<jupyter_output><empty_output><jupyter_text>Move October images into a seperate folder for first analysis<jupyter_code>import os
from shutil import copyfile

directory = "D:/Webcam Images/MI City/2017_imgprep/view2/view2OWGimg"
newdir = directory+"/october2017"
os.mkdir(newdir)

for filename in os.listdir(directory):
    if filename[4:6] == "10":
        shutil.copyfile("D:/Webcam Images/MI City/2017_imgprep/view2/view2OWGimg"+"/"+filename, newdir+"/"+filename)<jupyter_output><empty_output><jupyter_text>Create a new folder of training images at a higher resolution<jupyter_code>year = 2017
directory = "C:/njc/src/mcyimgs"
arrayshape = np.index_exp[100:700, 680:1280]
resolution = 256
version = 1
prepmicityimgs(year, directory, arrayshape, resolution, version)
year = 2019
directory = "C:/njc/src/mcyimgs"
arrayshape = np.index_exp[270:720, 830:1280]
resolution = 256
version = "test"
prepmicityimgs(year, directory, arrayshape, resolution, version)<jupyter_output><empty_output><jupyter_text>Make a Fall Data Directory<jupyter_code>directory = "C:/njc/src/mcyimgs/2019_imgpreptest"
newdir = directory+"/view2fall"
os.mkdir(newdir)

for filename in os.listdir(directory+"/"+"view2"):
    if filename[4:6] == "09" or filename[4:6] == "10":
        shutil.copyfile(directory+"/"+"view2"+"/"+filename, newdir+"/"+filename)<jupyter_output><empty_output><jupyter_text>Select out random images from a year to use as validation datacsv's generated from functions in csv_interpolation notebook<jupyter_code>import pandas as pd
df = pd.read_csv('C:/njc/src/SSF/OWG/mcyv22017trial3.csv', sep = ",", index_col=False)
df
# random sample ~30%
dfs = df.sample(n= int(.3*819))
dfs.reset_index(drop=True, inplace=True)
dfs
dfs.to_csv('C:/njc/src/SSF/OWG/mcyv22017(30trial3).csv', sep = ",")<jupyter_output><empty_output><jupyter_text>Drop rows of original dataframe if they have been randomly selected for validationthis creates the csv file that will be used to train the OWG<jupyter_code># load in the dataframes
total = pd.read_csv('C:/njc/src/SSF/OWG/mcyv22017trial3.csv', sep = ",", index_col=False)
validation = pd.read_csv('C:/njc/src/SSF/OWG/mcyv22017(30trial3).csv', sep = ",", index_col=False)

# if the imagename in the total dataframe exists in the validation dataframe do not include it in the train dataframe. 
# the ~ inverts the boolean indexing, which we need to do because only True values will be kept and they start as False as they are not in the validation dataframe
train = total[~total.id.isin(validation.id)]
train
train.to_csv('C:/njc/src/SSF/OWG/mcyv22017trial3train.csv', sep = ",")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10300253060666165, 'fraction_numerical': 0.0241433554476438, 'mean_word_length': 3.000547195622435, 'pattern_counts': {'":': 2, '<': 49, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '497355', 'license': 'MIT', 'n_tokens_mistral': 4721, 'n_tokens_neox': 4284, 'n_words': 1287, 'path': 'OWG/Prepdata.ipynb', 'repo_name': 'ncata/SSF', 'size': '14621'}