<jupyter_start><jupyter_text>This notebook covers: 🌎 Sea Surface Temperature Data Collection 🌎 Sea Surface Temperature (SST) is of critical importance for 🌀 hurricanes 🌀, who knew!The study of **Rapid Intensification (RI)** has grown considerably in the last decade. Scientists studying RI hypothesize that several different environmental factors may dramtically increase the liklihood of an RI event occuring. One such environmental factor is **Sea Surface Temperature**. Since tropical stroms draw their energy from warm water, it follows that higher SST may lead to an increased rate of hurricanes, creating a larger change of RI events.I ran into data access issues when searching for daily SST data at every coordinate in the Carribbean, so I scraped it from the NOAA images available [here](https://coralreefwatch.noaa.gov/product/5km/index.phpdata_access).<jupyter_code>Image(filename='../data/SST_images/20211111.jpg') # SST on November 11th, 2021<jupyter_output><empty_output><jupyter_text>Below we get the date and geographic coordinates of each storm observation in the HURDAT data set, then scrape the pixel color from the corresponding daily NOAA SST image. Next, the color is compared to the color scale at the bottom of the image and converted back into a temperature. When selecting pixels, we sample nearby pixels and average the colors to get an approximate temperature for the pixel. If the eye of the storm is over land, the SST is returned as NaN.   1) Download and Calibrate the ImageIf the NOAA SST image does not exist we download it, otherwise the images are saved in the`data/SST_images` folder, one for every day. We always want to ensure that the image is always centered and properly size, so whenever we use an image we call `check_alignment()`, which checks several key pixel colors and returns `True` is the image is properly aligned.<jupyter_code>def get_sst_image_url(d):
    """ 
    return the url for the NOAA SST image for the given date; args: datetime; 
    """
    return f"https://coralreefwatch.noaa.gov/data/5km/v3.1_op/image/daily/sst/png/{d.year}/{d:%m}/coraltemp_v3.1_caribbean_{d.year}{d:%m}{d:%d}.png"
def save_image_to_file(url_, name):
    """ 
    save an image to current directory from url
    args: 
        url (string): of image
        name (string): to save file as
    returns: 
        None
    """
    urllib.request.urlretrieve(url_, name)
""" List of tests and answers to determine if the NOAA SST image is properly aligned """

a1 = (62,102); a1_ = (255, 255, 255, 255)
b1 = (63,103); b1_ = (10, 10, 10, 255)
c1 = (64,104); c1_ = (150, 150, 150, 255)

a2 = (1595,102); a2_ = (255, 255, 255, 255)
b2 = (1594,103); b2_ = (10, 10, 10, 255)
c2 = (1593,104); c2_ = (150, 150, 150, 255)

a3 = (1595,869); a3_ = (255, 255, 255, 255)
b3 = (1594,868); b3_ = (10, 10, 10, 255)
c3 = (1593,867); c3_ = (150, 150, 150, 255)

a4 = (62,869); a4_ = (255, 255, 255, 255)
b4 = (63,868); b4_ = (10, 10, 10, 255)
c4 = (64,867); c4_ = (150, 150, 150, 255)

tests_sst = [a1,b1,c1,a2,b2,c2,a3,b3,c3,a4,b4,c4]
answers_sst = [a1_,b1_,c1_,a2_,b2_,c2_,a3_,b3_,c3_,a4_,b4_,c4_]
def get_pix_object(name):
    """
    returns the 'pix' object from PIL module used to check pixel color values; args: file name;
    """
    im = PIL.Image.open(name).convert('RGB')
    pix = im.load()
    return pix
def check_alignment(pix, tests, answers):
    """ 
    ensure that the image is aligned with where we 'think' it is. this method runs through a list of tests checking
    if expected pixel value (of the map border) is where/what we expect it to be.
    args:
        pix (pix): image object to test pixel colors
        tests (list): list of tuples of test pixels for
        answers (list): list of tuples of pixel colors (i.e. the answer key)
    returns:
        boolean: True is image is aligned, False if not
    """
    for i in range(len(tests)):
        try:
            pix[tests[i]] == answers[i]
        except:
            return False
    return True<jupyter_output><empty_output><jupyter_text>2) Get Sea Surface Temperature from ImageThere are several useful methods defined below that make the interaction between real life and the NOAA SST image a bit easier. First, we can calculate the pixel location of the geogrpahic coordinates with `get_pixel_location_from_coords`. This is made much simpler because the image map uses constant scaling for both latitude and longitude, allowing us to calculate the position linearly, thank goodness!Refering back to the SST image above, you may notice the possibility that the storm-eye coordinates fall directly on a reference line of the map. This would return a color (say black, from the line color) that is not the actual SST (which should have yellow, or whatever). To avoid this, we consider all the pixels within a radius of our desired pixel (using `find_neighbors_within_K`), and then average those temperatures together, thus avoiding this reference line issue.<jupyter_code>def get_pixel_location_from_coords(coords, x, y, lat, lon):
    """
    convert the coordinates tuple (lat,lon) into a pixel location for the NOAA SST image;
    note we must be sure the image is aligned first (using check_alignment()) to ensure that the
    minimum and maximum x and y values are correct
    args:
        coords (tuple): a tuple of the coordinates to find the pixel location of (lat,lon)
        x (tuple, length 2): the minimum and maximum x-pixel values of the map
        y (tuple, length 2): the minimum and maximum y-pixel values of the map 
        lat (tuple, length 2): the minimum and maximum latitude values of the map 
        lon (tuple, length 2): the minimum and maximum longitude values of the map 
    returns:
        (pixel_x,pixel,y) (tuple): tuple of the pixel coordinates for the argument lat/lon coordinates
    """
    # set min/max pixel values based on defualt image locations
    x_min = x[0];  y_min = y[0];
    x_max = x[1];  y_max = y[1];
    
    # caclulate width and height of image
    width = x_max - x_min
    height = y_max - y_min
    
    # set min/max lat/lon boundaries displayed in image
    # note: this works becuase the image is orthogonal (no change in latitude size as we go north )
    lat_min = lat[0];  lon_min = lon[0];
    lat_max = lat[1];  lon_max = lon[1];
    
    # calculate the percentage of lat/lon we are in (i.e. 50deg = 0%, 75deg = 50%, 100deg = 100%)
    percent_lat = (coords[0] - lat_min)/(lat_max - lat_min)
    percent_lon = (coords[1] - lon_min)/(lon_max - lon_min)
    
    # calculate pixel positions based on previous values
    pixel_x = round(x_min + (percent_lon * width))
    pixel_y = round(y_min + (percent_lat * height))
    
    return pixel_x, pixel_y    
def find_neighbors_within_K(p0, K, x, y):
    """
    find all pixels within K pixels (euclidean distance) of the argument pixel p0;
    args:
        p0 (tuple, length 2): pixel of interest, we are finding neightbors of this pixel
        K (float): distance parameter, we are finding all pixels within K units of p0
        x (tuple, length 2): the minimum and maximum x-pixel values of the map
        y (tuple, length 2): the minimum and maximum y-pixel values of the map 
    returns:
        neighbors (list of tuple): the points that are with K units of p0
    """
    neighbors = [] # list to store pixels close to p0

    # based on defulat image position
    x_min = x[0];  y_min = y[0];
    x_max = x[1];  y_max = y[1];
    
    # calculate width and height
    width = x_max - x_min
    height = y_max - y_min
    
    # iterate through all pixels in image, checking if they are within K units of p0
    for i in range(width):
        x = x_min + i # x pos to check for
        for j in range(height):
            y = y_min + j # y pos to check for
            if calculate_distance_between_pixels(p0,(x,y)) < K:
                neighbors.append((x,y)) # add if distance is less than K
    return neighbors
def calculate_distance_between_pixels(a,b):
    """ 
    calculate the euclidean distance between two pixels;
    args: 
        a,b (tuples, length 2) pixels to test, e.g. a=(10,20)
    returns: 
        distance between pixels (float)
    """
    summ = 0
    for i in range(2): # calculate euclidean distance for 2 coordinates
        summ += abs(a[i] - b[i])**2
    return summ**0.5
def get_mean_temp_of_neighbors(pix, neighbors):
    """
    get the mean temperature of a list of pixels, approximating the temperature of a specific pixel;
    args:
        neighbors (list of tuples): pixels within a radius of our pixel of interest
    returns:
        temperature (float): the average temperature of all of th heighbor pixels
    """
    temps = np.array([]) # empty array to store temperature of pixels
    for n in neighbors:
        new_temp = get_temperature_from_color(pix[n[0],n[1]]) # get temperature of single pixel
        temps = np.append(temps, new_temp)
    if len(temps) == 0: return np.nan
    return round(np.nanmean(temps),2) # return mean of list
    
def get_temperature_from_color(col):
    """
    use the color_temp_list list (painstakingly made) to calculate the temperature (in C) from a color;
    args: color (tuples, length 3) the color to get the tempertature of
    returns: temperature of color (float) between 0 and 40C based off scale in example photo
    """
    min_dist = np.inf
    min_temp = None
    min_color = None
    # iterate through all colors to find which is the closest
    for i in color_temp_list: 
        dist = calculate_distance_between_colors(col,i[0])
        if dist < min_dist - 0.001:
            min_dist = dist
            min_temp = i[1]
            min_color = i[0]
    # colors should be very close to those in color scale
    # so if the distance is too large, assume an error occurred and return temp=0
    if min_dist > 25: 
        return np.nan
    else: 
        return min_temp
def calculate_distance_between_colors(a,b):
    """ 
    calculate the euclidean distance between two colors using R,G,B as a vector between 0 and 255;
    args: 
        a,b (tuples, length 3) colors to test, e.g. a=(255,0,255)
    returns: 
        distance between colors (float)
    """
    summ = 0
    for i in range(3): # calculate euclidean distance for 3 coordinates (R,G,B)
        summ += abs(a[i] - b[i])**2
    return summ**0.5
def get_temperature_from_date_location(row, year, month, day, lat, lon):
    """
    this combo method uses the provided date and location to esimate SST using pixel colors from NOAA images;
    first, we check if the image exists and is properly aligned; then we calcualte the pixel coordinates from
    the geographic coordinates; then average the temperature of the neighbor pixels to estimate SST;
    args:
        row: row from HURDAT2 observation dataframe, to be supplied by lambda function
        year (int): the year of the HURDAT2 observation row
        month (int): the month of the HURDAT2 observation row
        day (int): the day of the HURDAT2 observation row
        lat (float): the latitude of the HURDAT2 observation row
        lon (float): the longitude of the HURDAT2 observation row
    returns:
        sst (float): the Sea Surface Temperature at the supplied location and time; if not appropriate value is found,
        (e.g. location is over land) then NaN is returned 
    """
    # Show progress in SST calculations, HURDAT2 dataframe should have 52,000 observations
    if int(row.name)%1000 == 0:
        print(row.name)
        
    try:
        d = datetime.datetime(year, month, day)
        name = f"../data/SST_images/{d.year}{d:%m}{d:%d}.jpg"
        pix = get_pix_object(name) # get pix object of image
    except:
        return np.nan # if image does not exists return NaN
    
    # make sure image is properly aligned
    check = check_alignment(pix, tests_sst, answers_sst) 
    
    if check:
        coords = (lat,lon) # get coordinates from argument
    
        # get pixel location using characteristics of default image format
        # Note: these numbers only work for the specific NOAA image, and are liable to change without notice
        pixel = get_pixel_location_from_coords(coords, [63,1594], [103,868], [32.5,7.5], [100,50])
        
        # get neighbor pixels of the pixel of interest
        neighbors = find_neighbors_within_K(pixel,3, [63,1594], [103,868])
        
        # calculate mean temperature of pixels
        sst = get_mean_temp_of_neighbors(pix, neighbors)
        return sst
    else:
        print(f'Image not aligned, {year, month, day}')
        return np.nan<jupyter_output><empty_output><jupyter_text>Main<jupyter_code># Read HURDAT2 observation dataframe from pickle
dat = pd.read_pickle('12_15_2021.pkl')

# Create SST column and set default vale to NaN
dat['SST'] = np.nan
# Calculate SST by applying the lamba function to each row 
dat['SST'] = dat.apply(lambda row:get_temperature_from_date_location(
                            row,
                            row.DateTime.year, 
                            row.DateTime.month, 
                            row.DateTime.day, 
                            row.Lat,
                            -1*row.Lon),axis=1)
# Save dataframe with SST as new pickle
dat.to_pickle(f"HURDAT2_with_SST.pkl")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08392816890825756, 'fraction_numerical': 0.03208355248328199, 'mean_word_length': 3.0382281553398056, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 18, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '637028', 'license': 'MIT', 'n_tokens_mistral': 4314, 'n_tokens_neox': 3965, 'n_words': 1661, 'path': 'scripts/SST_Data_Collection.ipynb', 'repo_name': 'SamLeBlanc/Hurricane-RI-Analysis', 'size': '13309'}