<jupyter_start><jupyter_text>Find most relevant saildrone data to compare to nearby CTD data Basic workflow- choose a CTD location- determine a time-window of relevance around when the CTD was deployed to use to match to Saildrone data- find relevant saildrone data in that timewindow.  Determine distance to ctd location (use subsampled data to speed this process up)- determine if distance between two platforms is close enough for comparison- if data exists in proper time range and distance range, compare instruments- repeat for all ctd casts and all saildrones<jupyter_code>#standard python import of libraries needed (pandas and datetime)
import pandas as pd
import datetime<jupyter_output><empty_output><jupyter_text>the following code is a generic and simple *great circle* calculator and can be found at [https://github.com/CodeDrome/great-circle-distances-python](https://github.com/CodeDrome/great-circle-distances-python)<jupyter_code>import math

DEGREES_IN_RADIAN = 57.29577951
MEAN_EARTH_RADIUS_KM = 6371
KILOMETRES_IN_MILE = 1.60934


class GreatCircle(object):

    """
    This class has attributes for the names and locations of a pair of cities,
    and for their distances.
    After the city attributes are set, call calculate to set the distance attributes.
    Validation is carried out and the valid attributes set. This should be checked before using
    calculated attributes.
    """

    def __init__(self):

        """
        Create a set of attributes with default values.
        """

        self.name1 = None
        self.latitude1_degrees = 0
        self.longitude1_degrees = 0
        self.latitude1_radians = 0
        self.longitude1_radians = 0

        self.name2 = None
        self.latitude2_degrees = 0
        self.longitude2_degrees = 0
        self.latitude2_radians = 0
        self.longitude2_radians = 0

        self.central_angle_radians = 0
        self.central_angle_degrees = 0
        self.distance_kilometres = 0
        self.distance_miles = 0
        self.valid = False

    def calculate(self):

        """
        Central method to set calculated attributes, which it
        does by calling other private functions.
        """

        self.__validate_degrees()

        if self.valid:
            self.__calculate_radians()
            self.__calculate_central_angle()
            self.__calculate_distance()

    def __validate_degrees(self):

        """
        Check latitudes and longitudes are within valid ranges,
        setting the valid attribute accordingly.
        """

        self.valid = True

        if self.latitude1_degrees < -90.0 or self.latitude1_degrees > 90.0:
            self.valid = False

        if self.longitude1_degrees < -180.0 or self.longitude1_degrees > 180.0:
            self.valid = False

        if self.latitude2_degrees < -90.0 or self.latitude2_degrees > 90.0:
            self.valid = False

        if self.longitude2_degrees < -180.0 or self.longitude2_degrees > 180.0:
            self.valid = False

    def __calculate_radians(self):

        """
        Calculate radians from degrees by dividing by constant.
        """

        self.latitude1_radians = self.latitude1_degrees / DEGREES_IN_RADIAN
        self.longitude1_radians = self.longitude1_degrees / DEGREES_IN_RADIAN

        self.latitude2_radians = self.latitude2_degrees / DEGREES_IN_RADIAN
        self.longitude2_radians = self.longitude2_degrees / DEGREES_IN_RADIAN

    def __calculate_central_angle(self):

        """
        Slightly complex formula for calculating the central angle
        between two points on the surface of a sphere.
        """

        if self.longitude1_radians > self.longitude2_radians:
            longitudes_abs_diff = self.longitude1_radians - self.longitude2_radians
        else:
            longitudes_abs_diff = self.longitude2_radians - self.longitude1_radians

        self.central_angle_radians = math.acos( math.sin(self.latitude1_radians)
                                         * math.sin(self.latitude2_radians)
                                         + math.cos(self.latitude1_radians)
                                         * math.cos(self.latitude2_radians)
                                         * math.cos(longitudes_abs_diff))

        self.central_angle_degrees = self.central_angle_radians * DEGREES_IN_RADIAN

    def __calculate_distance(self):

        """
        Because we are using radians, this is a simple formula multiplying the radius
        by the angle, the actual units used being irrelevant.
        Also the distance in miles is calculated from kilometres.
        """

        self.distance_kilometres = MEAN_EARTH_RADIUS_KM * self.central_angle_radians

        self.distance_miles = self.distance_kilometres / KILOMETRES_IN_MILE
#read in saildrone data via pandas - use subsampled data to spead up distance calculations
sd_df = pd.read_csv('data/sd-1033_data_6hr.csv',parse_dates=True,index_col='TIM')
sd_df.sample()
#read in healy data via pandas
he1901_df = pd.read_excel('data/he1901_merged_final.xlsx',sheet_name=0,skiprows=34,parse_dates=True)<jupyter_output><empty_output><jupyter_text>***ONCE YOU KNOW WHATS GOING ON, YOU CAN DELETE THE CELLS FROM HERE TO WHERE I SAY KEEP CELLS*** Lets just start with *CTD001* from the healy1901 cruise.  I don't know if there is a valid saildrone point nearby but it will show the mathWe can use pandas to only retrieve the CTD of interest by subsetting on name/value pairs.  Its a pandas thing and pretty easy to pick up on.<jupyter_code>he1901_df[he1901_df['Cast'] == 1] #this is pandas speak, and only one way to do it, to get all casts that equal 1 and index on those rows only<jupyter_output><empty_output><jupyter_text>The results above are the entire profile, you are only interested in near surface so you could also say that P_4 must equal some value: see following cell<jupyter_code>he1901_df[he1901_df['Cast'] == 1][he1901_df['P_4'] == 4]<jupyter_output><ipython-input-52-51847829b33a>:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  he1901_df[he1901_df['Cast'] == 1][he1901_df['P_4'] == 4]<jupyter_text>or, without knowing the depth, but just wanting the shallowest (minimum pressure... as the .min() function on the 'P_4' column will do) you could say:<jupyter_code>he1901_df[he1901_df['Cast'] == 1][he1901_df['P_4'] == he1901_df[he1901_df['Cast'] == 1]['P_4'].min()] #i'm stringing together index calls here and strongly suggest you look at pandas/python examples for how to index dataframes if the above is confusing.<jupyter_output><ipython-input-53-d5965d1aa1f9>:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  he1901_df[he1901_df['Cast'] == 1][he1901_df['P_4'] == he1901_df[he1901_df['Cast'] == 1]['P_4'].min()] #i'm stringing together index calls here and strongly suggest you look at pandas/python examples for how to index dataframes if the above is confusing.<jupyter_text>If you do the above, looping over all ctd casts, you will get a new list of only the shallowest datapoints in each cast... but perhaps you already did this in another routine... lets move on, to calculating distanceTaking the CTD001 example above, you can see I have latitude and Longitude... and same is true of the saildrone data.  Lets choose a 24hour window (+/- 12hrs) and see if there is any saildrone data in that time period<jupyter_code>ctdsfc = he1901_df[he1901_df['Cast'] == 1][he1901_df['P_4'] == he1901_df[he1901_df['Cast'] == 1]['P_4'].min()] #save a variable for ease of calculations to follow

# this will report the datetime of the cast and will be the central point of the window we want to look for in time
# the [0] at the end chooses the array value for furthur purposes and the datetime function is used to add 12 hours and subtract 12 hours from the central time
(ctdsfc['Date Time']-datetime.timedelta(hours=12))[0]
ctd_sd_match = sd_df[(pd.to_datetime(ctdsfc['Date Time'])-datetime.timedelta(hours=12))[0]:(pd.to_datetime(ctdsfc['Date Time'])+datetime.timedelta(hours=12))[0]]
ctd_sd_match<jupyter_output><empty_output><jupyter_text>So the 4 rows shown above are the relevant time values for the saildrone data, but are they close enough? lets use the great circle calculator between each of these four points and the ctd location<jupyter_code>#defined at the beginning of this notebook, its a python method and has some unique ways to call its functions
gc = GreatCircle()

#pt 1
gc.name1 = 'HLY Cast'
gc.latitude1_degrees = ctdsfc['Latitude'][0]
gc.longitude1_degrees = ctdsfc['Longitude'][0]

#choose just the first SD matched in time location for now, python is zero indexed
gc.name1 = 'SD Location'
gc.latitude2_degrees = ctd_sd_match['LATITUDE'][0]
gc.longitude2_degrees = ctd_sd_match['LONGITUDE'][0]

gc.calculate()
print(f'Distance between points are {gc.distance_kilometres} kilometers')
#loop over all valid saildrone points at this time and
#defined at the beginning of this notebook, its a python method and has some unique ways to call its functions
gc = GreatCircle()

#pt 1
gc.name1 = 'HLY Cast'
gc.latitude1_degrees = ctdsfc['Latitude'][0]
gc.longitude1_degrees = ctdsfc['Longitude'][0]

for i,rows in ctd_sd_match.iterrows():
    gc.name1 = '<NAME>'
    gc.latitude2_degrees = rows['LATITUDE']
    gc.longitude2_degrees = rows['LONGITUDE']

    gc.calculate()
    print(f'Distance between points are {gc.distance_kilometres} kilometers for sd at time {i}')<jupyter_output>Distance between points are 1308.9943672921042 kilometers for sd at time 2019-08-05 18:00:00
Distance between points are 1310.030787552367 kilometers for sd at time 2019-08-06 00:00:00
Distance between points are 1302.02454701064 kilometers for sd at time 2019-08-06 06:00:00
Distance between points are 1294.6123417742713 kilometers for sd at time 2019-08-06 12:00:00<jupyter_text>***KEEP ALL CELLS BELOW*** So for the example of Saildrone 1033 and CTD001 on HLY1901 - the saildrone was ~1300km away from the the ctd in the window of 1day around the cast.So now the goal would be to loop over all casts for this one Saildrone and see if there are any points of interest.  I'm gonna do it in one cell... this means that if you want to perform analysis's on other saildrones or cruises, you aught to be able to just change the data files and then run this cell.  **Warning** Some previous cells that are SD-1033 specific will break if you do this blindly...**NOTE** for the saildrone data, it looks to have some information about the distance between the drones... you may be able to use this information also when looking at all drones or you may not ... up to you<jupyter_code>#we are going to use another pandas trick call groupby to associate unique casts together and then loop through each cast
he1901_df.groupby('Cast').groups.keys() #will show all the unique 'Cast' numbers
for groups in he1901_df.groupby('Cast').groups.keys():
    cast = he1901_df.groupby('Cast').get_group(groups)
    print(f'Working on CTD {groups}')
    #get relevant CTD info
    ctdsfc = he1901_df[he1901_df['Cast'] == groups][he1901_df['P_4'] == he1901_df[he1901_df['Cast'] == groups]['P_4'].min()]
    #get matching saildrone info
    ctd_sd_match = sd_df[(pd.to_datetime(ctdsfc['Date Time'])-datetime.timedelta(hours=12)).iloc[0]:(pd.to_datetime(ctdsfc['Date Time'])+datetime.timedelta(hours=12)).iloc[0]]

    #calc great circle distance
    gc = GreatCircle()

    #pt 1
    gc.name1 = '<NAME>'
    gc.latitude1_degrees = ctdsfc['Latitude'].iloc[0]
    gc.longitude1_degrees = ctdsfc['Longitude'].iloc[0]

    for i,rows in ctd_sd_match.iterrows():
        gc.name1 = 'SD Location'
        gc.latitude2_degrees = rows['LATITUDE']
        gc.longitude2_degrees = rows['LONGITUDE']

        gc.calculate()
        print(f'Distance between points are {gc.distance_kilometres} kilometers for sd at time {i}')<jupyter_output>Working on CTD 1
Distance between points are 1308.9943672921042 kilometers for sd at time 2019-08-05 18:00:00
Distance between points are 1310.030787552367 kilometers for sd at time 2019-08-06 00:00:00
Distance between points are 1302.02454701064 kilometers for sd at time 2019-08-06 06:00:00
Distance between points are 1294.6123417742713 kilometers for sd at time 2019-08-06 12:00:00
Working on CTD 2
Distance between points are 1309.9089507683927 kilometers for sd at time 2019-08-06 00:00:00
Distance between points are 1301.7614441050487 kilometers for sd at time 2019-08-06 06:00:00
Distance between points are 1294.3086916567295 kilometers for sd at time 2019-08-06 12:00:00
Distance between points are 1291.5239550855442 kilometers for sd at time 2019-08-06 18:00:00
Working on CTD 3
Distance between points are 1276.541748549079 kilometers for sd at time 2019-08-06 06:00:00
Distance between points are 1269.119471961215 kilometers for sd at time 2019-08-06 12:00:00
Distance between points [...]
{'fraction_non_alphanumeric': 0.08072896471500582, 'fraction_numerical': 0.06172935246219465, 'mean_word_length': 3.7976190476190474, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '207102', 'license': 'MIT', 'n_tokens_mistral': 4537, 'n_tokens_neox': 3926, 'n_words': 1361, 'path': '2021/Samantha_SailDrone/FindSaildroneNearCTD.ipynb', 'repo_name': 'shaunwbell/ipythonnb', 'size': '12902'}