<jupyter_start><jupyter_text>Import and sort all caravela apogee_SL510 eurec4a files, creating a dataframe from this:<jupyter_code>files_list = glob(r'E:\Eurec4a_master\Caravela\apogee_SP_110\*\*'+ '/*SP_110*')  
files_list.sort()

li = []

for filename in tqdm(files_list):
    df = pd.read_csv(filename, header=0, sep=',',index_col=False)
    li.append(df)

sp = pd.concat(li, axis=0)
sp<jupyter_output><empty_output><jupyter_text>Parse the timestamp to a datetime format<jupyter_code>dt = []
for i in tqdm(sp['PC Timestamp[UTC]']):
    dt.append(datetime.fromisoformat([i]))
sp['dt [UTC]'] = dt
sp = sp[(sp['dt [UTC]'] >= '2020-01-22 00:00:00.000')] #select data from Caravela's launch onwards
sp<jupyter_output><empty_output><jupyter_text>Tidy this up to drop the columns we dont need<jupyter_code>sp = sp.reset_index()
sp = sp.drop(['index','PC Time Zone','PC Timestamp[UTC]' ],axis=1)<jupyter_output><empty_output><jupyter_text>check for any gaps in timeseries larger than 2 seconds<jupyter_code>time_diff = sp['dt [UTC]'].values[1:] - sp['dt [UTC]'].values[:-1]
for i in np.arange(1, len(time_diff)):
    if np.timedelta64(3599, 's') >= time_diff[i] > np.timedelta64(2, 's'):
        print('gap starts at', sp['dt [UTC]'][i], 'and lasts for', 
              np.timedelta64(time_diff[i],'s'))
    if time_diff[i] > np.timedelta64(3599, 's'):
        print('gap starts at', sp['dt [UTC]'][i], 'and lasts for', 
              np.timedelta64(time_diff[i],'s'), ' - approximately', np.timedelta64(time_diff[i],'h'))<jupyter_output>gap starts at 2020-01-22 20:36:23.464000 and lasts for 7352 seconds  - approximately 2 hours
gap starts at 2020-01-22 22:46:44.264000 and lasts for 13222 seconds  - approximately 3 hours
gap starts at 2020-01-27 13:02:10.301000 and lasts for 89486 seconds  - approximately 24 hours
gap starts at 2020-01-29 17:47:36.976000 and lasts for 20 seconds
gap starts at 2020-01-31 18:08:06.895000 and lasts for 14 seconds
gap starts at 2020-02-01 15:29:12.123000 and lasts for 2 seconds
gap starts at 2020-02-04 10:53:07.094000 and lasts for 15 seconds
gap starts at 2020-02-07 08:06:33.486000 and lasts for 2 seconds
gap starts at 2020-02-09 21:46:59.899000 and lasts for 3 seconds
gap starts at 2020-02-13 11:54:22.136000 and lasts for 2 seconds
gap starts at 2020-02-14 05:56:17.835000 and lasts for 2 seconds
gap starts at 2020-02-14 11:01:33.949000 and lasts for 2 seconds
gap starts at 2020-02-21 20:45:34.363000 and lasts for 2 seconds
gap starts at 2020-02-24 14:20:10.688000 and lasts for 11986 seco[...]<jupyter_text>Convert to iso time as this is a universally accepted format<jupyter_code>a = []
for i in tqdm(range(0,len(sp['dt [UTC]']))):
    a.append(sp['dt [UTC]'][i].isoformat())
sp['datetime [UTC]'] = a
sp = sp.drop(['dt [UTC]'], axis = 1)
sp.to_csv('CARAVELA_SP110.csv',index = None)
baa = pd.read_csv('CARAVELA_SP110.csv')# import file to test it
z = []
for i in tqdm(baa['datetime [UTC]']):
    z.append(datetime.fromisoformat(i))
baa['dt'] = z
import matplotlib.pyplot as plt
fig,ax = plt.subplots(1,1, figsize=(18, 15))
ax.plot(baa['dt'], baa['SP-110-SS[W m-2]'])
ax.set_ylabel('SP-110-SS[W m-2]')
ax.set_xlabel('Date')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15478424015009382, 'fraction_numerical': 0.12320200125078173, 'mean_word_length': 4.946096654275093, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '494621', 'license': 'MIT', 'n_tokens_mistral': 1530, 'n_tokens_neox': 1225, 'n_words': 326, 'path': 'apogee_SP110.ipynb', 'repo_name': 'ESiddle/Eurec4a-Analysis-scripts-AERIS', 'size': '3198'}