<jupyter_start><jupyter_text>MADATORY PYTHON LIBRARIES<jupyter_code>import ftputil
import numpy as np
import pandas as pd
from IPython.display import Image
%matplotlib inline<jupyter_output><empty_output><jupyter_text>SET YOUR CREDENTIALS<jupyter_code>user = 'protllan' #type CMEMS user name
password = '<PASSWORD>' #type CMEMS password<jupyter_output><empty_output><jupyter_text>INSITU PRODUCT LIST There are 2 types of In Situ Products: Near Real Time (NRT) and Reprocessed (REP) products that can be found on CMEMS FTPs servers.<jupyter_code>data = {'In Situ NRT products': []}
NRT_products, REP_products = [], []
#connect to CMEMS FTP
with ftputil.FTPHost('nrt.cmems-du.eu', user, password) as ftp_host: 
    ftp_host.chdir('Core')
    product_list = ftp_host.listdir(ftp_host.curdir)
    for product in product_list:
        items = product.split('_')
        if 'INSITU' in items:
            data['In Situ NRT products'].append(product)
pd.DataFrame(data=data)
data = {'In Situ REP products': []}
NRT_products, REP_products = [], []
#connect to CMEMS FTP
with ftputil.FTPHost('my.cmems-du.eu', user, password) as ftp_host: 
    ftp_host.chdir('Core')
    product_list = ftp_host.listdir(ftp_host.curdir)
    for product in product_list:
        items = product.split('_')
        if 'INSITU' in items:
            data['In Situ REP products'].append(product)
pd.DataFrame(data=data)<jupyter_output><empty_output><jupyter_text>INSITU INDEX FILES In Situ products are nothing else that lots of netCDFs containing the observations from the In Situ network of platforms in a given region (7 posible regions). Theese netCDFs are oganized in histocial, monthly and latest directories depending if they store a platform whole-life-span observations (one file per platform), the last 5 years of observations (oner file per platform per month) or the last 30 days of observations (one file per platform per day).<jupyter_code>Image(filename='../images/ftpStruture.png')<jupyter_output><empty_output><jupyter_text>For each product you will have a serie of index files to ease the navigation through the different directories and subdirectories:<jupyter_code>data = { 'FTP Host': ['nrt.cmems-du.eu', 'my.cmems-du.eu'], 'index_history.txt': [' ✔', ' ✔'], 'index_monthly.txt': [' ✔', '-'],'index_latest.txt': [' ✔', '-']}
pd.DataFrame(data=data, index=['In Situ NRT products', 'In Situ REP products'])<jupyter_output><empty_output><jupyter_text>index_history.txt contains the ftplink and some extra metadata for the netCDFs produced by In Situ platform network since ever index_monthly.txt contains the ftplink and some extra metadata for the netCDFs produced by In Situ platform network during the last 5 yearsindex_latest.txt contains the ftplink and some extra metadata for the netCDFs produced by In Situ platform network during the last 30 days Look at their content:<jupyter_code>product_name = 'INSITU_MED_NRT_OBSERVATIONS_013_035'
host = 'nrt.cmems-du.eu'
index_file = 'index_history.txt'
#connect to CMEMS FTP
with ftputil.FTPHost(host, user, password) as ftp_host: 
    #open the index file to read
    with ftp_host.open("Core"+'/'+product_name+'/'+index_file, "r") as indexfile:
        raw_index_info = pd.read_csv(indexfile, skiprows=5) #load it as pandas dataframe
print('ftplink and some metadata of the netCDFs produced by In Situ platform network since ever')
raw_index_info.head()
product_name = 'INSITU_MED_NRT_OBSERVATIONS_013_035'
host = 'nrt.cmems-du.eu'
index = 'index_monthly.txt'
#connect to CMEMS FTP
with ftputil.FTPHost(host, user, password) as ftp_host: 
    #open the index file to read
    with ftp_host.open("Core"+'/'+product_name+'/'+index_file, "r") as indexfile:
        raw_index_info = pd.read_csv(indexfile, skiprows=5) #load it as pandas dataframe
print('ftplink and metadata of the netCDFs produced by In Situ platform network during the last 5 years')
raw_index_info.head()
product_name = 'INSITU_MED_NRT_OBSERVATIONS_013_035'
host = 'nrt.cmems-du.eu'
index = 'index_latest.txt'
#connect to CMEMS FTP
with ftputil.FTPHost(host, user, password) as ftp_host: 
    #open the index file to read
    with ftp_host.open("Core"+'/'+product_name+'/'+index_file, "r") as indexfile:
        raw_index_info = pd.read_csv(indexfile, skiprows=5) #load it as pandas dataframe
print('ftplink and metadata to the netCDFs produced by In Situ platform network during the last 30 days')
raw_index_info.head()
data = {'NRT_products': [], 'REP_products': []}

NRT_products, REP_products = [], []
#connect to CMEMS FTP
with ftputil.FTPHost('my.cmems-du.eu', user, password) as ftp_host: 
    ftp_host.chdir('Core')
    product_list = ftp_host.listdir(ftp_host.curdir)
    for product in product_list:
        items = product.split('_')
        if 'INSITU' in items:
            data['REP_products'].append(product)
pd.DataFrame(data=data)
REP_products = []
#connect to CMEMS FTP
with ftputil.FTPHost('my.cmems-du', user, password) as ftp_host: 
    ftp_host.chdir('Core')
    product_list = ftp_host.listdir(ftp_host.curdir)
    for product in product_list:
        items = product.split('_')
        if 'INSITU' in items:
            REP_products.append(product)
            data['Description'].append('<a href="http://marine.copernicus.eu/services-portfolio/access-to-products/?option=com_csw&view=details&product_id='+product+'" target="_blank"></a>')
<br>
These products are nothing else that lots of netCDFs containing the observations from the In Situ network of platforms in a given area (7 regions). 
Depending on the product you will have a serie of <i>index files</i> to better deal with the outstanding amount of avaialable netCDFs in a given product:
data = { 'FTP Host': ['nrt.cmems-du.eu', 'my.cmems-du'], 'index_history.txt': [' ✔', ' ✔'], 'index_monthly.txt': [' ✔', '-'],'index_latest.txt': [' ✔', '-']}
pd.DataFrame(data=data, index=['In Situ NRT products', 'In Situ REP products'])
sorted(index)
ftp_host.listdir(ftp_host.curdir)
NRT_products = { 
    'FTP Host': 
    ['nrt.cmems-du.eu', 'my.cmems-du'], 
        'index_history.txt': [' ✔', ' ✔'], 
        'index_monthly.txt': [' ✔', '-'],
        'index_latest.txt': [' ✔', '-']
       }

data = { 
    'FTP Host': ['nrt.cmems-du.eu', 'my.cmems-du'], 
    'index_history.txt': [' ✔', ' ✔'], 
    'index_monthly.txt': [' ✔', '-'],
    'index_latest.txt': [' ✔', '-']
}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12546992481203006, 'fraction_numerical': 0.005169172932330827, 'mean_word_length': 4.680604982206406, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '643386', 'license': 'MIT', 'n_tokens_mistral': 2242, 'n_tokens_neox': 2187, 'n_words': 620, 'path': 'PythonNotebooks/.ipynb_checkpoints/In_Situ_Tips-checkpoint.ipynb', 'repo_name': 'CopernicusMarineInsitu/INSTACTraining--UPDATE-', 'size': '6386'}