<jupyter_start><jupyter_text>210103 Introduction to Earth System Data Accessing archives, see [Extra - resources](piggy/Fall2019/DE/DE-Earth/Extra) Archives can have various degrees of complexity. Typically the easier is the archive structure (e.g. just an FTP with several directories and files) the more workload is on the user/scientist side, i.e.:* discover data* find data* download data* access data* visualise data* analyse data  FTP/HTTP accessif the structure of the archive is either known to the user, or easy and well-documented, and file names meaningful, one could access directly via http. Example: Remote Sensinge.g. from the University of Maryland Global Land Cover Facility. Landsat data archive e.g.ftp://ftp.glcf.umd.edu/glcf/Landsat/```index of ftp://ftp.glcf.umd.edu/glcf/Landsat/WRS2/p001/r007/p001r007_5dx19880719.TM-GLS1990/Name 	Size 	Last ModifiedFile:p001r007_5dt19880719_z26_10.tif.gz	2340 KB 	23.04.09 	00:00:00 CESTFile:p001r007_5dt19880719_z26_20.tif.gz	16735 KB 	23.04.09 	00:00:00 CESTFile:p001r007_5dt19880719_z26_30.tif.gz	17603 KB 	23.04.09 	00:00:00 CESTFile:p001r007_5dt19880719_z26_40.tif.gz	16010 KB 	23.04.09 	00:00:00 CESTFile:p001r007_5dt19880719_z26_50.tif.gz	14610 KB 	23.04.09 	00:00:00 CESTFile:p001r007_5dt19880719_z26_60.tif.gz	8471 KB 	23.04.09 	00:00:00 CESTFile:p001r007_5dt19880719_z26_70.tif.gz	12709 KB 	23.04.09 	00:00:00 CESTFile:p001r007_5dx19880719.742.browse.jpg	145 KB 	04.08.09 	00:00:00 CESTFile:p001r007_5dx19880719.742.preview.jpg	9 KB 	04.08.09 	00:00:00 CESTFile:p001r007_5dx19880719.browse.jpg	101 KB 	04.08.09 	00:00:00 CESTFile:p001r007_5dx19880719.preview.jpg	6 KB 	04.08.09 	00:00:00 CESTFile:p001r007_5x19880719.met	4 KB 	23.04.09 	00:00:00 CEST```browse (quicklook) of data: Example: Climate ScienceE.g. from [NOAA data access](https://www.ncdc.noaa.gov/data-access), e.g. for [NOAA paleoclimatology data](https://www.ncdc.noaa.gov/data-access/paleoclimatology-data/datasets/historical):ftp://ftp.ncdc.noaa.gov/pub/data/paleo/historical/```Index of ftp://ftp.ncdc.noaa.gov/pub/data/paleo/historical/Up to higher level directoryName 	Size 	Last Modifiedalps		25.05.11 	00:00:00 CESTchina		25.05.11 	00:00:00 CESTfrance		25.05.11 	00:00:00 CESTjapan		25.05.11 	00:00:00 CESTswitzerland		25.05.11 	00:00:00 CESTafrica		18.05.15 	00:00:00 CESTasia		11.06.07 	00:00:00 CESTcaribbean		04.05.16 	00:00:00 CESTeurope		31.01.18 	00:09:00 CETeurope-seasonal-files		16.08.12 	00:00:00 CESTnorth_atlantic		04.05.16 	00:00:00 CESTnorthamerica		13.10.17 	13:05:00 CESTpacific		05.10.06 	00:00:00 CESTsouthamerica		15.09.09 	00:00:00 CESTFile:documentary.pdf	328 KB 	28.01.15 	00:00:00 CETFile:europe-seasonal.txt	38 KB 	14.11.06 	00:00:00 CETFile:kuettel2009readme.txt	7 KB 	05.05.09 	00:00:00 CESTFile:kuettel2009slp.txt	2900 KB 	20.04.09 	00:00:00 CEST```  Web access and form fillingE.g. from [NOAA data access](https://www.ncdc.noaa.gov/data-access), e.g. for [NOAA paleoclimatology data](https://www.ncdc.noaa.gov/data-access/paleoclimatology-data/datasets/historical):https://www.ncdc.noaa.gov/paleo-search/  Web mapping interfaces - searchE.g. from [NOAA data access](https://www.ncdc.noaa.gov/data-access), e.g. for [NOAA paleoclimatology data](https://www.ncdc.noaa.gov/data-access/paleoclimatology-data/datasets/historical):https://gis.ncdc.noaa.gov/maps/ncei/paleo?layers=0001  Web mapping interfaces - access and analyse onlineE.g. [Giovanni tool](https://giovanni.gsfc.nasa.gov/giovanni/)   API / programmatic data accessE.g. * [NASA Earth data access with python](https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+Python)* [Accessing Pangaea data programmatically - R](https://www.fromthebottomoftheheap.net/2016/12/16/pangaea-r-open-palaeo-data/)  WCS / OGC Data can be accessed via various stanards. Geographic data, either raster or vector, can use several of the [Open Geospatial Consortium](http://www.opengeospatial.org) standards and interfaces.An introduction to OGC web services is provided e.g. on the [EarthServer Geopython tutorial](https://nbviewer.jupyter.org/github/earthserver-eu/geopython_workshop_2017/blob/master/Intro_to_WCS.ipynb).Examples are taken and/or modified from the [WCS/WCPS domain examples (Geophython 2017)](https://nbviewer.jupyter.org/github/earthserver-eu/geopython_workshop_2017/blob/master/Domain_examples.ipynb)  Climate Science An introduction on time series extraction form climate science datasets is provided by the [ECMWF/EarthServer Time-series retrieval - Climate Sciences](https://nbviewer.jupyter.org/github/earthserver-eu/geospatial-workflow-with-wcps/blob/master/Time-series-retrieval_climate_sciences.ipynb) Some basic data extarction from weather databases is described on [Time-series retrieval - Climate Sciences](https://nbviewer.jupyter.org/github/earthserver-eu/geospatial-workflow-with-wcps/blob/master/Time-series-retrieval_climate_sciences.ipynb)The following example (courtesy <NAME>, ECMWF) creates a global map of the temperature at 2m height, dyncamically queried.  Load required libraries<jupyter_code>import time
import datetime
import requests
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg<jupyter_output><empty_output><jupyter_text>Helper functions See also [Example of calculating global monthly anomalies of surface temperature](https://nbviewer.jupyter.org/github/earthserver-eu/geospatial-workflow-with-wcps/blob/master/Monthly-averages_climate_sciences.ipynb)  Marine science  for northern latitudes I.E. europe coast you will get better results between apr-sept<jupyter_code>endpoint = "http://rsg.pml.ac.uk/rasdaman/ows"

coverage = "OCCCI_V3_monthly_chlor_a"<jupyter_output><empty_output><jupyter_text>see also e.g. [PML/EarthServer matchup tool](https://nbviewer.jupyter.org/github/earthserver-eu/geospatial-workflow-with-wcps/blob/master/Match-up_tool_marine_sciences.ipynb)  Earth Observation Materials are derived form the notebooks from [MEEO/EarthServer](http://jupyter.eodataservice.org/) (free registration required). Courtesy <NAME>.  Endpoint inizialisationUse the getCapabilities service to connect to the MEEO endpoint<jupyter_code>from owslib.wcs import WebCoverageService
import time

start = time.time()
my_wcs = WebCoverageService('https://eodataservice.org/dar05?', version='2.0.0')
print ( "time for GetCapabilities: %s s" %( time.time() - start ) )

print(my_wcs)
# for coverage_name in my_wcs.contents.keys():
#     if coverage_name.startswith( "SoilMoisture" ):
#         print ( coverage_name )<jupyter_output>time for GetCapabilities: 0.00013399124145507812 s
None<jupyter_text>see also (not working anymore ;( )https://nbviewer.jupyter.org/github/earthserver-eu/geopython_workshop_2017/blob/master/Exercises.ipynb<jupyter_code># see also http://sentinelhub-py.readthedocs.io/en/latest/examples/ogc_request.html<jupyter_output><empty_output><jupyter_text>Pangaea.de<jupyter_code># see https://git.geomar.de/isos/hands-on/blob/e2d93b6bb99e673a3dc0f432ef3dff6d7bbc79db/boknis.ipynb<jupyter_output><empty_output><jupyter_text>Search pangaea.de to retrieve the data URL  Retrieve the data URL and check<jupyter_code>data_url = 'https://doi.pangaea.de/10.1594/PANGAEA.855693?format=text' # replace this with your url
#get data from the url using package requests
import requests
r=requests.get(data_url)
data=r.text
print(data[:4000]) #see what's inside
#saving the data
with open('./output/boknis.txt', 'w+') as outfile: outfile.write(data)
# Header on Pangaea is marked with /* HEADER */
# split data and header:
h, d = data.split('*/')
print(h)
print(d[:400])
#parse data into a list
lines=d.split('\n')
#use pprint to pretty-print a list
import pprint
pprint.pprint(lines[:5])
#the first element is empty, so we remove it:
lines=lines[1:]
pprint.pprint(lines[:5])
#convert this to list by separating at '\t':
dd=[q.split('\t') for q in lines]
pprint.pprint(dd[:3])
# At this point you can do many of the data analysis 
# by working directly with the list we've just created, 
# e.g. using list comprehensions and numpy, etc. 
# This is useful for debugging, but could be cumbersome for non-trivial tasks,
# and slow for large datasets.
#
# Pandas is a python package for handling tabular data, which has methods 
# for data input-output and to various formats.
#
# First open the data file and find the number of lines in the header (remember that header ends with a '*/')

i=0 # counter is defined outside the scope of the "with" block
for line in open('./output/boknis.txt', 'r'): #read one line at a time
    if line.find('*/')>-1: break # if there's '*/' on a line, exit loop and close the file
    i+=1 # increment counter
print(i)

q=open('./output/boknis.txt', 'r').readlines()
q[31]
import pandas as pd
df = pd.read_csv('./output/boknis.txt', sep='\t', parse_dates=True, index_col=None, skiprows=i+1)
df.head()
#plot data from pandas using matplotlib
import seaborn as sns
import matplotlib.pyplot as plt
sns.boxplot(x='Depth water [m]', y='Temp [°C]', data=df)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11484841779154681, 'fraction_numerical': 0.08563841557866785, 'mean_word_length': 5.238095238095238, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 27, 'https://': 21, 'lorem ipsum': 0, 'www.': 9, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '430580', 'license': 'CC0-1.0', 'n_tokens_mistral': 3907, 'n_tokens_neox': 3357, 'n_words': 784, 'path': '210103_03-archives.ipynb', 'repo_name': 'oemuaghwor/awesome-open-geoscience', 'size': '9052'}