<jupyter_start><jupyter_text>Session 2 Reading data in Google Colab <NAME> Practice Colab, IHE Delft 2020 - 07- 28Session 2 **Contents**Reding rainfall data from different sources1. Reading Excel and CSV2. Read Era5 raster data from API3. Using google earth engine 4. From Kaggles website       Run in Google Colab        View source on GitHub    1) Reading from Excel and CSV This practice will aim at getting acquinted with the Google drive access and reading a simple data set created in a local machine.  Mount your GDriveThe first example on how to bring data will be done by GDrive Task: Read a file created by you in the goole colab by uploading it into GDrive and reading it using pandas.Create a file in your computer with the following dataA,B,C,D,E, target1,2,3,4,5, 0  12,15,121,124,135, 1213,322,243,214,326, 0Steps1. Open drive.google.com2. Create a folder named  "practice1" on the root of gDrive3. Upload a text file with the following content and name it "data.csv"Time estimated 10 minutes---  Now see in if the mount drive worksFunctions to remember in linux are $ls$ and $pwd$, which are meant to help list the contents of a directory and to show the $p$ath of the $w$orking $d$irectory<jupyter_code>!pwd
!ls<jupyter_output>/content
drive  sample_data<jupyter_text>Now we mount the drive, Note: if you have done it previously then there is no need for this step.<jupyter_code>from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

Enter your authorization code:
··········
Mounted at /content/drive<jupyter_text>To navigate the virtual machine hard drive structure you can also use the python os library. os.chdir makes a change in the location of the current directory<jupyter_code>import os
os.chdir("drive/My Drive/practice1")

!ls
!pwd<jupyter_output>data.gsheet
/content/drive/My Drive/practice1<jupyter_text>Now we import pandas and read it is done in a normal notebook in your local machine<jupyter_code>!ls
import pandas as pd
df= pd.read_csv("data.csv")
df.describe()
df.info()
df.summary()<jupyter_output><empty_output><jupyter_text>To plot it, we can add a matplotlib and visualize the contents<jupyter_code>df.plot()<jupyter_output><empty_output><jupyter_text>2) Reading Era5 Reanalysis dataWe would like to analyze the time series data from https://cds.climate.copernicus.eu/cdsapp!/dataset/reanalysis-era5-single-levels?tab=overview  Seeting up the environmentWe will use the API from the copernicus climate website, please register. And look for the API information and the process to register the API on your local machien (key and url).https://cds.climate.copernicus.eu/api-how-toYou need to register and obtain the two parameters below<jupyter_code>!echo "url: https://cds.climate.copernicus.eu/api/v2" > $HOME/.cdsapirc
!echo "key: 18475:0918427e-3da7-45da-991c-161d92b9b707" >> $HOME/.cdsapirc

!pip install cdsapi

#[-87.60483030714467,13.402246848143117]
#[-86.62704710401967,13.402246848143117]
#[-86.62704710401967,13.925339395097156]
#[-87.60483030714467,13.925339395097156]
#[-87.60483030714467,13.402246848143117]<jupyter_output>Collecting cdsapi
  Downloading https://files.pythonhosted.org/packages/4e/d7/8a0abd3e63c40ad2d7c6eac5fc4bdf13fa72e96a36688822a2fb0ca0814b/cdsapi-0.2.7.tar.gz
Requirement already satisfied: requests>=2.5.0 in /usr/local/lib/python3.6/dist-packages (from cdsapi) (2.23.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from cdsapi) (4.41.1)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.5.0->cdsapi) (1.24.3)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.5.0->cdsapi) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.5.0->cdsapi) (2020.4.5.1)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.5.0->cdsapi) (2.9)
Building wheels for collected packages: cdsapi
  Building wheel for cdsapi (setup.py) [...]<jupyter_text>To map the region where the rain is falling we are going to select the !echo "url: https://cds.climate.copernicus.eu/api/v2" > $HOME/.cdsapirc!echo "key: 18475:0918427e-3da7-45da-991c-161d92b9b707" >> $HOME/.cdsapirc<jupyter_code>import cdsapi
import xarray as xr
product = 'reanalysis-era5-single-levels'
request = {'product_type': 'reanalysis',
           'format': 'netcdf',
           'day': ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10',
                   '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21',
                   '22', '23', '24', '25', ' 26', '27', '28', '29', '30', '31'],
           'time': ['00:00', '01:00', '02:00', '03:00', '04:00', '05:00',
                    '06:00', '07:00', '08:00', '09:00', '10:00', '11:00', '12:00',
                    '13:00', '14:00', '15:00', '16:00', '17:00', '18:00', '19:00',
                    '20:00', '21:00', '22:00', '23:00'],
           'area': [-87.66, -87.65, 13.4, 13.92],
           'year': 2016,
           'month': 1,
           'variable': ['100m_u_component_of_wind',
                        '100m_v_component_of_wind',
                        '2m_temperature',
                        'runoff',
                        'soil_temperature_level_4',
                        'surface_net_solar_radiation',
                        'surface_pressure',
                        'surface_solar_radiation_downwards',
                        'toa_incident_solar_radiation',
                        'total_sky_direct_solar_radiation_at_surface']}

result = cdsapi.Client().retrieve(
    product,
    request
)

result.download("test.nc")

ds = xr.open_dataset("test.nc")
ds['ro'].to_series().describe()
import matplotlib.pyplot as plt

ds['ro'].to_series().plot(title="Test",rot=45 )<jupyter_output><empty_output><jupyter_text>3) Reading data from Google earth  Google earth engine cataloghttps://developers.google.com/earth-engine/datasets/catalog  google colab This is needed when you run the jupyter notebook from your own terminal<jupyter_code>from google.colab import auth
auth.authenticate_user()<jupyter_output><empty_output><jupyter_text>Authenticating Google Earth Engine (GEE)Google requires the authentication for each cloud service you would like to use. In this case, the library brings its own authentication procedure.<jupyter_code>import ee
ee.Authenticate()
ee.Initialize()<jupyter_output>To authorize access needed by Earth Engine, open the following URL in a web browser and follow the instructions. If the web browser does not start automatically, please manually browse the URL below.

    https://accounts.google.com/o/oauth2/auth?client_id=517222506229-vsmmajv00ul0bs7p89v5m89qs8eb9359.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fearthengine+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&code_challenge=TMm8CdDhHw3Zr6ZOIwYphlzjZiw3baWXp9GxnczPvhc&code_challenge_method=S256

The authorization workflow will generate a code, which you should paste in the box below. 
Enter verification code: 4/0AFeXaaQbRUOqrNdYW9ILupQbkM7-YHzbaEj5s7Nv8gDlK74UZV9rCY

Successfully saved authorization token.<jupyter_text>[Catalog for google earth](https://developers.google.com/earth-engine/datasets/catalog)  Javascrip translation into Python codingIn general, most of the google earth engine examples are in javascript so to be able to retrieve you data, you will need to translate code into python. For the sake of simplicity two of the most used translations are presented. However, on the [python API](https://developers.google.com/earth-engine/python_install) site of google you can find a complete table.**This is the code in Javascript**```{js, eval=FALSE}var dataset = ee.ImageCollection('LANDSAT/LC08/C01/T1_8DAY_NDWI')                  .filterDate('2017-01-01', '2017-12-31');var colorized = dataset.select('NDWI');var colorizedVis = {  min: 0.0,  max: 1.0,  palette: ['0000ff', '00ffff', 'ffff00', 'ff0000', 'ffffff'],};Map.setCenter(6.746, 46.529, 6);Map.addLayer(colorized, colorizedVis, 'Colorized');``` **This is the code in python**```{js, eval=FALSE}dataset = ee.ImageCollection('LANDSAT/LC08/C01/T1_8DAY_NDWI')                  .filterDate('2017-01-01', '2017-12-31');colorized = dataset.select('NDWI');colorizedVis = {  "min": 0.0,  "max": 1.0,  "palette": ['0000ff', '00ffff', 'ffff00', 'ff0000', 'ffffff'],  }Map.setCenter(6.746, 46.529, 6);Map.addLayer(colorized, colorizedVis, 'Colorized');```This is a typical example, where simple var is removed and the json type of information (colorizedVis) is updates such that it looks like a dictionary in python. So in general, only quotations are placed around the variable, in this case min, max and palette.Extra info: You can find more information in the website for GEE at the [Wagening website](https://geoscripting-wur.github.io/Earth_Engine/), about the python API and javacript functions and classes.  Normalized Difference Water Index (NDWI) This practice will focus on obtaining the NDWI from the [EE catallog](https://developers.google.com/earth-engine/datasets/catalog/LANDSAT_LC08_C01_T1_8DAY_NDWI)On the link this link  you will find the information about the data. Installing librariesThe first library we will use if **geehydro**, which is pretty handy for water resources. The following code runs the installation if you do now have it.<jupyter_code>import subprocess

try:
    import geehydro
except ImportError:
    print('geehydro package not installed. Installing ...')
    subprocess.check_call(["python", '-m', 'pip', 'install', 'geehydro'])<jupyter_output>geehydro package not installed. Installing ...<jupyter_text>The basic libraries required are google earth engine which appear in the following code as ee. However, to install it you will need to typeconda update earthengine-apiorpip install earthengine-api --upgrademore information can be found here:<jupyter_code>import ee
import folium
import geehydro
print(folium.__version__)<jupyter_output>0.8.3<jupyter_text>Now we need to initialize the library, since is connected to google services. If you have not authenticated the session, you need to uncomment the hash on the first line, in the following code.<jupyter_code>ee.Authenticate()
ee.Initialize()<jupyter_output>To authorize access needed by Earth Engine, open the following URL in a web browser and follow the instructions. If the web browser does not start automatically, please manually browse the URL below.

    https://accounts.google.com/o/oauth2/auth?client_id=517222506229-vsmmajv00ul0bs7p89v5m89qs8eb9359.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fearthengine+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&code_challenge=_2NjukStFwxgHw6F0UzulUEyoGOkiOzdT_mfxWkO9lI&code_challenge_method=S256

The authorization workflow will generate a code, which you should paste in the box below. 
Enter verification code: 4/0AGkMTxI3992j97VnSaL9FjtHJq1hG39_CBCvN8u-EW8GgCvjGdQdhQ

Successfully saved authorization token.<jupyter_text>Google Earth Engine (EE) library variablesAs tensorflow, pandas and numpy, Googel earth engine library (EE) has its own data structures. This is an example of the strings declaration.Normal python string<jupyter_code># traditional python string
print('Hello world!')<jupyter_output>Hello world!<jupyter_text>An with google earth<jupyter_code>Myimage=ee.Image('LANDSAT/LC08/C01/T1/LC08_044034_20140318')
print(Myimage.getInfo())<jupyter_output>{'type': 'Image', 'bands': [{'id': 'B1', 'data_type': {'type': 'PixelType', 'precision': 'int', 'min': 0, 'max': 65535}, 'dimensions': [7661, 7801], 'crs': 'EPSG:32610', 'crs_transform': [30, 0, 460785, 0, -30, 4264215]}, {'id': 'B2', 'data_type': {'type': 'PixelType', 'precision': 'int', 'min': 0, 'max': 65535}, 'dimensions': [7661, 7801], 'crs': 'EPSG:32610', 'crs_transform': [30, 0, 460785, 0, -30, 4264215]}, {'id': 'B3', 'data_type': {'type': 'PixelType', 'precision': 'int', 'min': 0, 'max': 65535}, 'dimensions': [7661, 7801], 'crs': 'EPSG:32610', 'crs_transform': [30, 0, 460785, 0, -30, 4264215]}, {'id': 'B4', 'data_type': {'type': 'PixelType', 'precision': 'int', 'min': 0, 'max': 65535}, 'dimensions': [7661, 7801], 'crs': 'EPSG:32610', 'crs_transform': [30, 0, 460785, 0, -30, 4264215]}, {'id': 'B5', 'data_type': {'type': 'PixelType', 'precision': 'int', 'min': 0, 'max': 65535}, 'dimensions': [7661, 7801], 'crs': 'EPSG:32610', 'crs_transform': [30, 0, 460785, 0, -30, 4264215]}, {'[...]<jupyter_text>Creating the first interactive mapSince in python you will have a local environment, the library folium will help to emulate the online environment. However, the folium library still uses google maps online to load maps.<jupyter_code>Map = folium.Map(location=[40, -100], zoom_start=12)
Map.setOptions('HYBRID')
Map<jupyter_output><empty_output><jupyter_text>**Task** Locate your country and change the zoom to 2 and then to 14. We select from a collection of data the period of images to be read<jupyter_code>dataset = ee.ImageCollection("LANDSAT/LC08/C01/T1_8DAY_NDWI").filterDate('2017-01-01', '2017-12-31');<jupyter_output><empty_output><jupyter_text>Then we select the variable to be read as an image<jupyter_code>image = dataset.select('NDWI');<jupyter_output><empty_output><jupyter_text>We center the Map created before and change the zoom level. And add our image<jupyter_code># Display the image.
Map.addLayer(image, {}, 'Landsat 8 Normalized Difference Water Index image')
Map.setCenter(6.746, 46.529, 6);

Map<jupyter_output><empty_output><jupyter_text>As you can see the map lacks of a color scale and therefore is not possible to interpret or analyse earily any information.So we add a color pallete<jupyter_code># Define visualization parameters in an object literal.
colorizedVis = {
    'min': 0.0,
    'max': 1.0,
    'palette': ['0000ff', '00ffff', 'ffff00', 'ff0000', 'ffffff'],
    };
Map.addLayer(image, colorizedVis, 'Colorized');
Map<jupyter_output><empty_output><jupyter_text>Now if we want political boundaries of the countries we can use one of the feature collections (which contains shape files or features)<jupyter_code>counties = ee.FeatureCollection('TIGER/2016/Counties')
Map.addLayer(counties, {}, 'counties')
Map<jupyter_output><empty_output><jupyter_text>As you can see it is not possible to see the layer since one is on top of the other. In this case would be handy to add the contro visibility for the layer.<jupyter_code>Map.setControlVisibility(layerControl=True, fullscreenControl=True, latLngPopup=True)
Map<jupyter_output><empty_output><jupyter_text>Examples websiteIf you want to knoe more about the geehydro examplesThis repository is a collection of 360+ Jupyter Python notebook examples.[Python notebooks](https://github.com/giswqs/earthengine-py-notebooks)<jupyter_code><jupyter_output><empty_output><jupyter_text>A good website to explore is the following https://github.com/giswqs/earthengine-py-notebooks  https://github.com/renelikestacos/Google-Earth-Engine-Python-Examples  Home work Short practice in Google earthAfter the following section you can try to read the same data but using the Earth Engine libraryhttps://code.earthengine.google.com/?scriptPath=Examples%3ADatasets%2FECMWF_ERA5_DAILYFor this exercise we will select precipitation data. Google earth provide a large variety of data from different sources. Lest select daily aggregates from Era5 .<jupyter_code>#@title Magic tools for jupyter notebook
%%html
<marquee style='width: 50%; color: blue;'><b>Welcome to Hydroinformatics Lab session</b></marquee>
<br>

#@markdown To be able to load any jupyter extension %load_ext

#@markdown To be able to use Jupyter Magic features just use %%<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12101136570838145, 'fraction_numerical': 0.07135476812739318, 'mean_word_length': 5.306630893062476, 'pattern_counts': {'":': 3, '<': 92, '<?xml version=': 0, '>': 108, 'https://': 17, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '614694', 'license': 'MIT', 'n_tokens_mistral': 6280, 'n_tokens_neox': 5552, 'n_words': 1563, 'path': 'ExeciseHiLabML1.ipynb', 'repo_name': 'HydroinformaticsLab/ColabWorkshop', 'size': '16467'}