<jupyter_start><jupyter_text>Tackle data formats<jupyter_code>%matplotlib inline
import matplotlib.pylab as plt
import numpy as np<jupyter_output><empty_output><jupyter_text>Binary data  Data from [Sea Ice Concentrations from Nimbus-7 SMMR and DMSP SSM/I-SSMIS Passive Microwave Data](http://nsidc.org/data/nsidc-0051.html).<jupyter_code>!wget ftp://sidads.colorado.edu/pub/DATASETS/nsidc0051_gsfc_nasateam_seaice/final-gsfc/north/monthly/nt_202012_f17_v1.1_n.bin<jupyter_output>--2022-02-03 12:49:13--  ftp://sidads.colorado.edu/pub/DATASETS/nsidc0051_gsfc_nasateam_seaice/final-gsfc/north/monthly/nt_202012_f17_v1.1_n.bin
           => ‘nt_202012_f17_v1.1_n.bin.1’
Resolving sidads.colorado.edu (sidads.colorado.edu)... 172.16.31.10
Connecting to sidads.colorado.edu (sidads.colorado.edu)|172.16.31.10|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /pub/DATASETS/nsidc0051_gsfc_nasateam_seaice/final-gsfc/north/monthly ... done.
==> SIZE nt_202012_f17_v1.1_n.bin ... 136492
==> PASV ... done.    ==> RETR nt_202012_f17_v1.1_n.bin ... done.
Length: 136492 (133K) (unauthoritative)

nt_202012_f17_v1.1_ 100%[===================>] 133.29K   156KB/s    in 0.9s    

2022-02-03 12:49:15 (156 KB/s) - ‘nt_202012_f17_v1.1_n.bin.1’ saved [136492]<jupyter_text>Create file id:<jupyter_code>ice = np.fromfile('nt_202012_f17_v1.1_n.bin', dtype='uint8')<jupyter_output><empty_output><jupyter_text>We use *uint8* data type. [List of numpy data types](https://numpy.org/doc/stable/user/basics.types.html) The file format consists of a 300-byte descriptive header followed by a two-dimensional array.<jupyter_code>ice
ice = ice[300:]<jupyter_output><empty_output><jupyter_text>Reshape<jupyter_code>ice = ice.reshape(448,304)<jupyter_output><empty_output><jupyter_text>Simple visualisation of array with imshow (Matplotlib function):<jupyter_code>plt.figure(figsize=(10,10))
plt.imshow(ice,extent=(0,10,0,10),cmap='jet')
plt.colorbar()<jupyter_output><empty_output><jupyter_text>To convert to the fractional parameter range of 0.0 to 1.0, divide the scaled data in the file by 250. If you prefer percent, divide by 2.5.<jupyter_code>ice = ice/250
plt.figure(figsize=(10,10))
plt.imshow(ice,extent=(0,10,0,10),cmap='jet')
plt.colorbar()<jupyter_output><empty_output><jupyter_text>You can mask all land or missing values:<jupyter_code>plt.figure(figsize=(10,10))
plt.imshow(ice,extent=(0,10,0,10),vmin=0, vmax=1.0,cmap='jet')
plt.colorbar()

ice_masked = np.ma.masked_greater(ice, 1.0)

plt.figure(figsize=(10,10))
plt.imshow(ice_masked,extent=(0,10,0,10),vmin=0, vmax=1.0,cmap='jet')
plt.colorbar()<jupyter_output><empty_output><jupyter_text>Masking in this case is similar to using *NaN* in Matlab. [More about NumPy masked arrays](https://numpy.org/doc/stable/reference/maskedarray.html)   Save binary file, if wanted<jupyter_code>fid = open('My_ice_2007.bin', 'wb')
ice.tofile(fid)
fid.close()<jupyter_output><empty_output><jupyter_text>In order to work with other data formats we need to use one of the SciPy submodules:   SciPy  General purpose scientific library (that consist of bunch of sublibraries) and builds on NumPy arrays.* Special functions ([scipy.special](http://docs.scipy.org/doc/scipy/reference/special.html))* Integration ([scipy.integrate](http://docs.scipy.org/doc/scipy/reference/integrate.html))* Optimization ([scipy.optimize](http://docs.scipy.org/doc/scipy/reference/optimize.html))* Interpolation ([scipy.interpolate](http://docs.scipy.org/doc/scipy/reference/interpolate.html))* Fourier Transforms ([scipy.fftpack](http://docs.scipy.org/doc/scipy/reference/fftpack.html))* Signal Processing ([scipy.signal](http://docs.scipy.org/doc/scipy/reference/signal.html))* Linear Algebra ([scipy.linalg](http://docs.scipy.org/doc/scipy/reference/linalg.html))* Sparse Eigenvalue Problems ([scipy.sparse](http://docs.scipy.org/doc/scipy/reference/sparse.html))* Statistics ([scipy.stats](http://docs.scipy.org/doc/scipy/reference/stats.html))* Multi-dimensional image processing ([scipy.ndimage](http://docs.scipy.org/doc/scipy/reference/ndimage.html))* File IO ([scipy.io](http://docs.scipy.org/doc/scipy/reference/io.html))We only use:   scipy.io  to open .mat files<jupyter_code>from scipy.io import loadmat<jupyter_output><empty_output><jupyter_text>We are going to use a matlab file [Polar science center Hydrographic Climatology (PHC)](http://psc.apl.washington.edu/nonwp_projects/PHC/Climatology.html).<jupyter_code>#!wget https://www.dropbox.com/s/0kuzvz03gw6d393/PHC_jan.mat<jupyter_output>--2021-05-12 14:58:23--  https://www.dropbox.com/s/0kuzvz03gw6d393/PHC_jan.mat
Resolving www.dropbox.com (www.dropbox.com)... fd00:a516:7c1b:17cd:6d81:2137:bd2a:2c5b, 172.16.17.32
Connecting to www.dropbox.com (www.dropbox.com)|fd00:a516:7c1b:17cd:6d81:2137:bd2a:2c5b|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: /s/raw/0kuzvz03gw6d393/PHC_jan.mat [following]
--2021-05-12 14:58:26--  https://www.dropbox.com/s/raw/0kuzvz03gw6d393/PHC_jan.mat
Reusing existing connection to [www.dropbox.com]:443.
HTTP request sent, awaiting response... 302 Found
Location: https://uc0cd569c1c9c75655770d0cea47.dl.dropboxusercontent.com/cd/0/inline/BOV6D50Idg5KqWD6cYnyX2cgqGTq_LVH1p7YVULtYdjSC3B_PdySfiZeVzt068eLdKCk59TWc8yUDKNnhrZANitu2W7NH667ZZCoDHGBaQrRKIMoqOvhCxLthsueJzefXQJg3A7a9y6uc4t2OggoSOkf/file# [following]
--2021-05-12 14:58:26--  https://uc0cd569c1c9c75655770d0cea47.dl.dropboxusercontent.com/cd/0/inline/BOV6D50Idg5KqWD6cYnyX2cgqGTq_LVH1p7YVULtYdjSC3B_PdySfiZeVzt068eLdKCk59TWc8[...]<jupyter_text>Open file:<jupyter_code>all_variables = loadmat('PHC_jan.mat')<jupyter_output><empty_output><jupyter_text>We can look at the names of variables stored in the file:<jupyter_code>all_variables.keys()<jupyter_output><empty_output><jupyter_text>We need only *PTEMP1* (3d potential temperature).<jupyter_code>temp = all_variables['PTEMP1']<jupyter_output><empty_output><jupyter_text>Check variable's shape:<jupyter_code>temp.shape<jupyter_output><empty_output><jupyter_text>Show surface level:<jupyter_code>plt.figure(figsize=(10,10))
plt.imshow(temp[0,:,:],extent=(0,10,0,10),cmap='jet')
plt.colorbar()<jupyter_output><empty_output><jupyter_text>Open netCDF files  Scipy have function for working with netCDF files, but please don't use it it's absolete (e.g. do not support netCDF4).The best basic option is to use [python netcdf4](https://github.com/Unidata/netcdf4-python) module that have a lot of nice functionality. Moreover NCEP reanalysis data, that we are going to work with are in netCDF4 format.Later in the course we will also have a look at [xarray](http://xarray.pydata.org/en/stable/) that is the most convinient option nowadays.  Import nessesary function:<jupyter_code>from netCDF4 import Dataset<jupyter_output><empty_output><jupyter_text>I am going to download [NCEP reanalysis data](http://www.esrl.noaa.gov/psd/data/reanalysis/reanalysis.shtml). Surface 4 daily air temperature for 2012.<jupyter_code># !wget ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface_gauss/air.2m.gauss.2016.nc
!wget ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface_gauss/air.2m.gauss.1948.nc<jupyter_output>--2021-05-12 16:11:54--  ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface_gauss/air.2m.gauss.1948.nc
           => ‘air.2m.gauss.1948.nc’
Resolving ftp.cdc.noaa.gov (ftp.cdc.noaa.gov)... 172.16.17.32
Connecting to ftp.cdc.noaa.gov (ftp.cdc.noaa.gov)|172.16.17.32|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /Datasets/ncep.reanalysis.dailyavgs/surface_gauss ... done.
==> SIZE air.2m.gauss.1948.nc ... 12586315
==> PASV ... done.    ==> RETR air.2m.gauss.1948.nc ... done.
Length: 12586315 (12M) (unauthoritative)

air.2m.gauss.1948.n 100%[===================>]  12,00M  1,63MB/s    in 8,2s    

2021-05-12 16:12:04 (1,46 MB/s) - ‘air.2m.gauss.1948.nc’ saved [12586315]<jupyter_text>Create file id:<jupyter_code>fnc = Dataset('air.2m.gauss.1948.nc')<jupyter_output><empty_output><jupyter_text>It's not really file id, it's *netcdf_file* object, that have some methods and attributes:<jupyter_code>fnc.description
fnc.history<jupyter_output><empty_output><jupyter_text>list variables<jupyter_code>fnc.variables<jupyter_output><empty_output><jupyter_text>Access information about variables<jupyter_code>air = fnc.variables['air']<jupyter_output><empty_output><jupyter_text>This time we create *netcdf_variable* object, that contain among other things attributes of the netCDF variable as well as data themselves.<jupyter_code>air.actual_range
air.long_name
air.units
air.shape<jupyter_output><empty_output><jupyter_text>We can access the data by simply using array syntax. Here we show first time step of our data set:<jupyter_code>plt.figure(figsize=(10,10))
plt.imshow(air[0,:,:],extent=(0,10,0,10),cmap='jet')
plt.colorbar()


#plt.imshow(air[0,:,:])
#plt.colorbar();<jupyter_output><empty_output><jupyter_text>Save netCDF file<jupyter_code>#!rm test_netcdf.nc
fw = Dataset('test_netcdf.nc', 'w')

fw.createDimension('t', 366)
fw.createDimension('y', 94)
fw.createDimension('x', 192)

air_var = fw.createVariable( 'air','float32', ('t', 'y', 'x'))
air_var[:] = air[:]
fw.close()<jupyter_output><empty_output><jupyter_text>More descriptive variant:<jupyter_code>!rm test_netcdf.nc
fw = Dataset('test_netcdf.nc', 'w')

fw.createDimension('TIME', 366)
fw.createDimension('LATITUDE', 94)
fw.createDimension('LONGITUDE', 192)

time = fw.createVariable('TIME', 'f', ('TIME',))
time[:] = fnc.variables['time'][:]
time.units = 'hours since 1-1-1 00:00:0.0' 

lat  = fw.createVariable('LATITUDE', 'f', ('LATITUDE',))
lat[:] = fnc.variables['lat'][:]

lon = fw.createVariable('LONGITUDE', 'f', ('LONGITUDE',))
lon[:] = fnc.variables['lon'][:]

ha = fw.createVariable('New_air','f', ('TIME', 'LATITUDE', 'LONGITUDE'))
ha[:] = air[:]
ha.missing_value = -9999.

fw.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15910898965791567, 'fraction_numerical': 0.06672633253778838, 'mean_word_length': 6.214490674318508, 'pattern_counts': {'":': 0, '<': 102, '<?xml version=': 0, '>': 120, 'https://': 8, 'lorem ipsum': 0, 'www.': 9, 'xml': 0}, 'pii_count': 5, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '372133', 'license': 'MIT', 'n_tokens_mistral': 4552, 'n_tokens_neox': 4100, 'n_words': 632, 'path': '2_Geospatial_Python/notebooks/Geoscience_intro/04_tackle_data_formats.ipynb', 'repo_name': 'eo2cube/datacube_eagles', 'size': '10041'}