<jupyter_start><jupyter_text>Python SVDIn this NoteBook the reader finds code to read GeoTiff files, single- or multi-band, from HDFS. The notebook will read GeoTiffs for two phenology products: phenology model and phenology remote sensing. The GeoTiffs are read as a **ByteArray** and then stored in memory using **MemFile** from **RasterIO** python package. The same package is then used to plot a GeoTiff's band, or multiple bands using sub-plots, histograms, etc. With this example the user can load GeoTiffs from HDFS and then explore all the features of Python packages such as [rasterio](https://github.com/mapbox/rasterio). It also show how to run kmeans from [**sklearn**](http://scikit-learn.org/stable/). The kmeans part in this notebook is inpired in [Unsupervised classification of imagery using scikit-learn](http://nbviewer.jupyter.org/gist/om-henners/c6c8d40389dab75cf535).  Dependencies<jupyter_code>#Add all dependencies to PYTHON_PATH
import sys
sys.path.append("/usr/lib/spark/python")
sys.path.append("/usr/lib/spark/python/lib/py4j-0.10.4-src.zip")
sys.path.append("/usr/lib/python3/dist-packages")

#Define environment variables
import os
os.environ["HADOOP_CONF_DIR"] = "/etc/hadoop/conf"
os.environ["PYSPARK_PYTHON"] = "python3"
os.environ["PYSPARK_DRIVER_PYTHON"] = "ipython"

#Load PySpark to connect to a Spark cluster
from pyspark import SparkConf, SparkContext

#from osgeo import gdal
#To read GeoTiffs as a ByteArray
from io import BytesIO
import rasterio
from rasterio.io import MemoryFile

import numpy as np
import matplotlib.pyplot as plt

import sklearn.cluster<jupyter_output><empty_output><jupyter_text>Connect to Spark<jupyter_code>appName = "plot_GeoTiff"
masterURL="spark://emma0.emma.nlesc.nl:7077"

#A context needs to be created if it does not already exist
try:
    sc.stop()
except NameError:
    print("A  new Spark Context will be created.")
    
sc = SparkContext(conf = SparkConf().setAppName(appName).setMaster(masterURL))<jupyter_output>A  new Spark Context will be created.<jupyter_text>Read data into NumPy Arrays  Load Satellite data<jupyter_code>data_dir = "hdfs:///user/hadoop/avhrr/SOST/"
start_year = 1989
end_year = 2014
years = list(range(start_year, end_year+1))

file_paths = []
sat_data = np.ndarray(shape=(30388736, len(years)), dtype=float , order='C')

for f in range (len(years)):
    file_path = data_dir + "/av_SOST" + str(years[f]) + "v4_SIx.tif"
    print(file_path)
    data = sc.binaryFiles(file_path).take(1)
    dataByteArray = bytearray(data[0][1])
    memfile = MemoryFile(dataByteArray)
    data_set = memfile.open()
    
    # Convert to NumPy array the 1st band
    data_array = np.asarray(data_set.read())[0]
    
    # Replace NaN
    #data_array[np.isnan(data_array)] = -1
    
    #Remove NaN
    data_array = data_array[~np.isnan(data_array)] 
                          
    sat_data[:,f] = data_array<jupyter_output>hdfs:///user/hadoop/avhrr/SOST//av_SOST1989v4_SIx.tif
hdfs:///user/hadoop/avhrr/SOST//av_SOST1990v4_SIx.tif
hdfs:///user/hadoop/avhrr/SOST//av_SOST1991v4_SIx.tif
hdfs:///user/hadoop/avhrr/SOST//av_SOST1992v4_SIx.tif
hdfs:///user/hadoop/avhrr/SOST//av_SOST1993v4_SIx.tif<jupyter_text>Load model data<jupyter_code>data_dir = "hdfs:///user/hadoop/spring-index/BloomFinal"
start_year = 1989
end_year = 2014
years = list(range(start_year, end_year+1))

file_paths = []
mod_data = np.ndarray(shape=(30388736, len(years)), dtype=float , order='C')

for f in range (len(years)):
    file_path = data_dir + "/" + str(years[f]) + ".tif"
    print(file_path)
    data = sc.binaryFiles(file_path).take(1)
    dataByteArray = bytearray(data[0][1])
    memfile = MemoryFile(dataByteArray)
    data_set = memfile.open()
    
    # Convert to NumPy array the 1st band
    data_array = np.asarray(data_set.read())[0]
    
    # Replace NaN
    #data_array[np.isnan(data_array)] = -1
    
    #Remove NaN
    data_array = data_array[~np.isnan(data_array)] 
                          
    mod_data[:,f] = data_array<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12034739454094293, 'fraction_numerical': 0.02382133995037221, 'mean_word_length': 4.346153846153846, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '269759', 'license': 'Apache-2.0', 'n_tokens_mistral': 1540, 'n_tokens_neox': 1438, 'n_words': 330, 'path': 'applications/notebooks/romulo/rasterio_svd.ipynb', 'repo_name': 'phenology/infrastructure', 'size': '4030'}