<jupyter_start><jupyter_text>Dask [![colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/davemlz/spyndex/blob/main/docs/tutorials/dask.ipynb)![level6](https://raw.githubusercontent.com/davemlz/spyndex/main/docs/_static/level6.png) It's parallel processing time: Level 6 - `spyndex + dask`!Remember to install `spyndex`!<jupyter_code>!pip install -U spyndex<jupyter_output><empty_output><jupyter_text>Now, let's start!First, import `spyndex` and `dask`:<jupyter_code>import spyndex
import dask<jupyter_output><empty_output><jupyter_text>`dask.Array`In Level 4 we worked with `xarray` and `DataArrays`. Well, let's do the same tutorial but know working in parallel!Let's use the `xarray.DataArray` that is stored in the `spyndex` datasets: `sentinel`:<jupyter_code>da = spyndex.datasets.open("sentinel")<jupyter_output><empty_output><jupyter_text>As you already know, this data array is very simple. We have 3 dimensions: `band`, `x` and `y`. Each band is one of the 10 m spectral bands of a Sentinel-2 image.<jupyter_code>da<jupyter_output><empty_output><jupyter_text>The data is stored as `int16`, so let's convert everything to `float`. The scale: 10000.<jupyter_code>da = da / 10000<jupyter_output><empty_output><jupyter_text>You can easily visualize the image with `rasterio`:<jupyter_code>from rasterio import plot<jupyter_output><empty_output><jupyter_text>Let's see the RGB visualization:<jupyter_code>plot.show((da.sel(band = ["B04","B03","B02"]).data / 0.3).clip(0,1),title = "RGB")<jupyter_output><empty_output><jupyter_text>Great! Everything is going great!Now, let's convert our normal `xarray.DataArray` to a rechunked one:<jupyter_code>da = da.chunk({"band":1,"x":100,"y":100})<jupyter_output><empty_output><jupyter_text>Let's see what we have:<jupyter_code>da<jupyter_output><empty_output><jupyter_text>Nice! Now let's compute the `NDVI`:<jupyter_code>NDVI = spyndex.computeIndex("NDVI",{"N": da.sel(band = "B08"),"R": da.sel(band = "B04")})<jupyter_output><empty_output><jupyter_text>Let's check our result:<jupyter_code>NDVI<jupyter_output><empty_output><jupyter_text>It's a chunked `xarray.DataArray`! This means that it is not computed yet, so let's compute it!<jupyter_code>NDVI = NDVI.compute()<jupyter_output><empty_output><jupyter_text>Now, if we take a look at our `NDVI`, you'll see that it is already computed! And it was done in parallel with `dask`!<jupyter_code>NDVI<jupyter_output><empty_output><jupyter_text>Let's visualize it!<jupyter_code>plot.show(NDVI,title = "NDVI")<jupyter_output><empty_output><jupyter_text>Amazing!  `dask.Series`In Level 3 we worked with `pandas.Series`. Well, let's do the same tutorial but know working in parallel again!Let's use the `pandas.DataFrame` that is stored in the `spyndex` datasets: `spectral`:<jupyter_code>df = spyndex.datasets.open("spectral")<jupyter_output><empty_output><jupyter_text>As you already know, each column of this dataset is the Surface Reflectance from Landsat 8 for 3 different classes. The samples were taken over Oporto:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Here you can see the classes stored in the `class` column:<jupyter_code>df["class"].unique()<jupyter_output><empty_output><jupyter_text>Each column of the data frame is a `pandas.Series` data type:<jupyter_code>type(df["SR_B2"])<jupyter_output><empty_output><jupyter_text>Everything great, right?Now, let's create a `dask` data frame:<jupyter_code>df = dask.dataframe.from_pandas(df,npartitions = 10)<jupyter_output><empty_output><jupyter_text>Let's see what we have:<jupyter_code>df<jupyter_output><empty_output><jupyter_text>That's a `dask` data frame! Now, let's compute our indices!<jupyter_code>indicesToCompute = ["NDVI","NDWI","NDBI"]<jupyter_output><empty_output><jupyter_text>Ready?... Go!<jupyter_code>df[indicesToCompute] = spyndex.computeIndex(
    index = indicesToCompute,
    params = {
        "G": df["SR_B3"],
        "R": df["SR_B4"],
        "N": df["SR_B5"],
        "S1": df["SR_B6"],
    }
)<jupyter_output><empty_output><jupyter_text>Let's see our result!<jupyter_code>df<jupyter_output><empty_output><jupyter_text>Our result is also a `dask` data frame! Now, just as we did with the chunked `xarray.DataArray`, we have to actually compute our indices!<jupyter_code>df = df.compute()<jupyter_output><empty_output><jupyter_text>Now, let's see what we have:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>OUR INDICES! COMPUTED IN PARALLEL! :D Let's visualize them!<jupyter_code>import seaborn as sns
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Define some colors for each one of the classes:<jupyter_code>colors = ["#E33F62","#3FDDE3","#4CBA4B"]<jupyter_output><empty_output><jupyter_text>Now, let's create our gorgeous pair grid!<jupyter_code>plt.figure(figsize = (15,15))
g = sns.PairGrid(df[['NDVI', 'NDWI', 'NDBI','class']],hue = "class",palette = sns.color_palette(colors))
g.map_lower(sns.scatterplot)
g.map_upper(sns.kdeplot,fill = True,alpha = .5)
g.map_diag(sns.kdeplot,fill = True)
g.add_legend()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1393363089462449, 'fraction_numerical': 0.012614011255579274, 'mean_word_length': 8.286486486486487, 'pattern_counts': {'":': 9, '<': 113, '<?xml version=': 0, '>': 113, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '151914', 'license': 'MIT', 'n_tokens_mistral': 2038, 'n_tokens_neox': 1943, 'n_words': 384, 'path': 'docs/tutorials/dask.ipynb', 'repo_name': 'davemlz/spyndex', 'size': '5153'}