<jupyter_start><jupyter_text>Python basics 4: Cleaning data This tutorial explores further concepts in Numpy such as categorical data, advanced indexing and dealing with Not-a-Number (NaN) data. Follow the instructions below to download the tutorial and open it in the Sandbox.  Download the tutorial notebook [Download the Python basics 4 tutorial notebook](../_static/python_basics/04_download-cleaning_data.ipynb)To view this notebook on the Sandbox, you will need to first download it to your computer, then upload it to the Sandbox. Ensure you have followed the set-up prerequisities listed in [Python basics 1: Jupyter](./01_jupyter.ipynb), and then follow these instructions:1. Download the notebook by clicking the link above.2. On the Sandbox, open the **Training** folder.3. Click the **Upload Files** button as shown below.4. Select the downloaded notebook using the file browser. Click **OK**.5. The solution notebook will appear in the **Training** folder. Double-click to open it.You can now use the tutorial notebook as an interactive version of this webpage.<jupyter_code>.. note::
    The tutorial notebook should look like the text and code below. However, the tutorial notebook outputs are blank (i.e. no results showing after code cells). Follow the instructions in the notebook to run the cells in the tutorial notebook. Refer to this page to check your outputs look similar.<jupyter_output><empty_output><jupyter_text>Numpy dictionaries and categorical data We will introduce a numpy structure called a **dictionary**. This will be useful for the next lesson on `xarray`.A dictionary represents a mapping between **keys** and **values**. The keys and values are Python objects of any type. We declare a dictionary using curly braces `{}`. Inside we specify the key then its associated value, with the keys and values separated by a colon `:`. Commas `,` are used to separate elements in the dictionary. ```pythondictionary_name = {key1: value1, key2: value2, key3: value3}```For example:<jupyter_code>d = {1: 'one', 
     2: 'two', 
     3: 'apple'}<jupyter_output><empty_output><jupyter_text>In the above dictionary `d`, we have three **keys** `1`, `2`, `3`, and their respective **values** `'one'`, `'two'` and `'apple'`.  We can look up elements in a dictionary using the `[ key_name ]` to address the value stored under a key. The syntax looks like:```pythondictionary_name[key_name]```In our example dictionary `d` above, we can call upon the value associated with the key name `1` like so:```pythond[1]```<jupyter_code>print(d[1], " + ", d[2], " = ", d[3])<jupyter_output><empty_output><jupyter_text>Elements in a dictionary can be modified or new elements added by using the `dictionary_name[key_name] = value` syntax.<jupyter_code>d[3] = 'three'
d[4] = 'four'

print(d[1], " + ", d[2], " = ", d[3])<jupyter_output><empty_output><jupyter_text>Again, the dictionary name, key name, and value must be specified. Dictionaries are useful for data analysis (including satellite data analysis) because they make it easy to assign **categorical values** to our dataset. Remote sensing can be used to create classification products that use categorical values. These products do not contain continuous values. They use discrete values to represent different classes individual pixels can belong to. As an example, the following cells simulate a very simple image containing three different land cover types. Value `1` represents area covered with grass, `2` croplands and `3` city.First, we import the libraries we want to use.<jupyter_code>%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt
from matplotlib import colors<jupyter_output><empty_output><jupyter_text>We will now create a 2-dimensional 100 pixel x 100 pixel numpy array where every value is `1`. This is done using the `numpy.ones` function. Then, we use array indexing to assign part of the area to have the value `2`, and another part to have the value `3`.<jupyter_code># grass = 1
area = np.ones((100,100))

# crops = 2
area[10:60,20:50] = 2

# city = 3
area[70:90,60:80] = 3

area.shape, area.dtype
area<jupyter_output><empty_output><jupyter_text>We now have a matrix filled with 1s, 2s and 3s. At this point, there is no association between the numbers and the different types of ground cover. If we want to show what the area looks like according to the grass/crops/city designation, we might want to give each of the classifications a colour.<jupyter_code># We map the values to colours
index = {1: 'green', 2: 'yellow', 3: 'grey'}

# Create a discrete colour map
cmap = colors.ListedColormap(index.values())

# Plot
plt.imshow(area, cmap=cmap)<jupyter_output><empty_output><jupyter_text>In the case above, every pixel had a value of either `1`, `2` or `3`. What happens if our dataset is incomplete and there is no data in some places?This is a common problem in real-life datasets. Real datasets can be incomplete and may be missing data at certain times or places. To deal with this, we use the special value known as `NaN`, which stands for **Not a Number**. `NaNs` are designated by the numpy `np.nan` function.<jupyter_code>arr = np.array([1,2,3,4,5,np.nan,7,8,9], dtype=np.float32)

arr<jupyter_output><empty_output><jupyter_text>To compute statistics on arrays containing NaN values, numpy has special versions of common functions such as `mean`, standard deviation `std`, and `sum` that ignore the `NaN` values. For example, the next cell shows the difference between using the usual `mean` function and the `nanmean` function.The `mean` function cannot handle `NaN` values so it will return `nan`. The `nanmean` function does not include `NaN` values in the calculation, and therefore returns a number value.<jupyter_code>print(np.mean(arr))

print(np.nanmean(arr))<jupyter_output>nan
4.875<jupyter_text>Note that `NaN` is generally not used as a key in dictionary key-value entries because there are different ways of expressing `NaN` in Python and they are not always equivalent. However, it is still possible to visualise data with `NaNs`; there will be gaps in the image where there is no data.  Exercises  4.1 The harvesting season has arrived and our cropping lands have changed colour to brown. Can you: 4.1.1 Modify the yellow area to contain the new value `4`? 4.1.2 Add a new entry to the `index` dictionary mapping number `4` to the value `brown`. 4.1.3 Plot the area.<jupyter_code># 4.1.1 Modify the yellow area to hold the value 4

# 4.1.2 Add a new key-value pair to index that maps 4 to 'brown'

# 4.1.3 Copy the cmap definition and re-run it to add the new colour

# Plot the area<jupyter_output><empty_output><jupyter_text>> **Hint:** If you want to plot the new area, you have to redefine `cmap` so the new value is assigned a colour in the colour map. Copy and paste the `cmap = ...` line from the original plot.  4.2 Set `area[20:40, 80:95] = np.nan`. Plot the area now.<jupyter_code># Set the nan area

# Plot the entire area<jupyter_output><empty_output><jupyter_text>4.3 Find the median of the `area` array from 4.2 using `np.nanmedian`. Does this match your visual interpretation? How does this compare to using `np.median`?<jupyter_code># Use np.nanmedian to find the median of the area<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08721970009629935, 'fraction_numerical': 0.018572018159306643, 'mean_word_length': 5.013234077750207, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '292406', 'license': 'Apache-2.0', 'n_tokens_mistral': 2143, 'n_tokens_neox': 2050, 'n_words': 1009, 'path': 'docs/python_basics/04_cleaning_data.ipynb', 'repo_name': 'digitalearthafrica/notebooks-training', 'size': '7269'}