<jupyter_start><jupyter_text>Cleaning up DataSometimes data comes to us in a form that requires some cleaning before we can begin with further analyses.  In this exercise we will explore some tools and strategies for that.We'll begin by reading in a modified version of the Ithaca climate dataset that we worked with previously.  You should notice that there is a new column in the dataframe, indicating the prevailing Sky conditions for each day (sunny, cloudy, etc.).Execute the code cell below.<jupyter_code>import pandas as pd

df = pd.read_csv('IthacaDailyClimateJan2018expanded.csv')
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 31 entries, 0 to 30
Data columns (total 8 columns):
 #   Column               Non-Null Count  Dtype  
---  ------               --------------  -----  
 0   Date                 31 non-null     object 
 1   Max T                30 non-null     float64
 2   Minimum Temp         28 non-null     float64
 3   Average Temperature  29 non-null     float64
 4   Precipitation        31 non-null     float64
 5   Snowfall             27 non-null     float64
 6   Snow Depth           29 non-null     float64
 7   Sky                  29 non-null     object 
dtypes: float64(6), object(2)
memory usage: 2.1+ KB<jupyter_text>Let's look at the dataframe in its entirety.  Execute the code cell below.<jupyter_code>df<jupyter_output><empty_output><jupyter_text>Step 1.Let's examine the column names in a bit more detail.  Execute the code cell below.<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>Notice that there are three different temperature columns, but the naming conventions differ for all three: "Max T", "Minimum Temp", and "Average Temperature".  This can lead to confusion because you need to keep track in your mind how temperature is labeled in each column name (T, Temp, Temperature).  Furthermore, the largest temperature is labeled with the shorthand "Max", while the smallest temperature is labeled with the full word "Minimum".  While we are free to choose whatever names we want for our data (within any syntactic rules), it is useful &mdash; for both you the developer and for anyone else who might be using the code &mdash; to establish some uniformity and consistency in naming.  Fortunately, we don't need to go back and modify the original csv file; we can just modify the column names in our code.In the code cell below, use the ```rename``` method on a dataframe to rename the columns as follows:* rename 'Max T' to be 'Maximum Temperature'* rename 'Minimum Temp' to be 'Minimum Temperature'Consult the [documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rename.html) for ```df.rename``` and find an example to "Rename columns using a mapping", which will demonstrate how to pass a dictionary to the method in order to change the column names. Hint: each element in the dictionary is a key/value pair in which the key is the original name and the value is the new name. For example: {original_name: new_name}.Note that the ```rename``` method will, by default, return a new dataframe with the modified names.  You can either assign that new dataframe to a variable (you can even just reassign it to the name ```df```) or you can use the ```inplace=True``` option to modify ```df``` directly. For this exercise, you will use the ```inplace=True``` option in the `rename` method to modify ```df``` directly. After you've done the renaming, inspect the column names of the dataframe to verify that you've changed the names as intended (and if necessary, modify your renaming code until the column names are as desired.)<jupyter_code>df.rename(columns={"Max T": "Maximum Temperature", "Minimum Temp": "Minimum Temperature"}, inplace=True)<jupyter_output><empty_output><jupyter_text>Self-CheckRun the cell below to test the correctness of your code in the cell above.<jupyter_code># Run this self-test cell to check your code; do not add code or delete code in this cell
from jn import testChangeNameMax, testChangeNameMin

try:
    print(testChangeNameMax(df))    
except Exception as e:
    print("Error!\n" + str(e))
    
try:
    print(testChangeNameMin(df))    
except Exception as e:
    print("Error!\n" + str(e))<jupyter_output>Correct!
Dataframe df contains column 'Maximum Temperature'.
Correct!
Dataframe df contains column 'Minimum Temperature'.<jupyter_text>Execute the cell below and inspect the column names of the dataframe.<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>Step 2.Let's examine the new data in the 'Sky' column.  The entries here are strings representing categorical data, such as "sunny", "partly sunny", "cloudy" and "partly cloudy".  Because they are text-based, it is often useful to verify that there are no mispellings or spelling variants.  A useful method on a Series, or on a column extracted from a DataFrame, is ```unique```, which returns an array of unique entries in that Series or column.The code cell below contains an expression to extract the unique entries of the 'Sky' column of the dataframe.<jupyter_code>df['Sky'].unique()<jupyter_output><empty_output><jupyter_text>While it might not have been obvious when looking at the entire dataframe, extracting the group of unique entries in the 'Sky' column makes it obvious that there are some misspellings and other textual problems, such as an extra space at the beginning of ' partly cloudy'.Pandas dataframes have a ```replace``` method that allows you to change values in a dataframe according to a specified rule, as described [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.replace.html).  If you want to replace text entries (strings), you either need to provide the full string to replace, or use the ```regex=True``` option to specify a regular expression (regex) for replacing part of a string.  In this exercise, let's just specify fully the strings we want to replace.  Note that the ```replace``` method is for changing values in the body of the dataframe, whereas the ```rename``` method used above is for changing the names of the index or column labels.  When called on an entire dataframe, the ```replace``` method will replace all instances of the specified text, regardless of what column it is in.  (If you wanted to replace values only in a particular column, you would first extract that column before doing the replacement.)In the code cell below, write and evaluate code to replace all the misspelled entries in the dataframe with their corrected versions.  The easiest way to do this is to provide all the corrections in a dictionary which is passed as an argument to the method.  Note that by default, the ```replace``` method will return a new dataframe, so you can either assign it to a variable, or modify the original dataframe in place by using the `inplace=True` option.For this exercise, you will modify `df` in place by using the `inplace=True` option.<jupyter_code>df.replace({"party sunny": "Partially Sunny", "couldy": "Cloudy", "partly cloudy": "Partially Cloudy"}, inplace=True)<jupyter_output><empty_output><jupyter_text>Self-CheckRun the cell below to test the correctness of your code in the cell above.<jupyter_code># Run this self-test cell to check your code; do not add code or delete code in this cell
from jn import testChangePartlySunny, testChangeCloudy, testChangePartlyCloudy

try:
    print(testChangePartlySunny(df))    
except Exception as e:
    print("Error!\n" + str(e))
    
try:
    print(testChangeCloudy(df))    
except Exception as e:
    print("Error!\n" + str(e))
    
try:
    print(testChangePartlyCloudy(df))    
except Exception as e:
    print("Error!\n" + str(e))<jupyter_output>Correct!
'party sunny' was corrected.
Correct!
'couldy' was corrected.
Incorrect!
' partly cloudy' was not corrected.<jupyter_text>After doing the text replacement, re-examine the unique entries of the 'Sky' column to verify that you have corrected the problems with the original data.  (There should be 5 unique entries.)  If you have not fixed all the problems, modify your replacement code above and continue until the data are corrected.<jupyter_code># YOUR CODE HERE
df['Sky'].unique()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07517249727635879, 'fraction_numerical': 0.006657789613848202, 'mean_word_length': 4.382410423452769, 'pattern_counts': {'":': 5, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '27532', 'license': 'Apache-2.0', 'n_tokens_mistral': 2286, 'n_tokens_neox': 2172, 'n_words': 1101, 'path': 'CleaningData.ipynb', 'repo_name': 'pujaroy280/AI-Foundations', 'size': '8261'}