<jupyter_start><jupyter_text>**** MEASUREMENTS DATA SECTION !! ****<jupyter_code># ID the first/measurements file:

hawaii_measurements = "Resources\hawaii_measurements.csv"
# Reference / "read in" the measurements CSV datafiles (raw - UNCLEANED - ORIGINAL):

hawaii_measurements = pd.read_csv("Resources/hawaii_measurements.csv")
hawaii_measurements.head()
# check some basic stats, to see if things look fairly normal:

hawaii_measurements.describe()
# Let's check the actual CSV sheet data "shape" to make sure it matches "as a whole" the dataframe head 
# ...from above.   Also to verify the correct number of columns, etc:

hawaii_measurements.shape

# The output of 4 columns matches the dataframe header.
# Checking for duplicates in the measurements DataFrame
hawaii_measurements.duplicated().sum()
# Now check for NULL values (making sure cells are filled):  We already saw 1 for prcp in the df above.

hawaii_measurements.isnull().sum()
# We see that the prcp column has some null values.  Let's remove:

# Removing the null values
hawaii_measurements = hawaii_measurements.dropna()
# After dropping the nulls, let's see how the shape of the data was affected:

# Rechecking the shape of the database
hawaii_measurements.shape
# It appears that (19550 - 18103) or 1447 null values were elimanted.   
#   **** This means that our new data set was reduced by about 7.4% of the observed values (rows).  
#  Based on this, I think it's ok to proceed with the analysis (dropping nulls) w/o bringing
# large shifts in our resulting analysis.  

# HOWEVER, let's test the above to be sure...................................
#  SINCE eliminating the null data sets will reduce the corresponding "tobs" data sets (rows),
# .. let's RE-RUN the basic summary stats to verify no massive shift in the MEAN values for tobs...

hawaii_measurements.describe()
#  From above, we see the "tobs" count falls by 1447 and the mean shifts from 73.09 (pre-drop dataframe)
# ... to 72.99 (above).   This shouldn't be too significant, so I think it's ok to elimnate the nulls.

# Now we created our new csv containing the "cleaned" data without null values in the prcp column:

hawaii_measurements.to_csv("Resources/clean_hawaii_measurements.csv", index=False)<jupyter_output><empty_output><jupyter_text>**** STATIONS DATA SECTION !! ****<jupyter_code># ID the second/stations file:

hawaii_stations = "Resources\hawaii_stations.csv"
# Reference / "read in" the measurements CSV datafiles (raw - UNCLEANED - ORIGINAL):

hawaii_stations = pd.read_csv("Resources/hawaii_stations.csv")
hawaii_stations.head()
# check some basic stats, to see if things look fairly normal:

hawaii_stations.describe()
# Let's check the actual CSV sheet data "shape" to make sure it matches "as a whole" the dataframe head 
# ...from above.   Also to verify the correct number of columns, etc:

hawaii_stations.shape

# The output of 5 columns matches the dataframe header.
# Checking for duplicates in the stations DataFrame
hawaii_stations.duplicated().sum()
# Now check for NULL values (making sure cells are filled):  

hawaii_stations.isnull().sum()

# NO Null datasets in the stations csv.
# Since we don't have null values, we can go and create the clean csv:

# Creating a new csv for the cleaned data
hawaii_stations.to_csv("Resources/clean_hawaii_stations.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09327805744743856, 'fraction_numerical': 0.009179745336097127, 'mean_word_length': 4.501628664495114, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '384548', 'license': 'MIT', 'n_tokens_mistral': 1093, 'n_tokens_neox': 1009, 'n_words': 413, 'path': 'data_engineering.ipynb', 'repo_name': 'APK500/HW-11_Surfs_UP', 'size': '3377'}