<jupyter_start><jupyter_text>Summarize total counts of trash by high-level categories for MDMAP dataset<jupyter_code>import pandas as pd
import numpy as np
import json
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Import `category_map.csv` and create a dictionary:<jupyter_code>cat_map = pd.read_csv('category_map.csv')
catdict = {key:value for key,value in zip(cat_map['Column Name'], cat_map['High-Level Category'])}<jupyter_output><empty_output><jupyter_text>Import cleaned MDMAP_Accumulation data:<jupyter_code>mdmap_all = pd.read_csv('data_processed/mdmap_accumulation_totalarea_zerosremoved.csv')<jupyter_output><empty_output><jupyter_text>Map MDMAP trash subcategories to their corresponding `High-Level Category`:<jupyter_code># Create a dataframe of the subcategories:
mdmap_subset = mdmap_all[['UniqueId',
                'Hard Plastic Fragments', 
                'Foamed Plastic Fragments', 
                'Filmed Plastic Fragments', 
                'Food Wrappers', 
                'Plastic Beverage Bottles', 
                'Other Jugs/Containers', 
                'Bottle/Container Caps', 
                'Cigar Tips', 
                'Cigarettes', 
                'Disposable Cigarette Lighters', 
                '6-Pack Rings', 
                'Bags', 
                'Plastic Rope/Net', 
                'Buoys & Floats', 
                'Fishing Lures & Line', 
                'Cups', 
                'Plastic Utensils', 
                'Straws', 
                'Balloons Mylar', 
                'Personal Care Products', 
                'Plastic Other', 
                'Metal', 
                'Aluminum/Tin Cans', 
                'Aerosol Cans', 
                'Metal Fragments', 
                'Metal Other', 
                'Glass', 
                'Glass Beverage Bottles', 
                'Jars', 
                'Glass Fragments', 
                'Glass Other', 
                'Rubber', 
                'Flip Flops', 
                'Rubber Gloves', 
                'Tires', 
                'Balloons Latex', 
                'Rubber Fragments', 
                'Rubber Other', 
                'Processed Lumber', 
                'Cardboard Cartons', 
                'Paper and Cardboard', 
                'Paper Bags', 
                'Lumber/Building Material', 
                'Processed Lumber Other', 
                'Cloth/Fabric', 
                'Clothing & Shoes', 
                'Gloves (non-rubber)', 
                'Towels/Rags', 
                'Rope/Net Pieces (non-nylon)', 
                'Fabric Pieces', 
                'Cloth/Fabric Other', 
                'Unclassified']] # Removed Total Debris and Debris Description
mdmap_long = pd.melt(mdmap_all, id_vars=['UniqueId'], 
                     value_vars=['Hard Plastic Fragments', 
                                 'Foamed Plastic Fragments', 
                                 'Filmed Plastic Fragments', 
                                 'Food Wrappers', 
                                 'Plastic Beverage Bottles', 
                                 'Other Jugs/Containers', 
                                 'Bottle/Container Caps', 
                                 'Cigar Tips', 
                                 'Cigarettes', 
                                 'Disposable Cigarette Lighters', 
                                 '6-Pack Rings', 
                                 'Bags', 
                                 'Plastic Rope/Net', 
                                 'Buoys & Floats', 
                                 'Fishing Lures & Line', 
                                 'Cups', 
                                 'Plastic Utensils', 
                                 'Straws', 
                                 'Balloons Mylar', 
                                 'Personal Care Products', 
                                 'Plastic Other', 
                                 'Metal', 
                                 'Aluminum/Tin Cans', 
                                 'Aerosol Cans', 
                                 'Metal Fragments', 
                                 'Metal Other', 
                                 'Glass', 
                                 'Glass Beverage Bottles', 
                                 'Jars', 
                                 'Glass Fragments', 
                                 'Glass Other', 
                                 'Rubber', 
                                 'Flip Flops', 
                                 'Rubber Gloves', 
                                 'Tires', 
                                 'Balloons Latex', 
                                 'Rubber Fragments', 
                                 'Rubber Other', 
                                 'Processed Lumber', 
                                 'Cardboard Cartons', 
                                 'Paper and Cardboard', 
                                 'Paper Bags', 
                                 'Lumber/Building Material', 
                                 'Processed Lumber Other', 
                                 'Cloth/Fabric', 
                                 'Clothing & Shoes', 
                                 'Gloves (non-rubber)', 
                                 'Towels/Rags', 
                                 'Rope/Net Pieces (non-nylon)', 
                                 'Fabric Pieces', 
                                 'Cloth/Fabric Other', 
                                 'Unclassified'], 
                     var_name='Subcategory', value_name='Count')
mdmap_long['Category'] = mdmap_long['Subcategory'].map(catdict)
# Remove non-numerical values under Count
mdmap_int = mdmap_long[mdmap_long.applymap(np.isreal).Count]
mdmap_int.info()
mdmap_int = mdmap_int.dropna()
mdmap_int.info()
mdmap_int.Count = mdmap_int.Count.astype('int64') 
# Calculate the total counts by the High-Level Category:
mdmap_group = mdmap_int.groupby(['UniqueId','Category'], as_index=False).sum()
mdmap_group.head()
mdmap_totals = mdmap_group.pivot(index='UniqueId', 
                                columns='Category',
                                values='Count')<jupyter_output><empty_output><jupyter_text>Merge the key location and survey stats with the trash totals:<jupyter_code># Store the key location and survey stats in a dataframe:
mdmap_orig = mdmap_all[['UniqueId',
                        'Organization', 
                        'Date', 
                        'Survey Year', 
                        'Survey ID', 
                        'Latitude Start', 
                        'Longitude Start', 
                        'Latitude End', 
                        'Longitude End', 
                        'Width', 
                        'Length',
                        'TotalArea',
                        'Total Debris',
                        'Plastic']]
mdmap_orig.columns = ['UniqueId',
                        'Organization', 
                        'Date', 
                        'Survey Year', 
                        'Survey ID', 
                        'Latitude Start', 
                        'Longitude Start', 
                        'Latitude End', 
                        'Longitude End', 
                        'Width', 
                        'Length',
                        'TotalArea',
                        'Total Debris',
                        'Plastic Count']
# Merge with map_totals:

# Test:
mdmap_final = pd.merge(mdmap_orig, mdmap_totals, how='outer', on='UniqueId', indicator=True)
mdmap_final.groupby('_merge').count()
mdmap_final = mdmap_final.drop('_merge', 1)
# Calculate debris relative to beach size
mdmap_final['Cloth Per Sq Meter'] = mdmap_final['Cloth']/mdmap_final['TotalArea']
mdmap_final['Fishing Gear  Per Sq Meter'] = mdmap_final['Fishing Gear']/mdmap_final['TotalArea']
mdmap_final['Glass Per Sq Meter'] = mdmap_final['Glass']/mdmap_final['TotalArea']
mdmap_final['Metal Per Sq Meter'] = mdmap_final['Metal']/mdmap_final['TotalArea']
mdmap_final['Other Per Sq Meter'] = mdmap_final['Other']/mdmap_final['TotalArea']
mdmap_final['Plastic Per Sq Meter'] = mdmap_final['Plastic']/mdmap_final['TotalArea']
mdmap_final['Processed Lumber Per Sq Meter'] = mdmap_final['Processed Lumber']/mdmap_final['TotalArea']
mdmap_final['Rubber Per Sq Meter'] = mdmap_final['Rubber']/mdmap_final['TotalArea']
mdmap_final['Total Debris Per Sq Meter'] = mdmap_final['Total Debris']/mdmap_final['TotalArea']
mdmap_final.head()<jupyter_output><empty_output><jupyter_text>How well does the computed plastic count align with the recorded plastic count?<jupyter_code>sns.regplot(x=mdmap_final["Plastic Count"], y=mdmap_final["Plastic"])<jupyter_output><empty_output><jupyter_text>Save to file.<jupyter_code>mdmap_final.to_csv('data_processed/mdmap_totals_by_category.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09446254071661238, 'fraction_numerical': 0.0005616084465910367, 'mean_word_length': 1.261046216353479, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 51, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '368985', 'license': 'MIT', 'n_tokens_mistral': 2495, 'n_tokens_neox': 2248, 'n_words': 461, 'path': 'mdmap_totals_by_category.ipynb', 'repo_name': 'DoshiHarsh/earth-challenge-2020-plastics', 'size': '8903'}