<jupyter_start><jupyter_text>We are going to look at OSHA recorded severe injuries between January 2015 and Spetmeber 2020. Interested to see if there is a pattern in what injuries lead to hospitalization and or amputation.<jupyter_code>injury.head()<jupyter_output><empty_output><jupyter_text>There are a number of columns that we can elminate to make the data more manageable. Since most labor laws are made on the state and federal level we will drop location data except for state. Further inspection could be done on the dropped columns, but we are going to focus our search for now. The source data noted the the Lat and Long columns may be unreliable so we will be dropping them.We can also drop the UPA ID as it is a duplicate ID, and the Final Narrative. We could run NLP on the FInal Narrative, but that is beyond the scope of this review. We will also drop titles columns after constructing dictionaries for labeling them later.<jupyter_code>injury.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 59469 entries, 0 to 59468
Data columns (total 26 columns):
 #   Column                  Non-Null Count  Dtype  
---  ------                  --------------  -----  
 0   ID                      59469 non-null  int64  
 1   UPA                     59469 non-null  int64  
 2   EventDate               59469 non-null  object 
 3   Employer                59469 non-null  object 
 4   Address1                59457 non-null  object 
 5   Address2                5488 non-null   object 
 6   City                    59457 non-null  object 
 7   State                   59469 non-null  object 
 8   Zip                     59455 non-null  float64
 9   Latitude                59441 non-null  float64
 10  Longitude               59441 non-null  float64
 11  Primary NAICS           59467 non-null  object 
 12  Hospitalized            59469 non-null  float64
 13  Amputation              59467 non-null  float64
 14  Inspection              20129 non-null[...]<jupyter_text>Before we drop them we will first fill in our missing fields. - Because we don't intend to use them in our analysis we will ignore the following: Address1, Address2, City, Zip, Latitude, Longitude. - Primary NAICS will be filled with 0 as will hospitalization and amputation. - Inspections will be converted to a binary column as the report numbers don't provide us with further information.- Secondary Souce will be filled with 0 and Secondary Source Title will be filled with None<jupyter_code>fill = {'Primary NAICS':0, 'Hospitalized':0, 'Amputation':0, 'Inspection':0, 
        'Secondary Source':0, 'Secondary Source Title':'None'}
injury.fillna(value=fill, inplace=True)
injury.loc[injury['Inspection']!=0, 'Inspection'] = 1
injury.head()
injury1 = injury.drop(columns=['UPA', 'Address1', 'Address2', 'City', 'Zip', 'Latitude', 'Longitude'])
injury1.info()
pob = injury1[['Part of Body', 'Part of Body Title']].groupby('Part of Body').first()
pob_labels = dict(zip(pob.index, pob['Part of Body Title']))
pob_labels
nature = injury1[['Nature', 'NatureTitle']].groupby('Nature').first()
nature_labels = dict(zip(nature.index, nature['NatureTitle']))
nature_labels
event = injury1[['Event', 'EventTitle']].groupby('Event').first()
event_labels = dict(zip(event.index, event['EventTitle']))
event_labels
source = injury1[['Source', 'SourceTitle']].groupby('Source').first()
source_labels = dict(zip(source.index, source['SourceTitle']))
source_labels
sec_source = injury1[['Secondary Source', 'Secondary Source Title']].groupby('Secondary Source').first()
sec_source_labels = dict(zip(sec_source.index, sec_source['Secondary Source Title']))
sec_source_labels
injury2 = injury1.drop(columns=['Part of Body Title', 'NatureTitle', 'EventTitle', 
                               'SourceTitle', 'Secondary Source Title','Final Narrative'])
injury2
injury2.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 59469 entries, 0 to 59468
Data columns (total 13 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   ID                59469 non-null  int64  
 1   EventDate         59469 non-null  object 
 2   Employer          59469 non-null  object 
 3   State             59469 non-null  object 
 4   Primary NAICS     59469 non-null  object 
 5   Hospitalized      59469 non-null  float64
 6   Amputation        59469 non-null  float64
 7   Inspection        59469 non-null  float64
 8   Nature            59469 non-null  int64  
 9   Part of Body      59469 non-null  int64  
 10  Event             59469 non-null  int64  
 11  Source            59469 non-null  int64  
 12  Secondary Source  59469 non-null  float64
dtypes: float64(4), int64(5), object(4)
memory usage: 5.9+ MB<jupyter_text>We should correct some of our datatypes now that we have cleaned out the columns we won't be looking at. Changing numerical columns to numerical datatypes makes for simplier analysis.<jupyter_code>injury2['EventDate'] = pd.to_datetime(injury2['EventDate'])
#injury2['Primary NAICS'] = injury2['Primary NAICS'].astype('int64')
injury2['Hospitalized'] = injury2['Hospitalized'].astype('int64')
injury2['Amputation'] = injury2['Amputation'].astype('int64')
injury2['Inspection'] = injury2['Inspection'].astype('int64')
injury2['Secondary Source'] = injury['Secondary Source'].astype('int64')
injury2.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 59469 entries, 0 to 59468
Data columns (total 13 columns):
 #   Column            Non-Null Count  Dtype         
---  ------            --------------  -----         
 0   ID                59469 non-null  int64         
 1   EventDate         59469 non-null  datetime64[ns]
 2   Employer          59469 non-null  object        
 3   State             59469 non-null  object        
 4   Primary NAICS     59469 non-null  object        
 5   Hospitalized      59469 non-null  int64         
 6   Amputation        59469 non-null  int64         
 7   Inspection        59469 non-null  int64         
 8   Nature            59469 non-null  int64         
 9   Part of Body      59469 non-null  int64         
 10  Event             59469 non-null  int64         
 11  Source            59469 non-null  int64         
 12  Secondary Source  59469 non-null  int64         
dtypes: datetime64[ns](1), int64(9), object(3)
memory usage: 5.9+ MB<jupyter_text>Let's use our new datetime column to create a month and year column<jupyter_code>injury2['year'] = injury2['EventDate'].dt.year
injury2['month'] = injury2['EventDate'].dt.month
injury2
fig, (ax1, ax2) = plt.subplots(2, figsize=(12,8))
injury2.groupby('year')['Amputation'].sum().plot(ax=ax1, kind='bar')
ax1.set_title('Amputations by Year')
ax1.set_xlabel('Year')
ax1.grid(b=True, axis='y')
ax1.set_ylabel('Amputations')

injury2.groupby('year')['Hospitalized'].sum().plot(ax=ax2, kind='bar')
ax2.set_title('Hospitalizations by Year')
ax2.set_xlabel('Year')
ax2.set_ylabel('Hospitalizations')
ax2.grid(b=True, axis='y')
fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Remember that we are missing 3 months of data for 2020, but much of the drop off was likely from COVID reducing the amount of total work. Both do show a trend upwards over the last few years, but it would be important to compare this to total hours worked to see if the number of incidents vs hours worked was increasing or decreasing.<jupyter_code>fig, (ax1, ax2) = plt.subplots(2, figsize=(12,8))
injury2.groupby('month')['Amputation'].sum().plot(ax=ax1, kind='bar')
ax1.set_title('Amputations by Month')
ax1.set_xlabel('Month')
ax1.set_ylabel('Amputations')
ax1.grid(b=True, axis='y')

injury2.groupby('month')['Hospitalized'].sum().plot(ax=ax2, kind='bar')
ax2.set_title('Hospitalizations by Month')
ax2.set_xlabel('Month')
ax2.set_ylabel('Hospitalizations')
ax2.grid(b=True, axis='y')
fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>There appears to be a trend up during the later summer months when more construction work is happening as well as in January. The January uptick is likely due to a return from holidays when people are not quite focused back on work yet.<jupyter_code>injury2['State'].unique()<jupyter_output><empty_output><jupyter_text>Let's quickly clean this up so that we don't have duplicates<jupyter_code>state_mapper = {'NY':'NEW YORK', 'WI':'WISCONSIN', 'PA':'PENNSYLVANIA', 'GA':'GEORGIA', 
                'FL':'FLORIDA', 'CO':'COLORADO', 'OK':'OKLAHOMA', 'TX':'TEXAS', 'LA':'LOUISIANA',
                'MI':'MISSISSIPPI','NJ':'NEW JERSEY', 'OH':'OHIO', 'IL':'ILLINOIS', 'NE':'NEBRASKA',
                'NH':'NEW HAMPSHIRE', 'KS':'KANSAS', 'MA':'MASSACHUSETTS', 'AR':'ARKANSAS', 
                'MI':'MICHIGAN', 'ID':'IDAHO', 'MI':'MISSOURI', 'ME':'MAINE', 'CT':'CONNECTICUT',
                'WV':'WEST VIRGINIA', 'ND':'NORTH DAKOTA', 'MT':'MONTANA', 'NC':'NORTH CAROLINA',
                'DE':'DELAWARE', 'CA':'CALIFORNIA', 'DC':'DISTRICT OF COLUMBIA', 'AL':'ALABAMA',
                'TN':'TENNESSEE', 'OR':'OREGON', 'SD':'SOUTH DAKOTA', 'RI':'RHODE ISLAND', 
                'IN':'INDIANA', 'VA':'VIRGINIA', 'NM':'NEW MEXICO', 'MD':'MARYLAND', 'UT':'UTAH', 
                'VT':'VERMONT', 'AZ':'ARIZONA', 'IA':'IOWA', 'KY':'KENTUCKY', 'MN':'MINNESOTA',
                'WA':'WASHINGTON', 'SC':'SOUTH CAROLINA', 'HI':'HAWAII', 'PR':'PUERTO RICO', 
                'VI':'VIRGIN ISLANDS','GU':'GUAM', 'NV':'NEVADA', 'WY':'WYOMING', 
                'AK':'ALASKA', 'NMI':'NORTHERN MARIANA ISLANDS', 'AS':'AMERICAN SAMOA'} 

injury2['State'].replace(state_mapper, inplace=True)
injury2['State'].unique()
injury2.groupby('State')['ID'].count().sort_values(ascending=False).head(15)<jupyter_output><empty_output><jupyter_text>While it isn't surprsing to see may of the higher population states represented in the top 10, it is surprising to see CA not in the top 15 as the most populous state. Let's take a look at what the top Nature of injury is for each state.<jupyter_code>nature_max = injury2.groupby('State')['Nature'].apply(lambda x: x.value_counts()
                                                      .sort_values(ascending=False).head(1)).unstack()
nature_max.head()<jupyter_output><empty_output><jupyter_text>Apparently 5 Natures of injury are consistenly make up all the max injuries for each state.<jupyter_code>nature_max.columns = nature_max.columns.map(nature_labels)
nature_max.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1258119984715323, 'fraction_numerical': 0.04585403133358808, 'mean_word_length': 3.3331953642384105, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '580288', 'license': 'MIT', 'n_tokens_mistral': 3742, 'n_tokens_neox': 3201, 'n_words': 967, 'path': 'OSHA.ipynb', 'repo_name': 'Bwils171/OSHA-Severe-Injuries', 'size': '10468'}