<jupyter_start><jupyter_text>Analysis & Visualization> Correlation analysis and visualization.<jupyter_code>#hide
from nbdev.showdoc import *
#export
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sully import eda
data = eda.load('bird-strikes')<jupyter_output><empty_output><jupyter_text>Correlations, take 2 Earlier we tried to look into correlations, but failed because too many columns were categorial. We can cast those categories into `pd.Categorical` types and then access their automagically created numeric codes so that a numerical analysis, such as `pd.DataFrame.corr()`, will produce proper results.<jupyter_code>#export
def makeCatCodes(data:pd.DataFrame, columns:list)->pd.DataFrame:
    for col in columns:
        data[col] = pd.Categorical(data[col])

    codes = {}
    for col in columns:
        codes[col] = data[col].cat.codes
        
    return pd.DataFrame(codes)
data_cc = makeCatCodes(data.copy(), eda.findCategoricalCandidates(data)['name'])
fig, ax = plt.subplots(figsize=(14,14))
sns.heatmap(data_cc.corr(), vmin=-1, vmax=1, cmap='coolwarm', annot=True, ax=ax);<jupyter_output><empty_output><jupyter_text>Several clusters can be identified directly:* `{'Effect: Indicated Damage', 'Altitude bin', 'Wildlife: Size', 'When: Time of day'}²`* `{'Aircraft: Airline/Operator', 'Aircraft: Make/Model', 'Wildlife: Species'} x {'Effect: Indicated Damage', 'Altitude bin', 'Wildlife: Size'}`* `{'When: Time of day', 'Effect: Impact to flight', 'Aircraft: Number of engines?'}²`* `{'When: Time of day', 'Effect: Impact to flight', 'Aircraft: Number of engines?'} x {'Speed (IAS) in knots', 'Aircraft: Make/Model', 'Wildlife: Species}` `Effect: {Impact to flight, Indicated Damage}` are both related, yet one is missing data for nearly half the records, whereas the other is mostly complete. Instead of attempting to fix `Effect: Impact to flight` we will stick to `Effect: Indicated Damage`.<jupyter_code>data['Effect: Impact to flight'].isna().sum() / len(data), data['Effect: Indicated Damage'].isna().sum() / len(data)<jupyter_output><empty_output><jupyter_text>Airport: Name &rarr; Effect: Indicated Damage Notice that we use crosstables to get margins on each aggregated sub table. Margins show up as an additional `All` column. This simplifies sorting and makes for easier "top x"-style queries.<jupyter_code>data_ct = pd.crosstab([data['Airport: Name']], data['Effect: Indicated Damage'], margins=True)
data_ct.sort_values(by='All', ascending=False)[1:].head(5)<jupyter_output><empty_output><jupyter_text>We limit our result to the first 20 records via dataframe slicing like so: `some_df[-21:-1]`. We skip the first record because in our reversely sorted dataframe, this would show the totals (margins) for each column<jupyter_code>data_ct.sort_values(by='All', ascending=True)[-21:-1].plot.barh(figsize=(10,10));<jupyter_output><empty_output><jupyter_text>We can see that certain airports among the top 20, such as Denver INTL Airport or Dallas/Fort Worth INTL ARPT, report bird strikes at a much higher rate. A quick check reveals those airports to be near military (air) bases.The majority of reported bird strikes cause no damage to the aircraft.  Aircraft: Airline/Operator, Airport: Name &rarr; Effect: Indicated Damage<jupyter_code>data_op_ct = pd.crosstab([data['Aircraft: Airline/Operator'], data['Airport: Name']], data['Effect: Indicated Damage'], margins=True)
data_op_ct.sort_values(by='All', ascending=True)[-21:-1].plot.barh(figsize=(8,10));<jupyter_output><empty_output><jupyter_text>Southwest Airlines shows up often, with Fedex Express at Memphis International taking the 2nd spot and UPS Airlines at Louisville INTL ARPT taking the 4th spot. Dallas/Fort Worth INTL ARPT and Denver INTL Airport are dominated by 'unknown'. The latter two are airfreight carriers, with Southwest Airlines running a [domestic airfreight branch](https://www.swacargo.com/). One can see that Fedex Express also claims the majority of bird strike reports at Memphis International (86.7%):<jupyter_code>data_op_ct.loc['FEDEX EXPRESS', 'MEMPHIS INTL']['All'] / data_ct.loc['MEMPHIS INTL']['All']<jupyter_output><empty_output><jupyter_text>UPS Airlines dominates bird strike reports at Louisville International Airport (76.4%):<jupyter_code>data_op_ct.loc['UPS AIRLINES', 'LOUISVILLE INTL ARPT']['All'] / data_ct.loc['LOUISVILLE INTL ARPT']['All']<jupyter_output><empty_output><jupyter_text>Let's check if we find other airports where bird strike reports are dominated by a single airline operator.<jupyter_code>operators = []
airports = []
ratios = []
totals = []
bounds = (.1, 1)
min_reports = 100

for idx in data_op_ct.index[:-1]:
    op = idx[0]
    ap = idx[1]
    
    t = data_ct.loc[ap]['All']
    r = data_op_ct.loc[idx]['All'] / max(1, t)

    if r >= bounds[0] and r < bounds[1] and t >= min_reports:
        operators.append(op)
        airports.append(ap)
        ratios.append(r)
        totals.append(t)

op_ap_r = pd.DataFrame({'operator': operators,
                        'airports': airports,
                        'ratio': ratios,
                        'totals': totals})
op_ap_r.sort_values(by='ratio', ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Military bases obviously dominate bird strike reports at their respective airport, so let's filter them out.<jupyter_code>op_ap_r[op_ap_r['operator'] != 'MILITARY'].sort_values(by='ratio', ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Memphis INTL and Louisville INTL ARPT look like air freight hubs (see operators). If air freight operations take mostly place at night, then their abnormal flight schedule could impose a higher risk of bird strikes, too.  Air freight vs. other Aircraft: Airline/Operators<jupyter_code>airfreight_pred = data['Aircraft: Airline/Operator'].isin(['FEDEX EXPRESS', 'UPS AIRLINES'])
data.groupby('When: cat').count()['When: Time (HHMM)'].plot.barh(figsize=(12,12))
data[airfreight_pred].groupby('When: cat').count()['When: Time (HHMM)'].plot.barh(color='orange');<jupyter_output><empty_output><jupyter_text>The time of day categories generated by `pd.qcut` are often close to one hour in range, apart from the late night/early hours categories. This indicates that flight schedules are mostly stable from noon to midnight (in terms of frequency).We can assume that airfreight carriers mostly operate during night time, which is why for them the risk of bird strikes is naturally higher in those hours.  Wildlife: Size, Altitude bin &rarr; Effect: Indicated Damage<jupyter_code>pd.crosstab([data['Wildlife: Size'],
             data['Altitude bin']],
             data['Effect: Indicated Damage'],
            margins=True).sort_values(by='All', ascending=True)[:-1].plot.barh(figsize=(10, 4));<jupyter_output><empty_output><jupyter_text>Bird strikes seem to happen with much higher frequency at lower altitudes. Smaller birds cause more reports! Large birds might be rarer, or perhaps it's the nature of smaller birds flying in large swarms.  When: Time (HHMM) &rarr; Effect: Indicated Damage Notice that we fold `When: Time (HHMM)` into hours for improved clarity. The column uses military time.<jupyter_code>pd.crosstab(data['When: Time (HHMM)'] // 100 * 100,
            data['Effect: Indicated Damage'],
            margins=True)[:-1].plot.bar(figsize=(12, 8));<jupyter_output><empty_output><jupyter_text>We see a strong daily increase in bird strike reports beginning at 0600 til 1100, then a decline during noon/early afternoon (1200-1600) and again a rise in the evening/late night (1700-2300), followed by a sharp drop during night hours (0000-0500).  When: Time of day, Aircraft: Number of engines? &rarr; Effect: Indicated Damage<jupyter_code>pd.crosstab([data['When: Time of day'],
             data['Aircraft: Number of engines?']],
             data['Effect: Indicated Damage'],
            margins=True).sort_values(by='All', ascending=True)[:-1].plot.barh(figsize=(12, 8));<jupyter_output><empty_output><jupyter_text>What shows up as strong correlation might just be a result of frequency wrt. twin-engine aircrafts flying during the day (read: almost all commercial airlines or modern military jets).  Aircraft: Make/Model &rarr; Effect: Indicated Damage<jupyter_code>pd.crosstab(data['Aircraft: Make/Model'],
            data['Effect: Indicated Damage'],
            margins=True).sort_values(by='All', ascending=True)[-21:-1].plot.barh(figsize=(12, 8));<jupyter_output><empty_output><jupyter_text>Aircraft models favored by (commercial) operators fly more often and therefore report more bird strikes, naturally. Military aircrafts report the majority of bird strikes.It's difficult to see whether the size of an aircraft, such as the massive Boeing 737 variants as compared to a much smaller model such as the DHC8 DASH 8, has an effect. Type of engine, such as turboprop vs jet engines, which is a distinctive feature among aircraft models, could also influence this figure.  Speed (IAS) in knots, When: Phase of flight &rarr; Effect: Indicated Damage<jupyter_code>pd.crosstab(data['Speed (IAS) in knots'],
            data['Effect: Indicated Damage'],
            margins=True).sort_values(by='All', ascending=True)[-21:-1].plot.bar(figsize=(12, 8));<jupyter_output><empty_output><jupyter_text>There is a speed interval around 100-180 knots that reports the majority of bird strike reports.<jupyter_code>pd.crosstab(data['Speed (IAS) in knots'],
            data['When: Phase of flight'],
            margins=True).sort_values(by='All', ascending=True)[-11:-1].plot.bar(figsize=(12, 8));<jupyter_output><empty_output><jupyter_text>Most bird strike seemingly happen during approach, with a speed (IAS) between 120 and 140 knots.Climb phase (at slightly higher speeds) and descent (at approx. twice the speed) also show an increase in bird strike reports.  Seasonal effects<jupyter_code>data.groupby(['FlightDate']).count()['When: Time of day'].plot(figsize=(12,8));
data['FlightDate: Month'] = data['FlightDate'].dt.strftime('%B') # dependent on locale, use .dt.month elsewhere!
data.groupby(['FlightDate: Month']).count()['FlightDate'].sort_values().plot.bar(figsize=(12,8));<jupyter_output><empty_output><jupyter_text>We see a seasonal increase in bird strikes during summer and autumn, from July to October. Most birds probably migrate to the south during the colder seasons (given that the data is from the US, which is in the northern hemisphere), so this is entirely expected.  When: Time (HHMM) &rarr; Effect: Indicated Damage over Bird Season vs. ~Bird Season<jupyter_code>bird_season = [7, 8, 9, 10]

data_bs = data[data['FlightDate'].dt.month.isin(bird_season)]
data_not_bs = data[~data['FlightDate'].dt.month.isin(bird_season)]

(pd.crosstab(data_bs['When: Time (HHMM)'] // 100 * 100,
             data_bs['Effect: Indicated Damage'],
             margins=True) - pd.crosstab(data_not_bs['When: Time (HHMM)'] // 100 * 100,
                                         data_not_bs['Effect: Indicated Damage'],
                                         margins=True))[:-1].plot.bar(figsize=(12, 8));<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11981030780243379, 'fraction_numerical': 0.015032211882605583, 'mean_word_length': 5.1479647964796476, 'pattern_counts': {'":': 0, '<': 78, '<?xml version=': 0, '>': 81, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '578575', 'license': 'Apache-2.0', 'n_tokens_mistral': 3594, 'n_tokens_neox': 3348, 'n_words': 1102, 'path': '01_analysis.ipynb', 'repo_name': 'mikhas/sully', 'size': '11176'}