<jupyter_start><jupyter_text>Let's make sure our data files are there. The `os` library will be very helpful.<jupyter_code>from os import listdir
from os.path import join

listdir('.')
listdir('../data')<jupyter_output><empty_output><jupyter_text>We will rely on `pandas`'s built in `read_csv` function.<jupyter_code>import pandas as pd
pd.read_csv?<jupyter_output><empty_output><jupyter_text>What other "read" functions does `pandas` have?<jupyter_code>import re
regex = re.compile(r'read')
list(filter(regex.match, dir(pd)))<jupyter_output><empty_output><jupyter_text>Let's focus on `read_csv` for now, and investigate some data from [openflights.org](openflights.org)<jupyter_code>routes = pd.read_csv(join('..', 'data', 'routes.csv'))
airports = pd.read_csv(join('..', 'data', 'airports.csv'))
routes.head()
routes.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 67663 entries, 0 to 67662
Data columns (total 9 columns):
airline       67663 non-null object
airline_id    67663 non-null object
source        67663 non-null object
source_id     67663 non-null object
dest          67663 non-null object
dest_id       67663 non-null object
codeshare     14597 non-null object
stops         67663 non-null int64
equipment     67645 non-null object
dtypes: int64(1), object(8)
memory usage: 4.6+ MB<jupyter_text>Looks like there's some missing data points...can we visualize what that looks like?<jupyter_code>import missingno as msno<jupyter_output><empty_output><jupyter_text>`missingno` is a library that wraps together handy data utilities to get a visual summary of data completeness. We will only scratch the surface of this library.<jupyter_code>%matplotlib inline
msno.matrix(routes)<jupyter_output><empty_output><jupyter_text>This gives a quick display to visually check out your data. The sparkline on the right shows the completeness of the data. Let's check out airports.<jupyter_code>airports.info()
msno.matrix(airports)<jupyter_output><empty_output><jupyter_text>Cool! Overall the csvs look pretty clean, with relatively complete data. We've got ~68000 airline routes and ~7000 airports around the world. Let's see if this list is real by traing to find everyone's favorite airport - LaGuardia (LGA).<jupyter_code>airports[airports['IATA'] == 'LGA']<jupyter_output><empty_output><jupyter_text>Awesome. How about finding some other things like the highest airport in the world, highest airport in the country that comes last alphabetically, and the highest airport in Australia.<jupyter_code>highest = airports.sort_values('altitude', ascending=False).iloc[0]
highestInLastAlpha = airports.sort_values(['country', 'altitude'], ascending=[False, False]).iloc[0]
highestInAustralia = airports[airports['country'] == 'Australia'].sort_values('altitude', ascending=False).iloc[0]
print(f'Highest airport: \n{highest}\n\n')
print(f'Highest airport in the country that comes last alphabetically: \n{highestInLastAlpha}\n\n')
print(f'Highest airport in Australia: \n{highestInAustralia}\n\n')<jupyter_output>Highest airport: 
id                             9310
name        <NAME>ading Airport
city                       Daocheng
country                       China
IATA                            DCY
ICAO                           ZUDC
lat                         29.3231
long                        100.053
altitude                      14472
timezone                          8
dst                               N
tzinfo                Asia/Shanghai
type                        airport
source                  OurAirports
Name: 6542, dtype: object


Highest airport in the country that comes last alphabetically: 
id                                                 1005
name        <NAME> International Airport
city                                             Harare
country                                        Zimbabwe
IATA                                                HRE
ICAO                                               FVHA
lat                                            -17[...]<jupyter_text>Let's do some more practical things with this data like finding the 10 busiest routes in the world.<jupyter_code>busiest = routes.groupby(['source', 'dest']).count()['airline_id'].nlargest(10)
busiest<jupyter_output><empty_output><jupyter_text>Merge in the names of the airports<jupyter_code>busiest = pd.DataFrame(busiest).reset_index()
busiest.head()
for sd in ['source', 'dest']:
    busiest = busiest.merge(airports[['IATA', 'name']].set_index('IATA'), how='left', left_on=sd, right_on='IATA')

busiest<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09379099256668125, 'fraction_numerical': 0.02470485351989506, 'mean_word_length': 3.0630550621669625, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '213326', 'license': 'Apache-2.0', 'n_tokens_mistral': 1437, 'n_tokens_neox': 1309, 'n_words': 385, 'path': 'notebooks/3_flights.ipynb', 'repo_name': 'jmn319/python-training', 'size': '4593'}