<jupyter_start><jupyter_text>Data Exploration<jupyter_code>import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Setup For the scope of this tutorial we are going to use AirBnb Scraped data for the city of Bologna. The data is freely available at **Inside AirBnb**: http://insideairbnb.com/get-the-data.html.A description of all variables in all datasets is avaliable [here](https://docs.google.com/spreadsheets/d/1iWCNJcSutYqpULSQHlNyGInUvHg2BoUGoNRIGa6Szc4/editgid=982310896).We are going to use 2 datasets:- listing dataset: contains listing-level information- pricing dataset: contains pricing data, over time  Importing Data Pandas has a variety of function to import data- `pd.read_csv()`- `pd.read_html()`- `pd.read_parquet()`Importatly for our purpose, `pd.read_csv()` can directly import data from the web. The first dataset that we are going to import is the dataset of Airbnb listings in Bologna. It contains listing-level information.<jupyter_code>url_listings = "http://data.insideairbnb.com/italy/emilia-romagna/bologna/2021-12-17/visualisations/listings.csv"
df_listings = pd.read_csv(url_listings)<jupyter_output><empty_output><jupyter_text>The second dataset that we are going to use is the dataset of calendar prices. This time the dataset is compressed but we can use the `compression` option to import it directly.<jupyter_code>url_prices = "http://data.insideairbnb.com/italy/emilia-romagna/bologna/2021-12-17/data/calendar.csv.gz"
df_prices = pd.read_csv(url_prices, compression="gzip")<jupyter_output><empty_output><jupyter_text>Inspecting Data > Methods>>- `info()`>- `head()`>- `describe()` The first way yo have a quick look at the data is the `info()` method. If called with the option `verbose=False`, it gives a quick overview of the dimensions of the data.<jupyter_code>df_listings.info(verbose=False)<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 3453 entries, 0 to 3452
Columns: 18 entries, id to license
dtypes: float64(4), int64(8), object(6)
memory usage: 485.7+ KB<jupyter_text>If we want to know how the data looks like, we can use the `head()` method. It prints the first 5 lines of the data by default.<jupyter_code>df_listings.head()<jupyter_output><empty_output><jupyter_text>We can print a description of the data using `describe()`. If we have many variables, it's best to print it transposed using the `.T` attribute.<jupyter_code>df_listings.describe().T[:5]<jupyter_output><empty_output><jupyter_text>You can select which variables to display using the `include` option. `include='all'` includes also categorical variables.<jupyter_code>df_listings.describe(include='all').T[:5]<jupyter_output><empty_output><jupyter_text>We can get the list of columns using the `.columns` attribute.<jupyter_code>df_listings.columns<jupyter_output><empty_output><jupyter_text>We can get the index using the `.index` attribute,<jupyter_code>df_listings.index<jupyter_output><empty_output><jupyter_text>Data Selection We can access single columns as if the DataFrame was a dictionary.<jupyter_code>df_listings['price']<jupyter_output><empty_output><jupyter_text>We can select rows and columns by index, using the `.iloc` attribute.<jupyter_code>df_listings.iloc[:7, 5:9]<jupyter_output><empty_output><jupyter_text>If we want to condition only on rows or columns, we have use `:` for the unrestricted dimesion, otherwise we get an error.<jupyter_code>df_listings.iloc[:, 5:9].head()<jupyter_output><empty_output><jupyter_text>Instead, the `.loc` attribute allows us to use row and column names.<jupyter_code>df_listings.loc[:, ['neighbourhood', 'latitude', 'longitude']].head()<jupyter_output><empty_output><jupyter_text>We can also select ranges.<jupyter_code>df_listings.loc[:, 'neighbourhood':'room_type'].head()<jupyter_output><empty_output><jupyter_text>There is an easy way to **select numerical columns**, the `.select_dtypes()` function.<jupyter_code>df_listings.select_dtypes(include=['number']).head()<jupyter_output><empty_output><jupyter_text>Other types include- `object` for strings- `bool` for booleans- `int` for integers- `float` for floats (numbers that are not integers) We can also use logical operators to selet rows.<jupyter_code>df_listings.loc[df_listings['number_of_reviews']>500, :].head()<jupyter_output><empty_output><jupyter_text>We can use logical operations as well. But remember to use paranthesis.**Note**: the `and` and `or` expressions do not work in this setting. We have to use `&` and `|` instead.<jupyter_code>df_listings.loc[(df_listings['number_of_reviews']>300) &
                (df_listings['reviews_per_month']>7), 
                :].head()<jupyter_output><empty_output><jupyter_text>For a single column (i.e. a Series), we can get the unique values using the `unique()` function.<jupyter_code>df_listings['neighbourhood'].unique()<jupyter_output><empty_output><jupyter_text>For multiple columns, we can use the `drop_duplicates` function.<jupyter_code>df_listings[['neighbourhood', 'room_type']].drop_duplicates()<jupyter_output><empty_output><jupyter_text>Aggregation and Pivot Tables We can compute statistics by group using `.groupby()`.<jupyter_code>df_listings.groupby('neighbourhood')[['price', 'reviews_per_month']].mean()<jupyter_output><empty_output><jupyter_text>If you want to perform more than one function, maybe on different columns, you can use `.aggregate()` which can be shortened to `.agg()`. It takes as argument a dictionary with variables as keys and lists of functions as values.<jupyter_code>df_listings.groupby('neighbourhood').agg({"reviews_per_month": ["mean"],
                                          "price": ["min", np.max]}).reset_index()<jupyter_output><empty_output><jupyter_text>The problem with this syntax is that it generates a hierarchical structure for variable names, which might not be so easy to work with. In the example above, to access the mean price, you have to use `df.price["min"]`.To perform variable naming and aggregation and the same time, you can ise the following syntax: `agg(output_var = ("input_var", function))`.<jupyter_code>df_listings.groupby('neighbourhood').agg(mean_reviews=("reviews_per_month", "mean"),
                                         min_price=("price", "min"),
                                         max_price=("price", np.max)).reset_index()<jupyter_output><empty_output><jupyter_text>We can make pivot tables with the `.pivot_table()` function. It takes the folling arguments:- `index`: rows- `columns`: columns- `values`: values- `aggfunc`: aggregation function<jupyter_code>df_listings.pivot_table(index='neighbourhood', columns='room_type', values='price', aggfunc='mean')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11594634873323398, 'fraction_numerical': 0.009985096870342772, 'mean_word_length': 6.342450765864332, 'pattern_counts': {'":': 2, '<': 93, '<?xml version=': 0, '>': 101, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '229170', 'license': 'MIT', 'n_tokens_mistral': 2223, 'n_tokens_neox': 2124, 'n_words': 590, 'path': 'notebooks/02_data_exploration.ipynb', 'repo_name': 'matteocourthoud/Data-Science-Python', 'size': '6710'}