<jupyter_start><jupyter_text>Pandas [Download relevant files here](https://melaniewalsh.org/Pandas.zip) In this lesson, we're going to learn about [Pandas](https://pandas.pydata.org/pandas-docs/stable/getting_started/overview.html), a powerful Python library for working with tabular data like CSV files. Things that were kind of clunky to accomplish with the csv module are much easier to accomplish with pandas.Here are some of the things you can do with Pandas:* Easily read and write CSV files* Sort and filter data* Analyze data* Combine datasets* Create data visualizations  Our Dataset  >Computationcould not, it seemed, capture the violent quandary that was the nation’shistory of and relationship to human bondage. Contemporary encounterswith digital technology have inherited this tension, with researchersstruggling to appreciate the inhumanity of bondage and the attendantdehumanization of black lives while also responding to the need for critical,rigorous, and engaged histories of slavery as histories of the present (18).> [D]isplaying dataalone could not and did not offer the atonement descendants of slavessought or capture the inhumanity of this archive’s formation. Cullingthe lives of women and children from the data set required approachingthe data with intention. It required a methodology attuned to black lifeand to dismantling the methods used to create the manifests in the firstplace, then designing and launching an interface responsive to the desireof descendants of slaves for reparation and redress (65).<NAME>,["Markup Bodies: Black [Life] Studies and Slavery [Death] Studies at the Digital Crossroads"](https://read.dukeupress.edu/social-text/article/36/4%20(137)/57/137032/Markup-BodiesBlack-Life-Studies-and-Slavery-Death) The dataset that we're going to be working with in this lesson is taken from [The Trans-Atlantic Slave Trade Database](https://www.slavevoyages.org/voyage/database), part of the [*Slave Voyages* project](https://www.slavevoyages.org/). This dataset was filtered to include only North American disembarkation locations and to include data about the percentage of men, women, and children on the voyages. Some column names have been altered or modified.We're working with this data for a number of reasons. First, the *Slave Voyages* project is a major data-driven contribution to the history of slavery and to the field of the digital humanities. Second, working with this data makes clear that, as Johnson writes, computation and data alone cannot capture "the violent quandary" of slavery or bring about justice. As we learn how to manipulate and analyze this dataset with Pandas, we should be reminded, at every step, of the challenges that are involved in using this data responsibly and intentionally.  Import Pandas To use the pandas library, we first need to `import` it, as below. This `import` statement not only imports the library but also gives it a handy little nickname, `as` "pd" instead of "pandas." You'll run across this alias convention a lot. It's a way to save time and not have to write the entire words "pandas" over and over again.<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Read in CSV File To read in a CSV file, you use the function `pd.read_csv()` and insert the name of your desired file path. You can also add the "delimiter" argument to specify the delimiter for your file.<jupyter_code>slave_voyages_df = pd.read_csv('../data/Slave-Voyages-Trans-Atlantic-North-America.csv', delimiter=",")<jupyter_output><empty_output><jupyter_text>**Pro tip!** If you use the `help()` function, you can see the documentation for almost any bit of code. If we run it on `pd.read_csv()`, we can see all the possible parameters that can be used with `pd.read_csv()`.<jupyter_code>help(pd.read_csv)<jupyter_output>Help on function read_csv in module pandas.io.parsers:

read_csv(filepath_or_buffer: Union[str, pathlib.Path, IO[~AnyStr]], sep=',', delimiter=None, header='infer', names=None, index_col=None, usecols=None, squeeze=False, prefix=None, mangle_dupe_cols=True, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skipinitialspace=False, skiprows=None, skipfooter=0, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=False, skip_blank_lines=True, parse_dates=False, infer_datetime_format=False, keep_date_col=False, date_parser=None, dayfirst=False, cache_dates=True, iterator=False, chunksize=None, compression='infer', thousands=None, decimal=b'.', lineterminator=None, quotechar='"', quoting=0, doublequote=True, escapechar=None, comment=None, encoding=None, dialect=None, error_bad_lines=True, warn_bad_lines=True, delim_whitespace=False, low_memory=True, memory_map=False, float_precision=None)
    Read a comma-separated values (csv) file into Dat[...]<jupyter_text>When you read in a CSV file with pandas, you transform it into a pandas object called a dataframe.<jupyter_code>type(slave_voyages_df)<jupyter_output><empty_output><jupyter_text>Display the Data In a Jupyter notebook, you can display a dataframe simply by running a cell with the name of the dataframe. The `NaN` value is the Pandas value for any missing data (`NaN` values have special properties that we'll get to later).<jupyter_code>slave_voyages_df<jupyter_output><empty_output><jupyter_text>You should always eyeball your dataset either by looking at the first few rows or by looking at a few random rows. To look at the first few rows, you can use a method called `.head()` (very similar to the `head` command on the command line).<jupyter_code>slave_voyages_df.head(10)<jupyter_output><empty_output><jupyter_text>To look at a random sample of rows, you can use the `.sample()` method. You might run this cell a few times to make sure everything looks ok.<jupyter_code>slave_voyages_df.sample(10)<jupyter_output><empty_output><jupyter_text>Examine the Data You can check the dataframe's "shape" or how many rows vs columns it contains with `.shape` (number of rows, number of columns)<jupyter_code>slave_voyages_df.shape<jupyter_output><empty_output><jupyter_text>You can (and always should) check the data types that each column contains with `.dtypes` | **Pandas dtypes** |                                                                                    ||:-------------:|:---------------------------------------------------------------------------------------------------:|| `object`         | string                                                                               || `float64`         | float                                               || `int64`       | integer                                                        || `datetime64`       |  date time<jupyter_code>slave_voyages_df.dtypes<jupyter_output><empty_output><jupyter_text>It's important to always check the data types in your dataframe. For example, sometimes numeric values, which you want to do mathematical calculations with, will accidentally be interpreted as a string object. To perform calculations on this data, you would need to first convert that column from a string to an integer.The data types in our Trans-Atlantic Slave Trade dataset look ok, except the column "year_of_arrival," which is currently interpreted as an integer value when ideally it should be a datetime value. You can also check the column names of your dataframe with `.columns`<jupyter_code>slave_voyages_df.columns<jupyter_output><empty_output><jupyter_text>Rename and Drop Columns Let's say we wanted to rename the "flag" column as "national_affiliation." One way to approach this goal would be to make a new column identical to the "flag" column but with a different name.Making a new column with Pandas is very similar to making a new Python variable. You simply type the name of the dataframe with square brackets `[]` and include the name of your new desired column that doesn't exist yet. Then you assign it the value of the already existing flag column.<jupyter_code>slave_voyages_df['national_affiliation'] = slave_voyages_df['flag']<jupyter_output><empty_output><jupyter_text>If you scroll all the way to the end of the dataframe, you can see that we added a new column called "national_affiliation" that's exactly the same as the flag column.<jupyter_code>slave_voyages_df.head()<jupyter_output><empty_output><jupyter_text>But what if we don't want multiple columns with the same information? What if we just want to rename the "flag" column? First, let's get rid of the "national_affiliation" column so we can try again. To remove columns, you can use the `.drop()` method and the `columns=` parameter, followed by the column you'd like to drop.<jupyter_code>slave_voyages_df = slave_voyages_df.drop(columns='national_affiliation')
slave_voyages_df.columns<jupyter_output><empty_output><jupyter_text>A better way to rename columns is with the `.rename()` method and the `columns=` parameter that includes the original name of the column followed by a colon and the new desired name of the column.<jupyter_code>slave_voyages_df.rename(columns={'flag': 'national_affiliation'})<jupyter_output><empty_output><jupyter_text>Renaming the "flag" column as we did above will only momentarily change that column's name. If we display our dataframe, we'll see that the column name has *not* changed permamently.<jupyter_code>slave_voyages_df.head(5)<jupyter_output><empty_output><jupyter_text>To save changes in your dataframe, you need to keep re-assigning your dataframe, as below.<jupyter_code>slave_voyages_df = slave_voyages_df.rename(columns={'flag': 'national_affiliation'})
slave_voyages_df.head(5)<jupyter_output><empty_output><jupyter_text>Select Columns To isolate a column with the csv module was kind of a pain. We had to loop through each row and index a certain value in each row's list. But with pandas it's very simple to select a column. You can do so by typing the name of the dataframe accompanied by square brackets `[]` with the name of the column in quotation marks.<jupyter_code>slave_voyages_df['vessel_name']
type(slave_voyages_df['vessel_name'])<jupyter_output><empty_output><jupyter_text>Each column in a dataframe is technically a pandas object called a "Series." We won't worry too much about the "Series" object right now, except to note that a Series object does not display as nicely as a DataFrame. To select a column as a dataframe, you can use two square brackets `[[]]`<jupyter_code>slave_voyages_df[['vessel_name']]
type(slave_voyages_df[['vessel_name']])<jupyter_output><empty_output><jupyter_text>With two square brackets, you can also select and isolate multiple columns at the same time.<jupyter_code>slave_voyages_df[['vessel_name', 'national_affiliation', 'year_of_arrival']]<jupyter_output><empty_output><jupyter_text>Filter Data You can also filter data based on different criteria. For example, let's say we wanted to look at the slave voyages whose national affiliation was American. You can evaluate each row in a column by isolating that column and then using the comparison operators that we discussed a few lessons ago, such as equals `==`.<jupyter_code>slave_voyages_df['national_affiliation'] == "U.S.A."<jupyter_output><empty_output><jupyter_text>The output above is an evaluation of whether each row in the "flag" column equals "U.S.A." or not. This function becomes much more helpful when we use this comparison as a selection criteria in its own right.Just as we can select a column by placing its name inside square brackets, we can select filtered data by placing a filter inside square brackets `[]`. Essentially, the line below is saying give me all the rows in the dataframe that match `slave_voyages_df['flag'] == "U.S.A."`<jupyter_code>slave_voyages_df[slave_voyages_df['national_affiliation'] == "U.S.A."]<jupyter_output><empty_output><jupyter_text>You can do the same thing with integers and other comparison operators. For example, if we wanted to filter for only the slave voyages that arrived after 1830, we could do so with `slave_voyages_df['year_of_arrival'] > 1830`<jupyter_code>slave_voyages_df['year_of_arrival'] > 1830
slave_voyages_df[slave_voyages_df['year_of_arrival'] > 1830]<jupyter_output><empty_output><jupyter_text>Sort Columns You can sort a dataframe with the `.sort_values()` method, inside of which you include the parameter `by=` and indicate the name of the column you want to sort by (written in quotation marks).<jupyter_code>slave_voyages_df.sort_values(by='total_disembarked')<jupyter_output><empty_output><jupyter_text>Take a look at the way the dataframe above is sorted. Do you notice that something seems a bit off?By default, pandas will sort in "ascending" order, from the smallest value to the largest value. If you want to sort the largest values first, you need to include another parameter `ascending=False`.<jupyter_code>slave_voyages_df.sort_values(by='total_disembarked', ascending=False)<jupyter_output><empty_output><jupyter_text>If you want to sort a Series object, you don't need to use the `by=` paramter.<jupyter_code>slave_voyages_df['total_disembarked'].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Count Values in Columns To count the unique values in a column, you can use the `.value_counts()` method.<jupyter_code>slave_voyages_df['national_affiliation'].value_counts()
slave_voyages_df['vessel_name'].value_counts()
slave_voyages_df['place_of_slave_disembarkation'].value_counts()<jupyter_output><empty_output><jupyter_text>Calculate Columns | Pandas calculations | Explanation                         ||----------|-------------------------------------|| .count()    | Number of observations    || .sum()      | Sum of values                       || .mean()     | Mean of values                      || .median()   | Median of values         || .min()      | Minimum                             || .max()      | Maximum                             || .mode()     | Mode                                || .std()      | Unbiased standard deviation         | You can do different calculations on columns with built-in Pandas functions, such as `.sum()` and `.max()<jupyter_code>slave_voyages_df['total_disembarked'].sum()
slave_voyages_df['total_embarked'].sum()
slave_voyages_df['total_embarked'].sum() - slave_voyages_df['total_disembarked'].sum()
slave_voyages_df['total_disembarked'].mean()
slave_voyages_df['total_disembarked'].max()<jupyter_output><empty_output><jupyter_text>Groupby Columns One powerful Pandas function is called `.groupby()`, which allows you to aggregate data and perform calculations on it. For example, if we wanted to see how many enslaved people were transported by each nation, we could group by, or aggregate our data based on, "national_affiliation". The first step to using groupby is to type the name of your dataframe followed by `.groupby()` with the column you'd like to aggregate based on.<jupyter_code>slave_voyages_df.groupby('national_affiliation')<jupyter_output><empty_output><jupyter_text>This action will created a "groupby" object, which you won't be able to access unless you perform a calculation on it, such as `.sum()`<jupyter_code>slave_voyages_df.groupby('national_affiliation').sum()<jupyter_output><empty_output><jupyter_text>If we want to isolate only the "total_disembarked" column from this groupby calculation, then we can add in "total_disembarked" in square brackets `[]`<jupyter_code>slave_voyages_df.groupby('national_affiliation')['total_disembarked'].sum()
slave_voyages_df.groupby('national_affiliation')[['total_disembarked']].sum()<jupyter_output><empty_output><jupyter_text>Let's save this groupby calculation into a new variable called `national_totals`<jupyter_code>national_totals = slave_voyages_df.groupby('national_affiliation')[['total_disembarked']].sum()<jupyter_output><empty_output><jupyter_text>Make Plots  We're going to talk about making plots and data visualizations more extensively later. But for now we're going to introduce the fact that you can make simple plots simply by using the [`.plot()` method](https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.DataFrame.plot.html) that's built into Pandas.<jupyter_code>national_totals.plot()
national_totals.plot(kind='bar')
national_totals.plot(kind='barh')<jupyter_output><empty_output><jupyter_text>Time Series<jupyter_code>slave_voyages_df['year_of_arrival'].dtypes<jupyter_output><empty_output><jupyter_text>Transform an integer into a datetime<jupyter_code>slave_voyages_df['year_of_arrival'] = pd.to_datetime(slave_voyages_df['year_of_arrival'], format='%Y', errors='coerce')
slave_voyages_df['year_of_arrival'].value_counts().plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09093088773002457, 'fraction_numerical': 0.0032368279086495235, 'mean_word_length': 5.156457564575645, 'pattern_counts': {'":': 0, '<': 137, '<?xml version=': 0, '>': 142, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '588987', 'license': 'MIT', 'n_tokens_mistral': 4928, 'n_tokens_neox': 4699, 'n_words': 1768, 'path': 'Pandas/Pandas-notebooks/Pandas.ipynb', 'repo_name': 'melaniewalsh/mwalsh', 'size': '16698'}