<jupyter_start><jupyter_text>Automate your analysis with `pandas`Automating your data analysis is one of the most powerful things you can do with Python in a newsroom. We're going to use a library called [`pandas`](http://pandas.pydata.org/) that will leave a replicable, transparent script for others to follow. Warmup: MLB salary dataRemember `data/mlb.csv`? Let's use that to practice.  Make a list of questions- What's the total MLB payroll?- What's the total payroll by team?- What's the typical (mean) salary for a designated hitter?(What else?)  Import pandas<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Read in the data into a data frameWe'll use the [`read_csv()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) method to read the CSV file in as a [`DataFrame`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html).<jupyter_code>df = pd.read_csv('data/mlb.csv')

# use head to check it out
df.head()<jupyter_output><empty_output><jupyter_text>_How many players?_Use the `.count()` method to find out how many pieces of data are in each column.<jupyter_code>df.count()<jupyter_output><empty_output><jupyter_text>_Total MLB payroll_To select a column from your data frame, you can use either dot or bracket notation. The result will be a [`Series`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html).Then, you can use the [`sum()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.sum.html) method to sum numeric data.<jupyter_code>df.SALARY.sum()<jupyter_output><empty_output><jupyter_text>_Get a list of teams_Sometimes you want to get a list of unique values in a column. You can use the [`unique()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.unique.html) method for this.<jupyter_code>df.TEAM.unique()<jupyter_output><empty_output><jupyter_text>_Total payroll by team_We want to group our data by team and sum the salaries -- analagous to a pivot table in Excel or a `GROUP BY` statement in SQL. Our steps:- Select the two columns we care about -- to select multiple columns, pass a list of columns to the data frame inside square brackets- Use the [`groupby()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html) function to group the data by team- Use the [`sum()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.core.groupby.GroupBy.sum.html) method to sum up the grouped data- Use the [`reset_index()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.reset_index.html) method to turn the Series back into a DataFrame- Index the results on the `TEAM` column (optional)- Use the [`sort_values()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sort_values.html) method to order the data by the `SALARY` column<jupyter_code>df[['TEAM', 'SALARY']].groupby('TEAM') \
                      .sum() \
                      .reset_index() \
                      .set_index('TEAM') \
                      .sort_values('SALARY', ascending=False)<jupyter_output><empty_output><jupyter_text>_Typical salary for DH_Let's find the median salary for designated hitters. To filter a data frame, you use square brackets to pass a filtering condition to the data frame: `df[YOUR CONDITION HERE]`.It's like a WHERE clause in SQL. In this case, we can use one of Python's [comparison operators](https://docs.python.org/3/reference/expressions.htmlcomparisons) to define our conditions. If the value in the `POS` column is `DH`, include it in the results.Then we're going to select the values in the `SALARY` column and calculate the typical salary with the [`mean()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.mean.html) function.<jupyter_code># filter data for designated hitters
dh = df[df['POS'] == 'DH']

# get median
dh.SALARY.median()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11468354430379747, 'fraction_numerical': 0.00025316455696202533, 'mean_word_length': 4.810294117647059, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '654864', 'license': 'MIT', 'n_tokens_mistral': 1284, 'n_tokens_neox': 1228, 'n_words': 392, 'path': 'completed/05. pandas? pandas! (Part 1).ipynb', 'repo_name': 'cjwinchester/cfj-2017', 'size': '3950'}