<jupyter_start><jupyter_text>Iterating with .iterrows()In the video, we discussed that `.iterrows()` returns each DataFrame row as a tuple of (index, `pandas` Series) pairs. But, what does this mean? Let's explore with a few coding exercises.A `pandas` DataFrame has been loaded into your session called `pit_df`. This DataFrame contains the stats for the Major League Baseball team named the Pittsburgh Pirates (abbreviated as `'PIT'`) from the year 2008 to the year 2012. It has been printed into your console for convenience.Instructions1. Use `.iterrows()` to loop over `pit_df` and print each row. Save the first item from `.iterrows()` as `i` and the second as `row`.2. Add **two** lines to the loop: one _before_ `print(row)` to print each index variable and one _after_ to print each row's type.3. Instead of using `i` and `row` in the for statement to store the output of `.iterrows()`, use **one** variable named `row_tuple`.4. Add a line in the for loop to print the type of each `row_tuple`.<jupyter_code># Import pandas
import pandas as pd

# Import dataset
pit_df = pd.read_csv('pit_df.csv')
# Iterate over pit_df and print each row
for i, row in pit_df.iterrows():
    print(row)
# Iterate over pit_df and print each index variable and then each row
for i, row in pit_df.iterrows():
    print(i)
    print(row)
    print(type(row))
# Use one variable instead of two to store the result of .iterrows()
for row_tuple in pit_df.iterrows():
    print(row_tuple)
# Print the row and type of each row
for row_tuple in pit_df.iterrows():
    print(row_tuple)
    print(type(row_tuple))<jupyter_output>(0, Unnamed: 0     PIT
Team            NL
League        2012
Year           651
RS             674
RA              79
W              162
G                0
Playoffs       NaN
Name: 0, dtype: object)
<class 'tuple'>
(1, Unnamed: 0     PIT
Team            NL
League        2011
Year           610
RS             712
RA              72
W              162
G                0
Playoffs       NaN
Name: 1, dtype: object)
<class 'tuple'>
(2, Unnamed: 0     PIT
Team            NL
League        2010
Year           587
RS             866
RA              57
W              162
G                0
Playoffs       NaN
Name: 2, dtype: object)
<class 'tuple'>
(3, Unnamed: 0     PIT
Team            NL
League        2009
Year           636
RS             768
RA              62
W              161
G                0
Playoffs       NaN
Name: 3, dtype: object)
<class 'tuple'>
(4, Unnamed: 0     PIT
Team            NL
League        2008
Year           735
RS             884
RA              67
W              162
G  [...]<jupyter_text>Run differentials with .iterrows()You've been hired by the San Francisco Giants as an analyst — congrats! The team's owner wants you to calculate a metric called the _run differential_ for each season from the year 2008 to 2012. This metric is calculated by subtracting the total number of runs a team allowed in a season from the team's total number of runs scored in a season. `'RS'` means runs scored and `'RA'` means runs allowed.The below function calculates this metric:```def calc_run_diff(runs_scored, runs_allowed):    run_diff = runs_scored - runs_allowed    return run_diff```A DataFrame has been loaded into your session as `giants_df` and printed into the console. Let's practice using `.iterrows()` to add a _run differential_ column to this DataFrame.Instructions1. Create an empty list called `run_diffs` that will be used to store the _run differentials_ you will calculate.2. Write a for loop that uses `.iterrows()` to loop over `giants_df` and collects each row's runs scored and runs allowed.3. Add a line to the for loop that uses the provided function to calculate each row's _run differential_.4. Add a line to the loop that appends each row's _run differential_ to the `run_diffs` list.<jupyter_code># Import dataset
giants_df = pd.read_csv('giants_df.csv')

# Define function
def calc_run_diff(runs_scored, runs_allowed):
    run_diff = runs_scored - runs_allowed
    return run_diffs
# Create an empty list to store run differentials
run_diffs = []

# Write a for loop and collect runs allowed and runs scored for each row
for i, row in giants_df.iterrows():
    runs_scored = row['RS']
    runs_allowed = row['RA']
    
    # Use the provided function to calculate run_diff for each row
    # run_diff = calc_run_diff(runs_scored, runs_allowed)    
    run_diff = row['RS'] - row['RA']
    
    # Append each run differential to the output list
    run_diffs.append(run_diff)

giants_df['RD'] = run_diffs
giants_df<jupyter_output><empty_output><jupyter_text>Iterating with .itertuples()Remember, `.itertuples()` returns each DataFrame row as a special data type called a **namedtuple**. You can look up an attribute within a namedtuple with a special syntax. Let's practice working with namedtuples.A pandas DataFrame has been loaded into your session called `rangers_df`. This DataFrame contains the stats (`'Team'`, `'League'`, `'Year'`, `'RS'`, `'RA'`, `'W'`, `'G'`, and `'Playoffs'`) for the Major League baseball team named the Texas Rangers (abbreviated as `'TEX'`).Instructions1. Use `.itertuples()` to loop over `rangers_df` and print each row.2. Loop over `rangers_df` with `.itertuples()` and save each row's `Index`, `Year`, and Wins (`W`) attribute as `i`, `year`, and `wins`.3. Now, loop over `rangers_df` and print these values **only for those rows** where the Rangers made the playoffs.<jupyter_code># Import dataset
rangers_df = pd.read_csv('rangers_df.csv')
# Loop over the DataFrame and print each row
for row in rangers_df.itertuples():
    print(row)
# Loop over the DataFrame and print each row's Index, Year and Wins (W)
for row in rangers_df.itertuples():
    i = row.Index
    year = row.Year
    wins = row.W
    print(i, year, wins)
# Loop over the DataFrame and print each row's Index, Year and Wins (W)
for row in rangers_df.itertuples():
    i = row.Index
    year = row.Year
    wins = row.W
  
    # Check if rangers made Playoffs (1 means yes; 0 means no)
    if row.Playoffs == 1:
        print(i, year, wins)<jupyter_output>0 2012 93
1 2011 96
2 2010 90
13 1999 95
14 1998 88
16 1996 90<jupyter_text>Run differentials with .itertuples()The New York Yankees have made a trade with the San Francisco Giants for your analyst contract— you're a hot commodity! Your new boss has seen your work with the Giants and now wants you to do something similar with the Yankees data. He'd like you to calculate `run differentials` for the Yankees from the year 1962 to the year 2012 and find which season they had the best `run differential`.You've remembered the function you used when working with the Giants and quickly write it down:```def calc_run_diff(runs_scored, runs_allowed):    run_diff = runs_scored - runs_allowed    return run_diff```Let's use `.itertuples()` to loop over the `yankees_df` DataFrame (which has been loaded into your session) and calculate _run differentials_.Instructions1. Use `.itertuples()` to loop over `yankees_df` and grab each row's runs scored and runs allowed values.2. Now, calculate each row's _run differential_ using `calc_run_diff()`. Be sure to append each row's _run differential_ to `run_diffs`.3. Append a new column called `'RD'` to the `yankees_df` DataFrame that contains the _run differentials_ you calculated.4. In what year within your DataFrame did the New York Yankees have the highest _run differential_? You'll need to rerun the code that creates the `'RD'` column if you'd like to analyze the DataFrame with code rather than looking at the console output.<jupyter_code># Import dataset
yankees_df = pd.read_csv('yankees_df.csv')

# Define function
def calc_run_diff(runs_scored, runs_allowed):
    run_diff = runs_scored - runs_allowed
    return run_diff
run_diffs = []

# Loop over the DataFrame and calculate each row's run differential
for row in yankees_df.itertuples():
    runs_scored = row.RS
    runs_allowed = row.RA

    run_diff = calc_run_diff(runs_scored, runs_allowed)
    run_diffs.append(run_diff)
    
# Append new column
yankees_df['RD'] = run_diffs
yankees_df<jupyter_output><empty_output><jupyter_text>Analyzing baseball stats with .apply()The Tampa Bay Rays want you to analyze their data.They'd like the following metrics:- The sum of each column in the data- The total amount of runs scored in a year ('`RS'` + `'RA'` for each year)- The `'Playoffs'` column in text format rather than using `1`'s and `0`'sThe below function can be used to convert the `'Playoffs'` column to text:```def text_playoffs(num_playoffs):     if num_playoffs == 1:        return 'Yes'    else:        return 'No' ```Use `.apply()` to get these metrics. A DataFrame (`rays_df`) has been loaded and printed to the console. This DataFrame is indexed on the `'Year'` column.Instructions1. Apply `sum()` to each **column** of `rays_df` to collect the sum of each column. Be sure to specify the correct `axis`.2. Apply `sum()` to each **row** of `rays_df`, only looking at the `'RS'` and `'RA'` columns, and specify the correct `axis`.3. Use `.apply()` and a `lambda` function to apply `text_playoffs()` to each **row**'s `'Playoffs'` value of the `rays_df` DataFrame.<jupyter_code># Import dataset
rays_df = pd.read_csv('rays_df.csv')

# Define function
def text_playoffs(num_playoffs): 
    if num_playoffs == 1:
        return 'Yes'
    else:
        return 'No' 
# Gather sum of all columns
stat_totals = rays_df.apply(sum, axis=0)
print(stat_totals)
# Gather total runs scored in all games per year
total_runs_scored = rays_df[['RS', 'RA']].apply(sum, axis=1)
print(total_runs_scored)
# Convert numeric playoffs to text
textual_playoffs = rays_df.apply(lambda row: text_playoffs(row['Playoffs']), axis=1)
print(textual_playoffs)<jupyter_output>0     No
1    Yes
2    Yes
3     No
4    Yes
dtype: object<jupyter_text>Settle a debate with .apply()Word has gotten to the Arizona Diamondbacks about your awesome analytics skills. They'd like for you to help settle a debate amongst the managers. One manager claims that the team has made the playoffs every year they have had a win percentage of `0.50` or greater. Another manager says this is not true.Let's use the below function and the `.apply()` method to see which manager is correct.```def calc_win_perc(wins, games_played):    win_perc = wins / games_played    return np.round(win_perc, 2)```A DataFrame named `dbacks_df` has been loaded.Instructions1. Print the first five rows of the `dbacks_df` DataFrame to see what the data looks like.2. Create a `pandas` Series called `win_percs` by applying the `calc_win_perc()` function to each **row** of the DataFrame with a `lambda` function.3. Create a new column in `dbacks_df` called `WP` that contains the win percentages you calculated in the above step.4. Which manager was correct in their claim?<jupyter_code># Import dataset
dbacks_df = pd.read_csv('dbacks_df.csv')

# Define function
def calc_win_perc(wins, games_played):
    win_perc = wins / games_played
    return np.round(win_perc, 2)
# Display the first five rows of the DataFrame
print(dbacks_df.head(), '\n')

# Create a win percentage Series 
win_percs = dbacks_df.apply(lambda row: calc_win_perc(row['W'], row['G']), axis=1)
print(win_percs, '\n')

# Append a new column to dbacks_df
dbacks_df['WP'] = win_percs
print(dbacks_df, '\n')

# Display dbacks_df where WP is greater than 0.50
print(dbacks_df[dbacks_df['WP'] >= 0.50])<jupyter_output>Team League  Year   RS   RA   W    G  Playoffs
0  ARI     NL  2012  734  688  81  162         0
1  ARI     NL  2011  731  662  94  162         1
2  ARI     NL  2010  713  836  65  162         0
3  ARI     NL  2009  720  782  70  162         0
4  ARI     NL  2008  720  706  82  162         0 

0     0.50
1     0.58
2     0.40
3     0.43
4     0.51
5     0.56
6     0.47
7     0.48
8     0.31
9     0.52
10    0.60
11    0.57
12    0.52
13    0.62
14    0.40
dtype: float64 

   Team League  Year   RS   RA    W    G  Playoffs    WP
0   ARI     NL  2012  734  688   81  162         0  0.50
1   ARI     NL  2011  731  662   94  162         1  0.58
2   ARI     NL  2010  713  836   65  162         0  0.40
3   ARI     NL  2009  720  782   70  162         0  0.43
4   ARI     NL  2008  720  706   82  162         0  0.51
5   ARI     NL  2007  712  732   90  162         1  0.56
6   ARI     NL  2006  773  788   76  162         0  0.47
7   ARI     NL  2005  696  856   77  162         0  0.48
8   ARI  [...]<jupyter_text>The manager who claimed the team **has not made** the playoffs every year they've had a win percentage of `0.50` or greater.  Replacing .iloc with underlying arraysNow that you have a better grasp on a DataFrame's internals let's update one of your previous analyses to leverage a DataFrame's underlying arrays. You'll revisit the win percentage calculations you performed row by row with the `.iloc` method:```def calc_win_perc(wins, games_played):    win_perc = wins / games_played    return np.round(win_perc,2)win_percs_list = []for i in range(len(baseball_df)):    row = baseball_df.iloc[i]    wins = row['W']    games_played = row['G']    win_perc = calc_win_perc(wins, games_played)    win_percs_list.append(win_perc)baseball_df['WP'] = win_percs_list```Let's update this analysis to use arrays instead of the `.iloc` method. A DataFrame (`baseball_df`) has been loaded into your session.Instructions1. Use _the right method_ to collect the underlying `'W'` and `'G'` arrays of `baseball_df` and pass them **directly into** the `calc_win_perc()` function. Store the result as a variable called `win_percs_np`.2. Create a new column in `baseball_df` called `'WP'` that contains the win percentages you just calculated.3. Use `timeit` in cell magic mode within your IPython console to compare the runtimes between the old code block using `.iloc` and the new code you developed using NumPy arrays. **Don't include the code that defines the `calc_win_perc()` function or the `print()` statements or when timing**. You should include **eight lines of code** when timing the old code block and **two lines** of code when timing the new code you developed. You may need to press `ENTER` when using `timeit` in cell magic mode to get to a new line within your IPython console. Which approach was the faster?<jupyter_code># Import dataset
baseball_df = pd.read_csv('baseball_df.csv')
# Use the W array and G array to calculate win percentages
win_percs_np = calc_win_perc(baseball_df['W'].values, baseball_df['G'].values)

# Append a new column to baseball_df that stores all win percentages
baseball_df['WP'] = win_percs_np
baseball_df.head()<jupyter_output><empty_output><jupyter_text>Bringing it all together: Predict win percentageA `pandas` DataFrame (`baseball_df`) has been loaded into your session. For convenience, a dictionary describing each column within `baseball_df` has been printed. You can reference these descriptions throughout the exercise.```OrderedDict([('Team', 'Abbreviated team name'),             ('League', 'Specifies National League or American League'),             ('Year', "Each season's year"),             ('RS', 'Runs scored in a season'),             ('RA', 'Runs allowed in a season'),             ('W', 'Wins in a season'),             ('G', 'Games played in a season'),             ('Playoffs', '`1` if a team made the playoffs; `0` if they did not'),             ('WP', 'True win percentage for a season')])```You'd like to attempt to _predict_ a team's win percentage for a given season by using the team's total runs scored in a season (`'RS'`) and total runs allowed in a season (`'RA'`) with the following function:```def predict_win_perc(RS, RA):    prediction = RS ** 2 / (RS ** 2 + RA ** 2)    return np.round(prediction, 2)```Let's compare the approaches you've learned to calculate a _predicted win percentage_ for each season (or row) in your DataFrame.Instructions1. Use a for loop and `.itertuples()` to predict the win percentage for each row of `baseball_df` with the `predict_win_perc()` function. Save each row's predicted win percentage as `win_perc_pred` and append each to the `win_perc_preds_loop` list.2. Apply `predict_win_perc()` to each row of the `baseball_df` DataFrame using a `lambda` function. Save the predicted win percentage as `win_perc_preds_apply`.3. Calculate the predicted win percentages by passing the underlying `'RS'` and `'RA'` **arrays** from `baseball_df` into `predict_win_perc()`. Save these predictions as `win_perc_preds_np`.4. Compare runtimes within your IPython console between **all three** approaches used to calculate the predicted win percentages. Use **`%%timeit`** (_cell magic mode_) to time the **six lines of code** (not including comment lines) for the `.itertuples()` approach. You may need to press `ENTER` after entering `%%timeit` to get to a new line within your IPython console. Use **`%timeit`** (_line magic mode_) to time the `.apply()` approach and the NumPy array approach separately. Each has only **one line of code** (not including comment lines). What is the order of approaches from fastest to slowest?<jupyter_code># Define function
def predict_win_perc(RS, RA):
    prediction = RS ** 2 / (RS ** 2 + RA ** 2)
    return np.round(prediction, 2)
win_perc_preds_loop = []

# Use a loop and .itertuples() to collect each row's predicted win percentage
for row in baseball_df.itertuples():
    runs_scored = row.RS
    runs_allowed = row.RA
    win_perc_pred = predict_win_perc(runs_scored, runs_allowed)
    win_perc_preds_loop.append(win_perc_pred)

# Apply predict_win_perc to each row of the DataFrame
win_perc_preds_apply = baseball_df.apply(lambda row: predict_win_perc(row['RS'], row['RA']), axis=1)

# Calculate the win percentage predictions using NumPy arrays
win_perc_preds_np = predict_win_perc(baseball_df['RS'].values, baseball_df['RA'].values)
baseball_df['WP_preds'] = win_perc_preds_np
baseball_df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08666555127990631, 'fraction_numerical': 0.03045005855780492, 'mean_word_length': 3.587362496802251, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '315615', 'license': 'MIT', 'n_tokens_mistral': 5921, 'n_tokens_neox': 5449, 'n_words': 2359, 'path': 'writing_efficient_python_code/4_basic_pandas_optimizations.ipynb', 'repo_name': 'vhsenna/datacamp-courses', 'size': '17931'}