<jupyter_start><jupyter_text>IntroductionCongratulations! In this section we will put all of the things that we learned together to do some truly interesting things with some datasets. The exercises in this section are therefore more difficult! While working through the exercises, tTry using method chaning syntax (use the resource below if you don't know what method chaining means). Also, take advantage the hints we provide. Relevant Resource* **[Method chaining resource](https://www.kaggle.com/residentmario/method-chaining-reference). ** Set Up**First, fork this notebook using the "Fork Notebook" button towards the top of the screen.**Run the code cell below to load data and the libraries you'll use.<jupyter_code>import pandas as pd
pd.set_option('max_rows', 5)

import sys
sys.path.append('../input/advanced-pandas-exercises/')
from method_chaining import *

chess_games = pd.read_csv("../input/chess/games.csv")<jupyter_output><empty_output><jupyter_text>Checking AnswersCheck your answers in each of the exercises that follow using the  `check_qN` function provided in the code cell above (replacing `N` with the number of the exercise). For example here's how you would check an incorrect answer to exercise 1:<jupyter_code>check_q1(pd.DataFrame())<jupyter_output><empty_output><jupyter_text>For the first set of questions, if you use `check_qN` on your answer, and your answer is right, a simple `True` value will be returned.For the second set of questions, using this function to check a correct answer will present you will an informative graph!If you get stuck, you may also use the companion `answer_qN` function to print the answer outright. Preview DataRun the cell below to preview the data<jupyter_code>chess_games.head()<jupyter_output><empty_output><jupyter_text>Exercises **Exercise 1**: It's well-known that in the game of chess, white has a slight first-mover advantage against black. Can you measure this effect in this dataset? Use the `winner` column to create a `pandas` `Series` showing how often white wins, how often black wins, and how often the result is a tie, as a ratio of total games played. In other words, a `Series` that looks something like this:    white    0.48    black    0.44    draw     0.08    Name: winner, dtype: float64    Hint: use `len` to get the length of the initial `DataFrame`, e.g. the count of all games played.<jupyter_code>temp = chess_games.winner.value_counts()/len(chess_games)
print (check_q1(temp), '\n\n', temp)<jupyter_output>True 

 white    0.498604
black    0.454033
draw     0.047363
Name: winner, dtype: float64<jupyter_text>**Exercise 2**: The `opening_name` field of the `chess_games` dataset provides interesting data on what the most commonly used chess openings are.  However, it gives a bit _too_ much detail, including information on the variation used for the most common opening types. For example, rather than giving `Queen's Pawn Game`, the dataset often includes `Queen's Pawn Game: Zukertort Variation`.This makes it a bit difficult to use for categorical purposes. Here's a function that can be used to separate out the "opening archetype": ```python lambda n: n.split(":")[0].split("|")[0].split("")[0].strip() ```Use this function to parse the `opening_name` field and generate a `pandas` `Series` counting how many times each of the "opening archetypes" gets used. Hint: use a map.<jupyter_code>temp = chess_games.opening_name.map(lambda n: n.split(":")[0].split("|")[0].split("#")[0].strip()).value_counts()
print (check_q2(temp), '\n\n', temp)<jupyter_output>True 

 Sicilian Defense       2632
French Defense         1412
                       ... 
Valencia Opening          1
Pterodactyl Defense       1
Name: opening_name, Length: 143, dtype: int64<jupyter_text>**Exercise 3**: In this dataset various players play variably number of games. Group the games by `{white_id, victory_status}` and count how many times each white player ended the game in `mate` , `draw`, `resign`, etcetera. The name of the column counting how many times each outcome occurred should be `n` (hint: `rename` or `assign` may help).<jupyter_code>temp = chess_games.assign(n=0).groupby(['white_id', 'victory_status']).n.apply(len).reset_index()
temp<jupyter_output><empty_output><jupyter_text>**Exercise 4**: There are a lot of players in the dataset who have only played one or a small handful of games. Create a `DataFrame` like the one in the previous exercise, but only include users who are in the top 20 users by number of games played. See if you can do this using method chaining alone! Hint: reuse the code from the previous example. Then, use `pipe`.<jupyter_code>#chess_games.white_id.value_counts().sort_index()
#temp['white_id'].value_counts().iloc[:20]
temp = temp.pipe(lambda x: x.loc[x.white_id.isin(chess_games.white_id.value_counts().head(20).index)])
print (check_q4(temp), '\n\n', temp)
chess_games.white_id.value_counts().head(20).index<jupyter_output><empty_output><jupyter_text>Next, let's do some visual exercises.The next exercise uses the following dataset:<jupyter_code>kepler = pd.read_csv("../input/kepler-exoplanet-search-results/cumulative.csv")
kepler<jupyter_output><empty_output><jupyter_text>**Exercise 5**: The Kepler space observatory is in the business of finding potential exoplanets (planets orbiting stars other suns) and, after collecting the evidence, generating whether or not to confirm, decline to confirm, or deny that a given space body is, in fact, an exoplanet. In the dataset above, the "before" status of the body is `koi_pdisposition`, and the "after" status is `koi_disposition`. Using the dataset above, generate a `Series` counting all of the possible transitions between pre-disposition and post-disposition. In other words, generate a `Series` whose index is a `MultiIndex` based on the `{koi_pdisposition, koi_disposition}` fields, and whose values is a count of how many times each possible combination occurred.<jupyter_code>kepler.koi_disposition.unique()
check_q5(kepler.groupby(['koi_pdisposition', 'koi_disposition']).rowid.count())<jupyter_output><empty_output><jupyter_text>The next few exercises use the following datasets:<jupyter_code>wine_reviews = pd.read_csv("../input/wine-reviews/winemag-data-130k-v2.csv", index_col=0)
wine_reviews.head()
ramen_reviews = pd.read_csv("../input/ramen-ratings/ramen-ratings.csv", index_col=0)
ramen_reviews.head()<jupyter_output><empty_output><jupyter_text>**Exercise 6**: As we demonstrated in previous workbooks, the `points` column in the `wine_reviews` dataset is measured on a 20-point scale between 80 and 100. Create a `Series` which normalizes the ratings so that they fit on a 1-to-5 scale instead (e.g. a score of 80 translates to 1 star, while a score of 100 is five stars). Set the `Series` name to "Wine Ratings", and sort by index value (ascending).<jupyter_code>temp2 = wine_reviews.points.map(lambda x: (x-80)/4).value_counts().sort_index().rename_axis("Wine Ratings")
print (check_q6(temp2))
#check_q6(pd.Series(temp2, name='Wine Ratings'))
#wine_reviews.points.sort_values().plot.hist()<jupyter_output>AxesSubplot(0.125,0.125;0.775x0.755)<jupyter_text>**Exercise 7**: The `Stars` column in the `ramen_reviews` dataset is the ramen equivalent to the similar data points in `wine_reviews`. Luckily it is already on a 0-to-5 scale, but it has some different problems...create a `Series` counting how many ramens earned each of the possible scores in the dataset. Convert the `Series` to the `float64` dtype and drop rames whose rating is `"Unrated"`. Set the name of the `Series` to "Ramen Ratings". Sort by index value (ascending).<jupyter_code>check_q7(ramen_reviews.Stars.replace('Unrated', None).dropna().astype('float64').value_counts().sort_index().rename_axis("Ramen Ratings"))
#answer_q7()<jupyter_output><empty_output><jupyter_text>**Exercise 8**: We can see from the result of the previous exercise that whilst the wine reviewers stick to a strict 20-point scale, ramen reviews occassionally deviate into fractional numbers. Modify your answer to the previous exercise by rounding review scores to the nearest half-point (so 0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, or 5).<jupyter_code>round(3.7, 0)
check_q8(ramen_reviews.Stars.replace('Unrated', None).dropna().astype('float64').map(lambda x: int(x) if x - int(x) < 0.5 else int(x) + 0.5).value_counts().sort_index().rename_axis("Ramen Reviews"))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10288017067678085, 'fraction_numerical': 0.01896408676069693, 'mean_word_length': 5.505782575173478, 'pattern_counts': {'":': 3, '<': 51, '<?xml version=': 0, '>': 50, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '511684', 'license': 'MIT', 'n_tokens_mistral': 2708, 'n_tokens_neox': 2553, 'n_words': 970, 'path': 'Tutorials/Pandas/Method Chaining.ipynb', 'repo_name': 'anshulrai/Kaggle-Challenges', 'size': '8437'}