<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Pokémon Overview2&nbsp;&nbsp;Pokémon Data2.1&nbsp;&nbsp;Indexes and Uniqueness2.2&nbsp;&nbsp;Selecting Specific Columns3&nbsp;&nbsp;Selecting Rows3.1&nbsp;&nbsp;Exact Text Matches3.2&nbsp;&nbsp;Partial Text Matching4&nbsp;&nbsp;Numerical Conditions5&nbsp;&nbsp;Grouping  and Aggregation5.1&nbsp;&nbsp;Counting in Groups5.2&nbsp;&nbsp;Selecting Record with a MultiIndex5.3&nbsp;&nbsp;Select a Record by Query5.4&nbsp;&nbsp;Renaming columns6&nbsp;&nbsp;Basic Plotting7&nbsp;&nbsp;Correlations7.1&nbsp;&nbsp;Resetting an Index7.2&nbsp;&nbsp;Correlation Heatmap7.2.1&nbsp;&nbsp;get_duplicate_indexes8&nbsp;&nbsp;Download Additional Pokemon Data9&nbsp;&nbsp;Types10&nbsp;&nbsp;Pokemon Types10.0.1&nbsp;&nbsp;Duplicated Pokemon Types11&nbsp;&nbsp;Joining DataFrames12&nbsp;&nbsp;Grouping and Plotting13&nbsp;&nbsp;Correlations13.1&nbsp;&nbsp;Abilities13.2&nbsp;&nbsp;Pokemon to Abilities Mapping14&nbsp;&nbsp;Categorical Data  Pokémon Overview> The Pokémon franchise revolves around 807 fictional species of collectible monsters, each having unique designs and skills. Conceived by <NAME> in early 1989, Pokémon are creatures that inhabit the fictional Pokémon World. The designs for the multitude of species can draw inspiration from anything, such as inanimate objects, real-world animals, or mythology. Many Pokémon are capable of evolving into more powerful species, while others can undergo form changes and achieve similar results. Originally, only a handful of artists led by <NAME> designed Pokémon. However, by 2013 a team of 20 artists worked together to create new species designs. Sugimori and <NAME> lead the team and determine the final designs. Each iteration of the series has brought about praise and criticism over the numerous creatures - https://en.wikipedia.org/wiki/List_of_Pok%C3%A9mon----> Nintendo/Creatures Inc./GAME FREAK inc. Pokémon, Pokémon character names, Nintendo 3DS, Nintendo DS, Wii, Wii U, and Wiiware are trademarks of Nintendo - https://www.pokemon.com/us/legal/This work is in no way affiliated with the fine people above. It is intended for educational purposes under fair use. Image Credit: https://veekun.com/dex/pokemon/bulbasaur<jupyter_code>import IPython
import requests
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression<jupyter_output><empty_output><jupyter_text>Pokémon DataDownload the files from the interwebs and save it to my hard drive. Found a Github repo that has pokemon data as handy CSVs: https://github.com/veekun/pokedex<jupyter_code>def download_file(url, destination, use_ssl = True):    
    r = requests.get(url, allow_redirects=True, verify = use_ssl)
    open(destination, "wb").write(r.content)
    return
download_file("https://github.com/veekun/pokedex/raw/master/pokedex/data/csv/pokemon.csv", 
              "pokemon.csv")<jupyter_output><empty_output><jupyter_text>Load our downloaded CSV into a Pandas DataFrame<jupyter_code>df_pokemon = pd.read_csv("pokemon.csv")<jupyter_output><empty_output><jupyter_text>So what columns do we have?<jupyter_code>df_pokemon.columns<jupyter_output><empty_output><jupyter_text>**Preview of Pokemon Data*** **id** - The unique id for the Pokemon* **identifier** - The name of the Pokemon* **species_id** - ???* **height** - The height of the Pokemon* **weight** - The weight of the Pokemon* **base_experience** - Experience required for the Pokemon to level up* **order** - ???* **is_default** - ???<jupyter_code>df_pokemon.head()<jupyter_output><empty_output><jupyter_text>How many Pokemon in total?<jupyter_code>len(df_pokemon)<jupyter_output><empty_output><jupyter_text>Indexes and UniquenessSo what's the very first column? It is the Index an automatically generated unique identifier for each row.<jupyter_code>df_pokemon.index<jupyter_output><empty_output><jupyter_text>Our dataset has an `id` column. Which if it is unique we could use as our unique index.So is the `id` column actually unique?<jupyter_code>df_pokemon["id"].is_unique<jupyter_output><empty_output><jupyter_text>Is the identifier column unique?<jupyter_code>df_pokemon["identifier"].is_unique<jupyter_output><empty_output><jupyter_text>So we could use it as the Index to uniquely identify each row.<jupyter_code>df_pokemon = df_pokemon.set_index("id")
df_pokemon.head()<jupyter_output><empty_output><jupyter_text>Selecting Specific ColumnsIf we only want the names:<jupyter_code>df_pokemon["identifier"]<jupyter_output><empty_output><jupyter_text>The above returns the column on it's own, however if we want multiple specific columns, as them as a list:<jupyter_code>df_pokemon[["identifier", "base_experience"]]<jupyter_output><empty_output><jupyter_text>Selecting Rows  Exact Text MatchesLets find the Pokemon called **electrode**.<jupyter_code>df_pokemon["identifier"] == "electrode"<jupyter_output><empty_output><jupyter_text>What we have above is an Indexer, which states for each Rows Index if the condition was met.We can use the Index to get the specific rows we want.<jupyter_code># get me electrode
is_electrode_index = df_pokemon["identifier"] == "electrode"
df_pokemon[is_electrode_index]<jupyter_output><empty_output><jupyter_text>Because we set the `id` to be the Index we can also lookup by Index value<jupyter_code># get the pokemon with the Index value of 101
df_pokemon.loc[101]<jupyter_output><empty_output><jupyter_text>We can also lookup by position in the DataFrame, which as position start counting from 0, we don't get **electrode** when at what is at position 101 we get **exeggcute**.<jupyter_code># get me the pokemon that's 10th in the dataframe
df_pokemon.iloc[101] # because 0 based index
df_pokemon[df_pokemon["identifier"] == "exeggcute"]<jupyter_output><empty_output><jupyter_text>Partial Text MatchingFor a series (column) get the values as string and then apply  string `contains` operation to determine if a specific value in the series matches `"pika"`.Also lets make a seperate DataFrame that just has Pikachus.<jupyter_code># find me the pikachus
pika_index = df_pokemon["identifier"].astype(str).str.contains("pika")
df_pikachus = df_pokemon[pika_index]
df_pikachus<jupyter_output><empty_output><jupyter_text>How many Pikachus do we have?<jupyter_code>len(df_pikachus)<jupyter_output><empty_output><jupyter_text>That's more than I expected...image credit: https://mainichi.jp/english/graphs/20170809/hpe/00m/0et/001000g/8  Numerical ConditionsLet's get some basic summary statistics for the height of our pokemon.<jupyter_code>df_pokemon["height"].mean()
df_pokemon["height"].std()
df_pokemon["height"].median()<jupyter_output><empty_output><jupyter_text>Or a single call to to get summary statistics.<jupyter_code>df_pokemon["height"].describe()<jupyter_output><empty_output><jupyter_text>Lets get the tallest Pokemon in the highest quartile.<jupyter_code>tallest_quartile_height = df_pokemon["height"].describe()["75%"]
print("tallest_quartile_height", tallest_quartile_height)
tallest_quartile_height_index = df_pokemon["height"] >= tallest_quartile_height<jupyter_output><empty_output><jupyter_text>How many Pokemon are in the tallest quartile<jupyter_code>len(df_pokemon[tallest_quartile_height_index])<jupyter_output><empty_output><jupyter_text>How many are not in the tallest quartile?In order to select rows that do **not** match an indexer use the '~' aka tilde character.<jupyter_code>len(df_pokemon[~tallest_quartile_height_index])<jupyter_output><empty_output><jupyter_text>Grouping  and AggregationLet's apply some grouping by height and weight<jupyter_code>df_pokemon.groupby(["height", "weight"])<jupyter_output><empty_output><jupyter_text>Counting in GroupsHowever we need to apply an operation to the Grouping in order for it be useful. So let's count.The call to the count is wrapped in a DataFrame so it displays in a formatted table.<jupyter_code>df_pokemon_grouped_height_weight = pd.DataFrame(
    df_pokemon.groupby(["height", "weight"])["identifier"].count()
)

# lets look at the first ten 10 rows
df_pokemon_grouped_height_weight.head(10)<jupyter_output><empty_output><jupyter_text>This DataFrame is a little different to the ones we've seen thus far. As it has two indexes, the columns we grouped by:<jupyter_code>df_pokemon_grouped_height_weight.index.names<jupyter_output><empty_output><jupyter_text>Selecting Record with a MultiIndex<jupyter_code>df_pokemon_grouped_height_weight.loc[(1,9999)]<jupyter_output><empty_output><jupyter_text>Surely this must be an exceptional Pokemon in terms of weight?<jupyter_code>df_pokemon["weight"].describe()<jupyter_output><empty_output><jupyter_text>Seems like it is the heaviest (or one of the heaviest) Pokemon, so which Pokemon is it?  Select a Record by QueryThat 1 pokemon with a height of 1 and a weight of 9999 is curious, what is it?We could look it up using multiple indexes or we could write a query.<jupyter_code>df_pokemon.query("height == 1 and weight == 9999")<jupyter_output><empty_output><jupyter_text>So which Pokemon happen to be...image credit: http://knowyourmeme.com/memes/its-over-9000<jupyter_code>df_pokemon.query("weight >= 9000")<jupyter_output><empty_output><jupyter_text>Renaming columnsThe counted identifiers doesn't have a descriptive column name, so lets rename it. What columns do we have in our grouped DataFrame?<jupyter_code>df_pokemon_grouped_height_weight.columns<jupyter_output><empty_output><jupyter_text>Let's rename that column by passing a list with the new column name.If we had multiple columns we'd need to pass all of the column names in.<jupyter_code>df_pokemon_grouped_height_weight.columns = ["pokemon_count"]
df_pokemon_grouped_height_weight.head(10)<jupyter_output><empty_output><jupyter_text>Basic Plotting A pairplot is a simple way to visualise the distrubtions and pairwise relationships between variables.<jupyter_code>plt.ion()
sns.pairplot(df_pokemon)<jupyter_output><empty_output><jupyter_text>We've got interactions that don't make for sensible plots.* `is_default` is a binary value (1 or 0 aka True or False), a scatter plot doesn't make sense.* `species_id` while numeric, seems to be an ordinal value (https://en.wikipedia.org/wiki/Ordinal_data), the "distance" between two values is not readily understandable or possibly even comparable.So let's only select specific columns<jupyter_code>sns.pairplot(df_pokemon[["height", "weight", "base_experience"]])<jupyter_output><empty_output><jupyter_text>Looking at the distributions (bar chat looking subplots aka histograms), the distribution of values tend to congregate in clumps.However for base_experience we've two peaks? Do we perhaps 2 subgroups in our dataset?<jupyter_code>def displot(df, column, ax, column2 = None):
    if column2 is not None:
        result = sns.distplot(df[column], df[column2], ax = ax)
    else:
        result = sns.distplot(df[column], ax = ax)
        
    return result
# I want to control the layout of the plots
plt.ioff()
fig, (ax1, ax2, ax3) = plt.subplots(ncols=1, nrows=3, figsize = (15, 11), sharex=False)
displot(df_pokemon, "height", ax1)
displot(df_pokemon, "weight", ax2)
displot(df_pokemon, "base_experience", ax3)
fig.subplots_adjust(hspace = 0.5)<jupyter_output><empty_output><jupyter_text>CorrelationsAs you may recall we have a number of Pikachus<jupyter_code>df_pikachus<jupyter_output><empty_output><jupyter_text>It seems that there is a relationship between `species_id` and the `id`. All of the Pikachus have a `species_id` of 25, which is the `id` of the first pikachu. Let's see what correlations we have in our dataset. Particularly those correlations with the `id`.The output of `df_pokemon.corr()` will generate a DataFrame of *correlations coefficients* measured from -1.0 to +1.0, indicating the pairwsie relationship between each numerical variable in the DataFrame of pokemons.*Correlations coefficients* are the measure of the strength and direction of a linear relationship between variables.* If the value is positive this denotes a positive relationship between the variables, i.e. increasing values of variable a are correlated with increasing values of variable b.* If the correlation coefficient is closer to 1, it indicates that changes are positively related:    * As variable a increases in value, variable b tends to increase in value.    * As variable a decreases in value, variable b tends to decrease in value.* If the correlation coefficient is closer to 0, it indicates that there isn't a relationship:    * Changes in variable a don't seem to be related to increasing or decreasing values of b.* If the correlation coefficient is closer to -1, it indicates that changes are negatively related:    * As variable a increases in value, variable b tends to decrease in value.    * As variable a decreases in value, variable b tends to increase in value.<jupyter_code>df_pokemon.corr()<jupyter_output><empty_output><jupyter_text>But `id` isn't listed above, as we moved `id` to the index - it's not available as a column.<jupyter_code>df_pokemon.columns<jupyter_output><empty_output><jupyter_text>Resetting an IndexSo let's reset the index back to an automatic incrementing number so we can use `id` as a column again.<jupyter_code>df_pokemon = df_pokemon.reset_index()
df_pokemon.head()
df_pokemon.corr()<jupyter_output><empty_output><jupyter_text>Correlation HeatmapThis is would be much more effective as a correlation heatmap.<jupyter_code>def plot_correlation(df, title, axis, figure_size = (11, 9)):
    global plt
    # Generate a mask for the upper triangle
    mask = np.zeros_like(df.corr(), dtype=np.bool)
    mask[np.triu_indices_from(mask)] = True
        
    # Generate a custom diverging colormap
    cmap = sns.diverging_palette(220, 10, as_cmap=True)

    # Draw the heatmap with the mask and correct aspect ratio
    result = sns.heatmap(df.corr(), mask=mask, vmin = -1, vmax= 1, cmap=cmap, center=0,
                square=True, linewidths=.5, cbar_kws={"shrink": .5}, ax = axis)
    
    result.set_title(title)
    
    return result
plt.ioff()
fig, ax1 = plt.subplots(figsize = (11, 9))
plot_correlation(df_pokemon, "All Pokemon Correlation", figure_size = (11, 9), axis = ax1)<jupyter_output><empty_output><jupyter_text>There is a strong negative correlation with `is_default` and `id`. Let's look into that.We know that `id` is a unique value, so lets split the data by the values of `is_default`.<jupyter_code>df_pokemon["is_default"].unique()<jupyter_output><empty_output><jupyter_text>Lets also look at id where varying values of `is_default`<jupyter_code>df_pokemon.query("is_default == 0")["id"].describe()
df_pokemon.query("is_default == 1")["id"].describe()
def display_figure(fig):
    IPython.display.display(fig)<jupyter_output><empty_output><jupyter_text>You'll notice below that there aren't any correlations for `is_default` as there aren't any changes for `is_default` in our two subsets.<jupyter_code>fig, (ax1, ax2) = plt.subplots(ncols = 2, nrows = 1, figsize = (15, 9), sharey = False)
plot_correlation(df_pokemon[df_pokemon["is_default"] == 0], "is_default == 0", ax1)
plot_correlation(df_pokemon[df_pokemon["is_default"] == 1], "is_default == 1", ax2)
fig.subplots_adjust(wspace = 0.5)<jupyter_output><empty_output><jupyter_text>Where `is_default == 0`* `height` and `id` - very weak negative correlation * `weight` and `id` - very weak negative correlation * `base_experience` and `id` - weak negative correlationWhere `is_default == 1`* `height` and `id` - very weak positive correlation * `weight` and `id` - weak positive correlation* `base_experience` and `id` - weak positive correlationLet's plot these relationships in some pair plots.<jupyter_code>def pair_plot_correlations(df, columns):
    return sns.pairplot(df[columns])
# some oddities around clearing the plot when switching from interactive
# mode off to interactive mode on. Need to look into why
plt.clf()
plt.cla()
plt.close()
plt.clf()
plt.ion()
pair_plot_correlations(df_pokemon[df_pokemon["is_default"] == 1], ["id", "height", "weight", "base_experience"])
pair_plot_correlations(df_pokemon[df_pokemon["is_default"] == 0], ["id", "height", "weight", "base_experience"])<jupyter_output><empty_output><jupyter_text>So are the variants (where the `id ` doesn't match the `species_id`) only different in name or is there some other variation?<jupyter_code>variant_pokemon_index = df_pokemon["id"] != df_pokemon["species_id"]
is_default_index = df_pokemon["is_default"] == 1<jupyter_output><empty_output><jupyter_text>get_duplicate_indexesFor a given DataFrame `df` find which rows have duplicated values for the `columns` (an array/list), keeping the first match.<jupyter_code>def get_duplicate_indexes(df, columns):
    return df.duplicated(columns, keep = "first")
df_duplicated_pokemon = df_pokemon[
    get_duplicate_indexes(df_pokemon, ["species_id", "height", "weight", "base_experience"])
]
df_duplicated_pokemon<jupyter_output><empty_output><jupyter_text>let's omit the rows in `df_pokemon` where the `id` appears in `df_duplicated_pokemon`.<jupyter_code>len(df_pokemon)
df_pokemon = df_pokemon[~df_pokemon["id"].isin(df_duplicated_pokemon["id"])]
len(df_pokemon)<jupyter_output><empty_output><jupyter_text>Let's drop the duplciated rows  Download Additional Pokemon DataNow that we've gotten a handle on the core Pokemon data lets get some additional data.<jupyter_code>def load_pokemon_data(filename):
    url = "https://github.com/veekun/pokedex/raw/master/pokedex/data/csv/"
    download_file(url + filename, filename)
    return pd.read_csv(filename)<jupyter_output><empty_output><jupyter_text>Types* **id** - Unique ID for Type* **identifier** - Name of the type* **generation_id** - When was the type introduced* **damage_class_id** - ???<jupyter_code>df_types = load_pokemon_data("types.csv")
df_types.sample(5)
df_types["id"].is_unique<jupyter_output><empty_output><jupyter_text>Pokemon TypesLinking Pokemon to Types<jupyter_code>df_pokemon_types = load_pokemon_data("pokemon_types.csv")
df_pokemon_types.sample(5)<jupyter_output><empty_output><jupyter_text>Duplicated Pokemon TypesCan a Pokemon have more than 1 type?<jupyter_code>duplicate_index = get_duplicate_indexes(df_pokemon_types, ["pokemon_id", "type_id"])
print("number of duplicated rows:", len(df_pokemon_types[duplicate_index]))<jupyter_output>number of duplicated rows: 0<jupyter_text>Joining DataFramesLet's join the seperated datasources, in a fashion not to dissimilar to a SQL style join.Credit: Pandas Project - https://pandas.pydata.org/pandas-docs/stable/merging.htmlbrief-primer-on-merge-methods-relational-algebra<jupyter_code>df_pokemon = df_pokemon.merge(df_pokemon_types, left_on="id", right_on="pokemon_id", how="inner")
df_pokemon.head()<jupyter_output><empty_output><jupyter_text>But now we've got some extranious columns, lets get rid of them.<jupyter_code>df_pokemon = df_pokemon.drop(["pokemon_id"], axis = 1)<jupyter_output><empty_output><jupyter_text>Speaking of columns, we have columns with the same names in our `df_pokemon` and `df_types` dataframes.Thankfully we can specify a suffix when merging dataframes.<jupyter_code>df_types.columns
df_pokemon.columns
df_pokemon = df_pokemon.merge(df_types, 
                              left_on="type_id", 
                              right_on="id", 
                              how="inner", 
                              suffixes=("", "_type"))
df_pokemon.head()
df_pokemon = df_pokemon.drop(["type_id", "id_type"], axis = 1)
df_pokemon.head()<jupyter_output><empty_output><jupyter_text>Grouping and Plotting Let's get an idea of the distribution of the Pokemon types.<jupyter_code>df_pokemon_type_group = df_pokemon[["identifier_type", "identifier"]].groupby(by="identifier_type").size()
df_pokemon_type_group = pd.DataFrame(df_pokemon_type_group)
df_pokemon_type_group<jupyter_output><empty_output><jupyter_text>So nice to have in our plot:* Sort the data display so that comparisons are easier, and we have an immediate sense of ranking.* Make the barplot horizontal so we don't have to tilt our heads sideways.* Give each bar it's own colour.<jupyter_code>df_pokemon_type_group = df_pokemon_type_group.reset_index()
df_pokemon_type_group.columns = ["type", "count"]
df_pokemon_type_group = df_pokemon_type_group.sort_values(by = "count", ascending = False)
sns.set_style('ticks')

fig, ax = plt.subplots()
fig.set_size_inches(11.7, 8.27)

locs, labels = plt.xticks()
plt.setp(labels, rotation = 0)

sns.barplot(y="type", x="count", data = df_pokemon_type_group)<jupyter_output><empty_output><jupyter_text>Correlations A Pokemon has abilities, abilities may be shared amongst pokemon.  Abilities<jupyter_code>df_abilities = load_pokemon_data("abilities.csv")
len(df_abilities)
df_abilities.head()<jupyter_output><empty_output><jupyter_text>Pokemon to Abilities Mapping<jupyter_code>df_pokemon_abilities = load_pokemon_data("pokemon_abilities.csv")
len(df_pokemon_abilities)
len(df_pokemon_abilities)
df_pokemon_abilities.head()
df_pokemon_abilities = pd.merge(df_pokemon_abilities, df_abilities, left_on = "ability_id", right_on = "id", how = "left")
len(df_pokemon_abilities)
df_pokemon_abilities = df_pokemon_abilities[["pokemon_id", "ability_id", "identifier", "slot","is_hidden"]]
df_pokemon_abilities.columns
df_pokemon_abilities.columns = ['pokemon_id', 'ability_id', 'ability_identifier', 'ability_slot', 'ability_is_hidden']
df_pokemon_abilities.head()
df_pokemon.head()
df_pokemon_skills = pd.merge(df_pokemon[["id", "identifier","identifier_type"]], df_pokemon_abilities, left_on = "id", right_on = "pokemon_id", how = "left")
df_pokemon_skills.head()<jupyter_output><empty_output><jupyter_text>What are the top 5 most common abilities in our pokemon skills dataframe?<jupyter_code>ability_counts_df = pd.DataFrame(
    df_pokemon_skills["ability_identifier"].value_counts()
)
ability_counts_df.columns = ["ability_count"]
ability_counts_df.head()<jupyter_output><empty_output><jupyter_text>Categorical DataWorking with categorical data, e.g. `identifier_type` sometimes require you to generate 'dummy' values.That is creating multiple columns of `True` and `False` from a single column<jupyter_code>df_pokemon[["identifier", "identifier_type"]].sample(5, random_state  = 42)<jupyter_output><empty_output><jupyter_text>Looking the sample above we want to create 4 additional columns.<jupyter_code>dummy_types_df = pd.get_dummies(df_pokemon["identifier_type"])
dummy_types_df.sample(5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1019139393401128, 'fraction_numerical': 0.009991562680403215, 'mean_word_length': 6.98864845689961, 'pattern_counts': {'":': 1, '<': 247, '<?xml version=': 0, '>': 251, 'https://': 9, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '61', 'dir': 'jupyter-structured-clean-dedup', 'id': '346917', 'license': 'MIT', 'n_tokens_mistral': 7715, 'n_tokens_neox': 7313, 'n_words': 1989, 'path': 'Pokedex.ipynb', 'repo_name': 'ryancollingwood/Pokedex', 'size': '22543'}