<jupyter_start><jupyter_text><NAME>  Research question/interests
I am interested to see which superheroes would make the ultimate Avengers team. I also want to see (if possible) which specific traits make one superhero stronger than the other.<jupyter_code>import pandas as pd
pd.read_csv("../data/raw/superheroes_nlp_dataset.csv")<jupyter_output><empty_output><jupyter_text>Milestone 3 (with edits for Milestone 5)<jupyter_code>import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
avengers = pd.read_csv("/Users/shiobhanbohach/school/school/year3/project-group04-project/data/raw/superheroes_nlp_dataset.csv")
avengers.shape
avengers.head()
avengers.columns
avengers.info()
avengers.head()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1450 entries, 0 to 1449
Data columns (total 81 columns):
 #   Column                            Non-Null Count  Dtype  
---  ------                            --------------  -----  
 0   name                              1448 non-null   object 
 1   real_name                         1301 non-null   object 
 2   full_name                         956 non-null    object 
 3   overall_score                     1450 non-null   object 
 4   history_text                      1360 non-null   object 
 5   powers_text                       1086 non-null   object 
 6   intelligence_score                1450 non-null   int64  
 7   strength_score                    1450 non-null   int64  
 8   speed_score                       1450 non-null   int64  
 9   durability_score                  1450 non-null   int64  
 10  power_score                       1450 non-null   int64  
 11  combat_score                      1450 non-null   int64  
 12  superp[...]<jupyter_text>Because I am interested in which superheroes would make the strongest avenger team, I am most interested in data relating to their strength & powers, rather than their backstory To simplify the data, I will be looking at my research question from the standpoint of overall scores, so will remove all individual score data<jupyter_code>avengers_cleaned = avengers.copy().drop(["has_energy_blasts","has_enhanced_senses","has_invulnerability","has_stealth","has_accelerated_healing","has_intelligence","has_electrokinesis","has_energy_constructs","has_mind_control", "has_matter_manipulation","has_flight","has_weapons_master","has_reflexes","has_telepathy","has_regeneration","has_immortality","has_teleportation","has_force_fields", "has_energy_manipulation","has_endurance","has_longevity","has_weapon-based_powers","has_marksmanship","has_super_speed","has_durability","has_energy_absorption","has_cold_resistance","has_magic","has_telekinesis","has_energy_beams","has_shapeshifting","has_jump","history_text","has_mind_control_resistance","has_enhanced_hearing","has_telepathy_resistance","has_dimensional_travel","type_race","gender","eye_color","img","relatives","teams","hair_color","skin_color","has_element_control","has_size_changing","alter_egos","aliases","place_of_birth","powers_text","first_appearance","alignment","occupation","base","height","weight","has_fire_resistance","has_self-sustenance","has_peak_human_condition","has_fire_control","has_dexterity","has_reality_warping","has_illusions","has_heat_resistance", "has_toxin_and_disease_resistance","has_stamina","has_agility","has_super_strength"],axis=1)
avengers.nunique(axis=0)
avengers.describe()
avengers_cleaned = avengers_cleaned.dropna(axis=0)
avengers_cleaned.shape<jupyter_output><empty_output><jupyter_text>The above data now only contains superheroes with positive integers in all columns I will now clean up the rows so that only superheroes who score in the 75% of strength score remain<jupyter_code>avengers1 = avengers_cleaned[avengers_cleaned["strength_score"].between(95, 100)]
avengers1.nunique(axis=0)
avengers1.describe()<jupyter_output><empty_output><jupyter_text>When you adjust for just the strenght score you still see all of the score means increasing, indicating that a high strength score is positively correlated to having high scores across the board<jupyter_code>corr = avengers1.corr()
corrmap = sns.heatmap(corr, xticklabels=corr.columns, yticklabels=corr.columns, annot=True, cmap=sns.diverging_palette(220, 20, as_cmap=True))
corrmap.set_title("Relationship Between Individual Score Catagories")<jupyter_output><empty_output><jupyter_text>As expected by previous findings, there is a fairly strong correlation between strength scores and all other scores I will now adjust all other scores to the 95-100 range, to see the effect on the overall scores and see if we can narrow down to our ultimate avengers team<jupyter_code>avengers2 = avengers1[avengers1["power_score"].between(95, 100)]
avengers3 = avengers2[avengers2["speed_score"].between(95, 100)]
avengers4 = avengers3[avengers3["combat_score"].between(95, 100)]
avengers5 = avengers4[avengers4["intelligence_score"].between(95, 100)]
avengers6 = avengers5[avengers5["durability_score"].between(95, 100)]
avengers6<jupyter_output><empty_output><jupyter_text>I am surprised that the only original avenger to make it onto this Ultimate Avengers Team is Thor (and it is more high powered versions of Thor that do make it on - not the version we see in the Marvel Avengers movie series). I am also suprirsed that a couple of our final heros still have fairly low overall scores leading me to believe that there are other factors I cleaned out of the data that may have a strong impact on overall score.<jupyter_code>graph1 = avengers_cleaned["power_score"].plot(kind='hist', bins=20, figsize=(15,6), facecolor='purple',edgecolor="white")
graph1.set_title("Power Score Frequency - Overall")
graph1b = avengers1["power_score"].plot(kind='hist', bins=20, figsize=(15,6), facecolor="purple",edgecolor="white")
graph1b.set_title("Power Score Frequency - Top 109 Strongest Superheroes")
graph2 = avengers_cleaned["intelligence_score"].plot(kind='hist', bins=20, figsize=(15,6), facecolor="turquoise",edgecolor="white")
graph2.set_title("Intelligence Score Frequency - Overall")
graph2b = avengers1["intelligence_score"].plot(kind='hist', bins=20, figsize=(15,6), facecolor="turquoise",edgecolor="white")
graph2b.set_title("Intelligence Score Frequency - Top 109 Strongest Superheroes")
graph3 = avengers_cleaned["combat_score"].plot(kind='hist', bins=20, figsize=(15,6), facecolor="orange",edgecolor="white")
graph3.set_title("Combat Score Frequency - Overall")
graph3a = avengers1["combat_score"].plot(kind='hist', bins=20, figsize=(15,6), facecolor="orange",edgecolor="white")
graph3a.set_title("Combat Score Frequency - Top 109 Strongest Superheroes")
graph4 = avengers_cleaned["durability_score"].plot(kind='hist', bins=20, figsize=(15,6), facecolor="green",edgecolor="white")
graph4.set_title("Durability Score Frequency - Overall")
graph4a = avengers1["durability_score"].plot(kind='hist', bins=20, figsize=(15,6), facecolor="green",edgecolor="white")
graph4a.set_title("Durability Score Frequency - Top 109 Strongest Superheroes")
graph5 = avengers_cleaned["speed_score"].plot(kind='hist', bins=20, figsize=(15,6), facecolor="brown",edgecolor="white")
graph5.set_title("Speed Score Frequency - Overall")
graph5a = avengers1["speed_score"].plot(kind='hist', bins=20, figsize=(15,6), facecolor="brown",edgecolor="white")
graph5a.set_title("Speed Score Frequency - Top 109 Strongest Superheroes")<jupyter_output><empty_output><jupyter_text>Based on comparing the frequency graphs for speed, power, intelligence, combat, and durability from the original cleaned dataframe, to the dataframe adjusted to include only superheroes scoring in the 95% tile, it is clear that a superhero that is higher in strength is more likely to have a higher score in the other catagories. However, the correlation is not incredibly strong as we can see that there are still low scores that come up. Unsurpisingly, power and combat appear to be the most highly correlated to strength. The following graphs will now compare the overall score with the individual catagory score of the "Ultimate Avengers Team" - aka the superheroes who scored in the 95% tile in each individual catagories (power, intelligence, combat, speed, durability, strength).<jupyter_code>sns.set(style="darkgrid")
plt.figure(figsize=(10, 7))
sns.barplot(
    x="overall_score", 
    y="power_score", 
    data=avengers6, 
    estimator=sum, 
    ci=None, 
    color="purple")
sns.set(style="darkgrid")
plt.figure(figsize=(10, 7))
sns.barplot(
    x="overall_score", 
    y="intelligence_score", 
    data=avengers6, 
    estimator=sum, 
    ci=None, 
    color="turquoise")

sns.set(style="darkgrid")
plt.figure(figsize=(10, 7))
sns.barplot(
    x="overall_score", 
    y="combat_score", 
    data=avengers6, 
    estimator=sum, 
    ci=None, 
    color="orange")
sns.set(style="darkgrid")
plt.figure(figsize=(10, 7))
sns.barplot(
    x="overall_score", 
    y="durability_score", 
    data=avengers6, 
    estimator=sum, 
    ci=None, 
    color="green")
sns.set(style="darkgrid")
plt.figure(figsize=(10, 7))
sns.barplot(
    x="overall_score", 
    y="speed_score", 
    data=avengers6, 
    estimator=sum, 
    ci=None, 
    color="brown")
sns.set(style="darkgrid")
plt.figure(figsize=(10, 7))
sns.barplot(
    x="overall_score", 
    y="strength_score", 
    data=avengers6, 
    estimator=sum, 
    ci=None, 
    color="yellow")<jupyter_output><empty_output><jupyter_text>As is shown in these graphs, having high scores in the individual catagories (i.e. speed, strength, power, etc) does not mean that a superhero will have a high overall score. I find this super interesting because it means that there are other factors that have been cleaned out of my dataset that impact the overall score of the superheroes.  Overall Findings and Answers to Research QuestionsI started this project wanting to find out 2 things: 1. Which superheroes would make the Ultimate Avengers Team?- What I have found is that the way you decide to analyze and clean the data will impact which superheroes make it onto the Ultimate Avengers Team. I decided to analyze my data by narrowing it down to the superheroes that scores in the top 95% of each individual score catagory. Based on this parameter here is the "Ultimate Avengers Team":   - Abraxas   - Cosmic King Thor   - General Zod   - Golden Ninja   - Man of Miracles   - <NAME>   - Monarch   - Old King Thor   - One-Above-All   - Ragnarok   - Shazam   - True Form Oblivion2. Which specific traits make one superheroe stronger (in overall score) than another.- The answer to this question is not as straight forward as the first one. Mainly because, according to my graphical analysis, the overall score is not strongly correlated to the individual scores. Taking a look at the overall scores of the superheroes on the "Ultimate Avengers Team", several of them have overall scores significantly below 100. Considering that each superhero on this list scored between 95 - 100 in each individual catagory, it is clear that the overall score is not a weighted average score of the individual catagory scores. The only strong correlation we see is that if a superhero scores high in one catagory, they are more likely to score higher in the other catagories, this finding is supported by what was shown by the frequency graphs.<jupyter_code>avengers1.to_csv('cleaned_Shiobhan.csv')
avengers6.to_csv('cleaned_top12_Shiobhan.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09908735332464146, 'fraction_numerical': 0.025380269448066057, 'mean_word_length': 5.146367521367521, 'pattern_counts': {'":': 1, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '569931', 'license': 'MIT', 'n_tokens_mistral': 3672, 'n_tokens_neox': 3335, 'n_words': 1030, 'path': 'notebooks/analysis2.ipynb', 'repo_name': 'data301-2021-winter1/project-group04-project', 'size': '11526'}