<jupyter_start><jupyter_text>The NarWolves: DC vs Marvel Data Analysis  Loading libraries and importing DC vs. Marvel dataThe dataset was obtained from [Kaggle](https://www.kaggle.com/hetulmehta/marvel-vs-dc-imdb-dataset). The dataset first cleaned and processed in this [notebook](https://github.com/nguyenjenny/spark_shared_repo/blob/main/group_02/CleanData.ipynb).<jupyter_code># Import libraries
import numpy as np # For math- and matrix-based functions
import pandas as pd # For DataFrame functionality
import seaborn as sns # For plotting/ data visualization
from sklearn.model_selection import train_test_split # For the train_test_split_function
import statsmodels.formula.api as smf # Stats models for  

# Figure size settings
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (16,7)

# Get path of data (where it is located on GitHub)
path = "https://raw.githubusercontent.com/nguyenjenny/spark_shared_repo/main/group_02/Marvel_DC_imdb_cleaned.csv"

# Load the data into a DataFrame
hero = pd.read_csv(path)

# Show DataFrame
hero<jupyter_output>/usr/local/lib/python3.7/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm<jupyter_text>Analyzing data  Getting to Know our Data: Aggregates and Basic Stats  Basic statisticsWe can use the `.describe()` function to get basic information about our numerical data.<jupyter_code># Use the `.describe()` to get information on the hero DataFrae

hero.describe()<jupyter_output><empty_output><jupyter_text>Mean IMDB_ScoreWe can use the `.mean()` function to get information about the average `"IMDB_Score"`First, we need to select the column of the DataFrame that we want.  We can use `.iloc[:, "column_name" ]` or we can use the short form of just `["column"]`<jupyter_code># Use loc to select the `"IMBD_Score"`
hero.loc[:, "IMDB_Score"]
# Use the short hand version ["column"] to get the "IMBD_Score"

hero['IMDB_Score'].mean()<jupyter_output><empty_output><jupyter_text>The mean IMDB score for both Marvel and DC movies is **7.99**.   Lowest and Highest IMDB_ScoreWe can use the `.min()` function to get information about the lowest `"IMDB_Score"` and `.max()` to get the highest `"IMDB_Score"` for comic/superhero movies.<jupyter_code># Get min/lowest IMBD_Rating
hero["IMDB_Score"].min()
# Get max/highest IMBD_Rating
hero["IMDB_Score"].max()<jupyter_output><empty_output><jupyter_text>The IMDB scores for both Marvel and DC movies range from **3.4** to **9.6**.   Standard Deviation of IMDB_ScoresStandard deviation represents the spread or variability of the data.![std.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAhwAAAD4CAYAAACntD/BAAAEGWlDQ1BrQ0dDb2xvclNwYWNlR2VuZXJpY1JHQgAAOI2NVV1oHFUUPrtzZyMkzlNsNIV0qD8NJQ2TVjShtLp/3d02bpZJNtoi6GT27s6Yyc44M7v9oU9FUHwx6psUxL+3gCAo9Q/bPrQvlQol2tQgKD60+INQ6Ium65k7M5lpurHeZe58853vnnvuuWfvBei5qliWkRQBFpquLRcy4nOHj4g9K5CEh6AXBqFXUR0rXalMAjZPC3e1W99Dwntf2dXd/p+tt0YdFSBxH2Kz5qgLiI8B8KdVy3YBevqRHz/qWh72Yui3MUDEL3q44WPXw3M+fo1pZuQs4tOIBVVTaoiXEI/MxfhGDPsxsNZfoE1q66ro5aJim3XdoLFw72H+n23BaIXzbcOnz5mfPoTvYVz7KzUl5+FRxEuqkp9G/Ajia219thzg25abkRE/BpDc3pqvphHvRFys2weqvp+krbWKIX7nhDbzLOItiM8358pTwdirqpPFnMF2xLc1WvLyOwTAibpbmvHHcvttU57y5+XqNZrLe3lE/Pq8eUj2fXKfOe3pfOjzhJYtB/yll5SDFcSDiH+hRkH25+L+sdxKEAMZahrlSX8ukqMOWy/jXW2m6M9LDBc31B9LFuv6gVKg/0Szi3KAr1kGq1GMjU/aLbnq6/lRxc4XfJ98hTargX++DbMJBSiYMIe9Ck1YAxFkKEAG3xbYaKmDDgYyFK0UGYpfoWYXG+fAPPI6tJnNwb7ClP7IyF+D+bjOtCpkhz6CFrIa/I6sFtNl8auFXGMTP34sNwI/JhkgEtmDz14ySfaRcTIBInmKPE32kxyyE2Tv+thKbEVePDfW/byMM1Kmm0XdObS7oGD/MypMXFPXrCwOtoYjyyn7BV29/MZfsVzpLDdRtuIZnbpXzvlf+ev8MvYr/Gqk4H/kV/G3csdazLuyTMPsbFhzd1UabQbjFvDRmcWJxR3zcfHkVw9GfpbJmeev9F08WW8uDkaslwX6avlWGU6NRKz0g/SHtCy9J30o/ca9zX3Kfc19zn3BXQKRO8ud477hLnAfc1/G9mrzGlrfexZ5GLdn6ZZrrEohI2wVHhZywjbhUWEy8icMCGNCUdiBlq3r+xafL549HQ5jH+an+1y+LlYBifuxAvRN/lVVVOlwlCkdVm9NOL5BE4wkQ2SMlDZU97hX86EilU/lUmkQUztTE6mx1EEPh7OmdqBtAvv8HdWpbrJS6tJj3n0CWdM6busNzRV3S9KTYhqvNiqWmuroiKgYhshMjmhTh9ptWhsF7970j/SbMrsPE1suR5z7DMC+P/Hs+y7ijrQAlhyAgccjbhjPygfeBTjzhNqy28EdkUh8C+DU9+z2v/oyeH791OncxHOs5y2AtTc7nb/f73TWPkD/qwBnjX8BoJ98VQNcC+8AAABEZVhJZk1NACoAAAAIAAIBEgADAAAAAQABAACHaQAEAAAAAQAAACYAAAAAAAKgAgAEAAAAAQAAAhygAwAEAAAAAQAAAPgAAAAAdk1rkAAAAVlpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6dGlmZj0iaHR0cDovL25zLmFkb2JlLmNvbS90aWZmLzEuMC8iPgogICAgICAgICA8dGlmZjpPcmllbnRhdGlvbj4xPC90aWZmOk9yaWVudGF0aW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KTMInWQAAKSdJREFUeAHtnc1v3MiVwKulBNjJ3LK3TIIsJsDOjNSyFxgkhwSb634AC4wtyZZl5N/YRWLLHsvjDXbzF+SawPqwJMt723M2yWUuGakl20Em2MHEOc4t4z3MmFuPVDWL3exuks2PKvJHQOpusvjqvV8VycdXj6xeoBfFAgEIQAACEIAABCoksFChbERDAAIQgAAEIACBkAAOBx0BAhCAAAQgAIHKCeBwVI6YCiAAAQhAAAIQwOGgD0AAAhCAAAQgUDkBHI7KEVMBBCAAAQhAAAI4HPQBCEAAAhCAAAQqJ4DDUTliKoAABCAAAQhAAIeDPgABCEAAAhCAQOUEcDgqR0wFEIAABCAAAQjgcNAHIAABCEAAAhConAAOR+WIqQACEIAABCAAARwO+gAEIAABCEAAApUT+ErlNXhYQa/XC7V+ob4Zfn4j+NRDK1AZAhCAAAQg4A4BIhzutAWaQAACEIAABFpLAIfDato/976lJLphIhtmk6xngQAEIAABCECgOAEcjgt24lS8of405mwYtD44Ha/21kOHSZwm++/vbp8aMyr5DM7uqr6uc/GdO+rsooa0dZVUXpLQ3dWvhsyqYlW1/JIwIAYCEIBAZQRwOCpDW69gcTYWbxzqSlfVoyBQQbCnrqooF6VKTaTehf720NGQutLWVanDfLIHamt5UW0+/mI+MRP3rlr+xIrZAAEIQMApAjgcF80xLbphWszlKMf+wZNQzcu37qr18Nt1dTjYUsv6++/PTNzBWFLe58LGgfpydy0hMG1dokDGHyZKcn0n4w6FivXV/bMv1c7VqvKnq5ZfyGh2ggAEIFA7ARwOjTwtb2NSS7jqdCwvvxOqfHp0MIw29JbvqYGOdnx+vDHJnFLW91aWQsfGFpa2zt6e5fvp/vHQlizl5ylj+M0jY9q+VcufVjfbIAABCLhAAIfDhVYoQYdL2zvq9tKCevXsvs6nWFH3UtI2Tu5cSuR2mDyP11fvhMMK5rd8fu3K3lCrtP3KijqM551Eukudl+9HRjy6GeWkSH7FLF3Sto/aI4aZnApjs6nLGJ0mx7Z5XO8ks1nyTT18QgACEOgKgc47HBKxGH0qxc/Gj0L3Ue7GmXr/UnyRFnvsC7j8vvYwzvP4/PF99cH5knr/JFCvBnfCaMXLJw8unJaBOjiKhmQu3zoZbn90c00diKC5ln21/sEz1X9rK4zEfLS1oqUN1Ln2My5tnwyHOSJdA/W7D3pTdRm18bX3doe5LC+f3FDGYRBnIMrZiPJdRoeERIdpNouzEeXLpBs/W376fqyFAAQg0GYCnXc4JHcj7+LqsEpkx3V1FLwaOgYfPbgURivkAh5d0JWSC/H+ppS+rtauLoa7Xb61o+7q631veV1d0ZGSeDGOjFzwxSEoc1lWS8G5Gjy/r76rIyqiY6CHgCLd0uqZrott48LbW+rDcCgptlEkSl7Ig4sE0WsPD8N8F8k5SeZwTK/HzlGRemTYSvSWoats8tNsYx0EIACBdhOwryzttrRj1kn+xk4YMVAqjlZEEP52WVJJk8tbS7OdCYkgjD6RkpSS95dc2D8Kh4IkAhENb2SLnMzSZWV1fSyvRLQLTs8v8kJW1VrodEU6T8qxmFTPpByVvPLzEqM8BCAAAV8JdNrh8GU4ZfYzJvtqbW1Nrb0X512U1yGjxzrFGRjNcyinjjiaEEVgjtSG9T6PZB3z6xI7Ckfq0Hr65ezsqVVV8Xqyybeq4isEIACBjhDotMNRZDjF9Is6h1XG4xFGi/gzOHqsjv7rhpXsua/u3R+EBRbevqLWrABG/Jiszpd49ios81wSJ8LlTJ2fB/pblEsRnB2o4/OojORSmGGZi8Lhh7mrf/XsXJ1fbEhbZ+8j303ipUlQ7S+9NVok/C26ne7uqd33N3WuyXRdjIA0G2WbPWRk8lDsYRDZr6jNWeUbHfmEAAQg0CkCeuy5s4tu6EAnjKb+Tdtm9nEH3F6gX/IlXsLYn84xCHSOQaBzFBLbvnZ1K9BPtSTWXb67NSZHOxlj+9r19Pv9hAz94rFgd391bJ1+GdnYopM1R8qJ/v1AJ6+Gi3ZuEtt10upUXWy95Pvra6up9gTB6Zjt9r46x2VqPWk2J+2bLX8MBisgAAEItJxAT+zTJ9vOLbMiFFleBMYssp3rNhgMAQhAAAIFCXR6SKUgM3aDAAQgAAEIQCAngc46HPPkbxjGs6IkphyfEIAABCAAga4T6KTD4cvTKV3vnNgPAQhAAALtIdBJh6M9zYclEIAABCAAAT8IdNLhKGM4xY/mRUsIQAACEICAGwQ66XCUOXcKeRxudGS0gAAEIAABtwl00uFwu0nK087MWCqzrJaxyGu+87x+fFKdZcmZJJ/1EIAABCDgHoHOvYcja0Qiy3s4THO69z4OeTX35eFbOWWW1/InXjPW8wkBCEAAAhCYTYAIx2xGM0tkdWJmCiqtQDQ/SXIG1NKE5xIkrw3v63lYzNTwuXamMAQgAAEItIYADkdrmnLckEkzoI6XrG7N6f7xxeys1dWBZAhAAAIQcJ8ADof7bZRZQ5OzEeVZpM/uGudP9Kx8jHh21GjfFXVPp33YZSUPxEy2tmjN5mqXMfWaaIZsMzPMProZ1TdJjhhp5Bs58mnnn6TVJWXM5G+ZQVEQAhCAAARqJ4DDUTvyaioUZ2Pz8Rda+KrSE4kpPTHaWEVSRhwAPTmZkil0Xg3uqGUl08E/UhtnD5WeAC7c5/KtHXVXzy57aXtHbW1tqa3bu+oXCzfV4o3DEZkDdXB0drHPyYU8pcwsrJe2T5QZ1pGZZqXOdDmRcxPJj/QPgr1Qn48eXAodCtt5kQojG6IyL5/cYMhmpGX4CQEIQMA1Ap1yOKrKtWj6vR729OrXHh6qdd3LFjYOhhd76XR2GblAS2Rgob8dDndE08pvqLtb/bB/mqndZZr2x/eP1eK1De18nKQ4MVGuiDgSWZNS0+SIbpvaEZLF6K/UdXWwuxque/nkgTpeP1F69tjwt54BV314vKG/X1drVxfDdfyDAAQgAAG3CXTK4ajKMSjzvR5Fuktwen6RJ7Gq1jZjCek5HH2lp38Pow3iKER/kZNyaftWGFWQC7wMqQxO3lG3d2+pteg6r3orSzoikr5IBMI4MOkl4rWT5fTV0kVdUjouN1DnkT8SCllZXZ+oR1wL3yAAAQhAwCUCnXI4XAJfpi7xhflIHe7Eks/OnsY/ht+SF+/h6vCLiRgM1M+219TB03N9oV+ecnGPcz9MrkZSXt5fSd1iRyrpiOSVSnkIQAACEGieQGccDhlOqTISUdVwTZYu0lteV1eWoqY0+RP2EIoMkfSW76lbV78Sint0M0oKlR9Srt+Lf29s/yR0MP5y+FydD5RaeRCHHIwDEA3ByL4H6vj8VShTcjTMkEe4YuTfcx2iON3dU3t7e2pwFkVkjBzRbediuCTWbV+t3zgKpbz23q0wp8SINEM+Ogajzp9F9ZttfEIAAhCAgKMEdEi9E4vGH2iHI/Nf3vIiu9nlNLi9tBCI3ml/OskyVE8ncY5t1zkRwcBS3pTRLwwbrtVJqCP7rQY6OTUwZdPqlP21E5LYr9/vJ37rJNdQjlQ0WlZkGh1G63l9bTXQSa4JWdrpYYEABCAAAUcJdOZNo5IkmSfCkedNo/rCGC7uvXHUaMYnBCAAAQhAoFkCnRlSyeNsFGmSqhJSjS4y9MECAQhAAAIQ8JVAZxyOqhtIHJom8ziqtg/5EIAABCAAgXkI4HDMQ499IQABCEAAAhDIRKATORxFIg9FcjhkH52rkwk8hSAAAQhAAAJdItCJCEfV+RWmw1SdJ2Lq4RMCEIAABCDgG4FOOBy+NQr6QgACEIAABNpGoBMOB5GHtnVb7IEABCAAAd8IdMLh8K1R0BcCEIAABCDQNgI4HCW3aJEE1ZJVQBwEIAABCEDAOQKtdzhwAJzrcygEAQhAAAIdJNB6h6ODbYrJEIAABCAAAecItN7hqOuRWOdaFoUgAAEIQAACDhFotcMhwylNPKHCMI5DPRxVIAABCEDACQKtdjicIIwSEIAABCAAAQioVjscDKfQwyEAAQhAAAJuEGi1w9HEcIo0K46OG50bLSAAAQhAwB0CrXY43MGMJhCAAAQgAIFuE8DhqKD9m4qsVGAKIiEAAQhAAAKlEMDhKAUjQiAAAQhAAAIQmEagtQ5H04+mNl3/tEZnGwQgAAEIQKBuAq11OEjcrLsrUR8EIAABCEBgMoHWOhyTTWYLBCAAAQhAAAJ1E2itw9F04qZEWBhWqbs7Ux8EIAABCLhKoLUOR9PAm3Z4mraf+iEAAQhAAAI2ARwOmwbfIQABCEAAAhCohEArHY7vqNfU338WVAIMoRCAAAQgAAEI5Cfwlfy7uL/HH3v/p2eJ7bmvKBpCAAIQgAAEOkKglRGOjrQdZkIAAhCAAAS8IYDDUXFTyfAOCwQgAAEIQKDrBFrpcHxn5wfOtKsM77BAAAIQ8JPAvlrtLaherzfyt6YO/DSoEa13V78a8vu726eV1F+1/LKUbqXDURYc5EAAAhDoNoHr6ih4pXauxul+r723q4LgUK13G0xG6wdqa3lRbT7+ImP5vMWqlp9Xn+nlW+dwMIQxvcHZCgEIQCAvgevr7w13+Zf1jeF3n78EZ3dVX0duru9UaUVf3T/7MuGwlVtb1fLL1bZ1Dke5eJAGAQhAAAJtJHC6f6zOajJsefmdSmuqWn5ZyrfO4XAtZ8KlfJKyOg1yIAABCKQROLlzaSTXIxlBeLW3Prb9a1f2QlFp20zuyOI7d0LnYFz+9FyScZkr6p5OoxA5l+9H+RSPbkb5KZJfMS4/qX/adtHR2GCYmJwKo7+py2xPk2NHWsb1TtYxS76px7XP1jkcrgFGHwhAAALdIDBQB0dRzODyrRP1anBHLWvDH92MnAK5iC7eOExFEW9bVY+CYLivUtHvL59uqxOdeCkX7iiHxJQ5UhsXzsi44H21/sEz1X9rSw20zI+2VnSRgTrXfsal7ZPhMMe1h4HOSQnU7z7oTdXfdlKkrkiPPXVVv/Pp5ZMbw6EZcQainI0L3XfXRlQrzkkEzZY/Up1DP3E4HGoMVIEABCDgL4EonyC6eMvFPbksbByoLy8uvgtvR06AlP38OM4Jee29tTAZtbe8rq4syeXpeeggSL7Fg4vES7m4S+Rgob8dRj1ePTtX58mqLn4tq6XgXA2e31ff1VEUcTKkvv3N1MJ65XT9Zf/IaVFK9P8w1Pu6Wru6OBRo63ntYZRYK3bbSbez6pnGKZv8oTrOfcHhqKFJSGStATJVQAACDREYqMFgoE5P4uolGmAcgnitUr2VpTDqYa+T7+FF+cqq+ufFw/Bx2+DsQB2fv9Jb3lJLCd+lr94/iSIS4jxEf5OemBEH4iN1Wzsuxknp9aYPwRi9Julvtq+srqfaEZyeX+SFrKo1y7GZlGMxqZ5JnPLKN/q68tk6h8PFnAnX8kpc6XzoAQEI+E9AnIONlRX1gR6rkEdAJfowmrMwy8rwzv34SB0dHalrw+iFDEmMOhPRkMgsefH2OGoRRSemDcFEj5gW0d/UFzsKR+rQevrl7OypKaI/i9eTTb5VlWNfW+dwOMYXdSAAAQh4T2D/4MnQBpOTISvEUVgxQxtPTVRCKcmLMMMPwx2l/EUEYHQYZG/rTL29GuU8pEUuesv31K2Ld4E8uhklfpr6+734t12XSbw0CZ39pbfszcPvz7WjdLq7p3bf39ROk0RVJutvdvr9mXm+RTtAz6J9ZFs8FBTnrtjDILJfHL2ZXM8kTlnkGx2d/NSN25rlzeCvAh3hCP9eqG8G8/zpxpprf1O36COyWCAAAQj4R2Av0EmR4TlMzmOT//qBHuoIdK7CxDL9fn9k22qgE0TDRSeYBjrBdGR79Fs7L8MlTb7Opwh0UujYovNFUuRFekph7RAltutE16n6j9r++trqGJtI19NAD+MkZNv76mTTqfVM4xQZOVv+GAxHVvREDw2jFYuEwsyQyq82P5nLpjfUn/SMs9+cS4bs/MOdb4cy/nDj13PLQgAEIACB9hGQIYbLYXRBkjFP9BMp8nRLNPQg65d03sapupvI5WgfhS5YxJBKF1oZGyEAAQg4S+BM9VZvqa2fnCh5/DVyNnS4Y5g46qziKJaTQGsdjh//6Ps5UZRf3AUdyrcKiRCAAATKJKAfLVVP1P1/T740LHrKRZ5KIbpRJu0mZcUz8jSpBXVDAAIQgEBnCYTvyNjurPmdMby1EQ7XWpB3cbjWIugDAQhAAAJ1EmhVhMMkjNYJMGtd4bs4WpOem9VqykEAApLMPrqkJaR/I/h0tBi/IdAqAq1yOFrVMhgDAQh4ReDPvW8N9ZWn3MxinAt7ndlmf9r743zYZPjeFgIMqbSlJbEDAhConYA4CeZPHArzJ06G+SuilJFpOyFF5FS1j5mtVGZYLWORV3xLJCjrq8cn1VmWnEnyWT8fgdZEOCRHoqfenY8Ge0MAAhDIQcCOWphIRo7dMxW1nY7mIx/xOzMyKZ+xUFlJo2XJyag2xXISIMKRExjFIQABCER3471hFKMqZ2OUtIl8jK6v73c0N0ly9tP6ardrkleG93VU5Lo1Z4m9ne/uEWiNw+H6BGkuJ7S61y3RCALuETAXe/mcZ7jEPcvyazRp9tP8korvcbp/fDEza3EZ7FkvgdY4HPViK1Ybj8YW48ZeEHCBgMnPsIdRmtLLdn7q0MHkbJjITtpssHH+hORimHyMeGbUaF000ZpdVvJAzERri+/cGToRdhlTr4lmyDajw6ObUX2T5AgfI9/IkU87/yStLiljJn6rg3EX6sDh6EIrYyMEIFCIgH1hN1GNuoZPCilcwU7ibGw+/kJLjmZz1ZOijdUiZcQB0BOTyUyVSk/Gpl9RLlPBP1IbZw+VngAu3OfyrZ1wTpRL2ztqa2tLbd3eVb9YuKkWbxyOyByog6NoRlY9qdqFvHgGVsnVMMM6MjOt1JkuRylxJiL5ZjbavVCfjx5cCh0K23kRJSIbojIvn9xgyGakZeb52RqHY3TI4qe//O08XErZd1QH14d9SjEaIRBoEQGXohqTsNpO0aQyRdfbU6tfe3io1rWghY2D4cVe5Npl5AItkYHoteQ6svDsXJ2rDXV3qx+qYKZ1l3lSHt8/VovXNpQ4D+NOTJQrIo7E7z7INmtbmhzRbVM7QrIY/ZW6rg52V8N1L588UMfrJ0rPHBv+lsnjPjze0N/169avLobr+FcegdY4HOUhQRIEINB1AuYi3tWohmn/4PT8YohjVa1tmrVKpedwyLwnUbRBHIXoL3JSLm3fCqMKcoG/p6//g5N31O3dW2rtwpforSwNJ22La4m+SQTCODCj20Z/T5bTV0uW3xKXG6jzyB8JRa2srk/UY7QufucngMORnxl7QAACLSdgIhu+mWkcpbL0ji/MR+rQehrk7OxpShXJi3eygIkYDNTPttfUwVMd+ThdnnJxj3M/TK5GUl7eX0ndYkcq6YjklUr5fARa4XCQjJmv0SkNAQiMEzAXa/k0kY3xUt1a01teV1eWosvEo5vaUdDm20MoMkTSW76nbl2NXun06GaUFCqUpFy/F//e2P5J6GD85fC5Oh8otfIgDjkYByAagklOTS85GmbII43+cx2iON3dU3t7e2pwFkVkjBzRbediuCTWbV+t3zgKRb323q0wp8TINUM+Ogajzp+9Mqv5LIlAT4e9vJ/hI3zp1+67CSS/2vwk8TvvD7nDKSM57Ic73x5W/fHmb8Iw43AFXyAAAWcISO6BWco49o0s82k/3VKFfFOPfJb7grDpL/uSJMvPdd5DnFwaayI5ESdPt4eRDFNGEkFNboY8QZJMGpXkzkP1xTBZNZZnvsn+kiRqRz/6/b4aDLQnM1wiOZJ3MpoYKkWMDkYns9vra6vqHw4fq8cqvjSK07NvDSmZsnzmI9AKh0NOFKNJoy46HNI0f7jx63wtRGkIQKAWAhLZqHKp0+Gw7SjX+bAl8x0C+Qi05tXm+cymNAQgAAEVzoMCBwhAoB4COBz1cKYWCEAAAo0QsCM3RDsaaQIqvSDQiqRRn1qTBFefWgtdIQABCECgLAKtiHCM5m+UBacKOeHLv+JcpCqqQCYEIDCFgH3HP6VYazbZuSOcelrTrF4a0gqHw0vyKA0BCDRCwL4AV/20SCMGjlTaBRtHTOanowRwOBxtGNSCAATKJWAiG12+ABsGQpZ8jnL7F9JmE/A+h4OciNmNTAkIQEApiWzY0Q2YQAAC9RIgwlEvb2qDAAQaItDlyIZBbjtc5HMYKnzWRcD7CIdvM7D6lOBaVyekHghAoB4C4nSZPxlesYdY6tGAWrpMgAhHl1sf2yHQcgJcUCc3sIl2EOmYzIgt5RLA4SiXZyZpknfysXqZqSyFIACB4gTMRVUkMKSS5AiPJA9+VU/Ae4fDxyEK3sVRfcemBggIAS6q9AMIuEPAe4fDHZRoAgEIQMA/AvawE4/K+td+PmmMw+FTa6ErBCAwk4B9AZ1ZmAIQgEBtBLx2OCQXoqferQ0WFUEAAu4TIG/D/TZCw24S8Nrh6GaTYTUEIJBGwEQ2yNtIo5NtnWEopRleycaMUtkJeP0eDt/ewWGaxcdEV6M7nxBwlYBENuzohqt6ohcEukqACEdXWx67IdAyAkQ25m9Q22Hj/Rzz80RCkgAOR5IHvyAAAQh0lgBOW2ebvhbDcThqwTxeCS//GmfCGgjkJWDnHOTdl/LTCRi25HJM58TW7AS8djh8zoXg5V/ZOyklITCJgD0EwN35JEqsh4AbBLx2ONxAiBYQgEBTBHAyqiNvnLkXepI3sxDtMCT4LELAW4eDd3AUaW72gQAEIJCNAM5cNk6Uyk7AW4cju4mUhAAE2kRAbjaGy2d/Pfz6P1/vDb/zBQIQcI+Atw6H5EB8Z8c9oFk18jn/JKuNlINAFQTs9++80O8aZoEABPwg4K3D4QdetIQABCDQHgJ2dOlj9bI9hmFJLQRwOGrBTCUQgEBZBBLRwc1PyhKLnAwE7OiS4s1gGYhRxCbg9avNbUN8/G7fLfioPzpDoC4CcqyYv7rqpB4IQKBcAt5GOBJ3OeUyqU0a7+KoDTUVeU7AvrP2OXfL82bQeXM/8N0E9G+QABGOBuFTNQQgAAEIQKArBLx0OBiK6Er3xE4IQMBVAmaIi/Oxqy3knl5eDqlIeLUNYVXCk+4dEGjkJgGOFffaxR7mIoHUvfZxUSMvIxwugkQnCECgfALmLrp8yUiEAATqJuBlhKNuSNQHAQg0Q8DcRbchotkMwepqJepUHdu2SsbhaGvLYhcEOkDgxz/6/tDKn/7yt8PvfCmPgM1YqRflCUZS5wgwpNJwk5Nw1XADUD0EIDA3AYa+5kbYCQFeRjjaFMoLQ8a8sa8TBxtG5ifQpmM9v/X+7GGGvkge9afNmtCUCEcT1KkTAhCAAAQg0DEC3jkcDEF0rIdibucImPA8x3rnmh6DW07AuyEVCd21KWOdkHHLjzDMy01gGJ7Xe7bpWM8NwqMdOI951FgNquqdw9EgK6qGAAQgAIEpBOyoFNPXTwHV0U04HA40vBykHJwONAQqOEEgz90yj8JW32Q24x/+07enVmhHp0ggnYqqkxv9y+Fo4WyFiYO0k90QoyEAAQhAoO0EiHC0vYWxDwIeELBD8T31rgcaoyIEIJCXgFcOh5yUOBnlbWLKQ8B9AnaUj0RR99trkoZ5hsMmyWB9ewl4NaRin5Ta1CQcpG1qTWyBAAQgAIE0Al5FONIMYB0EIAABCLhHwB4mIynevfZpQiMcjiaop9TJkyopUFjVGQJE+drX1ImINNM3tK+BC1jk1ZBKAfvYBQIQgAAEIAABBwh45XC0+S4ocTfgQMdABQjUQUAie3bovY46qQMCEGiGgDdDKnJS4gmVZjoJtUKgKgLG0ebJlKoINye3zTeIzVH1u2avIhx+o0Z7CEAAAt0kQCSrm+0+arU3EQ65E2rzXRB3A6Ndk99dIEC/70IrK2UiWbzuvBvtPclKIhyTyDSwnrHsBqBTJQQgAAEI1ELAmwhHLTSoBAIQqJyA7ViTl1U5biqAgDMEvHE4uhB6DcOOPK/uzMGBItUQGIbXtfg2D5NWQ89PqV04f/vZMvVq7cWQin1HVC+e+WsL/vF/5xeCBAhAAAIQgIDnBLyJcHjOGfUhAAEIdJ6AffPI68671x2cdzikg/r8hErvv/8mc6+SsKPYy4GYGRkFPSRAeN3DRitJZXs4jSdWSoLqkRjnHQ6PWKIqBCAwgYB9Z0ui6ARIrIZAywk473BE0Y0f5GqGKG+il2sfVwqHdwAkjrrSHOhREgH7zpZE0ZKgeiiG6JaHjVaiyl4kjea1N88wRl7ZVZfngKyaMPIhAAEIQKAJAs5HOJqAQp0QgAAEIFAtAXuYjby1alm7It1ph0M6pO93/DK8kzfiQuKoK4cHepRFwPfjuCwOyIkJ2MNsJJDGXNr8zekhlUSH9LQV8jobYqbYbXv/npqO2h0nEN4wyE2D/mOBAAQg4HSEg+aBAAT8JWDfMJAo6m87VqU5Ua+qyLor1+kIR1Fsvr/dUw5E+2RdlAP7QQACEIAABFwh4KzDEYZj9YW3yFJkGKNIPVXuI04HoegqCSMbAhBwhQDDb660RLV6MKRSLV+kQ6CzBAiZd7bpcxueiOjyHqLc/HzZwdkIR6ID+kIzRc95hnfawiAFC6taTMDcrbbYREyDAAQKECDCUQBanl3aMLyTx17KQsA4yiSK0heyEiAalpWU3+WcjXD4jRXtIQABCECgCAEiZEWo+bGPkxGOsMMVTBgV7D7PpWJ3G7x+mwbffSFAv/WlpdzU00TIeBmYm+0zj1ZOOhzzGMS+EIBA/QTkJsEszAZrSPAJAQjYBJwcUhl6uLamOb63KW+i1/Nz1tsczUXRFhCQY9b8tcAcTGiQgETIiJI12AAVVk2Eo0K4iIYABCAAgWIE7KgZk7sVY+jaXjgcrrUI+kDAQwLckXrYaI6rnIh0824Ox1srm3pODqm06eQ1z3s4pAmFRa/3KltrUgoCNRIwTxPYd6I1Vk9VEICAZwSci3BIzsKbn72Re0p3V7mXk0+yqM3DxXe1jbuql30Hyjs3utoLqrO7TTee1VHyS7JzDodf+NAWAhCAAASqJmBH0cjnqJp2dfKdczj+46MX6ufqe9VZ7KFkYcICAdcIcAfqWou0Vx87mkaw1992diqHg1yFyR0JNpPZsKVeAiZ3o95aqc0FApKTZv5c0Acd/CLgXITDL3xoC4FuELBD2uZuk7yNbrS9bWU5OWm2xGzf7Wia6YsMrWRj51IpxxwOSY5k+CC9g5A4ms6FtXUQME6G1GWf/Ouoe7SO5JNfvBhvlE/bfw/7Inn03jW1Yw6Hd/xQGAIQqJlA8i77k5prp7qmCTTt8DZtv8/1O+NwSI4CyZGTu5KwEUZB4FTazWSF2eI9ARO6FkM4yXvfnKUYYEeXko5fKeJzCbH7J8MrudA1VtgJhyNKiGQ4Ja0XhAf4p2bLonY6vsTpMDj4rJTAMHStayFfo1LU3ghv2smwQSX6ZxBPHojzYVNy67sTDodbSNAGAhBwmYB9l60UORwut1WVutlRt497v4mrIrcjZuHYt8YdDhPdYDhlds8QRv92meTR2aQoUZSAHaa2T+hF5VWxX/IumxyOKhj7JtPuq3YfJtrhVks27nAotUjuRo4+QS5HDlgUzU0gEabeyb07O7ScgB1dSjp+7hhu92FeEuZOu4gmDjgcbgFBGwhAAAIQSCfgqpORri1rXSPQqMMhE7UxlFKkS5A8WoQa+0wmYMLQdmh6culmt9h32eRwNNsWLtbuQx92kVsdOjX2jCWPwRZv3shJk1wOFgiUQ0DC0IlQdDliK5Eid9nmr5IKENoaAnJTK38sbhBozOFww3y0gAAEIAABCECgDgINDqnw3o0sDRyOmS6PlyR5dJwJa4oTIAxdnF2X9rSHs3zI5zD9OnoaMmopXp7YXI9txOEgd6OsBieXoyySXZRjh5rf/OyNEIEPF5EutpUrNvvbP+whaF7U0VR/asThaMpY6oUABGICdsL2z9X34g18g0DLCNh9vWWmeWVO7TkcJIuW1z9IHi2PZVckyfFn/rpiM3ZCwCZg+r98stRLoHaHo17zqA0CEEgSkNCy+Utu4RcEukHA9H97mKUbljdtZa1DKpFHKY38omm7W1N/lDza0xO6MS7ZmkYt2RD7To7QcslwEecdAfsYMMcGiaT1NGNtDodxNuzGrsfE9teC09H+Ni5ioTmZRhENIwFn35Dgs/0Ehk/VfDrJ1ijKIbNwmwXnw5Ao/7M2h6N81ZEIAQiMEoidDNkSnUxx8kcp8RsCEQFzbESTYhoqRIsNibI/a3E4zON3pnHLNgJ5KnxFvHBmaKXrvSFyMoQCx1vX+wL2ZyVgHyu20060IyvBbOUqdzh450a2hiijlBw0kXP3pXY8yAcug6kPMoxDL7raJ04fdEdHCLhHIHbaGWopt3UqdzjKVRdpEICAIWDuxHAyDBE+ITA/Aft4Yqhlfp62hEodDqIbNup6vsvBIgeJeOZEOeph3lwt5k6MRNDm2oCaXSYwfDNqyvQQWfS2nQ/j4HNezUIuvUwlDkfUMIuEd9OZV742djoUOR2V0663AoZP6uVNbRCICUQOPsMsMZG83ypxOPIqQXkIQGAyAXNnJSXsO67Je7AFAhAom4A59hhmKU62dIeD6EbxxihzT3NwyB0xT66USbYeWbaTYR5vjWpm+KSeFqAWCKQTMOdW2WpHHDnPpvOy15bucNjC+Q4BCOQjEDsaJj+DqEY+gpSGQH0EcD7ysS7V4TDent0I+dShdNkEpC1Mu+CBl023HHmmfWxpHEM2Db5DwH0C9jFrjmnOucl2K8XhMHBt4Mlq+NUkAWmXf730DRyPJhthpG5zzMhqjpsROPyEgOcEzDFtH+dK8fr0UhwOz/sG6kOgNgLmBGROSLVVTEUQgEDtBOzj3E42NU+6dO0R20IOhzlpmtYzYaP/PPmzWcWngwRMO9ntZ9Y5qK7XKsW5GGIG+RheNybKQ6AEAmnOh87pHy5dOBdncjjsC5TQ6QKYYS9o4Re7/UbblvYt3uA4GcXZsScEukTAdj6M3fa52D5Hm+1t+MzkcLTBUGyAQFUE7BOFqSPthGK28QkBCEBglIB9zkg7p7TBCZnpcIjhbTB0tHH5HRFIa9vRzp5Wpov8RrkYBvaJwqzjEwIQgEBRAvY5RRL+ZZl0/vHp/DxzSlGfjCnauOyXJCBtbv8lt7brlxzEWf9sJvb3dhHBGghAwEUC9jnH/j7r/OWSLTMjHC4piy4QyEpg0t3A6P5y4LJAAAIQ8JXArHNY1nOhbf8smXbZPN9xOPLQomyjBPIcOFUdMI0CoHIIQAACOQkUORdWda7F4cjZeBRvjkCRA6c5bakZAhCAgJ8EqjrXzszh8BMXWkMAAhCAAAQg4BIBHA6XWgNdIAABCEAAAi0lgMPR0obFLAhAAAIQgIBLBHp6rGZqmn6e5JGyDHvzszfUH7/+ohRxL9Q3C8l5Q/1Jyb7yWXSZd3+pt0wWRe2w95vRXeyipX9voi+WboSjAqWfyVLWcdekmUWP+ap1ts8lto72+qp1KCLfJ11H7WtTvx61zZff9jXDSYfDF5Bd1NPuPHXbj8NRN3HqgwAEIDAfAfuawVMq87Fk7xoJ2B23xmqpCgIQgAAESiAw0+HgJF8CZURAAAIQgAAEOk6ApNGOdwDMhwAEIAABCNRBAIejDsrUAQEIQAACEOg4ARyOjncAzIcABCAAAQjUQQCHow7K1AEBCEAAAhDoOAEcjo53AMyHAAQgAAEI1EEAh6MOytQBAQhAAAIQ6DgBHI6OdwDMhwAEIAABCNRBAIejDsrUAQEIQAACEOg4ARyOjncAzIcABCAAAQjUQQCHow7K1AEBCEAAAhDoOAEcjo53AMyHAAQgAAEI1EEAh6MOytQBAQhAAAIQ6DgBHI6OdwDMhwAEIAABCNRBAIejDsrUAQEIQAACEOg4ARyOjncAzIcABCAAAQjUQeD/AYi0x9sJMEMuAAAAAElFTkSuQmCC)↑ stdev = ↑ variability We can use the `.std()` function to get the standard deviation of the `"IMDB_Score"`.<jupyter_code>hero["IMDB_Score"].std()
std = hero["IMDB_Score"].std()
mean = hero["IMDB_Score"].mean()

print(f'Most scores are around {mean-std} to {mean+std}')<jupyter_output>Most scores are around 7.155963241175148 to 8.82973916946493<jupyter_text>The standard deviation of the IMDB scores for both Marvel and DC movies is **0.836**.   Mean Run Time of Movies and TV Series<jupyter_code># Calculate mean run time of the movies and tv series in this data using the `"RunTime"` column 

hero['RunTime'].mean()<jupyter_output><empty_output><jupyter_text>The mean run time is **40.5**.   Finding the highest grossing films (movies & TV shows) in the US using  `.sort_values()`We can sort our data set by a column by using the function called `.sort_values()`.We simply pass the name of the column we want to sort by as a parameter.- We can also set the parameter `ascending=True` or `ascending=False` if we want to sort by increasing or descreasing values    - `ascending=True`: Values are sorted from lowest to largest    - `ascending=False`: Values are sorted from largest to lowestThe format for `.sort_values()` looks like this:```    hero.sort_values("column_name", ascending=False)```<jupyter_code># Sort the hero DataFrame by `"USA_Gross"` and make `ascending=False`
hero.query('Type == "Movie" & Category == "DC"').sort_values("USA_Gross", ascending=False).iloc[[0, 1, -2, -1]]

# Sort the hero DataFrame by `"USA_Gross"` and make `ascending=False`
hero.query('Type == "Movie" & Category == "Marvel"').sort_values("USA_Gross", ascending=False).iloc[[0, 1, -2, -1]]<jupyter_output><empty_output><jupyter_text>The highest grossing film in the US was **Avengers: Infinity War**. It made **$858 million** in the US alone.   Finding the highest rated films (movies & TV shows) using `.sort_values()`<jupyter_code># Sort the hero DataFrame by `"IMDB_Score"` and make `ascending=False`
hero.sort_values("IMDB_Score", ascending=False).iloc[[0, 1, -2, -1]]<jupyter_output><empty_output><jupyter_text>The highest rated film according to the IMDB was **an episode of the Flash**. It was rated **9.6 by 11823 people**. The lowest rated film was a movie called **Catwoman with a rating of 3.4**.<jupyter_code><jupyter_output><empty_output><jupyter_text>Lowest and Highest Rated DC Movie:Highest rated DC film is the **Dark Knight** with at IMDB Rating of **9.0**.Second Highest rated DC film is the **Dark Knight Rises** with at IMDB Rating of **8.4**.Lowest rated DC film is the **Catwoman** with at IMDB Rating of **3.4**.Second Lowest rated DC film is the **Batman & Robin** with at IMDB Rating of **3.8**.<jupyter_code># Sort the hero DataFrame by `"IMDB_Score"` and make `ascending=False`
hero.query('Type == "Movie" & Category == "DC"').sort_values("IMDB_Score", ascending=False).iloc[[0, 1, -2, -1]]<jupyter_output><empty_output><jupyter_text>Lowest and Highest Rated Marvel Movie:Highest rated Marvel film is the **Avengers: Endgame** with at IMDB Rating of **8.4**.Second highest rated Marvel film is the **Avengers: Infinity War** with at IMDB Rating of **8.4**.Lowest rated Marvel film is the **Fantastic Four: Rise of Silver Surfer** with at IMDB Rating of **5.6**.Second lowest rated Marvel film is the **Punisher: War Zone** with at IMDB Rating of **5.9**.<jupyter_code># Sort the hero DataFrame by `"IMDB_Score"` and make `ascending=False`
hero.query('Type == "Movie" & Category == "Marvel"').sort_values("IMDB_Score", ascending=False).iloc[[0, 1, -2, -1]]<jupyter_output><empty_output><jupyter_text>Comparing DC vs Marvel cinematic universesThe battle of the cinematic universes begins! Who will reign supreme?  Figuring out the number of Marvel and DC films using `.value_counts()`The first order of business is figuring out how many of our datapoints are from either Marvel or DC? Are there more Marvel or DC films? To do this we need to use `.value_counts()` function. It is important to note that this function can only be applied on a single column of a DataFrameFor example:```    hero["column_name"].value_counts()```<jupyter_code># Figure out how many datapoints are from Marvel or the DC cinematic universe.  Use the column name "Category"
hero["Category"].value_counts()<jupyter_output><empty_output><jupyter_text>Our dataset has **116** films from the DC universe and **276** films from the Marvel universe.|<jupyter_code>hero.groupby(["Category", "Type"]).describe()<jupyter_output><empty_output><jupyter_text>Figuring out if Marvel or DC films are more highly rated using `.groupby()`Which movies are more highly rate on IMDB? DC or Marvel?To answer this question, we need to use the `.groupby()` method. We can select one our multiple columns to group the data by, and then we can aggregate that data to find out information about mean, min, max, standard deviation, sum, etc.The format that `.groupby()` uses is as follows:```hero.groupby("column_name").aggregate_function()```Where `"column_name"` is the name of the column and `.aggregate_function()` is the a specific aggregate function like `.mean()`, `.sum()`, `.max()`, `min()`.<jupyter_code># Figure out if DC or Marvel films have a greater average IMDB_Score, Metascore, Votes, USA_Gross.
# Remember the column that tells us is the film is DC or Marvel is called `"Category"``

hero.groupby("Category").mean()
hero.groupby("Category").mean()<jupyter_output><empty_output><jupyter_text>**DC** has a higher mean IMDB Score than **Marvel**.  And, **Marvel** has a higher mean Metacritic score than **DC**. In general, **Marvel** films tend to gross more in the US than **DC** films.   Grouping by both `"Category"` (i.e., DC vs Marvel) and `"Type"` (i.e., Movie vs Series)Our dataset also includes information if the film is a stand-alone movie (only one iteration of it) or if it is a series (multiple episodes).To get a break down of our data by both `"Category"` and `"Type"`, we need to pass two columns into `.groupby()`The format should be as follows: ```hero.groupby(["column_name_1", "column_name_2"]).aggregate_function()```<jupyter_code># Calculate the mean for the numerical data by grouping by "Category" and "Type"
hero.groupby(["Type", "Category"]).mean()<jupyter_output><empty_output><jupyter_text>When comparing **movies**, Marvel had a higher IMDB score than DC.  When comparing **TV Series**, DC had a higher IMDB rating than Marvel.  Looking films made after 2005 using `.query()` Everyone knows that there was a resurgence in superhero movies in the mid to late 2000s, what if we only look at data after 2005 using`.query()` to apply some sort of condition onto the data.  Usually the format is you set some sort of condition based on a column of your DataFrame.Some examples of queries include the following:- Only include films that were before the year 1995: `hero.query("Year_Start < 1995")`- Only include films that were directed by <NAME>: `hero.query("Director == 'TimBurton'")`- Only include films that have a IMDB score of greater or equal to 8 **and** were made after 2010: `hero.query("IMDB_Score >= 8 & Year_Start > 2010")`- Only include films that have a age rating of "R" **or** "M": `hero.query("Rating == 'R' | Rating == 'M'")`<jupyter_code># Query that hero data for films where the Year_Start is > 2005

hero_2005 = hero.query("Year_Start > 2005 & Votes > 1000 ") # Replace the year 
hero_2005
# Calculate the mean for the data only including 2005 and beyond by grouping by "Category" and "Type"
hero_2005.groupby(["Type", "Category"]).mean()<jupyter_output><empty_output><jupyter_text>When we subset/query our data to only include movies and series after 2015 and films that were rated by more than 1000, our results change slightly.  For both movies and TV series, Marvel seems to score higher on both IMBD and Metacritic, have more votes, and gross more in the US.  Graphing data  Why is visualizing data important?In data sciences, one of the most effective ways to get insights on data is to visualize it. Data can be visualized using a variety of plots, which make the data easy to understand and to convey general trends and key elements in the data.  SeabornIn python, there are a variety of plotting libraries. `matplotlib`, is the core plotting and visualization library that comes pre-packaged with python. It is extremely flexable, but is known for “tries to make easy things easy and hard things possible”.`seaborn`, which is built on top of `matplotlib`, is the library that we will be learning this week. It works well with pandas DataFrames and makes a well-defined set of complex graphs easy to do.  Import Seaborn To import `seaborn` we use the import command. `seaborn` is typically shorten to `sns`. ```  import seaborn as sns```<jupyter_code># Import seaborn as its shortened form, sns
import seaborn as sns

# Set seaborn to the dark  grid stylle(dark background)
sns.set_style("darkgrid") # The default is usually "lightgrid"<jupyter_output><empty_output><jupyter_text>Common Seaborn Parameters`seaborn` is built to be compatible with DataFrames. So we need to pass the DataFrame as a parameter as well as the column names that we hope to plot.Below are common `seaborn`  parameters:*   `data` - name of the DataFrame*   `x` - column name corresponding to the x-axis*   `y` - column name corresponding to the y-axis*   `hue` - column name corresponding to the colour of the plot shapes*   `estimator` - function to estimate data (e.g., `np.mean`, `np.std`, `np.max`, `np.min`)  Bar charts: Plotting aggregate data using `sns.barplot()`Bar plots are used for univariate or categorical data. They can be used to visualize the count of values of each category or to visualize some sort of aggregate value (e.g., mean, standard deviation, min, max) of catergorical data.`sns.barplot()` is the name of the function that creates bar plots. We can access the [documentation](https://seaborn.pydata.org/generated/seaborn.barplot.html) on the website. For example:```sns.barplot(    data=hero,    x='x_axis_column_name',    y='y_axis_column_name',    hue='color_column_name',    estimator=np.mean,)```  Barplot of Category, Type and mean IMBD_Score- The x-axis is `"Type"` which tells us if it is a series or movie- The y-axis is `"IMDB_Score"`- The colour/hue should be `"Category"`, which tells if the film is DC or Marvel<jupyter_code># Plot bar graph of Mean IMBD_score

sns.barplot(
    data=hero,
    x='Type',
    y='IMDB_Score',
    hue='Category',
    estimator=np.mean,
)<jupyter_output><empty_output><jupyter_text>Interpretation: This graph tells us Marvel movies are better than DC movies. DC TV shows are better than Marvel TV shows   Barplot of Category, Type and mean RunTime- The x-axis is `"Type"` which tells us if it is a series or movie- The y-axis is `"RunTime"`- The colour/hue should be `"Category"`, which tells if the film is DC or Marvel<jupyter_code># Plot bar graph of mean RunTime

sns.barplot(
    data=hero,
    x='Type',
    y='RunTime',
    hue='Category',
    estimator=np.mean,
)<jupyter_output><empty_output><jupyter_text>Interpretation: The first finding is that movies have a longer run time than TV shows. For Movies, Marvel and DC movies have the same run time. For TV shows, Marvel TV shows are longer than DC tv shows.   Barplot of Cateogry, Type and mean Metascore- The x-axis is `"Type"` which tells us if it is a series or movie- The y-axis is `"Metascore"`- The colour/hue should be `"Category"`, which tells if the film is DC or Marvel<jupyter_code># Plot bar graph of mean Metascore

sns.barplot(
    data=hero,
    x='Type',
    y='Metascore',
    hue='Category',
    estimator=np.mean,
)<jupyter_output><empty_output><jupyter_text>Interpretation: Marvel is way higher than DC in the movie rankings (Metacritic). The site does not collect TV show ratings.  Barplot of Category and  total Votes- The x-axis is `"Category"`, which tells if the film is DC or Marvel- The y-axis is `"Votes"`- The colour/hue should be `"Type"` which tells us if it is a series or movie- Make sure the esitmator is `np.sum`<jupyter_code># Plot bar graph of total Votes
sns.barplot(
    data=hero,
    x='Type',
    y='Votes',
    hue='Category',
    estimator=np.sum,
)<jupyter_output><empty_output><jupyter_text>Interpretion: Marvel movies get more votes than TV shows(because they are fun). Marvel has more total votes than DC when it comes to movies. DC has a littlemore votes than TV shows when it comes to TV showa.   Barplot of Category and  mean USA_Gross- The x-axis is `"Category"`, which tells if the film is DC or Marvel- The y-axis is `"USA_Gross"`<jupyter_code># Plot bar graph of mean USA_Gross
sns.barplot(
    data=hero,
    x='Category',
    y='USA_Gross',
    estimator=np.mean,
)<jupyter_output><empty_output><jupyter_text>Interpretation: Marvel movies make more than DC movies  Barplot of Genre, Type and  mean RunTime- The x-axis is `"Genre"` which tells us if it is a series or movie- The y-axis is `"RunTime"`- The colour/hue should be `"Category"`, which tells if the film is DC or Marvel<jupyter_code>sns.barplot(
    data=hero,
    x='Genre',
    y='RunTime',
    hue='Category',
    estimator=np.mean,
)<jupyter_output><empty_output><jupyter_text>Interpretation:  Barplot of Rating and  mean IMBD_Score- The x-axis is `"Rating"`, which tells if the film is DC or Marvel- The y-axis is `"IMDB_Score"`- The colour/hue should be `"Category"` which tells us if it is a series or movie<jupyter_code>sns.barplot(
    data=hero,
    x='Rating',
    y='IMDB_Score',
    hue='Category',
    estimator=np.mean,
)<jupyter_output><empty_output><jupyter_text>Interpretation: It shows us the different IMDB Scores for different age Ratings it compares Marvel and DC. for PG-13, Marvel Scores higher then DC, but for R, Rated movies DC Scores higher, for example, JOKER is an R Rated movie from DC in general Marvel movies tend to be more family friendly.   Scatter plots: Comparing two continuous variables using `sns.scatterplot()`Scatter plots are used for **bivariate data (continous, continuous)**. There are ideal for understanding the relationship between two continuous variables, and help to visualize the degree of correlation between two variables. `sns.scatterplot()` is the name of the function that creates scatter plots. More documentation about it can be found at [here](https://seaborn.pydata.org/generated/seaborn.scatterplot.html). For example:```sns.scatterplot(    data=hero,    x='x_axis_column_name',    y='y_axis_column_name',    hue='color_column_name',    size='size_column_name',)```  Relationship between Year_Start and IMDB_Score- The x-axis is `"Year_Start"`, which tells if the film is DC or Marvel- The y-axis is `"IMDB_Score"`- The colour/hue should be `"Category"` which tells us if it is a series or movie<jupyter_code>sns.scatterplot(
    data=hero,
    x='Year_Start',
    y='IMDB_Score',
    hue='Category'
)<jupyter_output><empty_output><jupyter_text>Interpretation: they're everywhere (DC films are way older than Marvel films) in the early 2000's, marvel released tonnes of low rated films. in 2010 & beyond, we get more high rated films!   Relationship between IMBD and Metacritic Score- The x-axis is `"IMBD_Score"`, which tells if the film is DC or Marvel- The y-axis is `"Metascore"`- The colour/hue should be `"Category"` which tells us if it is a series or movie<jupyter_code>sns.scatterplot(
    data=hero,
    x='IMDB_Score',
    y='Metascore',
    hue='Category',
)<jupyter_output><empty_output><jupyter_text>Interpretation:IMDB & metacritic scores are positively correlated. If a film is high on Metacrtitic, you score high on IMDB & vice versa!  Relationship between IMBD Score and USA_Gross- The x-axis is `"IMDB_Score"`, which tells if the film is DC or Marvel- The y-axis is `"USA_Gross"`- The colour/hue should be `"Category"` which tells us if it is a series or movie- The size should be the `"RunTime"`<jupyter_code>sns.scatterplot(
    data=hero,
    x='IMDB_Score',
    y='USA_Gross',
    hue='Category',
     size='RunTime'
)<jupyter_output><empty_output><jupyter_text>Interpretation:Higher ratings are correlated to bigger profit. They also have a longer run time.  Distribution PlotsSometimes we want to see how are data are distributed or spread out. This is where distribution plots come in handy for **univariate, continuous** data.There are several ways we can visualize the distribution of data, but here are four popular plots: 1. **box plots**: used to determine the variation in your continuous data2. **histograms**: used to visualize the frequency of our continuous variable within each predefined bins3.  **density plots**: a version of histogram smoothened by Kernel density estimation.4. **violin plots**: combined version of a box plot and density plot  Box Plots (`sns.boxplot`)Box plots are used to determine the variation in continuous data. In particular, they are often used to determine outliers.A box plot is comprised of **(1) the box** and **(2) the whiskers**.The **box part** of the plot is made up of three parts:1. **Quartile 1 (Q1):** 25th percentile2. **Quartile 2 (Q2)/Median:** The median or 50th percentile3. **Quartile 3 (Q3):** 75th percentileThe **Interquantile Range (IQR)** is the difference between Q3 and Q1. It gives us a sense of the spread of the data.The **whisker part** of the plot represents what is consider the borders (min and max) for outliers.1. **min:** any values greater than this range are considered outliers  - min = Q3 + 1.5*IQR2. **max:** any values less than this range are considered outliers  - max = Q1 + 1.5*IQR![Box Plot](https://i.imgur.com/RcdP4kr.png)`sns.boxplot` is the name of the function that creates these types of plots. More documentation about it can be found [here](https://seaborn.pydata.org/generated/seaborn.boxplot.html)Example syntax:```sns.boxplot(    data=hero,    x='x_axis_column_name',    y='y_axis_column_name',    hue='color_column_name',)```  Distribution of US_Gross- The x-axis is `"Category"`, which tells if the film is DC or Marvel- The y-axis is `"USA_Gross"`- The colour/hue should be `"Category"` which tells us if it is a series or movie`<jupyter_code>sns.boxplot(
    data=hero,
    x='Type',
    y='USA_Gross',
    hue="Category",
)<jupyter_output><empty_output><jupyter_text>Interpretation: The median USA gross amount is higher for Marvel than DC. The marvel movies has a greater spread than DC.  Histograms (`sns.histplot`)Histograms are used to visualize the frequency of continuous variables in what are known as **bins**.**Bins** A bin is an interval into which the dataset is divided. For example, if we have a range of values from 1-100, we can create bins of intervals of 20. So, we could have the first bin contain values from 1-20, the second 21-40, third 41-60, and so on until 100. This gives us an idea of how many of the values in a given data set fall within a certain range of values, i.e., the distribution of the data. Bins are usually equal in size, but do not necessarily have to be.![histogram bins](https://i.imgur.com/hDfK8vE.png)**Histogram Distributions**Histograms give us greater insight into how our data are distributed. Here are some common distributions. Note that the peaks are revered to local maxima. ![histogram distributions](https://i.imgur.com/ypLJkKJ.png)`sns.histplot` is the name of the function that creates these types of plots. More documentation about it can be found [here](https://seaborn.pydata.org/generated/seaborn.histplot.html)Example syntax:```sns.histplot(    data=hero,    x='x_axis_column_name',     hue='color_column_name',)```There is no `y` parameter, beause the y-axis is by default the count.  Distribution of RunTime- The x-axis is `"Category"`, which tells if the film is DC or Marvel- The y-axis is `"USA_Gross"`- The colour/hue should be `"Category"` which tells us if it is a series or movie`<jupyter_code>sns.histplot(
    data=hero,
    x='RunTime',
    hue="Category",
)<jupyter_output><empty_output><jupyter_text>Interpretation: Run Time has a bimodal distribution. The different peak represent different types of content. We have 25 min TV shows and 45 min TV shows - as well as longer movies.   Density Plots (`sns.kdeplot`)Density plots are similar to histograms but they use a kernal density estimate (usually gaussain) to smooth the plot.`sns.kdeplot` is the name of the function that creates these types of plots. More documentation about it can be found [here](https://seaborn.pydata.org/generated/seaborn.kdeplot.html)Example syntax:```sns.kdeplot(    data=hero,    x='x_axis_column_name',    hue='color_column_name',)```There is no `y` parameter, beause the y-axis is by default the kernal density estimate.  Distribution of Votes- The x-axis is `"Category"`, which tells if the film is DC or Marvel- The y-axis is `"Votes"`- The colour/hue should be `"Category"` which tells us if it is a series or movie`<jupyter_code>sns.kdeplot(
    data=hero,
    x='Votes',
    hue='Category',
)<jupyter_output><empty_output><jupyter_text>Interpretation: Most of our data points have very few Votes   Violin Plots (`sns.violinplot`)Violin plots are like if box plots and density plots had a baby. There great for showing distributions across different levels of categorical variables (e.g., like place of origin).The middle bar of the box plot gives similar information as the boxplot. Whereas the "violin" portion of the box plot encodes information about distribution and skewness. ![](https://i.imgur.com/PcGY1TK.png)`sns.violinplot` is the name of the function that creates these types of plots. More documentation about it can be found [here](https://seaborn.pydata.org/generated/seaborn.violinplot.html)Example syntax:```sns.violinplot(    data=hero,    x='x_axis_column_name',    y='y_axis_column_name',    hue='color_column_name',)``` Interpretation:  Distribution of IMDB_Score- The x-axis is `"Category"`, which tells if the film is DC or Marvel- The y-axis is `"IMDB_Score"`- The colour/hue should be `"Category"` which tells us if it is a series or movie`<jupyter_code>sns.violinplot(
    data=hero,
    x='Type',
    y='IMDB_Score',
    hue='Category',
)<jupyter_output><empty_output><jupyter_text>Interpretation: For movies, marvel movies has a more narrow distribution when compared to DC movies. When looking at TV Shows, Marvel movies have a biomodal distribution -- there are movies that are highly rated and not so highly rated. DC TV shows have a unimodal modal distribution and are consistently good.<jupyter_code><jupyter_output><empty_output><jupyter_text>Machine Learning What is machine learning?Machine learning isn't really a well defined term. There are, however, two defintions that are well accepted <NAME> (1959)    Field of study that gives computers the ability to learn without being explicitly programmed <NAME> (1999)    A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience EWatch this quick [video](https://www.youtube.com/watch?v=f_uwKZIAeM0) about machine learning. Types of Machine LearningThere are two major branches or machine learning that you different types of learning alorithms. 1. Supervised learningWe teach a computer how to do something, then let is use its new found knowledge to do it.- supervised learning requires **labelled** data- e.g., regression, classification 2. Unsupervised learningWe let the computer learn how to learn to do somethings, and use its knowledge to getermine structure and patterns in the data.- unsupervised learning requested **unlabelled** data- e.g., clustering Regression  Importing Machine Learning LibrariesThere are some specialized libraries in python that we can use for machine learning. sklearn[sklearn](https://scikit-learn.org/stable/) is a module of simple and efficient tools for machine learning problems like classification, regression and clustering. To import sklearn we use an import statement```    import sklearn```In particular, we want the `train_test_split` function, so we will import it this way```    from sklearn.model_selection import train_test_split``` statsmodels [statsmodels](https://www.statsmodels.org/stable/index.html) is a python module that provides class functions for many statistical models. Specifically, we are going to buse the [statsmodels.formula.api](https://www.statsmodels.org/stable/api.html?highlight=formula%20apistatsmodels-formula-api). This to long to type out, so we usually import and use the short for `smf`To import statsmodels.formula.api we use and import statement```    import statsmodels.formula.api as smf ```<jupyter_code># Lets import sklearn, along with our other data science packages

import numpy as np # For math- and matrix-based functions
import pandas as pd # For DataFrame functionality
import seaborn as sns # For plotting/ data visualization
from sklearn.model_selection import train_test_split # For the train_test_split_function
import statsmodels.formula.api as smf # Stats models for<jupyter_output><empty_output><jupyter_text>Regression AnalysisRegression analysis attempts to explain the relationship between variables. We want to see if we can predict or explain the **dependent variable** using one or more **explanatory variables**. In linear regression, we assume that the dependent variable can be somewhat explained by a combination of expanatory variables.A very simple example is one where the depedent variable is height and the explantatory variable is age. The age of a person does a good job of explaining their height for kids under 16. We can also think of **explanatory variables** as **features** in our linear model, and the **dependent variable** as the **target**.The pipeline is as follows:![image](https://i.imgur.com/tu0nVzO.png)  Lets pick a model: **Dependent/Target Variable** = `"IMDB_Score"`**Exploratory Variables/Features** = `["USA_Gross",  "RunTime", "Metascore"]`<jupyter_code># Define Variables
target = ["IMDB_Score"]
features = ["USA_Gross",  "RunTime", "Metascore", "Category"]

# Drop Data

hero_subset = hero[target + features].dropna()
hero_subset<jupyter_output><empty_output><jupyter_text>Lets plot the relationship between our target and each of our features using `sns.regplot()``sns.regplot()` plots the data as well as a linear regression fit. The syntax of this function is the same as all the other seaborn plots, but you can also take a look at the [documentation](https://seaborn.pydata.org/generated/seaborn.regplot.html)  Feature_1 vs. Target<jupyter_code>sns.regplot(
    data = hero_subset,
    x = "USA_Gross",
    y = "IMDB_Score"
    
)<jupyter_output><empty_output><jupyter_text>Feature_2 vs. Target<jupyter_code>sns.regplot(
    data = hero_subset,
    x = "RunTime",
    y = "IMDB_Score"
)<jupyter_output><empty_output><jupyter_text>Feature_3 vs. Target<jupyter_code>sns.regplot(
    data = hero_subset,
    x = "Metascore",
    y = "IMDB_Score"
)<jupyter_output><empty_output><jupyter_text>Splitting our data using `train_test_split()`Whenever we do machine learning, we always want to reserve data to **train** the model as well as data to **test** the model.  By default, we usually use 75% (.75) of our data for training and 25% (.25) of our data for testing.  Testing our data allow us to see how good of a job our linear regression model does at predicting the target variable. We use `train_test_split()` to shuffle and randomly split our data into the train dataset and the test dataset.Before we can use it, be sure to properly import it```    from sklearn.model_selection import train_test_split```The `train_test_split()` is formatted as follows:```    df_train, df_test = train_test_split(df[target], df[features], test_size=.25, train_size=.75, random_state=1)```- Where:    - `df_train`: the split dataset for training    - `df_test`: the split dataset for testing    - `test_size`: the proportion of the data set to include in the testing set    - `train_size`: the proportion of the data set to include in the training set    - `random_state`: controls the shuffling of the data, using an int allows for reproducible output<jupyter_code># Split the the car data into testing and training sets
hero_train, hero_test = train_test_split(hero_subset, test_size=.25, train_size=.75, random_state=1)

# The train dataset has 47 rows (notice how the indexes are randomly suffled)
hero_train
# The test dataset has 16 rows (notice how the indexes are randomly suffled)
hero_test<jupyter_output><empty_output><jupyter_text>Creating and Training our Linear Regression Machine Learning Model using `smf.ols()``smf.ols()` is the function we use to create a linear regression model.  OLS stands for ordinary least square models. It requires the following syntax```    model = smf.ols(formula="target ~ feature_1 + feature_2 + feature_3 + ... + feature_n", data=df).fit()```- where:    - `formula`: the formula of the model as string in the format of `"target ~ feature_1 + feature_2 + feature_3 + ... + feature_n"`        - The target and features are separated by `~`        - The features are seprated by `+`    - `data`: the name of your DataFrame    - `.fit()`: is a function that trains the model to fit the dataThis is the syntax of  `.predict()`:- If we pass nothing into the model, it will return the prediction based on what is was trained on: ```        model.predict()```- If we our pass our testing_data or other new data, it will return the prediction for the testing/new data|```        model.predict(df_test)```<jupyter_code># Lets create and train our model
hero_model = smf.ols(formula="IMDB_Score ~ USA_Gross + RunTime + Metascore + Category", data=hero_train,).fit()
hero_model.summary()
# Get prediciton from the linear model and make it a new colunn on the training DataFrame
hero_train['IMDB_Score_Prediction'] = hero_model.predict()

# Get the difference between the predicted and observed/actual value
hero_train['IMDB_Score_Difference'] = hero_train['IMDB_Score_Prediction']-hero_train['IMDB_Score']

# Show DataFrame
hero_train<jupyter_output>/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """<jupyter_text>Evaluating the Fit of the Model: Root Mean Squared Error (RMSE)Root Mean Squared Error (RMSE) is used to assess how well a regression model first the dataset. It tells the average distance between the predicted values from the model and the actual/observed values. The formula for RMSE is as follows:$$ RMSE =  \sqrt{\Sigma \frac{(P_{i} - O_{i})^2}{n}} $$Where:- $RMSE$ is root mean squared error- $P$ is predicted value- $O$ is observed/actual value- $i$ is a single instance- $n$ is the sample sizeA lower RMSE indicates a better fitting model.↓ RMSE = ↑ Fit<jupyter_code># Get sample size (n)
n = len(hero)

# Calculate RMSE
rmse_train = np.sqrt((hero_train['IMDB_Score_Difference'].pow(2).sum())/n)

# Print the RMSE
print(f'The RMSE of the training data is {rmse_train}')<jupyter_output>The RMSE of the training data is 0.12543098085424925<jupyter_text>Trying the model on the testing setWe need to test how our model performs on data it has never seen before. To do this we must again use `.predict()` Recall: This is the syntax of  `.predict()`:- If we pass nothing into the model, it will return the prediction based on what is was trained on: ```        model.predict()```- If we our pass our testing_data or other new data, it will return the prediction for the testing/new data|```        model.predict(df_test)```<jupyter_code># Get prediciton from the linear model and make it a new colunn on the training DataFrame
hero_test['IMDB_Score_Prediction'] = hero_model.predict(hero_test) # notice how we are passing the test data now, NOT train

# Get the difference between the predicted and observed/actual value
hero_test['IMDB_Score_Difference'] = hero_test['IMDB_Score_Prediction']- hero_test['IMDB_Score']

# Show DataFrame
hero_test
# Get sample size (n)
n = len(hero_test)

# Calculate RMSE
rmse_test = np.sqrt((hero_test['IMDB_Score_Difference'].pow(2).sum())/n)

# Print the RMSE
print(f'The RMSE of the testing data is {rmse_test}')<jupyter_output><empty_output><jupyter_text>Using the model to predict new scenerios. The RMSE values of the **machine learning model to predict IMDB_Scores from Votes, Metascore and USA_Gross** are pretty low. On the training set, the model had an RMSE value of  0.125- So on average, the model got the IMDB score correct within a range of 0.125 rating points.  So instead of guessing a rating of 6, our model might guess 6.125 instead.  On the testing set, the model had an RMSE value of  0.745- So on average, the model got the IMDB score was typically off by 0.745 points. So instead of guessing a score of 8.745, our model might guess 8 instead. - We expect the model to perform worse on the the testing set, because it has not yet seen that data- It's like taking a test where you've seen all the answers versus taking a test where you haven't yet seen the answers. Confidence in model and sample sizeBeing less than one rating point off, doesn't seem to bad for our model. If we had a larger sample size (i.e., more data points), the machine learning model can better at learning and our error would be less. But, for now lets say we have confidence in this model  Create a new data frame of our predicted dataFirst we must create a DataFrame that has the the exact same column names (case-senstive) as our feature columns. We will be creating a DataFrame using the dictionary method. Please read the [documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas).DataFrame.from_dict.html) for more details.If we have confidence in our model we can use machine learning to predict the IMDB_score for new scenerios Lets say we want estimate the IMDB rating (`"IMDB_Score"`) of for two movies that havent yet been added to IMDB website.**Attack of the NarWolves**- USA_Gross of  $ 600 M- RunTime of 120 minutes- Metascore of 80 - DC**Teeth or Horns?**- USA_Gross of  $ 400 M- RunTime of 90 minutes- Metascore of 60 - Marvel<jupyter_code># Create new_predictio DataFrame
new_prediction = pd.DataFrame({
    "Name": ["Attack of the NarWolves", "Teeth or Horns?"],
    "USA_Gross": [600, 400],
    "RunTime": [120, 90],
    "Metascore": [80, 60],
    "Category": ["DC", "Marvel"],
})

# Show DataFrame
new_prediction
# Lets use our model to predict the scores
new_prediction["IMDB_Score"] = hero_model.predict(new_prediction)

# Show DataFrame
new_prediction<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07479659763313609, 'fraction_numerical': 0.04970414201183432, 'mean_word_length': 7.176746295736317, 'pattern_counts': {'":': 6, '<': 169, '<?xml version=': 0, '>': 173, 'https://': 22, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '519662', 'license': 'MIT', 'n_tokens_mistral': 24528, 'n_tokens_neox': 22891, 'n_words': 4804, 'path': 'group_02/group_2_analysis.ipynb', 'repo_name': 'nguyenjenny/spark_shared_repo', 'size': '54095'}