<jupyter_start><jupyter_text>Pay DayUp until now, we have been building up our competencies in Python so that we have an appropriate (if not fully formed) sense of what the language looks like, its various components and their functionality, and how it treats its objects. It is only recently, however, that have we started working with the type of datasets that you would use in your own research. Even then, we have focused on how to clean your data to enable statistical analysis, without actually performing any statistical analysis. Given that many of you are likely experienced users in SPSS, this may have been a frustrating experience - even after several weeks, you are still yet to run the type of analyses that you could do in an afternoon with SPSS. This frustration is one of the reasons that people persist with programs like SPSS - there is less preparation work needed to be able to run statistical analysis, because you do not need to know how SPSS really works to produce an output, you just need to know where to point and click. The benefit of learning Python, however, is that you can make use of its flexibility to perform scalable, powerful, and transparent statistical analyses in the future. Additionally, it will also make you more powerful in the marketplace. As psychologists, you are or will be trained to communicate effectively. And while some of our best friends are programmers, a lot of those folks are not effective communicators. So if you can combine decent programming (and by decent, I mean you can run the statistical analyses you need to run, not that you need to be a master hacker) with effective communication skills, then you will be deadly (Recommended reading: Cal Newport's - So Good They Can't Ignore You). In short, Python (and R, but thats for later) will provide a return on investments that SPSS will never be able to achieve (if you persist with it, that is). So today, we want to starting paying out some of those dividends on your investment. In this workshop, we will perform various types of statistical analysis; we are going to learn how to sort our data, and describing our data (e.g., descriptive statistics). Load our Dataset and our PackagesLet's get our packages and our dataframe ready.<jupyter_code>#Import packages pandas, numpy and pathlib

# I have made adjustements to the df_movie dataframe to add more films and create an equal amount of films that have either a male/female lead.

director = ['<NAME>', '<NAME>', '<NAME>', '<NAME>', 'Coen Brothers', '<NAME>', '<NAME>', 'Coen Brothers', '<NAME>', '<NAME>', '<NAME>', '<NAME>', 'Annihilation', '<NAME>']
movie = ['The Thing', 'Drive', 'The Thing', 'Arrival', 'No Country for Old Men', 'Shrek 2', 'Hot Fuzz', 'Fargo', 'Alien', 'Lady Bird', 'Terminator 2', '10 Cloverfield Lane', 'Annihilation', 'Blade Runner 2049']
genre = ['Horror', 'Action', 'Horror', 'Sci-Fi', 'Drama', 'Comedy', 'Comedy', 'Dark Comedy', 'Horror', 'Drama', 'Action', 'Horror', 'Horror', 'Sci-Fi']
year = ['1982', '2011', '2011', '2016', '2007', '2004', '2007', '1996', '1979', '2017', '1992', '2016', '2018', '2017']
imdb = [82.0, 78.0, 62.0, 79.0, 82.0, 73.0, 78.0, 81.0, 85.0, 74.0, 86.0, 72.0, 68.00, 80.0]
rt_critics = [82.0, 93.0, 34.0, 94.0, 93.0, 89.0, 91.0, 94.0, 98.0, 99.0, 93.0, 90.0, 88.0, 88.0]
rt_fans = [92.0, 79.0, 42.0, 82.0, 86.0, 69.0, 89.0, 93.0, 94.0, 79.0, 95.0, 79.0, 66.0, 81.0]
gender_of_lead = ['Male', 'Male', 'Female', 'Male', 'Male', 'Male', 'Male', 'Female', 'Female', 'Female', 'Female', 'Female', 'Female', 'Male']

df = pd.DataFrame(# opening brackets but moving to new line for readability, note the uppercase D and F in the call
    list(zip(director, movie, genre, year, imdb, rt_critics, rt_fans, gender_of_lead)), #first argument note the comma at the end of it, this is a couple of nested functions
    columns = ['Director', 'Movie Title', 'Genre', 'Year of Release', 'ImdB Score', "Rotten Tomatoes' Critic Scores", "Rotten Tomatoes' Fan Scores", 'Gender of Lead']# second argument , passing a list to the columns arguments
    )# closing the first pair of brackets to complete the function call

df.head()<jupyter_output><empty_output><jupyter_text>Sorting our DataSorting is an important tool that can use to get an intuitive feel for your dataframe. In this section, we are going to look at how to sort by specific columns, by multiple columns, and how to sort from the smallest-largest or largest-smallest values within columns. Let's say we want to sort our dataframe by year of release - we can do that by running the following code cell.<jupyter_code>df.sort_values(by = "Year of Release")

#we apply the sort_values function onto our dataframe, df. 
#We use "by" to indicate the column that we want to sort our dataframe by.<jupyter_output><empty_output><jupyter_text>This returns our dataframe, but ordered by the year of the release. In this case, the dataframe is sorted in ascending order: it beings with the smallest value first and finishes with the largest value. If we want to order the dataframe in descending order, we need to add an argument (i.e., instruction) to our sort_values function. Run the following code cell and compare it to the one above.<jupyter_code>df.sort_values(by = "Year of Release", ascending= False)<jupyter_output><empty_output><jupyter_text>The ascending arguments specifies whether we want our data to be ascending. There are two values that we can pass to this argument: `True` (i.e., Yes, please sort our data from smallest to largest) or `False` (i.e., No, please sort our data from largest to smallest instead). If we do not add `ascending = False`, then sort_values orders by ascending by default. Now if you look at your dataframe, you may have noticed that the values in the `Year of Release` column are strings. If the strings are numerical, then ascending/descending works the same as if the values were integer or float. If the values are alphabetically, then ascending/descending still works, but it does so in an alphabetically order (i.e., A ('smallest value') to B to C....to Y to Z('largest value')). If a column contains both alphabetical and numerical values, numerical values come first. In the following code cell, sort the data by the `Movie Title` column in ascending order.<jupyter_code>#sort by movie column dataframe - ascending<jupyter_output><empty_output><jupyter_text>You can see that 10 Cloverfield Lane comes before Alien which comes before Annihiliation (Al comes before Ah), and that The Thing (x2) comes last. Now try sort the dataframe by the same column, but this time do so in descending order.<jupyter_code>#sort by movie column dataframe - descending<jupyter_output><empty_output><jupyter_text>Great. So we can see that `The Thing` comes in the first two rows of our sorted dataframe. John Carpenter's vastly superior (and one of the greatest horror movies ever made) 1982 version of the `The Thing` rightly comes ahead of the 2011 CGI crappy knock-off, but, Python has not decided not based on wonderful taste in film, but because in the original dataframe, the 1982 version of `The Thing` is entered before the 2011 version (or more technically, it has a lower index). Therefore, even though they have the same value, Python uses the original index to order between identical values. In cases where we have identical values in one column, we can sort by multiple columns to arrange the data in a more specific manner.<jupyter_code>df.sort_values(by = ["Movie Title", "Year of Release"], ascending= False)

# we pass a list to the "by" argument. The list enables us to say:

# by = [sort_by_this_value_first, then_this_value1, then_this_value2.......n]<jupyter_output><empty_output><jupyter_text>We can do a similiar thing with the ascending argument. Maybe we want to sort `Movie Title` by ascending order, but the year of release in descending order. In this case, both The Things should be at the bottom of the dataframe, but the 2011 version should come before the 1982 version.Run the code below to see what happens.<jupyter_code>df.sort_values(by = ["Movie Title", "Year of Release"], ascending= [True, False])<jupyter_output><empty_output><jupyter_text>While all the above commands sort our dataframe, it is a once-off output. If we want to save the ordering of our dataframe, we need to assign the output of our sorting to a variable. Additionally, we would want our new ordered dataframe to adjust its index accordingly (so that whatever is first in the new dataframe has a new index of 0, rather than its old index). To do that, we can use the `reset_index` method.<jupyter_code>df_new = df.sort_values(by = ["Movie Title", "Year of Release"], ascending= [True, False])

df_new = df_new.reset_index(drop=True)

df_new
#### In this cell, sort the movies by the genre of the movie (either in ascending or descending order)


#In this cell, sort the movies by genre of the movies first, then by Imdb Score. 
#Make sure to sort one column by ascending order, and the other by descending.



#Save the sorted dataframe to df_new<jupyter_output><empty_output><jupyter_text>Good job, now that we are sorted with dealing with dealing with dataframes, let's actually go about describing the data the data we have in those dataframes.   Descriptive Statistics Frequency AnalysisOne of the first things we do with our dataframes in SPSS is click "Analyse - Descriptive - Frequency Analysis". To do the equivalent in Python, we can use the value counts function<jupyter_code>#value count

df["Genre"].value_counts()<jupyter_output><empty_output><jupyter_text>The value counts goes through every item in the list for `Genre` and counts how many instances there are of that item. We can amend the `value_counts()` function to identify the frequency in which a particular value shows up within our column.<jupyter_code>## We can also identify the percentage for each value by adding an argument within our value_counts function

df["Genre"].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>The above code breakdowns the decimal frequency of each of the values within our Genre column. If we want to convert each value into a percentage output, we can apply two techniques:The `apply` method and the `lambda` function.The `apply()` method acts as an alternative to looping over a dataframe. It can apply a function across an entire Dataframe (or across all the values within a specified column in that dataframe).The `lambda` function assign multiple variables to an argument, and then perform operations on that argument (thereby performing that operation for each variable). Run the following code cell.<jupyter_code>df["Genre"].value_counts(normalize=True).apply(lambda x: f"{x * 100:.02f}%")

## The .02f, means = for any floats, we want there to be 2 (.02) decimal places. 

# The apply method is use here to apply a lambda function for each value within the Genre column

#The lambda function then takes each value and assigns it to the x (a placeholder variable), and then with "0.2f" - formats x as a float with two decimal places (0.2f)
#use the value counts function to return the number of males and females leads within our movies.
#make sure to use the print function to return both results. 


#use apply and lambda to convert and return that number into a percentage<jupyter_output><empty_output><jupyter_text>Calculate Central Tendency and DispersionLet's calculate the mean, median, and mode for the ImDb scores.<jupyter_code>mean = df_new["ImdB Score"].mean()
print(f"The mean ImDb score is {mean}")

med = df_new["ImdB Score"].median()
print(f"The median ImDb score is {med}")

mod = df_new["ImdB Score"].mode()
print(f"The mean ImDb score is {mod}") ##This will print out four values: index of mode1 - Mode1, index of mode2 - Mode2

#You will see this more clearly if you print the mode without the f-string below<jupyter_output><empty_output><jupyter_text>If we want to get a broad overview of our data, including calculating the measures of dispersion in our data, we can use the `describe` function.<jupyter_code>df_new.describe()<jupyter_output><empty_output><jupyter_text>The describe function identifies any column that contains numerical values and calculates a host of descriptive statistics for us, including the number of participants who have provided a value for that value (count), measures of dispersion. We can also use a function, `applymap`, to tidy up the output of these descriptive statistics (i.e., let's round to two decimal places)<jupyter_code>df_new.describe().applymap(lambda x: f'{x:0.2f}')<jupyter_output><empty_output><jupyter_text>We can also run the describe function on a single column. If we want to format the results for a single column however, we use the `apply` function rather than the `applymap` function, as the `applymap` function requires multiple lists to work with, rather than a single one.<jupyter_code>df_new["ImdB Score"].describe().apply(lambda x: f'{x:0.2f}')<jupyter_output><empty_output><jupyter_text>We can also specify the specific statistics that we need for a particular column, by using the `agg` method.<jupyter_code>df_new["ImdB Score"].agg(["count", "mean", "std"]).apply(lambda x: f'{x:0.2f}')<jupyter_output><empty_output><jupyter_text>Create new Variable - Mean Review ScoreBefore we start analysing data across differences in groups, let's create a new variable, called `Average Rating`. Average rating will consist of the mean score for Movie across ImDb, Rotten Tomatoes' Fans, and Rotten Tomatoes Critics ratings. We are then<jupyter_code>df_new["Average Rating"] = df_new[["ImdB Score", "Rotten Tomatoes' Fan Scores", "Rotten Tomatoes' Critic Scores"]].mean(axis = 1).apply(lambda x: f'{x:.02f}')

df_new

#This line of code creates a new column "Average Rating" in our df_new dataframe, that takes the scores from three columns, calculates their mean, formats it two decimal places,
# and then outputs the result into our new column. For the mean(axis = 1) means that are calculating the each new score from three seperate columns, rather than three seperate rows
#See what happens if you change axis = 0

#Now let's create a dataframe where we will focus our analysis on going forward. Let's call it df_analysis

df_analysis = df_new[['Movie Title', 'Director', 'Genre', 'Gender of Lead', "Average Rating", 'ImdB Score', "Rotten Tomatoes' Critic Scores", "Rotten Tomatoes' Fan Scores", 'Year of Release']]

#We will rearrange the column so that the data is a bit tidier given our aims

df_analysis<jupyter_output><empty_output><jupyter_text>Describing Data based on GroupsWe can also break up the results to determine the number and percentages of males and females per genre of movie. To do this, we can use a function called `groupby`. The `groupby` function involes i) splitting our object, ii) applying a function, and then iii) combining the results.<jupyter_code>#i splitting our object

gender_groups = df_analysis.groupby(["Gender of Lead"]) #creating a variable for later use

print(gender_groups)<jupyter_output><pandas.core.groupby.generic.DataFrameGroupBy object at 0x000001EA0F1D6E20><jupyter_text>The variable `gender_groups` returns a `DataFrameGroupBy object`. This object enables us to split our dataframe based on the gender of the participant. For example, imagine we only wanted a dataframe where the gender of the lead actor was male. We can use our gender_groups variable and then identify, using the `get_group` function, which of the groups we want to select.<jupyter_code>#ii applying a function to create gender-based dataframes

gender_groups.get_group("Male") ##get_groups enables us to identify only those 
#responses where the value Male is in the column "Gender of Lead"


#Return the values where the Gender of Lead is Female


##iii combining the results to get frequencies per gender

gender_groups["Genre"].value_counts()
#let's clean up the results a little bit. 

gender_groups["Genre"].value_counts(normalize=True).applymap(lambda x: f"{x * 100:.02f}%")

#we can get specific values, per gender, for specified columns

gender_groups["ImdB Score", "Rotten Tomatoes' Critic Scores", "Rotten Tomatoes' Fan Scores"].agg(["mean", "std"]).applymap(lambda x: f"{x:.02f}")

#Create a few code cells to run the following:

# 1. Run the groupby function with the Genre column and assign it to the variable genre_groups.
# 2. Return the value count of gender per genre (so the opposite of what we did earlier in the example)
# 3. Run descriptive statistics (describe) on the genre_groups
# 4. Pick one or more of the quantitative variables (i.e. columns) and return the mean and std for those columns, based on genre_groups<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08087883141762452, 'fraction_numerical': 0.0163433908045977, 'mean_word_length': 5.110095098756401, 'pattern_counts': {'":': 0, '<': 88, '<?xml version=': 0, '>': 88, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '601939', 'license': 'MIT', 'n_tokens_mistral': 4906, 'n_tokens_neox': 4517, 'n_words': 2393, 'path': 'src/week8.ipynb', 'repo_name': 'spider-z3r0/ULpsych_programing_club', 'size': '16806'}