<jupyter_start><jupyter_text>SETUP<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>data description This release is comprised of de-identified data from the first year (Academic Year 2013: Fall 2012, Spring 2013, and Summer 2013) of MITx and HarvardX courses on the edX platform along with related documentation. These data are aggregate records, and each record represents one individual's activity in one edX course. For more information about the existing analyses of these data and the first year of HarvardX and MITx courses, please see the HarvardX and MITx working paper "HarvardX and MITx: The first year of open online courses" by <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, and <NAME> (http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263). The first release of this dataset is the HarvardX-MITx Person-Course Academic Year 2013 De-Identified dataset, version 2.0, created on May 14, 2014. File name: HMXPC13_DI_v1_5-14-14.csv The md5sum for this release (HMXPC13_DI_v2_5-14-14.csv) is: 2b09c674af772d45dae429045cf7acfc (2014-05-24) --- Using the same data frame you created from the MITx/Harvardx dataset in the Week 1 assignment (D1), follow the directions below:1. How many people who enrolled in HarvardX/CS50x received certification in 2012?```Install.packages(dplyr)library(dplyr)D1b <- filter(D1, course_id == "HarvardX/CS50x/2012", certified== 1)View(D1b)```2. How many people registered for HarvardX/CS50x from Colombia on October 15, 2012?```COL<- filter(D1, course_id == "HarvardX/CS50x/2012", final_cc_cname_DI == "Colombia", start_time_DI == "2012-10-15")View(COL)```3. Create a new data frame (D2) that includes student ID and registration date for all students from the country label “Other Oceania”```D2 <- filter(D1, final_cc_cname_DI == "Other Oceania")D2 <- select(D2, userid_DI, start_time_DI)```4. Reflect: What kinds of questions could you answer using only student ID and start time? Is this information useful for any purpose?5. Create a new variable called “count” that only contains the number 1 for each entry.```D2(dollar)count <- 1```6. Using the “spread” function from the “tidyr” package create a new data frame (D3) in wide format in which the column names are the registration dates, with one column containing the student ids and all other cells indicating a 1 if the student registered on that date or an NA if they did not. This converts our data frame from long to wide format.```Install.packages(tidyr)library(tidyr)D2 <- unique(D2)D3 <- spread(D2, start_time_DI, count)View(D3)```7. Reflect: Why might we need data to be in wide format rather than long format?<jupyter_code>file = '/Users/quartz/data/learning_analytics/learning_analytics_fundamentals/HMXPC13_DI_v2_5-14-14.csv'
D1 = pd.read_csv(file)
D1.tail()
# 1. How many people who enrolled in HarvardX/CS50x received certification in 2012?

D1[(D1.course_id == "HarvardX/CS50x/2012") & (D1.certified == 1)].tail()
# 2. How many people registered for HarvardX/CS50x from Colombia on October 15, 2012?

COL = D1[(D1.course_id == "HarvardX/CS50x/2012") & (D1.final_cc_cname_DI == "Colombia") & (D1.start_time_DI == "2012-10-15")]
COL.tail()
# 3. Create a new data frame (D2) that includes student ID and registration date for all students from the country label “Other Oceania”

D2 = D1[D1.final_cc_cname_DI == "Other Oceania"]
D2 = D2[['userid_DI', 'start_time_DI']]
D2.tail()
# 4. Reflect: What kinds of questions could you answer using only student ID and start time? Is this information useful for any purpose?
# could notice who start the lectures. But that's all.

len(D2), len(np.unique(D2.userid_DI)), len(np.unique(D2.start_time_DI))
# 5. Create a new variable called “count” that only contains the number 1 for each entry.

D2['count'] = 1
D2.tail()
# 6. Using the “spread” function from the “tidyr” package create a new data frame (D3) in wide format 
# in which the column names are the registration dates, with one column containing the student ids and all other cells 
# indicating a 1 if the student registered on that date or an NA if they did not. This converts our data frame from long to wide format.

D2.groupby('start_time_DI').count().tail()
# 7. Reflect: Why might we need data to be in wide format rather than long format?<jupyter_output><empty_output><jupyter_text>---  EDA<jupyter_code>D1.tail()
D1.isna().sum().sort_values(ascending=False)
D1.describe()
# value_counts()

columns = D1.columns

for column in columns:
    
    try:
        content = D1[column].value_counts()[:20]
        n = len(D1[column].unique())
        line = "-" * 50

        print("'{}'s = {} \n\n {} \n{}\n".format(column, n, content, line))
    except:
        print("Error with the column '{}'\n{}\n".format(column, line))
# userid

n = len(np.unique(D1.userid_DI))

print("The number of students is %d" % (n))
print("Each students enrolled average %.3f courses" % (len(D1) / n))
# which continent have the most student.

data_certified = D1[D1.certified == 1]
data_certified.final_cc_cname_DI.value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10819286554292434, 'fraction_numerical': 0.04586436691493532, 'mean_word_length': 4.892609699769053, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '361291', 'license': 'MIT', 'n_tokens_mistral': 1788, 'n_tokens_neox': 1614, 'n_words': 612, 'path': '01_LearningAnalyticsFundamentals/Week 2_2(python).ipynb', 'repo_name': 'seokyeongheo/learninganalytics-study', 'size': '5144'}