<jupyter_start><jupyter_text>How Do We Design a Good Online Course for Business Analytics*** 1. Introduction  BackgroundWebsites for a massive open online course (MOOC), such as Udacity, Udemy, and Coursera, provide learners a low-cost, highly efficient way to acquire knowledge. Business analytics, which relies mainly on computer coding, is a popular topic in MOOC websites. Considering the impact of the COVID-19 pandemic on online learning and the job market where plenty of job openings on business analytics, we can foresee that the demand for business analytics online courses will continuously rise. In this blog, we aim to discover the secrete of designing an excellent online course for business analytics. We study the business analytics courses on the MOOC website, Udemy. Compared with MOOC websites like Udacity and Coursera, Udemy has many courses because it allows everyone to upload courses. The cost of learning Udemy courses is low as Udemy gives the $10 per course promotion regularly.  ObjectiveIn this project, we web scrap data of business analytics courses from Udemy and analyze to find insight on designing a good online course for business analytics.  Business Questions* What are the features of Business Analytics online course?* How are the course content introduced?* How are the instructor introduced?* What are the factors impacting the enrollments of the Business Analytics online course? DatasetWe use the package Selenium with Python to perform web scraping on the Udemy website.  The list of 1005 courses is obtained from Udemy's webpage for Business Analytics & Intelligence Courses. The information, such as enrollment number, course rating, and course description, is scrapped on each course's web page. * The preprosess of the non-text data is implemented in notebook ***ETL/ETL_nonText.ipynb***. * The preprosess of the text data is implemented in notebook ***ETL/ETL_Text.ipynb***.  Evaluation MetricsWe set the enrollment number as the depedent variable and use other variables to predict the enrollment number. We define the Root Mean Squared Logarithmic Error (RMSLE) as the evaluation metrics. $RMSLE=\sqrt{\frac{1}{n}\sum\limits_{i=1}^{n}{{{(\log ({{p}_{i}}+1)-\log ({{a}_{i}}+1))}^{2}}}}$* $n$: number of sample courses* ${p}_{i}$: the prediction of the enrollment number of the $i$ course.* ${a}_{i}$: the actural value of the enrollment number of the $i$ course.  2. Load Dataset<jupyter_code>import numpy as np
import pandas as pd
import math
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder, MinMaxScaler, StandardScaler
%matplotlib inline<jupyter_output><empty_output><jupyter_text>* The data is preprocessed in ***ETL/ETL_nonText.ipynb*** and ***ETL/ETL_Text.ipynb***.<jupyter_code>pd.options.display.max_columns = None
data = pd.read_pickle("../ETL/data_final.pkl")
data.head()
data.describe()<jupyter_output><empty_output><jupyter_text>***Comment:* In the dataset, several varaible is highly screwed. * The enrollment numbers has the maximum 342288, mean 5785, and standard deviation 21279. The 75% enrollment number is 3145, implying that most courses have enrollment numbers between 0 and 4000.* Because Udemy regularly has the $10 per course promotion, the price variable does not have a significant variance. Meanwhile, the original price variable has a big variance with the maximum 199.99, the mean 15.97, and the standard deviation of 17.07. * Hence, we need to perform a data exploration and feature transformation. ***  3. Data Exploration Enrollment<jupyter_code>plt.hist(data["enrollment"],  bins=10)
plt.ylabel("Number of courses")
plt.xlabel('Enrolloment')<jupyter_output><empty_output><jupyter_text>**** As the dependent variable, the enrollment variable should be close to normal distributin. * Hence, a log-transform is performed. As shown in the plot, the log-transformed enrollment variable is close to normal distribution. ***<jupyter_code>data["enrollment"] = data["enrollment"].apply(lambda x: np.log(x+1))
plt.hist(data["enrollment"], bins=10)
plt.ylabel('Number of courses')
plt.xlabel('Enrollments (Log-transformed)')<jupyter_output><empty_output><jupyter_text>Price<jupyter_code>plt.hist(data["price"],  bins=10)<jupyter_output><empty_output><jupyter_text>**** As one of the independent variable, the price variable should be close to normal distributin. * Hence, a log-transform and scaling are performed. As shown in the plot, the log-transformed and scaled price variable is less skewed. ***<jupyter_code>data["price"] = data["price"].apply(lambda x: np.log(x+1))
data["price"] = MinMaxScaler(feature_range=(-1, 1)).fit_transform(data["price"].values.reshape(-1,1))
plt.hist(data["price"], bins=10)<jupyter_output><empty_output><jupyter_text>Original_price<jupyter_code>plt.hist(data["original_price"],  bins=10)<jupyter_output><empty_output><jupyter_text>**** As one of the independent variable, the original_price variable should be close to normal distributin. * Hence, a log-transform and scaling are performed. As shown in the plot, the log-transformed and scaled original_price variable is less skewed. ***<jupyter_code># data["original_price"] = data["original_price"].apply(lambda x: np.log(x+1))
data["original_price"] = MinMaxScaler(feature_range=(-1, 1)).fit_transform(data["original_price"].values.reshape(-1,1))
plt.hist(data["price"], bins=10)<jupyter_output><empty_output><jupyter_text>lectures<jupyter_code>plt.hist(data["lectures"],  bins=10)<jupyter_output><empty_output><jupyter_text>**** As one of the independent variable, the lectures variable should be close to normal distributin. * Hence, a log-transform and scaling are performed. As shown in the plot, the log-transformed and scaled lectures variable is close to normal distribution. ***<jupyter_code>data["lectures"] = data["lectures"].apply(lambda x: np.log(x+1))
data["lectures"] = MinMaxScaler(feature_range=(-1, 1)).fit_transform(data["lectures"].values.reshape(-1,1))
plt.hist(data["lectures"], bins=10)<jupyter_output><empty_output><jupyter_text>hours<jupyter_code>plt.hist(data["hours"],  bins=10)<jupyter_output><empty_output><jupyter_text>**** As one of the independent variable, the hours variable should be close to normal distributin. * Hence, a log-transform and scaling are performed. As shown in the plot, the log-transformed and scaled hours variable is less skewed. ***<jupyter_code>data["hours"] = data["hours"].apply(lambda x: np.log(x+1))
data["hours"] = MinMaxScaler(feature_range=(-1, 1)).fit_transform(data["hours"].values.reshape(-1,1))
plt.hist(data["hours"], bins=10)<jupyter_output><empty_output><jupyter_text>rating<jupyter_code>plt.hist(data["rating"],  bins=10)<jupyter_output><empty_output><jupyter_text>**** As one of the independent variable, the rating variable should be close to normal distributin. * Hence, a scaling is performed. As shown in the plot, the log-transformed and scaled rating variable is less skewed. ***<jupyter_code># data["rating"] = data["rating"].apply(lambda x: np.log(x+1))
data["rating"] = MinMaxScaler(feature_range=(-1, 1)).fit_transform(data["rating"].values.reshape(-1,1))
plt.hist(data["rating"], bins=10)<jupyter_output><empty_output><jupyter_text>downloadable_resources<jupyter_code>plt.hist(data["downloadable_resources"],  bins=10)<jupyter_output><empty_output><jupyter_text>**** As one of the independent variable, the downloadable_resources variable should be close to normal distributin. * Hence, a log-transform and scaling are performed. As shown in the plot, the log-transformed and scaled downloadable_resources variable is less skewed. ***<jupyter_code>data["downloadable_resources"] = data["downloadable_resources"].apply(lambda x: np.log(x+1))
data["downloadable_resources"] = MinMaxScaler(feature_range=(-1, 1)).fit_transform(data["downloadable_resources"].values.reshape(-1,1))
plt.hist(data["downloadable_resources"], bins=10)<jupyter_output><empty_output><jupyter_text>instructor_no_courses<jupyter_code>plt.hist(data["instructor_no_courses"],  bins=10)<jupyter_output><empty_output><jupyter_text>**** As one of the independent variable, the instructor_no_courses variable should be close to normal distributin. * Hence, a log-transform and scaling are performed. As shown in the plot, the log-transformed and scaled instructor_no_courses variable is less skewed. ***<jupyter_code>data["instructor_no_courses"] = data["instructor_no_courses"].apply(lambda x: np.log(x+1))
data["instructor_no_courses"] = MinMaxScaler(feature_range=(-1, 1)).fit_transform(data["instructor_no_courses"].values.reshape(-1,1))
plt.hist(data["instructor_no_courses"], bins=10)<jupyter_output><empty_output><jupyter_text>4. Process categorical data<jupyter_code>dummy = pd.get_dummies(data["level"])
data = pd.concat([data,dummy],axis = 1)
data.drop(["level"],axis = 1, inplace=True)<jupyter_output><empty_output><jupyter_text>5. Regression Models The dependent variable and independent variables* The enrollment number is set as the dependent variables, and other variables are set as independent. * We drop the num_ratings variable, because the rating number obviously follows the enrollment. We can not obtain useful information from the variable.<jupyter_code>y = data["enrollment"]
X = data.drop(["enrollment","num_ratings"],axis = 1)<jupyter_output><empty_output><jupyter_text>Split dataset* The dataset is splitted into training and testing dataset for validating the prediction model.<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=99)<jupyter_output><empty_output><jupyter_text>RMSLE* We define a function to calculate Root Mean Squared Logarithmic Error (RMSLE)<jupyter_code>def rmsle(y, y_pred):
    assert len(y) == len(y_pred)
    to_sum = [(math.log(y_pred[i] + 1) - math.log(y[i] + 1)) ** 2.0 for i,pred in enumerate(y_pred)]
    return (sum(to_sum) * (1.0/len(y))) ** 0.5<jupyter_output><empty_output><jupyter_text>Ridge Regression* Ridge regression model is to use linear least squares with l2 regularization.* The model is to minimized the objective function, $||y - Xw||^2_2 + alpha * ||w||^2_2 $ * The Ridge regression is good for analyzing multiple regression data with multicollinearity. The multicollinearity is a situation in which more than two explanatory variables in a multiple regression model are highly linearly related. It makes least squares estimates have large variances are large.<jupyter_code>from sklearn.linear_model import Ridge
model = Ridge(alpha=0.9, fit_intercept=True, max_iter=1000,
   normalize=False, random_state=666, tol=0.00001)
model.fit(X_train, y_train)
y_pred = model.predict(X=X_test)
y_pred = np.expm1(y_pred)
y_true = np.expm1(y_test)
v_rmsle = rmsle(y_true.values, y_pred)
print("Ridge- RMSLE error on test dataset: "+str(v_rmsle))<jupyter_output>Ridge- RMSLE error on test dataset: 2.021496763033342<jupyter_text>**** The hyperparameter of the Ridge regression is alpha.* We use four values of alpha, 0.05, 0.1, 0.3, and 0.9. Among them, alpha of 0.9 gives the minimum RMSLE error on test dataset.***<jupyter_code>plt.scatter(y_true.values, y_pred, color = "red")
plt.title("True vs Prediction (Test dataset)")
plt.xlabel("True Value of Enrollment")
plt.ylabel("Prediction of Enrollment")
plt.show()<jupyter_output><empty_output><jupyter_text>Lasso Regression* Lasso regression model is to use linear least squares with l1 regularization.* The model is to minimized the objective function, $(1 / (2 * n_samples)) * ||y - Xw||^2_2 + alpha * ||w||_1$ * Lasso regression analysis is a shrinkage and variable selection method. * Shrinkage is where data values are shrunk towards a central point, like the mean. The lasso procedure obtain the subset of predictors.<jupyter_code>from sklearn.linear_model import Ridge, Lasso
model = Lasso(alpha=0.05, fit_intercept=True, max_iter=10000,
   normalize=False, precompute=False, random_state=666,
   selection='random', tol=0.00001)
model.fit(X_train, y_train)
y_pred = model.predict(X=X_test)
y_pred = np.expm1(y_pred)
y_true = np.expm1(y_test)
v_rmsle = rmsle(y_true.values, y_pred)
print("Lasso- RMSLE error on test dataset: "+str(v_rmsle))<jupyter_output>Lasso- RMSLE error on test dataset: 2.06452758703387<jupyter_text>**** The hyperparameter of the Lasso regression is alpha.* We use four values of alpha, 0.05, 0.1, 0.3, and 0.9. Among them, alpha of 0.05 gives the minimum RMSLE error on test dataset.***<jupyter_code>plt.scatter(y_true.values, y_pred, color = "red")
plt.title("True vs Prediction (Test dataset)")
plt.xlabel("True Value of Enrollment")
plt.ylabel("Prediction of Enrollment")
plt.show()<jupyter_output><empty_output><jupyter_text>Decision Tree Regression* Decision Tree model is to predicts the value of a target variable by learning simple decision rules inferred from the data features.* It breaks down a dataset into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed.<jupyter_code>from sklearn.tree import DecisionTreeRegressor
model = DecisionTreeRegressor(max_depth=4)
model.fit(X_train, y_train)
y_pred = model.predict(X=X_test)
y_pred = np.expm1(y_pred)
y_true = np.expm1(y_test)
v_rmsle = rmsle(y_true.values, y_pred)
print("Decision Tree Regression- RMSLE error on test dataset: "+str(v_rmsle))<jupyter_output>Decision Tree Regression- RMSLE error on test dataset: 1.5875228458514539<jupyter_text>**** The hyperparameter of the decision tree regression is max_depth.* We use four values of max_depth, 2, 4, 6, and 8. Among them, max_depth of 4 gives the minimum RMSLE error on test dataset.***<jupyter_code>plt.scatter(y_true.values, y_pred, color = "red")
plt.title("True vs Prediction (Test dataset)")
plt.xlabel("True Value of Enrollment")
plt.ylabel("Prediction of Enrollment")
plt.show()<jupyter_output><empty_output><jupyter_text>* Plot feature importance<jupyter_code>for feature, importance in zip(X.columns,model.feature_importances_):
    print(feature, importance)<jupyter_output>price 0.0
original_price 0.0
lectures 0.02914103102085937
hours 0.0
rating 0.0
five_stars 0.06699101518945215
four_stars 0.006111870442173789
three_stars 0.6356907963109857
two_stars 0.22888016127690103
downloadable_resources 0.0055633764925381015
instructor_no_courses 0.00483408015008962
descr_LDA_0 0.0
descr_LDA_1 0.00012595410699399614
descr_LDA_2 0.001318428927052887
descr_LDA_3 0.0
descr_LDA_4 0.0
descr_LDA_5 0.0
descr_LDA_6 0.0
descr_LDA_7 0.0
descr_LDA_8 0.0
descr_LDA_9 0.0
instr_LDA_0 0.01711448827730642
instr_LDA_1 0.0
instr_LDA_2 0.0
instr_LDA_3 0.0
instr_LDA_4 0.0
instr_LDA_5 0.0
instr_LDA_6 0.0
instr_LDA_7 0.0
instr_LDA_8 0.004228797805646816
instr_LDA_9 0.0
All Levels 0.0
Beginner 0.0
Expert 0.0
Intermediate 0.0<jupyter_text>***Model Comparision* Among the three regressions used in this study, the decision tree regression model shows the highest accuracy.* The three regreesion models are provided by the python package Scikit-learn. They are easy to use but can not indicate significant values of independent variables. * The decision tree regression model in Scikit-learn shows the feature importance of each independent variables. However, it can not show how the independent variable impact the dependent variable, such as positive or negative impact.  Feature importance* The variables that have importance in the regression are lectures, five_stars, four_stars, three_stars, two stars, downloadable_resources, instructor_no_courses, descr_LDA_1, descr_LDA_2, instr_LDA_0, and instr_LDA_8.* Key words of descr_LDA 1: curso, power, bi, descript, requir, show, dato, aprend, b, sico* Key words of descr_LDA 2: report, sql, pentaho, power, json, bi, postgr, use, server, gi* Key words of instr_LDA_0: scienc, cours, univers, show, market, technolog, analyt, experi, team, year* Key words of instr_LDA_8: jeff, year, step, program, technolog, workforc, industri, null, analyt, time* The above four topics can be interpreted as: power bi and visulization, project report involving sql, instructor in university, and many years of experience of instructor.***  OLS Regression* Here we use ordinary least squares (OLS) model to check the impacts of independent variables on the dependent variable (enrollment number).* The OLS model is one of the simplest regression model, but is good to exame the correlation between independent variable and dependent variable. * In the use of OLS model, we don't care about the acuracy of the prediction model. Hence, we don't split the dataset into train and test for validation, but use the whole dataset to fit the model.<jupyter_code>import statsmodels.api as sm
mod = sm.OLS(y,X)
fii = mod.fit()
p_values = fii.summary2().tables[1]['P>|t|']
print(fii.summary())<jupyter_output>OLS Regression Results                                
=======================================================================================
Dep. Variable:             enrollment   R-squared (uncentered):                   0.916
Model:                            OLS   Adj. R-squared (uncentered):              0.913
Method:                 Least Squares   F-statistic:                              302.8
Date:                Sat, 27 Feb 2021   Prob (F-statistic):                        0.00
Time:                        12:27:33   Log-Likelihood:                         -2100.1
No. Observations:                1005   AIC:                                      4270.
Df Residuals:                     970   BIC:                                      4442.
Df Model:                          35                                                  
Covariance Type:            nonrobust                                                  
================================[...]
{'fraction_non_alphanumeric': 0.1056100033795201, 'fraction_numerical': 0.03379520108144644, 'mean_word_length': 5.432971014492754, 'pattern_counts': {'":': 0, '<': 116, '<?xml version=': 0, '>': 117, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '347763', 'license': 'MIT', 'n_tokens_mistral': 5880, 'n_tokens_neox': 5347, 'n_words': 1794, 'path': 'model/Analysis.ipynb', 'repo_name': 'leizhipeng/analyze_mooc_course_data', 'size': '17754'}