<jupyter_start><jupyter_text>Assessing the Readiness of HEI in Malaysia to Accept Generation Alpha--- Data Exploration![orange-divider](https://user-images.githubusercontent.com/7065401/92672455-187a5f80-f2ef-11ea-890c-40be9474f7b7.png) Info ContentThis notebook provides the in depth analysis on the student performance in exams at public schools.**Info_Content.csv**The content in Junyi Academy contains exercises, videos, and exams.All the content in this dataset is in the type of exercise.An exercise is a basic unit for students to learn a certain concept.There are multiple problems in a single exercise that all relate to a certain concept.This table records the metadata and hierarchy structure of each exercise in Junyi Academy.There are three difficulty settings for each content, which indicates how hard it is to learn the concept.The learning stage is separated into three stages: Elementary, Junior, and Senior.The exercises in Junyi Academy are organized in a tree-like structure.The current dataset release has four levels in the hierarchy.| Variable Name | Description ||:-|:-|| ucid | The hashed unique ID of the content. || content_pretty_name | The Chinese display name of this content. || content_kind | The kind of this content. The current dataset release only includes `Exercise` || difficulty | The difficulty of this content. There are four possible values: `Easy`, `Normal`, `Hard` and `Unset`. Unset means || learning stage | The subject of this content. The current dataset release only includes `math` || subject | The learning stage of this content. There are three possible values: `Elementary`, `Junior` and `Senior`. || level1_id | The hashed level 1 layer ID of this content. The levels form the tree-like hierarchy structure of contents in Junyi || level2_id | The hashed level 2 layer ID of this content. The levels form the tree-like hierarchy structure of contents in Junyi || level3_id | The hashed level 3 layer ID of this content. The levels form the tree-like hierarchy structure of contents in Junyi || level4_id | The hashed level 4 layer ID of this content. The levels form the tree-like hierarchy structure of contents in Junyi |  Importing libraries<jupyter_code>import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style='whitegrid')
sns.set_color_codes('pastel')<jupyter_output><empty_output><jupyter_text>Loading data First, the dataset is imported using pandas. The read_csv() function is to load the data from the csv file to the dataframe which called `info_content_raw_df`.<jupyter_code>%%time

info_content_raw_df = pd.read_csv('../data/raw/Info_Content.csv', index_col='ucid')
info_content_raw_df.head()<jupyter_output>Wall time: 11 ms<jupyter_text>The process has taken 11 ms.  Exploratory Data Analysis (EDA) Before exploring the data, some observations can be seen.1. All columns consists of object data type only.2. `content_pretty_name` column has to be translated from Chinese to English. Let’s have a look at data dimensionality, feature names, and feature types.<jupyter_code>info_content_raw_df.shape
# Returns different datatypes for each columns (float, int, string, bool, etc.)
info_content_raw_df.dtypes<jupyter_output><empty_output><jupyter_text>From the output, we can see that the table contains 1330 rows and 9 columns.Now let's try printing out column names using columns:Returns number of rows and columns of the dataset<jupyter_code># Returns true for a column having null values, else false
info_content_raw_df.isnull().any()
# Returns basic information on all columns
info_content_raw_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 1330 entries, odIwFdIiecFwVUAEEV40K3MSuCSlIZkbq92Zp9tkZq8= to gvez7GFXUbuQl27U5+p/4QwFZZyXP2QFYQdoor8ZkeE=
Data columns (total 9 columns):
 #   Column               Non-Null Count  Dtype 
---  ------               --------------  ----- 
 0   content_pretty_name  1330 non-null   object
 1   content_kind         1330 non-null   object
 2   difficulty           1330 non-null   object
 3   subject              1330 non-null   object
 4   learning_stage       1330 non-null   object
 5   level1_id            1330 non-null   object
 6   level2_id            1330 non-null   object
 7   level3_id            1330 non-null   object
 8   level4_id            1330 non-null   object
dtypes: object(9)
memory usage: 103.9+ KB<jupyter_text>object is the data types of our features. With this same method, we can easily see if there are any missing values. Here, there are none because each column contains 1330 observations, the same number of rows we saw before with shape.<jupyter_code># Returns basic statistics on numeric columns
info_content_raw_df.describe().T<jupyter_output><empty_output><jupyter_text>Data Visualization Categorical Data For categorical (type object) feature we can use the value_counts method. Let's have a look at the distribution of Churn:<jupyter_code>info_content_raw_df['content_pretty_name'].value_counts()<jupyter_output><empty_output><jupyter_text>Some content_pretty_name has the frequency of 2.<jupyter_code>info_content_raw_df['content_kind'].value_counts()
sns.countplot(info_content_raw_df['content_kind']).set_title('content_kind distribution')<jupyter_output><empty_output><jupyter_text>1330 records are the kind of exercise. The current dataset release only includes `Exercise`. The `content_kind` column can be drop since all rows have the same values.<jupyter_code>info_content_raw_df['difficulty'].value_counts()
sns.countplot(info_content_raw_df['difficulty']).set_title('difficulty distribution')<jupyter_output><empty_output><jupyter_text>835 problems are easy, 305 problems are normal, 149 problems are hard difficulty and 41 problems are unset.<jupyter_code>info_content_raw_df['subject'].value_counts()
sns.countplot(info_content_raw_df['subject']).set_title('subject distribution')<jupyter_output><empty_output><jupyter_text>The subject of all 1330 content records are `math`. The current dataset release only includes `math`.<jupyter_code>info_content_raw_df['learning_stage'].value_counts()
sns.countplot(info_content_raw_df['learning_stage']).set_title('learning_stage distribution')<jupyter_output><empty_output><jupyter_text>There are 784 problems are elementary learning stage, 543 problems are junior learning stage and 3 problems are senior stage.<jupyter_code>info_content_raw_df['level1_id'].value_counts()
sns.countplot(info_content_raw_df['level1_id']).set_title('level1_id distribution')
info_content_raw_df['level2_id'].value_counts()
sns.countplot(info_content_raw_df['level2_id']).set_title('level2_id distribution')
info_content_raw_df['level3_id'].value_counts()
sns.countplot(info_content_raw_df['level3_id']).set_title('level3_id distribution')
info_content_raw_df['level4_id'].value_counts()
sns.countplot(info_content_raw_df['level4_id']).set_title('level4_id distribution')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07868056547194059, 'fraction_numerical': 0.024275310581179495, 'mean_word_length': 5.595103578154426, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '243518', 'license': 'MIT', 'n_tokens_mistral': 2149, 'n_tokens_neox': 2023, 'n_words': 707, 'path': 'notebooks/1.1-data-exploration-info-content.ipynb', 'repo_name': 'likweitan/final_year_project', 'size': '7003'}