<jupyter_start><jupyter_text>Libraries<jupyter_code>import numpy as np
import pandas as pd 
import matplotlib.pyplot as plt
import seaborn as sns

from warnings import filterwarnings 

pd.set_option('display.float_format', lambda num:'%1.3f'%num) 
pd.set_option('display.max_columns', 99)
filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Data Description | Feature                                                     	| Description||------------------------------------------------------------------------	|---	|| app_id                                        	| The unique application id  	|| category      	| The category under which app is categorized on the store.  	|| reviews                    	| The number of reviews received on the store.  	|| size                  	| Size of the app available for download (in KB/MB)  	|| installs  	| The number of people who had installed this app atleast once.  	|| price                                    	| The price of the app (in US $)  	|| suitable_for        	| Rating given to app based on the usage and content.  	|| last_update       	| When was the app updated last time by the developers.  	|| latest_ver        	| The latest version of the app available for download.  	|| popularity                                   	| User popularity (High/Low)  	|  Data Wrangling & Visualization<jupyter_code>train = pd.read_csv("../data/input/processed/train.csv")
train.head()
#Explore columns
train.columns
#Description
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 6275 entries, 0 to 6274
Data columns (total 10 columns):
 #   Column        Non-Null Count  Dtype 
---  ------        --------------  ----- 
 0   category      6275 non-null   object
 1   reviews       6275 non-null   object
 2   size          6275 non-null   object
 3   installs      6275 non-null   object
 4   price         6275 non-null   object
 5   suitable_for  6274 non-null   object
 6   last_update   6275 non-null   object
 7   latest_ver    6272 non-null   object
 8   app_id        6275 non-null   int64 
 9   popularity    6275 non-null   object
dtypes: int64(1), object(9)
memory usage: 490.4+ KB<jupyter_text>Handling Null Values<jupyter_code>train.isnull().sum()
train_na = (train.isnull().sum() / len(train)) * 100
train_na = train_na.drop(train_na[train_na == 0].index).sort_values(ascending=False)
missing_data = pd.DataFrame({'Missing Ratio' :train_na})
missing_data<jupyter_output><empty_output><jupyter_text>Since the ratio of missing values for the rest of the columns is less (<0.05), we proceed to fill these with the mode values instead of dropping these rows.<jupyter_code># Fill the three columns with the mode value
train['latest_ver'] = train['latest_ver'].fillna(train['latest_ver'].mode()[0])
train['suitable_for'] = train['suitable_for'].fillna(train['suitable_for'].mode()[0])<jupyter_output><empty_output><jupyter_text>Examine 'categories'<jupyter_code>categories = list(train["category"].unique())
print(categories)
# Remove Category 1.9
train = train[train.category != "1.9"]<jupyter_output><empty_output><jupyter_text>Data Cleaning Converting the "Installs", "Reviews", "Price" and "Size" columns to numeric values<jupyter_code># 'price' cleaning
train['price'] = train['price'].apply(lambda x : x.strip('$'))
# 'installs' cleaning
train['installs'] = train['installs'].apply(lambda x : x.strip('+').replace(',', ''))
train.head()
# 'size' cleaning
train["size"] = train["size"].str.replace("Varies with device", "0")
train["size"] = (train["size"].replace(r'[kM]+$', '', regex=True).astype(float) *\
        train["size"].str.extract(r'[\d\.]+([kM]+)', expand=False).fillna(1).replace(['k','M'], [10**3, 10**6]).astype(int))
# 'latest_ver'
train["latest_ver"] = train["latest_ver"].str.replace("Varies with device", "0")
# Convert the column types to numeric values
train['size'] = train['size'].apply(lambda x: float(x))
train['installs'] = train['installs'].apply(lambda x: float(x))
train['price'] = train['price'].apply(lambda x: float(x))
train['reviews'] = train['reviews'].apply(lambda x: int(x))<jupyter_output><empty_output><jupyter_text>**Handling Duplicates**<jupyter_code># Drop duplicates
train.drop_duplicates(inplace=True)
train.head()
# Display the dtypes of all the features in dataset
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 6067 entries, 0 to 6274
Data columns (total 10 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   category      6067 non-null   object 
 1   reviews       6067 non-null   int64  
 2   size          6067 non-null   float64
 3   installs      6067 non-null   float64
 4   price         6067 non-null   float64
 5   suitable_for  6067 non-null   object 
 6   last_update   6067 non-null   object 
 7   latest_ver    6067 non-null   object 
 8   app_id        6067 non-null   int64  
 9   popularity    6067 non-null   object 
dtypes: float64(3), int64(2), object(5)
memory usage: 521.4+ KB<jupyter_text>Visualization, Modeling, Machine Learning Build a model that can predict whether an app will become be trending or not and determine how different features influence the outcome. Please explain the findings effectively to technical and non-technical audiences using comments and visualizations, if appropriate. - Build an optimized model that effectively solves the business problem. - The model will be evaluated on the basis of Accuracy. - Read the test.csv file and prepare features for testing.<jupyter_code># Explore the distribution of the target column
sns.countplot(train['popularity'])<jupyter_output><empty_output><jupyter_text>There are approximately 3X more High labels than Low. Thus there is some class imbalance.<jupyter_code># Pairwise plot
sns.pairplot(train)<jupyter_output><empty_output><jupyter_text>There might be a need to remove outliers or standardize the data.<jupyter_code>sns.heatmap(train.corr(), annot=True)<jupyter_output><empty_output><jupyter_text>It seems there is a correlation between the number of 'installs' and number of 'reviews'.  Market breakdown  (installs vs category)<jupyter_code>ax = sns.jointplot(train['installs'], train['category'])
ax.fig.suptitle("Installs vs Category", y = 1.02, fontsize=14 )<jupyter_output><empty_output><jupyter_text>I broke down the apps by category and found that the Family and Game categories have the highest market prevalence. The Tools apps are also catching up.  Sizing strategy<jupyter_code># 'size' vs popularity
ax = sns.jointplot(train['size'], train['popularity'])
ax.fig.suptitle("Popularity vs Size of Apps", y = 1.02, fontsize=14 )<jupyter_output><empty_output><jupyter_text>Most popular apps are optimally sized between ~2MB to ~40MB i.e., neither too light nor too bulky.  Bulky Apps<jupyter_code>bulky_apps = train[train["size"]>40]
group_category = bulky_apps.groupby("category")["size"].count().sort_values(ascending=False).reset_index().head()

plt.figure(figsize=(8, 6))
sns.set(font_scale=1)
sns.set_style("darkgrid")
ax = sns.barplot(x="category", y="size", data=group_category, color = "darkcyan")
ax.set_xticklabels(ax.get_xticklabels(), rotation=90, ha="right")
plt.title('Count of Bulky Apps per Category',size = 14)<jupyter_output><empty_output><jupyter_text>Its found that the Game and Family categories have the highest number of bulky apps.We also observed that despite this, these bulky apps are fairly highly rated indicating that they are bulky for a purpose.  Pricing analysis  **vs Size**<jupyter_code>sns.scatterplot(x='size', y='price', data=train, s=5, color=".15")<jupyter_output><empty_output><jupyter_text>Majority of the paid apps have small sizes which implies that most paid apps are designed and developed to cater to specific functionalities and hence are not bulky. Users prefer to pay for apps that are light-weighted. A paid app that is bulky may not perform well in the market. **vs Category**<jupyter_code>sns.set_style("darkgrid")
ax = sns.catplot(x="price", y="category", kind="swarm", data=train)
plt.title('App pricing trend across categories',size = 14)<jupyter_output><empty_output><jupyter_text>Only categories Family, Lifestyle, Finance, and Business have apps with high prices. These could potentially be considered 'Junk' apps. **vs Installs**<jupyter_code># Adoption of paid vs free apps
sns.boxplot(x='installs', y='price', data=train, color=".15")
plt.show()<jupyter_output><empty_output><jupyter_text>Junk Apps<jupyter_code># Select apps priced below $100
train = train[train["price"] <100]

fig, ax = plt.subplots()
fig.set_size_inches(15, 8)

# Examine price vs category with the authentic apps (apps_under_100)
ax = sns.stripplot(x = "price", y = "category", data = train, jitter = True, linewidth = 1)
ax.set_title('App pricing trend across categories after filtering for junk apps')<jupyter_output><empty_output><jupyter_text>The apps now have a wider range of prices. The $80 cost for a Medical app might be a case for an app with extreme privacy and benefits. Apps with prices over 100 will be removed. It turns out that paid apps have a relatively lower number of installs than free apps.<jupyter_code># Finding categories with best popularity
train.value_counts(subset=['category', 'popularity']).head(10)<jupyter_output><empty_output><jupyter_text>Family, Game and Tools have the most number of bulky apps and they also have the highest popularity.  An app within one of these categores, which has a size less than 40MB, is either free or has a price less than $20 has a high chance of being popular.  Features encoding<jupyter_code>from sklearn import preprocessing
import time
import datetime
from matplotlib import pyplot

from sklearn.model_selection import train_test_split
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
# Category features encoding
category_list = train['category'].unique().tolist() 
category_list = ['cat_' + word for word in category_list]
train = pd.concat([train, pd.get_dummies(train['category'], prefix='cat')], axis=1)
# Encode 'suitable_for' features
le = preprocessing.LabelEncoder()
train['suitable_for'] = le.fit_transform(train['suitable_for'])
# 'last_update' encoding
train['last_update'] = train['last_update'].apply(lambda x : time.mktime(datetime.datetime.strptime(x, '%B %d, %Y').timetuple()))
# Bin 'installs'
bin_array = train.installs.sort_values().unique()

# convert to array
bins = [x for x in bin_array]

# Added 5 billion for the higher range of app installs
bins.append(5000000000)

# Create bins for installs
train['installs_binned'] = pd.cut(train['installs'], bins)

# Digitize the bins for encoding
installs_digitized = np.digitize(train['installs'], bins = bins )

# Add to the data frame as a column
train = train.assign(installs_d = pd.Series(installs_digitized).values)
# Drop irrelevant columns
train.drop(columns=['category', 'installs', 'installs_binned', 'app_id', 'latest_ver'], inplace=True)
train.info()
train.head(5)
# Split data into training and testing sets
features = ['reviews', 'size', 'installs_d', 'price', 'suitable_for', 'last_update']
features.extend(category_list)
X = train[features]
y = train['popularity']
# Select numerical columns for MinMaxScaling
column_trans = make_column_transformer(
        (MinMaxScaler(), features),
        remainder='passthrough'
)
X_trans = column_trans.fit_transform(X)
# Split the dataset into 75% train data and 25% test data.
X_train, X_test, y_train, y_test = train_test_split(X_trans, y, test_size = 0.25, random_state = 10)<jupyter_output><empty_output><jupyter_text>Why Accuracy?  .............  Modelling<jupyter_code># Make dict of models
models_dict = {
    "log_reg": LogisticRegression(max_iter=1000),
    "svm": LinearSVC(C=0.0001),
    "rf_clf": RandomForestClassifier(n_estimators=30, max_depth=9),
    "xgb_clf": XGBClassifier(n_estimators=30, max_depth=9),
}
for key, classifier in models_dict.items():
    print(f"Model: {classifier}")
    # fit the pipeline on the transformed data
    classifier.fit(X_train, y_train)   
    # make predictions
    y_pred = classifier.predict(X_test)
    # evaluate predictions
    accuracy = accuracy_score(y_test, y_pred)
    print("Accuracy: %.2f%%" % (accuracy * 100.0))<jupyter_output>Model: LogisticRegression(max_iter=1000)
Accuracy: 76.63%
Model: LinearSVC(C=0.0001)
Accuracy: 77.03%
Model: RandomForestClassifier(max_depth=9, n_estimators=30)
Accuracy: 77.23%
Model: XGBClassifier(base_score=None, booster=None, colsample_bylevel=None,
              colsample_bynode=None, colsample_bytree=None,
              enable_categorical=False, gamma=None, gpu_id=None,
              importance_type=None, interaction_constraints=None,
              learning_rate=None, max_delta_step=None, max_depth=9,
              min_child_weight=None, missing=nan, monotone_constraints=None,
              n_estimators=30, n_jobs=None, num_parallel_tree=None,
              predictor=None, random_state=None, reg_alpha=None,
              reg_lambda=None, scale_pos_weight=None, subsample=None,
              tree_method=None, validate_parameters=None, verbosity=None)
[11:06:02] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.5.1/src/learner.cc:1115: Starting in XGBoost 1.3.0, the[...]<jupyter_text>The XGBClassifier has the highest accuracy at 77.3%. Further new features can be generated through binning or discussions with a subject matter expert. Hyper parameter tuning can also be done to increase model accuracy score.  Show management the most important features in the model. -  Visualize the top 20 features and their feature importance.<jupyter_code>feat_importances = pd.Series(models_dict['xgb_clf'].feature_importances_, index=X.columns)
feat_importances.nlargest(20).plot(kind='barh')<jupyter_output><empty_output><jupyter_text>One-hot encoded app categories make up most of the top 20 features.  Features 'size', 'suitable_for' may need further feature engineering through binning to create more meaningful relations.  'last_update' can also be manipulated to be the difference between current date and last updated date.  Inference -  Submit the predictions on the test dataset using your optimized modelFor each record in the test set ( test.csv), predict the popularity variable. Submit a CSV file with a header row and one row per test entry.<jupyter_code># Loading Test data 
test_data=pd.read_csv('./data/input/processed/test.csv')
test_data.drop(columns=['popularity'], inplace=True)
test_data.head()
def row_operators(test):
    test = test.copy()
    test['suitable_for'] = test['suitable_for'].fillna(test['suitable_for'].mode()[0])
    test['price'] = test['price'].apply(lambda x : x.strip('$'))
    test['installs'] = test['installs'].apply(lambda x : x.strip('+').replace(',', ''))
    test["size"] = test["size"].str.replace("Varies with device", "0")
    test["size"] = (test["size"].replace(r'[kM]+$', '', regex=True).astype(float) *\
        test["size"].str.extract(r'[\d\.]+([kM]+)', expand=False).fillna(1).replace(['k','M'], [10**3, 10**6]).astype(int))

    test['size'] = test['size'].apply(lambda x: float(x))
    test['installs'] = test['installs'].apply(lambda x: float(x))
    test['price'] = test['price'].apply(lambda x: float(x))
    test['reviews'] = test['reviews'].apply(lambda x: int(x))
    
    test = test[test.category != "1.9"]
    test.drop_duplicates(inplace=True)
    test = test[test["price"] <100]
    
    return test
test_data = row_operators(test_data)
app_id_col = test_data['app_id']
app_id_col.shape
# Create function for performing feature engineering
def feature_engineering(test):
    category_list = test['category'].unique().tolist() 
    category_list = ['cat_' + word for word in category_list]
    test = pd.concat([test, pd.get_dummies(test['category'], prefix='cat')], axis=1)

    le = preprocessing.LabelEncoder()
    test['suitable_for'] = le.fit_transform(test['suitable_for'])
    test['last_update'] = test['last_update'].apply(lambda x : time.mktime(datetime.datetime.strptime(x, '%B %d, %Y').timetuple()))

    # Bin 'installs'
    bin_array = test.installs.sort_values().unique()
    # convert to array
    bins = [x for x in bin_array]
    # Added 5 billion for the higher range of app installs
    bins.append(5000000000)
    # Create bins for installs
    test['installs_binned'] = pd.cut(test['installs'], bins)
    # Digitize the bins for encoding
    installs_digitized = np.digitize(test['installs'], bins = bins )
    # Add to the data frame as a column
    test = test.assign(installs_d = pd.Series(installs_digitized).values)

    test.drop(columns=['category', 'installs', 'installs_binned', 'app_id', 'latest_ver'], inplace=True)

    features = ['reviews', 'size', 'installs_d', 'price', 'suitable_for', 'last_update']
    features.extend(category_list)
    X = test[features]

    column_trans = make_column_transformer(
        (MinMaxScaler(), features),
        remainder='passthrough'
    )
    test = column_trans.fit_transform(X)

    return test
test = feature_engineering(test_data)
y_pred = models_dict['xgb_clf'].predict(test)<jupyter_output><empty_output><jupyter_text>The file (submissions.csv) should have exactly 2 columns: - app_id- popularity<jupyter_code>submission_df = app_id_col.to_frame()
submission_df['popularity'] = y_pred.tolist()
submission_df.head(5)
# Submission
submission_df.to_csv('../data/output/submissions.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1261215507025563, 'fraction_numerical': 0.02025845042604819, 'mean_word_length': 4.6206787186806215, 'pattern_counts': {'":': 4, '<': 95, '<?xml version=': 0, '>': 93, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '566144', 'license': 'MIT', 'n_tokens_mistral': 5834, 'n_tokens_neox': 5500, 'n_words': 1541, 'path': 'notebook/app-popularity.ipynb', 'repo_name': 'jovi-s/app-popularity', 'size': '17721'}