<jupyter_start><jupyter_text>Android Play Store Dataset Cleaning        Dataset link<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>1. Dataset Reading    dicovering the dataset and get a notion of what the attributes describe.<jupyter_code>data = pd.read_csv("data/googleplaystore.csv")
data.sample(5)<jupyter_output><empty_output><jupyter_text>2. Dataset Discovery    dicovering the dataset and get a notion of what the attributes describe.<jupyter_code>data.count()
data.describe()
data.dtypes<jupyter_output><empty_output><jupyter_text>3. Dataset Pre-processing    dicovering the dataset and get a notion of what the attributes describe.      3.1 Checking for Missing values<jupyter_code>data.isnull().sum(0)
data[data["Rating"].isnull()].head()<jupyter_output><empty_output><jupyter_text>There are no missing reviews. However, there're missing rating values. Google play store forces you to rate an app if you're wiritng a review. Something must have gone wrong during data scraping for the rating values to be missing. We decided to leave the value as is.          3.2 Checking for duplicate values<jupyter_code>data.duplicated().value_counts()
data.drop_duplicates(inplace=True)
data["App"].duplicated().value_counts()
duplicated_app = data[data["App"].duplicated()].sort_values("App")<jupyter_output><empty_output><jupyter_text>We found that there're still some duplicate values based on the app name. After further investigation we found that the difference is the number of reviews. So, we dropped the duplicate rows and chose the row with the highest number of reviews to keep.        We first had to transform Reviews into a numeric column type and fix string values.          3.1 Checking for Missing values<jupyter_code>data[data['Reviews'].str.contains('[a-zA-Z]')]
replacement = dict(M=10**6)
data["Reviews"] = data["Reviews"].replace(replacement, regex=True).astype(int)
ordered_slice = data.sort_values(['App', 'Reviews'], ascending=[False, False])
data = ordered_slice.drop_duplicates(["App"])
data["App"].duplicated().value_counts()<jupyter_output><empty_output><jupyter_text>3.3 Data Cleaning            Change all size values to MB with a numeric type        Remove all + signs from installs and change the column type into a numeric type        Remove Dollar values from price column        Engineer genres so that they're a single value per row        Change formatting and type of column date          3.3.1 Change all size values to MB with a numeric type<jupyter_code>data["Size"].str.contains("[Mk+]").value_counts()
data[data["Size"].str.contains("+", regex=False)]<jupyter_output><empty_output><jupyter_text>Upon further investigation to row 10472 we found it to have a number of values that don't make sense. Thus, we decided to drop it.<jupyter_code>data = data.drop(10472)
data[data["Size"] == "Varies with device"].shape<jupyter_output><empty_output><jupyter_text>Values of size are either in KB or in MB. There are however string values such as varies with device. There are apps that have sizes varying per device as they follow the AAB format not the APK one. We'll change all KB values to MB and varies with device to zero.<jupyter_code>data["Size"] = data.Size.replace("Varies with device", "0")
replacement = dict(M=1, k=0.001)
data["Size"] = data["Size"].replace("[Mk]", "", regex=True).astype(float) * data["Size"].replace(replacement, regex=True).fillna(1).astype(float)<jupyter_output><empty_output><jupyter_text>3.3.2 Remove all + signs from installs and change the column type into a numeric type<jupyter_code>data["Installs"] = data["Installs"].str.replace("+", "")
data["Installs"] = data["Installs"].str.replace(",", "").astype(int)<jupyter_output><empty_output><jupyter_text>3.3.3 Remove Dollar values from price column<jupyter_code>data["Price"].unique()
data["Price"] = data["Price"].str.replace("$", "").astype(float)<jupyter_output><empty_output><jupyter_text>3.3.4 Engineer genres so that they're a single value per row<jupyter_code>genres = data["Genres"].unique()
genres
def length(x):
    return len(x.split(";"))
number_of_genres = np.vectorize(length)
number_of_genres(genres).max()<jupyter_output><empty_output><jupyter_text>We found that an app has at max 2 genres. we decided to engineer 2 new features genre1 and genre2, to replace the genre feature.<jupyter_code>data[["genre1", "genre2"]] = data["Genres"].str.split(";", expand=True)
data = data.drop(columns="Genres")<jupyter_output><empty_output><jupyter_text>We found some columns having the same genreswe decided to remove the values and replace them with a (None) in this case.     Some columns had the same value for both genre1 and genre2 we decided to replace genre2 with None in this case.<jupyter_code>data.loc[data["genre1"] == data["genre2"]].shape
data.loc[data["genre1"] == data["genre2"], "genre2"] = None
data.loc[data["genre1"] == data["genre2"]].shape<jupyter_output><empty_output><jupyter_text>3.3.5 Change formatting and type of column date<jupyter_code>date_expression = "(January|February|March|April|May|June|July|August|September|October|November|December)\s+\d{1,2},\s+\d{4}"
data["Last Updated"].str.contains(date_expression, regex=False).value_counts()
 data["Last Updated"] = pd.to_datetime(data['Last Updated'], format='%B %d, %Y')<jupyter_output><empty_output><jupyter_text>4. Dataset Visualization    Asking questions and aswering them.            Scatter Matrix        Percentage of free to paid apps        Number of installs for paid vs free apps        The dominant category<jupyter_code>plt.rcParams['figure.figsize']=(10,10)<jupyter_output><empty_output><jupyter_text>4.1 Scatter Matrix<jupyter_code>sns.pairplot(data, hue="Type", palette="GnBu")<jupyter_output><empty_output><jupyter_text>4.2 Percentage of free to paid apps<jupyter_code>size = data["Type"].value_counts()
labels = data["Type"].unique()[0:2]

colors=["#43a6cc", "#bae4bf"]

plt.pie(size, labels=labels, colors=colors, autopct='%1.1f%%')
plt.show()<jupyter_output><empty_output><jupyter_text>4.4 The dominant category<jupyter_code>categories = data["Category"].value_counts()
sns.barplot(x=categories.values, y=categories.index, palette="GnBu")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12517871326449564, 'fraction_numerical': 0.013661636219221604, 'mean_word_length': 6.355140186915888, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '92095', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 2051, 'n_tokens_neox': 1983, 'n_words': 554, 'path': '.ipynb_checkpoints/data-cleaning-checkpoint.ipynb', 'repo_name': 'grgmgd/playstore-eda', 'size': '6295'}