<jupyter_start><jupyter_text>Ho modificato il file csv perchè una riga appariva così:![title](DataSet_Problem.png) per cui ho spostato tutti i dati a destra di una cella e ho modificato la categoria cercando direttamente i dati corretti sull'app store<jupyter_code>import pandas as pd
import numpy as np
from itertools import chain
import re
app = pd.read_csv("https://raw.githubusercontent.com/mode95/progetto_foundation/master/googleplaystore.csv?token=ArUU7K8O6Zz2_MU4rR7Bw4brr_M7XXR6ks5cUF9<PASSWORD>%3D%3D")
app<jupyter_output><empty_output><jupyter_text>1. Convert the app sizes to a number and  3. Transform “Varies with device” into a missing value<jupyter_code>app["Size"] = app["Size"].str.replace("M", "")
app["Size"] = app["Size"].str.replace(",", "")
def app_size(string):
    if re.search("k$", string):
        return int(re.match("\d*", string).group()) / 1000
    else:
        re.search(".*", string)
        return re.match(".*", string).group()
app["Size"] = app.apply(lambda row: app_size(row["Size"]), axis = 1)
app = app.replace({"Varies with device" : np.nan})
app
app["Size"] = pd.to_numeric(app["Size"])   # Ho trasformato Size in una variabile numerica con unità di misura
type(app["Size"][1])                       # Megabyte<jupyter_output><empty_output><jupyter_text>2. Convert the number of installs to a number<jupyter_code>app["Installs"] = app["Installs"].str.replace("+", "")
app["Installs"] = app["Installs"].str.replace(",", "")
app["Installs"] = pd.to_numeric(app["Installs"])
app["Installs"] = pd.to_numeric(app["Installs"])
type(app["Installs"][0])<jupyter_output><empty_output><jupyter_text>4. Convert Current Ver and Android Ver into a dotted number (e.g. 4.0.3 or 4.2) (Si possono buttare via le righe dove "Android Ver" manca)<jupyter_code>app = app.dropna(subset=["Android Ver"])
app["Android Ver"] = app["Android Ver"].str.replace("[^0-9\.]", "")
app["Current Ver"] = app["Current Ver"].str.replace("[^0-9\.]", "")
app<jupyter_output><empty_output><jupyter_text>Nelle prossime quattro celle controllo che non ci siano valori differenti da dotted number in "Current Ver", per controlare "Android Ver" basta sostituire i nomi<jupyter_code>app["Current Ver_control"] = app["Current Ver"].astype(str)
def dot_num(string):
    if re.search("[a-zA-Z]", string):
        return re.match(".*", string).group()
a = app.apply(lambda row: dot_num(row["Current Ver_control"]), axis = 1).tolist()
for i in range(9477):
    if (a[i] !=  None) & (a[i] != "nan"):
        print(i)
app = app.drop("Current Ver_control", axis = 1)<jupyter_output><empty_output><jupyter_text>5. Remove the duplicates  Siccome sull'app store non possono esistere due app con lo stesso nome, lo standardizzo e rimuovo tutte le copie<jupyter_code>app.insert(1, "App_low", app["App"].str.lower())
app["App_low"] = app["App_low"].str.replace(" ", "")
ids = app["App_low"]
app[ids.isin(ids[ids.duplicated()])]               #Mostro i duplicati 
app = app.drop_duplicates(subset= "App_low", keep = "last")       # tengo l'ultimo che è la versione dell'app 
app                                                               # più aggiornata
app = app.drop("App_low", axis=1)
app<jupyter_output><empty_output><jupyter_text>6. For each category, compute the number of apps<jupyter_code>app.groupby("Category")["App"].count()<jupyter_output><empty_output><jupyter_text>7. For each category, compute the average rating<jupyter_code>app.groupby("Category")["Rating"].mean()<jupyter_output><empty_output><jupyter_text>8. Create two dataframes: one for the genres and one bridging apps and genres. So that, for instance, the app Pixel Draw - Number Art Coloring Book appears twice in the bridging table, once for Art & Design, once for Creativity<jupyter_code># lista dei generi separati da ;
def chainer(s):
    return list(chain.from_iterable(s.str.split(';')))

# Calcolo lunghezza split
lens = app["Genres"].str.split(";").map(len)
"""bridging = pd.DataFrame({"App":np.repeat(app["App"], lens), 
                       "Category" : np.repeat(app["Category"], lens),
                       "Rating" : np.repeat(app["Rating"], lens),
                       "Reviews" : np.repeat(app["Reviews"], lens),
                       "Size" : np.repeat(app["Size"], lens),
                       "Installs" : np.repeat(app["Installs"], lens),              questo è nel caso serva un 
                       "Type" : np.repeat(app["Type"], lens),                      dataframe completo
                       "Price" : np.repeat(app["Price"], lens),
                       "Content Rating" : np.repeat(app["Content Rating"], lens),
                       "Genres" : chainer(app["Genres"]),
                       "Last Updated" : np.repeat(app["Last Updated"], lens),
                       "Current Ver" : np.repeat(app["Current Ver"], lens),
                       "Android Ver" : np.repeat(app["Android Ver"], lens)
                      })
bridging """
genres = pd.DataFrame({"Genres" : chainer(app["Genres"])})
genres.head()
app_genres = pd.DataFrame({"App" : np.repeat(app["App"],lens),
                           "Genres" : chainer(app["Genres"])})
app_genres.head()<jupyter_output><empty_output><jupyter_text>9. For each genre, create a new column of the original dataframe. The new columns must have boolean values (True if the app has a given genre)<jupyter_code>binary = pd.get_dummies(app["Genres"]).astype(bool)
app = app.join(binary)
app<jupyter_output><empty_output><jupyter_text>10. For each genre, compute the average rating. What is the genre with highest average?<jupyter_code>app.groupby("Genres")["Rating"].mean().sort_values(ascending = False)<jupyter_output><empty_output><jupyter_text>11. For each app, compute the approximate income, obtain as a product of number of installs and price.<jupyter_code>app["Price"] = app["Price"].str.replace("$", "")
app["Price"] = pd.to_numeric(app["Price"])
app["Income"] = app["Price"] * app["Installs"]<jupyter_output><empty_output><jupyter_text>12. For each app, compute its minimum and maximum Sentiment_polarity<jupyter_code>app_reviews = pd.read_csv("https://raw.githubusercontent.com/mode95/progetto_foundation/master/googleplaystore_user_reviews.csv?token=<PASSWORD>")
app_reviews.head()
app_reviews.groupby("App")["Sentiment_Polarity"].max().head()
app_reviews.groupby("App")["Sentiment_Polarity"].min().head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15634771732332708, 'fraction_numerical': 0.008755472170106316, 'mean_word_length': 4.18815896188159, 'pattern_counts': {'":': 1, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '35326', 'license': 'CC0-1.0', 'n_tokens_mistral': 2147, 'n_tokens_neox': 2101, 'n_words': 484, 'path': 'progetto_foundation/Progetto.ipynb', 'repo_name': 'mode95/Data-Science', 'size': '6427'}