<jupyter_start><jupyter_text>Read data<jupyter_code>data_1 = pd.read_csv("./dataset/WineReviews/winemag-data-130k-v2.csv", index_col=0)
data_2 = pd.read_csv("./dataset/WineReviews/winemag-data_first150k.csv", index_col=0)
data = pd.concat([data_1, data_2], axis=0)
data.shape<jupyter_output><empty_output><jupyter_text>Data review- We can find that 'country','designation','province','region_1','region_2','taster_name','taster_twitter_handle','variety' and 'winery' are nominal attibutes. There are no sequence relation between attibutes.- 'points' and 'price' belong to numeric attribute.<jupyter_code>data.head(5)
data.describe()<jupyter_output><empty_output><jupyter_text>First of all, I will present the frequency of each category in each nominal attribute.<jupyter_code>Counter(data["country"])<jupyter_output><empty_output><jupyter_text>!!! In order to simplify the result, the following similar results are not implemented.<jupyter_code>Counter(data["designation"])
Counter(data["province"])
Counter(data["region_1"])
Counter(data["region_2"])
Counter(data["taster_name"])
Counter(data["taster_twitter_handle"])
Counter(data["variety"])
Counter(data["winery"])<jupyter_output><empty_output><jupyter_text>For numeric attributes, five-number summary and the count of null value will be presented.> points:  Minimum:80   Q1:86   Median:88   Q3:90   Maximum:100> price:  Minimum:4   Q1:16   Median:25   Q3:40   Maximum:3300<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>> The count of null value in attribute 'points' is 0.> The count of null value in attribute 'price' is 22691.<jupyter_code>data[["points", "price"]].isnull().sum()<jupyter_output><empty_output><jupyter_text>The data distribution of attribute 'points' basically conforms to the normal distribution.<jupyter_code>plt.figure(dpi=300)
plt.hist(x=data["points"], bins="auto")
plt.xlabel("points")
plt.ylabel("Frequency")<jupyter_output><empty_output><jupyter_text>The data are mainly distributed between 0 and 135.<jupyter_code>plt.figure(dpi=300)
plt.hist(x=data["price"], bins="auto")
plt.xlabel("price")
plt.ylabel("Frequency")
plt.ylim(0, 10000)<jupyter_output><empty_output><jupyter_text>Boxplot for numeric attributes.> For attribute 'points', IQR equals to 6. It's Q3 is 90, meaning that all the samples which points is higher than 96 will be regarded as outlier.<jupyter_code>plt.figure(dpi=300)
plt.boxplot(data["points"],patch_artist=True,boxprops = {'color':'black','facecolor':'#9999ff'},
 
            flierprops = {'marker':'o','markerfacecolor':'red','color':'black'},
 
            meanprops = {'marker':'D','markerfacecolor':'indianred'},
            medianprops = {'linestyle':'--','color':'orange'})<jupyter_output><empty_output><jupyter_text>Boxplot for attribute 'price'.<jupyter_code>import seaborn as sns
plt.figure(dpi=300)
# sns.boxplot(data["price"])
data.boxplot(column="price")
data.isnull().sum()<jupyter_output><empty_output><jupyter_text>At this part, I will use z-scores to detect extreme outliers which absolute value of z-scores are higher than 10.> Notably, everyone knows that wines could be extremely expensive. As a result, we shouldn't treat these samples as measuring errors.<jupyter_code>from scipy import stats
df_price = pd.DataFrame(data["price"])
price_zscore = df_price.apply(stats.zscore)
outlier_mask = ((price_zscore > 10).any(axis=1)).values + ((price_zscore < -10).any(axis=1)).values
df_price.loc[outlier_mask, :]<jupyter_output><empty_output><jupyter_text>Handle null values<jupyter_code>def nullplot(data):
    '''
    data: a dataframe on which we want to perform null plot
    '''
    df_null = data.isna()
    nulls_percent_per_col = df_null.sum(axis=0) / len(data)
    
    plt.figure(dpi=600)
    fig, (ax_1, ax_2) = plt.subplots(nrows=2, ncols=1, sharex=True, figsize=(8, 10))
    
    pla = matplotlib.cm.get_cmap("plasma")
    colormap = matplotlib.colors.ListedColormap([pla(0), 'gold'])
    sns.heatmap(data.isnull(), cmap=colormap, ax=ax_1, cbar=False, yticklabels=False)
    nulls_percent_per_col.plot(kind="bar", color="gold", x=nulls_percent_per_col, y=nulls_percent_per_col.index, 
                               ax=ax_2, label="Null value percent")
    ax_2.set_ylim((0,1))
    plt.show()<jupyter_output><empty_output><jupyter_text>The drawing below presents the null value in each attribute.<jupyter_code>nullplot(data)<jupyter_output><empty_output><jupyter_text>Eliminate null values --- drop rowsI will drop all rows containing "too much" null values.<jupyter_code>data_4 = data.copy()
data_4 = data_4.dropna(axis=0, thresh=10)
nullplot(data_4)<jupyter_output><empty_output><jupyter_text>Replace null values with most common value of the column.<jupyter_code>data_5 = data.copy()
def find_nost_common_value(series):
    '''
    To find the most common value of the column
    '''
    return series.value_counts().index[0]
most_common_value = data_5.apply(find_nost_common_value, axis=0)
print("Most common value of each column:\n", most_common_value)
data_5[data_5.columns.values] = data_5[data_5.columns.values].fillna(most_common_value)
nullplot(data_5)<jupyter_output>Most common value of each column:
 country                                                                 US
description              A little bit funky and unsettled when you pop ...
designation                                                        Reserve
points                                                                  87
price                                                                   20
province                                                        California
region_1                                                       Napa Valley
region_2                                                     Central Coast
taster_name                                                     <NAME>
taster_twitter_handle                                           @vossroger
title                    <NAME> NV Sonoma Brut Sparkling (Sonoma...
variety                                                         Pinot Noir
winery                                                     Willia[...]<jupyter_text>The missing values are filled by the correlation of attributes.Intuitively, 'taster_name' and 'taster-twitter-handle' have some implicit correlation. One taster may have several twitter IDs but these twitter IDs correspond to one single person. So it is significant to merge these two columns into one.<jupyter_code>data_6 = data.copy()
only_name = data_6.loc[data_6['taster_twitter_handle'].isnull() & data_6['taster_name'].notna(), 
                   ['taster_name', 'taster_twitter_handle']]
num_only_name = len(only_name.index)

only_twitter = data_6.loc[data_6['taster_name'].isnull() & data_6['taster_twitter_handle'].notna(), 
                      ['taster_name', 'taster_twitter_handle']]
num_only_twitter = len(only_twitter.index)

print(f'rows containing a name but no twitter handle: {num_only_name}'
      + f'\nrows containing a twitter handle but no taster name: {num_only_twitter}')<jupyter_output>rows containing a name but no twitter handle: 4969
rows containing a twitter handle but no taster name: 0<jupyter_text>So, we can simply drop the column 'taster_twitter_handle' as it doesn't provide any further information.<jupyter_code>data_6 = data_6.drop("taster_twitter_handle", axis=1)
data_6['taster_name'] = data_6['taster_name'].fillna('Unknown')
nullplot(data_6)<jupyter_output><empty_output><jupyter_text>The missing values are filled by the similarity between data objects.I will use attributes 'country','points','province','winery' to predict price.<jupyter_code>data_7 = data.copy()
data_feature = data_7[['country', 'points', 'province', 'winery', 'price']]
data_feature = data_feature.loc[data_feature['country'].notnull(), :]
data_feature = data_feature.loc[data_feature['province'].notnull(), :]
data_feature.head(5)<jupyter_output><empty_output><jupyter_text>Encode nominal attributes and normalization<jupyter_code>from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
data_feature['country'] = le.fit_transform(data_feature['country'])
data_feature['province'] = le.fit_transform(data_feature['province'])
data_feature['winery'] = le.fit_transform(data_feature['winery'])
print("After encode:\n", data_feature.head(5))

data_feature[['country', 'points', 'province', 'winery']] = data_feature[['country', 'points', 'province', 'winery']].apply(lambda x: (x - x.min()) / (x.max() - x.min()))
print("After normalization:\n", data_feature.head(5))<jupyter_output>After encode:
    country  points  province  winery  price
0       23      87       377   13219    NaN
1       35      87       116   14763   15.0
2       46      87       305   14836   14.0
3       46      87       245   16411   13.0
4       46      87       305   16669   65.0
After normalization:
     country  points  province    winery  price
0  0.469388    0.35  0.770961  0.689423    NaN
1  0.714286    0.35  0.237219  0.769949   15.0
2  0.938776    0.35  0.623722  0.773756   14.0
3  0.938776    0.35  0.501022  0.855899   13.0
4  0.938776    0.35  0.623722  0.869354   65.0<jupyter_text>Split training set and prediction set<jupyter_code>data_price_known = data_feature.loc[data_feature['price'].notnull(), :]
data_price_unknown = data_feature.loc[data_feature['price'].isnull(), :]

X_train = data_price_known.drop('price', axis=1)
y_train = data_price_known['price']

X_pred = data_price_unknown.drop('price', axis=1)<jupyter_output><empty_output><jupyter_text>Train model and predict<jupyter_code>from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
lr.fit(X=X_train, y=y_train)
X_pred['price'] = lr.predict(X_pred)
data_all = pd.concat([data_price_known,  X_pred], axis=0, ignore_index=True)
nullplot(data_all)
data_all.head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11977263499796996, 'fraction_numerical': 0.037758830694275276, 'mean_word_length': 3.9612286002014097, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 91, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '563354', 'license': 'MIT', 'n_tokens_mistral': 3380, 'n_tokens_neox': 3131, 'n_words': 691, 'path': 'Wine reviews.ipynb', 'repo_name': 'Unintented/data-mining-practice', 'size': '9863'}