<jupyter_start><jupyter_text>数据读取，所用数据集为“Wine Reviews”<jupyter_code>data_1 = pd.read_csv("./dataset/WineReviews/winemag-data-130k-v2.csv", index_col=0)
data_2 = pd.read_csv("./dataset/WineReviews/winemag-data_first150k.csv", index_col=0)
data = pd.concat([data_1, data_2], axis=0)
data.shape<jupyter_output><empty_output><jupyter_text>Data review- We can find that 'country','designation','province','region_1','region_2','taster_name','taster_twitter_handle','variety' and 'winery' are nominal attibutes. There are no sequence relation between attibutes.- 'points' and 'price' belong to numeric attribute.<jupyter_code>data.head(5)
data.isnull().sum()<jupyter_output><empty_output><jupyter_text>处理空值<jupyter_code>def nullplot(data):
    '''
    data: a dataframe on which we want to perform null plot
    '''
    df_null = data.isna()
    nulls_percent_per_col = df_null.sum(axis=0) / len(data)
    
    plt.figure(dpi=600)
    fig, (ax_1, ax_2) = plt.subplots(nrows=2, ncols=1, sharex=True, figsize=(8, 10))
    
    pla = matplotlib.cm.get_cmap("plasma")
    colormap = matplotlib.colors.ListedColormap([pla(0), 'gold'])
    sns.heatmap(data.isnull(), cmap=colormap, ax=ax_1, cbar=False, yticklabels=False)
    nulls_percent_per_col.plot(kind="bar", color="gold", x=nulls_percent_per_col, y=nulls_percent_per_col.index, 
                               ax=ax_2, label="Null value percent")
    ax_2.set_ylim((0,1))
    plt.show()<jupyter_output><empty_output><jupyter_text>下图展示了每个特征中的空值情况（空值以黄色标注）<jupyter_code>nullplot(data)<jupyter_output><empty_output><jupyter_text>直接将含有空值的样本删除<jupyter_code>data_4 = data.copy()
data_4 = data_4.dropna(axis=0)
nullplot(data_4)
data_4.isnull().sum()
data_4.shape<jupyter_output><empty_output><jupyter_text>删除“description”列，因为该列表示评论家对酒的评鉴，不适合用来进行关联规则 挖掘<jupyter_code>data_4 = data_4.drop(['description'], axis=1)
data_4.shape
data_4.head()
wineTuples = list(zip(*[data_4[c].values.tolist() for c in data_4]))<jupyter_output><empty_output><jupyter_text>由于数据各特征的属性值较多，当选择较大的支持度时只能够找到极少的关联规则，因此在后续的频繁模式挖掘中，我使用了0.25的支持度以及0.9的置信度<jupyter_code>from efficient_apriori import apriori
itemsets, rules = apriori(wineTuples, min_support=0.5, min_confidence=0.9)
print(rules)<jupyter_output>[{California} -> {US}]<jupyter_text>可以看出在此数据集中，绝大部分关联规则主要有关于两个方面1. 与品鉴师的身份信息相关，例如推特ID、姓名往往是一一映射的，所以二者同时出现的情况较多，也即置信度较高，如{Paul Gregutt} -> {@paulgwine }, {@paulgwine } -> {Paul Gregutt}2. 品鉴师的身份与酒的产地之间的关系也较为紧密，比如关联规则{@vboone, Virginie Boone} -> {US}说明了该品鉴师往往会对美国产地的酒较为钟爱<jupyter_code>itemsets, rules = apriori(wineTuples, min_support=0.25, min_confidence=0.9)
print(rules)<jupyter_output>[{Paul Gregutt} -> {@paulgwine }, {@paulgwine } -> {Paul Gregutt}, {@paulgwine } -> {US}, {Paul Gregutt} -> {US}, {@vboone} -> {California}, {@vboone} -> {US}, {Virginie Boone} -> {@vboone}, {@vboone} -> {Virginie Boone}, {California} -> {US}, {Virginie Boone} -> {California}, {Virginie Boone} -> {US}, {Washington} -> {US}, {<NAME>, US} -> {@paulgwine }, {@paulgwine , US} -> {<NAME>}, {@paulgwine , <NAME>utt} -> {US}, {Paul Gregutt} -> {@paulgwine , US}, {@paulgwine } -> {Paul Gregutt, US}, {@vboone, US} -> {California}, {@vboone, California} -> {US}, {@vboone} -> {California, US}, {California, Virginie Boone} -> {@vboone}, {@vboone, Virginie Boone} -> {California}, {@vboone, California} -> {Virginie Boone}, {Virginie Boone} -> {@vboone, California}, {@vboone} -> {California, Virginie Boone}, {US, Virginie Boone} -> {@vboone}, {@vboone, Virginie Boone} -> {US}, {@vboone, US} -> {Virginie Boone}, {Virginie Boone} -> {@vboone, US}, {@vboone} -> {US, Virginie Boone}, {US, V[...]<jupyter_text>下面展示了数据集中的频繁项集可以看出频繁项集的结果与上述关联规则一致，某些品鉴师出现的较为频繁，如<NAME>以及Virginie Boone，其占据了相当一部分的数据样本；此外品鉴师的姓名与推特ID同时出现的较多；某些品鉴师与特定国家同时出现的结果也较多。<jupyter_code>print(itemsets)
def lift(pa, pb, pab):
    '''计算提升度'''
    return pab / (pa * pb)<jupyter_output><empty_output><jupyter_text>品鉴师Virginie Boone与California之间的提升度大于1，说明二者之间是正相关的<jupyter_code>lift_of_Boone_and_California = lift(6422/22387, 12900/22387, 6422/22387)
print(lift_of_Boone_and_California)
import math
def consine(pa, pb, pab):
    '''计算余弦度量'''
    return pab / math.sqrt(pa * pb)<jupyter_output><empty_output><jupyter_text>余弦度量越大，说明二者之间的联系越紧密，这个结果与上述的提升度的结果一致<jupyter_code>consine_of_Boone_and_California = consine(6422/22387, 12900/22387, 6422/22387)
print(consine_of_Boone_and_California)<jupyter_output>0.7055703064644522<jupyter_text>通过词云的方式将频繁项集展示出来<jupyter_code>with open("txt_save.txt",'w') as file:
    for k, v in itemsets.items():
        for k1, v1 in v.items():
            for i in range(v1):
                file.write(str(k1).strip('(),') + ' ')
#                 list_word[str(k1).strip('(),')] = float(v1)
from wordcloud import WordCloud
# from scipy.misc import imread
import matplotlib.pyplot as plt
with open("txt_save.txt","r") as file:
        txt=file.read()
word=WordCloud(background_color="white").generate(txt)

plt.imshow(word)    #使用plt库显示图片
plt.axis("off")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16673354735152487, 'fraction_numerical': 0.028892455858747994, 'mean_word_length': 6.47376311844078, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 84, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '140397', 'license': 'MIT', 'n_tokens_mistral': 2411, 'n_tokens_neox': 2402, 'n_words': 296, 'path': 'Wine reviews(Data mining).ipynb', 'repo_name': 'Unintented/data-mining-practice', 'size': '5005'}