<jupyter_start><jupyter_text>实验二  预处理与可视化 实验目的+ 学习使用numpy模块进行数据预处理；+ 学习使用Matplotlib模块进行数据可视化； 实验要求+ 请在实验报告中说明以上两步操作的依据或思路；+ 请上传代码与处理后的数据，连同实验报告一起打包； 实验报告实验完成后要求提交 **实验报告** 与 **相应的实验结果**将实验报告和相关实验结果打包上传至服务器上相关目录下：> /上传作业/刘昆宏/数据挖掘/实验2/ 实验报告请在实验完成后*第二个双周周日前*上传<jupyter_code>%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>实验内容1 数据预处理1. 读取horse-colic.data数据文件，根据horse-colic.names文件中对属性的说明进行缺失值填充；2. 根据需要，对相关属性进行数据归一化操作；  1. 对`horse-colic.names`文件进行分析，整理出数据字典<jupyter_code># 数据字典
data_dict = {
    'surgery?': 'category',
    'Age':'category',
    'Hospital Number':'int64',
    'rectal temperature':'float64',
    'pulse':'float64',
    'respiratory rate':'float64',
    'temperature of extremities':'float64',
    'peripheral pulse':'category',
    'mucous membranes':'category',
    'capillary refill time':'category',
    'pain':'category',
    'peristalsis':'category',
    'abdominal distension':'category',
    'nasogastric tube':'category',
    'nasogastric reflux':'category',
    'nasogastric reflux PH':'float64',
    'rectal examination - feces':'category',
    'abdomen':'category',
    'packed cell volume':'float64',
    'total protein':'float64',
    'abdominocentesis appearance':'category',
    'abdomcentesis total protein':'float64',
    'outcome':'category',
    'surgical lesion?':'category',
    'type of lesion 1':'object',
    'type of lesion 2':'object',
    'type of lesion 3':'object',
    'cp_data':'category'
}
# 列名
col_name = list(data_dict.keys())
assert len(col_name) == 28<jupyter_output><empty_output><jupyter_text>2. 通过`d.read_table`方法读取文件`horse-colic.data`<jupyter_code># 读取文件
df = pd.read_table(
    'horse-colic.data',
    sep=' ',
    header=None,
    na_values='?',
    names=col_name,
    dtype=data_dict
)
df<jupyter_output><empty_output><jupyter_text>3. 对缺失值进行处理。1. 抛弃过多缺失值的样本(缺失率大于10%，则抛弃该行)2. 对于数值型缺失值，用均值填充，这样选择的原因是考虑之后的归一化3. 对于类别型，保留NaN<jupyter_code># the threshold of non_null values
threshold = int(df.shape[1] * 0.9)
df.dropna(thresh=threshold, inplace=True)
df.fillna(df.mean(), inplace=True)<jupyter_output><empty_output><jupyter_text>4. 分析需要归一化的列，并对其逐一归一化由于在上面分析数据字典时，已经对需要归一化的 **数值类** 数据定义成了 *float64* 类型，所以只需要提取出它们的key就能生成待归一化列的列表<jupyter_code>def z_score_normalize(series):
    '''
    Params:  a pandas serious

    Return: normalized version of the series
    '''
    return (series-series.mean())/series.std()
# 需要归一化的列表
to_be_normalized_cols = [col for col in col_name if data_dict[col] == 'float64']

# 对收集的列进行归一化操作
print('columns normalized:\n')
for col in to_be_normalized_cols:
    print('\t' + col)
    df[col] = z_score_normalize(df[col])

df<jupyter_output>columns normalized:

	rectal temperature
	pulse
	respiratory rate
	temperature of extremities
	nasogastric reflux PH
	packed cell volume
	total protein
	abdomcentesis total protein<jupyter_text>5. 将预处理过的数据保存到文件`preprocessed.csv`中备用<jupyter_code>df.to_csv('preprocess.csv')<jupyter_output><empty_output><jupyter_text>实验内容二     数据可视化+ 对iris.data进行可视化操作，分析各个属性在不同类别中分布是否有显著差异  1. 按照内容1的步骤进行数据预处理由于任务仅仅需要可视化，为了直观，这里不进行归一化。<jupyter_code># 数据字典
data_dict_iris = {
    'Sepal.Length':'float64',
    'Sepal.Width':'float64',
    'Petal.Length':'float64',
    'Petal.Width':'float64',
    'Type':'object'
}
# 列名
col_name_iris = list(data_dict_iris.keys())
to_be_normalized_cols_iris = col_name_iris[:4]
assert len(col_name_iris) == 5
for c in to_be_normalized_cols_iris:
    assert data_dict_iris[c] == 'float64'

# 读取文件
df_iris = pd.read_table(
    'iris.data.txt',
    sep=',',
    header=None,
    names=col_name_iris,
    dtype=data_dict_iris
)
df_iris<jupyter_output><empty_output><jupyter_text>2. 进行数据可视化首先实现对某种类型的绘图逻辑<jupyter_code>def plot_for_attributes(SL,SW,PL,PW,T):
    '''
    Params: lists of Sepal.Length, Sepal.Width, Petal.Length, Petal.Width of a given Type

    Return: None, but plot a figure
    '''
    plt.figure(figsize=(8,5))
    plt.title(T)

    plt.plot(np.arange(len(SL)), SL, label='Sepal.Length', color='red')
    plt.plot(np.arange(len(SW)), SW, label='Sepal.Width', color='yellow')
    plt.plot(np.arange(len(PL)), PL, label='Petal.Length', color='blue', linestyle='--')
    plt.plot(np.arange(len(PW)), PW, label='Petal.Width', color='green', linestyle='--')

    plt.xlabel('Individual NO.')
    plt.ylabel('cm.')

    plt.legend()
    plt.show()<jupyter_output><empty_output><jupyter_text>为了实现类别对比，需要对datafram按照Type进行分割对于每个子数据集，分别绘图<jupyter_code>types = df_iris['Type'].drop_duplicates().tolist()
for type in types:
    data = df_iris.loc[df_iris['Type'] == type]
    SL = data['Sepal.Length']
    SW = data['Sepal.Width']
    PL = data['Petal.Length']
    PW = data['Petal.Width']
    plot_for_attributes(SL, SW, PL, PW, type)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14971810398830654, 'fraction_numerical': 0.012737523491334307, 'mean_word_length': 5.543715846994536, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '594813', 'license': 'MIT', 'n_tokens_mistral': 2286, 'n_tokens_neox': 2321, 'n_words': 289, 'path': 'preprocess and visualization/lab2.ipynb', 'repo_name': 'AiRanthem/data-mining-playground', 'size': '4789'}