<jupyter_start><jupyter_text>数据预处理-2- 班级:2017211314- 学号:2017213508- 学生:蒋雪枫 作业3 **一些额外的Trick:**比如想要产生新的一列来统计同一时间内所有地方的PM指数和df['sumOfPM']=df['PM_US_POST']+df['PM_Nongzhanguan']+df['PM_Dongsi']+df['PM_Dongsihuan']比如想要按照某一列排序,升序def top(df,n=5,colunm='PRES'):    return df.sort_values(by=column)[-n:]apply函数是`pandas`里面所有函数中自由度最高的函数。该函数如下：DataFrame.apply(func, axis=0, broadcast=False, raw=False...)该函数最有用的是第一个参数，这个参数是函数，相当于C/C++的函数指针。这次预处理实验和上次预处理实验后,我明显感受到直接按列操作比遍历每一个csv中的cell效率高得多,减小了IO时间<jupyter_code>import numpy as np
import pandas as pd
import time
import matplotlib.pyplot as plt

df = pd.read_csv("BeijingPM20100101_20151231.csv",encoding='utf-8')
df.describe()
todo = ['HUMI', 'PRES', 'TEMP']
#进行线性插值
for each in todo:
    df[each]=df[each].interpolate(method="linear")
print("将dataframe其中一列抽出来是什么类型?{}".format(type(df['HUMI'])))
for each in todo:
    mean = df[each].mean()
    stdDoub = df[each].std()
    df[each]= df[each].apply(lambda x : mean + stdDoub*2 if x>mean + stdDoub * 2 
                                  else ( mean-2 * stdDoub if x < mean -  2 * stdDoub else x))

todo = ['PM_Dongsi','PM_Dongsihuan','PM_Nongzhanguan']
print("处理前")
#   PM:6,7,8,9 || HUMI PRES TEMP:11,12,13 || cbwd:14
print(df.iloc[30590:30600,6:9])
for each in todo:
    df[each] = df[each].apply(lambda x: 500 if x > 500 else x)
#     df['avr']+=df[each]
#     df['counter']+=(int)(int(df[each])!=0)
#     df['avr']=df['avr']/df['counter']
print("处理后")
print(df.iloc[30590:30600,6:9])
#下面第一种方式比较直观,但如果出现两个连续的cv,怎么处理?  
# Excel里面也告诉我们的确出现了
# for i in range(len(df['cbwd'])):
#     if df['cbwd'][i] == 'cv':
#         df.at[i:i,'cbwd'] =  df['cbwd'][i+1]
print("对CV处理前")
print(df.iloc[20:25,8:15])
print("对CV处理后")
#TODO:采用bfill替换方式
df['cbwd']=df['cbwd'].replace('cv',method='bfill')
print(df.iloc[20:25,8:15])
df.describe() 
df.to_csv("result.csv")<jupyter_output><empty_output><jupyter_text>比较前后的描述信息,可以较为明显地看出有关标准差采取的措施带来的改变  作业4<jupyter_code>from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
features = ['DEWP', 'TEMP']
df[features] = scaler.fit_transform(df[features])
df.head()
df.plot.scatter(x=['DEWP'],y=['TEMP'],s=10,figsize = (20,20))
from sklearn.preprocessing import StandardScaler
ss = StandardScaler()
features = ['DEWP', 'TEMP']
df[features] = ss.fit_transform(df[features])
df.head()
df.plot.scatter(x=['DEWP'],y=['TEMP'],s=10,figsize = (20,20))
sections = [0,50,100,150,200,300,1200] #划分为不同长度 的区间 
section_names=["green","yellow","orange","red","purple", "Brownish red"] #设置每个区间的标签
temp=df.groupby(["year","month","day"]).mean()
df1=pd.DataFrame(temp)
result = pd.cut(df1['avg'],sections,labels=section_names) 
print(pd.value_counts(result))<jupyter_output>green           699
yellow          655
orange          401
red             196
purple          153
Brownish red     51
Name: avg, dtype: int64
{'fraction_non_alphanumeric': 0.1917226742129466, 'fraction_numerical': 0.055182171913689425, 'mean_word_length': 6.0173697270471465, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '527152', 'license': 'MIT', 'n_tokens_mistral': 1439, 'n_tokens_neox': 1422, 'n_words': 169, 'path': 'Data PreProcess/Interpolation&ExceptionHandling/Homework34.ipynb', 'repo_name': 'Sprinter1999/PythonAdvanced', 'size': '2827'}