<jupyter_start><jupyter_text>处理缺失值  缺失值的表示  None：Python的表示法<jupyter_code>import numpy as np
import pandas as pd

vals1 = np.array([1, None, 3, 4])
vals1
vals1.sum()<jupyter_output><empty_output><jupyter_text>NaN<jupyter_code>vals2 = np.array([1, np.nan, 3, 4])
vals2.dtype
vals2.sum()
1 + np.nan
np.nansum(vals2)<jupyter_output><empty_output><jupyter_text>Pandas 中的 NaN 和 None<jupyter_code>pd.Series([1, np.nan, 2, None])
x = pd.Series(range(2), dtype=int)
x
x[0] = None
x<jupyter_output><empty_output><jupyter_text>缺失值的操作  判断缺失值：``isnull``<jupyter_code>data = pd.Series([1, np.nan, 'hello', None])
data.isnull()
data[data.notnull()]<jupyter_output><empty_output><jupyter_text>忽略缺失值：``dropna()``<jupyter_code>data.dropna()
df = pd.DataFrame([[1,      np.nan, 2],
                   [2,      3,      5],
                   [np.nan, 4,      6]])
df
df.dropna() # 默认去掉含有na的行
df.dropna(axis='columns')
df[3] = np.nan
df
df.dropna(axis='columns', how='all') # how 默认为 any
                                     # 改为all，只删除全部为NaN的行或者列
df.dropna(axis='rows', thresh=3) # 使用threshold<jupyter_output><empty_output><jupyter_text>填补缺失值：``fillna()``<jupyter_code>data = pd.Series([1, np.nan, 2, None, 3], index=list('abcde'))
data
data.fillna(0)
data.fillna(method='ffill') # forward-fill
data.fillna(method='bfill') # back-fill
df
df.fillna(method='ffill', axis=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1781317885590152, 'fraction_numerical': 0.02389572773352643, 'mean_word_length': 4.905982905982906, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '32992', 'license': 'Apache-2.0', 'n_tokens_mistral': 636, 'n_tokens_neox': 647, 'n_words': 88, 'path': 'Optional-01-Using-Pandas/04-Handling-Missing-Data/04-Handling-Missing-Data.ipynb', 'repo_name': 'mtianyan/Mtianyan-Play-with-Machine-Learning-Algorithms', 'size': '1381'}