<jupyter_start><jupyter_text>1. Cleaning Data<jupyter_code>weather = pd.read_csv("data/berlin_weather.csv")
weather.head(4)<jupyter_output><empty_output><jupyter_text>Looks like the value "-9999" is used to indicate a missing value.  Let's fix this to the normal NaN<jupyter_code>weather = weather.applymap(lambda x: np.nan if x == -9999 else x)
weather.head(4)<jupyter_output><empty_output><jupyter_text>Now, the 'DATE' column does not quite look like date strings.  Let's fix this next:<jupyter_code>weather['DATE'] = weather['DATE'].map(lambda x: datetime.strptime(str(x), '%Y%m%d').date())
weather.head(4)<jupyter_output><empty_output><jupyter_text>2.  Missing Data<jupyter_code>weather.notnull().head()<jupyter_output><empty_output><jupyter_text>N/A Strategy 1: Naive dropping missing valuesIf dropping any records with **any null**, we get:<jupyter_code>weather.dropna()<jupyter_output><empty_output><jupyter_text>N/A Strategy 2:  dropping rows with all missing valuesIf we are only interested in dropping records when all values in that record (row) are missing:<jupyter_code>weather.dropna(how='all')<jupyter_output><empty_output><jupyter_text>N/A Strategy 3:  dropping columns with all missing valuesWe can also drop columns where no value in that column is not missing<jupyter_code>weather.dropna(how='all', axis=1)<jupyter_output><empty_output><jupyter_text>N/A Strategy 4:  dropping with a threshold on missing valuesOnly drop row or column with missing value exceeding a threshold<jupyter_code># let's see how big our dataframe is

weather.shape<jupyter_output><empty_output><jupyter_text>Let's set a threshold for column missing values at 1000<jupyter_code>weather.dropna(thresh=1000, axis=1)<jupyter_output><empty_output><jupyter_text>3.  Duplicate records<jupyter_code>weather['DATE'].dtype<jupyter_output><empty_output><jupyter_text>Let's use the DATE column as the index for the dataframe<jupyter_code>weather = weather.set_index(pd.DatetimeIndex(weather['DATE']))<jupyter_output><empty_output><jupyter_text>Now we can test whether some indices are duplicated<jupyter_code>weather.index.duplicated()<jupyter_output><empty_output><jupyter_text>Definitely look like we have duplicate indices.Let's also look at whether we have issues at the STATION_NAME:<jupyter_code>weather['STATION_NAME'].value_counts()
weather.index.drop_duplicates().sort_values()<jupyter_output><empty_output><jupyter_text>4. ResamplingWe want to use "resampling" to fill in missing data.  The resampling frequency is "calendar day" or 'D'<jupyter_code>weather.groupby('STATION_NAME').resample('D').mean().head()<jupyter_output><empty_output><jupyter_text>Reference to additional sampling rule can be found at:        http://stackoverflow.com/questions/17001389/pandas-resample-documentation  5. Performance<jupyter_code>weather.shape<jupyter_output><empty_output><jupyter_text>Let's using timing code to check performance.<jupyter_code>weather.head()
#
# (1) use 'applymap' for all the column series of weather
#
%timeit weather.applymap(lambda x: np.nan if x == -9999 else x)
def return_row(row):
    return [x if x != -9999 else np.nan for x in row]
#
# (2) use 'apply' for all the rows of weather
#
%timeit weather.apply(return_row)
#
# (3) use 'map' to iterate over the columns
#
def iter_columns(df):
    for col in df.columns:
        df[col] = df[col].map(lambda x: x if x != -9999 else np.nan)
    return df
new_weather = weather.copy()
%timeit iter_columns(new_weather)<jupyter_output>1 loop, best of 3: 1.37 s per loop<jupyter_text>6.  ProfilingTo learn about how much time is spent on each portion of the program, a "profiler" is useful.Here is a timing profiling example:<jupyter_code>%prun weather.applymap(lambda x: np.nan if x == -9999 else x)<jupyter_output><jupyter_text>Let's look at how memory is consumed by our algorithms<jupyter_code>%load_ext memory_profiler
%memit weather.applymap(lambda x: np.nan if x == -9999 else x)<jupyter_output>peak memory: 198.21 MiB, increment: 82.08 MiB
{'fraction_non_alphanumeric': 0.10963704630788486, 'fraction_numerical': 0.01902377972465582, 'mean_word_length': 7.040241448692153, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '395750', 'license': 'MIT', 'n_tokens_mistral': 1439, 'n_tokens_neox': 1343, 'n_words': 359, 'path': 'lecture07.big.data/lecture07.1.clean.data.ipynb', 'repo_name': 'xjdong/datascience', 'size': '3995'}