<jupyter_start><jupyter_text>Data Science is Software Developer lifehacks for the Jupyter Data Scientist Section 4:  Don't let other people break your toys  Motivation > "Many machine learning algorithms have a curious property: they are robust against bugs. Since they’re designed to deal with noisy data, they can often deal pretty well with noise caused by math mistakes as well. If you make a math mistake in your implementation, the algorithm might still make sensible-looking predictions. This is bad news, not good news. It means bugs are subtle and hard to detect. Your algorithm might work well in some situations, such as small toy datasets you use for validation, and completely fail in other situations — high dimensions, large numbers of training examples, noisy observations, etc." — <NAME>, "[Testing MCMC code, part 1: unit tests](https://hips.seas.harvard.edu/blog/2013/05/20/testing-mcmc-code-part-1-unit-tests/)", Harvard Intelligent Probabilistic Systems group<jupyter_code>from __future__ import print_function

import os

import numpy as np
import pandas as pd

PROJ_ROOT = os.path.abspath(os.path.join(os.pardir, os.pardir))<jupyter_output><empty_output><jupyter_text>`numpy.testing`Provides useful assertion methods for values that are numerically close and for numpy arrays.<jupyter_code>data = np.random.normal(0.0, 1.0, 1000000)
assert np.mean(data) == 0.0
np.testing.assert_almost_equal(np.mean(data), 0.0, decimal=2)
a = np.random.normal(0, 0.0001, 10000)
b = np.random.normal(0, 0.0001, 10000)

np.testing.assert_array_equal(a, b)
np.testing.assert_array_almost_equal(a, b, decimal=3)<jupyter_output><empty_output><jupyter_text>[engarde]() decoratorsA new library that lets you practice defensive program--specifically with pandas `DataFrame` objects. It provides a set of decorators that check the return value of any function that returns a `DataFrame` and confirms that it conforms to the rules.<jupyter_code>import engarde.decorators as ed
test_data = pd.DataFrame({'a': np.random.normal(0, 1, 100),
                          'b': np.random.normal(0, 1, 100)})

@ed.none_missing()
def process(dataframe):
    dataframe.loc[10, 'a'] = 1.0
    return dataframe

process(test_data).head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09248554913294797, 'fraction_numerical': 0.02845709204090707, 'mean_word_length': 5.198347107438017, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '467155', 'license': 'MIT', 'n_tokens_mistral': 726, 'n_tokens_neox': 657, 'n_words': 247, 'path': 'notebooks/lectures/4.0-testing.ipynb', 'repo_name': 'gvvynplaine/data-science-is-software', 'size': '2254'}