<jupyter_start><jupyter_text>Input data for testsAs we saw in the last section, when using parametrisation it's often useful to split your test function into two logical parts:1. The data to be tested2. The code to do the testThis is because we had a situation where we had one test function and multiple examples to test. The opposite situation also happens where we have multiple test functions, all of which want the same input data.The name that pytest uses for "data which are provided to test functions" is *fixture* since it *fixes* a set of data against which to test.We'll start with the example of the `add_arrays` function to explain the syntax but soon we'll need to use a example which demonstates the benefits more.To make things clearer, we'll trim down the test file back to the basics. Just one test for `add_arrays`:<jupyter_code>%%writefile arrays.py

"""
This module contains functions for manipulating and combining Python lists.
"""

def add_arrays(x, y):
    """
    This function adds together each element of the two passed lists.

    Args:
        x (list): The first list to add
        y (list): The second list to add

    Returns:
        list: the pairwise sums of ``x`` and ``y``.

    Examples:
        >>> add_arrays([1, 4, 5], [4, 3, 5])
        [5, 7, 10]
    """

    if len(x) != len(y):
        raise ValueError("Both arrays must have the same length.")

    z = []
    for x_, y_ in zip(x, y):
        z.append(x_ + y_)

    return z
%%writefile test_arrays.py

from arrays import add_arrays

def test_add_arrays():
    a = [1, 2, 3]
    b = [4, 5, 6]
    expect = [5, 7, 9]
    
    output = add_arrays(a, b)
    
    assert output == expect<jupyter_output>Overwriting test_arrays.py<jupyter_text>To create our fixture we define a function which is decorated with the `pytest.fixture` decorator. Apart from that, all the function needs to do is return the data we want to provide to our tests, in this case, the two input lists:```pythonimport pytest@pytest.fixturedef pair_of_lists():    return [1, 2, 3], [4, 5, 6]```To make the test functions make use of the fixture, we use the name of the fixture (`pair_of_lists`) as a parameter of the test function, similar to how we did with parametrisation:```pythondef test_add_arrays(pair_of_lists):    ...```The data are now available inside the function using that name and we can use it however we wish:```pythondef test_add_arrays(pair_of_lists):    a, b = pair_of_lists    ...```This isn't how functions and arguments usually work in Python. pytest is doing something magic here and is matching up the names of things which it knows are fixtures (due to the decorator) with the names of parameters to test functions, automatically running the fixture and passing in the data.Note that `pair_of_lists` here is not a test function. It does not contain any `assert`s and will not  explicitly appear in the `pytest` output.Putting it all together, we end up with:<jupyter_code>%%writefile test_arrays.py

import pytest

from arrays import add_arrays

@pytest.fixture
def pair_of_lists():
    return [1, 2, 3], [4, 5, 6]

def test_add_arrays(pair_of_lists):
    a, b = pair_of_lists
    expect = [5, 7, 9]
    
    output = add_arrays(a, b)
    
    assert output == expect<jupyter_output>Overwriting test_arrays.py<jupyter_text>When we run the test suite, pytest will automatically run the `pair_of_lists` function for any test that has it as an input and pass in the result.<jupyter_code>!COLUMNS=60 pytest -v test_arrays.py<jupyter_output>[1m=================== test session starts ====================[0m
platform linux -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/matt/projects/courses/software_engineering_best_practices
plugins: attrib-0.1.3, requests-mock-1.8.0, cov-2.10.1
collected 1 item                                           [0m

test_arrays.py::test_add_arrays [32mPASSED[0m[32m               [100%][0m

[32m==================== [32m[1m1 passed[0m[32m in 0.01s[0m[32m =====================[0m<jupyter_text>A different exampleIt might be hard to see the benefit of fixtures with this rather contrived example in which there aren't repeated uses of the same input data. So lets take a look at a more sensible one where using a fixture makes sense.Make a new file called `books.py` which contains the following:<jupyter_code>%%writefile books.py

def word_count(text, word=''):
    """
    Count the number of occurences of ``word`` in a string.
    If ``word`` is not set, count all words.
    
    Args:
        text (str): the text corpus to search through
        word (str): the word to count instances of

    Returns:
        int: the count of ``word`` in ``text``
    """
    if word:
        count = 0
        for text_word in text.split():
            if text_word == word:
                count += 1
        return count
    else:
        return len(text.split())<jupyter_output>Writing books.py<jupyter_text>To test this function we want a corpus of text to test it on. For the purposes of this example and to simulate a complex data input, we will download the contents of a particularly long novel from Project Gutenberg. Our test function uses [`urllib.request`](https://docs.python.org/3/library/urllib.request.html) to download the text, converts it to a string and passes that to the `word_count` function.At first we will simply check that the word "hat" appears 33 times in the book:<jupyter_code>%%writefile test_books.py

import urllib.request

from books import word_count

def test_word_counts():
    url = "https://www.gutenberg.org/files/2600/2600-0.txt"
    book_text = urllib.request.urlopen(url).read().decode('utf-8')
    assert word_count(book_text, "hat") == 33
!COLUMNS=60 pytest -v test_books.py<jupyter_output>[1m=================== test session starts ====================[0m
platform linux -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/matt/projects/courses/software_engineering_best_practices
plugins: attrib-0.1.3, requests-mock-1.8.0, cov-2.10.1
collected 1 item                                           [0m

test_books.py::test_word_counts [32mPASSED[0m[32m               [100%][0m

[32m==================== [32m[1m1 passed[0m[32m in 2.77s[0m[32m =====================[0m<jupyter_text>The test has passed and it took about two seconds. This is because it takes some time to download the file from the internet. For this example we *want* it to take some time as it helps demonstrate the point. In reality you will come across test data inputs which take some time (more than a few milliseconds) to create.This creates a tension between wanting to have a large test suite which covers your code from lots of different angles and being able to run it very quickly and easily. An ideal test suite will run as quickly as possible as it will encourage you to run it more often. It's a good idea to have at least a subset of your tests which run through in some number of seconds rather than hours.Two seconds is not bad for this test but if we want to test against multiple examples, it could get slow. Let's parametrise the test to add in a bunch more inputs:<jupyter_code>%%writefile test_books.py

import urllib.request

import pytest

from books import word_count

@pytest.mark.parametrize('word, count',  [
    ('hat', 33),
    ('freedom', 71),
    ('electricity', 1),
    ('testing', 3),
    ('Prince', 1499),
    ('internet', 0),
    ('Russia', 71),
    ('Pierre', 1260),
    (None, 566317),
])
def test_word_counts(word, count):
    url = "https://www.gutenberg.org/files/2600/2600-0.txt"
    book_text = urllib.request.urlopen(url).read().decode('utf-8')
    assert word_count(book_text, word) == count
!COLUMNS=60 pytest -v test_books.py<jupyter_output>[1m=================== test session starts ====================[0m
platform linux -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/matt/projects/courses/software_engineering_best_practices
plugins: attrib-0.1.3, requests-mock-1.8.0, cov-2.10.1
collected 9 items                                          [0m

test_books.py::test_word_counts[hat-33] [32mPASSED[0m[32m       [ 11%][0m
test_books.py::test_word_counts[freedom-71] [32mPASSED[0m[32m   [ 22%][0m
test_books.py::test_word_counts[electricity-1] [32mPASSED[0m[32m [ 33%][0m
test_books.py::test_word_counts[testing-3] [32mPASSED[0m[32m    [ 44%][0m
test_books.py::test_word_counts[Prince-1499] [32mPASSED[0m[32m  [ 55%][0m
test_books.py::test_word_counts[internet-0] [32mPASSED[0m[32m   [ 66%][0m
test_books.py::test_word_counts[Russia-71] [32mPASSED[0m[32m    [ 77%][0m
test_books.py::test_word_counts[Pierre-1260] [32mPASSED[0m[32m  [ 88%][0m[...]<jupyter_text>You see here that it took about nine times as long. This is because the file is downloaded afresh for every test example where really, it only *needs* to be downloaded once.Let's move the slow setup into a fixture and give that as a parameter of the test function:<jupyter_code>%%writefile test_books.py

import urllib.request

import pytest

from books import word_count

@pytest.fixture()
def long_book():
    url = "https://www.gutenberg.org/files/2600/2600-0.txt"
    book_text = urllib.request.urlopen(url).read().decode('utf-8')
    return book_text

@pytest.mark.parametrize('word, count',  [
    ('hat', 33),
    ('freedom', 71),
    ('electricity', 1),
    ('testing', 3),
    ('Prince', 1499),
    ('internet', 0),
    ('Russia', 71),
    ('Pierre', 1260),
    (None, 566317),
])
def test_word_counts(long_book, word, count):
    assert word_count(long_book, word) == count
!COLUMNS=60 pytest -v test_books.py<jupyter_output>[1m=================== test session starts ====================[0m
platform linux -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/matt/projects/courses/software_engineering_best_practices
plugins: attrib-0.1.3, requests-mock-1.8.0, cov-2.10.1
collected 9 items                                          [0m

test_books.py::test_word_counts[hat-33] [32mPASSED[0m[32m       [ 11%][0m
test_books.py::test_word_counts[freedom-71] [32mPASSED[0m[32m   [ 22%][0m
test_books.py::test_word_counts[electricity-1] [32mPASSED[0m[32m [ 33%][0m
test_books.py::test_word_counts[testing-3] [32mPASSED[0m[32m    [ 44%][0m
test_books.py::test_word_counts[Prince-1499] [32mPASSED[0m[32m  [ 55%][0m
test_books.py::test_word_counts[internet-0] [32mPASSED[0m[32m   [ 66%][0m
test_books.py::test_word_counts[Russia-71] [32mPASSED[0m[32m    [ 77%][0m
test_books.py::test_word_counts[Pierre-1260] [32mPASSED[0m[32m  [ 88%][0m[...]<jupyter_text>Perhaps surprisingly, it is still taking very long time!By default a fixture will run once for every test function that uses it. In our case we only need it to run once for all the tests in the test session so we can pass in the `scope` parameter to `pytest.fixture` and set it to `session`:<jupyter_code>%%writefile test_books.py

import urllib.request

import pytest

from books import word_count

@pytest.fixture(scope="session")
def long_book():
    url = "https://www.gutenberg.org/files/2600/2600-0.txt"
    book_text = urllib.request.urlopen(url).read().decode('utf-8')
    return book_text

@pytest.mark.parametrize('word, count',  [
    ('hat', 33),
    ('freedom', 71),
    ('electricity', 1),
    ('testing', 3),
    ('Prince', 1499),
    ('internet', 0),
    ('Russia', 71),
    ('Pierre', 1260),
    (None, 566317),
])
def test_word_counts(long_book, word, count):
    assert word_count(long_book, word) == count
!COLUMNS=60 pytest -v test_books.py<jupyter_output>[1m=================== test session starts ====================[0m
platform linux -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/matt/projects/courses/software_engineering_best_practices
plugins: attrib-0.1.3, requests-mock-1.8.0, cov-2.10.1
collected 9 items                                          [0m

test_books.py::test_word_counts[hat-33] [32mPASSED[0m[32m       [ 11%][0m
test_books.py::test_word_counts[freedom-71] [32mPASSED[0m[32m   [ 22%][0m
test_books.py::test_word_counts[electricity-1] [32mPASSED[0m[32m [ 33%][0m
test_books.py::test_word_counts[testing-3] [32mPASSED[0m[32m    [ 44%][0m
test_books.py::test_word_counts[Prince-1499] [32mPASSED[0m[32m  [ 55%][0m
test_books.py::test_word_counts[internet-0] [32mPASSED[0m[32m   [ 66%][0m
test_books.py::test_word_counts[Russia-71] [32mPASSED[0m[32m    [ 77%][0m
test_books.py::test_word_counts[Pierre-1260] [32mPASSED[0m[32m  [ 88%][0m[...]
{'fraction_non_alphanumeric': 0.14216641971457536, 'fraction_numerical': 0.046400998206347965, 'mean_word_length': 3.9898832684824903, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 28, 'https://': 5, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '342331', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 4864, 'n_tokens_neox': 4356, 'n_words': 1317, 'path': '04 Fixtures.ipynb', 'repo_name': 'davidwilby/software_engineering_best_practices', 'size': '12823'}