<jupyter_start><jupyter_text>Testing in the Open"Automation is not testing"I recently attended a webinar where the speaker made this comment and it started me thinking about testing in general and what some of the challenges that can present. We all *know* that we should be testing, but there are a lot of potential pit falls that one can fall into when starting down this journey. I'm going to jump into the deep end of the pool here and deal with one of the struggles that it took me a while to deal with. Imagine the following:- You have a local device with a REST API- You write a library that accesses that REST API- You write tests for that library so that it runs against the local devices REST API- You push the library to GITHUB so that other people can leverage your work. ( You are a good person, right?) - You configure TravisCI or CircleCI for integration testing.- You realise all your tests fail because Travis/Circle doesn't have access to your internal device. And now your GITHUB badges all show red and no one trusts your code. Which brings us to *vcrpy* What's vcrpy?Wow! So glad you asked!  *vcrpy* is a RESTAPI library that helps to automatically record the responses from a RESTAPI and capture them on a local file that you can reuse later. Accoring to the *vcrpy* docs which are available [here](https://vcrpy.readthedocs.io/en/latest/)the three main benefits of this library are    The ability to work offline    Completely deterministic tests    Increased test execution speed    So let's dig in by writting a quick piece of code which is going to access a REST API. In this case, I'm going to be using a public API, but let's imagine that this was behind your firewall on a device where you don't want anyone fiddling with.  So in this case. We're going to be using the public API at https://api.kanye.rest.   This is a public API that respond to every GET request with a quote from Kanye West. Let's use python to create a small library that will access the API, print a message, and return the quote as JSON.If you want to take a look at the library, feel free to check out the GITHUB repository [here](INSERT_THE_REPO_HERE_WHEN_POSTED)If you don't want to leave this page, I've included the only function in this library right here.test<jupyter_code>import requests, json

def get_new_quote():
    url= 'https://api.kanye.rest'
    r = requests.get(url)
    quote = json.loads(r.text)
    print ("New Kanye Quote coming up!")
    return quote<jupyter_output><empty_output><jupyter_text>Now that we've got the new function built. Let's take a quick look at what we get. We will run the function and capture it in a python variable called *x*<jupyter_code>x = get_new_quote()<jupyter_output>New Kanye Quote coming up!<jupyter_text>Now let's use the json library to take a look at what was returned.<jupyter_code>json.dumps(x)<jupyter_output><empty_output><jupyter_text>Now that we've got some working code. Let's take run it again and capture this as the python variable *y*<jupyter_code>y = get_new_quote()
json.dumps(y)<jupyter_output><empty_output><jupyter_text>> HmmmmmSo as you can see, each time you perform a GET against the API another variable is returned. You can imagine that this probably isn't the best thing for testing as we really don't have a clear indication as to what *exactly* we're going to expect from this API. We do know a couple of things that can be used for testing- the returned object is a python dictionary- the returned object has a single key/value pair.- the returned object's first key is "quote"- the returned object's first value is going to be an object of type STR.Imagine what the code that could test for all of these things. > Take your time... I'll wait. While you were imagining it, I wrote a quick test to help make sure that the API is returning what we expect. If you don't expect anything, you deserve what you get, right?<jupyter_code>from unittest import TestCase
from pykanyerest.quotes import *

class TestGetNewQuote(TestCase):
    """
    Test Case for get_new_quote function from kanye.rest
    """

    def test_GetNewQuote(self):
        """
        """
        quote = get_new_quote()
        self.assertEqual(type(quote), dict)
        self.assertEqual(len(quote), 1)
        keys = quote.keys()
        self.assertIn('quote', keys)
        self.assertEqual(type(quote['quote']), str)<jupyter_output><empty_output><jupyter_text>If you were to run this test, you would find that it passes. You can trust me on this, right?Now that we've already got a lot that we can hang our tests on, but imagine that we want to also test the exact contents of the value. The code to test would now look like this<jupyter_code>class TestGetNewQuote(TestCase):
    """
    Test Case for get_new_quote function from kanye.rest
    """

    def test_GetNewQuote(self):
        """
        """
        quote = get_new_quote()
        self.assertEqual(type(quote), dict)
        self.assertEqual(len(quote), 1)
        keys = quote.keys()
        self.assertIn('quote', keys)
        self.assertEqual(type(quote['quote']), str)
        self.assertEqual(quote['quote'], "If I got any cooler I would freeze to death")<jupyter_output><empty_output><jupyter_text>If you ran THIS code, you would now find that it fails.```Traceback (most recent call last):  File "/Users/christopheryoung/PycharmProjects/OpenTestingBlog/tests/test_quotes.py", line 18, in test_GetNewQuote    self.assertEqual(quote['quote'], "If I got any cooler I would freeze to death")  File "/Applications/PyCharm.app/Contents/helpers/pycharm/teamcity/diff_tools.py", line 38, in _patched_equals    raise errorteamcity.diff_tools.EqualsAssertionError:  :: People only get jealous when they care. != If I got any cooler I would freeze to death```  Why did it fail?Well of course it failed because the API is supposed to return a *NEW* quote everytime we hit it. This is where the *vcrpy* library comes in SUPER handy as it can record and freeze the API response so that we can make sure that last test pasts every time. So the first thing we're going to do is to import the VCR library into our test file and configure the new test to record the API response into a file on the local filesystem.As you can see below, we've really only added two lines.The first is the "import vcr" line at the top which makes the vcrpy library available to the test script, and the second is the decorator on top of the "test_GetNewQuote" function which does two things.1) Tells the script where to copy the results of the API call the first time you run it2) Tells the script where to look for the results every subsequent time you run it.<jupyter_code>import vcr
from unittest import TestCase
from pykanyerest.quotes import *


class TestGetNewQuote(TestCase):
    """
    Test Case for get_new_quote function from kanye.rest
    """

    @vcr.use_cassette(cassette_library_dir='./test_pykanyerest/fixtures/cassettes')
    def test_GetNewQuote(self):
        """
        """
        quote = get_new_quote()
        self.assertEqual(type(quote), dict)
        self.assertEqual(len(quote), 1)
        keys = quote.keys()
        self.assertIn('quote', keys)
        self.assertEqual(type(quote['quote']), str)
        self.assertEqual(quote['quote'], "If I got any cooler I would freeze to death")<jupyter_output><empty_output><jupyter_text>Once you've run the test once, a new file will appear in the ./test_pykanyerest/fixtures/cassettes/test_GetNewQuote folder with the following contents```interactions:- request:    body: null    headers:      Accept:      - '*/*'      Accept-Encoding:      - gzip, deflate      Connection:      - keep-alive      User-Agent:      - python-requests/2.22.0    method: GET    uri: https://api.kanye.rest/  response:    body:      string: '{"quote":"I''m a creative genius"}'    headers:      Access-Control-Allow-Headers:      - Content-Type      Access-Control-Allow-Methods:      - GET      Access-Control-Allow-Origin:      - '*'      CF-RAY:      - 55a5b75cac3aecee-YUL      Connection:      - keep-alive      Content-Length:      - '33'      Content-Type:      - application/json      Date:      - Fri, 24 Jan 2020 23:16:38 GMT      Expect-CT:      - max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"      Server:      - cloudflare      Set-Cookie:      - __cfduid=d1c38c52e7d73039e047e768fdea93b5d1579907798; expires=Sun, 23-Feb-20        23:16:38 GMT; path=/; domain=.kanye.rest; HttpOnly; SameSite=Lax      Vary:      - Accept-Encoding    status:      code: 200      message: OKversion: 1```If you look closely, you can see a lot of information in there including the contents of the response body string which is now '{"quote":"I''m a creative genius"}'.Note: For those paying attention, you will probably have guessed that the test we created above will still be failing as the Kanye quote we were looking for has changed again. Let's change the test so that we are now looking for the new quote that's captured in the test_GetNewQuote file above.<jupyter_code>import vcr
from unittest import TestCase
from pykanyerest.quotes import *


class TestGetNewQuote(TestCase):
    """
    Test Case for get_new_quote function from kanye.rest
    """

    @vcr.use_cassette(cassette_library_dir='./test_pykanyerest/fixtures/cassettes')
    def test_GetNewQuote(self):
        """
        """
        quote = get_new_quote()
        self.assertEqual(type(quote), dict)
        self.assertEqual(len(quote), 1)
        keys = quote.keys()
        self.assertIn('quote', keys)
        self.assertEqual(type(quote['quote']), str)
        self.assertEqual(quote['quote'], "I'm a creative genius")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0778498609823911, 'fraction_numerical': 0.008649984553599012, 'mean_word_length': 3.682738669238187, 'pattern_counts': {'":': 2, '<': 32, '<?xml version=': 0, '>': 34, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '565285', 'license': 'Apache-2.0', 'n_tokens_mistral': 2853, 'n_tokens_neox': 2705, 'n_words': 1204, 'path': 'Testing in the Open.ipynb', 'repo_name': 'netmanchris/pykanyerest', 'size': '9711'}