<jupyter_start><jupyter_text>Postmodern error handling in Python 3.6 I'll be the first person to admit I have no idea what postmodernism actually means, but it sounds cool for an article on error-handling, and I would argue that the facilities (post?)modern Python provides for handling errors is pretty damn cool. derpRecently, an acquaintance of mine posed this question on a message board we both participate in:> okay nerdswhat do you call a union-type with three states?> so I was writing a Maybe implementation (in python) to deal with a processing pipeline, but then it turned into an Either, and then it turned into something with three states> basically there's either 1) the final value, 2) no value but a known error occurred, 3) no value but an unknown error occurred> the final states could be: 1) hey it succeeded, here's the object's id 2) no value but your json blob was missing a required attribute (or some other known error), or 3) we encountered an error that the programmer didn't forseeTo which another strapping young gentleman replied:>  if it were Rust you could enum your errors ಠ‿ಠWell, truth be told, I feel like I probably know about as little about [**fp**](https://www.smashingmagazine.com/2014/07/dont-be-scared-of-functional-programming/) as I do about postmodernism, but I thought this was an interesting question and I do at least know that Python's standard library does provide us Enums ┬─┬﻿ ノ( ゜-゜ノ) so I figured I would take a crack at it.--- To start, let's quickly talk about what an Enum (enumeration) is for those of us who have yet to encounter them.At a very basic level, I would define an Enum as a particular set of choices (or states).Let's say, for example, that you [lost a bet](https://www.youtube.com/watch?v=zBYz_RInbrQ), and now have a set of options to choose from as punishment.Those options are* **A**: Get slapped 10 times right now* **B**: Get slapped 5 times at any random time from now to eternityThose are the only 2 possibilities; you don't get any other choices. We could model this idea using an Enum.<jupyter_code>from enum import Enum

class SlapBet(Enum):
    TEN_SLAPS = 1 # get slapped ten times right now
    FIVE_SLAPS = 2 # fear slaps possibly for eternity
    
smart_choice = SlapBet.TEN_SLAPS

stupid_choice = SlapBet.FIVE_SLAPS

print(
    smart_choice, # notice the pretty repr
    smart_choice == SlapBet.TEN_SLAPS,
    smart_choice == stupid_choice,
    sep='\n'
)<jupyter_output>SlapBet.TEN_SLAPS
True
False<jupyter_text>So, why would we do this? Why go through the trouble of using an Enum when we could functionally do the same thing with something like integers? Well, the first reason, and the most important one, in my opinion, is readability. The second reason is that Enum instances can hold values we may want to use. Let's say you have a function that has an optional parameter.<jupyter_code>def connection(person_1, person_2, relationship='knows'):
    """Return a string that represents the relationship between two people."""
    
    # this sexy f"" string format is part of the new Python 3.6
    # hotness. An article for another day.
    
    return f"{person_1} {relationship} {person_2}"

connection("Ron", "Harry", relationship="is best friends with")<jupyter_output><empty_output><jupyter_text>Now let's say we want to restrain the possible relationships that people can have.Maybe we want to do this to prevent errors or to simply prevent weirdness like the following.<jupyter_code>connection("Ron", "Harry", relationship="is dating Hermoine but secretly wants")<jupyter_output><empty_output><jupyter_text>Enums to the rescue!<jupyter_code>from enum import Enum

class Relationship(Enum):
    knows = 'knows'
    likes = 'likes'
    loves = 'loves'
    detests = 'detests'
    bff = "is best friends with"

def connection(person_1, person_2, relationship=Relationship.likes):
    """Return a string that represents the relationship between two people."""
    
    # notice we use the .value of the Relationship instance to get the string
    
    return f"{person_1} {relationship.value} {person_2}"

connection("Ron", "Harry", relationship=Relationship.bff)<jupyter_output><empty_output><jupyter_text>InterludeNow, before tackle the original question that started us on this journey, let's quickly talk about another new and incredibly useful and important feature since Python 3.5 that has been vastly improved in Python 3.6 - type annotations.In our previous example, there would be nothing to prevent someone from doing the following:<jupyter_code>evil = lambda: 'power'
evil.value = 'corrupts'

connection("Ron", "Harry", relationship=evil)<jupyter_output><empty_output><jupyter_text>That's obviously not how we intended our function to be used, but part of what makes Python so powerful, its dynamic nature, is what allows such nefarious behavior. If only there were a way to save ourselves and people who use our code from making such mistakes... Behold!<jupyter_code>from enum import Enum

class Relationship(Enum):
    knows: str = 'knows'
    likes: str = 'likes'
    loves: str = 'loves'
    detests: str = 'detests'
    bff: str = "is best friends with"
    
def connection(person_1: str,
               person_2: str,
               relationship: Relationship=Relationship.knows) -> str:
    """Return a string that represents the relationship between two people."""
    
    # notice we use the .value of the Relationship instance to get the string
    
    return f"{person_1} {relationship.value} {person_2}"

connection("Ron", "Harry", relationship=Relationship.bff)<jupyter_output><empty_output><jupyter_text>"Huh?", you say. "It looks to me like things just got more verbosified. I like making up words." Cool word, and you would be right, things did get more wordy. Moreover, adding all those annotations won't prevent someone from doing what we mentioned before,```pythonevil = lambda: 'power'evil.value = 'corrupts'connection("Ron", "Harry", relationship=evil)  this still works, though we don't want it to```... **unless**, that is, we run that code using - [**mypy**](http://mypy-lang.org/)Mypy allows you to add type annotations and enforce them prior to running your program, so the only way the above function would run is if the relationship parameter was of type Relationship when called. Amazing!Now, anyone reading our code would know exactly the types of things that could be passed as parameters to functions, and mypy will help us to enforce the type annotations we set. That makes our code much more legible and provides us and people using our code some nice sanity checks - knowing certain kinds of human errors will be caught prior to our code being run.---  Now, back to our original question about modeling 3 possible conditions.1. Task ran without error. Data returned.2. A known error occurred during task execution. No data returned.3. A catastrophic runtime error occurred. No data returnedSo, before we look at the code, it's cool to note that as of Python 3.6 we now have typed NamedTuples that we can declare using a new syntax in 3.6. We had this typed namedtuples in Python 3.5 but I think the new syntax is much cleaner.```python old wayEmployee = NamedTuple('Employee', [('name', str), ('id', int)]) new sauceclass Employee(NamedTuple):    name : str    id   : int```Also, an Optional describes something that can be of a certain type, or None.```pythonpossible_integer: Optional[int] = None  we could make this an integer later on```You'll see the use of optional values a lot in functional programming. In fact, the following pattern of combining tuples and optional values is the primary way we handle errors in Go. Optionals are also ubiquitous in Swift - not only in the context of error-handling. Optionals are a really handy concept, and now Python has them as well as a lot of the other type safety goodness of other languages through mypy.<jupyter_code>from typing import NamedTuple, Optional
import requests
import logging
import json
import enum


class ApiInteraction(enum.Enum):
    """The 3 possible states we can expect when interacting with the API."""
    SUCCESS = 1
    ERROR = 2
    FAILURE = 3


class ApiResponse(NamedTuple):
    """
    This is sort of a really dumbed-down version of an HTTP response,
    if you think of it in terms of status codes and response bodies.
    """
    status: ApiInteraction
    payload: Optional[dict]


        
def hit_endpoint(url: str) -> ApiResponse:
    """
    1. Send an http request to a url
    2. Parse the json response as a dictionary
    3. Return an ApiResponse object
    """
    
    
    try:
        
        response = requests.get(url) # step 1
        payload = response.json() # step 2
        
    except json.decoder.JSONDecodeError as e:
        
        # something went wrong in step 2; we knew this might happen
        
        # log a simple error message
        
        logging.error(f'could not decode json from {url}')
        
        # log the full traceback at a lower level
        
        logging.info(e, exc_info=True)
        
        # since we anticipated this error, make the
        # ApiResponse.status an ERROR as opposed to a failure
        
        return ApiResponse(ApiInteraction.ERROR, None)
    
    except Exception as e:
        
        # something went wrong in step 1 or 2 that
        # we couldn't anticipate
        
        # log the exception with the traceback
        
        logging.error(f"Something bad happened trying to reach {url}")
        logging.info(e, exc_info=True)
        
        # Since something catastrophic happened that
        # we didn't anticipate i.e. (DivideByBananaError)
        # we set the ApiResponse.status to FAILURE
        
        return ApiResponse(ApiInteraction.FAILURE, None)
    
    else:
        
        # Everything worked as planned! No errors!
        
        return ApiResponse(ApiInteraction.SUCCESS, payload)


# Python is awesome. We can either use the function by itself
# or use it as a constructor for our ApiResponse class 
# by doing thefollowing:


ApiResponse.from_url = hit_endpoint
def test_endpoint_response():

    url = 'http://httpbin.org/headers'
    response = ApiResponse.from_url(url)
    assert response.status == ApiInteraction.SUCCESS
    assert response.status == hit_endpoint(url).status # our function and constructor work the same!
    assert response.payload is not None


    url = 'http://twitter.com'
    response = ApiResponse.from_url(url)
    assert response.status == ApiInteraction.ERROR
    assert response.status == hit_endpoint(url).status
    assert response.payload is None


    url = 'foo'
    response = ApiResponse.from_url(url)
    assert response.status == ApiInteraction.FAILURE
    assert response.status == hit_endpoint(url).status
    assert response.payload is None
        
    
test_endpoint_response()<jupyter_output>ERROR:root:could not decode json from http://twitter.com
ERROR:root:could not decode json from http://twitter.com
ERROR:root:Something bad happened trying to reach foo
ERROR:root:Something bad happened trying to reach foo
{'fraction_non_alphanumeric': 0.07023381294964029, 'fraction_numerical': 0.00539568345323741, 'mean_word_length': 3.726306842328942, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 34, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 1, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '237149', 'license': 'MIT', 'n_tokens_mistral': 3048, 'n_tokens_neox': 2902, 'n_words': 1409, 'path': 'notebooks/postmodern-error-handling.ipynb', 'repo_name': 'knowsuchagency/knowsuchagency.github.io.old', 'size': '11120'}