<jupyter_start><jupyter_text>Custom JSON Serialization As we saw in the previous video, certain data types cannot be serialized to JSON using Python's defaults. Here's a simple example of this:<jupyter_code>from datetime import datetime
current = datetime.utcnow()
current<jupyter_output><empty_output><jupyter_text>As we can see, this is a `datetime` object. Now let's try to serialize it to JSON:<jupyter_code>import json
json.dumps(current)<jupyter_output><empty_output><jupyter_text>As we can see Python raises a `TypeError` exception, stating that `datetime` objects are not JSON serializable.So, we'll need to come up with our own serialization format.For datetimes, the most common format is the **ISO 8601** format - you can read up more about it here (https://en.wikipedia.org/wiki/ISO_8601), but basically the format is:*YYYY-MM-DD* **T** *HH:MM:SS* There are some variations for encoding timezones, but to keep things simple I am going to use timezone naive timestamps, and just use UTC everywhere. We could use Python's string representation for datetimes:<jupyter_code>str(current)<jupyter_output><empty_output><jupyter_text>but this is not quite ISO-8601. We could write a custom formatter ourselves:<jupyter_code>def format_iso(dt):
    return dt.strftime('%Y-%m-%dT%H:%M:%S')<jupyter_output><empty_output><jupyter_text>(If you want more info and options on date and time formatting/parsing using `strftime` and `strptime`, which essentially pass through to their `C` counterparts, you can see the Python docs here: https://docs.python.org/3/library/datetime.htmlstrftime-strptime-behavior)<jupyter_code>format_iso(current)<jupyter_output><empty_output><jupyter_text>But Python actually provides us a function to do the same:<jupyter_code>current.isoformat()<jupyter_output><empty_output><jupyter_text>This is almost identical to our custom representation, but also includes fractional seconds. If you don't want fractional seconds in your representation, then you'll have to write some custom code like the one above.I'm just going to use Python's ISO-8601 representation.And now let's serialize our `datetime` object to JSON:<jupyter_code>log_record = {'time': datetime.utcnow().isoformat(), 'message': 'testing'}
json.dumps(log_record)<jupyter_output><empty_output><jupyter_text>OK, this works, but this is far from ideal. Normally, our dictionary will contain the `datetime` object, not it's string representation.For example, in the example I showed above, our record would likely be:<jupyter_code>log_record = {'time': datetime.utcnow(), 'message': 'testing'}<jupyter_output><empty_output><jupyter_text>The problem is that `log_record` is now not JSON serializable!What we have to do is write custom code to replace non-JSON serializable objects in our dictionary with custom representations. This can quickly become tedious and unmanageable if we deal with many dictionaries, and arbitrary structures.Fortunately, Python's `dump` and `dumps` functions have some ways for us to define general serializations for non-standard JSON objects.The simplest way is to specify a function that `dump`/`dumps` will call when it encounters something it cannot serialize:<jupyter_code>def format_iso(dt):
    return dt.isoformat()
json.dumps(log_record, default=format_iso)<jupyter_output><empty_output><jupyter_text>This will work even if we have more than one date in our dictionary:<jupyter_code>log_record = {
    'time1': datetime.utcnow(),
    'time2': datetime.utcnow(),
    'message': 'Testing...'
}
json.dumps(log_record, default=format_iso)<jupyter_output><empty_output><jupyter_text>So this works, but what happens if we introduce another non-serializable object:<jupyter_code>log_record = {
    'time': datetime.utcnow(),
    'message': 'Testing...',
    'other': {'a', 'b', 'c'}
}
json.dumps(log_record, default=format_iso)<jupyter_output><empty_output><jupyter_text>As you can see, Python encountered that `set`, and therefore called the `default` callable - but that callable was not designed to handle sets, and so we end up with an exception in the `format_iso` callable instead.We can remedy this by essentially adding code to our function to make it handle various data types. Essentially creating a dispatcher - this should remind you of the single-dispatch generic function decorator available in the `functools` module which we discussed in an earlier part of this series. You can also view more info about it here: https://docs.python.org/3/library/functools.htmlfunctools.singledispatch Let's first write it without the decorator to make sure we have our code correct:<jupyter_code>def custom_json_formatter(arg):
    if isinstance(arg, datetime):
        return arg.isoformat()
    elif isinstance(arg, set):
        return list(arg)
json.dumps(log_record, default=custom_json_formatter)<jupyter_output><empty_output><jupyter_text>To make things a little more interesting, let's throw in a custom object as well:<jupyter_code>class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
        self.create_dt = datetime.utcnow()
        
    def __repr__(self):
        return f'Person(name={self.name}, age={self.age})'
    
    def toJSON(self):
        return {
            'name': self.name,
            'age': self.age,
            'create_dt': self.create_dt.isoformat()
        }
p = Person('John', 82)
print(p)
print(p.toJSON())<jupyter_output>Person(name=John, age=82)
{'name': 'John', 'age': 82, 'create_dt': '2018-12-29T22:26:45.066252'}<jupyter_text>And we modify our custom JSON formatter as follows:<jupyter_code>def custom_json_formatter(arg):
    if isinstance(arg, datetime):
        return arg.isoformat()
    elif isinstance(arg, set):
        return list(arg)
    elif isinstance(arg, Person):
        return arg.toJSON()<jupyter_output><empty_output><jupyter_text>We can now serialize a more complex object:<jupyter_code>log_record = dict(time=datetime.utcnow(),
                  message='Created new person record',
                  person=p)
json.dumps(log_record, default=custom_json_formatter)
print(json.dumps(log_record, default=custom_json_formatter, indent=2))<jupyter_output>{
  "time": "2018-12-29T22:26:45.769929",
  "message": "Created new person record",
  "person": {
    "name": "John",
    "age": 82,
    "create_dt": "2018-12-29T22:26:45.066252"
  }
}<jupyter_text>One thing to note here is that for the `Person` class we returned a formatted string for the `created_dt` attribute. We don't actually need to do this - we can simply return a `datetime` object and let `custom_json_formatter` handle serializing the `datetime` object:<jupyter_code>class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
        self.create_dt = datetime.utcnow()
        
    def __repr__(self):
        return f'Person(name={self.name}, age={self.age})'
    
    def toJSON(self):
        return {
            'name': self.name,
            'age': self.age,
            'create_dt': self.create_dt
        }
p = Person('Monty', 100)
log_record = dict(time=datetime.utcnow(),
                  message='Created new person record',
                  person=p)
print(json.dumps(log_record, default=custom_json_formatter, indent=2))<jupyter_output>{
  "time": "2018-12-29T22:26:47.029102",
  "message": "Created new person record",
  "person": {
    "name": "Monty",
    "age": 100,
    "create_dt": "2018-12-29T22:26:46.749022"
  }
}<jupyter_text>In fact, we could simplify our class further by simply returning a dict of the attributes, since in this case we want to serialize everything as is.But using the `toJSON` callable means we can customize exactly how we want out objects to be serialized.So, if we weren't particular about the serialization we could do this:<jupyter_code>class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
        self.create_dt = datetime.utcnow()
        
    def __repr__(self):
        return f'Person(name={self.name}, age={self.age})'
    
    def toJSON(self):
        return vars(self)
p = Person('Python', 27)
p.toJSON()
log_record['person'] = p
print(log_record)
print(json.dumps(log_record, default=custom_json_formatter, indent=2))<jupyter_output>{
  "time": "2018-12-29T22:26:47.029102",
  "message": "Created new person record",
  "person": {
    "name": "Python",
    "age": 27,
    "create_dt": "2018-12-29T22:26:47.973930"
  }
}<jupyter_text>In fact, we could use this approach in our custom formatter - if an object does not have a `toJSON` callable, we'll just use a dictionary of the attributes - it it has any, it might not (like a complex number or a set as examples), so we need to watch out for that as well.<jupyter_code>'toJSON' in vars(Person)
def custom_json_formatter(arg):
    if isinstance(arg, datetime):
        return arg.isoformat()
    elif isinstance(arg, set):
        return list(arg)
    else:
        try:
            return arg.toJSON()
        except AttributeError:
            try:
                return vars(arg)
            except TypeError:
                return str(arg)<jupyter_output><empty_output><jupyter_text>Let's create another custom class that does not have a `toJSON` method:<jupyter_code>class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y
        
    def __repr__(self):
        return f'Point(x={self.x}, y={self.y})'
pt1 = Point(10, 10)
vars(pt1)
log_record = dict(time=datetime.utcnow(),
                  message='Created new point',
                  point=pt1,
                  created_by=p)
log_record<jupyter_output><empty_output><jupyter_text>And we can now serialize it to JSON:<jupyter_code>print(json.dumps(log_record, default=custom_json_formatter, indent=2))<jupyter_output>{
  "time": "2018-12-29T22:26:50.955039",
  "message": "Created new point",
  "point": {
    "x": 10,
    "y": 10
  },
  "created_by": {
    "name": "Python",
    "age": 27,
    "create_dt": "2018-12-29T22:26:47.973930"
  }
}<jupyter_text>So now, let's re-write our custom json formatter using the generic single dispatch decorator I mentioned earlier:<jupyter_code>from functools import singledispatch<jupyter_output><empty_output><jupyter_text>Our default approach is going to first try to use `toJSON`, if not it will try to use `vars`, and it that still fails we'll use the string representation, whatever that happens to be:<jupyter_code>@singledispatch
def json_format(arg):
    print(arg)
    try:
        print('\ttrying to use toJSON...')
        return arg.toJSON()
    except AttributeError:
        print('\tfailed - trying to use vars...')
        try:
            return vars(arg)
        except TypeError:
            print('\tfailed - using string representation...')
            return str(arg)<jupyter_output><empty_output><jupyter_text>And now we 'register' other data types:<jupyter_code>@json_format.register(datetime)
def _(arg):
    return arg.isoformat()
@json_format.register(set)
def _(arg):
    return list(arg)<jupyter_output><empty_output><jupyter_text>And we can now serialize just like before:<jupyter_code>print(json.dumps(log_record, default=json_format, indent=2))<jupyter_output>Point(x=10, y=10)
	trying to use toJSON...
	failed - trying to use vars...
Person(name=Python, age=27)
	trying to use toJSON...
{
  "time": "2018-12-29T22:26:50.955039",
  "message": "Created new point",
  "point": {
    "x": 10,
    "y": 10
  },
  "created_by": {
    "name": "Python",
    "age": 27,
    "create_dt": "2018-12-29T22:26:47.973930"
  }
}<jupyter_text>Let's change our Person class to emit some custom JSON instead of just using `vars`:<jupyter_code>class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
        self.create_dt = datetime.utcnow()
        
    def __repr__(self):
        return f'Person(name={self.name}, age={self.age})'
    
    def toJSON(self):
        return dict(name=self.name)
p = Person('Python', 27)
log_record['created_by'] = p
print(json.dumps(log_record, default=json_format, indent=2))<jupyter_output>Point(x=10, y=10)
	trying to use toJSON...
	failed - trying to use vars...
Person(name=Python, age=27)
	trying to use toJSON...
{
  "time": "2018-12-29T22:26:50.955039",
  "message": "Created new point",
  "point": {
    "x": 10,
    "y": 10
  },
  "created_by": {
    "name": "Python"
  }
}<jupyter_text>The way we wrote our default formatter, means that we can now also represent other unexpected data types, but using each object's string representation. If that's not acceptable, we can either not do this and let a `TypeError` exception get generated, or register more custom formatters:<jupyter_code>from decimal import Decimal
from fractions import Fraction

json.dumps(dict(a=1+1j, 
                b=Decimal('0.5'), 
                c=Fraction(1, 3),
                p=Person('Python', 27),
                pt=Point(0,0),
                time=datetime.utcnow()
               ), 
           default=json_format)<jupyter_output>(1+1j)
	trying to use toJSON...
	failed - trying to use vars...
	failed - using string representation...
0.5
	trying to use toJSON...
	failed - trying to use vars...
	failed - using string representation...
1/3
	trying to use toJSON...
	failed - trying to use vars...
	failed - using string representation...
Person(name=Python, age=27)
	trying to use toJSON...
Point(x=0, y=0)
	trying to use toJSON...
	failed - trying to use vars...<jupyter_text>Now, suppose we don't want that default representation for `Decimals` - we want to serialize it in this form: `Decimal(0.5)`.All we need to do is to register a new function to serialize `Decimal` types:<jupyter_code>@json_format.register(Decimal)
def _(arg):
    return f'Decimal({str(arg)})'
json.dumps(dict(a=1+1j, 
                b=Decimal(0.5), 
                c=Fraction(1, 3),
                p=Person('Python', 27),
                pt = Point(0,0),
                time = datetime.utcnow()
               ), 
           default=json_format)<jupyter_output>(1+1j)
	trying to use toJSON...
	failed - trying to use vars...
	failed - using string representation...
1/3
	trying to use toJSON...
	failed - trying to use vars...
	failed - using string representation...
Person(name=Python, age=27)
	trying to use toJSON...
Point(x=0, y=0)
	trying to use toJSON...
	failed - trying to use vars...<jupyter_text>One last example that clearly shows the `json_format` function gets called recursively when needed:<jupyter_code>print(json.dumps(dict(pt = Point(Person('Python', 27), 2+2j)),
          default=json_format, indent=2))<jupyter_output>Point(x=Person(name=Python, age=27), y=(2+2j))
	trying to use toJSON...
	failed - trying to use vars...
Person(name=Python, age=27)
	trying to use toJSON...
(2+2j)
	trying to use toJSON...
	failed - trying to use vars...
	failed - using string representation...
{
  "pt": {
    "x": {
      "name": "Python"
    },
    "y": "(2+2j)"
  }
}
{'fraction_non_alphanumeric': 0.11619200427864688, 'fraction_numerical': 0.02513705040780853, 'mean_word_length': 3.788412291933419, 'pattern_counts': {'":': 47, '<': 103, '<?xml version=': 0, '>': 103, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '253338', 'license': 'Apache-2.0', 'n_tokens_mistral': 5039, 'n_tokens_neox': 4716, 'n_words': 1452, 'path': 'dd_1/Part 3/Section 07 - Serialization and Deserialization/03 - Custom JSON Encoding.ipynb', 'repo_name': 'rebekka-halal/bg', 'size': '14958'}