<commit_before>Changelog
=========


Version 0.9 (in development)
----------------------------




Version 0.8.1
-------------

Drop Python 3.4 support.


Version 0.8
-----------

Add ``.crc32()``-method to ``Context`` and ``Definition`` returning the
hex-encoded crc32 of their context table string.

Include crc32 in the ``__repr__`` of context objects.
<commit_msg>document serialization and Lattice equivalence addition<commit_after>Changelog
=========


Version 0.9 (in development)
----------------------------

Add ``Context.tojson(path_or_fileonj)`` and ``Context.fromjson(path_or_fileonj)``
for long term storage format serialized as json. Long term storage format can include
the lattice structure information so that e.g. big lattices that are expensive to
compute can be stored and loaded without the need to re-calculate the lattice.

Add ``Context.todict()`` and ``Context.fromdict()`` for long term storage format
as Python dict. Can be (de)serialized with other tools such as pickle, yaml,
pprint.pprint() + ast.literal_eval, toml, xml, etc.

Add equivalence checking for Lattice. Note that while Context ignore their lattices for
equivalence checking, Lattices also compare their contexts. Lattice comparison is mostly
for unit-testing (not meant to be efficient), context-comparison should be superiour in most cases.


Version 0.8.1
-------------

Drop Python 3.4 support.


Version 0.8
-----------

Add ``.crc32()``-method to ``Context`` and ``Definition`` returning the
hex-encoded crc32 of their context table string.

Include crc32 in the ``__repr__`` of context objects.

{'fraction_non_alphanumeric': 0.1542319749216301, 'fraction_numerical': 0.018808777429467086, 'mean_word_length': 3.396694214876033, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3768105', 'n_tokens_mistral': 466, 'n_tokens_neox': 430, 'n_words': 186}