<jupyter_start><jupyter_text>Usage Categorizations Included categorizationsIn the `climate_categories` package, the categorizations are availabledirectly at the top-level namespace, and as a dictionary in `.cats`:<jupyter_code>import climate_categories

climate_categories.cats
climate_categories.IPCC2006
climate_categories.cats["IPCC2006"]<jupyter_output><empty_output><jupyter_text>Metadata for each categorization are accessible as properties:<jupyter_code>print(climate_categories.IPCC2006.name)
print(climate_categories.IPCC2006.title)
print(climate_categories.IPCC2006.comment)
print(climate_categories.IPCC2006.references)
print(climate_categories.IPCC2006.institution)
print(climate_categories.IPCC2006.last_update)
print(climate_categories.IPCC2006.version)<jupyter_output><empty_output><jupyter_text>The categorization can be used as a dictionary mapping category codes to categories:<jupyter_code>climate_categories.IPCC2006["1.A"]<jupyter_output><empty_output><jupyter_text>You can also query using alternative spellings of the code:<jupyter_code>climate_categories.IPCC2006["1A"]<jupyter_output><empty_output><jupyter_text>For the categories, metadata is also available: a `title`, maybe a`comment`, all of its `codes` and possibly additional non-standard informationin the `info` dictionary:<jupyter_code>one_a = climate_categories.IPCC2006["1.A"]
print(one_a.title)
print(one_a.comment)
print(one_a.codes)
print(one_a.info)<jupyter_output><empty_output><jupyter_text>For hierarchical categorizations, you can also query for parent and childcategories.Note that a list of sets of children is returned in case a categorycan be composed differently:<jupyter_code>climate_categories.IPCC2006["1.A"].children
climate_categories.IPCC2006["1.A"].parents<jupyter_output><empty_output><jupyter_text>Finally, you can check if a categorization is hierarchical, andfor hierarchical categorizations you can check if the sum of allchild categories should be equal to the sum of parent categories:<jupyter_code>print(f"Hierachical: {climate_categories.IPCC2006.hierarchical}")
print(f"Total sum: {climate_categories.IPCC2006.total_sum}")<jupyter_output><empty_output><jupyter_text>VisualizationThe relationships between categories in a hierarchical categorization can be visualizedin a tree-like fashion:<jupyter_code># Limit the maximum depth shown using `maxdepth`
print(climate_categories.IPCC2006.show_as_tree(maxdepth=2))
# Print only a part of tree using `root`
print(climate_categories.IPCC2006.show_as_tree(root="1A1"))<jupyter_output><empty_output><jupyter_text>Child sets in hierarchical categorizationsFor hierarchical categorizations, it is possible that a category can be composed of multiple child sets.As an example, in emissions reporting it is possible to report industrial emissions either by industry sectors, or by fuel.In this case, the parent `industry` category has two sets of children: either all the industry sectors, or all of the fuels.You can see this with two toy example categorizations included in `climate_categories`:<jupyter_code>import climate_categories.tests.examples

HierEx = climate_categories.tests.examples.HierEx()
print(
    "Hierarchical categorization with only one way to subdivide the top category:"
)
print(HierEx.show_as_tree())

HierAltEx = climate_categories.tests.examples.HierAltEx()
print(
    "\nHierarchical categorization with two ways to subdivide the top category:"
)
print(HierAltEx.show_as_tree())<jupyter_output><empty_output><jupyter_text>As you can see, alternative ways to subdivide a category are indicated with double lines in `show_as_tree`.Programmatically, the difference is clear because `children` contains a list of possible child sets:<jupyter_code>HierEx["0"].children
HierAltEx["0"].children<jupyter_output><empty_output><jupyter_text>Extending categorizationsOften, you want to use a common categorization, but for one reason oranother, you have to add a couple of categories. This is possible:<jupyter_code>IPCC2006_lulucf_extra = climate_categories.IPCC2006.extend(
    name="IPCC2006_lulucf_extra",
    categories={
        "M0.EL": {
            "title": "Total excluding lulucf",
            "comment": "All emissions and removals except emissions from land use, land use change, and forestry",
        }
    },
    children=[("M0.EL", ("1", "2", "4", "5"))],
)

print(IPCC2006_lulucf_extra.name)
print(IPCC2006_lulucf_extra.title)
print(IPCC2006_lulucf_extra.comment)
print(IPCC2006_lulucf_extra.show_as_tree(maxdepth=2))<jupyter_output><empty_output><jupyter_text>If the canonical top level category of hierarchicalcategorizations is defined, you can also calculate the level of a category in thehierarchy:<jupyter_code>print(climate_categories.IPCC2006["0"].level)
print(climate_categories.IPCC2006["1.A"].level)<jupyter_output><empty_output><jupyter_text>Pandas integrationFor each categorization, the categories are also available as a pandasDataFrame:<jupyter_code>climate_categories.IPCC2006.df<jupyter_output><empty_output><jupyter_text>Finding unknown codesSearching for a code in all included categorizations is possibleusing the `find_code` function:<jupyter_code>climate_categories.find_code("1A")<jupyter_output><empty_output><jupyter_text>Conversions Included conversionsYou can get the rules for conversion between categories usingthe conversion objects:<jupyter_code>conv = climate_categories.IPCC1996.conversion_to("IPCC2006")
conv<jupyter_output><empty_output><jupyter_text>The conversion object can be queried for metadata:<jupyter_code>print(conv.categorization_a)
print(conv.categorization_b)
print(conv.auxiliary_categorizations)
print(conv.comment)
print(conv.institution)
print(conv.references)
print(conv.version)
print(conv.last_update)<jupyter_output><empty_output><jupyter_text>More importantly, the conversion object also holds the actual conversion rules:<jupyter_code>len(conv.rules)
conv.rules[0]<jupyter_output><empty_output><jupyter_text>For the rules, the most important parts are the categories and factors for each side and the specification for which auxiliary categories the rule is valid:<jupyter_code>rule = conv.rules[0]
print(rule.factors_categories_a)
print(rule.factors_categories_b)
print(rule.auxiliary_categories)<jupyter_output><empty_output><jupyter_text>In this example, the category 1 of the IPCC1996 categorization equals exactly the category 1 of the IPCC2006 categorization (i.e. for both, the factor is unity) and the rule is valid for all gases (the set of gases is empty, meaning an unrestricted rule). Often, you might be interested in rules concerning a specific set of categories, which you can also fetch:<jupyter_code>one_or_two_rules = conv.relevant_rules(
    {climate_categories.IPCC1996["1"], climate_categories.IPCC1996["2"]}
)

for rule in one_or_two_rules:
    print("###")
    print(rule.format_human_readable())<jupyter_output><empty_output><jupyter_text>Here, we used the `format_human_readable()` function to get a nicely formatted output.You can see that for this conversion, the category 1 maps cleanly between the two categorizations, but for category 2, you need to combine categories 2 and 3 from IPCC1996 to get category 2 from IPCC2006.  Finding inconsistenciesDealing with large conversion rule sets can be daunting, so there are a few functions to help when checking conversions:<jupyter_code># describe all of the conversion rules
# We only print the start of the description because it is looong
print(conv.describe_detailed()[:200] + "…")
# Find potential over counting problems
suspected_problems = conv.find_over_counting_problems()
for p in suspected_problems:
    print(p)
    print()<jupyter_output><empty_output><jupyter_text>Note that `find_over_counting_problems` at the moment can't reliably detect all over counting problems and also some suspected problems might be fine under closer examination.Use this function only to generate hints for possible problems.<jupyter_code># Find unmapped categories
missing_1996, missing_2006 = conv.find_unmapped_categories()
# returns sets of missing categories
# unfortunately, this conversion is not very complete:
print(len(missing_1996))
print(len(missing_2006))
# the sets just contain regular categories which were forgotten
next(iter(missing_1996))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07948934120197519, 'fraction_numerical': 0.023846802360592556, 'mean_word_length': 7.919441460794844, 'pattern_counts': {'":': 3, '<': 85, '<?xml version=': 0, '>': 85, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '109076', 'license': 'Apache-2.0', 'n_tokens_mistral': 2642, 'n_tokens_neox': 2447, 'n_words': 709, 'path': 'docs/usage.ipynb', 'repo_name': 'rgieseke/climate_categories', 'size': '8303'}