<jupyter_start><jupyter_text>This notebook explores the structure of [Schema.org](http://schema.org/) types.<jupyter_code>import json
with open("../rocrate/data/schema.jsonld") as f:
    schema = json.load(f)
entities = schema["@graph"]
len(entities)
entities[0]<jupyter_output><empty_output><jupyter_text>Normalize: ensure that the value of `"@type"` and `"rdfs:subClassOf"` is always a list.<jupyter_code>for e in entities:
    types = e.get("@type", [])
    if isinstance(types, str):
        e["@type"] = [types]
    subclasses = e.get("rdfs:subClassOf", [])
    if isinstance(subclasses, dict):
        e["rdfs:subClassOf"] = [subclasses]<jupyter_output><empty_output><jupyter_text>Flatten `"rdfs:subClassOf"`<jupyter_code>for e in entities:
    try:
        subclasses = e["rdfs:subClassOf"]
    except KeyError:
        pass
    else:
        e["rdfs:subClassOf"] = [_["@id"] for _ in subclasses]
entities[0]<jupyter_output><empty_output><jupyter_text>Schema.org entities have one or more types. Since (some) entities are the actual types we are looking for, we'll call their types "metatypes". Is `rdfs:Class` the only metatype?<jupyter_code>metatypes = set.union(*(set(e["@type"]) for e in entities))
metatypes<jupyter_output><empty_output><jupyter_text>There's also `rdf:Property` and several other entities we'll explore later. We are interested in the root(s) of the type hierarchy. Which types have no superclass? Probably properties: not because they're root types, but because they're not types at all. Let's check this:<jupyter_code>for e in entities:
    if "rdf:Property" in e["@type"]:
        assert "rdfs:subClassOf" not in e<jupyter_output><empty_output><jupyter_text>Let's filter out properties then:<jupyter_code>types = [e for e in entities if "rdf:Property" not in e["@type"]]
len(types)<jupyter_output><empty_output><jupyter_text>Which types have no superclass?<jupyter_code>no_superclass = [e for e in types if "rdfs:subClassOf" not in e]
len(no_superclass)<jupyter_output><empty_output><jupyter_text>Are all these root types?<jupyter_code>no_superclass[0]<jupyter_output><empty_output><jupyter_text>This is not an `rdfs:Class`, but a `PhysicalExam`. That's one of the "other" metatypes seen above.<jupyter_code>type_map = {e["@id"]: e for e in types}
type_map['http://schema.org/PhysicalExam']<jupyter_output><empty_output><jupyter_text>`PhysicalExam` is a subclass of `MedicalEnumeration` (and of `MedicalProcedure`)<jupyter_code>type_map['http://schema.org/MedicalEnumeration']<jupyter_output><empty_output><jupyter_text>`MedicalEnumeration` is, in turn, a subclass of `Enumeration`.<jupyter_code>enumeration = type_map['http://schema.org/Enumeration']
enumeration<jupyter_output><empty_output><jupyter_text>Thus, `Neck` is one of the possible values for the `PhysicalExam` enum. Let's get all enums (descendants of `Enumeration`) recursively:<jupyter_code>def r_enumerations(pid='http://schema.org/Enumeration'):
    for e in types:
        if pid in e.get('rdfs:subClassOf', []):
            yield e
            for se in r_enumerations(e["@id"]):
                yield se
enums = set(_["@id"] for _ in r_enumerations())
len(enums)
next(iter(enums))<jupyter_output><empty_output><jupyter_text>Let's see what's left if we filter out these from the metatypes<jupyter_code>metatypes.difference(enums)
type_map["http://schema.org/DataType"]<jupyter_output><empty_output><jupyter_text>`DataType` is both an instance and a subclass of `rdfs:Class` (both a type and a metatype).<jupyter_code>[_ for _ in types if "rdfs:Class" in _.get("rdfs:subClassOf", [])]<jupyter_output><empty_output><jupyter_text>So `DataType` is the only type that's a subclass of `rdfs:Class`.<jupyter_code>datatype_values = [_ for _ in types if "http://schema.org/DataType" in _["@type"]]
datatype_values<jupyter_output><empty_output><jupyter_text>So `DataType` is effectively also an enum, with six possible values. What about `Boolean`?<jupyter_code>boolean_values = [_ for _ in types if "http://schema.org/Boolean" in _["@type"]]
boolean_values<jupyter_output><empty_output><jupyter_text>Also an enum, and a subclass of the `DataType` enum. To summarize, metatypes are either `rdf:Property`, `rdfs:Class` or enums.<jupyter_code>for t in "DataType", "Boolean":
    enums.add(f"https://schema.org/{t}")<jupyter_output><empty_output><jupyter_text>Some enum values like `Neck` or `False` have no superclass, so they are part of the candidate root types set we've built above. Let's build a set of all enum values and subtract it from the candidate root types.<jupyter_code>enum_values = set()
for entity in types:
    if set(entity["@type"]) <= enums:
        enum_values.add(entity["@id"])
len(enum_values)
next(iter(enum_values))<jupyter_output><empty_output><jupyter_text>Add values from the "special" `DataType` and `Boolean` enums.<jupyter_code>enum_values |= set(_["@id"] for _ in datatype_values)
enum_values |= {"http://schema.org/False", "http://schema.org/True"}<jupyter_output><empty_output><jupyter_text>Back to the types with no superclass:<jupyter_code>no_superclass_ids = set(_["@id"] for _ in no_superclass)
no_superclass_ids - enum_values<jupyter_output><empty_output><jupyter_text>So the only "real" root type is `Thing`. All other types that don't have a superclass are enum values. However, some enum values have a superclass:<jupyter_code>enum_values_with_superclass = enum_values - no_superclass_ids
enum_values_with_superclass<jupyter_output><empty_output><jupyter_text>These are the values of which enum?<jupyter_code>set.union(*(set(type_map[_]["@type"]) for _ in enum_values_with_superclass))<jupyter_output><empty_output><jupyter_text>And what are their superclasses?<jupyter_code>set.union(*(set(type_map[_]["rdfs:subClassOf"]) for _ in enum_values_with_superclass))<jupyter_output><empty_output><jupyter_text>Mapping the schema.org type hierarchy to Python classes We can map enums to Python [enums](https://docs.python.org/3.6/library/enum.html).<jupyter_code>from enum import Enum<jupyter_output><empty_output><jupyter_text>`Enum` can be considered as the mapping for `Enumeration`. All other enumerations can be mapped to Enum subclasses. Note that only leaf enumerations have values, while superclasses must have no members. We already know the `Enumeration -> MedicalEnumeration -> PhysicalExam` hierarchy: let's try to map it.<jupyter_code>physical_exam_values = sorted(
    _.rsplit("/", 1)[-1] for _ in enum_values
    if "http://schema.org/PhysicalExam" in type_map[_]["@type"]
)
physical_exam_values
class MedicalEnumeration(Enum):
    pass

PhysicalExam = MedicalEnumeration("PhysicalExam", physical_exam_values)
e = PhysicalExam.Neck
e == PhysicalExam.Ear
isinstance(e, MedicalEnumeration)<jupyter_output><empty_output><jupyter_text>With respect to the current ro-crate-py model, in principle, these types could avoid being derived from `Entity`, but then the logic for adding and manipulating entites must take into account this different kind of entity.What about "regular" (non-enum) types? We can have `Thing` map to our `Entity`, then it's only a matter of deriving other classes as required. Or is it?<jupyter_code>regular_types = set(_["@id"] for _ in types) - enums
superclasses = set.union(*[set(type_map[_].get("rdfs:subClassOf", [])) for _ in regular_types])
leaves = regular_types - superclasses
sorted(leaves)[0]<jupyter_output><empty_output><jupyter_text>Here's our first problem: `3DModel` is not a valid Python identifier since it starts with a digit. Let's ignore this for now and pick another one.<jupyter_code>sorted(leaves)[-1]
def parents(t):
    return [type_map[_] for _ in type_map[t]["rdfs:subClassOf"]]
parents('http://schema.org/Zoo')
parents('http://schema.org/CivicStructure')
parents('http://schema.org/Place')<jupyter_output><empty_output><jupyter_text>In the general case there might be multiple inheritance. Also, of course the above naive code is not the most efficient way of generating the whole class hierarchy. In this case, though, the structure could be:<jupyter_code>class Entity:  # in the real case this would be the ro-crate-py Entity class
    pass

class Place(Entity):
    pass

class CivicStructure(Place):
    pass

class Zoo(CivicStructure):
    pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12042074718897353, 'fraction_numerical': 0.0012090436464756377, 'mean_word_length': 6.398926654740608, 'pattern_counts': {'":': 1, '<': 114, '<?xml version=': 0, '>': 115, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '51346', 'license': 'Apache-2.0', 'n_tokens_mistral': 2793, 'n_tokens_neox': 2748, 'n_words': 733, 'path': 'notebooks/ExploreSchemaOrg.ipynb', 'repo_name': 'simleo/ro-crate-py', 'size': '8271'}