<jupyter_start><jupyter_text>Automatic table parsing - complex modelsThis example demonstrates how complex model structures can automatically be extracted from tables, taking the appropriate `required` flags into account, for each submodel. This is the table we want to parse (`./data/table_example_3.csv`):+----------------------------+--------+---------------+-------------------+--------------+-----------+| Composition                | TC (K) | Enthalpy (kJ) | −S diff(J kg−1 K) | CCK (J kg−1) | Ref.      |+----------------------------+--------+---------------+-------------------+--------------+-----------+| MnO2                       | 293    | 5             | 9.5               | 410          | 52–55     |+----------------------------+--------+---------------+-------------------+--------------+-----------+| MnO2                       | 293    | 1             | 3.25              | —            | 56        |+----------------------------+--------+---------------+-------------------+--------------+-----------+| La2 (PWD)                  | 337    | 1             | 2.70              | 68           | 57        |+----------------------------+--------+---------------+-------------------+--------------+-----------+| La0.67Ba0.33 (TF)          | 292    | 5             | 1.48              | 161          | 26        |+----------------------------+--------+---------------+-------------------+--------------+-----------+| La0.67Ca0.33 (TF)          | 252    | 5             | 2.08              | 175          | 26        |+----------------------------+--------+---------------+-------------------+--------------+-----------+| La0.67Sr0.33MnO3 (TF)      | 312    | 1.5           | 1.54              | 50.16        | 58        |+----------------------------+--------+---------------+-------------------+--------------+-----------+| La0.67Sr0.33MnO3 (TF)      | 321    | 1.5           | 1.47              | 34.24        | 58        |+----------------------------+--------+---------------+-------------------+--------------+-----------+| Ba0.33Mn0.98Ti0.02O3 (PWD) | 309    | 1             | 0.93              | 45           | 39        |+----------------------------+--------+---------------+-------------------+--------------+-----------+| Ba0.33Mn0.98Ti0.02O3 (PWD) | 309    | 5             | 3.19              | 307          | 39        |+----------------------------+--------+---------------+-------------------+--------------+-----------+| Ba0.33Mn0.98Ti0.02O3 (TF)  | 286    | 5             | 3.35              | 220          | This work |+----------------------------+--------+---------------+-------------------+--------------+-----------+| Ba0.33Mn0.98Ti0.02O3 (TF)  | 286    | 1             | 0.99              | 49           | This work |+----------------------------+--------+---------------+-------------------+--------------+-----------+ We are interested in the Curie Temperature `TC(K)`, so we create an appropriate model for it. Note that we are also assigning an enthalpy and a reference to the property, so we have to create models for those properties too, and nest them within the `CurieTemperature` model. The reference also contains a ficticious model that we call `AbsentTemperature` indicating that it will not be found in this table. Note that we are specifying `required` flags for each submodel. The `contextual = True` flag we have set for the `compound` model indicates that the compound could be merged from elsewhere in the document or the caption of the table.<jupyter_code>from chemdataextractor.doc import Caption, Document
from chemdataextractor.doc.table import Table
from chemdataextractor.model import TemperatureModel, StringType, Compound, ModelType, DimensionlessModel
from chemdataextractor.model.units.energy import EnergyModel
from chemdataextractor.parse import W, R
from chemdataextractor.parse.auto import AutoTableParser
class AbsentTemperature(TemperatureModel):
    specifier = StringType(parse_expression=W('Nothing'), required=True)
    compound = ModelType(Compound, required=True, contextual=True)
    parser = [AutoTableParser()]

class Reference(DimensionlessModel):
    specifier = StringType(parse_expression=R('Ref'), required=True)
    compound = ModelType(Compound, required=True, contextual=True)
    absent_temperature = ModelType(AbsentTemperature, required=True, contextual=True)
    parsers = [AutoTableParser()]

class Enthalpy(EnergyModel):
    specifier = StringType(parse_expression=R('Enthalpy'), required=True)
    compound = ModelType(Compound, required=True, contextual=True)
    parsers = [AutoTableParser()]

class CurieTemperature(TemperatureModel):
    specifier = StringType(parse_expression=R(r'^\[?T(C|c)(urie)?[1-2]?\]?$'), required=True)
    compound = ModelType(Compound, required=True, contextual=True)
    enthalpy = ModelType(Enthalpy, required=True, contextual=True)
    reference = ModelType(Reference, required=False, contextual=True)
    parsers = [AutoTableParser()]<jupyter_output><empty_output><jupyter_text>We then process a document containing only the table:<jupyter_code>doc = Document(Table(caption=Caption(""), table_data="./data/table_example_3.csv"))
doc.models = [CurieTemperature]

for record in doc.records:
    print(record.serialize())<jupyter_output>{'CurieTemperature': {'raw_value': '293', 'raw_units': '(K)', 'value': [293.0], 'units': 'Kelvin^(1.0)', 'specifier': 'TC', 'compound': {'Compound': {'names': ['MnO2']}}, 'enthalpy': {'Enthalpy': {'raw_value': '5', 'raw_units': '(kJ)', 'value': [5.0], 'units': '(10^3.0) * Joule^(1.0)', 'specifier': 'Enthalpy', 'compound': {'Compound': {'names': ['MnO2']}}}}}}
{'CurieTemperature': {'raw_value': '293', 'raw_units': '(K)', 'value': [293.0], 'units': 'Kelvin^(1.0)', 'specifier': 'TC', 'compound': {'Compound': {'names': ['MnO2']}}, 'enthalpy': {'Enthalpy': {'raw_value': '1', 'raw_units': '(kJ)', 'value': [1.0], 'units': '(10^3.0) * Joule^(1.0)', 'specifier': 'Enthalpy', 'compound': {'Compound': {'names': ['MnO2']}}}}}}
{'CurieTemperature': {'raw_value': '337', 'raw_units': '(K)', 'value': [337.0], 'units': 'Kelvin^(1.0)', 'specifier': 'TC', 'compound': {'Compound': {'names': ['La2']}}, 'enthalpy': {'Enthalpy': {'raw_value': '1', 'raw_units': '(kJ)', 'value': [1.0], 'units': '(10^3.0) * Joul[...]<jupyter_text>As we can see, the `reference` is not founds in the records, considering that it was not required and it contained a model that was required and not present.If we set the `reference` field to `required = True`, we will get no records at all:<jupyter_code>CurieTemperature.reference.required = True

for record in doc.records:
    print(record.serialize())<jupyter_output><empty_output><jupyter_text>Alternatively, if we set the `absent_temperature` to not required we will get everything, including the reference:<jupyter_code>Reference.absent_temperature.required = False

for record in doc.records:
    print(record.serialize())<jupyter_output>{'CurieTemperature': {'raw_value': '293', 'raw_units': '(K)', 'value': [293.0], 'units': 'Kelvin^(1.0)', 'specifier': 'TC', 'compound': {'Compound': {'names': ['MnO2']}}, 'enthalpy': {'Enthalpy': {'raw_value': '5', 'raw_units': '(kJ)', 'value': [5.0], 'units': '(10^3.0) * Joule^(1.0)', 'specifier': 'Enthalpy', 'compound': {'Compound': {'names': ['MnO2']}}}}, 'reference': {'Reference': {'raw_value': '52–55', 'value': [52.0, 55.0], 'specifier': 'Ref', 'compound': {'Compound': {'names': ['MnO2']}}}}}}
{'CurieTemperature': {'raw_value': '293', 'raw_units': '(K)', 'value': [293.0], 'units': 'Kelvin^(1.0)', 'specifier': 'TC', 'compound': {'Compound': {'names': ['MnO2']}}, 'enthalpy': {'Enthalpy': {'raw_value': '1', 'raw_units': '(kJ)', 'value': [1.0], 'units': '(10^3.0) * Joule^(1.0)', 'specifier': 'Enthalpy', 'compound': {'Compound': {'names': ['MnO2']}}}}, 'reference': {'Reference': {'raw_value': '56', 'value': [56.0], 'specifier': 'Ref', 'compound': {'Compound': {'names': ['MnO2']}}}}}}
{[...]
{'fraction_non_alphanumeric': 0.3146817895400126, 'fraction_numerical': 0.03982356647763075, 'mean_word_length': 1.9611940298507462, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '445720', 'license': 'MIT', 'n_tokens_mistral': 2480, 'n_tokens_neox': 2207, 'n_words': 582, 'path': 'examples/automatic_parsing_of_complex_models.ipynb', 'repo_name': 'Dingyun-Huang/chemdataextractor2', 'size': '7935'}