<jupyter_start><jupyter_text>ParsingBefore starting, it's important to look at the [parser API](../api/parser.rstmodule-yangify.parser) and [parse_indented_config](../api/parser.rstyangify.parser.text_tree.parse_indented_config). It may look like a lot and some parts may not make sense at this point but it will help understanding the tutorial.We are going to see how to parse configuration by example. To do that we are going to use a modified/simplified version of the ``openconfig-interfaces`` and ``openconfig-vlan`` models. Note they have been slightly modified for simplicity and brevity.Let's start by looking at the ASCII tree representation of the model:```+--rw openconfig-interfaces:interfaces|  +--rw interface* [name]|     +--rw config|     |  +--rw description? |     |  +--rw enabled? |     |  +--rw loopback-mode? |     |  +--rw mtu? |     |  +--rw name? |     |  +--rw type |     +--rw name |     +--rw subinterfaces|        +--rw subinterface* [index]|           +--rw config|           |  +--rw description? |           |  +--rw enabled? |           |  +--rw index? |           +--rw index +--rw openconfig-vlan:vlans   +--rw vlan* [vlan-id]      +--rw config      |  +--rw name?       |  +--rw status?       |  +--rw vlan-id?       +--rw vlan-id ```To write a parser you are going to need to write a class following the rules below:1. A grouping (either a list or a container) is represented by a class that inherits from ``yangify.parsers.Parser``.2. Classes that implement a part of the tree are nested in the parent object and are named as the grouping it implements.3. Each class may have a nested class named ``Yangify`` that inherits from ``yangify.parsers.ParserData``. This class may implement code to help with the processing. See the API documentation for details.4. Finally, leaves are processed by a function named after the leaf and will have to return an object of the correct type.Using the ``openconfig-vlans:vlans`` as an example, a parser may look like this:``` pythonclass Vlans(Parser):    class vlan(Parser):        class Yangify(ParserData):            def extract_elements(self):                 code to extra each element from the config goes here        class config(Parser):            def vlan_id(self):                 code to parse the vlan id goes here            def name(self):                 code to parse the name goes here            def status(self):                 code to parse the vlan status goes here        def vlan_id(self):                 code to parse the vlan id goes here```This format may be a bit cumbersome in deeply nested models, hence, an alternative maybe to use classes in the global namespace instead, for instance:``` pythonclass VlanConfig(Parser):    def vlan_id(self):         code to parse the vlan id goes here    def name(self):         code to parse the name goes here    def status(self):         code to parse the vlan status goes here            class Vlan(Parser):    class Yangify(ParserData):        def extract_elements(self):             code to extra each element from the config goes here    config = VlanConfig                def vlan_id(self):             code to parse the vlan id goes hereclass Vlans(Parser):    vlan = Vlan```Both are equivalent and both have their advantages and disadvantages when it comes to readibility. We are going to use the second method for this demo.  The `openconfig-interfaces` parserTo explain how this works we are going to write a parser that reads a configuration belonging to an IOS device and maps it into the model. Let's look at the configuration first:<jupyter_code>%cat data/ios/config.txt<jupyter_output>interface FastEthernet1
   description This is Fa1
   shutdown
   exit
!
interface FastEthernet1.1
   description This is Fa1.1
   exit
!
interface FastEthernet1.2
   description This is Fa1.2
   exit
!
interface FastEthernet3
   description This is Fa3
   no shutdown
   exit
!
interface FastEthernet4
   shutdown
   exit
!
vlan 10
   name prod
   no shutdown
   exit
!
vlan 20
   name dev
   shutdown
   exit
!<jupyter_text>Nothing very complex, just a few interfaces and subinterfaces and a couple of vlans. The code for this tutorial is in `tutorial_parser.py`, so let's start by importing it:<jupyter_code>import tutorial_parser<jupyter_output><empty_output><jupyter_text>Now that we have imported the code let's start looking at the code piece by piece. The starting point is going to be the class `Interfaces` which will be used to parse `openconfig-interfaces:interfaces`:<jupyter_code>show_code(tutorial_parser.Interfaces)<jupyter_output>[38;5;28;01mclass[39;00m [38;5;21;01mInterfaces[39;00m(Parser):
    [38;5;124m"""[39m
[38;5;124m    Implements openconfig-interfaces:interfaces[39m
[38;5;124m    """[39m

    interface [38;5;241m=[39m Interface<jupyter_text>Not much here. The container `openconfig-interfaces:interfaces` only has a YANG list in the `interface` node. As `interface` is a grouping a different `Parser` class is used. Let's look at it:<jupyter_code>show_code(tutorial_parser.Interface)<jupyter_output>[38;5;28;01mclass[39;00m [38;5;21;01mInterface[39;00m(Parser):
    [38;5;124m"""[39m
[38;5;124m    Implements openconfig-interfaces:interfaces/interface[39m
[38;5;124m    """[39m

    [38;5;28;01mclass[39;00m [38;5;21;01mYangify[39;00m(ParserData):
        [38;5;28;01mdef[39;00m [38;5;21mextract_elements[39m([38;5;28mself[39m) [38;5;241m-[39m[38;5;241m>[39m Iterator[Tuple[[38;5;28mstr[39m, Dict[[38;5;28mstr[39m, Any]]]:
            [38;5;124m"""[39m
[38;5;124m            IOS interfaces are in the root of the configuration. However,[39m
[38;5;124m            subinterfaces are also in the root of the configuration. For[39m
[38;5;124m            that reason we will have to make sure that we filter the[39m
[38;5;124m            subinterfaces as we iterate over all the interfaces in the root[39m
[38;5;124m            of the configuration. That's should be as easy as checking that[39m
[38;5;124m            the interface name has not dot in it.[39m
[[...]<jupyter_text>This one is more interesting. It has a few things:1. A ``Yangify`` class that implements `extract_elements`. This is mandatory as `openconfig-interfaces:interfaces/interface` is a YANG list.2. It has a ``config`` attribute that corresponds to the container `openconfig-interfaces:interfaces/interface/config` and points to the `Parser` class `InterfaceConfig`3. It also has a ``subinterfaces`` attribute that correspononds to the container `openconfig-interfaces:interfaces/interface/subinterfaces`4. Finally, it has a function to extract the leaf corresponding to `openconfig-interfaces:interfaces/interface/name`.Two things to note here:1. As explained in the API section linked in the note block early in this document, `Yangify.extract_elements` is used to extract each element returning its key and relevant configuration data.2. The key is automatically added to the dictionary inside `self.yy.keys` using the path to the node as the key and made easily available via the shortcut property `self.yy.key`. This can be used, for instance, when returning the name of the interface, which we already know as it's the same as key.Now let's look at the `InterfaceConfig` class that implements `openconfig-interfaces:interfaces/interface/config`:<jupyter_code>show_code(tutorial_parser.InterfaceConfig)<jupyter_output>[38;5;28;01mclass[39;00m [38;5;21;01mInterfaceConfig[39;00m(Parser):
    [38;5;124m"""[39m
[38;5;124m    Implements openconfig-interfaces:interfaces/interface/config[39m
[38;5;124m    """[39m

    [38;5;28;01mdef[39;00m [38;5;21mdescription[39m([38;5;28mself[39m) [38;5;241m-[39m[38;5;241m>[39m Optional[[38;5;28mstr[39m]:
        [38;5;28;01mreturn[39;00m cast(Optional[[38;5;28mstr[39m], [38;5;28mself[39m[38;5;241m.[39myy[38;5;241m.[39mnative[38;5;241m.[39mget([38;5;124m"[39m[38;5;124mdescription[39m[38;5;124m"[39m, {})[38;5;241m.[39mget([38;5;124m"[39m[38;5;124m#text[39m[38;5;124m"[39m))

    [38;5;28;01mdef[39;00m [38;5;21menabled[39m([38;5;28mself[39m) [38;5;241m-[39m[38;5;241m>[39m [38;5;28mbool[39m:
        shutdown [38;5;241m=[39m [38;5;28mself[39m[38;5;241m.[39myy[38;5;241m.[39mnative[38;5;241m.[39mget([38;5;124m"[39m[38;5;124mshutdown[39m[38;5;124m"[39m, {})[38;5;241m.[39mget([38;5;124m"[39m[38;5;12[...]<jupyter_text>This container only has leaves so we only have functions named after those leaves that will return the extracted data.Now let's circle back to `openconfig-interfaces:interfaces/interface` and head down to the `subinterfaces` container, which was parsed with the `Subinterfaces` class:<jupyter_code>show_code(tutorial_parser.Subinterfaces)<jupyter_output>[38;5;28;01mclass[39;00m [38;5;21;01mSubinterfaces[39;00m(Parser):
    [38;5;124m"""[39m
[38;5;124m    Implements openconfig-interfaces:interfaces/interface/subinterfaces[39m
[38;5;124m    """[39m

    subinterface [38;5;241m=[39m Subinterface<jupyter_text>Like the `Interface` class, not much to look at here, let's head down to the `Subinterface` class:<jupyter_code>show_code(tutorial_parser.Subinterface)<jupyter_output>[38;5;28;01mclass[39;00m [38;5;21;01mSubinterface[39;00m(Parser):
    [38;5;124m"""[39m
[38;5;124m    Implements openconfig-interfaces:interfaces/interface/subinterfaces/subinterface[39m
[38;5;124m    """[39m

    [38;5;28;01mclass[39;00m [38;5;21;01mYangify[39;00m(ParserData):
        [38;5;28;01mdef[39;00m [38;5;21mextract_elements[39m([38;5;28mself[39m) [38;5;241m-[39m[38;5;241m>[39m Iterator[Tuple[[38;5;28mstr[39m, Dict[[38;5;28mstr[39m, Any]]]:
            [38;5;124m"""[39m
[38;5;124m            IOS subinterfaces are in the root of the configuration and named following[39m
[38;5;124m            the format ``$parent_interface.$index``. The model specifies the key is[39m
[38;5;124m            the $index, which is just a number. These means we will need to:[39m

[38;5;124m            1. Iterate over all the interfaces[39m
[38;5;124m            2. Filter the ones that don't start by `$parent_interface.`[39m
[38;5;124m            3. Extract the [...]<jupyter_text>This almost looks identical to the `Interface` class. Extracting the elements is a bit tricker as it's explained in the code but it's still quite similar. The `index` function that extracts is also similar to the `name` function in the `Interface` class, however, the index is just what's after the `.` in the interface name, so we need to extract that from the key and convert it into an integer as that's what the mode expects.Finally, let's look at the `SubinterfaceConfig` class:<jupyter_code>show_code(tutorial_parser.SubinterfaceConfig)<jupyter_output>[38;5;28;01mclass[39;00m [38;5;21;01mSubinterfaceConfig[39;00m(Parser):
    [38;5;124m"""[39m
[38;5;124m    Implements openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/config[39m
[38;5;124m    """[39m

    [38;5;28;01mdef[39;00m [38;5;21mdescription[39m([38;5;28mself[39m) [38;5;241m-[39m[38;5;241m>[39m Optional[[38;5;28mstr[39m]:
        [38;5;28;01mreturn[39;00m cast(Optional[[38;5;28mstr[39m], [38;5;28mself[39m[38;5;241m.[39myy[38;5;241m.[39mnative[38;5;241m.[39mget([38;5;124m"[39m[38;5;124mdescription[39m[38;5;124m"[39m, {})[38;5;241m.[39mget([38;5;124m"[39m[38;5;124m#text[39m[38;5;124m"[39m))

    [38;5;28;01mdef[39;00m [38;5;21mindex[39m([38;5;28mself[39m) [38;5;241m-[39m[38;5;241m>[39m [38;5;28mint[39m:
        [38;5;28;01mreturn[39;00m [38;5;28mint[39m([38;5;28mself[39m[38;5;241m.[39myy[38;5;241m.[39mkey[38;5;241m.[39msplit([38;5;124m"[39m[38;5;124m.[39m[38;5;124m"[39m)[[38;5;241m[...]<jupyter_text>At this point I am sure there isn't much to add here :)Now that we have the parser classes we need to create the root parser. The root parser has the following functions:1. Allow the user pick and choose which parsers to use2. Initialize the configuration, if needed.3. Perform some post operations, if needed. Using the parserOur root class is going to load the `Interfaces` parser we explored before and use `parse_indented_config` to prepare the configuration:<jupyter_code>from yangify import parser
from yangify.parser.text_tree import parse_indented_config

class IOSParser(parser.RootParser):
    class Yangify(parser.ParserData):
        def init(self) -> None:
            self.root_native = parse_indented_config(self.native.splitlines())
            self.native = self.root_native

    interfaces = tutorial_parser.Interfaces<jupyter_output><empty_output><jupyter_text>Now we need to load the configuration file:<jupyter_code>with open("data/ios/config.txt", "r") as f:
    config = f.read()<jupyter_output><empty_output><jupyter_text>Now we will create the datamodel as we will need it later on:<jupyter_code>from yangson.datamodel import DataModel
dm = DataModel.from_file("yang/yang-library-data.json", ["yang/yang-modules/ietf", "yang/yang-modules/openconfig"])<jupyter_output><empty_output><jupyter_text>Finally, we are going to instantiate the `IOSParser` and call the `process` method:<jupyter_code>p = IOSParser(dm, native=config)
result = p.process()<jupyter_output><empty_output><jupyter_text>Now that we got the processed object, let's see the result:<jupyter_code>import json
print(json.dumps(result.raw_value(), indent=4))<jupyter_output>{
    "openconfig-interfaces:interfaces": {
        "interface": [
            {
                "name": "FastEthernet1",
                "config": {
                    "name": "FastEthernet1",
                    "type": "iana-if-type:ethernetCsmacd",
                    "description": "This is Fa1",
                    "enabled": false
                },
                "subinterfaces": {
                    "subinterface": [
                        {
                            "index": 1,
                            "config": {
                                "index": 1,
                                "description": "This is Fa1.1"
                            }
                        },
                        {
                            "index": 2,
                            "config": {
                                "index": 2,
                                "description": "This is Fa1.2"
                            }
                        }
                    ]
      [...]<jupyter_text>Adding a second parserIn the previous example we created a parser that only parses the `openconfig-interfaces` model, however, our `tutorial_parser.py` contains code to also parse the `openconfig-vlan` model, let's create a second `RootParser` class that can parse both models:<jupyter_code>from yangify import parser
from yangify.parser.text_tree import parse_indented_config

class IOSParser2(parser.RootParser):
    class Yangify(parser.ParserData):
        def init(self) -> None:
            self.root_native = parse_indented_config(self.native.splitlines())
            self.native = self.root_native

    interfaces = tutorial_parser.Interfaces
    vlans = tutorial_parser.Vlans
p = IOSParser2(dm, native=config)
result = p.process()
import json
print(json.dumps(result.raw_value(), indent=4))<jupyter_output>{
    "openconfig-interfaces:interfaces": {
        "interface": [
            {
                "name": "FastEthernet1",
                "config": {
                    "name": "FastEthernet1",
                    "type": "iana-if-type:ethernetCsmacd",
                    "description": "This is Fa1",
                    "enabled": false
                },
                "subinterfaces": {
                    "subinterface": [
                        {
                            "index": 1,
                            "config": {
                                "index": 1,
                                "description": "This is Fa1.1"
                            }
                        },
                        {
                            "index": 2,
                            "config": {
                                "index": 2,
                                "description": "This is Fa1.2"
                            }
                        }
                    ]
      [...]
{'fraction_non_alphanumeric': 0.12749052683046083, 'fraction_numerical': 0.07211832294340545, 'mean_word_length': 3.2468206592265765, 'pattern_counts': {'":': 36, '<': 48, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '468895', 'license': 'Apache-2.0', 'n_tokens_mistral': 6183, 'n_tokens_neox': 5212, 'n_words': 1360, 'path': 'docs/tutorials/parsing.ipynb', 'repo_name': 'SDN-Projects/yangify', 'size': '16362'}