<jupyter_start><jupyter_text>Detecting SequencesThe engine can detect sequential patterns of assertions. As these rules trigger, they get saved into memory as "arcs", or partially completed rules. Those arcs wait until they are completed before triggering their assertions.This can be useful in natural language and other real-world situations where it's important to assign meaning to a sequence of assertions that occur one after the other. These sequences can in turn form parts of larger sequences and structures, and you can carry information from the lower sequences and structures into the high-order sequences and structures to build up a structure of meaning.To begin, rev up the engine:<jupyter_code>import os, sys
sys.path.insert(1, os.path.abspath('..\\..'))
from thoughts.rules_engine import RulesEngine
from pprint import pprint

# start a new engine
engine = RulesEngine()<jupyter_output><empty_output><jupyter_text>Asserting a Partial SequenceLet's try a simple example in matching the sequence of letters A, B, and C.At first let's just assert only A and B and inspect the result.<jupyter_code>rules = { "#when": ["A", "B", "C"], "#then": "FOUND" }
engine.add_rule(rules)

result = engine.process(["A", "B"], extract_conclusions=True)
pprint(result)<jupyter_output>[]<jupyter_text>Only the incoming assertions were returned, and the rule did not trigger. This makes sense, as the engine is waiting on the final part of the sequence, "C". Asserting the Full SequenceNow let's try asserting the full sequence:<jupyter_code>result = engine.process(["A", "B", "C"], extract_conclusions=True)
pprint(result)<jupyter_output>['FOUND']<jupyter_text>Here all three parts, or *constituents*, of the sequence were found. This triggered the final "FOUND" fact.But why wasn't 'C' returned in the final result? This is an effect of using the extract_conclusions parameter, which only returns the leaf-node (bottom level) facts that were concluded.Let's take a look at the full tree to make this more apparent:<jupyter_code>result = engine.process(["A", "B", "C"], extract_conclusions=False)
pprint(result)<jupyter_output>[{'#assert': 'A', '#conclusions': [], '#seq-end': 1, '#seq-start': 0},
 {'#assert': 'B', '#conclusions': [], '#seq-end': 2, '#seq-start': 1},
 {'#assert': 'C',
  '#conclusions': [{'#assert': 'FOUND',
                    '#conclusions': [],
                    '#seq': [{'#assert': 'A', '#seq-end': 1, '#seq-start': 0},
                             {'#assert': 'B', '#seq-end': 2, '#seq-start': 1},
                             {'#assert': 'C', '#seq-end': 3, '#seq-start': 2}],
                    '#seq-end': 3,
                    '#seq-start': 0}],
  '#seq-end': 3,
  '#seq-start': 2}]<jupyter_text>Above, 'A' and 'B' have no sub-conclusions, so they are returned. 'C' has the sub-conclusion of 'FOUND' and so it is *not* returned. 'FOUND' has no sub-conclusions, so it is returned.  Inspecting ArcsNow let's run the rule again but this time we'll inspect the partial rules, or arcs, which are waiting on the next constituent in the sequence.<jupyter_code>result = engine.process(["A", "B"], extract_conclusions=True)

pprint(engine.context.arcs, sort_dicts=False)<jupyter_output>[{'#when': [{'#assert': 'A', '#seq-start': 0, '#seq-end': 1}, 'B', 'C'],
  '#then': 'FOUND',
  '#seq-idx': 1,
  '#seq-start': 0,
  '#seq-end': 1,
  '#unification': {'?#when': {'#assert': 'A', '#seq-start': 0, '#seq-end': 1}},
  '#is-arc': True},
 {'#when': [{'#assert': 'A', '#seq-start': 0, '#seq-end': 1},
            {'#assert': 'B', '#seq-start': 1, '#seq-end': 2},
            'C'],
  '#then': 'FOUND',
  '#seq-idx': 2,
  '#seq-start': 0,
  '#seq-end': 2,
  '#unification': {'?#when': {'#assert': 'B', '#seq-start': 1, '#seq-end': 2}},
  '#is-arc': True}]<jupyter_text>Above, there are two entries in the partial rules. Notice the system-generated seq-idx, seq-start, and seq-end attributes.The first entry has detected the 'A' constituent, "covering" positions 0 and 1. The rule is "sitting" at index position 1, meaning that it's waiting for the constiuent at that position, which in this example is 'B'. The second entry has detected the 'B' consituent, and has extended the original arc by covering the position from 0 to 2, which includes both the 'A' and 'B' positions. the index is sitting at position 2, meaning that it's waiting for the 'C' constituent.*In general, you can think of the arcs as both "covering" positions and "sitting" at the last position, waiting on the next constituent to arrive.*Also note that *both* arcs are kept in the partial rules. The engine keeps all partial matches to rules, even after extending an arc. This is how it keeps a list of all possibilities, in case another constituent arrives which could also match the rule.Finally, the arcs are cleared by the engine automatically each time the process function runs.  Constituents Must Follow Immediately After Each OtherBy default, the engine will only detect sequences where the next token constituent is immediately following the previous constituent. It assigns a seq-start and seq-end token to facts as they are asserted in the process function. Then it compares to make sure the fact immediately follows the previous constituent in the arc, based on the seq-end and seq-start positions. Compare:<jupyter_code>result = engine.process(["A", "B", "C"], extract_conclusions=True)
pprint(result)<jupyter_output>['FOUND']<jupyter_text>To:<jupyter_code>result = engine.process(["A", "B", "D", "C"], extract_conclusions=True)
pprint(result)<jupyter_output>[]<jupyter_text>Because "C" was not immediately following "A" and "B", the sequence is not detected.We'll see below how to chance this default behavior to allow for "junk" in the sequence.  Overlapping Sequences are DetectedSequences can overlap each other. This works differently than Sets, which by default prevent the same constituent from being matched multiple times in the same rule. See the tutorial on Sets for more details.The rule below looks for any ABA sequences.<jupyter_code>rule = { "#when": ["A", "B", "A"], "#then": "FOUND" }
engine.clear_rules()
engine.add_rule(rule)

result = engine.process(["A", "B", "A", "B", "A"], extract_conclusions=True)
pprint(result)<jupyter_output>['FOUND', 'FOUND']<jupyter_text>The ABA sequence appears twice in the assertion ABABA, so the rule triggers twice. The middle "A" is part of both sequences.  Allowing Disconnected Sequences (aka Junk in Sequences)By default, sequence matching requires that tokens immediately follow each other, with no "junk" tokens in between. You can change this behavior to allow junk by using the allow-junk attribute.<jupyter_code>rule = { "#when": ["A", "B", "C"], 
         "#seq-type": "allow-junk", 
         "#then": "FOUND" }

engine.clear_rules()
engine.add_rule(rule)
engine.process(["A", "B", "D", "E", "F", "C"])<jupyter_output><empty_output><jupyter_text>Allowing junk in sequences may at first appear to "overmatch" and generate more sequences than you expect. Consider the following:<jupyter_code>rule = { "#when": ["A", "B", "C"], 
         "#seq-type": "allow-junk", 
         "#then": "FOUND" }

engine.clear_rules()
engine.add_rule(rule)
engine.process(["A", "B", "X", "A", "B", "C"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13674508185445042, 'fraction_numerical': 0.005090108680698858, 'mean_word_length': 4.706436420722135, 'pattern_counts': {'":': 10, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '227754', 'license': 'MIT', 'n_tokens_mistral': 2259, 'n_tokens_neox': 2166, 'n_words': 884, 'path': 'notebooks/tutorial/025_sequence_based_rules.ipynb', 'repo_name': 'hofmanniac/thoughts', 'size': '7269'}