<commit_before>module AST
  class Processor
    # The processor module is a module which helps transforming one
    # AST into another.  In a nutshell, the {#process} method accepts
    # a {Node} and dispatches it to a handler corresponding to its
    # type, and returns a (possibly) updated variant of the node.
    #
    # The processor module has a set of associated design patterns.
    # They are best explained with a concrete example. Let's define a
    # simple arithmetic language and an AST format for it:
    #
    # Terminals (AST nodes which do not have other AST nodes inside):
    #
    #   * `(integer <int-literal>)`,
    #
    # Nonterminals (AST nodes with other nodes as children):
    #
    #   * `(add <node> <node>)`,
    #   * `(multiply <node> <node>)`,
    #   * `(divide <node> <node>)`,
    #   * `(negate <node>)`,
    #   * `(store <node> <string-literal>)`: stores value of `<node>`
    #     into a variable named `<string-literal>`,
    #   * `(load <string-literal>)`: loads value of a variable named
    #     `<string-literal>`,
    #   * `(each <node> ...): computes each of the `<node>`s and
    #     prints the result.
    #
    # All AST nodes have the same Ruby class, and therefore they don't
    # know how to traverse themselves. (A solution which dynamically
    # checks the type of children is possible, but is slow and
    # error-prone.) So, a class including the module which knows how
    # to traverse the entire tree should be defined.  Such classes
    # have a handler for each nonterminal node which recursively
    # processes children nodes:
    #
    #     require 'ast'
    #
    #     class ArithmeticsProcessor
    #       include AST::Processor::Module
    #       # This method traverses any binary operators such as (add)<commit_msg>correct Mixin module constant in code example (#13)<commit_after>module AST
  class Processor
    # The processor module is a module which helps transforming one
    # AST into another.  In a nutshell, the {#process} method accepts
    # a {Node} and dispatches it to a handler corresponding to its
    # type, and returns a (possibly) updated variant of the node.
    #
    # The processor module has a set of associated design patterns.
    # They are best explained with a concrete example. Let's define a
    # simple arithmetic language and an AST format for it:
    #
    # Terminals (AST nodes which do not have other AST nodes inside):
    #
    #   * `(integer <int-literal>)`,
    #
    # Nonterminals (AST nodes with other nodes as children):
    #
    #   * `(add <node> <node>)`,
    #   * `(multiply <node> <node>)`,
    #   * `(divide <node> <node>)`,
    #   * `(negate <node>)`,
    #   * `(store <node> <string-literal>)`: stores value of `<node>`
    #     into a variable named `<string-literal>`,
    #   * `(load <string-literal>)`: loads value of a variable named
    #     `<string-literal>`,
    #   * `(each <node> ...): computes each of the `<node>`s and
    #     prints the result.
    #
    # All AST nodes have the same Ruby class, and therefore they don't
    # know how to traverse themselves. (A solution which dynamically
    # checks the type of children is possible, but is slow and
    # error-prone.) So, a class including the module which knows how
    # to traverse the entire tree should be defined.  Such classes
    # have a handler for each nonterminal node which recursively
    # processes children nodes:
    #
    #     require 'ast'
    #
    #     class ArithmeticsProcessor
    #       include AST::Processor::Mixin
    #       # This method traverses any binary operators such as (add)
{'fraction_non_alphanumeric': 0.10030228084638637, 'fraction_numerical': 0.0005496015388843089, 'mean_word_length': 2.676767676767677, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6243748', 'n_tokens_mistral': 1060, 'n_tokens_neox': 1007, 'n_words': 442}