<commit_before># Katuv DSL redux

1. whitequark/ast now central
2. three main parts
  2.1. katuv/core
      The core implementation of the katuv dsl-dsl. Generally unseen by users
  2.2. katuv/dsl
      The implementation of the DSL, written as raw classes using katuv/core
  2.3. katuv/processor
      A Katuv-defined DSL for defining AST::Processors

  `require 'katuv'` pulls in the first two things. The processor must be pulled
  in manually.

3. Two modes, interpreter or compiler.
  3.1. Interpreter = dynamically define classes as needed within some namespace
  3.2. Compiler does interpreter in some namespace, unparses so they can be
       statically required (good for bootstrapping), and then removes from the
       namespace (if running in another program) or exits.

## Use:

The easiest way to show how to define a DSL with katuv (in the ideal case) is by
showing you:

    # starting the definition

    dsl "Some::Namespace::Here" do
    end

    #or

    dsl :Namespace do
    end

    ----------------------------------------------------------------------------

    # defining an actual dsl

    dsl :MyDsl do
      # I don't like the 'allowed_in_root' bit.
      terminal :Foo, allowed_in_root: false

      nonterminal :Bar, allowed_in_root: false do
        one :Foo
      end

      nonterminal :Baz do
        one :Foo
        one :Bar
      end

      nonterminal :Bingle do
        many :Foo
      end
    end

    # using this dsl

    my_dsl do

      baz 'name_attribute' do
        foo 'name_attribute'
        bar do #no name attr is fine
          foo 'more names'
        end
      end

      bingle 'naaames' do
        foo '1'
        foo '2'
        foo 'red'
        foo 'blue'
      end
    end

    ----------------------------------------------------------------------------

    # alternative to above:

    dsl :MyDSL do
      terminal :Foo

      nonterminal :Bar do
        one :Foo
      end

      nonterminal :Baz do
        one :Foo
        one :Bar
      end

      nonterminal :Bingle do
        many :Foo
      end

      root :my_dsl do
        maybe_one :Baz
        maybe_one :Bingle
      end
    end

It would be nice to use a ruby-parser or something so I could refer to the
constants AoT, basically suss out where the constants are and define them before
evaluating the DSL as normal against the defined comments.

It'd also be good to make these open, like classes. So that you can do:

    dsl :MyDSL do
      #stuff
    end

    # later, in another file

    dsl :MyDSL do
      #more stuff
    end

Which is nice from a maintenance point of view (don't need one big file for a
complicated DSL, can break it down / load it in peices if necessary.

## Things to give thought to

1. What can be done to make things easy to test? What helpers are there?
  1.1. `#parses?` might be nice, something like:

           dsl(:MyDSL).parses do
             some block
           end

       The implementation could be quite simple (just assert it doesn't raise a
       parse error

  1.2. Half-completed DSL elements should return their parse-tree, so that you
       can add to them dynamically. ie:

           dsl :MyDSL do
             f = bar 'something'
             f.foo 'something else'
           end

       would be equivalent to:

           dsl :MyDSL do
             bar 'something' do
               foo 'something else'
             end
           end

       This makes it easier to deal with elements peicemeal in tests.

<commit_msg>add some notes about processors<commit_after># Katuv DSL redux

1. whitequark/ast now central
2. three main parts
  2.1. katuv/core
      The core implementation of the katuv dsl-dsl. Generally unseen by users
  2.2. katuv/dsl
      The implementation of the DSL, written as raw classes using katuv/core
  2.3. katuv/processor
      A Katuv-defined DSL for defining AST::Processors

  `require 'katuv'` pulls in the first two things. The processor must be pulled
  in manually.

3. Two modes, interpreter or compiler.
  3.1. Interpreter = dynamically define classes as needed within some namespace
  3.2. Compiler does interpreter in some namespace, unparses so they can be
       statically required (good for bootstrapping), and then removes from the
       namespace (if running in another program) or exits.

## Use:

The easiest way to show how to define a DSL with katuv (in the ideal case) is by
showing you:

    # starting the definition

    dsl "Some::Namespace::Here" do
    end

    #or

    dsl :Namespace do
    end

    ----------------------------------------------------------------------------

    # defining an actual dsl

    dsl :MyDsl do
      # I don't like the 'allowed_in_root' bit.
      terminal :Foo, allowed_in_root: false

      nonterminal :Bar, allowed_in_root: false do
        one :Foo
      end

      nonterminal :Baz do
        one :Foo
        one :Bar
      end

      nonterminal :Bingle do
        many :Foo
      end
    end

    # using this dsl

    my_dsl do

      baz 'name_attribute' do
        foo 'name_attribute'
        bar do #no name attr is fine
          foo 'more names'
        end
      end

      bingle 'naaames' do
        foo '1'
        foo '2'
        foo 'red'
        foo 'blue'
      end
    end

    ----------------------------------------------------------------------------

    # alternative to above:

    dsl :MyDSL do
      terminal :Foo

      nonterminal :Bar do
        one :Foo
      end

      nonterminal :Baz do
        one :Foo
        one :Bar
      end

      nonterminal :Bingle do
        many :Foo
      end

      root :my_dsl do
        maybe_one :Baz
        maybe_one :Bingle
      end
    end

It would be nice to use a ruby-parser or something so I could refer to the
constants AoT, basically suss out where the constants are and define them before
evaluating the DSL as normal against the defined comments.

It'd also be good to make these open, like classes. So that you can do:

    dsl :MyDSL do
      #stuff
    end

    # later, in another file

    dsl :MyDSL do
      #more stuff
    end

Which is nice from a maintenance point of view (don't need one big file for a
complicated DSL, can break it down / load it in peices if necessary.

## Things to give thought to

1. What can be done to make things easy to test? What helpers are there?
  1.1. `#parses?` might be nice, something like:

           dsl(:MyDSL).parses do
             some block
           end

       The implementation could be quite simple (just assert it doesn't raise a
       parse error

  1.2. Half-completed DSL elements should return their parse-tree, so that you
       can add to them dynamically. ie:

           dsl :MyDSL do
             f = bar 'something'
             f.foo 'something else'
           end

       would be equivalent to:

           dsl :MyDSL do
             bar 'something' do
               foo 'something else'
             end
           end

       This makes it easier to deal with elements peicemeal in tests.

## Processors

We should be able to define a processor using a nice DSL. When defining a DSL,
we should automatically define an appropriate `AST::Processor` subclass. Again,
an example is informative:


    # I don't love the positional argument here, but it's hard to know which is
    # better, DSL name as the name attribute, or name-of-processor as the name
    # attribute
    processor :MyDSL, "Name of Processor" do
      on :Foo do |node|
        puts "Seen raw Foo #{node.name}"
        continue_with bar.foo(node)
      end

      on :Bar do |node|
        foo = *node
        puts "Seen in Bar #{foo.name}"
        # no continue, so it stops here
      end
    end

Or something roughly like that. The idea is to provide a couple helpers to make
defining these suckers easy (including stuff like having the DSL methods
available to make defining new syntax nodes easy). This will require a bit of
work to get right. The idea is that this peice of code should walk down the tree
and emit chunks of the tree to rebuild it (like proteins unzipping DNA into RNA)
{'fraction_non_alphanumeric': 0.08599629858112276, 'fraction_numerical': 0.004935225169648365, 'mean_word_length': 1.8214410024364776, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2051767', 'n_tokens_mistral': 2340, 'n_tokens_neox': 2185, 'n_words': 1064}