<commit_before># Forth model proposal

By *<NAME>*, `hwfwguy/at\gmail.com`

The traditional INTERPRET loop in Forth uses FIND as part of an outer interpreter. FIND returns a single *xt*, an execution token that can be used for compilation but not in a straightforward way. One solution is to have a dual-xt system. The header (in name space) contains an xt for execution and an xt for compilation. A new dialect of Forth, called DXT (for dual XT) is proposed. This can be a derivative of the 2012x Forth standard. Applications written in ANS can run on DXT with a compatibility layer written in DXT. Applications written in DXT may run on an ANS system with a non-standard compatibility layer. The DXT version of INTERPRET uses a version of FIND that takes a string token and finds its name token, *nt*, a namespace reference that easily converts to the appropriate xt.

- `PARSE-NAME`  ( "{spaces}name[space]" -- c-addr u )  *2012:* Skip leading space delimiters. Parse name delimited by a space.
- `NAME>COMP`  ( nt -- xt )  xt is the compilation token for the word nt.
- `NAME>INT`  ( nt -- xt )  *gforth:* xt represents the interpretation semantics of the word nt. If nt has no interpretation semantics (i.e. is compile-only), xt is the execution token for ticking-compile-only-error, which performs -2048 throw.
- `FIND-NAME`  ( c-addr u -- 0 nt | c-addr u )  Finds the word from its string token. Performs -16 throw if zero-length string.
- `NUMBER`  ( c-addr u -- ? )  Converts string to a number, performs -13 throw (undefined word) if it can't.
- `NAME>W`  ( nt -- w )  w is a general purpose parameter in the header.
- `NAME>NAME`  ( nt -- c-addr u )  Name of the word.

```
: INTERPRET ( -- )  // DXT interpreter loop
   BEGIN  PARSE-NAME  DUP WHILE
      FIND-NAME OVER IF NUMBER ELSE
         NIP  DUP NT !  STATE @ IF         
         NAME>COMP ELSE
         NAME>EXEC THEN  EXECUTE
      THEN
   REPEAT  2DROP
;
```
IMMEDIATE works by copying the last word's xtE to its xtC.


## Header structures

If there's anything computing has demonstrated, it's the persistence of data structures. Data is like rocks. Code is like sand. It's desirable to define a minimal amount of rock. For the benefit of embedded systems, bloat is optional.

The header structure would start with the counted name string. The MSB of the count byte would contain the *smudge* flag and bit 6 would contain the *immediate* flag. Empty bytes between the end of the string and the next CODE-ALIGNED address would be padded with 0. The rest of the header would contain:<commit_msg>Create xforth.md<commit_after># Forth model proposal

By *<NAME>*, `hwfwguy/at\gmail.com`

The traditional INTERPRET loop in Forth uses FIND as part of an outer interpreter. FIND returns a single *xt*, an execution token that can be used for compilation but not in a straightforward way. One solution is to have a dual-xt system. The header (in name space) contains an xt for execution and an xt for compilation. A new dialect of Forth, called DXT (for dual XT) is proposed. This can be a derivative of the 2012x Forth standard. Applications written in ANS can run on DXT with a compatibility layer written in DXT. Applications written in DXT may run on an ANS system with a non-standard compatibility layer. The DXT version of INTERPRET uses a version of FIND that takes a string token and finds its name token, *nt*, a namespace reference that easily converts to the appropriate xt.

- `PARSE-NAME`  ( "{spaces}name[space]" -- c-addr u )  *2012:* Skip leading space delimiters. Parse name delimited by a space.
- `NAME>COMP`  ( nt -- xt )  xt is the compilation token for the word nt.
- `NAME>INT`  ( nt -- xt )  *gforth:* xt represents the interpretation semantics of the word nt. If nt has no interpretation semantics (i.e. is compile-only), xt is the execution token for ticking-compile-only-error, which performs -2048 throw.
- `FIND-NAME`  ( c-addr u -- 0 nt | c-addr u )  Finds the word from its string token. Performs -16 throw if zero-length string.
- `NUMBER`  ( c-addr u -- ? )  Converts string to a number, performs -13 throw (undefined word) if it can't.
- `NAME>W`  ( nt -- w )  w is a general purpose parameter in the header.
- `NAME>NAME`  ( nt -- c-addr u )  Name of the word.

```
VARIABLE NT

: INTERPRET ( -- )  // DXT interpreter loop
   BEGIN  PARSE-NAME  DUP WHILE
      FIND-NAME OVER IF NUMBER ELSE
         NIP  DUP NT !  STATE @ IF         
         NAME>COMP ELSE
         NAME>EXEC THEN  EXECUTE
      THEN
   REPEAT  2DROP
;
```
IMMEDIATE works by copying the last word's xtE to its xtC.


## Header structures

If there's anything computing has demonstrated, it's the persistence of data structures. Data is like rocks. Code is like sand. It's desirable to define a minimal amount of rock. For the benefit of embedded systems, bloat is optional.

The header structure would start with the counted name string. The MSB of the count byte would contain the *smudge* flag and bit 6 would contain the *immediate* flag. Empty bytes between the end of the string and the next CODE-ALIGNED address would be padded with 0. The rest of the header would contain:
{'fraction_non_alphanumeric': 0.070873786407767, 'fraction_numerical': 0.007766990291262136, 'mean_word_length': 3.30685618729097, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '598951', 'n_tokens_mistral': 1566, 'n_tokens_neox': 1442, 'n_words': 790}