<commit_before># VM communication
The communication protocol is to be shared between "dumb terminal" and "stand-alone" modes. Basically, the thin client uses an escape 
sequence to intercept strings that would normally go to the TIB. It also returns results in an escape string that the terminal can treat 
appropriately. Escape characters should be in the 80-BF range so as to allow UTF-8 symbols. 
The escape char used is `ESC _` (0x9f, APC – Application Program Command) which marks the beginning of a thin-client command. 
`ESC \` (0x9c, ST – String Terminator) marks the end. 

The thin client avoids the use of bytes in the 80-9F range. In this case, byte b is re-mapped to 80 n, where n is (b & 0F) + 40. 
This allows the ISR to only care about looking for APC and ST when deciding where to direct the stream, and treat 80 as a special case. 
The host end gets the same simplicity.

If interrupt-driven communication is used, the stream could be managed by a UART (or I2C or whatever) ISR. 
The ISR can buffer incoming bytes in a FIFO, for example, and also look for a reset token. 
The hard reset token (80 00) could still be recognizable even if the system is hung.

Thin-client commands are evaluated by the DEBUG task. This task spends most of its time waiting for the next character. 
Spinning (invoking PAUSE if nothing is received) decreases efficiency slightly compared to having an ISR wake the DEBUG task. 
The PAUSE loop runs fast enought that you could get away with no interrupts and no UART buffering. DEBUG digests the UART data one byte 
at a time without impacting the timing of other real-time activity.

## Commands

DEBUG takes a command from the serial stream and returns a response based on the command. 
Some responses may be sent asynchronously depending on other conditions. For example, hard reset. Responses include:

- 0x07 = Hard reset
- 0x06 = ACK, command was successful
- 0x15 = NACK, command was not successful

### 01 = EXECUTE
Takes stack input, executes the xt, and returns the contents of the stack.

Input: {1 byte n, n*4 bytes stackInfo}. stackInfo is pushed onto the stack in order of reception.

Output: {1 byte 0, 1 byte ior, 1 byte n, n*4 bytes stackInfo}. The 0 byte marks the beginning of a response. 
ior is the return value of CATCH, which is used to execute the xt. stackInfo is popped from the stack in order.
### 02 = RAM_READ
<commit_msg>Update VMcommunication.md<commit_after># VM communication
The communication protocol is to be shared between "dumb terminal" and "stand-alone" modes. Basically, the thin client uses an escape 
sequence to intercept strings that would normally go to the TIB. It also returns results in an escape string that the terminal can treat 
appropriately. Escape characters should be in the 80-BF range so as to allow UTF-8 symbols. 
The escape char used is `ESC _` (0x9f, APC – Application Program Command) which marks the beginning of a thin-client command. 
`ESC \` (0x9c, ST – String Terminator) marks the end. 

The thin client avoids the use of bytes in the 80-9F range. In this case, byte b is re-mapped to 80 n, where n is (b & 0F) + 40. 
This allows the ISR to only care about looking for APC and ST when deciding where to direct the stream, and treat 80 as a special case. 
The host end gets the same simplicity.

If interrupt-driven communication is used, the stream could be managed by a UART (or I2C or whatever) ISR. 
The ISR can buffer incoming bytes in a FIFO, for example, and also look for a reset token. 
The hard reset token (80 00) could still be recognizable even if the system is hung.

Thin-client commands are evaluated by the DEBUG task. This task spends most of its time waiting for the next character. 
Spinning (invoking PAUSE if nothing is received) decreases efficiency slightly compared to having an ISR wake the DEBUG task. 
The PAUSE loop runs fast enought that you could get away with no interrupts and no UART buffering. DEBUG digests the UART data one byte 
at a time without impacting the timing of other real-time activity.

## Commands

DEBUG takes a command from the serial stream and returns a response based on the command. 
Some responses may be sent asynchronously depending on other conditions. For example, hard reset. Responses include:

- 0x07 = Hard reset
- 0x06 = ACK, command was successful
- 0x15 = NACK, command was not successful

### 01 = EXECUTE
Takes stack input, executes the xt, and returns the contents of the stack.

Input: {1 byte n, n\*4 bytes stackInfo}. stackInfo is pushed onto the stack in order of reception. 
The incoming stack is ( ... xt base ). The host expects ( ... ior base ) in the output stack. ior is the return value of CATCH, which is used to execute the xt.

Output: {1 byte ACK, 1 byte n, n\*4 bytes stackInfo}. The ACK marks the beginning of a response. stackInfo is popped from the stack in order.
### 02 = RAM_READ

{'fraction_non_alphanumeric': 0.04788213627992634, 'fraction_numerical': 0.016983834663392674, 'mean_word_length': 3.8977955911823647, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3927870', 'n_tokens_mistral': 1376, 'n_tokens_neox': 1287, 'n_words': 822}