<commit_before>Vaquero
=======

Vaquero is a scripting language for cowboy coders.

It's a Lisp variant with an unusual object/message system, delimited lexical environments, delimited continuations, restartable exceptions, and syntactic abstraction.

## Basic usage

The Vaquero bootstrap interpreter is written in [Chicken Scheme](http://www.call-cc.org/).  
You'll need this to compile and run it.

```bash
cd Vaquero
sh ./bin/get_eggs.sh   # may need to run this part as root
sh ./bin/compile.sh
sh ./bin/run_tests.sh
```

Compilation should take a minute or two.  run_tests.sh will execute a few hundred tests written in Vaquero. This should give the user a feel for whether the compilation worked.

```bash
./vaquero 

Usage:

vaquero repl
vaquero eval "<code string>"
vaquero run <filename>
vaquero check <filename>
vaquero expand <filename>
vaquero compile <filename>
vaquero clean
```

The Vaquero executable responds to the seven messages above.

* **repl** starts a Read-Eval-Print Loop.
* **eval** parses a string into code and evaluates it.
* **run** takes a filename and expands, links, compiles, caches, and executes it.
* **check** tests tests the syntax of a file.
* **expand** pretty-prints the named file with all operators expanded
* **compile** expands, links, compiles, and caches a file without executing it.
* **clean** clears the Vaquero cache in ~/.vaquero.


## Hello, world!

```bash
# you don't have to use rlwrap to run the REPL,
# but I can't recommend it enough

home> rlwrap ./vaquero repl
```

```scheme
vaquero> (say "Hello, I'm <NAME>.")
Hello, I'm <NAME>.
null
vaquero>
```

## Data types

[null](https://github.com/TurtleKitty/Vaquero/wiki/null) lacks value.  It's not equal to anything but itself.  It answers almost every message with **null**.

[true](https://github.com/TurtleKitty/Vaquero/wiki/bool) and [false](https://github.com/TurtleKitty/Vaquero/wiki/bool) are the boolean values.  null, 0, "", and empty lists, vectors, tables, and objects evaluate to false in a boolean context.  All other values evaluate to true.

Vaquero, like most programming languages, has [numbers](https://github.com/TurtleKitty/Vaquero/wiki/number).

A [symbol](https://github.com/TurtleKitty/Vaquero/wiki/symbol) names a value.  They make the best table keys and object messages.

A string of characters is called a [text](https://github.com/TurtleKitty/Vaquero/wiki/text).  They have built-in messages for dealing with regexen.  They are normally delimited by quotes ("Hi guys!\n"), but there are also reader literals for unescaped quoting #(text ...) and string interpolation #(template ...).

```scheme

"Hello, world!"

; text literals are useful for constructing regexen without needing to escape backslashes.
#(text (\w+)-(\d+))
   ->  "(\\w+)-(\\d+)"

; template literals allow interleaving text and expressions
(def my-variable 7)
#(template foo bar {{ my-variable }})
   -> "foo bar 7"

```

A [box](https://github.com/TurtleKitty/Vaquero/wiki/box) is the simplest container type.  It holds a single value and answers messages **val** and **set!**.

```scheme

(def phantom (box 5))
   -> #(box 5)

phantom.val
   -> 5

(phantom.set! 7)
   -> 7

phantom.val
   -> 7

```

The humble [pair](https://github.com/TurtleKitty/Vaquero/wiki/pair) is the simplest compound data structure.  Pairs form into lists and trees, and Vaquero code is made from these.

```scheme

(pair 'x 1)
   -> (x . 1)

(pair 1 (pair 2 (pair ())))
   -> (1 2)

(list 1 2)
   -> (1 2)

```

A [vector](https://github.com/TurtleKitty/Vaquero/wiki/vector) is a heterogeneous array.

```scheme
(def v (vector 2 3 5))
   -> #(vector 2 3 5)

; vectors answer the apply message
(v 1)
   -> 3

(v.set! 2 42)
   -> #(vector 2 3 42)

```

A [table](https://github.com/TurtleKitty/Vaquero/wiki/table) is an unordered set of pairs (hashtables, under the hood).

```scheme

(table 'x 1 'y (+ 1 1))
   -> #(table x 1 y 2)

; the : operator auto-quotes table keys while evaluating values
(: x 1 y (+ 1 1))
   -> #(table x 1 y 2)

```

Vaquero [lambdas](https://github.com/TurtleKitty/Vaquero/wiki/lambda) are simple functions with one expression.

```scheme
(def foo (lambda (x) (+ x 10))

(foo 3)
   -> 13
```

Vaquero [procedures](https://github.com/TurtleKitty/Vaquero/wiki/procedure) have a sequence of expressions, are variadic, can accept arbitrary optional arguments.

```scheme

(proc foo (x)
   (when (not x)
      (return 'ZERO))
   (say x)
   (list x opt.y opt.z rest))

(foo 1) ; prints 1
   -> (1 null null ())

(foo 2 3 5) ; prints 2
   -> (2 null null (3 5))

(foo 7 y: 11 13 17) ; prints 7
   -> (7 11 null (13 17))

(foo 19 y: 23 z: 29 31 37) ; prints 19
   -> (19 23 29 (31 37))

(foo 0)
   -> 'ZERO

```

Vaquero [operators](https://github.com/TurtleKitty/Vaquero/wiki/operator) are procedures that run at compile time.  They alter source code to create new syntax.

Vaquero [environments](https://github.com/TurtleKitty/Vaquero/wiki/environment) are first-class.  They can be accessed with the **env** operator.

```scheme

(let (x 2 y 3) env)
   -> #(env y 3 x 2)

```

A [stream](https://github.com/TurtleKitty/Vaquero/wiki/stream) is a sequence of characters, possibly from outside the program.
Input streams have a number of useful parsing primitives.
Output streams can write, print, or say things to the outside world.

Vaquero supports both [tcp](https://github.com/TurtleKitty/Vaquero/wiki/tcp-socket) and [unix](https://github.com/TurtleKitty/Vaquero/wiki/fs-socket) sockets.

Programmers can create their own data types with the [object](https://github.com/TurtleKitty/Vaquero/wiki/object) procedure.
There are no classes or prototypes - an object is just a box with slots.
Any procedures it contains will close over the environment of their creation.
The **auto:** option allows the creation of thunks that auto-execute on message reception.
The **resend:** option allows an object to easily delegate messages to other objects.
The **default:** option allows an object to answer arbitrary messages it does not understand.

```scheme

(def simple
   (object 'foo 2 'bar 3))

   -> #(object bar foo)

(def counter (box 0))
   -> #(box 0)

(proc counter-val ()
   counter.val)

(proc counter-incr ()
   (counter.set! counter.val.inc))

(def complex
   (object
      'baz 5
      'incr counter-incr
      'val  counter-val
      resend: (list (list simple 'foo 'bar))
      auto: '(incr val)
      default:
         (lambda (msg)
            (say 'do-not-grok))))

   -> #(object incr baz val bar foo)

simple.foo
   -> 2

simple.bar
   -> 3

simple.baz
   -> ERROR 'message-not-understood  ; the default default: is to toss an error on unknown messages

complex.foo
   -> 2

complex.bar
   -> 3

complex.baz
   -> 5

complex.val
   -> 0

complex.incr
   -> 1

complex.val
   -> 1

complex.foonballardy  ; prints do-not-grok
   -> null

```


## Binding 

Environments are append-only.
**def** inserts a new name and value.
Those who want mutable state will have to use mutable structures such as boxes, vectors, and tables.

```scheme

(def foo 1)
(say foo)
   -> 1

```

## Sequencing

Execute the given expressions in sequence, returning the value of the last one.
Procs and lets have an implied seq, so this form is mostly useful in conditionals.

```scheme
(seq
    (say "Here it comes!")
    (say 42)
    true)
```

## Conditionals

```scheme

(if true 1 2)
   -> 1

(if false 1 2)
   -> 2

(when (= x 0)
   (say "Zero!"))

(cond
   (= x 0) 'zero
   (= x 1) 'one
   default:
      (seq
         (say "Fell through!")
         'infinity))

(case x
   (0) 'zero
   (1 2 3 5 7) 'small-prime-number
   (4 6 8 9) 'small-composite-number
   default: 'way-too-big)

```

## Procedures

```scheme
(def square
   (proc (x)
       (* x x)))

(proc square-2 (x)  ; syntactic sugar for (def square-2 (proc (x) ...))
   (* x x))

(def square-3    ; the _ operator is useful to create quick procedures of one argument
   (_ (* _ _)))
```

## Recursion and loops

Vaquero has tail-call optimization, so a program written in tail-recursive form can recurse forever without blowing the stack or devouring all the RAM.
[loop](https://github.com/TurtleKitty/Vaquero/wiki/loop) is useful for anonymous recursion.
The [while](https://github.com/TurtleKitty/Vaquero/wiki/while) form has **next** and **last** operators.
[for](https://github.com/TurtleKitty/Vaquero/wiki/for) has **next**, **last**, and **redo**.

```scheme
(loop recur (x items.head xs items.tail acc ())
   (if items.tail
      (send (pair x acc) 'reverse)
      (recur xs.head xs.tail (pair x acc))))

(let (i 0 total 0)
   (while (< i 20)
      (set! total (+ total i))
      (set! i i.inc)
      (list i total)))

   -> (20 190)

(for (i 0 total 0) (<= i 20) (set! i i.inc)
   (set! total (+ total i))
   total)

   -> 210

(loop go (counter 1000000) (if (= counter 0) 'done (go counter.dec)))

   ; time passes...

   -> done

```

## Quoting

```scheme
(def foo 17)
(def bar (list 1 2 3))

foo
   -> 17

(quote foo)
   -> foo

'foo
   -> foo

bar
   -> (1 2 3)

'bar
   -> bar
<commit_msg>spacing<commit_after>Vaquero
=======

Vaquero is a scripting language for cowboy coders.

It's a Lisp variant with an unusual object/message system, delimited lexical environments, delimited continuations, restartable exceptions, and syntactic abstraction.

## Basic usage

The Vaquero bootstrap interpreter is written in [Chicken Scheme](http://www.call-cc.org/).  
You'll need this to compile and run it.

```bash
cd Vaquero
sh ./bin/get_eggs.sh   # may need to run this part as root
sh ./bin/compile.sh
sh ./bin/run_tests.sh
```

Compilation should take a minute or two.  run_tests.sh will execute a few hundred tests written in Vaquero. This should give the user a feel for whether the compilation worked.

```bash
./vaquero 

Usage:

vaquero repl
vaquero eval "<code string>"
vaquero run <filename>
vaquero check <filename>
vaquero expand <filename>
vaquero compile <filename>
vaquero clean
```

The Vaquero executable responds to the seven messages above.

* **repl** starts a Read-Eval-Print Loop.
* **eval** parses a string into code and evaluates it.
* **run** takes a filename and expands, links, compiles, caches, and executes it.
* **check** tests tests the syntax of a file.
* **expand** pretty-prints the named file with all operators expanded
* **compile** expands, links, compiles, and caches a file without executing it.
* **clean** clears the Vaquero cache in ~/.vaquero.


## Hello, world!

```bash
# you don't have to use rlwrap to run the REPL,
# but I can't recommend it enough

home> rlwrap ./vaquero repl
```

```scheme
vaquero> (say "Hello, I'm <NAME>.")
Hello, I'm <NAME>.
null
vaquero>
```

## Data types

[null](https://github.com/TurtleKitty/Vaquero/wiki/null) lacks value.  It's not equal to anything but itself.  It answers almost every message with **null**.

[true](https://github.com/TurtleKitty/Vaquero/wiki/bool) and [false](https://github.com/TurtleKitty/Vaquero/wiki/bool) are the boolean values.  null, 0, "", and empty lists, vectors, tables, and objects evaluate to false in a boolean context.  All other values evaluate to true.

Vaquero, like most programming languages, has [numbers](https://github.com/TurtleKitty/Vaquero/wiki/number).

A [symbol](https://github.com/TurtleKitty/Vaquero/wiki/symbol) names a value.  They make the best table keys and object messages.

A string of characters is called a [text](https://github.com/TurtleKitty/Vaquero/wiki/text).  They have built-in messages for dealing with regexen.  They are normally delimited by quotes ("Hi guys!\n"), but there are also reader literals for unescaped quoting #(text ...) and string interpolation #(template ...).

```scheme

"Hello, world!"

; text literals are useful for constructing regexen without needing to escape backslashes.
#(text (\w+)-(\d+))
   ->  "(\\w+)-(\\d+)"

; template literals allow interleaving text and expressions
(def my-variable 7)
#(template foo bar {{ my-variable }})
   -> "foo bar 7"

```

A [box](https://github.com/TurtleKitty/Vaquero/wiki/box) is the simplest container type.  It holds a single value and answers messages **val** and **set!**.

```scheme

(def phantom (box 5))
   -> #(box 5)

phantom.val
   -> 5

(phantom.set! 7)
   -> 7

phantom.val
   -> 7

```

The humble [pair](https://github.com/TurtleKitty/Vaquero/wiki/pair) is the simplest compound data structure.  Pairs form into lists and trees, and Vaquero code is made from these.

```scheme

(pair 'x 1)
   -> (x . 1)

(pair 1 (pair 2 (pair ())))
   -> (1 2)

(list 1 2)
   -> (1 2)

```

A [vector](https://github.com/TurtleKitty/Vaquero/wiki/vector) is a heterogeneous array.

```scheme
(def v (vector 2 3 5))
   -> #(vector 2 3 5)

; vectors answer the apply message
(v 1)
   -> 3

(v.set! 2 42)
   -> #(vector 2 3 42)

```

A [table](https://github.com/TurtleKitty/Vaquero/wiki/table) is an unordered set of pairs (hashtables, under the hood).

```scheme

(table 'x 1 'y (+ 1 1))
   -> #(table x 1 y 2)

; the : operator auto-quotes table keys while evaluating values
(: x 1 y (+ 1 1))
   -> #(table x 1 y 2)

```

Vaquero [lambdas](https://github.com/TurtleKitty/Vaquero/wiki/lambda) are simple functions with one expression.

```scheme
(def foo (lambda (x) (+ x 10))

(foo 3)
   -> 13
```

Vaquero [procedures](https://github.com/TurtleKitty/Vaquero/wiki/procedure) have a sequence of expressions, are variadic, can accept arbitrary optional arguments.

```scheme

(proc foo (x)
   (when (not x)
      (return 'ZERO))
   (say x)
   (list x opt.y opt.z rest))

(foo 1) ; prints 1
   -> (1 null null ())

(foo 2 3 5) ; prints 2
   -> (2 null null (3 5))

(foo 7 y: 11 13 17) ; prints 7
   -> (7 11 null (13 17))

(foo 19 y: 23 z: 29 31 37) ; prints 19
   -> (19 23 29 (31 37))

(foo 0)
   -> 'ZERO

```

Vaquero [operators](https://github.com/TurtleKitty/Vaquero/wiki/operator) are procedures that run at compile time.  They alter source code to create new syntax.

Vaquero [environments](https://github.com/TurtleKitty/Vaquero/wiki/environment) are first-class.  They can be accessed with the **env** operator.

```scheme

(let (x 2 y 3) env)
   -> #(env y 3 x 2)

```

A [stream](https://github.com/TurtleKitty/Vaquero/wiki/stream) is a sequence of characters, possibly from outside the program.
Input streams have a number of useful parsing primitives.
Output streams can write, print, or say things to the outside world.

Vaquero supports both [tcp](https://github.com/TurtleKitty/Vaquero/wiki/tcp-socket) and [unix](https://github.com/TurtleKitty/Vaquero/wiki/fs-socket) sockets.

Programmers can create their own data types with the [object](https://github.com/TurtleKitty/Vaquero/wiki/object) procedure.
There are no classes or prototypes - an object is just a box with slots.
Any procedures it contains will close over the environment of their creation.
The **auto:** option allows the creation of thunks that auto-execute on message reception.
The **resend:** option allows an object to easily delegate messages to other objects.
The **default:** option allows an object to answer arbitrary messages it does not understand.

```scheme

(def simple
   (object 'foo 2 'bar 3))

   -> #(object bar foo)

(def counter (box 0))
   -> #(box 0)

(proc counter-val ()
   counter.val)

(proc counter-incr ()
   (counter.set! counter.val.inc))

(def complex
   (object
      'baz 5
      'incr counter-incr
      'val  counter-val
      resend: (list (list simple 'foo 'bar))
      auto: '(incr val)
      default:
         (lambda (msg)
            (say 'do-not-grok))))

   -> #(object incr baz val bar foo)

simple.foo
   -> 2

simple.bar
   -> 3

simple.baz
   -> ERROR 'message-not-understood  ; the default default: is to toss an error on unknown messages

complex.foo
   -> 2

complex.bar
   -> 3

complex.baz
   -> 5

complex.val
   -> 0

complex.incr
   -> 1

complex.val
   -> 1

complex.foonballardy  ; prints do-not-grok
   -> null

```


## Binding 

Environments are append-only.
**def** inserts a new name and value.
Those who want mutable state will have to use mutable structures such as boxes, vectors, and tables.

```scheme

(def foo 1)
(say foo)
   -> 1

```

## Sequencing

Execute the given expressions in sequence, returning the value of the last one.
Procs and lets have an implied seq, so this form is mostly useful in conditionals.

```scheme
(seq
    (say "Here it comes!")
    (say 42)
    true)
```

## Conditionals

```scheme

(if true 1 2)
   -> 1

(if false 1 2)
   -> 2

(when (= x 0)
   (say "Zero!"))

(cond
   (= x 0) 'zero
   (= x 1) 'one
   default:
      (seq
         (say "Fell through!")
         'infinity))

(case x
   (0) 'zero
   (1 2 3 5 7) 'small-prime-number
   (4 6 8 9) 'small-composite-number
   default: 'way-too-big)

```

## Procedures

```scheme
(def square
   (proc (x)
       (* x x)))

(proc square-2 (x)  ; syntactic sugar for (def square-2 (proc (x) ...))
   (* x x))

(def square-3    ; the _ operator is useful to create quick procedures of one argument
   (_ (* _ _)))
```

## Recursion and loops

Vaquero has tail-call optimization, so a program written in tail-recursive form can recurse forever without blowing the stack or devouring all the RAM.
[loop](https://github.com/TurtleKitty/Vaquero/wiki/loop) is useful for anonymous recursion.
The [while](https://github.com/TurtleKitty/Vaquero/wiki/while) form has **next** and **last** operators.
[for](https://github.com/TurtleKitty/Vaquero/wiki/for) has **next**, **last**, and **redo**.

```scheme
(loop recur (x items.head xs items.tail acc ())
   (if items.tail
      (send (pair x acc) 'reverse)
      (recur xs.head xs.tail (pair x acc))))

(let (i 0 total 0)
   (while (< i 20)
      (set! total (+ total i))
      (set! i i.inc)
      (list i total)))

   -> (20 190)

(for (i 0 total 0) (<= i 20) (set! i i.inc)
   (set! total (+ total i))
   total)

   -> 210

(loop go (counter 1000000)
   (if (= counter 0)
      'done
      (go counter.dec)))

   ; time passes...

   -> done

```

## Quoting

```scheme
(def foo 17)
(def bar (list 1 2 3))

foo
   -> 17

(quote foo)
   -> foo

'foo
   -> foo

bar
   -> (1 2 3)

'bar
   -> bar

{'fraction_non_alphanumeric': 0.1359651541048685, 'fraction_numerical': 0.019187296686331807, 'mean_word_length': 3.2457865168539324, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 113, 'https://': 42, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '582579', 'n_tokens_mistral': 6620, 'n_tokens_neox': 6028, 'n_words': 2286}