<commit_before># Copyright 2017 <NAME>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#+TITLE: Hacking on SHCL
#+AUTHOR: <NAME>

So you want to contribute to SHCL?  Here's everything you need to know!

* File Organization
The root folder of the SHCL source tree contains a few loose files and
several sub-directories.  The root folder should not contain any SHCL
source files.  Let's explore the sub-directories!

** =/core/=
This directory contains the "core" of SHCL.  The core contains just
the common functionality that both the SHCL library and the SHCL shell
need.  Think of it as a toolbox that you can use to build your own
shell.

*** =/core/support/=
This folder contains the C functionality that cannot be written in
Common Lisp (even with the help of CFFI).  Currently, it contains two
things:
1. function wrappers around POSIX APIs which are defined as macros, and
2. the ~shcl_spawn~ function, which provides a bit more flexibility
   than ~posix_spawn~.

You might be thinking "sure, I see why you need #1, but can't you
implement #2 in Common Lisp?".  Well, yes.  Sort of.  Some Common Lisp
compilers (namely, SBCL) won't let you fork when there are multiple
threads.  SHCL uses threads.  We can go behind the compiler's back and
call ~fork~ directly, but that puts us into an unsupported
configuration.  Its easier and more portable to just write the ~fork~
~exec~ logic in straight C.  The lisp compiler should (hopefully)
never notice that we forked.  That's the theory, at least.

** =/shell/=
This directory contains things that are only relevant to SHCL when it
is used as a shell.  For example, shell builtins.

** =/library/=
This directory doesn't yet exist, but it will contain high-level
abstractions which allow Common Lisp programs to easily run shell
commands.

** =/test/=
This directory contains tests that ensure SHCL is working correctly.
The tests are run by the =test.lisp= script in the root directory.

** =/linters/=
This directory contains scripts which detect stylistic issues with
SHCL's source.

* The Pipeline
SHCL is, at its heart, a pipeline.  Generally, each stage is a
transformation from one iterator to another.

** Lexing
The lexer takes a character stream and produces an iterator for the
tokens found in the stream.

The lexer can be extended using lexer readtables (much like Common
Lisp supports adding reader macros).

** Baking
Some tokens can't be complete until "run time".  For example, consider
a token that expands to the value of a lisp variable.  Lexing might
occur at compile time, but the user would probably rather get the
value of the variable at run time.  The baking phase allows each token
to provide a lisp form that needs to be evaluated before the shell
command is run.  If possible, these forms should be compiled at
compile time and then run at run time.  If the baking phase runs at
run time, these forms will be passed to ~EVAL~.

Very little actually happens during the baking phase.  The bake forms
are collected into a queue that is expected to be drained later.  The
tokens that enter the bake phase are returned unmodified.  The bake
forms typically modify the tokens when they are evaluated.

** Parsing
The parsing phase consumes a token iterator and produces a syntax tree
iterator.  That is, the values of the resulting iterator are syntax
trees.

** Evaluating
This is the final phase of the pipeline.  In this phase, a syntax tree
is pulled from the input iterator.  Before calling ~EVALUATE~ on the
syntax tree, all of the bake forms in the bakery queue should be
evaluated.

*** Expansion
Just before the ~EVALUATE~ function runs a command or builtin, it
expands each "word" being passed to the command/builtin.
* Assorted Style Guidelines
- Every exported symbol should have documentation.  Documenting
  internal functions is also a Good Thing.  Document methods at your
  discretion.
- Only tests are allowed to access unexported symbols.
- Treat all exported symbols as public API.  No packages are private.
- Long lines should be avoided.
- Prefer immutable data structures (e.g. fset or define-data).<commit_msg>Small update to HACKING<commit_after># Copyright 2017 <NAME>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#+TITLE: Hacking on SHCL
#+AUTHOR: <NAME>

So you want to contribute to SHCL?  Here's everything you need to know!

* File Organization
The root folder of the SHCL source tree contains a few loose files and
several sub-directories.  The root folder should not contain any SHCL
source files.  Let's explore the sub-directories!

** =/core/=
This directory contains the "core" of SHCL.  The core contains just
the common functionality that both the SHCL library and the SHCL shell
need.  Think of it as a toolbox that you can use to build your own
shell.

*** =/core/support/=
This folder contains the C functionality that cannot be written in
Common Lisp (even with the help of CFFI).  Currently, it contains two
things:
1. function wrappers around POSIX APIs which are defined as macros, and
2. the ~shcl_spawn~ function, which provides a bit more flexibility
   than ~posix_spawn~.

You might be thinking "sure, I see why you need #1, but can't you
implement #2 in Common Lisp?".  Well, yes.  Sort of.  Some Common Lisp
compilers (namely, SBCL) won't let you fork when there are multiple
threads.  SHCL uses threads.  We can go behind the compiler's back and
call ~fork~ directly, but that puts us into an unsupported
configuration.  Its easier and more portable to just write the ~fork~
~exec~ logic in straight C.  The lisp compiler should (hopefully)
never notice that we forked.  That's the theory, at least.

** =/shell/=
This directory contains things that are only relevant to SHCL when it
is used as a shell.  For example, shell builtins.

** =/test/=
This directory contains tests that ensure SHCL is working correctly.
The tests are run by the =test.lisp= script in the root directory.

** =/linters/=
This directory contains scripts which detect stylistic issues with
SHCL's source.

* The Pipeline
SHCL is, at its heart, a pipeline.  Generally, each stage is a
transformation from one iterator to another.

** Lexing
The lexer takes a character stream and produces an iterator for the
tokens found in the stream.

The lexer can be extended using lexer readtables (much like Common
Lisp supports adding reader macros).

** Baking
Some tokens can't be complete until "run time".  For example, consider
a token that expands to the value of a lisp variable.  Lexing might
occur at compile time, but the user would probably rather get the
value of the variable at run time.  The baking phase allows each token
to provide a lisp form that needs to be evaluated before the shell
command is run.  If possible, these forms should be compiled at
compile time and then run at run time.  If the baking phase runs at
run time, these forms will be passed to ~EVAL~.

Very little actually happens during the baking phase.  The bake forms
are collected into a queue that is expected to be drained later.  The
tokens that enter the bake phase are returned unmodified.  The bake
forms typically modify the tokens when they are evaluated.

** Parsing
The parsing phase consumes a token iterator and produces a syntax tree
iterator.  That is, the values of the resulting iterator are syntax
trees.

** Evaluating
This is the final phase of the pipeline.  In this phase, a syntax tree
is pulled from the input iterator.  Before calling ~EVALUATE~ on the
syntax tree, all of the bake forms in the bakery queue should be
evaluated.

Note that this phase is undergoing a transformation and a new phase is
being inserted.  Before evaluation, the shell AST will be translated
into a DSL for expressing shell commands.  That DSL will then be
translated into Common Lisp and evaluated like any other lisp form.

*** Expansion
Just before the ~EVALUATE~ function runs a command or builtin, it
expands each "word" being passed to the command/builtin.

* Assorted Style Guidelines
- Every exported symbol should have documentation.  Documenting
  internal functions is also a Good Thing.  Document methods at your
  discretion.
- Only tests are allowed to access unexported symbols.
- Treat all exported symbols as public API.  No packages are private.
- Long lines should be avoided.
- Prefer immutable data structures (e.g. fset or define-data).
{'fraction_non_alphanumeric': 0.05093087952065055, 'fraction_numerical': 0.0025679435052428845, 'mean_word_length': 4.088187261839956, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1910931', 'n_tokens_mistral': 2511, 'n_tokens_neox': 2405, 'n_words': 1478}