<commit_before>Implementation
==============

Components of the project
-------------------------

![**Major components that make up Oboe.js** showing flow from http
transport to registered callbacks. Every component is not shown here.
Particularly, components whose responsibility it is to initialise the
oboe instance but have no role once it is running are omitted. UML
facet/receptacle notation is used to show the flow of events with event
names in capitals. \label{overallDesign}](images/overallDesign.png)

Oboe's architecture has been designed to so that I may have as much
confidence as possible regarding the correct working of the library
through automated testing. Designing a system to be amenable to testing
in this case meant splitting into many co-operating parts each with an
easily specified remit.

Although there is limited encapsulation to follow an OO-style
arrangement, data is hidden. Outside of Oboe, only a restricted public
API is exposed. Not only are the internals inaccessible, they are
unaddressable. With no references attached to any external data
structures, most data exists only as captured within closures.

Internally, communication between components is facilitated by an event
bus which is local to to Oboe instance, with most components interacting
solely by picking up events, processing them and publishing further
events in response. Essentially, Oboe's architecture resembles a fairly
linear pipeline visiting a series of units, starting with http data and
sometimes ending with callbacks being notified. This use of an event bus
is a variation on the Observer pattern which removes the need for each
unit to obtain a reference to the previous one so that it may observe
it, giving a highly decoupled shape to the library. Once everything is
wired into the bus very little central control is required and the
larger behaviours emerge as the consequence of this interaction between
finer ones. One downside is perhaps that a central event bus does not
lend itself to a UML class diagram, giving a diagram shape with an event
bus as a central hub and everything else hanging off it as spokes.

Automated testing
-----------------

Automated testing improves what can be written, not just making what is
written more reliable. Tests deal with the problem of "irreducible
complexity" - when a program is made out of parts whose correct
behaviour cannot be observed without all of the program. Allows smaller
units to be verified before verifying the whole.

![The testing pyramid is a common concept, relying on the assumption
that verification of small parts provides a solid base from which to
compose system-level behaviours. A Lot of testing is done on the
low-level components of the system, less on the component level and less
still on a whole-system level where only smoke tests are provided.
\label{testpyramid}](images/pyramid)

The testing itself is a non-trivial undertaking with 80% of code written
for this project being test specifications. Based on the idea that a
correct system must be built from individually correct units, the
majority of the specifications are unit tests, putting each unit under
the microscope and describing the correct behaviour as completely as
possible. Component tests zoom out from examining individual components
to focus on their correct composition, falsifying only the http traffic.
To avoid testing implementation details the component tests do not look
at the means of coupling between the code units but rather check for the
behaviours which should emerge as a consequence of their composition. At
the apex of the test pyramid are a small number of integration tests.
These tests check all of Oboe, automatically spinning up a REST service
so that the correctness of the whole library may be examined against an
actual server.

The desire to be amenable to testing influences the boundaries on which
the application splits into components. Black-box unit testing of a
stateful unit is difficult; because of side-effects it may later react
differently to the same calls. For this reason where state is required
it is stored in very simple state-storing units with intricate program
logic removed. The logic may then be separately expressed as functions
which map from one state to the next. Although comprehensive coverage is
of course impossible and tests are inevitably incomplete, for whatever
results the functions give while under test, uninfluenced by state I can
be sure that they will continue to give in any future situation. The
separate unit holding the state is trivial to test, having exactly one
responsibility: to store the result of a function call and later pass
that result to the next function. This approach clearly breaks with
object oriented style encapsulation by not hiding data behind the logic
which acts on them but I feel the departure is worthwhile for the
greater certainty it allows over the correct functioning of the program.

Largely for the sake of testing Oboe has also embraced dependency
injection. This means that components do not create the further
components that they require but rather rely on them being provided by
an external wiring. The file `wire.js` performs the actual injection.
One such example is the streamingHttp component which hides various
incompatible http implementations by publishing their downloaded content
progressively via the event bus. This unit does not know how to create
the underlying browser XHR which it hides. Undoubtedly, by not
instantiating its own dependencies a it presents a less friendly
interface, although this is mitigated somewhat by the interface being
purely internal, the objects it depends on are no longer a hidden
implementation detail but exposed as a part of the component's API. The
advantage of dependency injection here is that unit testing is much
simpler. Unit tests should test exactly one behaviour of one unit. Were
the streaming http object to create its own transport, that part would
also be under test, plus whichever external service that it connects to.
Because Javascript allows redefinition of built in types, this could be
avoided by overwriting the XHR constructor to return a mock but
modifying the built in types for tests opens up the possibilities of
changes leaking between cases. Dependency injection allows a much
simpler test style because it is trivial to inject a stub in place of
the XHR.

Integration tests run against a node service which returns known content
according to known timings, somewhat emulating downloading via a slow
internet connection. For example, the url `/tenSlowNumbers` writes out a
JSON array of the first ten natural numbers at a rate of one per second,
while `/echoBackHeaders` writes back the http headers that it received
as a JSON object. The test specifications which use these services
interact with Oboe through the public API alone as an application author
would and try some tricky cases. For example, requesting ten numbers but
registering a listener against the fifth and aborting the request on
seeing it. The correct behaviour is to get no callback for the sixth,
even when running on platforms where the http is buffered so that all
ten will have already been downloaded.

Running the tests
-----------------

![**Relationship between various files and test libraries** *other half
of sketch from notebook*](images/placeholder.png)

The Grunt task runner was used to automate routine tasks such as
executing the tests and building. Unit and component tests run
automatically whenever a source file changes. As well as being correct
execution, the project is required to not surpass a certain size so the
built size is also checked. As a small, tightly focused project the
majority of programming is refactoring already working code. Running
tests on save provides quick feedback so that mistakes are found as soon
as they are made. Agile practitioners emphasise the importance of tests
that execute quickly [@cleancode P314, T9], the 220 unit and component
tests run in less than a second so discovering mistakes is near instant.
If the "content of any medium is always another medium” [@media p8], we
might say that the content of programming is the program that is
realised by its execution. A person working in arts and crafts sees the
thing as they work but a programmer will usually not see the execution
simultaneously as they program. Conway observed that an artisan works by
transform-in-place "start with the working material in place and you
step by step transform it into its final form" whereas software is
created through intermediate proxies, and attempts to close this gap by
merging programming with the results of programming [@humanize side8-9].
When we bring together the medium and the message the cost of small
experimentation is very low and I feel that programming becomes more
explorative and expressive.

The integration tests are not run on save because they intentionally
simulate slow transfers and take some time to run. The integration tests
are used as a final check against built code before a branch in git can
be merged into the master. Once the code has been packaged for
distribution the internals are no longer visible the integration tests
which are coded against the public API are the only runnable tests.
While these tests don't individually test every component, they are
designed to exercise the whole codebase so that a mistake in any
component will be visible through them. Grunt executes the build,
including starting up the test REST services that give the integration
tests something to fetch.

Packaging as a single distributable file
----------------------------------------

![**Packaging of many javascript files into multiple single-file
packages.** The packages are individually targeted at different
execution contexts, either browsers or node *get from notebook, split
sketch diagram in half*](images/placeholder.png)

As an interpreted language, Javascript may of course be ran directly
without any prior compilation. While running the same code as I see in
the editor is convenient while programming, it is much less so for
distribution. Although the languages imposes no compulsory build phase,
in practice one is necessary. Dependency managers have not yet become
standard for client-side web development (although Bower is looking
good) so most files are manually downloaded. For a developer wishing to
include my library in their own project a single file is much more
convenient. Should they not have a build process of their own, a single
file is also much faster to transfer to their users, mostly because of
the cost of establishing connections and the http overhead.

Javascript files are interpreted in series by the browser so
dependencies must come before dependants, tricks for circular
dependencies notwithstanding. Unsurprisingly, files concatenated in the
same order as they would be delivered to the browser will be
functionally equivalent to the same files delivered separately. Several
tools are available to automate this stage of the build process,
including the topological sort of the dependency digraph that finds a
working concatenation order.

Early in this project I chose Require.js although it was later abandoned
because I found it to be too heavyweight. Javascript as a language
doesn't have an import statement so Require implements importing in
Javascript itself as a normally executable function. When running raw
source, this function AJAXes in the imported source but Require also has
an 'optimise mode' which uses static analysis to deduce a workable
source order and concatenates into a single file. This of course is
impossible in the general case and if, for example, the import is
subject to branching require falls back to lazy loading, fetching only
when needed. In practice undecidability isn't an issue because importing
is typically unconditional and may even be an advantage in larger
projects. To speed up initial loading of larger web applications,
*Asynchronous Module Definition* (AMD) imports rarely-loaded
functionality in response to events, resisting static analysis and so
downloading the code only as it is needed.

I hoped to use Require's `optimise` to generate my distributable
Javascript library. However, require's structure necessitates that calls
to the importing functions stay in the code and that the require.js
run-time component is available to facilitate these calls. For a small
project I found this constant-size overhead too large in relation to the
rest of the project. Require also prefers to be used as the sole means
of loading scripts meaning that Oboe would not fit naturally into web
applications not already using require. Overall, Require seems more
suited to developing whole applications than programming libraries.

Having abandoned require rather than look for another sophisticated
means to perform concatenation I decided to pick up the simplest tool
which could possibly work, a Grunt module which works like the unix
`cat` command. With only 15 source Javascript files manually finding a
working order by drawing a graph on paper isn't a daunting task. As new
files are added it is simple to find a place to insert them into the
list. I adjusted each Javascript file to, when loaded directly, place
its API in the global namespace, then post-concatenation wrapped the
combined in a single function, converting the APIs inside the function
from global to the scope of that function, thereby hiding the
implementation for code outside of Oboe.

For future consideration there is Browserify. This library reverses the
'browser first' image of Javascript by converting applications targeted
at Node into a single file efficiently packaged for delivery to a web
browser, conceptually making Node the primary environment for Javascript
and adapting browser execution to match. Significantly, require leaves
no trace of itself in the concatenated Javascript other than Adaptors
presenting browser APIs as the Node equivalents. Browserify's http
adaptor[^1] is complete but more verbose compared to Oboe's version[^2].

As well as combining into a single file, Javascript source can made
significantly smaller by removing comments and reducing inaccessible
tokens to a single character. For Oboe the popular library *Uglify* is
used for minification. Uglify performs only surface optimisations,
operating on the AST level but concentrating mostly on compact syntax. I
also considered Google's Closure compiler. Closure resembles a
traditional compiler optimiser by leveraging a deeper understanding to
search for smaller representations, unfortunately at the cost of safety.
Decidability in highly dynamic languages is often impossible and Closure
operates on a well-advised subset of Javascript, delivering no
reasonable guarantee of equivalence when code is not written as the
Closure authors expected. Integration tests should catch any such
failures but for the time being I have a limited appetite for a workflow
which forces me to be suspicious of the project's build process.

Styles of Programming
---------------------

Built own functional libs and list library (compose, fold etc)

Programming is finished when each line reads as a statement of fact
rather than the means of making the statement so.

How doing data hiding in JS without copying an OO concept of data
hiding.

Interestingly, the mixed paradigm design hasn't changed the top-level
design very much from how it'd be as a pure OO project (IoC, decorators,
event filters, pub/sub etc).

The code presented is the result of the development many prior versions,
it has never been rewritten in the sense of starting again. Nonetheless,
every part has been complely renewed several times. I am reviewing only
the final version. Git promotes regular commits, there have been more
than 1000.

some of it is pure functional (jsonPath, controller) ie, only
semantically different from a Haskell programme others, syntactically
functional but stateful to fit in with expected APIs etc

The style of implementation of the generator of functions corresponding
to json path expressions is reminiscent of a traditional parser
generator, although rather than generating source, functions are
dynamically composed. Reflecting on this, parser gens only went to
source to break out of the ability to compose the expressive power of
the language itself from inside the language itself. With a functional
approach, assembly from very small pieces gives a similar level of
expressivity as writing the logic out as source code.

Why could implement Function\#partial via prototype. Why not going to.
Is a shame. However, are using prototype for minimal set of polyfills.
Not general purpose.

Javascript: not the greatest for 'final' elegant presentation of
programming. Does allow 'messy' first drafts which can be refactored
into beautiful code. Ie, can write stateful and refactor in small steps
towards being stateless. An awareness of beautiful languages lets us
know the right direction to go in. An ugly language lets us find
something easy to write that works to get us started. Allows a very
sketchy program to be written, little more than a programming
scratchpad.

Without strict typing, hard to know if program is correct without
running it. In theory (decidability) and in practice (often find errors
through running and finding errors thrown). Echo FPR: once compiling,
good typing tends to give a reasonable sureness that the code is
correct.

### Performance implications of functional javascript

V8 and other modern JS engines are often said to be 'near-native' speed,
meaning it runs at close to the speed of a similarly coded C program.
However, this relies on the programmer also coding in the style of a C
programmer, for example with only mono-morphic callsites and without a
functional style. Once either of those programming techniques is taken
up performance drops rapidly
[http://rfrn.org/\~shu/2013/03/20/two-reasons-functional-style-is-slow-in-spidermonkey.html]
9571 ms vs 504 ms. When used in a functional style, not 'near-native' in
the sense that not close to the performance gained by compiling a well
designed functional language to natively executable code. Depends on
style coded in, comparison to native somewhat takes C as the description
of the operation of an idealised CPU rather than an abstract machine
capable of executing on an actual CPU.

(perhaps move to background, or hint at it, eg "although there are still
some performance implications involved in a functional style, javascript
may be used in a non-pure functional style") - with link to here

The performance degradation, even with a self-hosted forEach, is due to
the JIT’s inability to efficiently inline both the closures passed to
forEach

Lambda Lifting, currently not implemented in SpiderMonkey or V8:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.48.4346

The transformations to enable the above criteria are tedious and are
surely the purview of the compiler. All that’s needed are brave compiler
hackers

JS is much faster with "monomorphic call sites"

However, js execution time is not much of a problem,

### Preferring functions over constructors (subsume into above section?)

What constructors are in js. Any function, but usually an uppercase
initial char indicates that it is intended to be used as a constructor.

Inheritence is constructed using the language itself. While this is more
flexible and allows each project to define a bespoke version of
inherience to suit their particular needs or preferences, it also
hampers portability more than an 'extends' keyword would.

> So far, the JavaScript community has not agreed on a common
> inheritance library (which would help tooling and code portability)
> and it is doubtful that that will ever happen. That means, we’re stuck
> with constructors under ECMAScript 5.
> http://www.2ality.com/2013/07/defending-constructors.html

Functions can be like Factories, gives me the flexability to chagne how
something is created but by exposing a constructor are stuck with using
'new' to create an instance of exactly one type. 'new' is inconsistent
invocation with rest of language.

Dart has 'factory' constructors which are called like constructors but
act like factory functions:
(http://www.dartlang.org/docs/dart-up-and-running/contents/ch02.html\#ch02-constructor-factory)

Incrementally building up the content
-------------------------------------

As shown in figure \ref{overallDesign}, there is an incremental content
builder and ascent tracer which handle the output from the Clarinet JSON
SAX parser. Taken together, these might be considered as a variant of
the Adaptor pattern, providing to the controller a simpler interface
than is presented by Clarinet. However, this is not a model
implementation of the pattern; it is even-driven rather than
call-driven: we receive six kinds of event and in response emmit a
smaller vocabulary of two, creating an adaptor over pushes rather that
pulls.

To perform matching on JSONPath expressions, the controller requires the
path from the root of the document to the current node is required, this
is provided in the NODE\_FOUND and PATH\_FOUND events emitted from the
incremental content builder. For each Clarinet event this builder
provides a corresponding function which takes the current path and
returns the path after the event has been applied. For example, the
`objectopen` and `arrayopen` events add new items to the path, whereas
`closeobject` and `closearray` remove them. Over the course of the parse
of the whole JSON file the path in this way will visit every node,
allowing each to be tested against the registered JSONPath expressions.
Internally, the builder's event handlers are created from the
combination of a smaller number of more basic reusable handlers; Oboe is
largely indiscriminate regarding the type of nodes found in the JSON
whereas Clarinet's events often differ only in the type of node that was
encountered. This reuse of smaller instructions to build up larger ones
is slightly reminiscent of the use of composed micro-instructions in
CISC CPU design to implement the chip's higher-level instructions.
Consider the `value` event which is fired when Clarinet encounters a
String or Number, because primitive nodes are always leaves the internal
handling is as a node which instantaneously starts and ends, expressed
programmatically as the functional composition of the `nodeFound` and
`curNodeFinished` handlers.

Although the builder functions are stateless, ultimately the state
regarding the current path needs to be stored between clarinet calls.
This is handled by the ascent tracker. This tiny component merely serves
as a holder for this data, starting from an empty path it passes the
path to each builder function and stores the result to be given to the
next one.

![List representation of an ascent from leaf to root of a JSON tree.
Note the special ROOT token which represents the path mapping to the
root node (of course nothing maps to the root) - this is an object,
taking advantage of object identity to ensure that the token is unequal
to anything but itself. This list form is built up by the incremental
content builder and is the format that compiled JSONPath expressions
test against for matches \label{ascent}](images/ascent.png)

The path of the current node is maintained as a singly linked list, with
each list element holding the field name and the node and the node
itself, see figure \ref{ascent}. The list is arranged with the JSON root
at the far end and the current node at the head. As we traverse the JSON
the current node is appended and removed many times whereas the root is
immutable. This ordering was chosen because it is computationally very
efficient since all updates to the list are at the head. Each link in
the list is immutable, enforced by newer Javascript engines as frozen
objects [^3].

Linked lists were chosen in preference to the more conventional approach
of using native Javascript Arrays for several reasons. Firstly, I find
this area of the program more easy to test and debug given immutable
data structures. Treating the native Arrays as immutable would be very
expensive because on each change to the path the array contents would
have to be copied to a new array. As well as promoting mutability, most
Javascript virtual machines are quite poor at array growing and
shrinking so for collections whose size changes often are outperformed
by linked lists. Finally, this is a very convenient format for the
JSONPath engine to perform matching on as will be discussed in the next
section. The Javascript file lists.js implements the lists with some
appropriate recursive operations: `map`, `foldR`, `all`.

Because it is more common to quote paths as descents rather than ascent,
on the boundary to the outside world Oboe reverses the order and,
because Javascript programmers will not be familiar with this structure,
converts to arrays.

* * * * *

Stateless makes using a debugger easier - can look back in stack trace
and because of no reassignment, can see the whole, unchanged state of
the parent call. What the params are now are what they always have been,
no chance of reassignment (some code style guides recommend not to
reassign parameters but imperative languages generally do not forbid it)
No Side effects: can type expressions into debugger to see evaluation
without risk of changing program execution.

Oboe JSONPath Implementation
----------------------------

Not surprisingly given its importance, the JSONPath implementation is
one of the most refactored and considered parts of the Oboe codebase.
Like many small languages, on the first commit it was little more than a
series of regular expressions[^4] but has slowly evolved into a
featureful and efficient implementation[^5]. The extent of the rewriting
was possible because the correct behaviour is well defined by test
specifications[^6].

The JSONPath compiler exposes a single higher-order function to the rest
of Oboe. This function takes a JSONPath as a String and, proving it is a
valid expression, returns a function which tests for matches to the
JSONPath. Both the compiler and the functions that it generates benefit
from being stateless. The type of the compiler, expressed as Haskell
syntax would be:

~~~~ {.haskell}
String -> Ascent -> JsonPathMatchResult
~~~~

The match result is either a failure to match, or a hit, with the node
that matched. In the case of path matching, the node may currently be
unknown. If the pattern has a clause prefixed with `$`, the node
matching that clause is captured and returned as the result. Otherwise,
the last clause is implicitly capturing.

The usage profile for JSONPath expressions in Oboe is to be compiled
once and then evaluated many times, once for each node encountered while
parsing the JSON. Because matching is performed perhaps hundreds of
times per file the most pressing performance consideration is for
matching to execute quickly, the time required to compile is relatively
unimportant. Oboe's JSONPath design contrasts with JSONPath's reference
implementation which, because it provides a first order function,
freshly reinterprets the JSONPath string each time it is invoked.

The compilation is performed by recursively by examining the left-most
side of the string for a JSONPath clause. For each kind of clause there
is a function which matches ascents against that clause, for example by
checking the name field. By partial completion this function is
specialised to match against one particular name. Once a clause function
is generated, compilation recurs by passing to itself the remaining
unparsed portion of the JSONPath string. This continues until it is
called with a zero-length JSONPath. On each recursive call the clause
function is wrapped in the result from the next recursive call,
resulting ultimately in a linked series of clause functions. When
evaluated against an ascent, each clause functions examines the head of
the ascent and passes the ascent onto the next function if it passes. A
special clause functions, `skip1` is used for the `.` syntax and places
no condition on the head of the ascent but passes on to the next clause
only the tail, thus moving evaluation of the ascent one node up the
parsed JSON tree. Similarly, there is a `skipMany` which maps onto the
`..` syntax and recursively consumes nodes until it can find a match in
the next clause.

JsonPath implementation allows the compilation of complex expressions
into an executable form, but each part implementing the executable form
is locally simple. By using recursion, assembling the simple functions
into a more function expressing a more complex rule also follows as
being locally simple but gaining a usefully sophisticated behaviour
through composition of simple parts. Each recursive call of the parser
identifies one token for non-empty input and then recursively digests
the rest.

As an example, the pattern `!.$person..{height tShirtSize}` once
compiled to a Javascript functional representation would roughly
resemble this:

~~~~ {.javascript}
statementExpr(             // wrapper, added when JSONPath is zero-length 
   duckTypeClause(         // token 6, {height tShirtSize}
      skipMany(            // token 5, '..'  
         capture(          // token 4, css4-style '$' notation
            nameClause(    // token 3, 'person'
               skip1(      // token 2, '.'  
                  rootExpr // token 1, '!' at start of JSONPath expression
               ) 
            'person' )
         )
   ), ['height', 'tShirtSize'])
)      
~~~~

Since I am only using a side-effect free subset of Javascript for this
segment of Oboe it would be safe to use a functional cache. As well as
saving time by avoiding repeated execution, this could potentially also
save memory because where two JSONPath strings contain a common start
they could share the inner parts of their functional expression.
Although Javascript doesn't come with functional caching, it can be
added using the language itself [^7]. I suspect, however, that hashing
the parameters might be slower than performing the matching. Although
the parameters are all immutable and could in theory be hashed by object
identity, in practice there is no way to access an object id from inside
the language so any hash of a node parsed out of JSON would have to walk
the entire subtree rooted from that node.

The JSONPath tokenisation is split out into its own file and separately
tested. The tokenisation implementation is based on regular expressions,
they are the simplest form able to express the clause patterns. The
regular expressions are hidden to the outside the tokenizer and only
functions are exposed to the main body of the compiler. The regular
expressions all start with `^` so that they only match at the head of
the string. A more elegant alternative is the 'y' [^8] flag but as of
now this lacks wide browser support.

By verifying the tokens through their own unit tests it is simpler to
thoroughly specify the tokenisation, producing simpler failure messages
than if it were done through the full JSONPath engine. We might consider
the unit test layer of the pyramid (figure \ref{testpyramid}) is further
split into two sub-layers. Arguably, the upper of these sub-layer is not
a unit test because it is verifying two units together. There is some
redundancy with the tokens being tested both individually and as full
expressions. I maintain that this is the best approach regardless
because stubbing out the tokenizer functions would be a considerable
effort and would not improve the rigor of the JSONPath specification.

![Some kind of diagram showing jsonPath expressions and functions
partially completed to link back to the previous function. Include the
statementExpr pointing to the last clause](images/placeholder)

[^1]: https://github.com/substack/http-browserify

[^2]: https://github.com/jimhigson/oboe.js/blob/master/src/streamingHttp.js
    I can't claim superior programming ability, this version is shorter
    because it is not a generic solution

[^3]: See
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global\_Objects/Object/freeze.
    Although older engines don't provide any ability to create immutable
    objects at run-time, we can be fairly certain that the code does not
    mutate these objects or the tests would fail when run in
    environments which are able to enforce this.

[^4]: JSONPath compiler from the first commit can be found at line 159
    here:
    https://github.com/jimhigson/oboe.js/blob/a17db7accc3a371853a2a0fd755153b10994c91e/src/main/progressive.js\#L159

[^5]: for contrast, the current source can be found at
    https://github.com/jimhigson/oboe.js/blob/master/src/jsonPath.js

[^6]: The current tests are viewable at
    https://github.com/jimhigson/oboe.js/blob/master/test/specs/jsonPath.unit.spec.js
    and
    https://github.com/jimhigson/oboe.js/blob/master/test/specs/jsonPathTokens.unit.spec.js

[^7]: Probably the best known example being `memoize` from
    Underscore.js: http://underscorejs.org/\#memoize

[^8]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular\_Expressions
<commit_msg>fold last jungle into civilised block<commit_after>Implementation
==============

Components of the project
-------------------------

![**Major components that make up Oboe.js** showing flow from http
transport to registered callbacks. Every component is not shown here.
Particularly, components whose responsibility it is to initialise the
oboe instance but have no role once it is running are omitted. UML
facet/receptacle notation is used to show the flow of events with event
names in capitals. \label{overallDesign}](images/overallDesign.png)

Oboe's architecture has been designed to so that I may have as much
confidence as possible regarding the correct working of the library
through automated testing. Designing a system to be amenable to testing
in this case meant splitting into many co-operating parts each with an
easily specified remit.

Although there is limited encapsulation to follow an OO-style
arrangement, data is hidden. Outside of Oboe, only a restricted public
API is exposed. Not only are the internals inaccessible, they are
unaddressable. With no references attached to any external data
structures, most data exists only as captured within closures.

Internally, communication between components is facilitated by an event
bus which is local to to Oboe instance, with most components interacting
solely by picking up events, processing them and publishing further
events in response. Essentially, Oboe's architecture resembles a fairly
linear pipeline visiting a series of units, starting with http data and
sometimes ending with callbacks being notified. This use of an event bus
is a variation on the Observer pattern which removes the need for each
unit to obtain a reference to the previous one so that it may observe
it, giving a highly decoupled shape to the library. Once everything is
wired into the bus very little central control is required and the
larger behaviours emerge as the consequence of this interaction between
finer ones. One downside is perhaps that a central event bus does not
lend itself to a UML class diagram, giving a diagram shape with an event
bus as a central hub and everything else hanging off it as spokes.

Automated testing
-----------------

Automated testing improves what can be written, not just making what is
written more reliable. Tests deal with the problem of "irreducible
complexity" - when a program is made out of parts whose correct
behaviour cannot be observed without all of the program. Allows smaller
units to be verified before verifying the whole.

![The testing pyramid is a common concept, relying on the assumption
that verification of small parts provides a solid base from which to
compose system-level behaviours. A Lot of testing is done on the
low-level components of the system, less on the component level and less
still on a whole-system level where only smoke tests are provided.
\label{testpyramid}](images/pyramid)

The testing itself is a non-trivial undertaking with 80% of code written
for this project being test specifications. Based on the idea that a
correct system must be built from individually correct units, the
majority of the specifications are unit tests, putting each unit under
the microscope and describing the correct behaviour as completely as
possible. Component tests zoom out from examining individual components
to focus on their correct composition, falsifying only the http traffic.
To avoid testing implementation details the component tests do not look
at the means of coupling between the code units but rather check for the
behaviours which should emerge as a consequence of their composition. At
the apex of the test pyramid are a small number of integration tests.
These tests check all of Oboe, automatically spinning up a REST service
so that the correctness of the whole library may be examined against an
actual server.

The desire to be amenable to testing influences the boundaries on which
the application splits into components. Black-box unit testing of a
stateful unit is difficult; because of side-effects it may later react
differently to the same calls. For this reason where state is required
it is stored in very simple state-storing units with intricate program
logic removed. The logic may then be separately expressed as functions
which map from one state to the next. Although comprehensive coverage is
of course impossible and tests are inevitably incomplete, for whatever
results the functions give while under test, uninfluenced by state I can
be sure that they will continue to give in any future situation. The
separate unit holding the state is trivial to test, having exactly one
responsibility: to store the result of a function call and later pass
that result to the next function. This approach clearly breaks with
object oriented style encapsulation by not hiding data behind the logic
which acts on them but I feel the departure is worthwhile for the
greater certainty it allows over the correct functioning of the program.

Largely for the sake of testing Oboe has also embraced dependency
injection. This means that components do not create the further
components that they require but rather rely on them being provided by
an external wiring. The file `wire.js` performs the actual injection.
One such example is the streamingHttp component which hides various
incompatible http implementations by publishing their downloaded content
progressively via the event bus. This unit does not know how to create
the underlying browser XHR which it hides. Undoubtedly, by not
instantiating its own dependencies a it presents a less friendly
interface, although this is mitigated somewhat by the interface being
purely internal, the objects it depends on are no longer a hidden
implementation detail but exposed as a part of the component's API. The
advantage of dependency injection here is that unit testing is much
simpler. Unit tests should test exactly one behaviour of one unit. Were
the streaming http object to create its own transport, that part would
also be under test, plus whichever external service that it connects to.
Because Javascript allows redefinition of built in types, this could be
avoided by overwriting the XHR constructor to return a mock but
modifying the built in types for tests opens up the possibilities of
changes leaking between cases. Dependency injection allows a much
simpler test style because it is trivial to inject a stub in place of
the XHR.

Integration tests run against a node service which returns known content
according to known timings, somewhat emulating downloading via a slow
internet connection. For example, the url `/tenSlowNumbers` writes out a
JSON array of the first ten natural numbers at a rate of one per second,
while `/echoBackHeaders` writes back the http headers that it received
as a JSON object. The test specifications which use these services
interact with Oboe through the public API alone as an application author
would and try some tricky cases. For example, requesting ten numbers but
registering a listener against the fifth and aborting the request on
seeing it. The correct behaviour is to get no callback for the sixth,
even when running on platforms where the http is buffered so that all
ten will have already been downloaded.

Running the tests
-----------------

![**Relationship between various files and test libraries** *other half
of sketch from notebook*](images/placeholder.png)

The Grunt task runner was used to automate routine tasks such as
executing the tests and building. Unit and component tests run
automatically whenever a source file changes. As well as being correct
execution, the project is required to not surpass a certain size so the
built size is also checked. As a small, tightly focused project the
majority of programming is refactoring already working code. Running
tests on save provides quick feedback so that mistakes are found as soon
as they are made. Agile practitioners emphasise the importance of tests
that execute quickly [@cleancode P314, T9], the 220 unit and component
tests run in less than a second so discovering mistakes is near instant.
If the "content of any medium is always another medium” [@media p8], we
might say that the content of programming is the program that is
realised by its execution. A person working in arts and crafts sees the
thing as they work but a programmer will usually not see the execution
simultaneously as they program. Conway observed that an artisan works by
transform-in-place "start with the working material in place and you
step by step transform it into its final form" whereas software is
created through intermediate proxies, and attempts to close this gap by
merging programming with the results of programming [@humanize side8-9].
When we bring together the medium and the message the cost of small
experimentation is very low and I feel that programming becomes more
explorative and expressive.

The integration tests are not run on save because they intentionally
simulate slow transfers and take some time to run. The integration tests
are used as a final check against built code before a branch in git can
be merged into the master. Once the code has been packaged for
distribution the internals are no longer visible the integration tests
which are coded against the public API are the only runnable tests.
While these tests don't individually test every component, they are
designed to exercise the whole codebase so that a mistake in any
component will be visible through them. Grunt executes the build,
including starting up the test REST services that give the integration
tests something to fetch.

Packaging as a single distributable file
----------------------------------------

![**Packaging of many javascript files into multiple single-file
packages.** The packages are individually targeted at different
execution contexts, either browsers or node *get from notebook, split
sketch diagram in half*](images/placeholder.png)

As an interpreted language, Javascript may of course be ran directly
without any prior compilation. While running the same code as I see in
the editor is convenient while programming, it is much less so for
distribution. Although the languages imposes no compulsory build phase,
in practice one is necessary. Dependency managers have not yet become
standard for client-side web development (although Bower is looking
good) so most files are manually downloaded. For a developer wishing to
include my library in their own project a single file is much more
convenient. Should they not have a build process of their own, a single
file is also much faster to transfer to their users, mostly because of
the cost of establishing connections and the http overhead.

Javascript files are interpreted in series by the browser so
dependencies must come before dependants, tricks for circular
dependencies notwithstanding. Unsurprisingly, files concatenated in the
same order as they would be delivered to the browser will be
functionally equivalent to the same files delivered separately. Several
tools are available to automate this stage of the build process,
including the topological sort of the dependency digraph that finds a
working concatenation order.

Early in this project I chose Require.js although it was later abandoned
because I found it to be too heavyweight. Javascript as a language
doesn't have an import statement so Require implements importing in
Javascript itself as a normally executable function. When running raw
source, this function AJAXes in the imported source but Require also has
an 'optimise mode' which uses static analysis to deduce a workable
source order and concatenates into a single file. This of course is
impossible in the general case and if, for example, the import is
subject to branching require falls back to lazy loading, fetching only
when needed. In practice undecidability isn't an issue because importing
is typically unconditional and may even be an advantage in larger
projects. To speed up initial loading of larger web applications,
*Asynchronous Module Definition* (AMD) imports rarely-loaded
functionality in response to events, resisting static analysis and so
downloading the code only as it is needed.

I hoped to use Require's `optimise` to generate my distributable
Javascript library. However, require's structure necessitates that calls
to the importing functions stay in the code and that the require.js
run-time component is available to facilitate these calls. For a small
project I found this constant-size overhead too large in relation to the
rest of the project. Require also prefers to be used as the sole means
of loading scripts meaning that Oboe would not fit naturally into web
applications not already using require. Overall, Require seems more
suited to developing whole applications than programming libraries.

Having abandoned require rather than look for another sophisticated
means to perform concatenation I decided to pick up the simplest tool
which could possibly work, a Grunt module which works like the unix
`cat` command. With only 15 source Javascript files manually finding a
working order by drawing a graph on paper isn't a daunting task. As new
files are added it is simple to find a place to insert them into the
list. I adjusted each Javascript file to, when loaded directly, place
its API in the global namespace, then post-concatenation wrapped the
combined in a single function, converting the APIs inside the function
from global to the scope of that function, thereby hiding the
implementation for code outside of Oboe.

For future consideration there is Browserify. This library reverses the
'browser first' image of Javascript by converting applications targeted
at Node into a single file efficiently packaged for delivery to a web
browser, conceptually making Node the primary environment for Javascript
and adapting browser execution to match. Significantly, require leaves
no trace of itself in the concatenated Javascript other than Adaptors
presenting browser APIs as the Node equivalents. Browserify's http
adaptor[^1] is complete but more verbose compared to Oboe's version[^2].

As well as combining into a single file, Javascript source can made
significantly smaller by removing comments and reducing inaccessible
tokens to a single character. For Oboe the popular library *Uglify* is
used for minification. Uglify performs only surface optimisations,
operating on the AST level but concentrating mostly on compact syntax. I
also considered Google's Closure compiler. Closure resembles a
traditional compiler optimiser by leveraging a deeper understanding to
search for smaller representations, unfortunately at the cost of safety.
Decidability in highly dynamic languages is often impossible and Closure
operates on a well-advised subset of Javascript, delivering no
reasonable guarantee of equivalence when code is not written as the
Closure authors expected. Integration tests should catch any such
failures but for the time being I have a limited appetite for a workflow
which forces me to be suspicious of the project's build process.

Styles of Programming
---------------------

Built own functional libs and list library (compose, fold etc)

Programming is finished when each line reads as a statement of fact
rather than the means of making the statement so.

How doing data hiding in JS without copying an OO concept of data
hiding.

Interestingly, the mixed paradigm design hasn't changed the top-level
design very much from how it'd be as a pure OO project (IoC, decorators,
event filters, pub/sub etc).

The code presented is the result of the development many prior versions,
it has never been rewritten in the sense of starting again. Nonetheless,
every part has been complely renewed several times. I am reviewing only
the final version. Git promotes regular commits, there have been more
than 1000.

some of it is pure functional (jsonPath, controller) ie, only
semantically different from a Haskell programme others, syntactically
functional but stateful to fit in with expected APIs etc

The style of implementation of the generator of functions corresponding
to json path expressions is reminiscent of a traditional parser
generator, although rather than generating source, functions are
dynamically composed. Reflecting on this, parser gens only went to
source to break out of the ability to compose the expressive power of
the language itself from inside the language itself. With a functional
approach, assembly from very small pieces gives a similar level of
expressivity as writing the logic out as source code.

Why could implement Function\#partial via prototype. Why not going to.
Is a shame. However, are using prototype for minimal set of polyfills.
Not general purpose.

Javascript: not the greatest for 'final' elegant presentation of
programming. Does allow 'messy' first drafts which can be refactored
into beautiful code. Ie, can write stateful and refactor in small steps
towards being stateless. An awareness of beautiful languages lets us
know the right direction to go in. An ugly language lets us find
something easy to write that works to get us started. Allows a very
sketchy program to be written, little more than a programming
scratchpad.

Without strict typing, hard to know if program is correct without
running it. In theory (decidability) and in practice (often find errors
through running and finding errors thrown). Echo FPR: once compiling,
good typing tends to give a reasonable sureness that the code is
correct.

### Performance implications of functional javascript

V8 and other modern JS engines are often said to be 'near-native' speed,
meaning it runs at close to the speed of a similarly coded C program.
However, this relies on the programmer also coding in the style of a C
programmer, for example with only mono-morphic callsites and without a
functional style. Once either of those programming techniques is taken
up performance drops rapidly
[http://rfrn.org/\~shu/2013/03/20/two-reasons-functional-style-is-slow-in-spidermonkey.html]
9571 ms vs 504 ms. When used in a functional style, not 'near-native' in
the sense that not close to the performance gained by compiling a well
designed functional language to natively executable code. Depends on
style coded in, comparison to native somewhat takes C as the description
of the operation of an idealised CPU rather than an abstract machine
capable of executing on an actual CPU.

(perhaps move to background, or hint at it, eg "although there are still
some performance implications involved in a functional style, javascript
may be used in a non-pure functional style") - with link to here

The performance degradation, even with a self-hosted forEach, is due to
the JIT’s inability to efficiently inline both the closures passed to
forEach

Lambda Lifting, currently not implemented in SpiderMonkey or V8:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.48.4346

The transformations to enable the above criteria are tedious and are
surely the purview of the compiler. All that’s needed are brave compiler
hackers

JS is much faster with "monomorphic call sites"

However, js execution time is not much of a problem,

### Preferring functions over constructors (subsume into above section?)

What constructors are in js. Any function, but usually an uppercase
initial char indicates that it is intended to be used as a constructor.

Inheritence is constructed using the language itself. While this is more
flexible and allows each project to define a bespoke version of
inherience to suit their particular needs or preferences, it also
hampers portability more than an 'extends' keyword would.

> So far, the JavaScript community has not agreed on a common
> inheritance library (which would help tooling and code portability)
> and it is doubtful that that will ever happen. That means, we’re stuck
> with constructors under ECMAScript 5.
> http://www.2ality.com/2013/07/defending-constructors.html

Functions can be like Factories, gives me the flexability to chagne how
something is created but by exposing a constructor are stuck with using
'new' to create an instance of exactly one type. 'new' is inconsistent
invocation with rest of language.

Dart has 'factory' constructors which are called like constructors but
act like factory functions:
(http://www.dartlang.org/docs/dart-up-and-running/contents/ch02.html\#ch02-constructor-factory)

Incrementally building up the content
-------------------------------------

As shown in figure \ref{overallDesign}, there is an incremental content
builder and ascent tracer which handle the output from the Clarinet JSON
SAX parser. Taken together, these might be considered as a variant of
the Adaptor pattern, providing to the controller a simpler interface
than is presented by Clarinet. However, this is not a model
implementation of the pattern; it is even-driven rather than
call-driven: we receive six kinds of event and in response emmit a
smaller vocabulary of two, creating an adaptor over pushes rather that
pulls.

To perform matching on JSONPath expressions, the controller requires the
path from the root of the document to the current node is required, this
is provided in the NODE\_FOUND and PATH\_FOUND events emitted from the
incremental content builder. For each Clarinet event this builder
provides a corresponding function which takes the current path and
returns the path after the event has been applied. For example, the
`objectopen` and `arrayopen` events add new items to the path, whereas
`closeobject` and `closearray` remove them. Over the course of the parse
of the whole JSON file the path in this way will visit every node,
allowing each to be tested against the registered JSONPath expressions.
Internally, the builder's event handlers are created from the
combination of a smaller number of more basic reusable handlers; Oboe is
largely indiscriminate regarding the type of nodes found in the JSON
whereas Clarinet's events often differ only in the type of node that was
encountered. This reuse of smaller instructions to build up larger ones
is slightly reminiscent of the use of composed micro-instructions in
CISC CPU design to implement the chip's higher-level instructions.
Consider the `value` event which is fired when Clarinet encounters a
String or Number, because primitive nodes are always leaves the internal
handling is as a node which instantaneously starts and ends, expressed
programmatically as the functional composition of the `nodeFound` and
`curNodeFinished` handlers.

Although the builder functions are stateless, ultimately the state
regarding the current path needs to be stored between clarinet calls.
This is handled by the ascent tracker. This tiny component merely serves
as a holder for this data, starting from an empty path it passes the
path to each builder function and stores the result to be given to the
next one.

![List representation of an ascent from leaf to root of a JSON tree.
Note the special ROOT token which represents the path mapping to the
root node (of course nothing maps to the root) - this is an object,
taking advantage of object identity to ensure that the token is unequal
to anything but itself. This list form is built up by the incremental
content builder and is the format that compiled JSONPath expressions
test against for matches \label{ascent}](images/ascent.png)

The path of the current node is maintained as a singly linked list, with
each list element holding the field name and the node and the node
itself, see figure \ref{ascent}. The list is arranged with the JSON root
at the far end and the current node at the head. As we traverse the JSON
the current node is appended and removed many times whereas the root is
immutable. This ordering was chosen because it is computationally very
efficient since all updates to the list are at the head. Each link in
the list is immutable, enforced by newer Javascript engines as frozen
objects [^3].

Linked lists were chosen in preference to the more conventional approach
of using native Javascript Arrays for several reasons. Firstly, I find
this area of the program more easy to test and debug given immutable
data structures. Handling native Arrays without mutating would be very
expensive because on each new path the array would have to be copied
rather than edited in-place. Unpicking a stack trace is easier if I know
that every value revealed is the value that has always occupied that
space because I don't have to think four-dimensionally projecting my
mind forwards and back in time to different values that were there when
the variable was used. The lack of side effects means I can try explore
new commands in the debugger's CLI without worrying about breaking the
execution of the program. As well as promoting mutability, most
Javascript virtual machines are quite poor at array growing and
shrinking so for collections whose size changes often are outperformed
by linked lists. Finally, this is a very convenient format for the
JSONPath engine to perform matching on as will be discussed in the next
section. The Javascript file lists.js implements the list functions:
`cons`, `head`, `tail`, `map`, `foldR`, `all`.

Because it is more common to quote paths as descents rather than ascent,
on the boundary to the outside world Oboe reverses the order and,
because Javascript programmers will not be familiar with this structure,
converts to arrays.

Oboe JSONPath Implementation
----------------------------

Not surprisingly given its importance, the JSONPath implementation is
one of the most refactored and considered parts of the Oboe codebase.
Like many small languages, on the first commit it was little more than a
series of regular expressions[^4] but has slowly evolved into a
featureful and efficient implementation[^5]. The extent of the rewriting
was possible because the correct behaviour is well defined by test
specifications[^6].

The JSONPath compiler exposes a single higher-order function to the rest
of Oboe. This function takes a JSONPath as a String and, proving it is a
valid expression, returns a function which tests for matches to the
JSONPath. Both the compiler and the functions that it generates benefit
from being stateless. The type of the compiler, expressed as Haskell
syntax would be:

~~~~ {.haskell}
String -> Ascent -> JsonPathMatchResult
~~~~

The match result is either a failure to match, or a hit, with the node
that matched. In the case of path matching, the node may currently be
unknown. If the pattern has a clause prefixed with `$`, the node
matching that clause is captured and returned as the result. Otherwise,
the last clause is implicitly capturing.

The usage profile for JSONPath expressions in Oboe is to be compiled
once and then evaluated many times, once for each node encountered while
parsing the JSON. Because matching is performed perhaps hundreds of
times per file the most pressing performance consideration is for
matching to execute quickly, the time required to compile is relatively
unimportant. Oboe's JSONPath design contrasts with JSONPath's reference
implementation which, because it provides a first order function,
freshly reinterprets the JSONPath string each time it is invoked.

The compilation is performed by recursively by examining the left-most
side of the string for a JSONPath clause. For each kind of clause there
is a function which matches ascents against that clause, for example by
checking the name field. By partial completion this function is
specialised to match against one particular name. Once a clause function
is generated, compilation recurs by passing to itself the remaining
unparsed portion of the JSONPath string. This continues until it is
called with a zero-length JSONPath. On each recursive call the clause
function is wrapped in the result from the next recursive call,
resulting ultimately in a linked series of clause functions. When
evaluated against an ascent, each clause functions examines the head of
the ascent and passes the ascent onto the next function if it passes. A
special clause functions, `skip1` is used for the `.` syntax and places
no condition on the head of the ascent but passes on to the next clause
only the tail, thus moving evaluation of the ascent one node up the
parsed JSON tree. Similarly, there is a `skipMany` which maps onto the
`..` syntax and recursively consumes nodes until it can find a match in
the next clause.

JsonPath implementation allows the compilation of complex expressions
into an executable form, but each part implementing the executable form
is locally simple. By using recursion, assembling the simple functions
into a more function expressing a more complex rule also follows as
being locally simple but gaining a usefully sophisticated behaviour
through composition of simple parts. Each recursive call of the parser
identifies one token for non-empty input and then recursively digests
the rest.

As an example, the pattern `!.$person..{height tShirtSize}` once
compiled to a Javascript functional representation would roughly
resemble this:

~~~~ {.javascript}
statementExpr(             // wrapper, added when JSONPath is zero-length 
   duckTypeClause(         // token 6, {height tShirtSize}
      skipMany(            // token 5, '..'  
         capture(          // token 4, css4-style '$' notation
            nameClause(    // token 3, 'person'
               skip1(      // token 2, '.'  
                  rootExpr // token 1, '!' at start of JSONPath expression
               ) 
            'person' )
         )
   ), ['height', 'tShirtSize'])
)      
~~~~

Since I am only using a side-effect free subset of Javascript for this
segment of Oboe it would be safe to use a functional cache. As well as
saving time by avoiding repeated execution, this could potentially also
save memory because where two JSONPath strings contain a common start
they could share the inner parts of their functional expression.
Although Javascript doesn't come with functional caching, it can be
added using the language itself [^7]. I suspect, however, that hashing
the parameters might be slower than performing the matching. Although
the parameters are all immutable and could in theory be hashed by object
identity, in practice there is no way to access an object id from inside
the language so any hash of a node parsed out of JSON would have to walk
the entire subtree rooted from that node.

The JSONPath tokenisation is split out into its own file and separately
tested. The tokenisation implementation is based on regular expressions,
they are the simplest form able to express the clause patterns. The
regular expressions are hidden to the outside the tokenizer and only
functions are exposed to the main body of the compiler. The regular
expressions all start with `^` so that they only match at the head of
the string. A more elegant alternative is the 'y' [^8] flag but as of
now this lacks wide browser support.

By verifying the tokens through their own unit tests it is simpler to
thoroughly specify the tokenisation, producing simpler failure messages
than if it were done through the full JSONPath engine. We might consider
the unit test layer of the pyramid (figure \ref{testpyramid}) is further
split into two sub-layers. Arguably, the upper of these sub-layer is not
a unit test because it is verifying two units together. There is some
redundancy with the tokens being tested both individually and as full
expressions. I maintain that this is the best approach regardless
because stubbing out the tokenizer functions would be a considerable
effort and would not improve the rigor of the JSONPath specification.

![Some kind of diagram showing jsonPath expressions and functions
partially completed to link back to the previous function. Include the
statementExpr pointing to the last clause](images/placeholder)

[^1]: https://github.com/substack/http-browserify

[^2]: https://github.com/jimhigson/oboe.js/blob/master/src/streamingHttp.js
    I can't claim superior programming ability, this version is shorter
    because it is not a generic solution

[^3]: See
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global\_Objects/Object/freeze.
    Although older engines don't provide any ability to create immutable
    objects at run-time, we can be fairly certain that the code does not
    mutate these objects or the tests would fail when run in
    environments which are able to enforce this.

[^4]: JSONPath compiler from the first commit can be found at line 159
    here:
    https://github.com/jimhigson/oboe.js/blob/a17db7accc3a371853a2a0fd755153b10994c91e/src/main/progressive.js\#L159

[^5]: for contrast, the current source can be found at
    https://github.com/jimhigson/oboe.js/blob/master/src/jsonPath.js

[^6]: The current tests are viewable at
    https://github.com/jimhigson/oboe.js/blob/master/test/specs/jsonPath.unit.spec.js
    and
    https://github.com/jimhigson/oboe.js/blob/master/test/specs/jsonPathTokens.unit.spec.js

[^7]: Probably the best known example being `memoize` from
    Underscore.js: http://underscorejs.org/\#memoize

[^8]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular\_Expressions

{'fraction_non_alphanumeric': 0.0367232911278331, 'fraction_numerical': 0.0033944637197915258, 'mean_word_length': 4.5111331843390445, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 17, 'https://': 16, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6624762', 'n_tokens_mistral': 15809, 'n_tokens_neox': 14816, 'n_words': 10097}