<commit_before># Installing a zprint filter

Would you be interested in a capability where you could
type a few keystrokes, wait about 1.5 seconds, and have 
a Clojure(script) function pretty-printed from scratch 
(not just indented) right in your
editor?

If so, read on -- or jump right in:

* [What is a Unix filter](#what-is-a-unix-filter)
* [What would a zprint filter do](#what-would-a-zprint-filter-do)
* [Why would I want a zprint filter](#why-would-i-want-a-zprint-filter)
* [We all know Clojure startup is so slow that this will not work](#we-all-know-that-clojure-startup-is-so-slow-that-this-will-not-work)
  * [Performance Results](#performance-results)
   * [Benchmarks](#benchmarks)
* [Okay, how do I set this up for myself](#okay-how-do-i-set-this-up-for-myself)
  * [bootclasspath version](#bootclasspath-version)
  * [appcds version](#appcds-version)
* [Now, how do I use it](#now-how-do-i-use-it)
  * [vim](#vim)
  * [emacs](#emacs)
  * [other editors](#other-editors)
* [Configuring the filter](#configuring-the-filter)
  * [Different files for different styles](#different-files-for-different-styles)


## What is a Unix filter

In the Unix world (which includes Linux and MacOS for the purposes
of this discussion), a __filter__ is a program that reads from
standard input, performs some operation on the data, and writes to
standard output.  Examples are `grep`, `sort`, `cat`, `uniq`, `tail`
and `fmt`.  All of these process and transform or select text from
standard in (stdin) and put the results of these transformations
on standard out (stdout).

`fmt` is an interesting example, because it doesn't actually change
the content of the text passed through it, but rather it changes
the __layout__ or __formatting__ of the text -- by word wrapping
the result.  Back in the day, `fmt` was used to word-wrap email and
even today people use it make text fit conveniently into some given
number of columns.  The source text for this paragraph, for instance,
was piped through `fmt` to make it fit nicely inside of an 80 column
editor window.

## What would a zprint filter do

In the same mold as `fmt`, which reformats plain ASCII text, a `zprint`
filter would accept Clojure(script) source on stdin, and produce pretty-printed
Clojure(script) source on stdout.

## Why would I want a zprint filter

Most editors or IDE's have some way to take a section of the source
and pipe it though a filter and replace the source with whatever the
filter returns.  Thus, I can take one of the paragraphs of this text
file and pipe it through `fmt` with the vi clone editor that I use.
Certainly vim and emacs both can take text (which might be Clojure(script)
source) and pipe it through a filter and then replace the source with
whatever is returned.

So, if you had a zprint filter, by typing a few keystrokes, you
could have any individual function pretty-printed by zprint while
you were sitting in an editor or IDE looking at the function.

## We all know Clojure startup is so slow that this is never going to work

Yes, I thought that Clojure startup would be __way__ too slow, so I spent
some significant time exploring Clojurescript specifically so that I could
build a zprint formatting filter using Clojurescript.  And I did, and it
does work in Clojurescript -- you can see the results of my efforts in
`lein-zprint` [here](https://github.com/kkinnear/lein-zprint#a-zprint-formatting-filter-using-planck-or-lumo)

However, I have found that, if I AOT compile all of zprint and
the associated libraries (and use just minimal libraries), and use the
right Java command line arguments (and caching techniques), that I can
get a Clojure zprint filter to startup slightly __faster__ than a Clojurescript
zprint filter.  And to do so in just under 1 second.  Which isn't zero, but
which is fast enough to be perfectly useful.

Moreover, as the amount of source to be formatted grows, the Clojure
version runs considerably faster than the Clojurescript version.

### Performance Results

I have built two Clojurescript zprint filters, one using `planck`, which
uses JavascriptCore on MacOS, and one using `lumo`, which uses node.js.
There are also three different invocations of the Clojure zprint filter,
which I will explain later.  

__The bottom line__: If you download an uberjar from github, you can
have a filter that will pretty-print your Clojure(script) source
code in about 1.5 seconds for a reasonable sized function.
There are two ways to set it up to be fast, with the "appcds"
approach being about 10% faster than the "bootclasspath" approach.
Both are quite simple, and there are complete instructions below
on how to do it.

#### Benchmarks

If you want the details, here they are.

The machine is a MacBook Air (13-inch, Mid 2012), 2 GHz Intel Core
i7, 8 GB 16000 MHz DDR3.  SSD disk, but mostly it runs out of the
file cache.  

All numbers are an average of at least 7 runs, using
whatever kind of cache is available (both Clojurescript filters
cache the compiled Javascript, and the zprint-filter appcds has a
cache as well).

The table below has a lot of data in it.  There are three basic
experiments:

  * "Startup Secs" -- how long does it take to format the
  string "hello world", which measures startup and actually
  doing something -- but not much.

  * "Format defn" -- how long does it take to format the
  1.8 source for the function `defn`.  This is 49 lines
  long, and it a reasonable example for a function you might
  want to format.

  * "Format 3K loc" -- how long does it take to format
  a source file in zprint itself, which has 3K lines in it.
  It contains 113 lines of comments.  It has a number of
  of normal sized functions and a couple of very large ones.
  This is the entire source file, something you could certainly
  use the filter for, but not the expected use case. 

  I've included this edge case because it shows how Java really
  does run faster than Javascript, but it can take some work to
  get it to start up quickly.

Here are the numbers:

| Filter | Startup Secs | % of basic Startup | Format defn (49 loc) | % of basic format defn | Format 3K loc | % of basic format 3K loc|
|--------|------|-------------------------------|----------|----|----|----|
| planck (JavascriptCore) | 3.100 | 158% | 3.530 | 146% | 13.854 | 194% |
| zprint-filter basic | 1.955 | 100% | 2.415 | 100% | 7.107 | 100% |
| zprint-filter bootclasspath | 1.179 | 60% | 1.611 | 67% | 5.914 | 83% |
| lumo (node.js)| 1.003 | 51% | 2.188 | 91% | 20.442 | 288%  |
| zprint-filter appcds | .996 | 51% | 1.407 | 58%  | 5.900 | 83%  |

Your machine may very well be faster than mine, so you would see
even better numbers.  

## Okay, how do I set this up for myself

The two lines you should be interested in are the zprint-filter bootclasspath
and zprint-filter appcds lines.  As you might imagine, the bootclasspath
is easier to set up, and the appcds runs faster for small functions.

I have provided an uberjar in the releases area, which you need to
download.  It doesn't have a .jar extension because I find that various
programs get all excited about .jar files.   Why upset them needlessly?

First, you need to find a directory which is on your path (or, if you don't<commit_msg>Fix link to latest release.<commit_after># Installing a zprint filter

Would you be interested in a capability where you could
type a few keystrokes, wait about 1.5 seconds, and have 
a Clojure(script) function pretty-printed from scratch 
(not just indented) right in your
editor?

If so, read on -- or jump right in:

* [What is a Unix filter](#what-is-a-unix-filter)
* [What would a zprint filter do](#what-would-a-zprint-filter-do)
* [Why would I want a zprint filter](#why-would-i-want-a-zprint-filter)
* [We all know Clojure startup is so slow that this will not work](#we-all-know-that-clojure-startup-is-so-slow-that-this-will-not-work)
  * [Performance Results](#performance-results)
   * [Benchmarks](#benchmarks)
* [Okay, how do I set this up for myself](#okay-how-do-i-set-this-up-for-myself)
  * [bootclasspath version](#bootclasspath-version)
  * [appcds version](#appcds-version)
* [Now, how do I use it](#now-how-do-i-use-it)
  * [vim](#vim)
  * [emacs](#emacs)
  * [other editors](#other-editors)
* [Configuring the filter](#configuring-the-filter)
  * [Different files for different styles](#different-files-for-different-styles)


## What is a Unix filter

In the Unix world (which includes Linux and MacOS for the purposes
of this discussion), a __filter__ is a program that reads from
standard input, performs some operation on the data, and writes to
standard output.  Examples are `grep`, `sort`, `cat`, `uniq`, `tail`
and `fmt`.  All of these process and transform or select text from
standard in (stdin) and put the results of these transformations
on standard out (stdout).

`fmt` is an interesting example, because it doesn't actually change
the content of the text passed through it, but rather it changes
the __layout__ or __formatting__ of the text -- by word wrapping
the result.  Back in the day, `fmt` was used to word-wrap email and
even today people use it make text fit conveniently into some given
number of columns.  The source text for this paragraph, for instance,
was piped through `fmt` to make it fit nicely inside of an 80 column
editor window.

## What would a zprint filter do

In the same mold as `fmt`, which reformats plain ASCII text, a `zprint`
filter would accept Clojure(script) source on stdin, and produce pretty-printed
Clojure(script) source on stdout.

## Why would I want a zprint filter

Most editors or IDE's have some way to take a section of the source
and pipe it though a filter and replace the source with whatever the
filter returns.  Thus, I can take one of the paragraphs of this text
file and pipe it through `fmt` with the vi clone editor that I use.
Certainly vim and emacs both can take text (which might be Clojure(script)
source) and pipe it through a filter and then replace the source with
whatever is returned.

So, if you had a zprint filter, by typing a few keystrokes, you
could have any individual function pretty-printed by zprint while
you were sitting in an editor or IDE looking at the function.

## We all know Clojure startup is so slow that this is never going to work

Yes, I thought that Clojure startup would be __way__ too slow, so I spent
some significant time exploring Clojurescript specifically so that I could
build a zprint formatting filter using Clojurescript.  And I did, and it
does work in Clojurescript -- you can see the results of my efforts in
`lein-zprint` [here](https://github.com/kkinnear/lein-zprint#a-zprint-formatting-filter-using-planck-or-lumo)

However, I have found that, if I AOT compile all of zprint and
the associated libraries (and use just minimal libraries), and use the
right Java command line arguments (and caching techniques), that I can
get a Clojure zprint filter to startup slightly __faster__ than a Clojurescript
zprint filter.  And to do so in just under 1 second.  Which isn't zero, but
which is fast enough to be perfectly useful.

Moreover, as the amount of source to be formatted grows, the Clojure
version runs considerably faster than the Clojurescript version.

### Performance Results

I have built two Clojurescript zprint filters, one using `planck`, which
uses JavascriptCore on MacOS, and one using `lumo`, which uses node.js.
There are also three different invocations of the Clojure zprint filter,
which I will explain later.  

__The bottom line__: If you download an uberjar from github, you can
have a filter that will pretty-print your Clojure(script) source
code in about 1.5 seconds for a reasonable sized function.
There are two ways to set it up to be fast, with the "appcds"
approach being about 10% faster than the "bootclasspath" approach.
Both are quite simple, and there are complete instructions below
on how to do it.

#### Benchmarks

If you want the details, here they are.

The machine is a MacBook Air (13-inch, Mid 2012), 2 GHz Intel Core
i7, 8 GB 16000 MHz DDR3.  SSD disk, but mostly it runs out of the
file cache.  

All numbers are an average of at least 7 runs, using
whatever kind of cache is available (both Clojurescript filters
cache the compiled Javascript, and the zprint-filter appcds has a
cache as well).

The table below has a lot of data in it.  There are three basic
experiments:

  * "Startup Secs" -- how long does it take to format the
  string "hello world", which measures startup and actually
  doing something -- but not much.

  * "Format defn" -- how long does it take to format the
  1.8 source for the function `defn`.  This is 49 lines
  long, and it a reasonable example for a function you might
  want to format.

  * "Format 3K loc" -- how long does it take to format
  a source file in zprint itself, which has 3K lines in it.
  It contains 113 lines of comments.  It has a number of
  of normal sized functions and a couple of very large ones.
  This is the entire source file, something you could certainly
  use the filter for, but not the expected use case. 

  I've included this edge case because it shows how Java really
  does run faster than Javascript, but it can take some work to
  get it to start up quickly.

Here are the numbers:

| Filter | Startup Secs | % of basic Startup | Format defn (49 loc) | % of basic format defn | Format 3K loc | % of basic format 3K loc|
|--------|------|-------------------------------|----------|----|----|----|
| planck (JavascriptCore) | 3.100 | 158% | 3.530 | 146% | 13.854 | 194% |
| zprint-filter basic | 1.955 | 100% | 2.415 | 100% | 7.107 | 100% |
| zprint-filter bootclasspath | 1.179 | 60% | 1.611 | 67% | 5.914 | 83% |
| lumo (node.js)| 1.003 | 51% | 2.188 | 91% | 20.442 | 288%  |
| zprint-filter appcds | .996 | 51% | 1.407 | 58%  | 5.900 | 83%  |

Your machine may very well be faster than mine, so you would see
even better numbers.  

## Okay, how do I set this up for myself

The two lines you should be interested in are the zprint-filter bootclasspath
and zprint-filter appcds lines.  As you might imagine, the bootclasspath
is easier to set up, and the appcds runs faster for small functions.

I have provided an uberjar in the
[releases](https://github.com/kkinnear/zprint/releases/latest)
area, which you need to
download.  It doesn't have a .jar extension because I find that various
programs get all excited about .jar files.   Why upset them needlessly?

First, you need to find a directory which is on your path (or, if you don't
{'fraction_non_alphanumeric': 0.07888583218707015, 'fraction_numerical': 0.018707015130674003, 'mean_word_length': 3.7722349852313752, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6045522', 'n_tokens_mistral': 4553, 'n_tokens_neox': 4119, 'n_words': 2213}