<commit_before># smallex

Smallex is a small application/library used for emitting small, hopefully
somewhat readable lexers. The lexers emitted are intended to be designed for
customisability and readability.

You often want to have control of your lexer, but let's face it: You very seldom
want to create the control flow graph by hand. Whenever I create
lexers/scanners, they end up just having a "backtrack" function which puts
values on top of the stack. This actually turned out to be a performance issue
for me, and although it seldom is, I don't want to end up with the same problem
twice.

To state the somewhat obvious here: This is a *lexer* library. If you wanted to
do easy *parsing*, then I would recommend the amazing
[instaparse](https://github.com/Engelberg/instaparse) library. If you want a
general lexer library for Clojure/JVM, I'd recommend
[ANTLR](http://www.antlr.org/). Hopefully I can someday recommend this lexer
generator instead.

## (Subsections go here)

TODO:
* What does the library do?
* Why did you write it (or didn't use an existing solution)?
* Who should be using it, and for what reasons?
* How should it be used (how to build it, how to configure it, how to run it and get started)?
* Where can additional information be found? (typically doc/tutorial.md)
From [Don't be a jerk: Write documentation](http://ferd.ca/don-t-be-a-jerk-write-documentation.html)
by <NAME>.

## License

Copyright © 2014 by [hyPiRion](https://github.com/hyPiRion) and
[contributors](https://github.com/hyPiRion/smallex).
<commit_msg>Smallex commands.<commit_after># smallex

Smallex is a small application/library used for emitting small, hopefully
somewhat readable lexers. The lexers emitted are intended to be designed for
customisability and readability.

You often want to have control of your lexer, but let's face it: You very seldom
want to create the control flow graph by hand. Whenever I create
lexers/scanners, they end up just having a "backtrack" function which puts
values on top of the stack. This actually turned out to be a performance issue
for me, and although it seldom is, I don't want to end up with the same problem
twice.

To state the somewhat obvious here: This is a *lexer* library. If you wanted to
do easy *parsing*, then I would recommend the amazing
[instaparse](https://github.com/Engelberg/instaparse) library. If you want a
general lexer library for Clojure/JVM, I'd recommend
[ANTLR](http://www.antlr.org/). Hopefully I can someday recommend this lexer
generator instead.

Smallex only emits an rather limited lexer as of now. I'll see if I can pop up a
formal definition when the implementation is done.

## Usage

This is best shown with the output from the help command.

```
usage: java -jar smallex.jar [-v|--version] [-h|--help] [-H|--langhelp lang]
  [--loglevel level] [--logfile file] [-o|--out outdir] [-i|--inputfile file]
  [--list-languages] [-l|--lang language] <language-specific options> ...

Options

 -v, --version
     Prints Smallex' version name.

 -h, --help
     Prints this help.

 -H, --langhelp
     Prints additional information about the specific language, along with other
     flags, the version number and its use.

 --loglevel level
     Specifies the logging level: NONE, INFO or DEBUG. Is by default INFO.

 --logfile file
     Sets the logfile to the specified file. If none is specified, logging is
     sent to stdout.

 --out, -o outdir
     Sets the output directory, and places generated files in that directory. If
     the directory or any parent directory does not exist, will generate those.
     The output directory must be specified.

 --inputfile, -i file
     Specifies the file to read the lexer from.

 --list-languages
     Lists all languages (not necessarily programming languages) supported by
     this version of smallex, along with their version number.

 --lang, -l language
     Specifies the language to emit for. Must be explicitly set if code is to be
     generated.
```

Here is a quick example of how to use the tool: Say I have the file
`lexer.smlx`, and want to generate a java->clojure lexer which should reside in
the directory `mylexer`. Then, the following command is issued (assuming
smallex.jar is in my current directory):

```bash
java -jar smallex.jar -i lexer.smlx --lang java->clojure --out mylexer \
  --pkgname com.hypirion.mylexer
```


## License

Copyright © 2014 by [hyPiRion](https://github.com/hyPiRion) and
[contributors](https://github.com/hyPiRion/smallex).

{'fraction_non_alphanumeric': 0.08107507774322523, 'fraction_numerical': 0.001776988005330964, 'mean_word_length': 3.800639658848614, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1234538', 'n_tokens_mistral': 1348, 'n_tokens_neox': 1268, 'n_words': 634}