<commit_before># ETF-Java
A Java parser/writer for the ETF format.

This api aims to provide high speed low and high level abstractions
for ETF. Additionally it supports multiple subsets of ETF: standard ETF,
BERT-rpc and Hammer & Chisel's (Discord's) Loqui.

## Using ETF-Java
**TODO**

## A note about performance
While this project attempts to be as well optimized as possible, the JVM
does however creates a limit to how fast this can be just due to how it 
manages memory. 

Performance can be improved via JNI or usage of the `Unsafe` class
but these have issues which has kept me using pure java. 

Regarding JNI: I do not know enough C/C++ to properly optimize and there
would still be a cost for interfacing natives with java (meaning that for 
smaller ETF data there won't be much of a benefit).

Regarding `Unsafe`: Unsafe is *unsafe*. Meaning it is not guaranteed to 
work across java versions and jvm implementations. So despite its 
significant performance benefits, it's impossible to be sure that it will
always work how I expect it to.

## Caveats
Due to java not implementing proper primitive unsigned data types, I have
been forced to use much larger data types than should be necessary to 
prevent potential underflows (ex. `long` instead of `int` or `BigInteger` 
instead of `long`). This causes some strange internal data type handling
and some slight overhead.

## Helpful Resources
* [ETF Documentation](http://erlang.org/doc/apps/erts/erl_ext_dist.html)
* [BERT Documentation](http://bert-rpc.org/)
* [Loqui](https://github.com/hammerandchisel/loqui)<commit_msg>For Panda :eyes:<commit_after># ETF-Java
A Java parser/writer for the ETF format.

This api aims to provide a set of high-speed abstractions for ETF. 
Additionally, it supports multiple subsets of ETF: standard ETF, BERT-rpc
and Hammer & Chisel's (Discord's) Loqui.

## Using ETF-Java
**TODO**

## A note about performance
While this project attempts to be as well optimized as possible, the JVM
creates a limit to how fast this can be just due to how it manages memory. 

Performance can be improved via JNI or usage of the `Unsafe` class
but these have issues which has kept me using pure java. 

Regarding JNI: I do not know enough C/C++ to properly optimize and there
would still be a cost for interfacing natives with java (meaning that for 
smaller ETF data there won't be much of a benefit).

Regarding `Unsafe`: Unsafe is *unsafe*. Meaning it is not guaranteed to 
work across java versions and jvm implementations. So despite its 
significant performance benefits, it's impossible to be sure that it will
always work how I expect it to.

## Caveats
Due to java not implementing proper primitive unsigned data types, I have
been forced to use much larger data types than should be necessary to 
prevent potential underflows (ex. `long` instead of `int` or `BigInteger` 
instead of `long`). This causes some strange internal data type handling
and some slight overhead.

## Helpful Resources
* [ETF Documentation](http://erlang.org/doc/apps/erts/erl_ext_dist.html)
* [BERT Documentation](http://bert-rpc.org/)
* [Loqui](https://github.com/hammerandchisel/loqui)
{'fraction_non_alphanumeric': 0.06656101426307448, 'fraction_numerical': 0.0, 'mean_word_length': 4.441379310344828, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1806471', 'n_tokens_mistral': 927, 'n_tokens_neox': 857, 'n_words': 461}