<commit_before>---
layout: post
title:  "Building a Better Java Parser"
date:   2015-01-15 19:33:00.000000000 +2:00
---
During my [last post]({% post_url 2015-01-06-porting-java-inner-and-anonymous-classes-to-typescript %}), I claimed all that was needed was to add the ability to my Transpiler to emit nested and anonymous classes, and by now I expected to be done with that. It turned out that what I though was a "good enough" Transpiler, was in fact, not.

In particular, its major failing was that its parsing was far too fragile. Even aside from trying to parse nested and anonymous classes, which should have been *relatively* easy to add, depending on the way the source Java was written, often enough it would incorrectly recognise a control structure, method call, etc and the parsing would fail.

Truthfully, my parser was pretty rubbish, but it was my first attempt at a parser and, unsurprisingly, I learnt many lessons along the way as I came to better understand the domain of parsing. While it's time consuming and disappointing that I had to resort to re-writing my parser, software developers learning such lessons in their inexperienced domain is actually pretty normal, hence why well experienced developers can normally demand a higher salary.

I have no formal training on how to write a parser and hence "winged it" at first, but having written one which mostly worked before, I know what worked well, and what I should change. My parser was broken into three parts:

1. [The Lexer](http://en.wikipedia.org/wiki/Lexical_analysis), which does tokenization of the input file(s)
2. The Parser, which uses the tokens to identify Java structures and then creates an [AST (Abstract Syntax Tree)](http://en.wikipedia.org/wiki/Abstract_syntax_tree).
3. The TypeScript Emitter, which traverses the AST and emits TypeScript code.

Lexing
---
To summarize lexing, it's the process of taking an input like:<commit_msg>And trying again.<commit_after>---
layout: post
title:  "Building a Better Java Parser"
date:   2015-01-15 12:33:00.000000000 +2:00
---
During my [last post]({% post_url 2015-01-06-porting-java-inner-and-anonymous-classes-to-typescript %}), I claimed all that was needed was to add the ability to my Transpiler to emit nested and anonymous classes, and by now I expected to be done with that. It turned out that what I though was a "good enough" Transpiler, was in fact, not.

In particular, its major failing was that its parsing was far too fragile. Even aside from trying to parse nested and anonymous classes, which should have been *relatively* easy to add, depending on the way the source Java was written, often enough it would incorrectly recognise a control structure, method call, etc and the parsing would fail.

Truthfully, my parser was pretty rubbish, but it was my first attempt at a parser and, unsurprisingly, I learnt many lessons along the way as I came to better understand the domain of parsing. While it's time consuming and disappointing that I had to resort to re-writing my parser, software developers learning such lessons in their inexperienced domain is actually pretty normal, hence why well experienced developers can normally demand a higher salary.

I have no formal training on how to write a parser and hence "winged it" at first, but having written one which mostly worked before, I know what worked well, and what I should change. My parser was broken into three parts:

1. [The Lexer](http://en.wikipedia.org/wiki/Lexical_analysis), which does tokenization of the input file(s)
2. The Parser, which uses the tokens to identify Java structures and then creates an [AST (Abstract Syntax Tree)](http://en.wikipedia.org/wiki/Abstract_syntax_tree).
3. The TypeScript Emitter, which traverses the AST and emits TypeScript code.

Lexing
---
To summarize lexing, it's the process of taking an input like:
{'fraction_non_alphanumeric': 0.059295701708959085, 'fraction_numerical': 0.019161056447436563, 'mean_word_length': 4.534383954154728, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1364283', 'n_tokens_mistral': 1057, 'n_tokens_neox': 954, 'n_words': 581}