<commit_before># XLParser
A C# Excel formula parser with the following properties:

* **High compatiblity**<br/>
  XLParser has been tested on over a million real-world formulas and has a 99.9% succesful parse rate.
* **Compact parse trees**<br/>
  XLParser was designed for and is used in research of Excel spreadsheets and refactoring, all of which are easier if parse trees are smaller
* **Compact grammar**<br />
  [Our grammar](https://github.com/PerfectXL/XLParser/blob/master/src/XLParser/ExcelFormulaGrammar.cs) contains less than 100 tokens and production rules, and is thus easy to implement in another language or parser generator.

### Quickstart

You can parse formulas and view parse trees in [the online demo](https://xlparser.perfectxl.nl/demo).

**or**

Use [the NuGet package](https://www.nuget.org/packages/XLParser/).

**or**

1. Download the [latest release](https://github.com/PerfectXL/XLParser/releases/latest)
2. Extract somewhere convenient
3. Build Irony GrammarExplorer, see the Debugging section below, and open it
4. Click on the `...` button at the top and select `Add Grammar`
5. Point to the `XLParser.dll` file in the folder you extracted and click ok
6. You can now parse formulas and see the trees in the `Test` tab

## Background

XLParser is the reference implementation of the Excel grammar published in the paper ["A Grammar for Spreadsheet Formulas Evaluated on Two Large Datasets" by <NAME>, <NAME> and <NAME>](https://fenia266781730.files.wordpress.com/2019/01/07335408.pdf).

XLParser can parse Excel formulas and is intended to facilitate the analysis of spreadsheet formulas, and for that purpose produces compact parse trees.
XLParser  has a 99.99% success rate on the [Enron](http://www.felienne.com/archives/3634) and [EUSES](http://eusesconsortium.org/resources.php) datasets.
Note however that XLParser is not very restrictive, and thus might parse formulas that Excel would reject as invalid, keep this in mind when parsing user input with XLParser.

XLParser is based on the C# [Irony parser framework](https://github.com/IronyProject/Irony).

## Using XLParser

### Building

Open the `XLParser.sln` file in `src/` in Visual Studio and press build. The dependencies are already included in compiled form in this repository.

### Using in your project

The easiest way to add the dependency to your project is with [NuGet](https://www.nuget.org/packages/XLParser/)

The `ExcelFormulaParser` class is your main entry point. You can parse a formula through `ExcelFormulaParser.Parse("yourformula")`.

`ExcelFormulaParser` has several useful methods that operate directly on the parse tree like `AllNodes` to traverse the whole tree or `GetFunction` to get the function name of a node that represents a function call. You can `Print` any node.

`FormulaAnalyzer` provides functionality for analyzing the parse tree. For example, `FormulaAnalyzer.ParserReferences()` returns all references (to cells, cell ranges, named ranges, horizontal or vertical ranges) that the formula contains.        

### Debugging

Irony, the parser framework XLParser uses, includes a tool called the "grammar explorer". This is a great way to play around with the grammar and parse trees.
To use this tool, you first need to build it once by opening the Irony solution (`lib/Irony/Irony_All.2012.sln`) and building it with release configuration. After that you can use the binary in `lib/Irony/Irony.GrammarExplorer/bin/Release/Irony.GrammarExplorer.exe`.

To load the XLParser grammar, first make sure you have built XLParser. Then open the GrammarExplorer and add the grammar (`...` button) from `src/XLParser/bin/Debug/XLParser.dll`.

In Visual Studio you can see the printed version of any node during debugging by adding `yournode.Print(),ac` in the watch window.

## Documentation

See [the doc folder](doc) for further documentation
<commit_msg>Add link to API documentation<commit_after># XLParser
A C# Excel formula parser with the following properties:

* **High compatiblity**<br/>
  XLParser has been tested on over a million real-world formulas and has a 99.9% succesful parse rate.
* **Compact parse trees**<br/>
  XLParser was designed for and is used in research of Excel spreadsheets and refactoring, all of which are easier if parse trees are smaller
* **Compact grammar**<br />
  [Our grammar](https://github.com/PerfectXL/XLParser/blob/master/src/XLParser/ExcelFormulaGrammar.cs) contains less than 100 tokens and production rules, and is thus easy to implement in another language or parser generator.

### Quickstart

You can parse formulas and view parse trees in [the online demo](https://xlparser.perfectxl.nl/demo).

**or**

Use [the NuGet package](https://www.nuget.org/packages/XLParser/).

**or**

1. Download the [latest release](https://github.com/PerfectXL/XLParser/releases/latest)
2. Extract somewhere convenient
3. Build Irony GrammarExplorer, see the Debugging section below, and open it
4. Click on the `...` button at the top and select `Add Grammar`
5. Point to the `XLParser.dll` file in the folder you extracted and click ok
6. You can now parse formulas and see the trees in the `Test` tab

## Background

XLParser is the reference implementation of the Excel grammar published in the paper ["A Grammar for Spreadsheet Formulas Evaluated on Two Large Datasets" by <NAME>, <NAME> and <NAME>](https://fenia266781730.files.wordpress.com/2019/01/07335408.pdf).

XLParser can parse Excel formulas and is intended to facilitate the analysis of spreadsheet formulas, and for that purpose produces compact parse trees.
XLParser  has a 99.99% success rate on the [Enron](http://www.felienne.com/archives/3634) and [EUSES](http://eusesconsortium.org/resources.php) datasets.
Note however that XLParser is not very restrictive, and thus might parse formulas that Excel would reject as invalid, keep this in mind when parsing user input with XLParser.

XLParser is based on the C# [Irony parser framework](https://github.com/IronyProject/Irony).

## Using XLParser

### Building

Open the `XLParser.sln` file in `src/` in Visual Studio and press build. The dependencies are already included in compiled form in this repository.

### Using in your project

The easiest way to add the dependency to your project is with [NuGet](https://www.nuget.org/packages/XLParser/)

The `ExcelFormulaParser` class is your main entry point. You can parse a formula through `ExcelFormulaParser.Parse("yourformula")`.

`ExcelFormulaParser` has several useful methods that operate directly on the parse tree like `AllNodes` to traverse the whole tree or `GetFunction` to get the function name of a node that represents a function call. You can `Print` any node.

`FormulaAnalyzer` provides functionality for analyzing the parse tree. For example, `FormulaAnalyzer.ParserReferences()` returns all references (to cells, cell ranges, named ranges, horizontal or vertical ranges) that the formula contains.        

### Debugging

Irony, the parser framework XLParser uses, includes a tool called the "grammar explorer". This is a great way to play around with the grammar and parse trees.
To use this tool, you first need to build it once by opening the Irony solution (`lib/Irony/Irony_All.2012.sln`) and building it with release configuration. After that you can use the binary in `lib/Irony/Irony.GrammarExplorer/bin/Release/Irony.GrammarExplorer.exe`.

To load the XLParser grammar, first make sure you have built XLParser. Then open the GrammarExplorer and add the grammar (`...` button) from `src/XLParser/bin/Debug/XLParser.dll`.

In Visual Studio you can see the printed version of any node during debugging by adding `yournode.Print(),ac` in the watch window.

## Documentation

Browse the [API documentation](http://spreadsheetlab.github.io/XLParser/api/index.html)

See [the doc folder](doc) for further documentation.

{'fraction_non_alphanumeric': 0.08083375699034062, 'fraction_numerical': 0.011947127605490595, 'mean_word_length': 4.706308919506889, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 14, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2158629', 'n_tokens_mistral': 2303, 'n_tokens_neox': 2125, 'n_words': 1008}