<commit_before># Assem's Arabic Stemmer [![Gitter](https://badges.gitter.im/arabicstemmer/Lobby.svg)](https://gitter.im/arabicstemmer/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

This is an algorithm for Arabic stemming written on Snowball framework language. If offers light stemming and text normalization. voc



## Requirements

- [Snowball framework](https://github.com/snowballstem/snowball)
- [Snowball-data](https://github.com/snowballstem/snowball-data)

You can download it automatically using:

```sh
    $ make download
```
or manually by:
- extracting snowball into the root folder `{Root}/snowball`
- extracting snowball-data/arabic/voc.txt.gz into `{Root}/test_snowball/voc.txt`


## Build

```sh
    $ make build
```

## Run

```sh
	$ make run
	الطالب
	طالب

```

## Test 
We configured tests to run against snowball-data arabic sample. 

- time:

    ```sh
	    $ make time
	```

- grouping effect:
    
    ```sh
    $ make grouping
	```
- all:
	
	```sh
    $ make test
	```

# Distributions
```sh
    $ make dist
```
<commit_msg>Fix path in README file<commit_after># Assem's Arabic Stemmer [![Gitter](https://badges.gitter.im/arabicstemmer/Lobby.svg)](https://gitter.im/arabicstemmer/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

This is an algorithm for Arabic stemming written on Snowball framework language. If offers light stemming and text normalization. voc



## Requirements

- [Snowball framework](https://github.com/snowballstem/snowball)
- [Snowball-data](https://github.com/snowballstem/snowball-data)

You can download it automatically using:

```sh
    $ make download
```
or manually by:
- extracting snowball into the root folder `{Root}/snowball`
- extracting snowball-data/arabic/voc.txt.gz into `{Root}/test_data/voc.txt`


## Build

```sh
    $ make build
```

## Run

```sh
	$ make run
	الطالب
	طالب

```

## Test 
We configured tests to run against snowball-data arabic sample. 

- time:

    ```sh
	    $ make time
	```

- grouping effect:
    
    ```sh
    $ make grouping
	```
- all:
	
	```sh
    $ make test
	```

# Distributions
```sh
    $ make dist
```

{'fraction_non_alphanumeric': 0.13786764705882354, 'fraction_numerical': 0.0, 'mean_word_length': 3.7429193899782134, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5335629', 'n_tokens_mistral': 838, 'n_tokens_neox': 759, 'n_words': 182}