<commit_before># Counts the number of times each element occurs in a list

This project implements the same frequency counter a number of times,
trying to systematically increase the parallelism of the program with each implementation.

It tries to implement parallel techniques from chapter 2 and 3 in
[Simon Marlow's book](http://chimera.labs.oreilly.com/books/1230000000929/index.html).

The frequency counter accepts a text file as input from the command-line, and then prints the frequency of each word
and the frequency of each character.

## Running the command-line executables

You can run all of these with summarised output by using the following script:
```
$ ./summary.sh flatland.txt
1-seq-datalist.sh
  Total   time    0.534s  (  0.549s elapsed)

f2-par-datalist.sh
  TASKS: 6 (1 bound, 5 peak workers (5 total), using -N2)
  SPARKS: 2 (2 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
  Total   time    0.756s  (  0.445s elapsed)

f3-seq-datamap.sh
  Total   time    0.168s  (  0.178s elapsed)<commit_msg>How to get sample text<commit_after># Counts the number of times each element occurs in a list

This project implements the same frequency counter a number of times,
trying to systematically increase the parallelism of the program with each implementation.

It tries to implement parallel techniques from chapter 2 and 3 in
[Simon Marlow's book](http://chimera.labs.oreilly.com/books/1230000000929/index.html).

The frequency counter accepts a text file as input from the command-line, and then prints the frequency of each word
and the frequency of each character.

## Sample Text Files

[Flatland](https://en.wikipedia.org/wiki/Flatland) by <NAME> is a small text and included in this repository.

I mostly ran my programs against [the longest book](https://en.wikipedia.org/wiki/List_of_longest_novels) I could find:
A French novel called [Artamène](https://en.wikipedia.org/wiki/Artam%C3%A8ne).

You can download it with the following command:
```
curl http://www.artamene.org/documents/cyrus[1-10].txt | iconv --from-code=ISO-8859-1 --to-code=UTF8 > artamene.txt
```

## Running the command-line executables

You can run all of these with summarised output by using the following script:
```
$ ./summary.sh flatland.txt
1-seq-datalist.sh
  Total   time    0.534s  (  0.549s elapsed)

f2-par-datalist.sh
  TASKS: 6 (1 bound, 5 peak workers (5 total), using -N2)
  SPARKS: 2 (2 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
  Total   time    0.756s  (  0.445s elapsed)

f3-seq-datamap.sh
  Total   time    0.168s  (  0.178s elapsed)
{'fraction_non_alphanumeric': 0.0861598440545809, 'fraction_numerical': 0.0456140350877193, 'mean_word_length': 3.8876190476190478, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 5, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3646461', 'n_tokens_mistral': 886, 'n_tokens_neox': 799, 'n_words': 323}