<commit_before>giga
====

Concurrent File I/O on Arbitrarily-Sized Files

Overview
----

`giga` is a concurrent file I/O library which seeks to ameliorate large-file memory problems.

Internally, a file is represented in memory with a linked-list bit-array keeping track of dirty blocks of data. On file save, all the dirty blocks are then written to 
the file; intermediate states of dirty blocks are stored on disk in `/tmp/`. To keep the memory footprint small, `giga` only keeps recently-accessed blocks in RAM with a 
least-used allocation policy (generalized second-chance).

Current Features
----
* unit tests covering concurrency usage
* concurrent and sequential reads implemented

Planned Features
----
* `overwrite(buf)` (cf. `fwrite(..., buf, ...)`), `insert(buf)`, and `delete(n)` write modes
* `file.save()`
* possible network integration
* performance testing

Installation
----

```
git clone --recursive https://github.com/cripplet/giga.git
```

cf. `test/concurrent.cc` for usage example

Testing<commit_msg>documentation edits<commit_after>giga
====

Concurrent File I/O on Arbitrarily-Sized Files

Overview
----

`giga` is a concurrent file I/O library which seeks to ameliorate large-file memory problems.

Internally, a file is represented in memory with a linked-list bit-array keeping track of dirty blocks of data. On file save, all the dirty blocks are then written to 
the file; intermediate states of dirty blocks are stored on disk in `/tmp/`. To keep the memory footprint small, `giga` only keeps recently-accessed blocks in RAM with a 
least-used allocation policy (generalized second-chance).

Current Features
----
* unit tests covering concurrency usage
* concurrent and sequential reads implemented

Planned Features
----
* `overwrite(buf)` (cf. `fwrite(..., buf, ...)`), `insert(buf)`, and `delete(n)` write modes
* `save()`
* possible network integration
* performance testing

Known Bugs
----
* memory leak when calling file destructor, due to linked list

Installation
----

```
git clone --recursive https://github.com/cripplet/giga.git
```

cf. `test/concurrent.cc` for usage example

Testing
{'fraction_non_alphanumeric': 0.10548523206751055, 'fraction_numerical': 0.0, 'mean_word_length': 4.105263157894737, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1709593', 'n_tokens_mistral': 616, 'n_tokens_neox': 587, 'n_words': 260}