<commit_before># gogrep

In my daily job, I'm working on an Windows application that 
generates numberous and detailled log files. During 24h, we have 
100 thousands of files, for several gigabytes. They are zipped daily.

To search something into logs, archives must be unziped 
before using a search tool like grep or FINDSTR. This 
process need some disk space, and is very time consuming.

This was my motivation for wrtiting this utility.

## Requierments for a tool to search into Zip archives

* faster than unzipping then search
* simple to use
* windows and linux bianries

And, that would be great to have a CSV, or better Excel output.


### Faster than unzipping then search
Grep is fast! 5 seconds on my testing file set. So, I don't expect 
to beat it on pure performance area. 

Speed gain comes from skipping itermediate file storage.


### Simple to use
No need to implement the rich palette of grep options. Let go
 strait to the point: search pattern in archived files.

### Cross platform
Thanks to Go compiler, it is (really) easy to compile 
binaries for a varierty of OSes and hardware. 

# Performances

## Standard tools

Test set|Command|Duration|Remark
--------|-------|--------|-------<commit_msg>readme editing<commit_after># gogrep

In my daily job, I'm working on an Windows application that 
generates numberous and detailled log files. During 24h, we have 
100 thousands of files, for several gigabytes. They are zipped daily.

To search something into logs, archives must be unziped 
before using a search tool like grep or FINDSTR. This 
process need some disk space, and is very time consuming.

This was my motivation for wrtiting this utility.

## Requierments for a tool to search into Zip archives

* faster than unzipping then search
* simple to use, one tool for searching files, folders, archives
* read hudge .zip files
* works with plain ascii, but also with utf-8, utf-16 encoded files
* windows and linux binaries

And, nice to have features:
* CSV output
* Excel output, why not?
* Read tgz files,
* Read tar files

## TODO:
- [*] ascii and UTF-8 
- [ ] UTF-16 reading
- [ ] CSV output

### Faster than unzipping then search
Grep is fast! 5 seconds on my testing file set. So, I don't expect 
to beat it on pure performance area. 

Speed gain comes from skipping itermediate file storage.


### Simple to use
No need to implement the rich palette of grep options. Let go
 strait to the point: search pattern in archived files.

### Cross platform
Thanks to Go compiler, it is (really) easy to compile 
binaries for a varierty of OSes and hardware. 

# Performances

## Standard tools

Test set|Command|Duration|Remark
--------|-------|--------|-------
{'fraction_non_alphanumeric': 0.07447202667654687, 'fraction_numerical': 0.006669136717302705, 'mean_word_length': 3.5996592844974447, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1804620', 'n_tokens_mistral': 798, 'n_tokens_neox': 739, 'n_words': 411}