<commit_before>
# zstdmt

zstdmt depends currently on posix threads for multi-threading and is
only tested on linux

I will add support for windows and CreateThread() together with
WaitForMultipleObjects() / WaitForSingleObject() later...

## Building for Linux with gcc

 - Run `make`.

## Building for Linux with Clang

 - Run `make CC=clang`

## Overview

 - three tools:
   - `./zstd-st` - single threaded multi stram mode
   - `./zstd-mt` - multi threaded via phread_create / pthread_join
   - `./zstd-mt2` - multi threaded via X workers that read/compress/write
     (decompress has to be changed a bit)

## See also

 - [zstd Extremely Fast Compression algorithm](https://github.com/Cyan4973/zstd)
 - [7-Zip with zstd support](https://github.com/mcmilk/7-Zip-Zstd)

## Tables with some first testing...

 - System: Arch Linux, Kernel 4.3.6-ck
 - Intel(R) Core(TM) i7-3632QM, limited to 1,2 .. 1,8 GHz
 - data.tar contains the tarred silesia directory

Level|Threads|InSize|OutSize|Frames*|Real|User|Sys|MaxMem
-----|-------|------|-------|--------|----|----|---|-------
ST-L1|1|211957760|73662797|1618|2,8|1,964|0,39|2524
MT-L1|1|211957760|73662797|1618|2,114|1,332|0,2|2644
MT2-L1|1|211957760|73662815|1619|2,323|2,266|0,50|2580
ST-L1|2|211957760|74127590|809|2,14|1,975|0,34|3768
MT-L1|2|211957760|74127590|809|1,129|1,299|0,6|3688
MT2-L1|2|211957760|74112206|1620|1,12|1,981|0,38|7256
ST-L1|4|211957760|74473001|405|2,54|2,7|0,41|5860
MT-L1|4|211957760|74473001|405|1,33|1,609|0,8|6068
MT2-L1|4|211957760|74442594|1622|0,589|2,296|0,47|7264
ST-L1|8|211957760|74689944|203|2,175|2,125|0,44|10460
MT-L1|8|211957760|74689944|203|0,848|2,639|0,122|20968
MT2-L1|8|211957760|74675136|1626|0,450|3,456|0,50|14808
ST-L1|16|211957760|74852878|102|2,247|2,191|0,50|19420
MT-L1|16|211957760|74852878|102|0,639|2,668|0,38|19748
MT2-L1|16|211957760|74824351|1634|0,460|3,542|0,60|28764

Level|Threads|InSize|OutSize|Frames*|Real|User|Sys|MaxMem
-----|-------|------|-------|--------|----|----|---|-------
ST-L5|1|211957760|65010435|1618|4,977|4,925|0,38|4116
MT-L5|1|211957760|65010435|1618|5,273|4,448|0,3|4312
MT2-L5|1|211957760|65010463|1619|5,216|5,154|0,49|4204
ST-L5|2|211957760|65648113|809|6,602|6,513|0,54|7032
MT-L5|2|211957760|65648113|809|3,300|5,340|0,3|7044
MT2-L5|2|211957760|65689018|1620|2,699|5,349|0,34|7392
ST-L5|4|211957760|66124197|405|6,545|6,462|0,53|12488
MT-L5|4|211957760|66124197|405|2,644|6,902|0,10|12656
MT2-L5|4|211957760|66124630|1622|1,740|6,874|0,46|13904
ST-L5|8|211957760|66589424|203|6,451|6,375|0,49|23564
MT-L5|8|211957760|66589424|203|2,77|11,244|0,16|23856
MT2-L5|8|211957760|66629780|1626|1,563|12,167|0,75|25764
ST-L5|16|211957760|67098645|102|6,389|6,308|0,61|45748
MT-L5|16|211957760|67098645|102|1,831|11,531|0,39|46312
MT2-L5|16|211957760|67106059|1634|1,622|12,610|0,58|54296

Level|Threads|InSize|OutSize|Frames*|Real|User|Sys|MaxMem
-----|-------|------|-------|--------|----|----|---|-------
ST-L10|1|211957760|60182718|1618|14,258|14,154|0,50|14396
MT-L10|1|211957760|60182718|1618|13,652|12,766|0,3|14396
MT2-L10|16|211957760|63582919|1634|3,425|26,495|0,90|213340
ST-L10|2|211957760|61052907|809|15,70|14,976|0,39|27268
MT-L10|2|211957760|61052907|809|8,553|15,443|0,7|27440
MT2-L10|8|211957760|62696412|1626|3,534|27,529|0,82|106624
ST-L10|4|211957760|61822686|405|15,89|14,984|0,56|53112
MT-L10|4|211957760|61822686|405|6,260|20,20|0,11|53048
MT2-L10|4|211957760|61872471|1622|4,676|18,538|0,60|54312
ST-L10|8|211957760|62672817|203|14,930|14,799|0,75|104500
MT-L10|8|211957760|62672817|203|4,304|26,701|0,24|105016
MT2-L10|2|211957760|61100119|1620|7,739|15,384|0,47|27724
ST-L10|16|211957760|63567841|102|14,102|13,966|0,81|207516
MT-L10|16|211957760|63567841|102|3,836|26,19|0,48|208028
MT2-L10|1|211957760|60182727|1619|14,623|14,526|0,54|14520

Level|Threads|InSize|OutSize|Frames*|Real|User|Sys|MaxMem
-----|-------|------|-------|--------|----|----|---|-------
ST-L15|1|211957760|58020189|1618|56,645|56,405|0,58|22584
MT-L15|1|211957760|58020189|1618|56,101|55,118|0,6|22560
MT2-L15|1|211957760|58020195|1619|58,125|57,895|0,84|22736
ST-L15|2|211957760|59057591|809|54,941|54,718|0,63|43728
MT-L15|2|211957760|59057591|809|30,711|58,773|0,10|43760
MT2-L15|2|211957760|59088525|1620|30,270|60,294|0,78|44168
ST-L15|4|211957760|60087164|405|52,292|52,81|0,69|85796
MT-L15|4|211957760|60087164|405|18,830|67,256|0,16|85928
MT2-L15|4|211957760|60149893|1622|15,722|62,551|0,73|87268
ST-L15|8|211957760|61218687|203|50,817|50,554|0,85|170632
MT-L15|8|211957760|61218687|203|11,79|76,665|0,74|180596
MT2-L15|8|211957760|61245001|1626|9,909|77,522|0,98|173648
ST-L15|16|211957760|62444174|102|45,451|45,219|0,98|339264
MT-L15|16|211957760|62444174|102|10,251|73,660|0,75|340800
MT2-L15|16|211957760|62499547|1634|9,443|73,615|0,126|345232

Level|Threads|InSize|OutSize|Frames*|Real|User|Sys|MaxMem
-----|-------|------|-------|--------|----|----|---|-------
ST-L20|1|211957760|53017204|1618|138,233|137,762|0,91|330768
MT-L20|1|211957760|53017204|1618|138,18|136,772|0,34|330696
ST-L20|2|211957760|54572596|809|129,424|128,913|0,129|659372
MT-L20|2|211957760|54572596|809|75,60|144,82|0,69|659616
ST-L20|4|211957760|56001912|405|120,462|119,885|0,198|1317632
MT-L20|4|211957760|56001912|405|42,304|152,492|0,129|1317664
ST-L20|8|211957760|57295695|203|108,370|107,757|0,317|2586776
MT-L20|8|211957760|57295695|203|24,264|169,436|0,253|2585756
ST-L20|16|211957760|58551804|102|100,35|99,183|0,564|4965152
MT-L20|16|211957760|58551804|102|22,791|162,112|0,505|4966428

Frames - was FrameSet first... will be changed to Frames written<commit_msg>	geändert:       README.md<commit_after>
# zstdmt

zstdmt depends currently on posix threads for multi-threading and is
only tested on linux

I will add support for windows and CreateThread() together with
WaitForMultipleObjects() / WaitForSingleObject() later...

## Building for Linux with gcc

 - Run `make`.

## Building for Linux with Clang

 - Run `make CC=clang`

## Overview

 - three tools:
   - `./zstd-st` - single threaded multi stram mode
   - `./zstd-mt` - multi threaded via phread_create / pthread_join
   - `./zstd-mt2` - multi threaded via X workers that read/compress/write
     (decompress has to be changed a bit)

## See also

 - [zstd Extremely Fast Compression algorithm](https://github.com/Cyan4973/zstd)
 - [7-Zip with zstd support](https://github.com/mcmilk/7-Zip-Zstd)

## Tables with some first testing...

 - System: Arch Linux, Kernel 4.3.6-ck
 - Intel(R) Core(TM) i7-3632QM, limited to 1,2 .. 1,8 GHz
 - data.tar contains the tarred silesia directory

Level|Threads|InSize|OutSize|Frames*|Real|User|Sys|MaxMem
-----|-------|------|-------|--------|----|----|---|-------
ST-L1|1|211957760|73662797|1618|2,8|1,964|0,39|2524
MT-L1|1|211957760|73662797|1618|2,114|1,332|0,2|2644
MT2-L1|1|211957760|73662815|1619|2,323|2,266|0,50|2580
ST-L1|2|211957760|74127590|809|2,14|1,975|0,34|3768
MT-L1|2|211957760|74127590|809|1,129|1,299|0,6|3688
MT2-L1|2|211957760|74112206|1620|1,12|1,981|0,38|7256
ST-L1|4|211957760|74473001|405|2,54|2,7|0,41|5860
MT-L1|4|211957760|74473001|405|1,33|1,609|0,8|6068
MT2-L1|4|211957760|74442594|1622|0,589|2,296|0,47|7264
ST-L1|8|211957760|74689944|203|2,175|2,125|0,44|10460
MT-L1|8|211957760|74689944|203|0,848|2,639|0,122|20968
MT2-L1|8|211957760|74675136|1626|0,450|3,456|0,50|14808
ST-L1|16|211957760|74852878|102|2,247|2,191|0,50|19420
MT-L1|16|211957760|74852878|102|0,639|2,668|0,38|19748
MT2-L1|16|211957760|74824351|1634|0,460|3,542|0,60|28764

Level|Threads|InSize|OutSize|Frames*|Real|User|Sys|MaxMem
-----|-------|------|-------|--------|----|----|---|-------
ST-L5|1|211957760|65010435|1618|4,977|4,925|0,38|4116
MT-L5|1|211957760|65010435|1618|5,273|4,448|0,3|4312
MT2-L5|1|211957760|65010463|1619|5,216|5,154|0,49|4204
ST-L5|2|211957760|65648113|809|6,602|6,513|0,54|7032
MT-L5|2|211957760|65648113|809|3,300|5,340|0,3|7044
MT2-L5|2|211957760|65689018|1620|2,699|5,349|0,34|7392
ST-L5|4|211957760|66124197|405|6,545|6,462|0,53|12488
MT-L5|4|211957760|66124197|405|2,644|6,902|0,10|12656
MT2-L5|4|211957760|66124630|1622|1,740|6,874|0,46|13904
ST-L5|8|211957760|66589424|203|6,451|6,375|0,49|23564
MT-L5|8|211957760|66589424|203|2,77|11,244|0,16|23856
MT2-L5|8|211957760|66629780|1626|1,563|12,167|0,75|25764
ST-L5|16|211957760|67098645|102|6,389|6,308|0,61|45748
MT-L5|16|211957760|67098645|102|1,831|11,531|0,39|46312
MT2-L5|16|211957760|67106059|1634|1,622|12,610|0,58|54296

Level|Threads|InSize|OutSize|Frames*|Real|User|Sys|MaxMem
-----|-------|------|-------|--------|----|----|---|-------
ST-L10|1|211957760|60182718|1618|14,258|14,154|0,50|14396
MT-L10|1|211957760|60182718|1618|13,652|12,766|0,3|14396
MT2-L10|1|211957760|60182727|1619|14,623|14,526|0,54|14520
ST-L10|2|211957760|61052907|809|15,70|14,976|0,39|27268
MT-L10|2|211957760|61052907|809|8,553|15,443|0,7|27440
MT2-L10|2|211957760|61100119|1620|7,739|15,384|0,47|27724
ST-L10|4|211957760|61822686|405|15,89|14,984|0,56|53112
MT-L10|4|211957760|61822686|405|6,260|20,20|0,11|53048
MT2-L10|4|211957760|61872471|1622|4,676|18,538|0,60|54312
ST-L10|8|211957760|62672817|203|14,930|14,799|0,75|104500
MT-L10|8|211957760|62672817|203|4,304|26,701|0,24|105016
MT2-L10|8|211957760|62696412|1626|3,534|27,529|0,82|106624
ST-L10|16|211957760|63567841|102|14,102|13,966|0,81|207516
MT-L10|16|211957760|63567841|102|3,836|26,19|0,48|208028
MT2-L10|16|211957760|63582919|1634|3,425|26,495|0,90|213340

Level|Threads|InSize|OutSize|Frames*|Real|User|Sys|MaxMem
-----|-------|------|-------|--------|----|----|---|-------
ST-L15|1|211957760|58020189|1618|56,645|56,405|0,58|22584
MT-L15|1|211957760|58020189|1618|56,101|55,118|0,6|22560
MT2-L15|1|211957760|58020195|1619|58,125|57,895|0,84|22736
ST-L15|2|211957760|59057591|809|54,941|54,718|0,63|43728
MT-L15|2|211957760|59057591|809|30,711|58,773|0,10|43760
MT2-L15|2|211957760|59088525|1620|30,270|60,294|0,78|44168
ST-L15|4|211957760|60087164|405|52,292|52,81|0,69|85796
MT-L15|4|211957760|60087164|405|18,830|67,256|0,16|85928
MT2-L15|4|211957760|60149893|1622|15,722|62,551|0,73|87268
ST-L15|8|211957760|61218687|203|50,817|50,554|0,85|170632
MT-L15|8|211957760|61218687|203|11,79|76,665|0,74|180596
MT2-L15|8|211957760|61245001|1626|9,909|77,522|0,98|173648
ST-L15|16|211957760|62444174|102|45,451|45,219|0,98|339264
MT-L15|16|211957760|62444174|102|10,251|73,660|0,75|340800
MT2-L15|16|211957760|62499547|1634|9,443|73,615|0,126|345232

Level|Threads|InSize|OutSize|Frames*|Real|User|Sys|MaxMem
-----|-------|------|-------|--------|----|----|---|-------
ST-L20|1|211957760|53017204|1618|138,233|137,762|0,91|330768
MT-L20|1|211957760|53017204|1618|138,18|136,772|0,34|330696
ST-L20|2|211957760|54572596|809|129,424|128,913|0,129|659372
MT-L20|2|211957760|54572596|809|75,60|144,82|0,69|659616
ST-L20|4|211957760|56001912|405|120,462|119,885|0,198|1317632
MT-L20|4|211957760|56001912|405|42,304|152,492|0,129|1317664
ST-L20|8|211957760|57295695|203|108,370|107,757|0,317|2586776
MT-L20|8|211957760|57295695|203|24,264|169,436|0,253|2585756
ST-L20|16|211957760|58551804|102|100,35|99,183|0,564|4965152
MT-L20|16|211957760|58551804|102|22,791|162,112|0,505|4966428

\* Frames - was FrameSet first... will be changed to Frames written
{'fraction_non_alphanumeric': 0.23082411701748126, 'fraction_numerical': 0.5114163396361042, 'mean_word_length': 7.598926380368098, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '432185', 'n_tokens_mistral': 8922, 'n_tokens_neox': 5927, 'n_words': 389}