<commit_before># kiavertor

kiavertor.pl -f FILENAME -e TARGETENCODING [OPTIONS...]

"-f [filename] or -d [dirname]  and -e [encode format] is not optional

Options:

 --help or -h                                         : this help
 
 --filesource=/path/file or -f /path/file             : input file name
 
 --encodesource=cp1256 or -s cp1256                   : input file source encoding,
 
                                                        if dont use it encoding determined automaticly
                                                        
                                                        if possible
                                                        
 --encodetarget=utf8 or -e utf8                       : input file target encoding
 
 --directory=/path or -d /path                        : choose all file in directory,
 
                                                        use with -e switche,
                                                        
                                                        if no -s switch source encoding
                                                        
                                                        determined automaticly if possible
                                                        
 if use [--directory=here] or [-d here] use current path that you run $0 from
 
 --fileextension = txt or -x txt                      : if use -d then you can determine
 
                                                        file extensions you want
                                                        
                                                        default file name is /path/file-encodetarget)
                                                        
 --dirtarget=/path/dirpath/ or -t /path/dirpath/      : dir path for new files
                                                        <commit_msg>change md file<commit_after># kiavertor

simple file decode-encode tools like iconv but with different opts

```sh
kiavertor.pl -f FILENAME -d DIRNAME -e TARGETENCODING [OPTIONS...]
```

"-f [filename] or -d [dirname]  and -e [file encoding types] is not optional

### Options:

* **--help** or **-h**
* **--filesource=**/path/filename or **-f** /path/filename
* **--encodesource=**DecodingType or **-s** DecodingType
  * if dont use -s opt, decoding determine automaticly if possible
* **--encodetarget=**EncodingType or **-e** EncodingType
* **--directory=**/path/ or **-d** /path/
 * choose all files in directory
    if use [--directory=here] or [-d here] use current path that you run
* **--fileextension**=fileExtension or **-x** FileExtesnsion
* **--dirtarget=**/path/dirpath/ or **-t** /path/dirpath/
                                                        

{'fraction_non_alphanumeric': 0.08620065193770374, 'fraction_numerical': 0.00398406374501992, 'mean_word_length': 0.9423347398030942, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 34, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '998002', 'n_tokens_mistral': 607, 'n_tokens_neox': 551, 'n_words': 216}