<commit_before>CIRCLean
========
![CIRCLean logo](https://www.circl.lu/assets/images/logos/circlean.png)
![Cleaner in action](http://www.circl.lu/assets/images/CIRCLean/CIRCLean.png)

How To
======

[Graphical how-to and pre-built image](http://circl.lu/projects/CIRCLean/).

To prepare the SD card on Windows, you can use [Win32DiskImager](http://sourceforge.net/projects/win32diskimager/).

And the linux way is in the command line, via dd (see in copy_to_final.sh)

If you'd like to contribute to the project or build the image yourself, see
[contributing](CONTRIBUTING.md) and the [setup readme](README_setup.md).

Why/What
========

This project aims to be useful when you get/find a USB key that you can't trust,
and you want to have a look at its contents without taking the risk of plugging it into your
main computer directly.

This is a work in progress - contributions are welcome:

The content of the first key will be copied or/and converted to the second key
following these rules (based on the mime type, as determined by libmagic):
- Direct copy of:
  - Plain text files (mime type: text/*)
  - Audio files (mime type: audio/*)
  - Video files (mime type: video/*)
  - Example files (mime type: example/*)
  - Multipart files (mime type: multipart/*)
  - *xml* files, after being converted to text files
  - Octet-stream files
- Copied after verification:
  - Image files after verifying that they are not compression bombs (mime type: image/*)
  - PDF files, after marking as dangerous if they contain malicious content
  - msword|vnd.openxmlformats-officedocument.*|vnd.ms-*|vnd.oasis.opendocument*, after
    parsing with oletools/olefile and marking as dangerous if the parsing fails.
- Copied but marked as dangerous (DANGEROUS_filename_DANGEROUS)
  - Message files (mime type: message/*)
  - Model files (mime type: model/*)
  - x-dosexec (executable)
- Compressed files (zip|x-rar|x-bzip2|x-lzip|x-lzma|x-lzop|x-xz|x-compress|x-gzip|x-tar|*compressed):<commit_msg>Readme update with filesystem info<commit_after>CIRCLean
========
![CIRCLean logo](https://www.circl.lu/assets/images/logos/circlean.png)
![Cleaner in action](http://www.circl.lu/assets/images/CIRCLean/CIRCLean.png)

How To
======

[Graphical how-to and pre-built image](http://circl.lu/projects/CIRCLean/).

To prepare the SD card on Windows, you can use [Win32DiskImager](http://sourceforge.net/projects/win32diskimager/).

And the linux way is in the command line, via dd (see in copy_to_final.sh)

If you'd like to contribute to the project or build the image yourself, see
[contributing](CONTRIBUTING.md) and the [setup readme](README_setup.md).
This is a work in progress - contributions are welcome.

Why/What
========

This project aims to be useful when you get/find a USB key that you can't trust,
and you want to look at its contents without taking the risk of plugging it into
your computer directly.

CIRCLean is currently tested to work with USB keys that have FAT32, NTFS, or
ext2/3/4 filesystems. Currently, exFAT is not supported due to lack of support for
this format in pmount. The vast majority of USB keys will be FAT32 or NTFS.

The content of the untrusted key will be copied or/and converted to the second
(blank) key following these rules (based on the mime type as determined bylibmagic):
- Direct copy of:
  - Plain text files (mime type: text/*)
  - Audio files (mime type: audio/*)
  - Video files (mime type: video/*)
  - Example files (mime type: example/*)
  - Multipart files (mime type: multipart/*)
  - *xml* files, after being converted to text files
  - Octet-stream files
- Copied after verification:
  - Image files after verifying that they are not compression bombs (mime type: image/*)
  - PDF files, after marking as dangerous if they contain malicious content
  - msword|vnd.openxmlformats-officedocument.*|vnd.ms-*|vnd.oasis.opendocument*, after
    parsing with oletools/olefile and marking as dangerous if the parsing fails.
- Copied but marked as dangerous (DANGEROUS_filename_DANGEROUS)
  - Message files (mime type: message/*)
  - Model files (mime type: model/*)
  - x-dosexec (executable)
- Compressed files (zip|x-rar|x-bzip2|x-lzip|x-lzma|x-lzop|x-xz|x-compress|x-gzip|x-tar|*compressed):
{'fraction_non_alphanumeric': 0.10569874674864034, 'fraction_numerical': 0.0040198628517379995, 'mean_word_length': 4.235148514851486, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5572532', 'n_tokens_mistral': 1394, 'n_tokens_neox': 1311, 'n_words': 506}