<commit_before>[![Build Status](https://travis-ci.org/azumakuniyuki/p5-Sisimai.svg?branch=master)](https://travis-ci.org/azumakuniyuki/p5-Sisimai) 
[![Coverage Status](https://img.shields.io/coveralls/azumakuniyuki/p5-Sisimai.svg)](https://coveralls.io/r/azumakuniyuki/p5-Sisimai)

         ____  _     _                 _ 
        / ___|(_)___(_)_ __ ___   __ _(_)
        \___ \| / __| | '_ ` _ \ / _` | |
         ___) | \__ \ | | | | | | (_| | |
        |____/|_|___/_|_| |_| |_|\__,_|_|
                                 

What is Sisimai ? | シシマイ?
=============================

Sisimai is the system formerly known as bounceHammer 4, is a Perl module for 
analyzing bounce emails and output parsed results as a JSON format. "Sisimai"
is a coined word: Sisi (the number 4 is pronounced "Si" in Japanese) and MAI
(acronym of "Mail Analyzing Interface").

"シシマイ"はbounceHammer version 4として開発していたエラーメール解析モジュール
で、解析結果をJSONで出力します。Version 4なので"シ"から始まりマイ(MAI: Mail 
Analyzing Interface)を含む名前になりました。

System requirements | 動作環境
------------------------------

* [Perl 5.10.1 or later](http://www.perl.org/)

Dependencies | 依存モジュール
-----------------------------
Sisimai relies on:

* [__Class::Accessor::Lite__](https://metacpan.org/pod/Class::Accessor::Lite)
* [__JSON__](https://metacpan.org/pod/JSON)

Sisimaiは上記のモジュールに依存しています。

Install | インストール
----------------------

    % sudo cpanm Sisimai
    --> Working on Sisimai
    Fetching http://www.cpan.org/authors/id/A/AK/AKXLIX/Sisimai-4.1.20.tar.gz ... OK
    ...
    1 distribution installed
    % perldoc -l Sisimai
    /usr/local/lib/perl5/site_perl/5.20.0/Sisimai.pm

OR
    
    % cd /usr/local/src
    % git clone https://github.com/azumakuniyuki/p5-Sisimai.git
    % cd ./Sisimai
    % sudo cpanm .
    --> Working on .
    Configuring Sisimai-4.1.20 ... OK
    1 distribution installed


Basic usage | 基本的な使い方
----------------------------
make() method provides feature for getting parsed data from bounced email 
messages like following.

```perl
#! /usr/bin/env perl
use Sisimai;
my $v = Sisimai->make( '/path/to/mbox' );   # or Path to Maildir

if( defined $v ) {
    for my $e ( @$v ) {
        print ref $e;                   # Sisimai::Data
        print ref $e->recipient;        # Sisimai::Address
        print ref $e->timestamp;        # Sisimai::Time

        print $e->addresser->address;   # <EMAIL> # From
        print $e->recipient->address;   # <EMAIL>   # To
        print $e->recipient->host;      # example.jp
        print $e->deliverystatus;       # 5.1.1
        print $e->reason;               # userunknown

        my $h = $e->damn();             # Convert to HASH reference
        my $j = $e->dump('json');       # Convert to JSON string
        print $e->dump('json');         # JSON formatted bounce data
    }

    # OR
    use JSON '-convert_blessed_universally';
    my $json = JSON->new->allow_blessed->convert_blessed;

    printf "%s\n", $json->encode( $v );
}
```

上記のようにSisimaiのmake()メソッドをmboxかMaildirのPATHを引数にして実行すると
解析結果が配列リファレンスで返ってきます。


Differences between ver.2 and ver.4 | 新旧の違い
------------------------------------------------
The followings are the differences between version 2 (bounceHammer 2.7.13) and
version 4 (Sisimai).

| Features                                       | bounceHammer  | Sisimai     |
|------------------------------------------------|---------------|-------------|
| Command line tools                             | OK            | N/A         |
| Modules for Commercial MTAs                    | N/A           | Included    |
| WebUI/API                                      | OK            | N/A         |
| Database schema for storing parsed bounce data | Available     | N/A(1)      |
| Analysis accuracy ratio(2)                     | 0.64          | 1.00        |
| Parse 2 or more bounces in a single email      | Only 1st rcpt | ALL         |
| Parse FeedBack Loop Message/ARF format mail    | N/A           | OK          |
| Classification based on recipient domain       | Available     | N/A         |
| Output format of parsed data                   | YAML,JSON,CSV | JSON only(3)|
| The speed of parsing email(1484 files)         | 6.15s         | 3.16s       |
| Easy to install                                | No            | Yes         |
| Install using cpan or cpanm command            | N/A           | OK          |
| Dependencies                                   | 24 modules    | 2 modules   |
| LOC:Source lines of code                       | 18200 lines   | 7500 lines  |
| The number of tests in t/ directory            | 27365 tests   | 70500 tests |
| License                                        | GPLv2 or Perl | 2 clause BSD|
| Support Contract provided by Developer         | Available     | Coming soon |

1. Implement yourself with using DBI or any O/R Mapper you like
2. See ./ANALYSIS-ACCURACY
3. YAML format is available if "YAML" module has been installed

公開中のbouncehammer version 2.7.13とversion 4(シシマイ)は下記のような違いがあります。

| 機能                                           | bounceHammer  | Sisimai     |
|------------------------------------------------|---------------|-------------|
| コマンドラインツール                           | あり          | 無し        |
| 商用MTA対応解析モジュール                      | 無し          | あり(標準)  |
| WebUIとAPI                                     | あり          | 無し        |
| 解析済バウンスデータを保存するDBスキーマ       | あり          | 無し(1)     |
| 解析精度の割合(2)                              | 0.64          | 1.00        |
| 2件以上のバウンスがあるメールの解析            | 1件目だけ     | 全件対応    |<commit_msg>Add dump data as a JSON<commit_after>[![Build Status](https://travis-ci.org/azumakuniyuki/p5-Sisimai.svg?branch=master)](https://travis-ci.org/azumakuniyuki/p5-Sisimai) 
[![Coverage Status](https://img.shields.io/coveralls/azumakuniyuki/p5-Sisimai.svg)](https://coveralls.io/r/azumakuniyuki/p5-Sisimai)

         ____  _     _                 _ 
        / ___|(_)___(_)_ __ ___   __ _(_)
        \___ \| / __| | '_ ` _ \ / _` | |
         ___) | \__ \ | | | | | | (_| | |
        |____/|_|___/_|_| |_| |_|\__,_|_|
                                 

What is Sisimai ? | シシマイ?
=============================

Sisimai is the system formerly known as bounceHammer 4, is a Perl module for 
analyzing bounce emails and output parsed results as a JSON format. "Sisimai"
is a coined word: Sisi (the number 4 is pronounced "Si" in Japanese) and MAI
(acronym of "Mail Analyzing Interface").

"シシマイ"はbounceHammer version 4として開発していたエラーメール解析モジュール
で、解析結果をJSONで出力します。Version 4なので"シ"から始まりマイ(MAI: Mail 
Analyzing Interface)を含む名前になりました。

System requirements | 動作環境
------------------------------

* [Perl 5.10.1 or later](http://www.perl.org/)

Dependencies | 依存モジュール
-----------------------------
Sisimai relies on:

* [__Class::Accessor::Lite__](https://metacpan.org/pod/Class::Accessor::Lite)
* [__JSON__](https://metacpan.org/pod/JSON)

Sisimaiは上記のモジュールに依存しています。

Install | インストール
----------------------

    % sudo cpanm Sisimai
    --> Working on Sisimai
    Fetching http://www.cpan.org/authors/id/A/AK/AKXLIX/Sisimai-4.1.20.tar.gz ... OK
    ...
    1 distribution installed
    % perldoc -l Sisimai
    /usr/local/lib/perl5/site_perl/5.20.0/Sisimai.pm

OR
    
    % cd /usr/local/src
    % git clone https://github.com/azumakuniyuki/p5-Sisimai.git
    % cd ./Sisimai
    % sudo cpanm .
    --> Working on .
    Configuring Sisimai-4.1.20 ... OK
    1 distribution installed


Basic usage | 基本的な使い方
----------------------------
make() method provides feature for getting parsed data from bounced email 
messages like following.

```perl
#! /usr/bin/env perl
use Sisimai;
my $v = Sisimai->make( '/path/to/mbox' );   # or Path to Maildir

if( defined $v ) {
    for my $e ( @$v ) {
        print ref $e;                   # Sisimai::Data
        print ref $e->recipient;        # Sisimai::Address
        print ref $e->timestamp;        # Sisimai::Time

        print $e->addresser->address;   # <EMAIL> # From
        print $e->recipient->address;   # <EMAIL>   # To
        print $e->recipient->host;      # example.jp
        print $e->deliverystatus;       # 5.1.1
        print $e->reason;               # userunknown

        my $h = $e->damn();             # Convert to HASH reference
        my $j = $e->dump('json');       # Convert to JSON string
        print $e->dump('json');         # JSON formatted bounce data
    }

    # OR
    use JSON '-convert_blessed_universally';
    my $json = JSON->new->allow_blessed->convert_blessed;

    printf "%s\n", $json->encode( $v );
}
```

```json
[{"recipient": "<EMAIL>", "addresser": "<EMAIL>", "feedbacktype": "", "action": "failed", "subject": "Nyaaaaan", "smtpcommand": "DATA", "diagnosticcode": "550 Unknown user <EMAIL>", "listid": "", "destination": "example.jp", "smtpagent": "Courier", "lhost": "1jo.example.org", "deliverystatus": "5.0.0", "timestamp": 1291954879, "messageid": "201012100421.<EMAIL>", "diagnostictype": "SMTP", "timezoneoffset": "+0900", "reason": "filtered", "token": "ce999a4c869e3f5e4d8a77b2e310b23960fb32ab", "alias": "", "senderdomain": "1jo.example.org", "rhost": "mfsmax.example.jp"}, {"diagnostictype": "SMTP", "timezoneoffset": "+0900", "reason": "userunknown", "timestamp": 1381900535, "messageid": "E1C50F1B-1C83-4820-BC36-AC6<EMAIL>", "token": "9fe754876e9133aae5d20f0fd8dd7f05b4e9d9f0", "alias": "", "senderdomain": "example.org", "rhost": "mx.bouncehammer.jp", "action": "failed", "addresser": "<EMAIL>", "recipient": "<EMAIL>", "feedbacktype": "", "smtpcommand": "DATA", "subject": "バウンスメールのテスト(日本語)", "destination": "bouncehammer.jp", "listid": "", "diagnosticcode": "550 5.1.1 <<EMAIL>>... User Unknown", "deliverystatus": "5.1.1", "lhost": "p0000-ipbfpfx00kyoto.kyoto.example.co.jp", "smtpagent": "Sendmail"}]
```

上記のようにSisimaiのmake()メソッドをmboxかMaildirのPATHを引数にして実行すると
解析結果が配列リファレンスで返ってきます。


Differences between ver.2 and ver.4 | 新旧の違い
------------------------------------------------
The followings are the differences between version 2 (bounceHammer 2.7.13) and
Sisimai.

| Features                                       | bounceHammer  | Sisimai     |
|------------------------------------------------|---------------|-------------|
| Command line tools                             | OK            | N/A         |
| Modules for Commercial MTAs and MPSs           | N/A           | Included    |
| WebUI/API                                      | OK            | N/A         |
| Database schema for storing parsed bounce data | Available     | N/A(1)      |
| Analysis accuracy ratio(2)                     | 0.64          | 1.00        |
| Parse 2 or more bounces in a single email      | Only 1st rcpt | ALL         |
| Parse FeedBack Loop Message/ARF format mail    | N/A           | OK          |
| Classification based on recipient domain       | Available     | N/A         |
| Output format of parsed data                   | YAML,JSON,CSV | JSON only(3)|
| The speed of parsing email(1484 files)         | 6.15s         | 3.16s       |
| Easy to install                                | No            | Yes         |
| Install using cpan or cpanm command            | N/A           | OK          |
| Dependencies                                   | 24 modules    | 2 modules   |
| LOC:Source lines of code                       | 18200 lines   | 7500 lines  |
| The number of tests in t/ directory            | 27365 tests   | 70500 tests |
| License                                        | GPLv2 or Perl | 2 clause BSD|
| Support Contract provided by Developer         | Available     | Coming soon |

1. Implement yourself with using DBI or any O/R Mapper you like
2. See ./ANALYSIS-ACCURACY
3. YAML format is available if "YAML" module has been installed

公開中のbouncehammer version 2.7.13とSisimai(シシマイ)は下記のような違いがあります。

| 機能                                           | bounceHammer  | Sisimai     |
|------------------------------------------------|---------------|-------------|
| コマンドラインツール                           | あり          | 無し        |
| 商用MTAとMSP対応解析モジュール                 | 無し          | あり(標準)  |
| WebUIとAPI                                     | あり          | 無し        |
| 解析済バウンスデータを保存するDBスキーマ       | あり          | 無し(1)     |
| 解析精度の割合(2)                              | 0.64          | 1.00        |
| 2件以上のバウンスがあるメールの解析            | 1件目だけ     | 全件対応    |
{'fraction_non_alphanumeric': 0.17265428594598978, 'fraction_numerical': 0.026437434109155788, 'mean_word_length': 1.6537551108241877, 'pattern_counts': {'":': 42, '<': 16, '<?xml version=': 0, '>': 56, 'https://': 14, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 41, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6899449', 'n_tokens_mistral': 4540, 'n_tokens_neox': 4153, 'n_words': 910}