<commit_before>
Go-Guerrilla SMTPd
====================

An minimalist SMTP server written in Go, made for receiving large volumes of mail.

![Go Guerrilla](https://raw.github.com/flashmob/go-guerrilla/master/GoGuerrilla.png)

### What is Go Guerrilla SMTPd?

It's a small SMTP server written in Go, for the purpose of receiving large volume of email.
Written for GuerrillaMail.com which processes tens of thousands of emails
every hour.

The purpose of this daemon is to grab the email, save it to the database
and disconnect as quickly as possible.

A typical user of this software would probably want to customize the saveMail function for
their own systems.

This server does not attempt to filter HTML, check for spam or do any sender 
verification. These steps should be performed by other programs.
The server does NOT send any email including bounces. This should
be performed by a separate program.

The software is using MIT License (MIT) - contributors welcome.

### Roadmap / Contributing & Bounties

Wow, this project did not expect to get so many stars! 
However, not that much pull requests...
To encourage more pull requests, we are now offering bounties funded 
from our bitcoin donation address:

`1grr11aWtbsyMUeB4EGfHvTuu7eFzkJ4A`

So far we have the following bounties:

- Client Pooling: When a client is finished, it should be placed into a 
pool instead of being destroyed. Looking for a idiomatic 
Go solution with channels. (0.5 BTC for a successful merge)

- Modularize: Ability for the server to be used as a module. If it used
as a module, the new program would be able to start several servers on 
different ports, would be possible to specify a config file for 
each server, and specify its own saveMail function (otherwise, revert to
default). Ideas on how to best refactor this welcome too. 
(0.5 BTC for a successful merge)

- Analytics: A web based admin panel that displays live statistics,
including the number of clients, memory usage, graph the number of
connections/bytes/memory used for the last 24h. 
Show the top senders by: IP, by domain & by HELO message. 
Using websocket via https & password protected. 
(1 BTC for a successful merge)

If you need more clarification / details, please open an issue on Github.
Also, welcome your suggestions for adding things to this Roadmap - please open an issue.

Another way to contribute is to donate to our bitcoin address to help
us fund more bounties!
`1grr11aWtbsyMUeB4EGfHvTuu7eFzkJ4A`

### Brief History and purpose

Go-Guerrilla is used as the primary server for receiving email at 
Guerrilla Mail. As of 2016, it's handling all connections without any 
proxy (Nginx).

Originally, Guerrilla Mail ran Exim which piped email to a php script (2009). 
As as the site got popular and more email came through, this approach
eventually swamped the server.

The next solution was to decrease the heavy setup into something more 
lightweight. A small script was written to implement a basic SMTP server (2010).
Eventually that script also got swamped, so it was re-written to use
event driven I/O (2012). A year later, the latest script also became inadequate
 so it was ported to Go and has served us well since.
 

Getting started
===========================

To build, just run

$ go build

Rename goguerrilla.conf.sample to goguerrilla.conf

By default, the saveMail() function saves the meta-data of an email
into MySQL while the body is saved in Redis.

If you want to use the default saveMail() function, setup the following table
in MySQL:

	CREATE TABLE IF NOT EXISTS `new_mail` (
	  `mail_id` int(11) NOT NULL auto_increment,
	  `date` datetime NOT NULL,
	  `from` varchar(128) character set latin1 NOT NULL,
	  `to` varchar(128) character set latin1 NOT NULL,
	  `subject` varchar(255) NOT NULL,
	  `body` text NOT NULL,
	  `charset` varchar(32) character set latin1 NOT NULL,
	  `mail` longblob NOT NULL,
	  `spam_score` float NOT NULL,
	  `hash` char(32) character set latin1 NOT NULL,
	  `content_type` varchar(64) character set latin1 NOT NULL,
	  `recipient` varchar(128) character set latin1 NOT NULL,
	  `has_attach` int(11) NOT NULL,
	  `ip_addr` varchar(15) NOT NULL,
	  `delivered` bit(1) NOT NULL default b'0',
	  `attach_info` text NOT NULL,
	  `dkim_valid` tinyint(4) default NULL,
	  PRIMARY KEY  (`mail_id`),
	  KEY `to` (`to`),
	  KEY `hash` (`hash`),
	  KEY `date` (`date`)
	) ENGINE=InnoDB  DEFAULT CHARSET=utf8

The above table does not store the body of the email which makes it quick
to query and join, while the body of the email is fetched from Redis 
if needed.

You can implement your own saveMail function to use whatever storage /
backend fits for you.


Configuration
============================================
The configuration is in strict JSON format. Here is an annotated configuration.
Copy goguerrilla.conf.sample to goguerrilla.conf


{
    "allowed_hosts": "guerrillamail.com,guerrillamailblock.com,sharklasers.com,guerrillamail.net,guerrillamail.org" // What hosts to accept mail for
    "primary_mail_host":"sharklasers.com", // main domain
    "verbose":false, // report all events to log
    "mysql_db":"gmail_mail", // name of mysql database
    "mysql_host":"127.0.0.1:3306", // mysql host and port (tcp)
    "mysql_pass":"ok", // mysql password
    "mysql_user":"gmail_mail", // mysql username
    "mail_table":"new_mail", // mysql save table. Email meta-data is saved there
    "redis_interface" : "127.0.0.1:6379", // redis host and port, email data payload is saved there
	"redis_expire_seconds" : 3600, // how long to keep in redis
	"save_workers_size" : 3, // number workers saving email from all servers
	"pid_file" : "/var/run/go-guerrilla.pid", // pid = process id, so that other programs can send signals to our server
    	"servers" : [ // the following is an array of objects, each object represents a new server that will be spawned
    	    {
                "is_enabled" : true, // boolean
                "host_name":"mail.test.com", // the hostname of the server as set by MX record
                "max_size": 1000000, // maximum size of an email in bytes
                "private_key_file":"/path/to/pem/file/test.com.key",  // full path to pem file private key
                "public_key_file":"/path/to/pem/file/test.com.crt", // full path to pem file certificate
                "timeout":180, // timeout in number of seconds before an idle connection is closed
                "listen_interface":"127.0.0.1:25", // listen on ip and port
                "start_tls_on":true, // supports the STARTTLS command?
                "tls_always_on":false, // always connect using TLS? If true, start_tls_on will be false
                "max_clients": 1000, // max clients at one time
                "log_file":"/dev/stdout" // where to log to
    	    },
    	    // the following is a second server, but listening on port 465 and always using TLS
    	    {
                "is_enabled" : true,
                "host_name":"mail.test.com",
                "max_size":1000000,
                "private_key_file":"/path/to/pem/file/test.com.key",
                "public_key_file":"/path/to/pem/file/test.com.crt",
                "timeout":180,
                "listen_interface":"127.0.0.1:465",
                "start_tls_on":false,
                "tls_always_on":true,
                "max_clients":500,
                "log_file":"/dev/stdout"
            }
            // repeat as many servers as you need
    	]
}

	

Releases
=========================================================

1.4 - 25th Oct 2016
- New Feature: multiple servers!
- Changed the configuration file format to support multiple servers,
this means that a new configuration file would need to be created form the
sample (goguerrilla.conf.sample)
- Organised code into separate files. Config is now strongly typed, etc
- Deprecated nginx proxy support


1.3 14th July 2016
- Number of saveMail workers added to config (GM_SAVE_WORKERS) 
- convenience function for reading int values form config
- advertise PIPELINING
- added HELP command
- rcpt to host validation: now case insensitive and done earlier (after DATA)
- iconv switched to: go get gopkg.in/iconv.v1

1.2 1st July 2016
- Reload config on SIGHUP
- Write current process id (pid) to a file, /var/run/go-guerrilla.pid by default


Using Nginx as a proxy<commit_msg>README formatting<commit_after>
Go-Guerrilla SMTPd
====================

An minimalist SMTP server written in Go, made for receiving large volumes of mail.

![Go Guerrilla](https://raw.github.com/flashmob/go-guerrilla/master/GoGuerrilla.png)

### What is Go Guerrilla SMTPd?

It's a small SMTP server written in Go, for the purpose of receiving large volume of email.
Written for GuerrillaMail.com which processes tens of thousands of emails
every hour.

The purpose of this daemon is to grab the email, save it to the database
and disconnect as quickly as possible.

A typical user of this software would probably want to customize the saveMail function for
their own systems.

This server does not attempt to filter HTML, check for spam or do any sender 
verification. These steps should be performed by other programs.
The server does NOT send any email including bounces. This should
be performed by a separate program.

The software is using MIT License (MIT) - contributors welcome.

### Roadmap / Contributing & Bounties

Wow, this project did not expect to get so many stars! 
However, not that much pull requests...
To encourage more pull requests, we are now offering bounties funded 
from our bitcoin donation address:

`1grr11aWtbsyMUeB4EGfHvTuu7eFzkJ4A`

So far we have the following bounties:

- Client Pooling: When a client is finished, it should be placed into a 
pool instead of being destroyed. Looking for a idiomatic 
Go solution with channels. (0.5 BTC for a successful merge)

- Modularize: Ability for the server to be used as a module. If it used
as a module, the new program would be able to start several servers on 
different ports, would be possible to specify a config file for 
each server, and specify its own saveMail function (otherwise, revert to
default). Ideas on how to best refactor this welcome too. 
(0.5 BTC for a successful merge)

- Analytics: A web based admin panel that displays live statistics,
including the number of clients, memory usage, graph the number of
connections/bytes/memory used for the last 24h. 
Show the top senders by: IP, by domain & by HELO message. 
Using websocket via https & password protected. 
(1 BTC for a successful merge)

If you need more clarification / details, please open an issue on Github.
Also, welcome your suggestions for adding things to this Roadmap - please open an issue.

Another way to contribute is to donate to our bitcoin address to help
us fund more bounties!
`1grr11aWtbsyMUeB4EGfHvTuu7eFzkJ4A`

### Brief History and purpose

Go-Guerrilla is used as the primary server for receiving email at 
Guerrilla Mail. As of 2016, it's handling all connections without any 
proxy (Nginx).

Originally, Guerrilla Mail ran Exim which piped email to a php script (2009). 
As as the site got popular and more email came through, this approach
eventually swamped the server.

The next solution was to decrease the heavy setup into something more 
lightweight. A small script was written to implement a basic SMTP server (2010).
Eventually that script also got swamped, so it was re-written to use
event driven I/O (2012). A year later, the latest script also became inadequate
 so it was ported to Go and has served us well since.
 

Getting started
===========================

To build, just run

$ go build

Rename goguerrilla.conf.sample to goguerrilla.conf

By default, the saveMail() function saves the meta-data of an email
into MySQL while the body is saved in Redis.

If you want to use the default saveMail() function, setup the following table
in MySQL:

	CREATE TABLE IF NOT EXISTS `new_mail` (
	  `mail_id` int(11) NOT NULL auto_increment,
	  `date` datetime NOT NULL,
	  `from` varchar(128) character set latin1 NOT NULL,
	  `to` varchar(128) character set latin1 NOT NULL,
	  `subject` varchar(255) NOT NULL,
	  `body` text NOT NULL,
	  `charset` varchar(32) character set latin1 NOT NULL,
	  `mail` longblob NOT NULL,
	  `spam_score` float NOT NULL,
	  `hash` char(32) character set latin1 NOT NULL,
	  `content_type` varchar(64) character set latin1 NOT NULL,
	  `recipient` varchar(128) character set latin1 NOT NULL,
	  `has_attach` int(11) NOT NULL,
	  `ip_addr` varchar(15) NOT NULL,
	  `delivered` bit(1) NOT NULL default b'0',
	  `attach_info` text NOT NULL,
	  `dkim_valid` tinyint(4) default NULL,
	  PRIMARY KEY  (`mail_id`),
	  KEY `to` (`to`),
	  KEY `hash` (`hash`),
	  KEY `date` (`date`)
	) ENGINE=InnoDB  DEFAULT CHARSET=utf8

The above table does not store the body of the email which makes it quick
to query and join, while the body of the email is fetched from Redis 
if needed.

You can implement your own saveMail function to use whatever storage /
backend fits for you.


Configuration
============================================
The configuration is in strict JSON format. Here is an annotated configuration.
Copy goguerrilla.conf.sample to goguerrilla.conf


    {
        "allowed_hosts": "guerrillamail.com,guerrillamailblock.com,sharklasers.com,guerrillamail.net,guerrillamail.org" // What hosts to accept 
        "primary_mail_host":"sharklasers.com", // main domain
        "verbose":false, // report all events to log
        "mysql_db":"gmail_mail", // name of mysql database
        "mysql_host":"127.0.0.1:3306", // mysql host and port (tcp)
        "mysql_pass":"ok", // mysql password
        "mysql_user":"gmail_mail", // mysql username
        "mail_table":"new_mail", // mysql save table. Email meta-data is saved there
        "redis_interface" : "127.0.0.1:6379", // redis host and port, email data payload is saved there
        "redis_expire_seconds" : 3600, // how long to keep in redis
        "save_workers_size" : 3, // number workers saving email from all servers
        "pid_file" : "/var/run/go-guerrilla.pid", // pid = process id, so that other programs can send signals to our server
            "servers" : [ // the following is an array of objects, each object represents a new server that will be spawned
                {
                    "is_enabled" : true, // boolean
                    "host_name":"mail.test.com", // the hostname of the server as set by MX record
                    "max_size": 1000000, // maximum size of an email in bytes
                    "private_key_file":"/path/to/pem/file/test.com.key",  // full path to pem file private key
                    "public_key_file":"/path/to/pem/file/test.com.crt", // full path to pem file certificate
                    "timeout":180, // timeout in number of seconds before an idle connection is closed
                    "listen_interface":"127.0.0.1:25", // listen on ip and port
                    "start_tls_on":true, // supports the STARTTLS command?
                    "tls_always_on":false, // always connect using TLS? If true, start_tls_on will be false
                    "max_clients": 1000, // max clients at one time
                    "log_file":"/dev/stdout" // where to log to
                },
                // the following is a second server, but listening on port 465 and always using TLS
                {
                    "is_enabled" : true,
                    "host_name":"mail.test.com",
                    "max_size":1000000,
                    "private_key_file":"/path/to/pem/file/test.com.key",
                    "public_key_file":"/path/to/pem/file/test.com.crt",
                    "timeout":180,
                    "listen_interface":"127.0.0.1:465",
                    "start_tls_on":false,
                    "tls_always_on":true,
                    "max_clients":500,
                    "log_file":"/dev/stdout"
                }
                // repeat as many servers as you need
            ]
    }

	

Releases
=========================================================

1.4 - 25th Oct 2016
- New Feature: multiple servers!
- Changed the configuration file format to support multiple servers,
this means that a new configuration file would need to be created form the
sample (goguerrilla.conf.sample)
- Organised code into separate files. Config is now strongly typed, etc
- Deprecated nginx proxy support


1.3 14th July 2016
- Number of saveMail workers added to config (GM_SAVE_WORKERS) 
- convenience function for reading int values form config
- advertise PIPELINING
- added HELP command
- rcpt to host validation: now case insensitive and done earlier (after DATA)
- iconv switched to: go get gopkg.in/iconv.v1

1.2 1st July 2016
- Reload config on SIGHUP
- Write current process id (pid) to a file, /var/run/go-guerrilla.pid by default


Using Nginx as a proxy
{'fraction_non_alphanumeric': 0.09092521430682826, 'fraction_numerical': 0.01950931126219332, 'mean_word_length': 3.152184585174276, 'pattern_counts': {'":': 56, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 8, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5035175', 'n_tokens_mistral': 5169, 'n_tokens_neox': 4721, 'n_words': 2112}