license:unknown region:us
Welcome to the SpamAssassin public mail corpus.  This is a selection of mail
messages, suitable for use in testing spam filtering systems.  Pertinent
points:

  - All headers are reproduced in full.  Some address obfuscation has taken
    place, and hostnames in some cases have been replaced with
    "spamassassin.taint.org" (which has a valid MX record).  In most cases
    though, the headers appear as they were received.

  - All of these messages were posted to public fora, were sent to me in the
    knowledge that they may be made public, were sent by me, or originated as
    newsletters from public news web sites.

  - relying on data from public networked blacklists like DNSBLs, Razor, DCC
    or Pyzor for identification of these messages is not recommended, as a
    previous downloader of this corpus might have reported them!

  - Copyright for the text in the messages remains with the original senders.


OK, now onto the corpus description.  It's split into three parts, as follows:

  - spam: 500 spam messages, all received from non-spam-trap sources.

  - easy_ham: 2500 non-spam messages.  These are typically quite easy to
    differentiate from spam, since they frequently do not contain any spammish
    signatures (like HTML etc).

  - hard_ham: 250 non-spam messages which are closer in many respects to
    typical spam: use of HTML, unusual HTML markup, coloured text,
    "spammish-sounding" phrases etc.

  - easy_ham_2: 1400 non-spam messages.  A more recent addition to the set.

  - spam_2: 1397 spam messages.  Again, more recent.

Total count: 6047 messages, with about a 31% spam ratio.
---
license: unknown
---

# Dataset Card for the SpamAssassin public mail corpus

## Dataset Description

- **Homepage:** https://spamassassin.apache.org/old/publiccorpus/readme.html

### Dataset Summary

This is a selection of mail messages, suitable for use in testing spam filtering systems assembled by members of the SpamAssassin project.

### Supported Tasks and Leaderboards

[More Information Needed]

### Languages

[More Information Needed]

## Dataset Structure

### Data Instances

 - The `text` config normalizes all character sets to utf8 and dumps the
   MIME tree as a JSON list of lists.
 - The `unprocessed` config does not parse messages at all, leaving the
   full headers and content as binary.

### Data Fields

 - `label`: `spam` or `ham`
 - `group`: SpamAssassin has grouped these samples into categories
   {'hard_ham', 'spam_2', 'spam', 'easy_ham', 'easy_ham_2'}
 - `text`: normalized text of the message bodies
 - `raw`: full binary headers and contents of messages

### Data Splits

Only a _train_ split has been provided.

## Dataset Creation

### Curation Rationale

It is hoped this dataset can help verify that modern NLP tools can solve
old NLP problems.

### Source Data

#### Initial Data Collection and Normalization

[The upstream corpus description](https://spamassassin.apache.org/old/publiccorpus/readme.html)
goes into detail on collection methods.  The work here to recover text bodies
is largely done with [email.parser](https://docs.python.org/3/library/email.parser.html)
and [ftfy](https://pypi.org/project/ftfy/).

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

[More Information Needed]

### Contributions

[More Information Needed]