license:apache-2.0 region:us
This benchmark environment contains a dataset comprised of human-spoken text and Large Language Models (LLM) generated spoken text.
We also have three benchmark tasks - AA (multi-class classification problem on human datasets), AV (binary classification problem on whether two spoken texts are from same human),
and TT (Turing test problem, determining human vs AI spoken texts problem).
@InProceedings{huggingface:dataset,
title = {A great new dataset},
author={huggingface, Inc.
},
year={2023}
}
---
license: apache-2.0
---

# HANSEN

Human and AI Spoken Text Benchmark for Authorship Analysis.

**We are updating the HANSEN to the following specific format **

The various portions of the 
(1) open-source data/existing datasets that we are free to re-distribute (All AA and AV datasets except for FTN and CEO)
(2) open-source data that we may not freely re-distribute but users have to download/scrape themselves (AA and AV datasets for FTN and CEO due to redistribution issue)
and (3) AI-generated data that we have generated (TT datasets that can be accessible after submitting the form https://forms.gle/WZt7KrxTcmfPXuho9 and accept to the terms 
of good usage of the datasets.

## Description

HANSEN comprises 17 human "spoken-text" datasets. It also contains spoken texts generated from three LLMs: ChatGPT, PaLM2, and Vicuna13B.
Spoken text is the text/transcript version of what people say, such as speech, conversation, interviews.
HANSEN can be used for different authorship analysis tasks.
Currently three tasks are defined.

1. AA (Author Attribution): A multi-class classification problem.  Given a spoken text T, identifies the speaker from a list of candidate speakers. 
2. AV (Author Attribution): A binary classification problem.  Given a pair of spoken texts (T1, T2), detects whether they were generated by the same speakers or different speakers.
3. TT (Turing Test/Human vs AI text detection): A binary classification problem. Given a spoken text T, identifies whether the speaker is a human or an LLM. 

## AA Task

Currently there are 17 human datasets. Each dataset has two version: small (number of speaker, N=10) and large (number of speaker, N=100 in most cases, for USP, SEC N=30, for TED N=50. for PAN N=56).
So, AA_TED_small will be loading the dataframes of 10 class classification problem in TED dataset.

The dataframes have two columns: author_id (0 to N-1) and text. The list of datasets are as follows.

Dataset  | Description
------------- | -------------
TED  | TED talks
Spotify  | Spotify podcasts
BASE    | British Academic Spoken English (BASE) corpus (Nesi and Thompson, 2003)
BNC | British National Corpus
BNC14 | Contemporary version of BNC
MSU | MSU Switchboard Dialogue Act (Telephone conversation)
PAN | Spoken portion of PAN'23 AV datasets
Tennis  | Post-match Interview of Tennis players
CEO     | CEO and other financial interviews
Voxceleb    | Interview of YouTube celebrities
BP  | British Parliament Question and Answers
Voxpopuli   | European Parliament Events recording
FTN | Face the Nation tv program transcripts
USP     | US Life Podcast radio program transcripts
SEC     | Security Exchange Commission speeches
Debate  |  Debates held as part of Intelligence Squared Debates
Court   | U.S. Supreme Court oral arguments transcripts

For the CEO and FTN datasets, they do not contain the original text due to the redistribution issue. We have added url and line number (in the text) for each sample in these datasets.
(The script to donwload the original text will be provided soon)

## AV Task

The dataframes have three columns: label (0 if different speaker, 1 if same speaker), text1, and text2. Dataset descriptsions are same as AA task.


## TT Task

Currently HANSEN has three LLMs in five categories (from human dataset settings: TED, Spotify, SEC, CEO, Tennis) spoken texts. 

LLM  | Description
------------- | -------------
ChatGPT  | gpt-3.5-turbo
PALM  | PaLM2 (chat-bison@001)
Vicuna13B   | Vicuna 13B version finetuned on Llama 13B

So, TT_ChatGPT_TED will be loading the dataframes from human (0) vs ChatGPT (1) dataset in TED category. 
The dataframes have two columns: label (0 for human, 1 for AI) and text. 
To access the HANSEN-TT dataset, please fill up the form and agree to the terms & conditions.
https://forms.gle/WZt7KrxTcmfPXuho9