task_categories:question-answering size_categories:10K<n<100K language:en license:apache-2.0 arxiv:2210.03350 region:us
---
license: apache-2.0
task_categories:
- question-answering
language:
- en
pretty_name: ehc-contrived-financial
size_categories:
- 10K<n<100K
---
# Everything Has Context | contrived company research example (ehc-contrived-financial)

### 📝 Description 

`train.csv` dataset contains 12,514 rows of high-quality contrived<sup>1</sup> research patterns in the public market equities category for Q/A pairs with a high perplexity<sup>2</sup>. 

The data is generated from `davinci-turbo` using the OpenAI API with prompts engineered to do several things which incite a grounded hallucinatory research example each call:

1. Generate one-shot Q/A example with a mask for the subject using the syntax `[Company]` which has a high perplexity thus requires multiple follow up questions (or the answer itself requires two sources of external context).
2. Between the question and answer of each one-shot example, hallucinate context from a search of equity filings data required to get to the answer.
3. Replace `[Company]` instances with a random company from a list in our case of 118 companies<sup>*</sup>
4. Filter on all rows for conditions which suit your needs (we choose higher perplexity which we define in a contrived dataset as: `∀(context,followup)∈S, where S is the dataset, and ∣{(context,followup)}∣>2`)

### 🙈 Contrived!
It's not real context. We are researching what this means for compositionality gaps in the respective domain for the model finetuning. There are perhaps more obvious limitations around the ability to reason on questions with high perplexity involved which the model has not been finetuned on, especially as reasoning about the question's context requirements could grow. Naively-posed questions, loaded questions, or questions of a contradictory manner may throw off the reasoning and context retrieval abilities of a finetuned model derived from a contrived 'environment', if you will. These are just some of the challenges which may be posed using a contrived set of Q/A context-driven dataset.

## 🧑‍💻 Other Datasets for Everything Has Context

1️⃣ <i>real world context:</i> not out yet but it's comin'. I have the context though I don't have the generations, give it a week max from this README commit's date.

2️⃣ <i>databricks-dolly-15k x real world context:</i> see 1

----
#### 💬 Citation

<sup>*</sup> <small>we do this after the work in 1, 2 because it removes the potential of sticky base model knowledge affecting the context and Q/A diversity! we do only 118 companies because the company names don't matter, facts in context do</small>

<sup>1</sup> <small>contrived is a term we use here to say there was a prompt engineered to create the data from a world-class model

<sup>2</sup> <small>@misc{press2023measuring,
      title={Measuring and Narrowing the Compositionality Gap in Language Models}, 
      author={Ofir Press and Muru Zhang and Sewon Min and Ludwig Schmidt and Noah A. Smith and Mike Lewis},
      year={2023},
      eprint={2210.03350},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}</small>