size_categories:1M<n<10M license:mit deprecated region:us
---
license: mit
size_categories:
- 1M<n<10M
tags:
- deprecated
---

# DEPRECATION NOTICE
As of August 1, 2023, the OpenValidators Mining dataset has been officially deprecated and discontinued. We are no longer updating or maintaining this dataset.
If this data has any relevance to your current or future projects, or you have any questions or concerns related to the deprecation, please do not hesitate to contact us. We are more than willing to assist you and provide guidance for alternative solutions where possible.

---
# Dataset Card for Openvalidators mining dataset

## Dataset Description


- Repository: https://github.com/opentensor/validators

### Dataset Summary


The OpenValidators Mining dataset, developed by the OpenTensor Foundation, is an ever-expanding collection of carefully
curated data produced by the [OpenValidators](https://github.com/opentensor/validators) project within 
[W&B](https://wandb.ai/opentensor-dev/openvalidators/table). This dataset consists of hundreds of thousands of records
for miners aiming to fine-tune their models using unique data sourced from the bittensor network.
It offers unique pairs of (`base_prompt`, `best_followup`) and (`answer_prompt`, `best_answer`) extracted from the 
openvalidators logs, adhering to the [OpenAI fine-tuning schema](https://platform.openai.com/docs/guides/fine-tuning/prepare-training-data):
```json lines
{"prompt": "<base prompt>", "completion": "<best followup>"}
{"prompt": "<answer prompt>", "completion": "<best answer>"}
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
...
```

### How to use

The current available OpenAI fine-tuning dataset can be consumed by the `datasets` library or using `pandas` directly.

**Consuming the dataset using the `datasets` library**

To download the OpenAI fine-tuning data using the `datasets` library, one can use the following code snippet:

```python
from datasets import load_dataset

dataset = load_dataset('opentensor/openvalidators-mining', data_files='openai/openvalidators-openai.jsonl')
```

**Consuming the dataset using pandas**

To download the OpenAI fine-tuning data using `pandas`, one can use the following code snippet:

```python
import pandas as pd

path = 'hf://datasets/opentensor/openvalidators-mining/openai/openvalidators-openai.jsonl'
df = pd.read_json(path, lines=True)
```

## Dataset Structure

### Data Instances

**OpenAI fine-tuning dataset**

The unique pair of (prompt, response) collected from the openvalidators logs for the 
[OpenAI fine-tuning schema](https://platform.openai.com/docs/guides/fine-tuning/prepare-training-data):

### Data Fields

**OpenAI fine-tuning dataset**

The versioned raw_data collected from W&B follows the following schema:
- `prompt`: (string) Unique prompt from `base_prompt` or `answer_prompt` fields
- `completion`: (string) Unique prompt from `best_followup` or `best_answer` fields


## Dataset Creation

### Curation Rationale

The purpose of this database is to offer a collection of data ready to be consumed by the Bittensor mining community.
The idea was to create a clean and continuous database that aggregates the network's unique set of prompts and most rewarded answers.
We believe that empowering the miner with quality data can increase network efficiency and the quality of trained models.


### Source Data

#### Initial Data Collection and Normalization

The initial collection process for this dataset involves recurrent collection by a specialized worker responsible for 
extracting data from `openvalidators dataset` , cleaning and ingesting it into the Hugging Face `openvalidators-mining`
structure. The collected data is cleaned by ensuring that `prompt` and `completion` pairs are unique and not empty.

Please note that the code for this data collection process will be released for transparency and reproducibility.

#### Who are the source language producers?

The language producers for this dataset are the openvalidators in the bittensor network. The original data collected
in [wandb](https://wandb.ai/opentensor-dev/openvalidators/table) is processed into the Hugging Face [`openvalidators`](https://huggingface.co/datasets/opentensor/openvalidators) dataset structure that is used as main source for 
the mining datasets. 

### Licensing Information
The dataset is licensed under the [MIT License](https://github.com/opentensor/validators/blob/main/LICENSE)


### Supported Tasks and Leaderboards

[More Information Needed]

### Citation Information

[More Information Needed]

### Contributions

[More Information Needed]