license:mit region:us
---
license: mit
dataset_info:
  features:
  - name: id
    dtype: string
  - name: description
    dtype: string
  - name: embedding
    sequence: float64
  splits:
  - name: train
    num_bytes: 94758018
    num_examples: 29718
  download_size: 78891306
  dataset_size: 94758018
---

# Hugging Face Spaces Descriptions and Embeddings Dataset

I parsed all the available public 🤗 spaces as of May 22, 2023, generated concise descriptions of their functionality, and created embeddings for them.

The descriptions were generated using various LLMs from each space's app file (README.md -> app_file). The embeddings were created using the [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) SentenceTransformer model.

The dataset comprises approximately 30,000 spaces that meet specific criteria: having more than 40 lines of code and over 1000 characters in the app file.

The descriptions provide an overview of the spaces and their features.

## Dataset Details

- **Name**: HF Spaces Descriptions and Embeddings
- **Creator**: [anzorq](https://huggingface.co/anzorq)
- **License**: MIT

## Dataset Usage

You can use this dataset for various natural language processing (NLP) tasks such as semantic search, clustering, etc.

## Loading the Dataset
You can load the dataset using the datasets library:
```python
from datasets import load_dataset

# Load the dataset
dataset = load_dataset("anzorq/hf-spaces-descriptions-embeddings")

# Access the different splits
train_split = dataset['train']
valid_split = dataset['valid']
test_split = dataset['test']
```

## Semantic Search Example
Performing a semantic search using the dataset's embeddings:
```python
import torch
from sentence_transformers import SentenceTransformer
from datasets import load_dataset
import numpy as np

# Load the dataset
dataset = load_dataset("anzorq/hf-spaces-descriptions-embeddings")

# Load the SentenceTransformer model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Example query
query = "Removing background from images"

# Encode the query
query_embedding = model.encode([query], convert_to_tensor=True)

# Get the space descriptions and embeddings
descriptions = dataset['train']['description']
embeddings = np.array(dataset['train']['embedding'])

# Calculate cosine similarity
cosine_scores = torch.nn.functional.cosine_similarity(query_embedding, torch.tensor(embeddings))

# Sort the results
top_k = torch.topk(cosine_scores, k=5)

# Print the top-k results
print("Query:", query)
for idx in top_k.indices[0]:
    print("Space ID:", dataset['train']['id'][idx])
    print("Description:", descriptions[idx])
    print("Score:", cosine_scores[idx].item())
```

## License

This dataset is distributed under the [MIT License](https://opensource.org/licenses/MIT).

