license:cc-by-sa-3.0 region:us
---
license: cc-by-sa-3.0
---
## Semantic search over the 44 million of English Wikipedia paragraphs using sentence transformers encoder. 

The dataset contains:
- 43 911 155 paragraphs from 6 458 670 wikipedia articles stored in a zip archive;
- FAISS index with the embeddings;
- Retriever module for semantic search over the paragraphs. 

The size of each paragraph varies from 20 to 2000 characters.   
The embedding vector size is 768.  
The index is 4-bit-quantized 2-level IVF16384_HNSW32 constructed with the [FAISS library](https://github.com/facebookresearch/faiss).  
Sentence encoder: [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2).