langchain.document_transformers.embeddings_redundant_filter.EmbeddingsClusteringFilter¶
class langchain.document_transformers.embeddings_redundant_filter.EmbeddingsClusteringFilter[source]¶
Bases: BaseDocumentTransformer, BaseModel
Perform K-means clustering on document vectors.
Returns an arbitrary number of documents closest to center.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param embeddings: langchain.schema.embeddings.Embeddings [Required]¶
Embeddings to use for embedding document contents.
param num_closest: int = 1¶
The number of closest vectors to return for each cluster center.
param num_clusters: int = 5¶
Number of clusters. Groups of documents with similar meaning.
param random_state: int = 42¶
Controls the random number generator used to initialize the cluster centroids.
If you set the random_state parameter to None, the KMeans algorithm will use a
random number generator that is seeded with the current time. This means
that the results of the KMeans algorithm will be different each time you
run it.
param remove_duplicates: bool = False¶
By default duplicated results are skipped and replaced by the next closest
vector in the cluster. If remove_duplicates is true no replacement will be done:
This could dramatically reduce results when there is a lot of overlap between
clusters.
param sorted: bool = False¶
By default results are re-ordered “grouping” them by cluster, if sorted is true
result will be ordered by the original position from the retriever
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.