langchain.document_transformers.embeddings_redundant_filter.EmbeddingsRedundantFilter¶
class langchain.document_transformers.embeddings_redundant_filter.EmbeddingsRedundantFilter[source]¶
Bases: BaseDocumentTransformer, BaseModel
Filter that drops redundant documents by comparing their embeddings.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param embeddings: langchain.schema.embeddings.Embeddings [Required]¶
Embeddings to use for embedding document contents.
param similarity_fn: Callable = <function cosine_similarity>¶
Similarity function for comparing documents. Function expected to take as input
two matrices (List[List[float]]) and return a matrix of scores where higher values
indicate greater similarity.
param similarity_threshold: float = 0.95¶
Threshold for determining when two documents are similar enough
to be considered redundant.
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶