langchain.document_transformers.beautiful_soup_transformer.BeautifulSoupTransformer¶
class langchain.document_transformers.beautiful_soup_transformer.BeautifulSoupTransformer[source]¶
Transform HTML content by extracting specific tags and removing unwanted ones.
Example
Initialize the transformer.
This checks if the BeautifulSoup4 package is installed.
If not, it raises an ImportError.
Methods
__init__()
Initialize the transformer.
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
extract_tags(html_content, tags)
Extract specific tags from a given HTML content.
remove_unnecessary_lines(content)
Clean up the content by removing unnecessary lines.
remove_unwanted_tags(html_content, unwanted_tags)
Remove unwanted tags from a given HTML content.
transform_documents(documents[, ...])
Transform a list of Document objects by cleaning their HTML content.
__init__() → None[source]¶
Initialize the transformer.
This checks if the BeautifulSoup4 package is installed.
If not, it raises an ImportError.
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.
static extract_tags(html_content: str, tags: List[str]) → str[source]¶
Extract specific tags from a given HTML content.
Parameters
html_content – The original HTML content string.
tags – A list of tags to be extracted from the HTML.
Returns
A string combining the content of the extracted tags.
static remove_unnecessary_lines(content: str) → str[source]¶
Clean up the content by removing unnecessary lines.
Parameters
content – A string, which may contain unnecessary lines or spaces.
Returns
A cleaned string with unnecessary lines removed.