Returns
A cleaned string with unnecessary lines removed.
static remove_unwanted_tags(html_content: str, unwanted_tags: List[str]) → str[source]¶
Remove unwanted tags from a given HTML content.
Parameters
html_content – The original HTML content string.
unwanted_tags – A list of tags to be removed from the HTML.
Returns
A cleaned HTML string with unwanted tags removed.
transform_documents(documents: Sequence[Document], unwanted_tags: List[str] = ['script', 'style'], tags_to_extract: List[str] = ['p', 'li', 'div', 'a'], remove_lines: bool = True, **kwargs: Any) → Sequence[Document][source]¶
Transform a list of Document objects by cleaning their HTML content.
Parameters
documents – A sequence of Document objects containing HTML content.
unwanted_tags – A list of tags to be removed from the HTML.
tags_to_extract – A list of tags whose content will be extracted.
remove_lines – If set to True, unnecessary lines will be
content. (removed from the HTML) – 
Returns
A sequence of Document objects with transformed content.
Examples using BeautifulSoupTransformer¶
Beautiful Soup
Set env var OPENAI_API_KEY or load from a .env file: