langchain.document_loaders.news.NewsURLLoader¶
class langchain.document_loaders.news.NewsURLLoader(urls: List[str], text_mode: bool = True, nlp: bool = False, continue_on_failure: bool = True, show_progress_bar: bool = False, **newspaper_kwargs: Any)[source]¶
Load news articles from URLs using Unstructured.
Parameters
urls – URLs to load. Each is loaded into its own document.
text_mode – If True, extract text from URL and use that for page content.
Otherwise, extract raw HTML.
nlp – If True, perform NLP on the extracted contents, like providing a summary
and extracting keywords.
continue_on_failure – If True, continue loading documents even if
loading fails for a particular URL.
show_progress_bar – If True, use tqdm to show a loading progress bar. Requires
tqdm to be installed, pip install tqdm.
**newspaper_kwargs – Any additional named arguments to pass to
newspaper.Article().
Example
from langchain.document_loaders import NewsURLLoader
loader = NewsURLLoader(
    urls=["<url-1>", "<url-2>"],
)
docs = loader.load()
Newspaper reference:https://newspaper.readthedocs.io/en/latest/
Initialize with file path.
Methods
__init__(urls[, text_mode, nlp, ...])
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(urls: List[str], text_mode: bool = True, nlp: bool = False, continue_on_failure: bool = True, show_progress_bar: bool = False, **newspaper_kwargs: Any) → None[source]¶