langchain.document_loaders.recursive_url_loader.RecursiveUrlLoader¶
class langchain.document_loaders.recursive_url_loader.RecursiveUrlLoader(url: str, max_depth: Optional[int] = 2, use_async: Optional[bool] = None, extractor: Optional[Callable[[str], str]] = None, metadata_extractor: Optional[Callable[[str, str], str]] = None, exclude_dirs: Optional[Sequence[str]] = (), timeout: Optional[int] = 10, prevent_outside: Optional[bool] = True, link_regex: Optional[Union[str, Pattern]] = None, headers: Optional[dict] = None, check_response_status: bool = False)[source]¶
Load all child links from a URL page.
Initialize with URL to crawl and any subdirectories to exclude.
:param url: The URL to crawl.
:param max_depth: The max depth of the recursive loading.
:param use_async: Whether to use asynchronous loading.
If True, this function will not be lazy, but it will still work in the
expected way, just not lazy.
Parameters
extractor – A function to extract document contents from raw html.
When extract function returns an empty string, the document is
ignored.
metadata_extractor – A function to extract metadata from raw html and the
source url (args in that order). Default extractor will attempt
to use BeautifulSoup4 to extract the title, description and language
of the page.
exclude_dirs – A list of subdirectories to exclude.
timeout – The timeout for the requests, in the unit of seconds. If None then
connection will not timeout.
prevent_outside – If True, prevent loading from urls which are not children
of the root url.
link_regex – Regex for extracting sub-links from the raw html of a web page.
check_response_status – If True, check HTTP response status and skip