check_response_status – If True, check HTTP response status and skip
URLs with error responses (400-599).
Methods
__init__(url[, max_depth, use_async, ...])
Initialize with URL to crawl and any subdirectories to exclude. :param url: The URL to crawl. :param max_depth: The max depth of the recursive loading. :param use_async: Whether to use asynchronous loading. If True, this function will not be lazy, but it will still work in the                   expected way, just not lazy. :param extractor: A function to extract document contents from raw html. When extract function returns an empty string, the document is                   ignored. :param metadata_extractor: A function to extract metadata from raw html and the                            source url (args in that order). Default extractor will attempt                            to use BeautifulSoup4 to extract the title, description and language                            of the page. :param exclude_dirs: A list of subdirectories to exclude. :param timeout: The timeout for the requests, in the unit of seconds. If None then                 connection will not timeout. :param prevent_outside: If True, prevent loading from urls which are not children                         of the root url. :param link_regex: Regex for extracting sub-links from the raw html of a web page. :param check_response_status: If True, check HTTP response status and skip                               URLs with error responses (400-599).
lazy_load()
Lazy load web pages.
load()
Load web pages.
load_and_split([text_splitter])
Load Documents and split into chunks.