<jupyter_start><jupyter_text>Python Reference - Selenium**Author:** <NAME> Definitionuse chrome, ie or firefox drivers to automate browser interaction - the perfect tool for a webcrawler! Linkshttps://www.selenium.dev/  chrome options  list of all available options:  https://peter.sh/experiments/chromium-command-line-switches/<jupyter_code>from selenium import webdriver
chrome_options = webdriver.chrome.options.Options()
webdriver_chrome = webdriver.Chrome(options=chrome_options)<jupyter_output><empty_output><jupyter_text>run chrome without a browser window<jupyter_code>chrome_options.add_argument('--headless')<jupyter_output><empty_output><jupyter_text>required to use the '--headless' argument<jupyter_code>chrome_options.add_argument('--no-sandbox')<jupyter_output><empty_output><jupyter_text>The /dev/shm partition is too small in certain VM environments, causing Chrome to fail or crash (see http://crbug.com/715363).Use this flag to work-around this issue (a temporary directory will always be used to create anonymous shared memory files).<jupyter_code>chrome_options.add_argument('--disable-dev-shm-usage')<jupyter_output><empty_output><jupyter_text>set download directory ->  add it to the 'prefs' dictionary using add_experimental_option<jupyter_code>download_directory = os.path.abspath("C:\Documents\Downloads")
chrome_options.add_experimental_option("prefs", {"download.default_directory": download_directory,
                                                 "plugins.always_open_pdf_externally": True})<jupyter_output><empty_output><jupyter_text>get current url<jupyter_code>driver.current_url<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10030395136778116, 'fraction_numerical': 0.00364741641337386, 'mean_word_length': 6.764150943396227, 'pattern_counts': {'":': 2, '<': 26, '<?xml version=': 0, '>': 27, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '263624', 'license': 'MIT', 'n_tokens_mistral': 508, 'n_tokens_neox': 486, 'n_words': 100, 'path': 'Webcrawlers/Selenium.ipynb', 'repo_name': 'Architect0711/PythonReference', 'size': '1653'}