<jupyter_start><jupyter_text>Step 1 - Scraping NASA Mars News* Scrape the [NASA Mars News Site](https://mars.nasa.gov/news/) and collect the latest News Title and Paragraph Text. Assign the text to variables that you can reference later.<jupyter_code># initialize dict for mars_dict_list in pymongo
article_dict = {}

# Setup splinter
executable_path = {'executable_path': ChromeDriverManager().install()}
browser = Browser('chrome', **executable_path, headless=False)

# 1. URL of website
mars_news_url = "https://mars.nasa.gov/news/"

# 2. Get website response
browser.visit(mars_news_url)

# 3. Store response as text
mars_news_html = browser.html

# 4. Create soup object
news_soup = BeautifulSoup(mars_news_html, 'html.parser')

#Isolate the first news article title <div class="content-title"/>
all_content_titles = news_soup.find_all("div", class_="content_title")

#capture result in variable
article_dict['latest_title'] = all_content_titles[1].text.strip()

#Isolate the paragraph for that article <div class="article_teaser_body"/>
all_article_teaser_body = news_soup.find_all('div', class_="article_teaser_body")

#capture result in variable
article_dict['latest_paragraph'] = all_article_teaser_body[0].text.strip()

# add values to mars_dict_list for pymongo
mars_dict_list.append(article_dict)
#quit broswer session
browser.quit()<jupyter_output><empty_output><jupyter_text>JPL Mars Space Images - Featured Image* Visit the url for JPL Featured Space Image [here](https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars).* Navigate the site and find the image url for the current Featured Mars.<jupyter_code># initialize dict for mars_dict_list in pymongo
featured_img_dict = {}   
    
# Setup splinter
executable_path = {'executable_path': ChromeDriverManager().install()}
browser = Browser('chrome', **executable_path, headless=False)

# 1. URL of website
jpl_imgs_url ="https://www.jpl.nasa.gov/images?search=&category=Mars"

# 2. Get website response
browser.visit(jpl_imgs_url)

# 3. Store response as text
jpl_imgs_html = browser.html

# 4. Create soup object
jpl_soup = BeautifulSoup(jpl_imgs_html, 'html.parser')

#Find & click first img on page <img src = browser.find_by_css('img[class="BaseImage  object-contain"]').first />
featured_img_click = browser.find_by_css('img[class="BaseImage  object-contain"]').first.click()

# Store response as text
featured_img_html = browser.html

# Create soup object of new page
featured_soup = BeautifulSoup(featured_img_html, 'html.parser')

# capture fullsize featured img url
featured_img_dict['featured_image_url'] = featured_soup.find('img', class_= 'BaseImage')['src']

# add values to mars_dict_list for pymongo
mars_dict_list.append(featured_img_dict)

# quit browser session
browser.quit()<jupyter_output>[WDM] - ====== WebDriver manager ======
[WDM] - Current google-chrome version is 89.0.4389
[WDM] - Get LATEST driver version for 89.0.4389
[WDM] - Driver [C:\Users\Lola\.wdm\drivers\chromedriver\win32\89.0.4389.23\chromedriver.exe] found in cache<jupyter_text>Mars Facts* Visit the Mars Facts webpage [here](https://space-facts.com/mars/)* Scrape facts about the planet including Diameter, Mass, etc.<jupyter_code># URL of website
facts_url ="https://space-facts.com/mars/"

# Store response as text
facts_html = pd.read_html(facts_url)

# get first table
facts_table = facts_html[0]

#give columns headers
facts_table.columns = ['Title', 'Fact']

# convert df into html
pprint(facts_table.to_html())<jupyter_output>('<table border="1" class="dataframe">\n'
 '  <thead>\n'
 '    <tr style="text-align: right;">\n'
 '      <th></th>\n'
 '      <th>Title</th>\n'
 '      <th>Fact</th>\n'
 '    </tr>\n'
 '  </thead>\n'
 '  <tbody>\n'
 '    <tr>\n'
 '      <th>0</th>\n'
 '      <td>Equatorial Diameter:</td>\n'
 '      <td>6,792 km</td>\n'
 '    </tr>\n'
 '    <tr>\n'
 '      <th>1</th>\n'
 '      <td>Polar Diameter:</td>\n'
 '      <td>6,752 km</td>\n'
 '    </tr>\n'
 '    <tr>\n'
 '      <th>2</th>\n'
 '      <td>Mass:</td>\n'
 '      <td>6.39 × 10^23 kg (0.11 Earths)</td>\n'
 '    </tr>\n'
 '    <tr>\n'
 '      <th>3</th>\n'
 '      <td>Moons:</td>\n'
 '      <td>2 (Phobos &amp; Deimos)</td>\n'
 '    </tr>\n'
 '    <tr>\n'
 '      <th>4</th>\n'
 '      <td>Orbit Distance:</td>\n'
 '      <td>227,943,824 km (1.38 AU)</td>\n'
 '    </tr>\n'
 '    <tr>\n'
 '      <th>5</th>\n'
 '      <td>Orbit Period:</td>\n'
 '      <td>687 days (1.9 years)</td>\n'
 '    </tr>\n'
 '    <tr>\n'
 '      <th>6</th>\n'
 '  [...]<jupyter_text>USGS Astrogeology site - Mars Hemispheres* Visit the USGS Astrogeology site [here](https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars) to obtain high resolution images for each of Mar's hemispheres.* Save both the image url string for the full resolution hemisphere image, and the Hemisphere title containing the hemisphere name.<jupyter_code>#Open the page with splinter
executable_path = {'executable_path': ChromeDriverManager().install()}
browser = Browser('chrome', **executable_path, headless=False)

#1. URL of website
hemi_url ="https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars"

#2. Get website response
browser.visit(hemi_url)

#3. Store response as text
all_hemi_html = browser.html

#4. Create soup object
all_hemi_soup = BeautifulSoup(all_hemi_html, 'html.parser')
all_hemi_soup

#find all imgs
hemi_all_img = all_hemi_soup.find_all('img', class_='thumb')

# initialize dict for mars_dict_list in pymongo
hemi_dict = {}

# Start loop to get img and title on all hemishpere
for img in hemi_all_img:
    #Locate img link
    img_target = 'img[class="thumb"]'
    #Click the link
    browser.find_by_tag(img_target).click()
    #Store response as text
    one_hemi_html = browser.html
    #Create soup object
    one_hemi_soup = BeautifulSoup(one_hemi_html, 'html.parser')
    #Save img url and title to variable        
    hemi_dict['img_url'] = one_hemi_soup.find("a", text="Original")['href']
    #Save hemishpere title
    hemi_dict['title'] = one_hemi_soup.find("h2", class_="title").text
    #Append the dict to list
    mars_dict_list.append(hemi_dict)
    #Click the back button
    browser.back()
#check your list
hemi_dict<jupyter_output>[WDM] - ====== WebDriver manager ======
[WDM] - Current google-chrome version is 89.0.4389
[WDM] - Get LATEST driver version for 89.0.4389
[WDM] - Driver [C:\Users\Lola\.wdm\drivers\chromedriver\win32\89.0.4389.23\chromedriver.exe] found in cache<jupyter_text>Step 2 - MongoDB and Flask ApplicationUse MongoDB with Flask templating to create a new HTML page that displays all of the information that was scraped from the URLs above.* Start by converting your Jupyter notebook into a Python script called `scrape_mars.py` with a function called `scrape` that will execute all of your scraping code from above and return one Python dictionary containing all of the scraped data.* Next, create a route called `/scrape` that will import your `scrape_mars.py` script and call your `scrape` function.  * Store the return value in Mongo as a Python dictionary.* Create a root route `/` that will query your Mongo database and pass the mars data into an HTML template to display the data.* Create a template HTML file called `index.html` that will take the mars data dictionary and display all of the data in the appropriate HTML elements. Use the following as a guide for what the final product should look like, but feel free to create your own design.<jupyter_code># create connection to mongo database
conn = 'mongodb://localhost:27017'
client = pymongo.MongoClient(conn)

# create new data base
db = client.mars_db

# drop the collection if it already exists
db.mars.drop()

# instert our list of mini dictionaries from scraping
db.mars.insert_many(quotes_list)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13071227980597397, 'fraction_numerical': 0.015445494000510596, 'mean_word_length': 4.440972222222222, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 84, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '143001', 'license': 'ADSL', 'n_tokens_mistral': 2814, 'n_tokens_neox': 2691, 'n_words': 734, 'path': 'mission_to_mars/mission_to_mars.ipynb', 'repo_name': 'eglulu6/web-scraping-challenge', 'size': '7834'}