<jupyter_start><jupyter_text>Mission to Mars![mission_to_mars](Images/mission_to_mars.jpg)In this assignment, you will build a web application that scrapes various websites for data related to the Mission to Mars and displays the information in a single HTML page. The following outlines what you need to do.  Step 1 - ScrapingComplete your initial scraping using Jupyter Notebook, BeautifulSoup, Pandas, and Requests/Splinter.* Create a Jupyter Notebook file called `mission_to_mars.ipynb` and use this to complete all of your scraping and analysis tasks. The following outlines what you need to scrape.  NASA Mars News* Scrape the [NASA Mars News Site](https://mars.nasa.gov/news/) and collect the latest News Title and Paragragh Text. Assign the text to variables that you can reference later.```python Example:news_title = "NASA's Next Mars Mission to Investigate Interior of Red Planet"news_p = "Preparation of NASA's next spacecraft to Mars, InSight, has ramped up this summer, on course for launch next May from Vandenberg Air Force Base in central California -- the first interplanetary launch in history from America's West Coast."```<jupyter_code># Dependencies
import pandas as pd
import re
import requests
import pymongo
from splinter import Browser
from bs4 import BeautifulSoup
# Scraping NASA Mars News
news_url = 'https://mars.nasa.gov/news/'
news_html = requests.get(news_url)
# Parsing w/ BS4
news_soup = BeautifulSoup(news_html.text, 'html.parser')
# Print body
print(news_soup.body.prettify())
# Print news article titles
news_titles = news_soup.find_all('div', class_='content_title')
news_titles
# Print title text only
for news in news_titles:
    title = news.find('a')
    news_title = title.text
    print(news_title)
# Print News Paragraphs
mars_parag = news_soup.find_all('div', class_='rollover_description')
mars_parag
# Print paragraph text only
for paragraph in mars_parag:
    p_text = paragraph.find('div', class_='rollover_description_inner')
    mars_p = p_text.text
    print(mars_p)<jupyter_output>NASA is investing in technology concepts, including several from JPL, that may one day be used for future space exploration missions.


NASA is about to go on a journey to study the center of Mars.


NASA’s next mission to Mars will be the topic of a media briefing Thursday, March 29, at JPL. The briefing will air live on NASA Television and the agency’s website.


NASA spacecraft travel to far-off destinations in space, but a new mobile app produced by NASA's Jet Propulsion Laboratory, Pasadena, California, brings spacecraft to users.


NASA invites digital creators to apply for social media credentials to cover the launch of the InSight mission to Mars, May 3-5, at California's Vandenberg Air Force Base.


NASA's InSight spacecraft has arrived at Vandenberg Air Force Base in central California to begin final preparations for a launch this May.<jupyter_text>JPL Mars Space Images - Featured Image* Visit the url for JPL's Featured Space Image [here](https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars).* Use splinter to navigate the site and find the image url for the current Featured Mars Image and assign the url string to a variable called `featured_image_url`.* Make sure to find the image url to the full size `.jpg` image.* Make sure to save a complete url string for this image.```python Example:featured_image_url = 'https://www.jpl.nasa.gov/spaceimages/images/largesize/PIA16225_hires.jpg'```<jupyter_code># Open browser of Mars space images# Open b 
mars_img_browser = Browser('chrome', headless=False)
jpl_url = 'https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars'
mars_img_browser.visit(jpl_url)
# Parse html file with BeautifulSoup
mars_img_html = mars_img_browser.html
jpl_soup = BeautifulSoup(mars_img_html, 'html.parser')
# Print body of html# Print  
print(jpl_soup.body.prettify())
# Find all images on page
mars_imgs = jpl_soup.find_all('div', class_='carousel_items')
mars_imgs
# Retrieve our featured image url
for img in mars_imgs:
    large_pic = img.find(class_='button fancybox').get('data-fancybox-href')
    
    featured_image_url = f'https://www.jpl.nasa.gov{large_pic}'
    print(featured_image_url)<jupyter_output>https://www.jpl.nasa.gov/spaceimages/images/mediumsize/PIA18358_ip.jpg<jupyter_text>Mars Weather* Visit the Mars Weather twitter account [here](https://twitter.com/marswxreport?lang=en) and scrape the latest Mars weather tweet from the page. Save the tweet text for the weather report as a variable called `mars_weather`.```python Example:mars_weather = 'Sol 1801 (Aug 30, 2017), Sunny, high -21C/-5F, low -80C/-112F, pressure at 8.82 hPa, daylight 06:09-17:55'```<jupyter_code># Retrieve tweets using Splinter
twitter_browser = Browser('chrome', headless=False)
twitter_url = 'https://twitter.com/marswxreport?lang=en'
twitter_browser.visit(twitter_url)
# Parse with BeautifulSoup
twitter_html = twitter_browser.html
twitter_soup = BeautifulSoup(twitter_html, 'html.parser')
# Print body of twitter html
print(twitter_soup.body.prettify())
# Find all mars weather tweets with BeautifulSoup
weather_tweets = twitter_soup.find_all('p', class_='TweetTextSize')
weather_tweets
# Filter tweets beginning with 'sol' to retrieve weather text
weather_filter = 'Sol'
for tweet in weather_tweets:
    if weather_filter in tweet.text and 'Happy' not in tweet.text:
        mars_weather = tweet.text
        print(mars_weather)<jupyter_output>Sol 2030 (April 22, 2018), Sunny, high -4C/24F, low -73C/-99F, pressure at 7.21 hPa, daylight 05:25-17:21
Sol 2029 (April 21, 2018), Sunny, high -11C/12F, low -72C/-97F, pressure at 7.22 hPa, daylight 05:25-17:21
Sol 2026 (April 18, 2018), Sunny, high -6C/21F, low -73C/-99F, pressure at 7.19 hPa, daylight 05:26-17:21
Sol 2024 (April 16, 2018), Sunny, high -7C/19F, low -76C/-104F, pressure at 7.20 hPa, daylight 05:26-17:21
Sol 2022 (April 14, 2018), Sunny, high -4C/24F, low -73C/-99F, pressure at 7.19 hPa, daylight 05:27-17:21
Sol 2019 (April 11, 2018), Sunny, high -6C/21F, low -75C/-103F, pressure at 7.18 hPa, daylight 05:27-17:21
Sol 2017 (April 09, 2018), Sunny, high -6C/21F, low -75C/-103F, pressure at 7.17 hPa, daylight 05:28-17:21
Sol 2016 (April 08, 2018), Sunny, high -9C/15F, low -73C/-99F, pressure at 7.18 hPa, daylight 05:28-17:21
Sol 2012 (April 04, 2018), Sunny, high -7C/19F, low -74C/-101F, pressure at 7.15 hPa, daylight 05:29-17:22
Sol 2011 (April 03, 2018), Sunny, high -1[...]<jupyter_text>Mars Facts* Visit the Mars Facts webpage [here](http://space-facts.com/mars/) and use Pandas to scrape the table containing facts about the planet including Diameter, Mass, etc.* Use Pandas to convert the data to a HTML table string.<jupyter_code># Store mars facts URL
facts_url = 'http://space-facts.com/mars/'
# Read table from URL
facts_table = pd.read_html(facts_url)
facts_table
# Use pandas for table format
mars_facts = facts_table[0]

# Transpose table to make header
mars_df = mars_facts.transpose()

# Re-name all headers
mars_df.columns = [
    'Equatorial Diameter',
    'Polar Diameter',
    'Mass',
    'Moons',
    'Orbit Distance',
    'Orbit Period',
    'Surface Temperature',
    'First Record',
    'Recorded By'
]

# Drop sub-header
mars_facts_df = mars_df.iloc[1:]

mars_facts_df
# Convert the table to HTML data string
mars_facts_html = mars_facts_df.to_html()
print(mars_facts_html)<jupyter_output><table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Equatorial Diameter</th>
      <th>Polar Diameter</th>
      <th>Mass</th>
      <th>Moons</th>
      <th>Orbit Distance</th>
      <th>Orbit Period</th>
      <th>Surface Temperature</th>
      <th>First Record</th>
      <th>Recorded By</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>6,792 km</td>
      <td>6,752 km</td>
      <td>6.42 x 10^23 kg (10.7% Earth)</td>
      <td>2 (Phobos &amp; Deimos)</td>
      <td>227,943,824 km (1.52 AU)</td>
      <td>687 days (1.9 years)</td>
      <td>-153 to 20 °C</td>
      <td>2nd millennium BC</td>
      <td>Egyptian astronomers</td>
    </tr>
  </tbody>
</table><jupyter_text>Mars Hemispheres* Visit the USGS Astrogeology site [here](https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars) to obtain high resolution images for each of Mar's hemispheres.* You will need to click each of the links to the hemispheres in order to find the image url to the full resolution image.* Save both the image url string for the full resolution hemipshere image, and the Hemisphere title containing the hemisphere name. Use a Python dictionary to store the data using the keys `img_url` and `title`.* Append the dictionary with the image url string and the hemisphere title to a list. This list will contain one dictionary for each hemisphere.```python Example:hemisphere_image_urls = [    {"title": "Valles Marineris Hemisphere", "img_url": "..."},    {"title": "Cerberus Hemisphere", "img_url": "..."},    {"title": "Schiaparelli Hemisphere", "img_url": "..."},    {"title": "Syrtis Major Hemisphere", "img_url": "..."},]```<jupyter_code># Retrieve hemisphere images and links using Splinter
usgs_browser = Browser('chrome', headless=False)
usgs_url = 'https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars'
usgs_browser.visit(usgs_url)
# Parse hemisphere html using BS
hemi_html = usgs_browser.html
hemi_soup = BeautifulSoup(hemi_html, "html.parser")
# Print body
print(hemi_soup.body.prettify())
# Find image link and title
hemi_description = hemi_soup.find_all('div', class_='description')
print(hemi_description)
# Create list of dictionaries containing img title and link
hemi_img_urls = []

for image in hemi_description:
    found_hemi_url = image.find('a', class_='itemLink')
    partial_hemi_url = found_hemi_url.get('href')
    complete_hemi_url = 'https://astrogeology.usgs.gov' + partial_hemi_url
    print(complete_hemi_url)
    
    # Visit each link
    usgs_browser.visit(complete_hemi_url)
    
    # Create dictionary containing title and image url
    hemi_img_dict = {}
    
    # Parse html again
    mars_hemi_html = usgs_browser.html
    mars_hemi_soup = BeautifulSoup(mars_hemi_html, 'html.parser')
    
    # Get image link
    hemi_link = mars_hemi_soup.find('a', text='Original').get('href')
    
    # Get title text
    hemi_title = mars_hemi_soup.find('h2', class_='title').text.replace(' Enhanced', '')
    
    # Append title and image urls of hemisphere to dictionary
    hemi_img_dict['title'] = hemi_title
    hemi_img_dict['img_url'] = hemi_link
    
    # Append dictionaries to list
    hemi_img_urls.append(hemi_img_dict)

print(hemi_img_urls)
# Convert jupyter notebook file into python script 'scrape_mars.py'
! jupyter nbconvert --to script --template basic Mission_to_Mars.ipynb --output scrape_mars<jupyter_output>[NbConvertApp] Converting notebook Mission_to_Mars.ipynb to script
[NbConvertApp] Writing 8636 bytes to scrape_mars.py
{'fraction_non_alphanumeric': 0.10512913786831575, 'fraction_numerical': 0.03355765732993816, 'mean_word_length': 4.590747330960854, 'pattern_counts': {'":': 8, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 12, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '561239', 'license': 'ADSL', 'n_tokens_mistral': 4042, 'n_tokens_neox': 3595, 'n_words': 1170, 'path': 'Mission_to_Mars.ipynb', 'repo_name': 'JasonBLeeDataAnalyses/Mission_to_Mars', 'size': '10996'}