<jupyter_start><jupyter_text>Nasa Mission to Mars  Jupyter lab notebook  This notebook was created to facilitate the web scrapping portion of the assignment  This notebook was developed by <NAME> 2021  Instructions1. Scrape the [NASA Mars News Site](https://mars.nasa.gov/news/) and collect the latest News Title and Paragraph Text. Assign the text to variables that you can reference later.2. JPL Mars Space Images - Featured Image* Visit the url for JPL Featured Space Image [here](https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars).* Use splinter to navigate the site and find the image url for the current Featured Mars Image and assign the url string to a variable called `featured_image_url`.* Make sure to find the image url to the full size `.jpg` image.* Make sure to save a complete url string for this image.3. Mars Facts* Visit the Mars Facts webpage [here](https://space-facts.com/mars/) and use Pandas to scrape the table containing facts about the planet including Diameter, Mass, etc.* Use Pandas to convert the data to a HTML table string.4. Mars Hemispheres* Visit the USGS Astrogeology site [here](https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars) to obtain high resolution images for each of Mar's hemispheres.* You will need to click each of the links to the hemispheres in order to find the image url to the full resolution image.* Save both the image url string for the full resolution hemisphere image, and the Hemisphere title containing the hemisphere name. Use a Python dictionary to store the data using the keys `img_url` and `title`.* Append the dictionary with the image url string and the hemisphere title to a list. This list will contain one dictionary for each hemisphere.  Step 11. Scrape the [NASA Mars News Site](https://mars.nasa.gov/news/) and collect the latest News Title and Paragraph Text. Assign the text to variables that you can reference later.<jupyter_code># Importing the necessarcy dependencies 
from splinter import Browser
from bs4 import BeautifulSoup as bs
import time
import pandas as pd
import urllib.request
import requests
from webdriver_manager.chrome import ChromeDriverManager
# first step initiate browser
def init_browser():
    # Set up Splinter
    executable_path = {'executable_path': ChromeDriverManager().install()}
    browser = Browser('chrome', **executable_path, headless=False)
    return browser
# Initiate scrape for articel title and paragraph 
#For my flask application I will set this as a function called scrape in the following form:
#def scrape():
browser = init_browser()

# Visit https://mars.nasa.gov/news/
url= "https://mars.nasa.gov/news/?page=0&per_page=40&order=publish_date+desc%2Ccreated_at+desc&search=&category=19%2C165%2C184%2C204&blank_scope=Latest"
browser.visit(url)
    
# Set time sleep for it not to crash 
time.sleep(1)

# Scrape page into Soup
html = browser.html
soup = bs(html, "html.parser")

#News title
news_title = soup.find('div', class_= "content_title").text

#News paragraph
news_p = soup.find('div', class_ = "article_teaser_body").text
    
print(f"News_Titel:{news_title}")
print(f"News_Paragraph :{news_p}")

# Close the browser after scraping
browser.quit()<jupyter_output>====== WebDriver manager ======
Current google-chrome version is 90.0.4430
Get LATEST driver version for 90.0.4430
Driver [C:\Users\danie\.wdm\drivers\chromedriver\win32\90.0.4430.24\chromedriver.exe] found in cache<jupyter_text>Step 22. JPL Mars Space Images - Featured Image* Visit the url for JPL Featured Space Image [here](https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars).* Use splinter to navigate the site and find the image url for the current Featured Mars Image and assign the url string to a variable called `featured_image_url`.* Make sure to find the image url to the full size `.jpg` image.* Make sure to save a complete url string for this image.<jupyter_code>#Step 2 
# Scarpping the site for the feature photo and create a final url for it 
# For my falsk application this part of the code will join with the code on the previouse cell above the "browser.quit() " funciton 
# Scraping the image name 
browser = init_browser()
url = 'https://spaceimages-mars.com/'
browser.visit(url)

# Set time sleep for it not to crash 
time.sleep(1)

# Scrape page into Soup
html = browser.html
soup = bs(html, "html.parser")

#scrapping for images name
#image_name = soup.find('div', class_='floating_text_area')

image_name = soup.find('img', class_= 'headerimage fade-in')['src']
print(image_name)

# Close the browser after scraping
browser.quit()
featured_image_url = url + image_name
print(f"The image complete url for the featured image is: {featured_image_url}")<jupyter_output>The image complete url for the featured image is: https://spaceimages-mars.com/image/featured/mars1.jpg<jupyter_text>Step 33. Mars Facts* Visit the Mars Facts webpage [here](https://space-facts.com/mars/) and use Pandas to scrape the table containing facts about the planet including Diameter, Mass, etc.* Use Pandas to convert the data to a HTML table string.<jupyter_code># Step 3 
#Scrapping the data on the table using pandas and converting it into an html file 
#visit https://space-facts.com/mars/

#url
url = 'https://space-facts.com/mars/'

# using padnas to read the html 
tables = pd.read_html(url)
tables

#using pandas to extract only the table for mars facts [0]
mars_df = tables[0]
mars_df
#Setting the column names for mars_df
mars_df.columns = ['Mars_Planet_Profile', 'Values']
mars_df
#Turning df into html table
html_table = mars_df.to_html(header = None, index = False)
html_table.replace('\n','')
html_table
#Creating table.html file in repo
mars_df.to_html('table.html')<jupyter_output><empty_output><jupyter_text>Step 44. Mars Hemispheres* Visit the USGS Astrogeology site [here](https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars) to obtain high resolution images for each of Mar's hemispheres.* You will need to click each of the links to the hemispheres in order to find the image url to the full resolution image.* Save both the image url string for the full resolution hemisphere image, and the Hemisphere title containing the hemisphere name. Use a Python dictionary to store the data using the keys `img_url` and `title`.* Append the dictionary with the image url string and the hemisphere title to a list. This list will contain one dictionary for each hemisphere.<jupyter_code>#Step 4
browser = init_browser()
url = 'https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars'
browser.visit(url)

# Set time sleep for it not to crash 
time.sleep(1)

# Scrape page into Soup
html = browser.html
soup = bs(html, "html.parser")

#scrapping 

product = soup.find('div', class_ = 'collapsible results')
hemisphere_items = product.find_all('div', class_='item')
hemisphere_img_urls = []

#Creating conditional while scrapping 

for i in hemisphere_items:
    #Create a try statement
    try:
        hemisphere_title = i.find('h3').text
        hemisphere_href = i.find('a')['href']
        hemisphere_url = "https://astrogeology.usgs.gov/" + hemisphere_href
        hemisphere_page = requests.get(hemisphere_url).text
        soup = bs(hemisphere_page, 'html.parser')
        
        hemisphere_page_img = soup.select('#wide-image > div > ul > li:nth-child(1) > a')
        hemisphere_img_url = hemisphere_page_img[0]['href']
        hemisphere_img_dict = {"title": hemisphere_title, "img_url": hemisphere_img_url}
        hemisphere_img_urls.append(hemisphere_img_dict)
        
    #Except clause
    except Exception as e:
        print(e)

# Close the browser after scraping
browser.quit()

# Print hemispheres img title and url 
hemisphere_img_urls<jupyter_output>====== WebDriver manager ======
Current google-chrome version is 90.0.4430
Get LATEST driver version for 90.0.4430
Driver [C:\Users\danie\.wdm\drivers\chromedriver\win32\90.0.4430.24\chromedriver.exe] found in cache
{'fraction_non_alphanumeric': 0.09152203770757897, 'fraction_numerical': 0.012860531901610689, 'mean_word_length': 4.808556925308195, 'pattern_counts': {'":': 2, '<': 15, '<?xml version=': 0, '>': 19, 'https://': 16, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '559257', 'license': 'ADSL', 'n_tokens_mistral': 2546, 'n_tokens_neox': 2392, 'n_words': 910, 'path': 'mission_to_mars.ipynb', 'repo_name': 'danielatuesta55/web-scraping-challenge', 'size': '8027'}