<jupyter_start><jupyter_text>1. Write a python program to display all the header tags from ‘en.wikipedia.org/wiki/Main_Page’.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://en.wikipedia.org/wiki/Main_Page")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting all header tags from the page
header_tags=soup.find_all('span',class_="mw-headline")
header_tags

#Extracting the text from these tags one by one and it can be done by looping over these tags using for loop
Header=[]  #Empty list
for i in header_tags:
    Header.append(i.text.replace('\xa0...', ''))
Header 

#Saving the tags in a dataframe
import pandas as pd
tags=pd.DataFrame(Header)
tags<jupyter_output><empty_output><jupyter_text>2. Write a python program to display IMDB’s Top rated 100 movies’ data (i.e. Name, IMDB rating, Year of release) and save it in form of a CSV file.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://www.imdb.com/search/title/?count=100&groups=top_100&sort=user_rating")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting all the tags having movie names
names=soup.find_all('h3',class_="lister-item-header")
names[0:10]

#Extracting the text alone from the tags
movies=[]  #Empty list
for i in names:
    for j in i.find_all('a'):   #As we need only the movie names but not the year and index no
        movies.append(j.text)
movies[0:10]   

#Extracting all the tags having ratings
rating=soup.find_all('strong')
rating[0:10]

#Extracting only the rating
ratings=[]   #Empty list
for i in rating:
    ratings.append(i.get_text().replace('Detailed', " ").replace('User Rating', " "))
ratings[:10]  

#Extracting all the tags having the year of release
year=soup.find_all('span',class_="lister-item-year text-muted unbold")
year[0:10]

#Extracting only the year value
release_year=[]   #empty list
for i in year:
    release_year.append(i.get_text().replace('(I) ', " "))
release_year[:10]

#Checking the length of each of the list obtained
print(len(movies),len(ratings),len(release_year))

#Creating a new dataframe with the data we extracted
import pandas as pd
IMDB_100=pd.DataFrame({})
IMDB_100['Movie_name']=movies
IMDB_100['Rating']=ratings[2:]
IMDB_100['Year_of_Release']=release_year

#Checking the dataframe 
IMDB_100
#Saving the data into a csv file
IMDB_100.to_csv('IMDB_top100movies.csv')<jupyter_output><empty_output><jupyter_text>3. Write a python program to display IMDB’s Top rated 100 Indian movies’ data (i.e. Name, IMDB rating, Year of release) and save it in form of a CSV file.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://www.imdb.com/india/top-rated-indian-movies/?pf_rd_m=A2FGELUUNOQJNL&pf_rd_p=461131e5-5af0-4e50-bee2-223fad1e00ca&pf_rd_r=KW9XY49QFKX5SR23QR6F&pf_rd_s=center-1&pf_rd_t=60601&pf_rd_i=india.toprated&ref_=fea_india_ss_toprated_india_tr_india250_hd")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting all the tags having movie names
names=soup.find_all('td',class_='titleColumn')
names[0:10]

#Extracting the text alone from the tags
movies=[]  #Empty list
for i in names:
    for j in i.find_all('a'):   #As we need only the movie names but not the year and index no
        movies.append(j.text)
movies[0:10]  

#Extracting all the tags having ratings
ratings=soup.find_all('strong')
ratings[0:10]

#Extracting only the rating
rating=[]   #Empty list
for i in ratings:
    rating.append(i.get_text())
rating[:10] 

#Extracting all the tags having the year of release
year=soup.find_all('span',class_="secondaryInfo")
year[0:10]

#Extracting only the year value
release_year=[]   #empty list
for i in year:
    release_year.append(i.get_text())
release_year[:10]

#Checking the length of each of the list obtained
print(len(movies),len(rating),len(release_year))

#Creating a new dataframe with the data we extracted
import pandas as pd
IMDB_Indian100=pd.DataFrame({})
IMDB_Indian100['Movie_name']=movies[:100]
IMDB_Indian100['Rating']=rating[:100]
IMDB_Indian100['Year_of_Release']=release_year[:100]

#Checking the dataframe 
IMDB_Indian100
#Saving the data into a csv file
IMDB_Indian100.to_csv('IMDB_top100_IndianMovies.csv')<jupyter_output><empty_output><jupyter_text>4. Write a python program to scrap book name, author name, genre and book review of any 5 books from ‘www.bookpage.com’.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://bookpage.com/reviews")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting all the tags having book name
book_names=soup.find_all('h4',class_='italic')
book_names[:5]

#Extracting the text alone from the tags
book_title=[]  #Empty list
for i in book_names:
    for j in i.find_all('a'):   #As we need only the movie names but not the year and index no
        book_title.append(j.text)
book_title[:5]

#Extracting all the tags having the author name
author=soup.find_all('p',class_="sans bold")
author[:5]

#Extracting only the text from the tags
authors=[]   #Empty list
for i in author:
    authors.append(i.get_text().replace('\n', " "))
authors[:5] 

#Extracting all the tags having the genre of the book
genre=soup.find_all('p',class_="genre-links hidden-phone")
genre[:5]

#Extracting only the text from the tags
book_genre=[]   #Empty list
for i in genre:
    book_genre.append(i.get_text().replace('\n', ""))
book_genre[:5]  

#Extracting the tags having book review
review=soup.find_all('p',class_="excerpt")
review[:5]

#Extracting only the text from the tags
book_review=[]  #Empty list
for i in review:
    book_review.append(i.get_text().replace('\n', '').replace('\xa0', ''))
book_review[:5]

#Checking the length of each of the list obtained
print(len(book_title),len(authors),len(book_genre),len(book_review))

#Creating a new dataframe with the data we extracted
import pandas as pd
books=pd.DataFrame({})
books['Title']=book_title[:5]
books['Author']=authors[:5]
books['Genre']=book_genre[:5]
books['Review']=book_review[:5]

#Checking the dataframe
books<jupyter_output>10 10 10 10<jupyter_text>5. Write a python program to scrape cricket rankings from ‘www.icc-cricket.com’. i) Top 10 ODI teams in men’s cricket along with the records for matches, points and rating.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://www.icc-cricket.com/rankings/mens/team-rankings/odi")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting all the tags having team names
team=soup.find_all('span',class_="u-hide-phablet")
team[:10]

#Extracting the text alone from the tags
teams=[]  #Empty list
for i in team:
    teams.append(i.get_text())
teams[:10]

#Extracting the tags having the number of matches for Rank1 team
match=soup.find_all('td',class_="rankings-block__banner--matches")
match

#Saving matches to matches list
Matches=[]  
for i in match:
    Matches.append(i.get_text().replace('\n',''))
Matches

#Extracting remaining matches
match=soup.find_all('td',class_="table-body__cell u-center-text")
match[:9]

del match[1::2]   #As it extracted other column data, we are deleting it

for i in match:
    Matches.append(i.get_text().replace('\n',''))
Matches[:10]

#Extracting the tags having the points for Rank1 team
points=soup.find_all('td',class_="rankings-block__banner--points")
points

#Saving points to points list
Points=[]  
for i in points:
    Points.append(i.get_text().replace('\n',''))
Points

#Extracting remaining points
points=soup.find_all('td',class_="table-body__cell u-center-text")
points[:9]

del points[::2]   #As other data are obtained, we are deleting them

#Saving points of respective teams
for i in points:
    Points.append(i.get_text().replace('\n',''))
Points[:10]

#Extracting the rating from tags of rank1 team
rating=soup.find_all('td',class_="rankings-block__banner--rating u-text-right")
rating

#Saving rating to rating list
Rating=[]  
for i in rating:
    Rating.append(i.get_text().replace('\n','').strip())
Rating

#Extracting other team ratings data
rating=soup.find_all('td',attrs={'class':'table-body__cell u-text-right rating'})
rating[:9]

#Saving other teams ratings 
for i in rating:
    Rating.append(i.get_text().replace('\n',''))
Rating=Rating[0:10]   #Saving only first 10 ratings
Rating

#Checking the length of each of the list obtained
print(len(teams),len(Matches),len(Points),len(Rating))

#Creating a new dataframe with the data we extracted
import pandas as pd
Team_Rankings=pd.DataFrame({})
Team_Rankings['Team']=teams[:10]
Team_Rankings['Matches']=Matches[:10]
Team_Rankings['Points']=Points[:10]
Team_Rankings['Rating']=Rating

#Checking the dataframe
Team_Rankings<jupyter_output>25 20 20 10<jupyter_text>ii) Top 10 ODI Batsmen in men along with the records of their team and rating.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://www.icc-cricket.com/rankings/mens/player-rankings/odi/batting")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting the tag having player name for Rank1
name=soup.find_all('div',class_="rankings-block__banner--name-large")
name

#Extracting the text alone from tag
names=[]  #Empty list
for i in name:
    names.append(i.get_text())
names

#Extracting the tag having player names
name=soup.find_all('td',class_="table-body__cell rankings-table__name name")
name[:9]

#Extracting the text from tags
for i in name:
    names.append(i.get_text().replace('\n', ' '))
names[:10]    

#Extracting the tag for team for rank1
team=soup.find_all('div',class_="rankings-block__banner--nationality")
team

#Extracting the text alone from tag
teams=[]  #Empty list
for i in team:
    teams.append(i.get_text().replace('\n', ''))
teams   

#Extracting the tags for other teams
team=soup.find_all('span',class_="table-body__logo-text")
team[:9]

#Extracting the text alone from tags
for i in team:
    teams.append(i.get_text())
teams[:10] 

#Extracting the tag having rating for rank1
rating=soup.find_all('div',class_="rankings-block__banner--rating")
rating

#Extracting the text alone from tag
ratings=[]  #Empty list
for i in rating:
    ratings.append(i.get_text())
ratings 

#Extracting the tag for ratings of other players
rating=soup.find_all('td',class_="table-body__cell rating")
rating[:9]

#Extracting the text alone from tags
for i in rating:
    ratings.append(i.get_text())
ratings[:10] 

#Extracting the tags having career best rating for rank1
career_best=soup.find_all('span',class_="rankings-block__career-best-text")
career_best

#Extracting the text alone from tag
career_best_rating=[]  #Empty list
for i in career_best:
    career_best_rating.append(i.get_text().replace('\n', ' ').replace('  ',''))
career_best_rating    

#Extracting the tags for remaining players career best rating
career_best=soup.find_all('td',class_="table-body__cell u-text-right u-hide-phablet")
career_best[:9]

#Extracting the text alone from tags
for i in career_best:
    career_best_rating.append(i.get_text().replace('\n', ' ').replace('                        ',' '))
career_best_rating[:10] 

#Checking the length of each of the list obtained
print(len(names),len(teams),len(ratings),len(career_best_rating))

#Creating a new dataframe with the data we extracted
import pandas as pd
top10=pd.DataFrame({})
top10['Name']=names[:10]
top10['Team']=teams[:10]
top10['Rating']=ratings[:10]
top10['Career_best_rating']=career_best_rating[:10]

#Checking the dataframe
top10<jupyter_output>100 100 100 100<jupyter_text>iii) Top 10 ODI bowlers along with the records of their team and rating.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://www.icc-cricket.com/rankings/mens/player-rankings/odi/bowling")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting the tag having player name for Rank1
name=soup.find_all('div',class_="rankings-block__banner--name-large")
name

#Extracting the text alone from tag
names=[]  #Empty list
for i in name:
    names.append(i.get_text())
names

#Extracting the tag having player names
name=soup.find_all('td',class_="table-body__cell rankings-table__name name")
name[:9]

#Extracting the text from tags
for i in name:
    names.append(i.get_text().replace('\n', ' '))
names[:10]    

#Extracting the tag for team for rank1
team=soup.find_all('div',class_="rankings-block__banner--nationality")
team

#Extracting the text alone from tag
teams=[]  #Empty list
for i in team:
    teams.append(i.get_text().replace('\n', ''))
teams   

#Extracting the tags for other teams
team=soup.find_all('span',class_="table-body__logo-text")
team[:9]

#Extracting the text alone from tags
for i in team:
    teams.append(i.get_text())
teams[:10] 

#Extracting the tag having rating for rank1
rating=soup.find_all('div',class_="rankings-block__banner--rating")
rating

#Extracting the text alone from tag
ratings=[]  #Empty list
for i in rating:
    ratings.append(i.get_text())
ratings 

#Extracting the tag for ratings of other players
rating=soup.find_all('td',class_="table-body__cell rating")
rating[:9]

#Extracting the text alone from tags
for i in rating:
    ratings.append(i.get_text())
ratings[:10] 

#Extracting the tags having career best rating for rank1
career_best=soup.find_all('span',class_="rankings-block__career-best-text")
career_best

#Extracting the text alone from tag
career_best_rating=[]  #Empty list
for i in career_best:
    career_best_rating.append(i.get_text().replace('\n', ' ').replace('  ',''))
career_best_rating    

#Extracting the tags for remaining players career best rating
career_best=soup.find_all('td',class_="table-body__cell u-text-right u-hide-phablet")
career_best[:9]

#Extracting the text alone from tags
for i in career_best:
    career_best_rating.append(i.get_text().replace('\n', ' ').replace('                        ',' '))
career_best_rating[:10] 

#Checking the length of each of the list obtained
print(len(names),len(teams),len(ratings),len(career_best_rating))

#Creating a new dataframe with the data we extracted
import pandas as pd
top10=pd.DataFrame({})
top10['Name']=names[:10]
top10['Team']=teams[:10]
top10['Rating']=ratings[:10]
top10['Career_best_rating']=career_best_rating[:10]

#Checking the dataframe
top10<jupyter_output>100 100 100 100<jupyter_text>6. Write a python program to scrape cricket rankings from ‘www.icc-cricket.com’. i) Top 10 ODI teams in women’s cricket along with the records for matches, points and rating.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://www.icc-cricket.com/rankings/womens/team-rankings/odi")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting all the tags having team names
team=soup.find_all('span',class_="u-hide-phablet")
team[:10]

#Extracting the text alone from the tags
teams=[]  #Empty list
for i in team:
    teams.append(i.get_text())
teams[:10]

#Extracting the tags having the number of matches for Rank1 team
match=soup.find_all('td',class_="rankings-block__banner--matches")
match

#Saving matches to matches list
Matches=[]  
for i in match:
    Matches.append(i.get_text().replace('\n',''))
Matches

#Extracting remaining matches
match=soup.find_all('td',class_="table-body__cell u-center-text")
match[:9]

del match[1::2]   #As it extracted other column data, we are deleting it

for i in match:
    Matches.append(i.get_text().replace('\n',''))
Matches[:10]

#Extracting the tags having the points for Rank1 team
points=soup.find_all('td',class_="rankings-block__banner--points")
points

#Saving points to points list
Points=[]  
for i in points:
    Points.append(i.get_text().replace('\n',''))
Points

#Extracting remaining points
points=soup.find_all('td',class_="table-body__cell u-center-text")
points[:9]

del points[::2]   #As other data are obtained, we are deleting them

#Saving points of respective teams
for i in points:
    Points.append(i.get_text().replace('\n',''))
Points[:10]

#Extracting the rating from tags of rank1 team
rating=soup.find_all('td',class_="rankings-block__banner--rating u-text-right")
rating

#Saving rating to rating list
Rating=[]  
for i in rating:
    Rating.append(i.get_text().replace('\n','').strip())
Rating

#Extracting other team ratings data
rating=soup.find_all('td',attrs={'class':'table-body__cell u-text-right rating'})
rating[:9]

#Saving other teams ratings 
for i in rating:
    Rating.append(i.get_text().replace('\n',''))
Rating=Rating[0:10]   #Saving only first 10 ratings
Rating

#Checking the length of each of the list obtained
print(len(teams),len(Matches),len(Points),len(Rating))

#Creating a new dataframe with the data we extracted
import pandas as pd
Team_Rankings=pd.DataFrame({})
Team_Rankings['Team']=teams[:10]
Team_Rankings['Matches']=Matches[:10]
Team_Rankings['Points']=Points[:10]
Team_Rankings['Rating']=Rating

#Checking the dataframe
Team_Rankings<jupyter_output>15 10 10 10<jupyter_text>ii) Top 10 ODI Batswomen in women along with the records of their team and rating.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://www.icc-cricket.com/rankings/womens/player-rankings/odi/batting")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting the tag having player name for Rank1
name=soup.find_all('div',class_="rankings-block__banner--name-large")
name

#Extracting the text alone from tag
names=[]  #Empty list
for i in name:
    names.append(i.get_text())
names

#Extracting the tag having player names
name=soup.find_all('td',class_="table-body__cell rankings-table__name name")
name[:9]

#Extracting the text from tags
for i in name:
    names.append(i.get_text().replace('\n', ' '))
names[:10]    

#Extracting the tag for team for rank1
team=soup.find_all('div',class_="rankings-block__banner--nationality")
team

#Extracting the text alone from tag
teams=[]  #Empty list
for i in team:
    teams.append(i.get_text().replace('\n', ''))
teams   

#Extracting the tags for other teams
team=soup.find_all('span',class_="table-body__logo-text")
team[:9]

#Extracting the text alone from tags
for i in team:
    teams.append(i.get_text())
teams[:10] 

#Extracting the tag having rating for rank1
rating=soup.find_all('div',class_="rankings-block__banner--rating")
rating

#Extracting the text alone from tag
ratings=[]  #Empty list
for i in rating:
    ratings.append(i.get_text())
ratings 

#Extracting the tag for ratings of other players
rating=soup.find_all('td',class_="table-body__cell rating")
rating[:9]

#Extracting the text alone from tags
for i in rating:
    ratings.append(i.get_text())
ratings[:10] 

#Extracting the tags having career best rating for rank1
career_best=soup.find_all('span',class_="rankings-block__career-best-text")
career_best

#Extracting the text alone from tag
career_best_rating=[]  #Empty list
for i in career_best:
    career_best_rating.append(i.get_text().replace('\n', ' ').replace('  ',''))
career_best_rating    

#Extracting the tags for remaining players career best rating
career_best=soup.find_all('td',class_="table-body__cell u-text-right u-hide-phablet")
career_best[:9]

#Extracting the text alone from tags
for i in career_best:
    career_best_rating.append(i.get_text().replace('\n', ' ').replace('                        ',' '))
career_best_rating[:10] 

#Checking the length of each of the list obtained
print(len(names),len(teams),len(ratings),len(career_best_rating))

#Creating a new dataframe with the data we extracted
import pandas as pd
top10=pd.DataFrame({})
top10['Name']=names[:10]
top10['Team']=teams[:10]
top10['Rating']=ratings[:10]
top10['Career_best_rating']=career_best_rating[:10]

#Checking the dataframe
top10<jupyter_output>100 100 100 100<jupyter_text>iii) Top 10 women’s ODI all-rounders along with the records of their team and rating.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://www.icc-cricket.com/rankings/womens/player-rankings/odi/all-rounder")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting the tag having player name for Rank1
name=soup.find_all('div',class_="rankings-block__banner--name-large")
name

#Extracting the text alone from tag
names=[]  #Empty list
for i in name:
    names.append(i.get_text())
names

#Extracting the tag having player names
name=soup.find_all('td',class_="table-body__cell rankings-table__name name")
name[:9]

#Extracting the text from tags
for i in name:
    names.append(i.get_text().replace('\n', ' '))
names[:10]    

#Extracting the tag for team for rank1
team=soup.find_all('div',class_="rankings-block__banner--nationality")
team

#Extracting the text alone from tag
teams=[]  #Empty list
for i in team:
    teams.append(i.get_text().replace('\n', ''))
teams   

#Extracting the tags for other teams
team=soup.find_all('span',class_="table-body__logo-text")
team[:9]

#Extracting the text alone from tags
for i in team:
    teams.append(i.get_text())
teams[:10] 

#Extracting the tag having rating for rank1
rating=soup.find_all('div',class_="rankings-block__banner--rating")
rating

#Extracting the text alone from tag
ratings=[]  #Empty list
for i in rating:
    ratings.append(i.get_text())
ratings 

#Extracting the tag for ratings of other players
rating=soup.find_all('td',class_="table-body__cell rating")
rating[:9]

#Extracting the text alone from tags
for i in rating:
    ratings.append(i.get_text())
ratings[:10] 

#Extracting the tags having career best rating for rank1
career_best=soup.find_all('span',class_="rankings-block__career-best-text")
career_best

#Extracting the text alone from tag
career_best_rating=[]  #Empty list
for i in career_best:
    career_best_rating.append(i.get_text().replace('\n', ' ').replace('  ',''))
career_best_rating    

#Extracting the tags for remaining players career best rating
career_best=soup.find_all('td',class_="table-body__cell u-text-right u-hide-phablet")
career_best[:9]

#Extracting the text alone from tags
for i in career_best:
    career_best_rating.append(i.get_text().replace('\n', ' ').replace('                        ',' '))
career_best_rating[:10] 

#Checking the length of each of the list obtained
print(len(names),len(teams),len(ratings),len(career_best_rating))

#Creating a new dataframe with the data we extracted
import pandas as pd
top10=pd.DataFrame({})
top10['Name']=names[:10]
top10['Team']=teams[:10]
top10['Rating']=ratings[:10]
top10['Career_best_rating']=career_best_rating[:10]

#Checking the dataframe
top10<jupyter_output>20 20 20 20<jupyter_text>8. Write a python program to extract information about the local weather from the National Weather Service website of USA, https://www.weather.gov/ for the city, San Francisco. You need to extract data about 7 day extended forecast display for the city. The data should include period, short description, temperature and description.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://forecast.weather.gov/MapClick.php?lat=37.7749&lon=-122.4194#.YFhT9K8zZPY")
page

#Webpage source code
page.content

#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup

#Extracting all the tags having period data
period=soup.find_all('p',class_="period-name")
period

#Extracting the text alone from the tags
Period=[]   #Empty list
for i in period:
    Period.append(i.get_text().replace('Night', ' Night'))
Period    

#Extracting the tags having short description
short_desc=soup.find_all('p',class_="short-desc")
short_desc

#Extracting the text alone from the tags
sd=[]   #Empty list
for i in short_desc:
    sd.append(i.get_text())
sd    

#Extracting the tags having temperature
Temp=soup.find_all('p',class_="temp")
Temp

#Extracting the text alone from the tags
temp=[]  #Empty list
for i in Temp:
    temp.append(i.text)
temp   

#Extracting the tags having long description
desc=soup.find_all('div',class_="col-sm-10 forecast-text")
desc

#Extracting the text from tags
description=[]   #Empty text
for i in desc:
    description.append(i.get_text())
description    

#Checking the length of each of the list obtained
print(len(Period),len(sd),len(temp),len(description))

#Creating a new dataframe with the data we extracted
import pandas as pd
weather=pd.DataFrame({})
weather['Period']=Period
weather['Short_Description']=sd
weather['Temperature']=temp
weather['Detailed_Description']=description[:9]

#Checking the dataframe
weather<jupyter_output>9 9 9 13<jupyter_text>7. Write a python program to scrape details of all the mobile phones under Rs. 20,000 listed on Amazon.in. The scraped data should include Product Name, Price, Image URL and Average Rating.<jupyter_code>#Sending get request to webpage server to get the source code of it
page=requests.get("https://www.amazon.in/s?k=mobile+phones+under+20000&ref=nb_sb_noss")
page
#Webpage source code
page.content
#Parsing the source code by using BeautifulSoup and by html.parser
soup=BeautifulSoup(page.content,'html.parser')
soup
#Extracting all tags having the product name
product=soup.find_all('span',class_="a-size-medium a-color-base a-text-normal")
product
#Extracting the text alone from the tags
Product=[]  #Empty list
for i in product:
    Product.append(i.get_text())
Product    
#Extracting the tags having price
price=soup.find_all('span',class_="a-price-whole")
price
#Extracting the text from tags
Price=[]  #Empty list
for i in price:
    Price.append(i.get_text())
Price   
#Extracting the tags having the Image URL
url=soup.find_all('img',class_="s-image")
url
#Extracting the text alone from tag
Image=[]  #Empty list
for i in url:
    Image.append(i['src'])
Image    
#Extracting the tags having Average Rating
rating=soup.find_all('span',class_="a-icon-alt")
rating
#Extracting the text alone from tag
Rating=[]  #Empty list
for i in rating:
    Rating.append(i.get_text().replace('amp;',' '))
Rating  
#Checking the length of the data obtained
print(len(Product),len(Price),len(Image),len(Rating))
#Creating a new dataframe with the data we extracted
import pandas as pd
mobile=pd.DataFrame({})
mobile['Product_Name']=Product[:15]
mobile['Price']=Price
mobile['Image_Url']=Image[:15]
mobile['Average_Rating']=Rating[:15]
#Checking the dataframe
mobile<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12122311291668178, 'fraction_numerical': 0.017302042148790307, 'mean_word_length': 5.648179406800097, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 13, 'lorem ipsum': 0, 'www.': 13, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '494009', 'license': 'MIT', 'n_tokens_mistral': 9644, 'n_tokens_neox': 9317, 'n_words': 2684, 'path': 'Webscraping/Webscraping_Assignment1.ipynb', 'repo_name': 'Jerish7/FlipRoboTechnologies-Internship', 'size': '27569'}