<jupyter_start><jupyter_text>Chapter 6 -  Data Sourcing via Web Segment 3 - Data parsing<jupyter_code>from bs4 import BeautifulSoup

import urllib
import urllib.request
import re
with urllib.request.urlopen('https://raw.githubusercontent.com/BigDataGal/Data-Mania-Demos/master/IoT-2018.html') as response:
    html = response.read()
soup = BeautifulSoup(html, "lxml")
type(soup)<jupyter_output><empty_output><jupyter_text>Parsing your data<jupyter_code>print(soup.prettify()[0:100])<jupyter_output><html>
 <head>
  <title>
   IoT Articles
  </title>
 </head>
 <body>
  <p class="title">
   <b><jupyter_text>Getting data from a parse tree<jupyter_code>text_only = soup.get_text()
print(text_only)<jupyter_output>IoT Articles

2018 Trends: Best New IoT Device Ideas for Data Scientists and Engineers
It’s almost 2018 and IoT is on the cusp of an explosive expansion. In this article, I offer you a listing of new IoT device ideas that you can use...


It’s almost 2018 and IoT is on the cusp of an explosive expansion. In this article, I offer you a listing of new IoT device ideas that you can use to get practice in designing your first IoT applications.
Looking Back at My Coolest IoT Find in 2017
Before going into detail about best new IoT device ideas, here’s the backstory. Last month Ericsson Digital invited me to tour the Ericsson Studio in Kista, Sweden. Up until that visit, IoT had been largely theoretical to me. Of course, I know the usual mumbo-jumbo about wearables and IoT-connected fitness trackers. That stuff is all well and good, but it’s somewhat old hat – plus I am not sure we are really benefiting so much from those, so I’m not that impressed.

It wasn’t until I got to the Ericss[...]<jupyter_text>Searching and retrieving data from a parse tree  Retrieving tags by filtering with name arguments<jupyter_code>soup.find_all("li")<jupyter_output><empty_output><jupyter_text>Retrieving tags by filtering with keyword arguments<jupyter_code>soup.find_all(id="link 7")<jupyter_output><empty_output><jupyter_text>Retrieving tags by filtering with string arguments<jupyter_code>soup.find_all('ol')<jupyter_output><empty_output><jupyter_text>Retrieving tags by filtering with list objects<jupyter_code>soup.find_all(['ol', 'b'])<jupyter_output><empty_output><jupyter_text>Retrieving tags by filtering with regular expressions<jupyter_code>t = re.compile("t")
for tag in soup.find_all(t):
    print(tag.name)<jupyter_output>html
title
strong
strong
strong
strong
strong
strong<jupyter_text>Retrieving tags by filtering with a Boolean value<jupyter_code>for tag in soup.find_all(True):
    print(tag.name)<jupyter_output>html
head
title
body
p
b
p
br
br
h1
span
strong
a
a
a
img
a
span
strong
a
h1
ol
li
strong
li
strong
li
strong
h1
a
a
a
h2
ol
li
li
li
li
li
li
h2
ol
li
li
li
li
li
a
img
h2
ol
li
li
li
li
li
h2
ol
li
li
li
li
span
strong
a
em
p<jupyter_text>Retrieving weblinks by filtering with string objects<jupyter_code>for link in soup.find_all('a'):
    print(link.get('href'))<jupyter_output>http://bit.ly/LPlNDJj
http://www.data-mania.com/blog/m2m-vs-iot/
bit.ly/LPlNDJj
http://mat.se/
http://bit.ly/LPlNDJj
https://click.linksynergy.com/deeplink?id=*JDLXjeE*wk&mid=39197&murl=https%3A%2F%2Fwww.udemy.com%2Ftopic%2Finternet-of-things%2F%3Fsort%3Dhighest-rated
http://www.skyfilabs.com/iot-online-courses
https://www.coursera.org/specializations/iot
bit.ly/LPlNDJj
http://bit.ly/LPlNDJj<jupyter_text>Retrieving strings by filtering with regular expressions<jupyter_code>soup.find_all(string=re.compile("data"))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0920794850265883, 'fraction_numerical': 0.01371396585502379, 'mean_word_length': 5.643122676579925, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 3, 'lorem ipsum': 0, 'www.': 4, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '130927', 'license': 'MIT', 'n_tokens_mistral': 1356, 'n_tokens_neox': 1280, 'n_words': 384, 'path': 'Pt_1/06_03_Data_parsing/06_03_end.ipynb', 'repo_name': 'vladcuevas/Python-Data-Science-Labs', 'size': '3573'}