<jupyter_start><jupyter_text>A Useful PageAn Interesting TitleLorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.<jupyter_code>**html title with tags:**<jupyter_output><empty_output><jupyter_text>bs.title<jupyter_code>**html title without tags or just the string/text:**<jupyter_output><empty_output><jupyter_text>bs.title.get_text()<jupyter_code>**This gives same results:**<jupyter_output><empty_output><jupyter_text>bs.html.title.get_text() from urllib.request import urlopenimport requests  Alternatively, we can use requests library instead of urllibfrom bs4 import BeautifulSouphtml = urlopen('http://www.pythonscraping.com/pages/warandpeace.html') html = requests.get('http://www.pythonscraping.com/pages/page1.html')bs = BeautifulSoup(html, 'html.parser')<jupyter_code><html>
<head>
<style>
.green{
	color:#55ff55;
}
.red{
	color:#ff5555;
}
#text{
	width:50%;
}
</style>
</head>
<body>
<h1>War and Peace</h1>
<h2>Chapter 1</h2>
<div id="text">
"<span class="red">Well, Prince, so Genoa and Lucca are now just family estates of the
Buonapartes. But I warn you, if you don't tell me that this means war,
if you still try to defend the infamies and horrors perpetrated by
that Antichrist- I really believe he is Antichrist- I will have
nothing more to do with you and you are no longer my friend, no longer
my 'faithful slave,' as you call yourself! But how do you do? I see
I have frightened you- sit down and tell me all the news.</span>"
<p/>
It was in July, 1805, and the speaker was the well-known <span class="green">Anna
<NAME></span>, maid of honor and favorite of the <span class="green">Empress Marya
Fedorovna</span>. With these words she greeted <span class="green">Prince <NAME></span>, a man
of high rank and importance, who was the first to arrive at her
reception. <span class="green"><NAME></span> had had a cough for some days. She was, as
she said, suffering from la grippe; grippe being then a new word in
<span class="green">St. Petersburg</span>, used only by the elite.
<p/>
All her invitations without exception, written in French, and
delivered by a scarlet-liveried footman that morning, ran as follows:
<p/>
"<span class="red">If you have nothing better to do, Count [or Prince], and if the
prospect of spending an evening with a poor invalid is not too
terrible, I shall be very charmed to see you tonight between 7 and 10-
<NAME>er.</span>"
<p/>
"<span class="red">Heavens! what a virulent attack!</span>" replied <span class="green">the prince</span>, not in the
least disconcerted by this reception. He had just entered, wearing
an embroidered court uniform, knee breeches, and shoes, and had
stars on his breast and a serene expression on his flat face. He spoke
in that refined French in which our grandfathers not only spoke but
thought, and with the gentle, patronizing intonation natural to a
man of importance who had grown old in society and at court. He went
up to <span class="green"><NAME></span>, kissed her hand, presenting to her his bald,
scented, and shining head, and complacently seated himself on the
sofa.
<p/>
"<span class="red">First of all, dear friend, tell me how you are. Set your friend's
mind at rest,</span>" said he without altering his tone, beneath the
politeness and affected sympathy of which indifference and even
irony could be discerned.
<p/>
"<span class="red">Can one be well while suffering morally? Can one be calm in times
like these if one has any feeling?</span>" said <span class="green"><NAME></span>. "<span class="red">You are
staying the whole evening, I hope?</span>"
<p/>
"<span class="red">And the fete at the English ambassador's? Today is Wednesday. I
must put in an appearance there,</span>" said <span class="green">the prince</span>. "<span class="red">My daughter is
coming for me to take me there.</span>"
<p/>
"<span class="red">I thought today's fete had been canceled. I confess all these
festivities and fireworks are becoming wearisome.</span>"
<p/>
"<span class="red">If they had known that you wished it, the entertainment would
have been put off,</span>" said <span class="green">the prince</span>, who, like a wound-up clock, by
force of habit said things he did not even wish to be believed.
<p/>
"<span class="red">Don't tease! Well, and what has been decided about Novosiltsev's
dispatch? You know everything.</span>"
<p/>
"<span class="red">What can one say about it?</span>" replied <span class="green">the prince</span> in a cold,
listless tone. "<span class="red">What has been decided? They have decided that
Buonaparte has burnt his boats, and I believe that we are ready to
burn ours.</span>"
<p/>
<span class="green">Prince Vasili</span> always spoke languidly, like an actor repeating a
stale part. <span class="green"><NAME></span> Scherer on the contrary, despite her forty
years, overflowed with animation and impulsiveness. To be an
enthusiast had become her social vocation and, sometimes even when she
did not feel like it, she became enthusiastic in order not to
disappoint the expectations of those who knew her. The subdued smile
which, though it did not suit her faded features, always played
round her lips expressed, as in a spoiled child, a continual
consciousness of her charming defect, which she neither wished, nor
could, nor considered it necessary, to correct.
<p/>
In the midst of a conversation on political matters <span class="green"><NAME></span>
burst out:
<p/>
"<span class="red">Oh, don't speak to me of Austria. Perhaps I don't understand
things, but Austria never has wished, and does not wish, for war.
She is betraying us! Russia alone must save Europe. Our gracious
sovereign recognizes his high vocation and will be true to it. That is
the one thing I have faith in! Our good and wonderful sovereign has to
perform the noblest role on earth, and he is so virtuous and noble
that God will not forsake him. He will fulfill his vocation and
crush the hydra of revolution, which has become more terrible than
ever in the person of this murderer and villain! We alone must
avenge the blood of the just one.... Whom, I ask you, can we rely
on?... England with her commercial spirit will not and cannot
understand the Emperor Alexander's loftiness of soul. She has
refused to evacuate Malta. She wanted to find, and still seeks, some
secret motive in our actions. What answer did Novosiltsev get? None.
The English have not understood and cannot understand the
self-abnegation of our Emperor who wants nothing for himself, but only
desires the good of mankind. And what have they promised? Nothing! And
what little they have promised they will not perform! Prussia has
always declared that Buonaparte is invincible, and that all Europe
is powerless before him.... And I don't believe a word that Hardenburg
says, or Haugwitz either. This famous Prussian neutrality is just a
trap. I have faith only in God and the lofty destiny of our adored
monarch. He will save Europe!</span>"
<p/>
She suddenly paused, smiling at her own impetuosity.
<p/>
"<span class="red">I think,</span>" said <span class="green">the prince</span> with a smile, "<span class="red">that if you had been
sent instead of our dear <span class="green">Wintzingerode</span> you would have captured the
<span class="green">King of Prussia</span>'s consent by assault. You are so eloquent. Will you
give me a cup of tea?</span>"
<p/>
"<span class="red">In a moment. A propos,</span>" she added, becoming calm again, "<span class="red">I am
expecting two very interesting men tonight, <span class="green">le <NAME></span>,
who is connected with the <span class="green">Montmorencys</span> through the <span class="green">Rohans</span>, one of
the best French families. He is one of the genuine emigres, the good
ones. And also the <span class="green"><NAME></span>. Do you know that profound thinker? He
has been received by <span class="green">the Emperor</span>. Had you heard?</span>"
<p/>
"<span class="red">I shall be delighted to meet them,</span>" said <span class="green">the prince</span>. "<span class="red">But tell me,</span>"
he added with studied carelessness as if it had only just occurred
to him, though the question he was about to ask was the chief motive
of his visit, "<span class="red">is it true that the Dowager Empress wants <NAME>
to be appointed first secretary at Vienna? The baron by all accounts
is a poor creature.</span>"
<p/>
<span class="green">Prince Vasili</span> wished to obtain this post for his son, but others
were trying through the <span class="green">Dowager Empress <NAME></span> to secure it
for <span class="green">the baron</span>.
<p/>
<span class="green"><NAME></span> almost closed her eyes to indicate that neither she
nor anyone else had a right to criticize what <span class="green">the Empress</span> desired or
was pleased with.
<p/>
"<span class="red"><NAME> has been recommended to the Dowager Empress by her
sister,</span>" was all she said, in a dry and mournful tone.
<p/>
As she named <span class="green">the Empress</span>, <span class="green"><NAME>'s</span> face suddenly assumed an
expression of profound and sincere devotion and respect mingled with
sadness, and this occurred every time she mentioned her illustrious
patroness. She added that <span class="green">Her Majesty</span> had deigned to show <span class="green">Baron
Funke</span>, and again her face clouded over with sadness.
<p/>
<span class="green">The prince</span> was silent and looked indifferent. But, with the
womanly and courtierlike quickness and tact habitual to her, <span class="green">Anna
Pavlovna</span> wished both to rebuke him (for daring to speak he had done of
a man recommended to <span class="green">the Empress</span>) and at the same time to console him,
so she said:
<p/>
"<span class="red">Now about your family. Do you know that since your daughter came
out everyone has been enraptured by her? They say she is amazingly
beautiful.</span>"
<p/>
<span class="green">The prince</span> bowed to signify his respect and gratitude.
<p/>
"<span class="red">I often think,</span>" she continued after a short pause, drawing nearer
to the prince and smiling amiably at him as if to show that
political and social topics were ended and the time had come for
intimate conversation- "<span class="red">I often think how unfairly sometimes the
joys of life are distributed. Why has fate given you two such splendid
children? I don't speak of <span class="green">Anatole</span>, your youngest. I don't like
him,</span>" she added in a tone admitting of no rejoinder and raising her
eyebrows. "<span class="red">Two such charming children. And really you appreciate
them less than anyone, and so you don't deserve to have them.</span>"
<p/>
And she smiled her ecstatic smile.
<p/>
"<span class="red">I can't help it,</span>" said <span class="green">the prince</span>. "<span class="red">Lavater would have said I
lack the bump of paternity.</span>"
<p/>
"<span class="red">Don't joke; I mean to have a serious talk with you. Do you know I
am dissatisfied with your younger son? Between ourselves</span>" (and her
face assumed its melancholy expression), "<span class="red">he was mentioned at Her
Majesty's and you were pitied....</span>"
<p/>
<span class="green">The prince</span> answered nothing, but she looked at him significantly,
awaiting a reply. He frowned.
<p/>
"<span class="red">What would you have me do?</span>" he said at last. "<span class="red">You know I did all
a father could for their education, and they have both turned out
fools. Hippolyte is at least a quiet fool, but Anatole is an active
one. That is the only difference between them.</span>" He said this smiling
in a way more natural and animated than usual, so that the wrinkles
round his mouth very clearly revealed something unexpectedly coarse
and unpleasant.
<p/>
"<span class="red">And why are children born to such men as you? If you were not a
father there would be nothing I could reproach you with,</span>" said <span class="green"><NAME></span>, looking up pensively.
<p/>
"<span class="red">I am your faithful slave and to you alone I can confess that my
children are the bane of my life. It is the cross I have to bear. That
is how I explain it to myself. It can't be helped!</span>"
<p/>
He said no more, but expressed his resignation to cruel fate by a
gesture. <span class="green"><NAME></span> meditated.
</div>
</body>
</html><jupyter_output><empty_output><jupyter_text>**Using .findAll(), find all span tags with a CSS class='green'**<jupyter_code>help(bs.findAll)
green_text = bs.findAll('span', class_='green') # or green_text = bs.findAll('span', {'class': 'green'})
type(green_text)
len(green_text)
green_text[0]
for i, text in enumerate(green_text):
    print(i, text.get_text())<jupyter_output>0 Anna
<NAME>
1 Empress Marya
Fedorovna
2 <NAME>
3 <NAME>
4 St. Petersburg
5 the prince
6 <NAME>
7 <NAME>
8 the prince
9 the prince
10 the prince
11 <NAME>
12 <NAME>
13 <NAME>
14 the prince
15 Wintzingerode
16 King of Prussia
17 le Vicomte de Mortemart
18 Montmorencys
19 Rohans
20 <NAME>
21 the Emperor
22 the prince
23 Pr<NAME>
24 Dowager Empress <NAME>
25 the baron
26 <NAME>
27 the Empress
28 the Empress
29 <NAME>'s
30 Her Majesty
31 Baron
Funke
32 The prince
33 Anna
Pavlovna
34 the Empress
35 The prince
36 Anatole
37 the prince
38 The prince
39 Anna
Pavlovna
40 <NAME><jupyter_text>**It appears there is a new line character in a few of the results.  Let's check the first result:**<jupyter_code>green_text[0].get_text()<jupyter_output><empty_output><jupyter_text>**Let's remove new line character from our results:**<jupyter_code>for i, text in enumerate(green_text):
    print(i, text.get_text().replace('\n', ''))<jupyter_output>0 <NAME>
1 Empress MaryaFedorovna
2 <NAME>
3 <NAME>
4 St. Petersburg
5 the prince
6 <NAME>
7 <NAME>
8 the prince
9 the prince
10 the prince
11 <NAME>
12 <NAME>
13 <NAME>
14 the prince
15 Wintzingerode
16 King of Prussia
17 le Vicomte de Mortemart
18 Montmorencys
19 Rohans
20 <NAME>
21 the Emperor
22 the prince
23 Prince Vasili
24 Dowager Empress <NAME>
25 the baron
26 <NAME>
27 the Empress
28 the Empress
29 <NAME>'s
30 Her Majesty
31 BaronFunke
32 The prince
33 AnnaPavlovna
34 the Empress
35 The prince
36 Anatole
37 the prince
38 The prince
39 AnnaPavlovna
40 <NAME><jupyter_text>**Although, for the purpose of this exercise, I didn't need to remove the new line character, but you'll encounter white spaces and so this is one way to deal with them.**  Let's get all text Looking at the HTML above, we see that all the text is under a single ```div``` tag and with attribute ```id``` equal to 'text'<jupyter_code>all_divs = bs.findAll('div', id='text')
type(all_divs)
len(all_divs)
all_divs[0].get_text()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.096932028563872, 'fraction_numerical': 0.011306532663316583, 'mean_word_length': 5.030701754385965, 'pattern_counts': {'":': 0, '<': 273, '<?xml version=': 0, '>': 273, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '235161', 'license': 'MIT', 'n_tokens_mistral': 4645, 'n_tokens_neox': 4427, 'n_words': 2219, 'path': 'web_scraping/beautifulsoup/BeautifulSoup_Learning_Guide.ipynb', 'repo_name': 'manual123/Nacho-Jupyter-Notebooks', 'size': '15460'}