<jupyter_start><jupyter_text>NLTK ![nltk.png](attachment:nltk.png)  1-Import NLTK ✔<jupyter_code>!pip install nltk
import nltk
text=""" Perhaps one of the most significant advances made by Arabic mathematics began at this time with the work of al-Khwarizmi, namely the beginnings of algebra. It is important to understand just how significant this new idea was. It was a revolutionary move away from the Greek concept of mathematics which was essentially geometry. Algebra was a unifying theory which allowed rational numbers, irrational numbers, geometrical magnitudes, etc., to all be treated as "algebraic objects". It gave mathematics a whole new development path so much broader in concept to that which had existed before, and provided a vehicle for future development of the subject. Another important aspect of the introduction of algebraic ideas was that it allowed mathematics to be applied to itself in a way which had not happened before."""<jupyter_output><empty_output><jupyter_text>2-Tokenization ✔<jupyter_code>tokens = nltk.word_tokenize(text)
tokens<jupyter_output><empty_output><jupyter_text>3-POS-tagging✔<jupyter_code>tagged = nltk.pos_tag(tokens)

tagged<jupyter_output><empty_output><jupyter_text>4-Stemming✔<jupyter_code>from nltk.stem import PorterStemmer
stemmer= PorterStemmer()
for word in tokens :
    print(f"{word} >>> {stemmer.stem(word)}")<jupyter_output>Perhaps >>> perhap
one >>> one
of >>> of
the >>> the
most >>> most
significant >>> signific
advances >>> advanc
made >>> made
by >>> by
Arabic >>> arab
mathematics >>> mathemat
began >>> began
at >>> at
this >>> thi
time >>> time
with >>> with
the >>> the
work >>> work
of >>> of
al-Khwarizmi >>> al-khwarizmi
, >>> ,
namely >>> name
the >>> the
beginnings >>> begin
of >>> of
algebra >>> algebra
. >>> .
It >>> it
is >>> is
important >>> import
to >>> to
understand >>> understand
just >>> just
how >>> how
significant >>> signific
this >>> thi
new >>> new
idea >>> idea
was >>> wa
. >>> .
It >>> it
was >>> wa
a >>> a
revolutionary >>> revolutionari
move >>> move
away >>> away
from >>> from
the >>> the
Greek >>> greek
concept >>> concept
of >>> of
mathematics >>> mathemat
which >>> which
was >>> wa
essentially >>> essenti
geometry >>> geometri
. >>> .
Algebra >>> algebra
was >>> wa
a >>> a
unifying >>> unifi
theory >>> theori
which >>> which
allowed >>> allow
rational >>> ration
numbers >>> [...]<jupyter_text>5- Lemmatization✔<jupyter_code>from nltk.stem import WordNetLemmatizer
lemmatizer=WordNetLemmatizer()
for word in tokens:
    print(f"{word} >> {lemmatizer.lemmatize(word)}")<jupyter_output>Perhaps >> Perhaps
one >> one
of >> of
the >> the
most >> most
significant >> significant
advances >> advance
made >> made
by >> by
Arabic >> Arabic
mathematics >> mathematics
began >> began
at >> at
this >> this
time >> time
with >> with
the >> the
work >> work
of >> of
al-Khwarizmi >> al-Khwarizmi
, >> ,
namely >> namely
the >> the
beginnings >> beginning
of >> of
algebra >> algebra
. >> .
It >> It
is >> is
important >> important
to >> to
understand >> understand
just >> just
how >> how
significant >> significant
this >> this
new >> new
idea >> idea
was >> wa
. >> .
It >> It
was >> wa
a >> a
revolutionary >> revolutionary
move >> move
away >> away
from >> from
the >> the
Greek >> Greek
concept >> concept
of >> of
mathematics >> mathematics
which >> which
was >> wa
essentially >> essentially
geometry >> geometry
. >> .
Algebra >> Algebra
was >> wa
a >> a
unifying >> unifying
theory >> theory
which >> which
allowed >> allowed
rational >> rational
numbers >> number
, >> ,
irrational >> [...]<jupyter_text>6-Identify named entities✔<jupyter_code>entities = nltk.chunk.ne_chunk(tagged)
entities<jupyter_output>The Ghostscript executable isn't found.
See http://web.mit.edu/ghostscript/www/Install.htm
If you're using a Mac, you can try installing
https://docs.brew.sh/Installation then `brew install ghostscript`<jupyter_text>7-Parsing✔<jupyter_code>from nltk.corpus import treebank
import sys
grammar="NP: {<DT>?<JJ>*<NN>}"
parser=nltk.RegexpParser(grammar)
output=parser.parse(tagged)
output.draw()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1357815897559797, 'fraction_numerical': 0.0016912297656438754, 'mean_word_length': 5.061493411420205, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 368, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '538869', 'license': 'MIT', 'n_tokens_mistral': 1346, 'n_tokens_neox': 1281, 'n_words': 449, 'path': 'nltk.ipynb', 'repo_name': 'Hind-EL-BANANE1/NLP-TP', 'size': '4139'}