<jupyter_start><jupyter_text>TP1 NLTK Natural Language ToolkitThe Natural Language Toolkit, or more commonly NLTK, is a suite of libraries and programs for symbolic and statistical natural language processing (NLP) for English written in the Python programming language.<jupyter_code>## install nltk
!pip install nltk
import nltk
 
from nltk.tokenize import sent_tokenize, word_tokenize,PunktSentenceTokenizer


arabic_text="""ربما كانت أحد أهم التطورات التي قامت بها الرياضيات العربية التي بدأت في هذا الوقت بعمل الخوارزمي و هي بدايات الجبر، و من المهم فهم كيف كانت هذه الفكرة الجديدة مهمة، فقد كانت خطوة ثورية بعيدا عن المفهوم اليوناني للرياضيات التي هي في جوهرها هندسة، الجبر كان نظرية موحدة تتيح الأعداد الكسرية و الأعداد اللا كسرية، و قدم وسيلة للتنمية في هذا الموضوع مستقبلا. و جانب آخر مهم لإدخال أفكار الجبر و هو أنه سمح بتطبيق الرياضيات على نفسها بطريقة لم تحدث من قبل"""

english_text= """Perhaps one of the most significant advances made by Arabic mathematics began at this time with the work of al-Khwarizmi, namely 
the beginnings of algebra. It is important to understand just how significant this new idea was. It was a revolutionary move away from 
the Greek concept of mathematics which was essentially geometry. Algebra was a unifying theory which allowed rational 
numbers, irrational numbers, geometrical magnitudes, etc., to all be treated as "algebraic objects". It gave mathematics a whole new 
development path so much broader in concept to that which had existed before, and provided a vehicle for future development of the 
subject. Another important aspect of the introduction of algebraic ideas was that it allowed mathematics to be applied to itself in a 
way which had not happened before.
"""<jupyter_output><empty_output><jupyter_text>***Tokenize text by Syllable***<jupyter_code>tk = nltk.SyllableTokenizer()
print(tk.tokenize(english_text))
 
tk = nltk.SyllableTokenizer()
print(tk.tokenize(arabic_text))<jupyter_output>['رب', 'م', 'ا', ' ', 'ك', 'ا', 'ن', 'ت', ' ', 'أ', 'ح', 'د', ' ', 'أ', 'ه', 'م', ' ', 'ا', 'ل', 'ت', 'ط', 'و', 'ر', 'ا', 'ت', ' ', 'ا', 'ل', 'ت', 'ي', ' ', 'ق', 'ا', 'م', 'ت', ' ', 'ب', 'ه', 'ا', ' ', 'ا', 'ل', 'ر', 'ي', 'ا', 'ض', 'ي', 'ا', 'ت', ' ', 'ا', 'ل', 'ع', 'ر', 'ب', 'ي', 'ة', ' ', 'ا', 'ل', 'ت', 'ي', ' ', 'ب', 'د', 'أ', 'ت', ' ', 'ف', 'ي', ' ', 'ه', 'ذ', 'ا', ' ', 'ا', 'ل', 'و', 'ق', 'ت', ' ', 'ب', 'ع', 'م', 'ل', ' ', 'ا', 'ل', 'خ', 'و', 'ا', 'ر', 'ز', 'م', 'ي', ' ', 'و', ' ', 'ه', 'ي', ' ', 'ب', 'د', 'ا', 'ي', 'ا', 'ت', ' ', 'ا', 'ل', 'ج', 'ب', 'ر', '،', ' ', 'و', ' ', 'م', 'ن', ' ', 'ا', 'ل', 'م', 'ه', 'م', ' ', 'ف', 'ه', 'م', ' ', 'ك', 'ي', 'ف', ' ', 'ك', 'ا', 'ن', 'ت', ' ', 'ه', 'ذ', 'ه', ' ', 'ا', 'ل', 'ف', 'ك', 'ر', 'ة', ' ', 'ا', 'ل', 'ج', 'د', 'ي', 'د', 'ة', ' ', 'م', 'ه', 'م', 'ة', '،', ' ', 'ف', 'ق', 'د', ' ', 'ك', 'ا', 'ن', 'ت', ' ', 'خ', 'ط', 'و', 'ة', ' ', 'ث', 'و', 'ر', 'ي', 'ة', ' ', 'ب', 'ع', 'ي', 'د', 'ا', ' ', 'ع', 'ن', ' ', 'ا', 'ل', 'م', 'ف', 'ه', 'و', 'م'[...]<jupyter_text>tokenize text by word<jupyter_code>tokenize english text by word<jupyter_output><empty_output><jupyter_text>import nltkfrom nltk.tokenize import sent_tokenize, word_tokenize,PunktSentenceTokenizerenglish_text= """Perhaps one of the most significant advances made by Arabic mathematics began at this time with the work of al-Khwarizmi, namely the beginnings of algebra. It is important to understand just how significant this new idea was. It was a revolutionary move away from the Greek concept of mathematics which was essentially geometry. Algebra was a unifying theory which allowed rational numbers, irrational numbers, geometrical magnitudes, etc., to all be treated as "algebraic objects". It gave mathematics a whole new development path so much broader in concept to that which had existed before, and provided a vehicle for future development of the subject. Another important aspect of the introduction of algebraic ideas was that it allowed mathematics to be applied to itself in a way which had not happened before."""for i in range(len(word_tokenize((english_text)))) :   print(word_tokenize((english_text))[i])<jupyter_code># tokenize arabic text by word<jupyter_output><empty_output><jupyter_text>import nltkfrom nltk.tokenize import sent_tokenize, word_tokenize,PunktSentenceTokenizerarabic_text="""ربما كانت أحد أهم التطورات التي قامت بها الرياضيات العربية التي بدأت في هذا الوقت بعمل الخوارزمي و هي بدايات الجبر، و من المهم فهم كيف كانت هذه الفكرة الجديدة مهمة، فقد كانت خطوة ثورية بعيدا عن المفهوم اليوناني للرياضيات التي هي في جوهرها هندسة، الجبر كان نظرية موحدة تتيح الأعداد الكسرية و الأعداد اللا كسرية، و قدم وسيلة للتنمية في هذا الموضوع مستقبلا. و جانب آخر مهم لإدخال أفكار الجبر و هو أنه سمح بتطبيق الرياضيات على نفسها بطريقة لم تحدث من قبل"""for i in range(len(word_tokenize((arabic_text)))) :   print(word_tokenize((arabic_text))[i])<jupyter_code># Sentence Tokenizing<jupyter_output><empty_output><jupyter_text>----------------------tokenize english text by sentence<jupyter_code>for i in range(len(sent_tokenize(english_text))) :
     print(sent_tokenize(english_text)[i])
     print()<jupyter_output>Perhaps one of the most significant advances made by Arabic mathematics began at this time with the work of al-Khwarizmi, namely 
the beginnings of algebra.

It is important to understand just how significant this new idea was.

It was a revolutionary move away from 
the Greek concept of mathematics which was essentially geometry.

Algebra was a unifying theory which allowed rational 
numbers, irrational numbers, geometrical magnitudes, etc., to all be treated as "algebraic objects".

It gave mathematics a whole new 
development path so much broader in concept to that which had existed before, and provided a vehicle for future development of the 
subject.

Another important aspect of the introduction of algebraic ideas was that it allowed mathematics to be applied to itself in a 
way which had not happened before.<jupyter_text>----------------------tokenize arabic text by sentence<jupyter_code>for i in range(len(sent_tokenize(arabic_text))) :
     print(sent_tokenize(arabic_text)[i])
     print()<jupyter_output>ربما كانت أحد أهم التطورات التي قامت بها الرياضيات العربية التي بدأت في هذا الوقت بعمل الخوارزمي و هي بدايات الجبر، و من المهم فهم كيف كانت هذه الفكرة الجديدة مهمة، فقد كانت خطوة ثورية بعيدا عن المفهوم اليوناني للرياضيات التي هي في جوهرها هندسة، الجبر كان نظرية موحدة تتيح الأعداد الكسرية و الأعداد اللا كسرية، و قدم وسيلة للتنمية في هذا الموضوع مستقبلا.

و جانب آخر مهم لإدخال أفكار الجبر و هو أنه سمح بتطبيق الرياضيات على نفسها بطريقة لم تحدث من قبل<jupyter_text>Filtering  Stop Words<jupyter_code>nltk.download("stopwords")<jupyter_output>[nltk_data] Downloading package stopwords to
[nltk_data]     C:\Users\DELL\AppData\Roaming\nltk_data...
[nltk_data]   Package stopwords is already up-to-date!<jupyter_text>NLTK has by default a bunch of words that it considers to be stop words. It can be accessed via the NLTK corpus with:<jupyter_code>from nltk.corpus import stopwords<jupyter_output><empty_output><jupyter_text>Example to incorporate the stop_words set to remove the stop words from a given text:<jupyter_code>english_text= """Perhaps one of the most significant advances made by Arabic mathematics began at this time with the work of al-Khwarizmi, namely 
the beginnings of algebra. It is important to understand just how significant this new idea was. It was a revolutionary move away from 
the Greek concept of mathematics which was essentially geometry. Algebra was a unifying theory which allowed rational 
numbers, irrational numbers, geometrical magnitudes, etc., to all be treated as "algebraic objects". It gave mathematics a whole new 
development path so much broader in concept to that which had existed before, and provided a vehicle for future development of the 
subject. Another important aspect of the introduction of algebraic ideas was that it allowed mathematics to be applied to itself in a 
way which had not happened before.
"""
words = nltk.word_tokenize(english_text)
print("Unfiltered: ", words)

stopwords = nltk.corpus.stopwords.words("english")

cleaned = [word for word in words if word not in stopwords]
print("Filtered: ", cleaned)<jupyter_output>Unfiltered:  ['Perhaps', 'one', 'of', 'the', 'most', 'significant', 'advances', 'made', 'by', 'Arabic', 'mathematics', 'began', 'at', 'this', 'time', 'with', 'the', 'work', 'of', 'al-Khwarizmi', ',', 'namely', 'the', 'beginnings', 'of', 'algebra', '.', 'It', 'is', 'important', 'to', 'understand', 'just', 'how', 'significant', 'this', 'new', 'idea', 'was', '.', 'It', 'was', 'a', 'revolutionary', 'move', 'away', 'from', 'the', 'Greek', 'concept', 'of', 'mathematics', 'which', 'was', 'essentially', 'geometry', '.', 'Algebra', 'was', 'a', 'unifying', 'theory', 'which', 'allowed', 'rational', 'numbers', ',', 'irrational', 'numbers', ',', 'geometrical', 'magnitudes', ',', 'etc.', ',', 'to', 'all', 'be', 'treated', 'as', '``', 'algebraic', 'objects', "''", '.', 'It', 'gave', 'mathematics', 'a', 'whole', 'new', 'development', 'path', 'so', 'much', 'broader', 'in', 'concept', 'to', 'that', 'which', 'had', 'existed', 'before', ',', 'and', 'provided', 'a', 'vehicle', 'for', 'future', 'development[...]<jupyter_text>You can observe the differences between the two lists Unfiltered & Filtered<jupyter_code><jupyter_output><empty_output><jupyter_text>Stemming  English Text<jupyter_code># importing modules
from nltk.stem import PorterStemmer
from nltk.tokenize import word_tokenize
  
ps = PorterStemmer()

words = word_tokenize(english_text)
  
for w in words:
    print(w, " : ", ps.stem(w))<jupyter_output>Perhaps  :  perhap
one  :  one
of  :  of
the  :  the
most  :  most
significant  :  signific
advances  :  advanc
made  :  made
by  :  by
Arabic  :  arab
mathematics  :  mathemat
began  :  began
at  :  at
this  :  thi
time  :  time
with  :  with
the  :  the
work  :  work
of  :  of
al-Khwarizmi  :  al-khwarizmi
,  :  ,
namely  :  name
the  :  the
beginnings  :  begin
of  :  of
algebra  :  algebra
.  :  .
It  :  it
is  :  is
important  :  import
to  :  to
understand  :  understand
just  :  just
how  :  how
significant  :  signific
this  :  thi
new  :  new
idea  :  idea
was  :  wa
.  :  .
It  :  it
was  :  wa
a  :  a
revolutionary  :  revolutionari
move  :  move
away  :  away
from  :  from
the  :  the
Greek  :  greek
concept  :  concept
of  :  of
mathematics  :  mathemat
which  :  which
was  :  wa
essentially  :  essenti
geometry  :  geometri
.  :  .
Algebra  :  algebra
was  :  wa
a  :  a
unifying  :  unifi
theory  :  theori
which  :  which
allowed  :  allow
rational  :  ration
numbers  :  [...]<jupyter_text>Stemming Arabic Text<jupyter_code># importing modules
from nltk.stem import PorterStemmer
from nltk.tokenize import word_tokenize
  
ps = PorterStemmer()

words = word_tokenize(arabic_text)
  
for w in words:
    print(w, " : ", ps.stem(w))<jupyter_output>ربما  :  ربما
كانت  :  كانت
أحد  :  أحد
أهم  :  أهم
التطورات  :  التطورات
التي  :  التي
قامت  :  قامت
بها  :  بها
الرياضيات  :  الرياضيات
العربية  :  العربية
التي  :  التي
بدأت  :  بدأت
في  :  في
هذا  :  هذا
الوقت  :  الوقت
بعمل  :  بعمل
الخوارزمي  :  الخوارزمي
و  :  و
هي  :  هي
بدايات  :  بدايات
الجبر،  :  الجبر،
و  :  و
من  :  من
المهم  :  المهم
فهم  :  فهم
كيف  :  كيف
كانت  :  كانت
هذه  :  هذه
الفكرة  :  الفكرة
الجديدة  :  الجديدة
مهمة،  :  مهمة،
فقد  :  فقد
كانت  :  كانت
خطوة  :  خطوة
ثورية  :  ثورية
بعيدا  :  بعيدا
عن  :  عن
المفهوم  :  المفهوم
اليوناني  :  اليوناني
للرياضيات  :  للرياضيات
التي  :  التي
هي  :  هي
في  :  في
جوهرها  :  جوهرها
هندسة،  :  هندسة،
الجبر  :  الجبر
كان  :  كان
نظرية  :  نظرية
موحدة  :  موحدة
تتيح  :  تتيح
الأعداد  :  الأعداد
الكسرية  :  الكسرية
و  :  و
الأعداد  :  الأعداد
اللا  :  اللا
كسرية،  :  كسرية،
و  :  و
قدم  :  قدم
وسيلة  :  وسيلة
للتنمية  :  للتنمية
في  :  في
هذا  :  هذا
الموضوع  :  الموضوع
مستقبلا  :  مستقبلا
.  :  .
و  :  و
جانب  :  جانب
آخر  :[...]<jupyter_text>Tagging Parts of Speech<jupyter_code>eng_pos_tag = nltk.pos_tag(word_tokenize(english_text))
nltk.help.upenn_tagset()
en_pos_tag = nltk.pos_tag(word_tokenize(english_text))<jupyter_output>$: dollar
    $ -$ --$ A$ C$ HK$ M$ NZ$ S$ U.S.$ US$
'': closing quotation mark
    ' ''
(: opening parenthesis
    ( [ {
): closing parenthesis
    ) ] }
,: comma
    ,
--: dash
    --
.: sentence terminator
    . ! ?
:: colon or ellipsis
    : ; ...
CC: conjunction, coordinating
    & 'n and both but either et for less minus neither nor or plus so
    therefore times v. versus vs. whether yet
CD: numeral, cardinal
    mid-1890 nine-thirty forty-two one-tenth ten million 0.5 one forty-
    seven 1987 twenty '79 zero two 78-degrees eighty-four IX '60s .025
    fifteen 271,124 dozen quintillion DM2,000 ...
DT: determiner
    all an another any both del each either every half la many much nary
    neither no some such that the them these this those
EX: existential there
    there
FW: foreign word
    gemeinschaft hund ich jeux habeas <NAME> K'ang-si vous
    lutihaw alai je jour objets salutaris fille quibusdam pas trop Monte
    terram fiche oui corporis ...
IN: preposition or[...]<jupyter_text>Lemmatizing<jupyter_code>from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
lemmatized_words = [lemmatizer.lemmatize(word) for word in word_tokenize(english_text)]
print(lemmatized_words)<jupyter_output>['Perhaps', 'one', 'of', 'the', 'most', 'significant', 'advance', 'made', 'by', 'Arabic', 'mathematics', 'began', 'at', 'this', 'time', 'with', 'the', 'work', 'of', 'al-Khwarizmi', ',', 'namely', 'the', 'beginning', 'of', 'algebra', '.', 'It', 'is', 'important', 'to', 'understand', 'just', 'how', 'significant', 'this', 'new', 'idea', 'wa', '.', 'It', 'wa', 'a', 'revolutionary', 'move', 'away', 'from', 'the', 'Greek', 'concept', 'of', 'mathematics', 'which', 'wa', 'essentially', 'geometry', '.', 'Algebra', 'wa', 'a', 'unifying', 'theory', 'which', 'allowed', 'rational', 'number', ',', 'irrational', 'number', ',', 'geometrical', 'magnitude', ',', 'etc.', ',', 'to', 'all', 'be', 'treated', 'a', '``', 'algebraic', 'object', "''", '.', 'It', 'gave', 'mathematics', 'a', 'whole', 'new', 'development', 'path', 'so', 'much', 'broader', 'in', 'concept', 'to', 'that', 'which', 'had', 'existed', 'before', ',', 'and', 'provided', 'a', 'vehicle', 'for', 'future', 'development', 'of', 'the', 'subject[...]<jupyter_text>mplementation: Chunking in NLP using nltk let us try to extract all the noun phrases from a sentence using the steps defined above. First, we’ll import the required libraries and then tokenize the sentence before applying POS_tagging to it.<jupyter_code># Importing the required libraries
from nltk import pos_tag
from nltk import word_tokenize
from nltk import RegexpParser
# english_text
english_text= """Perhaps one of the most significant advances made by Arabic mathematics began at this time with the work of al-Khwarizmi, namely 
the beginnings of algebra. It is important to understand just how significant this new idea was. It was a revolutionary move away from 
the Greek concept of mathematics which was essentially geometry. Algebra was a unifying theory which allowed rational 
numbers, irrational numbers, geometrical magnitudes, etc., to all be treated as "algebraic objects". It gave mathematics a whole new 
development path so much broader in concept to that which had existed before, and provided a vehicle for future development of the 
subject. Another important aspect of the introduction of algebraic ideas was that it allowed mathematics to be applied to itself in a 
way which had not happened before.
"""
# Splitiing the sentence into words
list_of_words = word_tokenize(english_text)
# Applying POS_tagging
tagged_words = pos_tag(list_of_words)<jupyter_output><empty_output><jupyter_text>We then define our chunk keeping in mind that our aim is to extract all the noun phrases present in our sentence.<jupyter_code># Extracting the Noun Phrases
chunk_to_be_extracted = r''' Chunk: {<DT>*<NNP>*<NN>*} '''
# Applying chunking to the text
chunkParser = nltk.chunk.RegexpParser(chunk_to_be_extracted)
chunked_sentence = chunkParser.parse(tagged_words)<jupyter_output><empty_output><jupyter_text>The ‘chunked_sentence’ variable is an NLTK tree which can be viewed using the draw() method.<jupyter_code># To view the NLTK tree
chunked_sentence.draw()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13336999022482893, 'fraction_numerical': 0.0018328445747800588, 'mean_word_length': 4.4929530201342285, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 68, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '511100', 'license': 'MIT', 'n_tokens_mistral': 6810, 'n_tokens_neox': 6231, 'n_words': 2031, 'path': 'nltk .ipynb', 'repo_name': 'Aoutman-ait-mbarK/nltk', 'size': '16378'}