<jupyter_start><jupyter_text>FUNCIONES DE AYUDA AL PROCESADO DE TEXTO  MLearner**MLearner** pretende ser una libreria de herramientas utiles para desarrollar algoritmos de Machine Learning e IA de manera mas facil e intuitiva. El desarrollo de esta libreria me ha servido para adquirir conocimientos de creacion de un proyecto de desarrollo software:- Integracion Continua y Despliegue Continuo.- Gestion de Repositorio a nivel de proyecto.- Gestion de Repositorio OpenSource.- Clean Code en desarrollo software.- Frameworks Machine Learning y Deep Learning.- Automatizacion de testing.- Documentacion de codigo.- Documentacion de la Libreria entorno Web.- Empaquetacion y publicacion en Pypi. ![mlearner_web.png](images/mlearner_web.png)  PyPIPara instalar **MLearner** ejecute:```pythonpip install mlearner```Como alternativa, puede descargar el paquete directamente desde PyPI https://pypi.python.org/pypi/mlearner, posteriormente desarchivelo y navegue hasta la ruta del paquete, una vez alli ejecute el siguiente comando:```pythonpython setup.py install```  Links- **Documentation:** https://jaisenbe58r.github.io/MLearner/- **Source code repository:** https://github.com/jaisenbe58r/MLearner- **PyPI:** https://pypi.python.org/pypi/mlearner ***  Text Helper FunctionsImplementation of functions for Natural language Processing> from mlearner.nlp.helpers import *  Overview HelpersList of implemented modules:- URL- Emoticons- Email- Hash- Mention- Number- Phone Number- Year- Non Alphanumeric- Punctuations- Repetitive Character- Dollar- Number-Greater- Number-Lesser- Dates- Only Words- Only Numbers- Boundaries- Search- Pick Sentence- Duplicate Sentence- Caps Words- Length of Words- Length of Characters- Get ID- Specific String Rows- Hex code to Color- Tags- IP Address- Mac Address- Subword- Latitude & Longitude- PAN- Phone Number Country Code- Domain  Importacion de Librerias<jupyter_code>import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from mlearner import nlp 
from mlearner.preprocessing import DataAnalyst
from mlearner.utils import keras_checkpoint

import emoji

%load_ext autoreload
%autoreload 2

%matplotlib inline<jupyter_output>The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload<jupyter_text>Carga del dataset**SMS Spam Collection v.1**The SMS Spam Collection v.1 (hereafter the corpus) is a set of SMS tagged messages that have been collected for SMS Spam research. It contains one set of SMS messages in English of 5,574 messages, tagged acording being ham (legitimate) or spam.The Grumbletext Web site is: http://www.grumbletext.co.uk/<jupyter_code>file = "data/SMSSpamCollection.txt"
file_csv = "data/SMSSpamCollection.csv"

if not os.path.isfile(file_csv):
    text = nlp.open_txt(file).replace("\t", "\n").split("\n")
    del text[-1] # Se borra la linea del final

    d = {'target': text[0::2], 'text': text[1::2]}
    sms = pd.DataFrame(d)
    sms.to_csv("data/SMSSpamCollection.csv", index=False)
else:
    sms = pd.read_csv(file_csv)

sms.head(5)
dataset = DataAnalyst.load_dataframe(sms)
dataset.distribution_targets(target=["target"])<jupyter_output><empty_output><jupyter_text>***  Procesado del Texto<jupyter_code>data_clean = dataset.data.copy()<jupyter_output><empty_output><jupyter_text>URLs* **find_url(text)**: Busqueda de URLs en el texto<jupyter_code>from mlearner.nlp.helpers import find_url
data_clean['url'] = dataset.data['text'].apply(lambda x : find_url(x))
data_clean[data_clean["url"]!=""][["target", "url"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Emojis* **find_emoji(text)**: Busqueda de emojis en el texto.* **remove_emoji(text)**: Borra los _emoji_ del texto.<jupyter_code>from mlearner.nlp.helpers import find_emoji, remove_emoji
sentence="I play () ... ()"
find_emoji(sentence)
data_clean['emoji'] = dataset.data['text'].apply(lambda x : find_emoji(x))
data_clean['text'] = dataset.data['text'].apply(lambda x : remove_emoji(x))<jupyter_output><empty_output><jupyter_text>***  Email* **find_email(text)**: Extraccion de emails del texto.<jupyter_code>from mlearner.nlp.helpers import find_email
data_clean['emails'] = dataset.data['text'].apply(lambda x : find_email(x))
data_clean[data_clean["emails"]!=""][["target", "emails"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Hash* **find_hash(text)**: Busqueda de Hashtags en el texto<jupyter_code>from mlearner.nlp.helpers import find_hash
data_clean['Hash'] = dataset.data['text'].apply(lambda x : find_hash(x))
data_clean[data_clean["Hash"]!=""][["target", "Hash"]].head(5)
data_clean["text"].iloc[471]<jupyter_output><empty_output><jupyter_text>***  Mention* **find_at(text)**: Busqueda de menciones "@" en el texto<jupyter_code>from mlearner.nlp.helpers import find_at
data_clean['Mention'] = dataset.data['text'].apply(lambda x : find_at(x))
data_clean[data_clean["Mention"]!=""][["target", "Mention"]].tail(5)<jupyter_output><empty_output><jupyter_text>***  Numbers* **find_number(text)**: Busqueda de numeros en el texto.<jupyter_code>from mlearner.nlp.helpers import find_number
data_clean['Numbers'] = dataset.data['text'].apply(lambda x : find_number(x))
data_clean[data_clean["Numbers"]!=""][["target", "Numbers"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Phone Number* **find_phone_number(text)**: Busqueda de numeros de telefono españoles en el texto.<jupyter_code>from mlearner.nlp.helpers import find_phone_number
data_clean['phone_number'] = dataset.data['text'].apply(lambda x : find_phone_number(x))
find_phone_number("+34666999666")<jupyter_output><empty_output><jupyter_text>***  Find Year* **find_year(text)**: Busqueda de años de nacimiento en el texto [1940-2040]<jupyter_code>from mlearner.nlp.helpers import find_year
data_clean['Years'] = dataset.data['text'].apply(lambda x : find_year(x))
data_clean[data_clean["Years"]!=""][["target", "Years"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Non Alphanumeric characters* **find_nonalp(text)**: Extraccion de caracteres no alfanumericos.<jupyter_code>from mlearner.nlp.helpers import find_nonalp
data_clean['nonalp'] = dataset.data['text'].apply(lambda x : find_nonalp(x))
data_clean[data_clean["nonalp"]!=""][["target", "nonalp"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Retrieve punctuations from sentence* **find_punct(text)**: Signos de puntuacion.<jupyter_code>from mlearner.nlp.helpers import find_punct
data_clean['find_punct'] = dataset.data['text'].apply(lambda x : find_punct(x))
data_clean[data_clean["find_punct"]!=""][["target", "find_punct"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Unique Char* **unique_char(sentence)**: Elimina los caracteres repetidos de una palabra.<jupyter_code>from mlearner.nlp.helpers import unique_char
sentence="I lovee Machinee learning!"
unique_char(sentence)<jupyter_output><empty_output><jupyter_text>***  Prices* **find_coin(text, symbol="$")**: Busqueda de precios en el texto<jupyter_code>from mlearner.nlp.helpers import find_coin
data_clean['find_coin$'] = dataset.data['text'].apply(lambda x : find_coin(x, symbol="$"))
data_clean[data_clean["find_coin$"]!=""][["target", "find_coin$"]].head(2)
dataset.data['text'].iloc[60]<jupyter_output><empty_output><jupyter_text>***  Numbers great* **num_great(text)**: Busqueda de numeros mayores a 930.<jupyter_code>from mlearner.nlp.helpers import num_great
data_clean['num_great'] = dataset.data['text'].apply(lambda x : num_great(x))
data_clean[data_clean["num_great"]!=""][["target", "num_great"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Numbers less* **num_less(text)**: Busqueda de numeros menores a 930.<jupyter_code>from mlearner.nlp.helpers import num_less
data_clean['num_less'] = dataset.data['text'].apply(lambda x : num_less(x))
data_clean[data_clean["num_less"]!=""][["target", "num_less"]].tail(5)<jupyter_output><empty_output><jupyter_text>***  Find Dates* **find_dates(text)**: Busqueda de fechas [mm-dd-yyyy]<jupyter_code>from mlearner.nlp.helpers import find_dates
sentence="Todays date is 04/28/2020 for format mm/dd/yyyy, not 28/04/2020"
find_dates(sentence)
data_clean['find_dates'] = dataset.data['text'].apply(lambda x : find_dates(x))<jupyter_output><empty_output><jupyter_text>***  Only Words* **only_words(text)**: Eliminar los numeros del texto.<jupyter_code>from mlearner.nlp.helpers import only_words
data_clean['only_words'] = dataset.data['text'].apply(lambda x : only_words(x))
data_clean[data_clean["only_words"]!=""][["target", "only_words"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Search Key* **search_string(text, key)**: Comprobar existencia de palabras en la frase.<jupyter_code>from mlearner.nlp.helpers import search_string
data_clean['search_string'] = dataset.data['text'].apply(lambda x : search_string(x,' day '))
data_clean[data_clean["search_string"]==True][["target", "search_string"]].head(5)
data_clean["text"].iloc[163]<jupyter_output><empty_output><jupyter_text>***  pick only key sentence* **pick_only_key_sentence(text, keyword)**: Devuelve las frases que contiene la palabra (Keyword) seleccionada.<jupyter_code>from mlearner.nlp.helpers import pick_only_key_sentence
data_clean['pick_only_key_sentence'] = dataset.data['text'].apply(lambda x : pick_only_key_sentence(x,' day '))
data_clean[data_clean["pick_only_key_sentence"]!=""][["target", "pick_only_key_sentence"]].iloc[163:165]<jupyter_output><empty_output><jupyter_text>***  pick unique sentence* **pick_unique_sentence(text)**: Elimina frases duplicadas<jupyter_code>from mlearner.nlp.helpers import pick_unique_sentence
sentence="I thank doctors\nDoctors are working very hard in this pandemic situation\nI thank doctors"
pick_unique_sentence(sentence)
data_clean['pick_unique_sentence'] = dataset.data['text'].apply(lambda x : pick_unique_sentence(x))<jupyter_output><empty_output><jupyter_text>***  Capital words* **find_capital(text)**: Busqueda de palabras con primera letra en mayuscula.<jupyter_code>from mlearner.nlp.helpers import find_capital
data_clean['find_capital'] = dataset.data['text'].apply(lambda x : find_capital(x))
data_clean[data_clean["find_capital"]!=""][["target", "find_capital"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Remove tag html* **remove_tag(text)**: Elimina los tags de html.<jupyter_code>from mlearner.nlp.helpers import remove_tag
sentence="Markdown sentences can use <br> for breaks and <i></i> for italics"
remove_tag(sentence)
data_clean['remove_tag'] = dataset.data['text'].apply(lambda x : remove_tag(x))<jupyter_output><empty_output><jupyter_text>***  Mac address * **mac_add(text)**: Busqueda de _Mac address_ en el texto<jupyter_code>from mlearner.nlp.helpers import mac_add
sentence="MAC ADDRESSES of this laptop - 00:24:17:b1:cc:cc. Other details will be mentioned"
mac_add(sentence)
data_clean['mac_add'] = dataset.data['text'].apply(lambda x : mac_add(x))<jupyter_output><empty_output><jupyter_text>***  IP address * **ip_add(text)**: Busqueda de _IP address_ en el texto<jupyter_code>from mlearner.nlp.helpers import ip_add
sentence="An example of ip address is 172.16.58.3"
ip_add(sentence)
data_clean['ip_add'] = dataset.data['text'].apply(lambda x : ip_add(x))<jupyter_output><empty_output><jupyter_text>***  Extract number of subwords* **subword(string, sub)**: Devuelve el numero de veces que aparece la raiz de la palabra.<jupyter_code>from mlearner.nlp.helpers import subword
sentence = 'Fundamentalism and constructivism are important skills'
subword(sentence,'ism') # change subword and try for others
data_clean['subword'] = dataset.data['text'].apply(lambda x : subword(x, "on"))
data_clean[["target", "text", "subword"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Valid latitude & longitude* **lat_lon(string)**: Devuelve si el dato de Latitud y longitud es correcto.<jupyter_code>from mlearner.nlp.helpers import lat_lon
lat_lon('28.6466772,76.8130649', display=True)
lat_lon('2324.3244,3423.432423', display=True)
data_clean['lat_lon'] = dataset.data['text'].apply(lambda x : lat_lon(x))
data_clean[["target", "text", "lat_lon"]].head(5)<jupyter_output><empty_output><jupyter_text>***  Valid latitude & longitude* **find_domain(string)**: Devuelve si el dato de Latitud y longitud es correcto.<jupyter_code>from mlearner.nlp.helpers import find_domain
data_clean['find_domain'] = dataset.data['text'].apply(lambda x : find_domain(x))
data_clean[["target", "text", "find_domain"]].iloc[10:15]
data_clean["text"].iloc[12]<jupyter_output><empty_output><jupyter_text>***<jupyter_code>data_clean.T<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1460879225519759, 'fraction_numerical': 0.011506110141247422, 'mean_word_length': 8.260102865540045, 'pattern_counts': {'":': 0, '<': 123, '<?xml version=': 0, '>': 124, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '491181', 'license': 'MIT', 'n_tokens_mistral': 4558, 'n_tokens_neox': 4402, 'n_words': 923, 'path': 'docs/sources/user_guide/nlp/Text Helper Functions.ipynb', 'repo_name': 'jaisenbe58r/MLearner', 'size': '12603'}