<jupyter_start><jupyter_text>Introduction<jupyter_code><jupyter_output><empty_output><jupyter_text>Import dependencies<jupyter_code># Common imports
import numpy as np
import os

# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)


import pandas as pd
import seaborn as sns
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)<jupyter_output><empty_output><jupyter_text>Getting sample data<jupyter_code>!git clone https://github.com/wizelineacademy/data-engineering-bootcamp.git<jupyter_output>Cloning into 'data-engineering-bootcamp'...
remote: Enumerating objects: 21, done.[K
remote: Counting objects: 100% (21/21), done.[K
remote: Compressing objects: 100% (15/15), done.[K
remote: Total 21 (delta 5), reused 15 (delta 3), pack-reused 0[K
Unpacking objects: 100% (21/21), done.<jupyter_text>Exploratory analysis for sample data<jupyter_code>df = pd.read_csv('/content/data-engineering-bootcamp/data/sample.csv')
df.head(5)
df.describe()
df.columns
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1999 entries, 0 to 1998
Data columns (total 15 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   producto         1999 non-null   object 
 1   presentacion     1999 non-null   object 
 2   marca            1999 non-null   object 
 3   categoria        1999 non-null   object 
 4   catalogo         1999 non-null   object 
 5   precio           1999 non-null   float64
 6   fechaRegistro    1999 non-null   object 
 7   cadenaComercial  1999 non-null   object 
 8   giro             1999 non-null   object 
 9   nombreComercial  1999 non-null   object 
 10  direccion        1999 non-null   object 
 11  estado           1999 non-null   object 
 12  municipio        1999 non-null   object 
 13  latitud          1229 non-null   float64
 14  longitud         1229 non-null   float64
dtypes: float64(3), object(12)
memory usage: 234.4+ KB<jupyter_text>Exploring numerical features<jupyter_code>sns.set(rc={'figure.figsize':(20,10)})

sns.countplot('precio', data=df)
plt.title('Precio', fontsize=14)
plt.plot()

print('Precio Count')
df['precio'].value_counts()<jupyter_output>Precio Count<jupyter_text>Exploring text features * producto - object  * presentacion - object  * marca - object  * categoria - object  * catalogo - object  * fechaRegistro - object  * cadenaComercial - object  * giro - object  * nombreComercial - object  * direccion - object  * estado - object  * municipio - object  Feature `producto`<jupyter_code>print('INFO: There are {} categories in the feature `producto`.'.format(len(df.producto.value_counts())))
print('INFO: Some examples')
df.producto.value_counts()<jupyter_output>INFO: There are 330 categories in the feature `producto`.
INFO: Some examples<jupyter_text>Feature `presentacion`<jupyter_code>print('INFO: There are {} categories in the feature `presentacion`.'.format(len(df.presentacion.value_counts())))
print('INFO: Some examples')
df.presentacion.value_counts()<jupyter_output>INFO: There are 767 categories in the feature `presentacion`.
INFO: Some examples<jupyter_text>Feature `marca`<jupyter_code>print('INFO: There are {} categories in the feature `marca`.'.format(len(df.marca.value_counts())))
print('INFO: Some examples')
df.marca.value_counts()<jupyter_output>INFO: There are 395 categories in the feature `marca`.
INFO: Some examples<jupyter_text>Feature `categoria`<jupyter_code>print('INFO: There are {} categories in the feature `categoria`.'.format(len(df.categoria.value_counts())))
print('INFO: Some examples')
df.categoria.value_counts()<jupyter_output>INFO: There are 36 categories in the feature `categoria`.
INFO: Some examples<jupyter_text>Feature `catalogo`<jupyter_code>print('INFO: There are {} categories in the feature `catalogo`.'.format(len(df.catalogo.value_counts())))
print('INFO: Some examples')
df.catalogo.value_counts()<jupyter_output>INFO: There are 6 categories in the feature `catalogo`.
INFO: Some examples<jupyter_text>Feature `fechaRegistro`<jupyter_code>print('INFO: There are {} categories in the feature `fechaRegistro`.'.format(len(df.fechaRegistro.value_counts())))
print('INFO: Some examples')
df.fechaRegistro.value_counts()<jupyter_output>INFO: There are 2 categories in the feature `fechaRegistro`.
INFO: Some examples<jupyter_text>Feature `cadenaComercial`<jupyter_code>print('INFO: There are {} categories in the feature `cadenaComercial`.'.format(len(df.cadenaComercial.value_counts())))
print('INFO: Some examples')
df.cadenaComercial.value_counts()<jupyter_output>INFO: There are 4 categories in the feature `cadenaComercial`.
INFO: Some examples<jupyter_text>Feature `giro`<jupyter_code>print('INFO: There are {} categories in the feature `giro`.'.format(len(df.giro.value_counts())))
print('INFO: Some examples')
df.giro.value_counts()<jupyter_output>INFO: There are 2 categories in the feature `giro`.
INFO: Some examples<jupyter_text>Feature `nombreComercial`<jupyter_code>print('INFO: There are {} categories in the feature `nombreComercial`.'.format(len(df.nombreComercial.value_counts())))
print('INFO: Some examples')
df.nombreComercial.value_counts()<jupyter_output>INFO: There are 5 categories in the feature `nombreComercial`.
INFO: Some examples<jupyter_text>Feature `direccion`<jupyter_code>print('INFO: There are {} categories in the feature `direccion`.'.format(len(df.direccion.value_counts())))
print('INFO: Some examples')
df.direccion.value_counts()<jupyter_output>INFO: There are 5 categories in the feature `direccion`.
INFO: Some examples<jupyter_text>Feature `estado`<jupyter_code>print('INFO: There are {} categories in the feature `estado`.'.format(len(df.estado.value_counts())))
print('INFO: Some examples')
df.estado.value_counts()<jupyter_output>INFO: There are 2 categories in the feature `estado`.
INFO: Some examples<jupyter_text>Feature `municipio`<jupyter_code>print('INFO: There are {} categories in the feature `municipio`.'.format(len(df.municipio.value_counts())))
print('INFO: Some examples')
df.municipio.value_counts()<jupyter_output>INFO: There are 5 categories in the feature `municipio`.
INFO: Some examples<jupyter_text>Getting all data<jupyter_code>!git clone https://github.com/chentinghao/download_google_drive.git
%cd download_google_drive/
!python download_gdrive.py 1crFR4yrS_6DV3_Fh-HrhANJg1-qlyE-O profeco.zip
!unzip /content/download_google_drive/profeco.zip<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12196601941747573, 'fraction_numerical': 0.02624393203883495, 'mean_word_length': 5.6194779116465865, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '373791', 'license': 'MIT', 'n_tokens_mistral': 2377, 'n_tokens_neox': 2225, 'n_words': 514, 'path': 'Technical_Challenge_Part_2.ipynb', 'repo_name': 'carloslme/Practical-Deep-Learning-Book', 'size': '6592'}