<jupyter_start><jupyter_text>NOTA: Uso de memoria reducido en una sexta parte<jupyter_code>#Cantidad de Nan en los datos
data_values.isna().sum().sum()
#Cantidad de Nulls en los datos
data_values.isnull().sum().sum()
data_values_int_columns=data_values.loc[:, data_values.dtypes == np.int64]
data_values_int_columns.head()
#Transformamos los tipos int64 a booleanos para reducir espacio y aplicar el tipo correspondiente
boolean_columns = [col for col in data_values if data_values[col].isin([0,1]).all()]

for col in list(boolean_columns):
    data_values[col] = data_values[col].astype(bool)

data_values.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 260601 entries, 0 to 260600
Data columns (total 39 columns):
building_id                               260601 non-null int64
geo_level_1_id                            260601 non-null int64
geo_level_2_id                            260601 non-null int64
geo_level_3_id                            260601 non-null int64
count_floors_pre_eq                       260601 non-null int64
age                                       260601 non-null int64
area_percentage                           260601 non-null int64
height_percentage                         260601 non-null int64
land_surface_condition                    260601 non-null category
foundation_type                           260601 non-null category
roof_type                                 260601 non-null category
ground_floor_type                         260601 non-null category
other_floor_type                          260601 non-null category
position                                  [...]<jupyter_text>NOTA: Uso de memoria reducido a la mitad<jupyter_code>#Analisis de 'data_labels'
data_labels = pd.read_csv('data/train_labels.csv')
data_labels.head()
data_labels.info()
data_labels.shape
data_labels.size
#Cantidad de Nan en los datos
data_labels.isna().sum().sum()
#Merge de labels y values. 
#Se realiza un outer para contemplar todos los posibles casos
data = data_values.merge(right=data_labels, how='outer', left_on='building_id', right_on='building_id')
data.head()
data.info()
#Detalles de columnas numericas
data_int_columns=data.loc[:, data.dtypes == np.int64].columns
data[data_int_columns].describe()
print('max value: ',data[data_int_columns].max().max(), ' - min value: ', data[data_int_columns].min().min())<jupyter_output>max value:  1052934  - min value:  0<jupyter_text>Rangos de int:    int64 = -9,223,372,036,854,775,808 a 9,223,372,036,854,775,807    int32 = -2,147,483,648 a 2,147,483,647     int16 = -32768 a 32767    int8 = -128 a 127Por lo que se puede restringir el tipo numerico de acuerdo a su maximo valor.De antemano ya se observa que el rango maximo aplica para int32.<jupyter_code>#Transformamos los tipos int64 a int32 para reducir espacio

for col in list(data_int_columns):
    if(data[col].max() > 32767):
        data_values[col] = data[col].astype(np.int32)
    elif(data[col].max() > 127):
        data_values[col] = data[col].astype(np.int16)
    else:
        data_values[col] = data[col].astype(np.int8)

data_values.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 260601 entries, 0 to 260600
Data columns (total 40 columns):
building_id                               260601 non-null int32
geo_level_1_id                            260601 non-null int8
geo_level_2_id                            260601 non-null int16
geo_level_3_id                            260601 non-null int16
count_floors_pre_eq                       260601 non-null int8
age                                       260601 non-null int16
area_percentage                           260601 non-null int8
height_percentage                         260601 non-null int8
land_surface_condition                    260601 non-null category
foundation_type                           260601 non-null category
roof_type                                 260601 non-null category
ground_floor_type                         260601 non-null category
other_floor_type                          260601 non-null category
position                                  2606[...]<jupyter_text>NOTA: Uso de memoria reducido en una cuarta parte<jupyter_code>#Cantidad de Nan en los datos
data.isna().sum().sum()
#Cantidad de Nulls en los datos
data.isnull().sum().sum()
data.to_csv('data/clean_data.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08342805182996135, 'fraction_numerical': 0.07729029324846556, 'mean_word_length': 2.4348165495706477, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '344852', 'license': 'MIT', 'n_tokens_mistral': 1569, 'n_tokens_neox': 1333, 'n_words': 340, 'path': 'tp2/data_cleaning.ipynb', 'repo_name': 'alanbach89/orga_datos_1c_2021', 'size': '4399'}