<jupyter_start><jupyter_text>Importing the dataset and libraries.<jupyter_code>import pandas as pd
import numpy as np

inspections = pd.read_csv('Data/sedec_vistorias.csv',sep=";")<jupyter_output><empty_output><jupyter_text>Taking a look at the dataset.<jupyter_code>inspections.info()
inspections.head()
inspections['vistoria_risco'].value_counts()
inspections['vistoria_localidade'].value_counts()
inspections['ano'].value_counts()<jupyter_output><empty_output><jupyter_text>We are going to drop all the columns that don't relate to the info that we need, all of these relate to the inspection process and log so these features aren't related to slidings. Additionally, we are dropping the date and locations columns since we'll be extracting these values from a different dataset when cross-referencing the ID.<jupyter_code>inspections = inspections.drop(columns=['ano','mes','vistoria_data','avaliador',
                        'vistoria_localidade','vistoria_rpa_codigo','vistoria_setor',
                        'vistoria_microrregiao'],axis=1)
inspections.head()<jupyter_output><empty_output><jupyter_text>Renaming the dataset for better understanding. This dataset relates to an inspection conducted on that place, the feature that relates to our problem is the risk evaluated by the inspector at the scene.<jupyter_code>inspections = inspections.rename(columns={'vistoria_risco':'risk','processo_numero':'ID'})
inspections.head()<jupyter_output><empty_output><jupyter_text>We will transform the risk column into:4- Very High risk.3- High.2- Medium.1- Low.0- Not informed.<jupyter_code>mapping = {'Não informado': 0, 'R1 Baixo': 1,'R2 Médio': 2,'R3 Alto': 3,'R4 Muito Alto': 4 }

inspections = inspections.applymap(lambda s: mapping.get(s) if s in mapping else s)
inspections.head()<jupyter_output><empty_output><jupyter_text>Finally there are some duplicate ID entries, so we'll drop the ones that might have 0 values for risk assessment.<jupyter_code>inspections = inspections.sort_values('risk', ascending=False).drop_duplicates('ID').sort_index()<jupyter_output><empty_output><jupyter_text>Taking a look at the unique values of this dataset.<jupyter_code>inspections['risk'].value_counts()
inspections.to_csv(path_or_buf='inspections_prepared.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10698689956331878, 'fraction_numerical': 0.0069868995633187774, 'mean_word_length': 7.095406360424028, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '66839', 'license': 'MIT', 'n_tokens_mistral': 757, 'n_tokens_neox': 703, 'n_words': 186, 'path': '00 - Inspections_preparation.ipynb', 'repo_name': 'NExt-Machine-Learning-Project/RecifeLandSlidings', 'size': '2290'}