<jupyter_start><jupyter_text>Importação dos dados<jupyter_code>base = pd.read_csv('dados/BRAZIL_CITIES.csv', sep=';', decimal=',')
base.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5576 entries, 0 to 5575
Data columns (total 81 columns):
CITY                      5576 non-null object
STATE                     5576 non-null object
CAPITAL                   5576 non-null int64
IBGE_RES_POP              5568 non-null float64
IBGE_RES_POP_BRAS         5568 non-null float64
IBGE_RES_POP_ESTR         5568 non-null float64
IBGE_DU                   5566 non-null float64
IBGE_DU_URBAN             5566 non-null float64
IBGE_DU_RURAL             5495 non-null float64
IBGE_POP                  5568 non-null float64
IBGE_1                    5568 non-null float64
IBGE_1-4                  5568 non-null float64
IBGE_5-9                  5568 non-null float64
IBGE_10-14                5568 non-null float64
IBGE_15-59                5568 non-null float64
IBGE_60+                  5568 non-null float64
IBGE_PLANTED_AREA         5573 non-null float64
IBGE_CROP_PRODUCTION_$    5573 non-null float64
IDHM Ranking 2010         5568 no[...]<jupyter_text>De acordo com a descrição encontrada no Kaggle e com a quantidade de valores nulos, selecionei algumas colunas para entrar na analise<jupyter_code>colunas = [
    'CITY', 'CAPITAL', 'IBGE_POP', 'IBGE_PLANTED_AREA', 'IDHM', 'ALT',  'AREA', 'Pu_Bank'
]
base[colunas].info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5576 entries, 0 to 5575
Data columns (total 8 columns):
CITY                 5576 non-null object
CAPITAL              5576 non-null int64
IBGE_POP             5568 non-null float64
IBGE_PLANTED_AREA    5573 non-null float64
IDHM                 5568 non-null float64
ALT                  5567 non-null float64
AREA                 5574 non-null float64
Pu_Bank              3345 non-null float64
dtypes: float64(6), int64(1), object(1)
memory usage: 348.6+ KB<jupyter_text>Algumas variaveis irei preencher os valores nulo com zero para indicar que ela não possui a caracteristica, enquanto outras colunas que deveriam estar todas preenchidas vou preencher com a média.Tambem irei renomear as colunas para que todas sigam um mesmo padrão<jupyter_code>base['CAPITAL'] = base['CAPITAL'].fillna(base['CAPITAL'].median())
base['IBGE_POP'] = base['IBGE_POP'].fillna(base['IBGE_POP'].median())
base['IBGE_PLANTED_AREA'] = base['IBGE_PLANTED_AREA'].fillna(base['IBGE_PLANTED_AREA'].median())
base['IDHM'] = base['IDHM'].fillna(base['IDHM'].median())
base['ALT'] = base['ALT'].fillna(base['ALT'].median())
base['AREA'] = base['AREA'].fillna(base['AREA'].median())
base['PU_BANK'] = base['Pu_Bank'].fillna(0)
colunas = [
    'CITY', 'CAPITAL', 'IBGE_POP', 'IBGE_PLANTED_AREA', 'IDHM', 'ALT',  'AREA', 'PU_BANK'
]<jupyter_output><empty_output><jupyter_text>Analisando a correlação entre as variaveis'<jupyter_code>plt.figure(figsize=[15, 10])
sns.heatmap(base[colunas].corr(), annot=True, fmt='.2f', vmax=1, vmin=0, cmap=sns.cubehelix_palette(10))
plt.show()<jupyter_output><empty_output><jupyter_text>Analisando as distribuições das variaveis<jupyter_code>for coluna in base[colunas].drop('CITY', axis=1).columns:
    print('Maximo: ' + str(max(base[coluna])))
    print('Minimo: ' + str(min(base[coluna])))
    print('Media: ' + str(np.median(base[coluna])))
    print('Mediana: ' + str(np.median(base[coluna])))

    plt.figure(figsize=[15, 3])
    sns.distplot(base[coluna])
    plt.show()
base[colunas].to_csv('dados/base_cidades.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12266517981600224, 'fraction_numerical': 0.061053805408419294, 'mean_word_length': 3.582375478927203, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '39150', 'license': 'MIT', 'n_tokens_mistral': 1484, 'n_tokens_neox': 1308, 'n_words': 276, 'path': 'preparacao_dos_dados.ipynb', 'repo_name': 'natansilva/clusterizacao', 'size': '3587'}