<jupyter_start><jupyter_text>**Instalação de dependências**<jupyter_code>!pip install seaborn==0.9.0
import pandas as pd
from google.colab import files

from sklearn.cluster import KMeans, AgglomerativeClustering
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
import seaborn as sns
from sklearn.metrics import pairwise_distances_argmin
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()  # for plot styling
import numpy as np<jupyter_output>Requirement already satisfied: seaborn==0.9.0 in /usr/local/lib/python3.6/dist-packages (0.9.0)
Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn==0.9.0) (0.24.2)
Requirement already satisfied: matplotlib>=1.4.3 in /usr/local/lib/python3.6/dist-packages (from seaborn==0.9.0) (3.0.3)
Requirement already satisfied: numpy>=1.9.3 in /usr/local/lib/python3.6/dist-packages (from seaborn==0.9.0) (1.16.4)
Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn==0.9.0) (1.3.0)
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn==0.9.0) (2018.9)
Requirement already satisfied: python-dateutil>=2.5.0 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn==0.9.0) (2.5.3)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.4.3->seaborn==0.9.0) ([...]<jupyter_text>**Upload do CSV (Tabela 4)**<jupyter_code>tabela4Uploaded = files.upload()<jupyter_output><empty_output><jupyter_text>**Importação dos arquivos CSV na biblioteca do Pandas**<jupyter_code>tabela4 = pd.read_csv("tabela4.csv", sep=";", header=0, decimal=',', names=["Trechos", "ATR (mm)", "N", "FN", "Espessura total do revestimento (cm)", "Tempo de solicitação (meses)", "Região", "Temperatura média do ar (grau C)"])<jupyter_output><empty_output><jupyter_text>**Funções auxiliares**<jupyter_code>def calculaRazao(atr,n):
  return atr/n
def obtemCriterio(fn):
  if fn>100 and fn<=300:
    return 'Médio'
  elif fn>300 and fn<=1000:
    return 'Pesado'
  elif fn>1000:
    return 'Extremamente Pesado'
  else:
    return 'Leve'
def obtemCriterioAtrN(atrN):
  if atrN<3000000:
    return 'Leve'
  elif atrN>10000000 and atrN<=30000000:
    return 'Leve|Médio|Pesado'
  elif atrN>30000000:
    return 'Leve|Médio|Pesado|Extremamente Pesado'
  else:
    return 'Leve|Médio'
tabela4["ATR/N"] = tabela4.apply(lambda x: calculaRazao(x['ATR (mm)'], x['N']), axis=1)
#tabela4["criterio"] = tabela4["FN"].apply(obtemCriterio)
tabela4["criterio_n"] = tabela4["N"].apply(obtemCriterioAtrN)<jupyter_output><empty_output><jupyter_text>**Exibição da Tabela**<jupyter_code>tabela4<jupyter_output><empty_output><jupyter_text>**Extraindo os dummies**Cria uma nova tabela com os criterios (em colunas) para populá-los com dados binários a fim de identificar os trechos por critério. Este é um processo muito comum em ciência de dados quando se deseja clusterizar ou utilizar uma variável categórica.<jupyter_code>criterio = tabela4.criterio_n.str.get_dummies()
criterio<jupyter_output><empty_output><jupyter_text>**Concatenando a Tabela 4 (CSV) com os dummies**<jupyter_code>dados_dos_trechos = pd.concat([tabela4,criterio], axis=1) # axis=1 indica que sera concatenado ao lado (na horizontal)
dados_dos_trechos<jupyter_output><empty_output><jupyter_text>**Escalando os dados**<jupyter_code>scaler = StandardScaler()
criterios_escalado = scaler.fit_transform(criterio)
criterios_escalado.shape<jupyter_output><empty_output><jupyter_text>**Agrupando os dados**Coloca os itens parecidos nos mesmos grupos e itens diferentes em grupos diferentes<jupyter_code>modelo = KMeans(n_clusters=3)
modelo.fit(criterios_escalado)

print("Grupos {}".format(modelo.labels_))
tsne = TSNE()
visualizacao = tsne.fit_transform(criterios_escalado)
visualizacao
sns.set(rc={'figure.figsize': (13, 13)})

sns.scatterplot(x=visualizacao[:, 0],
               y=visualizacao[:, 1],
               hue=modelo.labels_,
               palette=sns.color_palette('Set1', 3))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14056896963901505, 'fraction_numerical': 0.04470475735118336, 'mean_word_length': 6.176672384219554, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '335363', 'license': 'Apache-2.0', 'n_tokens_mistral': 1746, 'n_tokens_neox': 1673, 'n_words': 319, 'path': 'deformacao_permanente.ipynb', 'repo_name': 'gabrieltavaresmelo/petran', 'size': '4183'}