<jupyter_start><jupyter_text>Outliers are mainly found in columns Age<jupyter_code>plt.figure(figsize=(16,5))
df.boxplot()
print()
df[['Water','Age']].describe()<jupyter_output><empty_output><jupyter_text>Removing Outliers in Age using IQR<jupyter_code>plt.figure(figsize=(8, 4))
sns.boxplot(x=df['Age'])
plt.show()
Q1 = df['Age'].quantile(.25)
Q3 = df['Age'].quantile(.75)

IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

df = df[df['Age'] >= lower] 
df = df[df['Age'] <= upper]
plt.figure(figsize=(16,5))
df.boxplot()
print()
df.describe()
df.shape<jupyter_output><empty_output><jupyter_text>Duplicated Values<jupyter_code>print(df.duplicated().sum()) # 25 duplicate values 
df = df.drop_duplicates() # Values have been removed<jupyter_output><empty_output><jupyter_text>Correlations<jupyter_code># Correlation between columns
plt.figure(figsize=(12,7))
correlation = df.corr()
sns.heatmap(correlation, annot = True);
correlations = df.corr()['Concrete compressive strength'].drop('Concrete compressive strength')
correlations.sort_values()<jupyter_output><empty_output><jupyter_text>Clustering<jupyter_code>from sklearn.cluster import KMeans
from sklearn import metrics
from scipy.spatial.distance import cdist
X = np.array(df.drop('Concrete compressive strength',1))
X<jupyter_output><empty_output><jupyter_text>Método de Elbow para descobrir o nº de Clusters<jupyter_code>## O NÚMERO MÁXIMO DE POSSÍVEIS CLUSTERS É N. A QUANTIDADE DE EXPERIÊNCIAS
K = range(1,10)
# k means determine k
distortions = []

for k in K:
    kmeanModel = KMeans(n_clusters=k).fit(X)
    kmeanModel.fit(X)
    distortions.append(sum(np.min(cdist(X, kmeanModel.cluster_centers_, 'euclidean'), axis=1)) / X.shape[0])
# Plot the elbow
plt.plot(K, distortions, 'bx-')
plt.xlabel('k')
plt.ylabel('Distortion')
plt.title('Método do cotovelo para K ótimo')
plt.show()

distortions<jupyter_output><empty_output><jupyter_text>> O melhor nº de K = 3  K-Means<jupyter_code>kmeans = KMeans(n_clusters=3, random_state=0)
kmeans.fit_predict(X)
kmeans.labels_<jupyter_output><empty_output><jupyter_text>Comparando os valores existentes com as classes criadas<jupyter_code>df['Classes'] = kmeans.labels_
df
df['Classes'].value_counts()
plt.title('Classes', fontsize = 15)
sns.countplot(df['Classes'], order = df['Classes'].value_counts().index)
plt.show()
plt.figure(figsize=(12,8))
plt.scatter(X[:, 0], X[:,1], s = 100, c = kmeans.labels_)
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], s = 110, c = 'red', label = 'Centroids')
plt.title('Concrete Clusters and Centroids')
plt.legend()
plt.ylabel('Spending Score (0-400)') , plt.xlabel('Concrete compressive strength')
plt.show()<jupyter_output><empty_output><jupyter_text>> A partir do número de três clusters as distâncias dos erros quadráticos praticamente se estabiliza, formando-se 3 grupos.  Prevendo os dados<jupyter_code>prediction = kmeans.labels_
prediction[80]<jupyter_output><empty_output><jupyter_text>Prever qual classe pertencerá determinado concreto se dermos as seguintes condições<jupyter_code>df.head()
# Cement, Blast Furnace Slag, Fly Ash, Water, Superplasticizer, Coarse Aggregate, Fine Aggregate, Age

test = np.array([[148.5, 116.0, 115.1, 196.0, 2.5, 932.0, 761.5, 22]])
kmeans.predict(test) #Prevê qual classe pertencerá<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14641464146414643, 'fraction_numerical': 0.025802580258025804, 'mean_word_length': 7.252475247524752, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '252189', 'license': 'MIT', 'n_tokens_mistral': 1385, 'n_tokens_neox': 1280, 'n_words': 259, 'path': 'Concrete - Clustering.ipynb', 'repo_name': 'Daniel-RPS/Concrete-Repository-UCI', 'size': '3333'}