<jupyter_start><jupyter_text>Classificação de galáxias * Determição do tipo de galáxia* Dados obtidos na NASA (https://heasarc.gsfc.nasa.gov/db-perl/W3Browse/w3table.pl?tablehead=name%3Dneargalcat&Action=More+Options) * Importando bibliotecas<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>* Lendo arquivo Excel<jupyter_code>dados = pd.read_excel('galaxias.xls',sheet_name='neargalcat')<jupyter_output>*** No CODEPAGE record, no encoding_override: will use 'ascii'
*** No CODEPAGE record, no encoding_override: will use 'ascii'
*** No CODEPAGE record, no encoding_override: will use 'ascii'
*** No CODEPAGE record, no encoding_override: will use 'ascii'<jupyter_text>* Exibindo cinco primeiras linhas<jupyter_code>dados.head()<jupyter_output><empty_output><jupyter_text>* Verificando quantidade de galáxias para cada tipo<jupyter_code>sns.countplot(x=dados['class'])
plt.xticks(rotation=90)
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>* Renomeando classes para uma forma mais geral<jupyter_code>def fix_classe(x):
    if(x=="IRREGULAR GALAXY"):
        return "Irregular"
    elif(x=="SPIRAL GALAXY"):
         return "Espiral"
    elif(x=="LENTICULAR GALAXY"):
         return "Lenticular"
    elif(x=="IRREGULAR GALAXY HII REGION"):
         return "Irregular"
    else:
         return x
dados['class'] = dados['class'].apply(fix_classe)
sns.countplot(x=dados['class'])
plt.xticks(rotation=90)
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>* Removendo galáxias sem classe definida<jupyter_code>dados = dados[dados['class']!="UNIDENTIFIED"]
dados.head()<jupyter_output><empty_output><jupyter_text>* Exibindo informações dos dados<jupyter_code>dados.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 868 entries, 0 to 868
Data columns (total 10 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   name             868 non-null    object 
 1   ra               868 non-null    float64
 2   dec              868 non-null    float64
 3   bmag             863 non-null    float64
 4   ks_mag           862 non-null    float64
 5   linear_diameter  861 non-null    float64
 6   distance         868 non-null    float64
 7   radial_velocity  760 non-null    float64
 8   abs_bmag         863 non-null    float64
 9   class            868 non-null    object 
dtypes: float64(8), object(2)
memory usage: 74.6+ KB<jupyter_text>* Verificando existência de NaNs<jupyter_code>dados.isna().sum()<jupyter_output><empty_output><jupyter_text>* Removendo linhas das colunas com menos elementos vazios<jupyter_code>dados = dados.dropna(subset=['bmag','ks_mag','linear_diameter','abs_bmag'])
dados.isna().sum()<jupyter_output><empty_output><jupyter_text>* Separando cada grupo de galáxias num dataframe<jupyter_code>irregular = dados[dados['class']=='Irregular']
lenticular = dados[dados['class']=='Lenticular']
espiral = dados[dados['class']=='Espiral']<jupyter_output><empty_output><jupyter_text>* Verificando existência de NaNs nas irregulares e substituindo pela mediana<jupyter_code>irregular.isna().sum()
irregular['radial_velocity'] = irregular['radial_velocity'].fillna(irregular['radial_velocity'].mean())<jupyter_output><ipython-input-16-46fd05ba8252>:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  irregular['radial_velocity'] = irregular['radial_velocity'].fillna(irregular['radial_velocity'].mean())<jupyter_text>* Verificando existência de NaNs nas lenticulares e substituindo pela mediana<jupyter_code>lenticular.isna().sum()
lenticular['radial_velocity'] = lenticular['radial_velocity'].fillna(lenticular['radial_velocity'].mean())<jupyter_output><ipython-input-18-e8ad1879e9d7>:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  lenticular['radial_velocity'] = lenticular['radial_velocity'].fillna(lenticular['radial_velocity'].mean())<jupyter_text>* Verificando existência de NaNs nas espirais e substituindo pela mediana<jupyter_code>espiral['radial_velocity'] = espiral['radial_velocity'].fillna(espiral['radial_velocity'].mean())<jupyter_output><ipython-input-19-2d8240883313>:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  espiral['radial_velocity'] = espiral['radial_velocity'].fillna(espiral['radial_velocity'].mean())<jupyter_text>* Juntando dataframes<jupyter_code>dados = pd.concat([espiral,lenticular,irregular])
dados.head()
dados.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 861 entries, 0 to 868
Data columns (total 10 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   name             861 non-null    object 
 1   ra               861 non-null    float64
 2   dec              861 non-null    float64
 3   bmag             861 non-null    float64
 4   ks_mag           861 non-null    float64
 5   linear_diameter  861 non-null    float64
 6   distance         861 non-null    float64
 7   radial_velocity  861 non-null    float64
 8   abs_bmag         861 non-null    float64
 9   class            861 non-null    object 
dtypes: float64(8), object(2)
memory usage: 74.0+ KB<jupyter_text>* Analisando posição dos objetos<jupyter_code>sns.lmplot(x='ra', y='dec', data=dados, hue='class', fit_reg=False, palette='coolwarm', size=6, aspect=2)
plt.title('Coordenadas equatoriais (RA-DEC)')
plt.xlabel('Ascenção reta (graus)')
plt.ylabel('Declinação (graus)')<jupyter_output>/home/usuario/anaconda3/lib/python3.8/site-packages/seaborn/regression.py:573: UserWarning: The `size` parameter has been renamed to `height`; please update your code.
  warnings.warn(msg, UserWarning)<jupyter_text>* Conforme esperado objetos não estão numa posição preferencial * Removendo colunas name, ra e dec do dataframe<jupyter_code>dados = dados.drop(['name','ra','dec'],axis=1)
dados.head()<jupyter_output><empty_output><jupyter_text>* Analisando distribuição das variáveis<jupyter_code>fig, ax = plt.subplots(2,3,figsize=(10,8))
sns.histplot(data=dados,x='bmag',hue='class',ax=ax[0][0])
sns.histplot(data=dados,x='ks_mag',hue='class',ax=ax[0][1])
sns.histplot(data=dados,x='linear_diameter',hue='class',ax=ax[0][2])

sns.histplot(data=dados,x='distance',hue='class',ax=ax[1][0])
sns.histplot(data=dados,x='radial_velocity',hue='class',ax=ax[1][1])
sns.histplot(data=dados,x='abs_bmag',hue='class',ax=ax[1][2])
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>* Analisando outliers<jupyter_code>fig, ax = plt.subplots(2,3,figsize=(10,8))
sns.boxplot(data=dados,y='bmag',x='class',ax=ax[0][0])
sns.boxplot(data=dados,y='ks_mag',x='class',ax=ax[0][1])
sns.boxplot(data=dados,y='linear_diameter',x='class',ax=ax[0][2])

sns.boxplot(data=dados,y='distance',x='class',ax=ax[1][0])
sns.boxplot(data=dados,y='radial_velocity',x='class',ax=ax[1][1])
sns.boxplot(data=dados,y='abs_bmag',x='class',ax=ax[1][2])
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>* Convertendo variável classe para numérica<jupyter_code>from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
dados['class'] = le.fit_transform(dados['class'].values)
dados.head()<jupyter_output><empty_output><jupyter_text>* Determinando variáveis X e Y<jupyter_code>X = dados.drop('class',axis=1).values
Y = dados['class'].values<jupyter_output><empty_output><jupyter_text>* Criando amostras de treino e teste<jupyter_code>from sklearn.model_selection import train_test_split
X_treino, X_teste, Y_treino, Y_teste = train_test_split(X,Y,test_size=0.3,random_state=42)<jupyter_output><empty_output><jupyter_text>* Importando métricas de classificação<jupyter_code>from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
modelos = []
acuracia = []
precisao = []
recall = []
f1 = []<jupyter_output><empty_output><jupyter_text>* Modelo 1: Regressão logistica<jupyter_code>from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
logreg.fit(X_treino,Y_treino)
Y_pred_logreg = logreg.predict(X_teste)
acc_logreg = accuracy_score(Y_teste,Y_pred_logreg)
prec_logreg = precision_score(Y_teste,Y_pred_logreg,average='macro')
rec_logreg = recall_score(Y_teste,Y_pred_logreg,average='macro')
f1_logreg = f1_score(Y_teste,Y_pred_logreg,average='macro')
print("Acurácia = {:0.2f}%".format(acc_logreg*100))
print("Precisão = {:0.2f}%".format(prec_logreg*100))
print("Recall = {:0.2f}%".format(rec_logreg*100))
print("F1 = {:0.2f}%".format(f1_logreg*100))
modelos.append("Regressão logistica")
acuracia.append(acc_logreg)
precisao.append(prec_logreg)
recall.append(rec_logreg)
f1.append(f1_logreg)<jupyter_output><empty_output><jupyter_text>* Modelo 2: Support Vector Machine<jupyter_code>from sklearn.svm import SVC
svc = SVC()
svc.fit(X_treino,Y_treino)
Y_pred_svc = svc.predict(X_teste)
acc_svc = accuracy_score(Y_teste,Y_pred_svc)
prec_svc = precision_score(Y_teste,Y_pred_svc,average='macro')
rec_svc = recall_score(Y_teste,Y_pred_svc,average='macro')
f1_svc = f1_score(Y_teste,Y_pred_svc,average='macro')
print("Acurácia = {:0.2f}%".format(acc_svc*100))
print("Precisão = {:0.2f}%".format(prec_svc*100))
print("Recall = {:0.2f}%".format(rec_svc*100))
print("F1 = {:0.2f}%".format(f1_svc*100))
modelos.append("SVC")
acuracia.append(acc_svc)
precisao.append(prec_svc)
recall.append(rec_svc)
f1.append(f1_svc)<jupyter_output><empty_output><jupyter_text>* Modelo 3: Naive-Bayes<jupyter_code>from sklearn.naive_bayes import GaussianNB
nb = GaussianNB()
nb.fit(X_treino,Y_treino)
Y_pred_nb = nb.predict(X_teste)
acc_nb = accuracy_score(Y_teste,Y_pred_nb)
prec_nb = precision_score(Y_teste,Y_pred_nb,average='macro')
rec_nb = recall_score(Y_teste,Y_pred_nb,average='macro')
f1_nb = f1_score(Y_teste,Y_pred_nb,average='macro')
print("Acurácia = {:0.2f}%".format(acc_nb*100))
print("Precisão = {:0.2f}%".format(prec_nb*100))
print("Recall = {:0.2f}%".format(rec_nb*100))
print("F1 = {:0.2f}%".format(f1_nb*100))
modelos.append("Naive-Bayes")
acuracia.append(acc_nb)
precisao.append(prec_nb)
recall.append(rec_nb)
f1.append(f1_nb)<jupyter_output><empty_output><jupyter_text>* Modelo 4: Decision Tree<jupyter_code>from sklearn.tree import DecisionTreeClassifier
dtc = DecisionTreeClassifier()
dtc.fit(X_treino,Y_treino)
Y_pred_dtc = dtc.predict(X_teste)
acc_dtc = accuracy_score(Y_teste,Y_pred_dtc)
prec_dtc = precision_score(Y_teste,Y_pred_dtc,average='macro')
rec_dtc = recall_score(Y_teste,Y_pred_dtc,average='macro')
f1_dtc = f1_score(Y_teste,Y_pred_dtc,average='macro')
print("Acurácia = {:0.2f}%".format(acc_dtc*100))
print("Precisão = {:0.2f}%".format(prec_dtc*100))
print("Recall = {:0.2f}%".format(rec_dtc*100))
print("F1 = {:0.2f}%".format(f1_dtc*100))
modelos.append("Decision Tree")
acuracia.append(acc_dtc)
precisao.append(prec_dtc)
recall.append(rec_dtc)
f1.append(f1_dtc)<jupyter_output><empty_output><jupyter_text>* Modelo 5: Random Forest<jupyter_code>from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier()
rfc.fit(X_treino,Y_treino)
Y_pred_rfc = rfc.predict(X_teste)
acc_rfc = accuracy_score(Y_teste,Y_pred_rfc)
prec_rfc = precision_score(Y_teste,Y_pred_rfc,average='macro')
rec_rfc = recall_score(Y_teste,Y_pred_rfc,average='macro')
f1_rfc = f1_score(Y_teste,Y_pred_rfc,average='macro')
print("Acurácia = {:0.2f}%".format(acc_rfc*100))
print("Precisão = {:0.2f}%".format(prec_rfc*100))
print("Recall = {:0.2f}%".format(rec_rfc*100))
print("F1 = {:0.2f}%".format(f1_rfc*100))
modelos.append("Random Forest")
acuracia.append(acc_rfc)
precisao.append(prec_rfc)
recall.append(rec_rfc)
f1.append(f1_rfc)<jupyter_output><empty_output><jupyter_text>* Dataframe com resultados<jupyter_code>dicionario = {"Modelo" : modelos, "Acuracia" : acuracia, "Precisao" : precisao,
             "Recall" : recall, "F1" : f1}
pd_di = pd.DataFrame(dicionario)
pd_di<jupyter_output><empty_output><jupyter_text>* Determinando melhor modelo<jupyter_code>pd_di = pd_di.sort_values(by='Recall',ascending=False)
pd_di<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1450483756784394, 'fraction_numerical': 0.02965468418154645, 'mean_word_length': 6.785670545009186, 'pattern_counts': {'":': 0, '<': 115, '<?xml version=': 0, '>': 115, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '526510', 'license': 'MIT', 'n_tokens_mistral': 5314, 'n_tokens_neox': 4992, 'n_words': 745, 'path': 'Data_Science/Astronomia/Tipo_Galaxia/Galaxias.ipynb', 'repo_name': 'maledicente/cursos', 'size': '12713'}