<jupyter_start><jupyter_text>Resolvendo o problema do Titanic no Kaggle (<NAME>)Para esse exemplo vamos seguir a playlist do **<NAME> no Youtube** [Resolva o Titanic Como um Campeão do Kaggle](https://www.youtube.com/playlist?list=PLwnip85KhroW8Q1JSNbgl06iNPeC0SDkx).  Lesson 01  01.1 - Importa as bibliotecas necessárias:<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>01.2 - Pega os conjunto de dados de treino e teste:<jupyter_code>train = pd.read_csv("datasets/train.csv")
test = pd.read_csv("datasets/test.csv")<jupyter_output><empty_output><jupyter_text>01.3 - Exibe os dados de treino e dados de teste:<jupyter_code>train.info()
train.head()
test.info()
test.head()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 418 entries, 0 to 417
Data columns (total 11 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  418 non-null    int64  
 1   Pclass       418 non-null    int64  
 2   Name         418 non-null    object 
 3   Sex          418 non-null    object 
 4   Age          332 non-null    float64
 5   SibSp        418 non-null    int64  
 6   Parch        418 non-null    int64  
 7   Ticket       418 non-null    object 
 8   Fare         417 non-null    float64
 9   Cabin        91 non-null     object 
 10  Embarked     418 non-null    object 
dtypes: float64(2), int64(4), object(5)
memory usage: 36.0+ KB<jupyter_text>**NOTE:**  Veja que os dados de teste não tem a **feature "Survived"**, ou seja, a que vamos tentar prever.  01.4 - importa e instancia o módulo RandomForestClassifier:<jupyter_code>from sklearn.ensemble import RandomForestClassifier

# Instância.
model = RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=0)<jupyter_output><empty_output><jupyter_text>01.5 - Cria uma função para categorizar (binarizar) os sexos: - **1** se for feminino (female) - **0** se for masculuno (male)<jupyter_code>def sex_transform(value):
  if value == 'female':
    return 1
  else:
    return 0<jupyter_output><empty_output><jupyter_text>01.6 - Cria uma nova feature (binary_sex) com a função sex_transform()<jupyter_code>train['Binary_sex'] = train['Sex'].map(sex_transform)
train.head()<jupyter_output><empty_output><jupyter_text>**NOTE:**  Veja que agora nós temos a *feature* **"binary_sex"**.  01.7 - Contabiliza quantas das amostras são do sexo feminino (female/1) ou masculino (male/0):<jupyter_code>train['Sex'].value_counts()
train['Binary_sex'].value_counts()<jupyter_output><empty_output><jupyter_text>01.8 - Seleciona as features que nós vamos utilizar para treinar nosso modelo:<jupyter_code>variables = ['Binary_sex', 'Age']
variables<jupyter_output><empty_output><jupyter_text>01.9 - Por fim, abstrai as variáveis que serão enviadas para o modelo treinar: - **x** - Variáveis independentes; - **y** - Variável dependente (target).<jupyter_code>x = train[variables]
y = train['Survived']<jupyter_output><empty_output><jupyter_text>01.10 - Exibe as variáveis antes de iniciar o treinamento para nós verificarmos o que está acontecendo:<jupyter_code>x.head()
y.head()<jupyter_output><empty_output><jupyter_text>01.11 - Tratando os dados missing (ausentes):  Outra observação é que nosso modelo não consegue treinar com dados **missing (ausentes / NaN)**. Sabendo disso nós vamos: - Verificar o número de dados missing em cada features; - Ver quanto porcento representa esses dados faltantes; - E por fim, aplicar alguma técnica para tratar esses dados que estão faltando.<jupyter_code># Data missing sum.
missing = x.isnull().sum()
missing
# Data missing in percent.
percentMissing = (missing / len(x.index)) * 100
percentMissing<jupyter_output><empty_output><jupyter_text>---<jupyter_code># Data missing sum.
missing = y.isnull().sum()
missing
# Data missing in percent.
percentMissing = (missing / len(y.index)) * 100
percentMissing<jupyter_output><empty_output><jupyter_text>**NOTE:**  Preenche os dados missing com **-1** da *feature* **"Age"**.<jupyter_code>x = x.fillna(-1)
# Data missing sum.
missing = x.isnull().sum()
missing
# Data missing in percent.
percentMissing = (missing / len(x.index)) * 100
percentMissing<jupyter_output><empty_output><jupyter_text>Lesson 02  02.1 - Treinando o modelo<jupyter_code>model.fit(x, y)<jupyter_output><empty_output><jupyter_text>02.2 - Classifica (binariza) a feature "Sex" dos dados de teste com o método sex_transform:<jupyter_code>test['Binary_sex'] = test['Sex'].map(sex_transform)<jupyter_output><empty_output><jupyter_text>02.3 - Prepara os dados de teste:Aqui nós vamos aplicar o mesmo processamento dos dados treino para testar nosso modelo.<jupyter_code>x_test = test[variables]
x_test = x_test.fillna(-1)
x_test.head()<jupyter_output><empty_output><jupyter_text>**NOTE:**   - Vejam que aqui nós utilizamos as mesmas features do dados de treino (Binary_sex e Age) - E Removemos os dados ausentes (missing).  02.4 - Tenta prever com os dados de treino:<jupyter_code># tentou prever quais sobreviveram e quais morreram.
predicted = model.predict(x_test)
predicted<jupyter_output><empty_output><jupyter_text>02.5 - Prepara os dados para submeter a avaliação do Kaggle:<jupyter_code># Relaciona o ID do passageiro dos dados de treino com o resultado da nossa previsão
sub = pd.Series(predicted, index=test['PassengerId'], name='Survived')
sub
# Verifica o as dimensões (shape).
sub.shape
# Salva o nosso resulta no formato ".csv".
sub.to_csv("first_model.csv", header=True)
!head -n20 first_model.csv<jupyter_output>PassengerId,Survived
892,0
893,1
894,0
895,1
896,1
897,0
898,1
899,0
900,1
901,0
902,0
903,0
904,1
905,0
906,1
907,1
908,0
909,0
910,1<jupyter_text>Lesson 03  03.1 - Demonstra como o método "train_test_split" funciona:<jupyter_code>from sklearn.model_selection import train_test_split
x_false = np.arange(10)
x_false
train_test_split(x_false, test_size=0.5, random_state=0)<jupyter_output><empty_output><jupyter_text>03.2 - Divide os dados do conjunto de treino em: - Dados de treino; - Dados de validação.<jupyter_code>x_train, x_valid, y_train, y_valid = train_test_split(x, y, test_size=0.5)<jupyter_output><empty_output><jupyter_text>03.3 - Verifica as dimensões (shape):<jupyter_code>x_train.shape, x_valid.shape, y_train.shape, y_valid.shape<jupyter_output><empty_output><jupyter_text>03.4 - Treina o modelo com o conjunto de dados separado para treino:<jupyter_code>model.fit(x_train, y_train)<jupyter_output><empty_output><jupyter_text>03.5 - Faz predições com os dados de validação (x_valid):<jupyter_code>predicted2 = model.predict(x_valid)
predicted2<jupyter_output><empty_output><jupyter_text>03.6 - Verifica se os valores previstos foram iguais aos de validação (y_valid):<jupyter_code>y_valid == predicted2<jupyter_output><empty_output><jupyter_text>03.7 - Verifica accuracy:  "O número de acertos dividido pelo número total de exemplos (amostras)"<jupyter_code>np.mean(y_valid == predicted2)<jupyter_output><empty_output><jupyter_text>Lesson 04  04.1 - Demonstra como a função KFold funciona em um exemplo abstrato (lista qualquer):<jupyter_code># Lista de teste:
x_false
from sklearn.model_selection import KFold
#?KFold # Ver a documentação do método KFold.
kf = KFold(3, shuffle=True, random_state=0)
for train_line, valid_line in kf.split(x_false):
  print("Train", train_line)
  print("Valid", valid_line)
  print()<jupyter_output>Train [0 1 3 5 6 7]
Valid [2 4 8 9]

Train [0 2 3 4 5 8 9]
Valid [1 6 7]

Train [1 2 4 6 7 8 9]
Valid [0 3 5]<jupyter_text>04.2 - Aplica o método KFold nos nossos dados de treino e validação e ver "accuracy" em cada bloco e por fim verifica a média de todas "accuracy":<jupyter_code>kf = KFold(3, shuffle=True, random_state=0)
results = []

for train_line, valid_line in kf.split(x):
  print("Train", train_line.shape[0])
  print("Valid", valid_line.shape[0])
    
  x_train, x_valid = x.iloc[train_line], x.iloc[valid_line]
  y_train, y_valid = y.iloc[train_line], y.iloc[valid_line]

  model = RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=0)
  model.fit(x_train, y_train)
    
  predited3 = model.predict(x_valid)
  accuracy = np.mean(y_valid == predited3)
  print("Current accuracy:", accuracy)
  print()

  results.append(accuracy)
results
np.mean(results)<jupyter_output><empty_output><jupyter_text>Lesson 5  05.1 - Aplica o método KFold com "RepeatedKFold":<jupyter_code>from sklearn.model_selection import RepeatedKFold
#?RepeatedKFold # Ver a documentação do método.
results = []

# 2 divisões (n_splits) e repetir 10x (n_repeats)
kf = RepeatedKFold(n_splits=2, n_repeats=10, random_state=10)

for train_line, valid_line in kf.split(x):
  print("Train", train_line.shape[0])
  print("Valid", valid_line.shape[0])
    
  x_train, x_valid = x.iloc[train_line], x.iloc[valid_line]
  y_train, y_valid = y.iloc[train_line], y.iloc[valid_line]

  model = RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=0)
  model.fit(x_train, y_train)

  predited4 = model.predict(x_valid)
  accuracy = np.mean(y_valid == predited4)
  print("Current accuracy:", accuracy)
  print()

  results.append(accuracy)<jupyter_output>Train 445
Valid 446
Current accuracy: 0.7511210762331838

Train 446
Valid 445
Current accuracy: 0.7617977528089888

Train 445
Valid 446
Current accuracy: 0.7533632286995515

Train 446
Valid 445
Current accuracy: 0.7595505617977528

Train 445
Valid 446
Current accuracy: 0.7690582959641256

Train 446
Valid 445
Current accuracy: 0.7573033707865169

Train 445
Valid 446
Current accuracy: 0.7399103139013453

Train 446
Valid 445
Current accuracy: 0.7595505617977528

Train 445
Valid 446
Current accuracy: 0.7354260089686099

Train 446
Valid 445
Current accuracy: 0.797752808988764

Train 445
Valid 446
Current accuracy: 0.7690582959641256

Train 446
Valid 445
Current accuracy: 0.7640449438202247

Train 445
Valid 446
Current accuracy: 0.7757847533632287

Train 446
Valid 445
Current accuracy: 0.7573033707865169

Train 445
Valid 446
Current accuracy: 0.7556053811659192

Train 446
Valid 445
Current accuracy: 0.7640449438202247

Train 445
Valid 446
Current accuracy: 0.7443946188340808

Train 446
Valid[...]<jupyter_text>05.2 - Demons visualmente os nossos resultados (KFolds):<jupyter_code># Exibe os gráficos na prória celular
%matplotlib inline
%pylab inline
pylab.hist(results)
np.mean(results)<jupyter_output><empty_output><jupyter_text>05.3 - Separa novas features para o nosso modelo aprender:<jupyter_code>train.head()
new_variables = [
  'Binary_sex',
  'Age',
  'Pclass',
  'SibSp',
  'Parch',
  'Fare'
]
x = train[new_variables].fillna(-1)
y = train['Survived']
x.head()
y.head()<jupyter_output><empty_output><jupyter_text>05.4 - Treina e verifica "accuracy" do modelo com as novas features:<jupyter_code>results = []
kf = RepeatedKFold(n_splits=2, n_repeats=10, random_state=10)

for train_line, valid_line in kf.split(x):
  print("Train", train_line.shape[0])
  print("Valid", valid_line.shape[0])
    
  x_train, x_valid = x.iloc[train_line], x.iloc[valid_line]
  y_train, y_valid = y.iloc[train_line], y.iloc[valid_line]

  model = RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=0)
  model.fit(x_train, y_train)

  predited5 = model.predict(x_valid)
  accuracy = np.mean(y_valid == predited5)
  print("Current accuracy:", accuracy)
  print()

  results.append(accuracy)
pylab.hist(results)
np.mean(results)<jupyter_output><empty_output><jupyter_text>05.5 - Prepara os dados para validar com o conjunto de dados de teste: **Pega os conjuntos de dados de treino e teste:**<jupyter_code>df_train = pd.read_csv("datasets/train.csv") # Training set.
df_test = pd.read_csv("datasets/test.csv") # Testing set.<jupyter_output><empty_output><jupyter_text>**Categoriza (binariza) a feature "Sex" para os dois conjuntos de dados:**<jupyter_code># Categoriza (binariza) a feature "Sex" para os dados de treino e teste.
df_train['Binary_sex'] = df_train['Sex'].map(sex_transform)
df_test['Binary_sex'] = df_test['Sex'].map(sex_transform)<jupyter_output><empty_output><jupyter_text>**Overview dos conjuntos de dados de treino e teste:**<jupyter_code>df_train.head()
df_test.head()<jupyter_output><empty_output><jupyter_text>**Separa as variáveis (features) independentes e dependente (target) que nosso modelo utilizará para aprender:**<jupyter_code>x = df_train[new_variables].fillna(-1)
y = df_train['Survived']
x.head()
y.head()<jupyter_output><empty_output><jupyter_text>**Treina o modelo com o conjunto de dados de treino:**<jupyter_code>model = RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=0)
model.fit(x, y)<jupyter_output><empty_output><jupyter_text>**Faz predições com os dados de teste e as novas features:**  Antes de fazer as previsões com os dados de teste vamos: - Configurar quais features serão utilizadas para fazer a previsão; - E preencher os valores ausentes (missing).<jupyter_code>x_test = df_test[new_variables].copy()
x_test = x_test.fillna(-1)
x_test.head()
predicted6 = model.predict(x_test)
predicted6<jupyter_output><empty_output><jupyter_text>Prepara os dados novamente para fazer outra submisssão no Kaggle:<jupyter_code># Relaciona o ID do passageiro dos dados de treino com o resultado da nossa previsão
sub2 = pd.Series(predicted6, index=test['PassengerId'], name='Survived')
sub2
# Verifica o as dimensões (shape).
sub2.shape
# Salva o nosso resulta no formato ".csv".
sub.to_csv("second_model.csv", header=True)
!head -n20 second_model.csv<jupyter_output>PassengerId,Survived
892,0
893,1
894,0
895,1
896,1
897,0
898,1
899,0
900,1
901,0
902,0
903,0
904,1
905,0
906,1
907,1
908,0
909,0
910,1
{'fraction_non_alphanumeric': 0.11233318668426455, 'fraction_numerical': 0.06261915236838246, 'mean_word_length': 5.557211538461538, 'pattern_counts': {'":': 2, '<': 151, '<?xml version=': 0, '>': 151, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '169881', 'license': 'MIT', 'n_tokens_mistral': 5789, 'n_tokens_neox': 5213, 'n_words': 1252, 'path': 'modules/lives/modules/ai/titanic/titanic-notebook.ipynb', 'repo_name': 'drigols/Studies', 'size': '13648'}