<jupyter_start><jupyter_text>Grupos de features:- **Categorial Ordinal:**    - TP_ (17-4-3 = 10)    - Questions: ```["Q001", "Q002", "Q003", "Q004", "Q005","Q006", "Q007", "Q008", "Q009", "Q010", "Q011", "Q012", "Q013", "Q014", "Q015", "Q016", "Q017", "Q019", "Q022", "Q024"]``` (20)                     - **Categorial Nominal:**    - IN_ : All Binary (52)     - TP_ : ```["TP_SEXO", "TP_ESTADO_CIVIL", "TP_COR_RACA", "TP_NACIONALIDADE"]``` (4)    - SG_ : (4-1 = 3)    - Questions: ```["Q018", "Q020", "Q021", "Q023", "Q025"]``` (5)    - **Numerical:**    - NU_IDADE (1)- Droped:    - Identificator: ```[NU_INSCRICAO]``` (1)    - More than 40% missing: ```['CO_ESCOLA', 'NO_MUNICIPIO_ESC', 'SG_UF_ESC', 'TP_DEPENDENCIA_ADM_ESC', 'TP_LOCALIZACAO_ESC', 'TP_SIT_FUNC_ESC']``` (4)    - NO_M: (To many categories): ```['NO_MUNICIPIO_RESIDENCIA', 'NO_MUNICIPIO_NASCIMENTO', 'NO_MUNICIPIO_PROVA']``` (3)    - NU_NOTA: Targets variables (5)<jupyter_code>train_df = pd.read_parquet("data/train.parquet")
clean_target(train_df)
#test= pd.read_parquet("data/test.parquet")
categorical_ordinal_columns = get_categorical_ordinal_columns(train_df)
qtd_categorical_ordinal_columns=len(categorical_ordinal_columns)
print(f"Number of categorial ordinal features: {qtd_categorical_ordinal_columns}")
categorical_nominal_columns = get_categorical_nominal_columns(train_df)
qtd_categorical_nominal_columns = len(categorical_nominal_columns)
print(f"Number of categorial nominal features: {qtd_categorical_nominal_columns}")
drop_columns = ["NU_INSCRICAO", 
                "CO_ESCOLA", 
                "NO_MUNICIPIO_ESC", 
                "SG_UF_ESC", 
                "TP_DEPENDENCIA_ADM_ESC", 
                "TP_LOCALIZACAO_ESC", 
                "TP_SIT_FUNC_ESC", 
                "NO_MUNICIPIO_RESIDENCIA", 
                "NO_MUNICIPIO_NASCIMENTO", 
                "NO_MUNICIPIO_PROVA"]
qtd_drop_columns = len(drop_columns)
print(f"Number of columns dropped: {qtd_drop_columns}")
target_columns = train_df.filter(regex="NU_NOTA").columns.tolist()
qtd_target_columns = len(target_columns)
print(f"Number of targets: {qtd_target_columns}")
numerical_columns = ["NU_IDADE"]
qtd_numerical_columns = len(numerical_columns)
print(f"Number of targets: {qtd_numerical_columns}")
target_columns = train_df.filter(regex="NU_NOTA").columns.tolist()
qtd_target_columns = len(target_columns)
print(f"Number of targets: {qtd_target_columns}")
all_columns = drop_columns + categorical_nominal_columns + categorical_ordinal_columns + numerical_columns + target_columns
qtd_total = qtd_drop_columns + qtd_categorical_nominal_columns + qtd_categorical_ordinal_columns + qtd_numerical_columns + qtd_target_columns
print(f"Total columns: {qtd_total}")<jupyter_output>Total columns: 110<jupyter_text>**Create Pipeline**<jupyter_code>"""
Variáveis categóricas com dados ordinais que tem dados faltantes:
- TP_ENSINO: Suposto que NaN representa a categoria faltante descrita nos metadados.
- TP_STATUS_REDACAO: Mapeado para outra classe (Faltou na prova)
"""
categorical_ordinal_pipe = Pipeline([
    ('selector', ColumnSelector(categorical_ordinal_columns)),
    ('imputer', SimpleImputer(missing_values=np.nan, 
                              strategy='constant', 
                              fill_value=0)),
    ('encoder', OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1))
])

"""
Variáveis categóricas com dados ordinais que tem dados faltantes:
- SG_UF_NASCIMENTO: Mapeado para uma nova categoria
"""
categorical_nominal_pipe = Pipeline([
    ('selector', ColumnSelector(categorical_nominal_columns)),
    ('imputer', SimpleImputer(missing_values=np.nan, 
                              strategy='constant', 
                              fill_value="missing")),
    ('encoder', OneHotEncoder(drop="first", handle_unknown='ignore'))
])

numerical_pipe = Pipeline([
    ('selector', ColumnSelector(numerical_columns)),
    ('imputer', SimpleImputer(missing_values=np.nan, 
                              strategy='constant', 
                              fill_value=0)),
    ('scaler', MinMaxScaler())
])


preprocessor = FeatureUnion([
    ('categorical_ordinal', categorical_ordinal_pipe),
    ('categorical_nominal', categorical_nominal_pipe),
    ('numerical', numerical_pipe)
])

kwargs_regressor = {"n_estimators":100,
                    "max_depth":20,
                    "n_jobs":-1,
                    "verbose":2}

pipe = Pipeline([
    ('preprocessor', preprocessor),
    ('model', RandomForestRegressor(**kwargs_regressor))
])
def presence_filter(df, key):
    cond = df.filter(regex=f"PRESENCA_{key}|STATUS_{key}").iloc[:,0] == 1
    return df.loc[cond, :], df.loc[~cond, :]


y_structure = {"CN":[], 
               "CH":[],
               "LC":[],
               "MT":[],
               "REDACAO":[]}
from joblib import dump
n_samples = 20000
sample = train_df.sample(n_samples)
    
for key, ys in tqdm(y_structure.items()):
    filtered_sample, missed_exam = presence_filter(sample, key)
    X = filtered_sample.drop(columns=target_columns+drop_columns)
    y = filtered_sample.filter(regex=f"NU_NOTA_{key}")
    
    X_train, X_test, y_train, y_test =  train_test_split(X, y, test_size=0.1, random_state=42)
    pipe.fit(X_train, y_train)
    dump(pipe, f"models/model_{key}.joblib") 
    y_train_hat = pipe.predict(X_train)
    y_test_hat = pipe.predict(X_test)
    ys.extend([y_train, y_test, y_train_hat, y_test_hat])
mean_train = []
mean_test = []
for key, ys in tqdm(y_structure.items()):
    train_error = mean_squared_error(ys[0], ys[2], squared=False)
    mean_train.append(train_error)
    test_error = mean_squared_error(ys[1], ys[3], squared=False)
    mean_test.append(test_error)
    print(key)
    print(f"Train: {train_error}")
    print(f"Test: {test_error}\n")
np.mean(mean_train)
np.mean(mean_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13993174061433447, 'fraction_numerical': 0.021331058020477817, 'mean_word_length': 4.145741878841089, 'pattern_counts': {'":': 9, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '427774', 'license': 'MIT', 'n_tokens_mistral': 2244, 'n_tokens_neox': 2104, 'n_words': 338, 'path': 'pipeline_3.ipynb', 'repo_name': 'rocabrera/kaggles_enem', 'size': '5860'}