<jupyter_start><jupyter_text>ШАГ 1Посмотреть количество пропусков и как их заполнять; проверить на наличие дубликатов.<jupyter_code>quantity_nans = dict(df_train.isnull().sum())
quantity_nans_list = list(quantity_nans.items())
quantity_nans_list.sort(key = lambda x: x[1], reverse = True)
for item in quantity_nans_list:
    print(item[0],':',item[1])
np.round(100 - (df_train.dropna().shape[0] * 100 / df_train.shape[0] ))<jupyter_output><empty_output><jupyter_text>Если мы решим удалить все пропуски, мы удалим 75% датасета, поэтому нам необходимо заполнять пропуски.<jupyter_code>print('Количество дубликатов в датасете: {}'.format(df_train.shape[0] - df_train.drop_duplicates().shape[0]))<jupyter_output>Количество дубликатов в датасете: 0<jupyter_text>Прежде чем заполнять данные с помощью SimpleImputer, необходимо разобраться с 2 столбцами с категориальными данными. Будет оптимально взять столбцы с категориальными признаками, заполнить na в этих столбцах SimpleImputer(category='most_frequent'), затем разбить их OneHotEncoder ( может быть нужно будет сначала использовать LabelEncoder)затем все остальные столбцы заполнить SimpleImputer()<jupyter_code>from sklearn.impute import SimpleImputer 
most_frequent = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
median = SimpleImputer(missing_values=np.nan, strategy='median')
X_train_categorial = df_train[['GmWt_Desc1','GmWt_Desc2']]
most_frequent.fit(X_train_categorial)
X_train_filled_categorial = most_frequent.transform(X_train_categorial)
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder(handle_unknown = 'ignore')
ohe.fit(X_train_filled_categorial)
X_train_filled_categorial_ohe  = ohe.transform(X_train_filled_categorial)
X_train_filled_categorial_ohe  
X_train_filled_categorial_ohe.toarray().shape
df_categorial_filled_features = pd.DataFrame(X_train_filled_categorial_ohe.toarray())

df_categorial_filled_features.columns = list(ohe.categories_[0]) + list(ohe.categories_[1])
df_categorial_filled_features.shape<jupyter_output><empty_output><jupyter_text>Удалим категориальные столбцы и таргет из нашего датафрейма, заменив их на X_train_filled_categorial_ohe<jupyter_code>y_train = df_train['Energy_Kcal'].values
len(y_train) 
columns_to_del = ['Short_Description','GmWt_Desc1','GmWt_Desc2', 'Energy_Kcal']
for column in columns_to_del:
    if column in df_train:
        del df_train[column]
df_train.shape
df_train.shape, df_categorial_filled_features.shape
df_train_without_cat = pd.concat([df_train, df_categorial_filled_features],axis=1)
df_train_without_cat.shape
df_train_without_cat
median.fit(df_train_without_cat)
X_train_prepared = median.transform(df_train_without_cat)
X_train_prepared<jupyter_output><empty_output><jupyter_text>ШАГ 2Произведём нормировку данных<jupyter_code>from sklearn import feature_selection
VT = feature_selection.VarianceThreshold(.001).fit(X_train_prepared)
X_train_VT = VT.transform(X_train_prepared)
X_train_VT.shape
scaler = preprocessing.StandardScaler()
scaler.fit(X_train_VT)
X_train_norm = scaler.transform(X_train_VT)<jupyter_output><empty_output><jupyter_text>ШАГ 3Попробуем посчитать регрессию на наших данных X_train_prepared, y_train<jupyter_code>from sklearn.model_selection import GridSearchCV
import xgboost
model = xgboost.XGBRegressor(
    learning_rate = 0.1,
    n_estimators = 1000,
    max_depth = 8,
    n_jobs = -1
)
model.fit(X_train_norm, y_train)
metrics.mean_absolute_error(y_train, model.predict(X_train_norm))
model.predict(X_train_norm)<jupyter_output><empty_output><jupyter_text>Шаг 4Предскажем данные на тестовой выборке<jupyter_code>df_test = pd.read_csv('test.csv')
df_test.columns = columns
df_test.head()
X_test_categorial = df_test[['GmWt_1','GmWt_2']]
X_test_categorial.dtypes
most_frequent
X_test_filled_categorial = most_frequent.transform(X_test_categorial)
X_test_filled_categorial_ohe  = ohe.transform(X_test_filled_categorial)

df_categorial_filled_features_test = pd.DataFrame(X_test_filled_categorial_ohe.toarray())
df_categorial_filled_features_test.columns = list(ohe.categories_[0]) + list(ohe.categories_[1])

columns_to_del = ['Short_Description','GmWt_1','GmWt_2', 'Energy_Kcal']
for column in columns_to_del:
    if column in df_test:
        del df_test[column]

df_test.shape, df_categorial_filled_features_test.shape
df_test_without_cat = pd.concat([df_test, df_categorial_filled_features_test],axis=1)
df_test_without_cat.shape
df_test_without_cat
X_test_prepared = median.transform(df_test_without_cat)
X_test_VT = VT.transform(X_test_prepared)
X_test_norm = scaler.transform(X_test_VT)
model
y_pred = model.predict(X_test_norm)
y_pred
import csv
f = open("Pred_main.csv", "w")

with f:
    writer = csv.writer(f)
    
    writer.writerow(y_pred)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09214659685863874, 'fraction_numerical': 0.009633507853403141, 'mean_word_length': 9.140127388535031, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '424974', 'license': 'MIT', 'n_tokens_mistral': 1904, 'n_tokens_neox': 1936, 'n_words': 273, 'path': 'Main final.ipynb', 'repo_name': 'shannels/best-hack-2020', 'size': '4775'}