<jupyter_start><jupyter_text>Содержание* [Импорт библиотек](Импортбиблиотек)* [Загрузка данных](Загрузкаданных)* [Модели](Модели)  + [raw_desc_tfidf_RF_specialization](raw_desc_tfidf_RF_specialization)    - [Моделируем](Моделируемraw_desc_tfidf_RF_specialization)    - [Отчёт](Отчётraw_desc_tfidf_RF_specialization)  + [raw_desc_tfidf_RF_prof_area](raw_desc_tfidf_RF_prof_area)    - [Моделируем](Моделируемraw_desc_tfidf_RF_prof_area)    - [Отчёт](Отчётraw_desc_tfidf_RF_prof_area)  + [lem_desc_tfidf_RF_prof_area](lem_desc_tfidf_RF_prof_area)    - [Моделируем](Моделируемlem_desc_tfidf_RF_prof_area)    - [Отчёт](Отчётlem_desc_tfidf_RF_prof_area)  + [lem_desc_tfidf_RF_specialization](lem_desc_tfidf_RF_specialization)    - [Моделируем](Моделируемlem_desc_tfidf_RF_specialization)    - [Отчёт](Отчётlem_desc_tfidf_RF_specialization)  + [raw_desc_tfidf_LR_prof_area_with_len](raw_desc_tfidf_LR_prof_area_with_len)    - [Моделируем](Моделируемraw_desc_tfidf_LR_prof_area_with_len)    - [Отчёт](Отчётraw_desc_tfidf_LR_prof_area_with_len)  + [lem_desc_tfidf_LR_prof_area_14000](lem_desc_tfidf_LR_prof_area_14000)    - [Моделируем](Моделируемlem_desc_tfidf_LR_prof_area_14000)    - [Отчёт](Отчётlem_desc_tfidf_LR_prof_area_14000)  + [lem_desc_tfidf_LR_prof_area_14000_only_Russia](lem_desc_tfidf_LR_prof_area_14000_only_Russia)    - [Моделируем](Моделируемlem_desc_tfidf_LR_prof_area_14000_only_Russia)    - [Отчёт](Отчётlem_desc_tfidf_LR_prof_area_14000_only_Russia)  + [lem_desc_resp_tfidf_LR_prof_area_10000_only_Russia](lem_desc_resp_tfidf_LR_prof_area_10000_only_Russia)    - [Моделируем](Моделируемlem_desc_resp_tfidf_LR_prof_area_10000_only_Russia)    - [Отчёт](Отчётlem_desc_resp_tfidf_LR_prof_area_10000_only_Russia)  + [lem_desc_req_tfidf_LR_prof_area_10000_only_Russia](lem_desc_req_tfidf_LR_prof_area_10000_only_Russia)    - [Моделируем](Моделируемlem_desc_req_tfidf_LR_prof_area_10000_only_Russia)    - [Отчёт](Отчётlem_desc_req_tfidf_LR_prof_area_10000_only_Russia)  + [lem_desc_con_tfidf_LR_prof_area_10000_only_Russia](lem_desc_con_tfidf_LR_prof_area_10000_only_Russia)    - [Моделируем](Моделируемlem_desc_con_tfidf_LR_prof_area_10000_only_Russia)    - [Отчёт](Отчётlem_desc_con_tfidf_LR_prof_area_10000_only_Russia)  + [lem_desc_resp_req_tfidf_LR_prof_area_10_Russia](lem_desc_resp_req_tfidf_LR_prof_area_10_Russia)    - [Моделируем](Моделируемlem_desc_resp_req_tfidf_LR_prof_area_10_Russia)    - [Отчёт](Отчётlem_desc_resp_req_tfidf_LR_prof_area_10_Russia)  + [lem_desc_resp_con_tfidf_LR_prof_area_10_Russia](lem_desc_resp_con_tfidf_LR_prof_area_10_Russia)    - [Моделируем](Моделируемlem_desc_resp_con_tfidf_LR_prof_area_10_Russia)    - [Отчёт](Отчётlem_desc_resp_con_tfidf_LR_prof_area_10_Russia)  + [lem_desc_req_con_tfidf_LR_prof_area_10_Russia](lem_desc_req_con_tfidf_LR_prof_area_10_Russia)    - [Моделируем](Моделируемlem_desc_req_con_tfidf_LR_prof_area_10_Russia)    - [Отчёт](Отчётlem_desc_req_con_tfidf_LR_prof_area_10_Russia)     + [lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia](lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia)    - [Моделируем](Моделируемlem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia)    - [Отчёт](Отчётlem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia)      + [lem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia](lem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia)    - [Моделируем](Моделируемlem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia)    - [Отчёт](Отчётlem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia)       Импорт библиотек[к содержанию](ксодержанию)<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
!pip install -U colabcode
import colabcode
colabcode.ColabCode(
    port=12000,
    authtoken='<KEY>',
    lab=True
)
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.ensemble import RandomForestClassifier
import joblib
from sklearn.linear_model import LogisticRegression
import pandas as pd
import sys, os
import pickle
import scipy.sparse as sp<jupyter_output><empty_output><jupyter_text>Загрузка данных[к содержанию](ксодержанию)<jupyter_code>!gdown --id 1A-59GMNuJWbrbCc17zvBPkOycSdCBdCA
df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['language'] == 'ru']
sys.path.append(os.path.abspath('../'))
from src.models import utils
from src.data import eda_utils
utils.salary_process(df)<jupyter_output>C:\Users\Lisa\MSU_diploma\src\models\utils.py:23: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df['salary_from'][df['salary_currency'] == curr] = df['salary_from'][df['salary_currency'] == curr].apply(lambda x: x*currency[curr])<jupyter_text>Модели[к содержанию](ксодержанию)  raw_desc_tfidf_RF_specialization[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = df[['raw_description', 'specialization', 'specializations_name']].dropna()
specializations = dict()
df['specializations_name'].apply(lambda x: eda_utils.fill_prof_area(specializations, eval(x)));
df['specialization'] = df['specialization'].apply(utils.decrease_specializations)
X = df[['raw_description', 'specialization']]
X_train, X_test, y_train, y_test = train_test_split(X[['raw_description']], X['specialization'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=4000)
tfidf.fit(X_train['raw_description'])
features = tfidf.transform(X_train['raw_description'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("raw_desc_tfidf.pkl","wb"))
del X_train
model = RandomForestClassifier()
model.fit(features, y_train)
del features
features_test = tfidf.transform(X_test['raw_description'])
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, 'raw_desc_tfidf_RF_specialization.pkl', compress=8)
joblib.dump(y_pred, 'y_pred_raw_desc_tfidf_RF_specialization.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['language'] == 'ru']
df['specialization'] = df['specialization'].apply(utils.decrease_specializations)
regression_report, salary = utils.report(df, y_test, y_pred, 'specialization')
report = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report.style.format(formatter='{:,.2f}').background_gradient()
report = pd.DataFrame(regression_report)
report.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
report.index = salary['true'].unique()
report.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>raw_desc_tfidf_RF_prof_area[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['language'] == 'ru']
utils.salary_process(df)
X = df[['raw_description', 'prof_area']]
X_train, X_test, y_train, y_test = train_test_split(X[['raw_description']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=4000)
tfidf.fit(X_train['raw_description'])
features = tfidf.transform(X_train['raw_description'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("raw_desc_tfidf.pkl","wb"))
del X_train
model = RandomForestClassifier()
model.fit(features, y_train)
del features
features_test = tfidf.transform(X_test['raw_description'])
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, 'raw_desc_tfidf_RF_prof_area.pkl', compress=8)
joblib.dump(y_pred, 'y_pred_raw_desc_tfidf_RF_prof_area.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
report_raw = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report_raw.style.format(formatter='{:,.2f}').background_gradient()
regression_report_raw = pd.DataFrame(regression_report)
regression_report_raw.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report_raw.index = salary['true'].unique()
regression_report_raw.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_tfidf_RF_prof_area[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>X = df[['lem_raw_desc', 'prof_area']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=4000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("lem_desc_tfidf.pkl","wb"))
del X_train
model = RandomForestClassifier()
model.fit(features, y_train)
del features
features_test = tfidf.transform(X_test['lem_raw_desc'])
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, 'lem_desc_tfidf_RF_prof_area.pkl', compress=8)
joblib.dump(y_pred, 'y_pred_lem_desc_tfidf_RF_prof_area.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report_lem = pd.DataFrame(regression_report)
regression_report_lem.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report_lem.index = salary['true'].unique()
regression_report_lem.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_tfidf_RF_specialization[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = df[['lem_raw_desc', 'specialization', 'specializations_name']].dropna()
specializations = dict()
df['specializations_name'].apply(lambda x: eda_utils.fill_prof_area(specializations, eval(x)));
df['specialization'] = df['specialization'].apply(utils.decrease_specializations)
X = df[['lem_raw_desc', 'specialization']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc']], X['specialization'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=4000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("lem_desc_tfidf.pkl","wb"))
del X_train
model = RandomForestClassifier()
model.fit(features, y_train)
del features
features_test = tfidf.transform(X_test['lem_raw_desc'])
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, 'lem_desc_tfidf_RF_specialization.pkl', compress=8)
joblib.dump(y_pred, 'y_pred_lem_desc_tfidf_RF_specialization.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['language'] == 'ru']
df['specialization'] = df['specialization'].apply(utils.decrease_specializations)
regression_report, salary = utils.report(df, y_test, y_pred, 'specialization')
report_lem = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report_lem.style.format(formatter='{:,.2f}').background_gradient()
report = pd.DataFrame(regression_report)
report.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
report.index = salary['true'].unique()
report.style.format(formatter='{:,.1f}').background_gradient()
report_lem.to_csv('lem_desc_tfidf_RF_specialization.csv')
report.to_csv('lem_desc_tfidf_RF_specialization.csv')<jupyter_output><empty_output><jupyter_text>raw_desc_tfidf_LR_prof_area_with_len[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>X = df[['len_description_responsibilities', 'len_description_requirements', 'len_description_conditions', 'raw_description', 'prof_area']]
X_train, X_test, y_train, y_test = train_test_split(X[['raw_description', 'len_description_responsibilities', 'len_description_requirements', 'len_description_conditions']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=4000)
tfidf.fit(X_train['raw_description'])
features = tfidf.transform(X_train['raw_description'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("raw_desc_tfidf.pkl","wb"))
# Добавляем фичи с длинами массивов
feat = ['len_description_responsibilities', 'len_description_requirements', 'len_description_conditions']
features = pd.DataFrame(features.toarray())
for feature in feat:
  features[feature] = X_train[feature]
  features[feature]= features[feature].fillna(features[feature].mode().values[0])
features = features.to_numpy()
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features
features_test = tfidf.transform(X_test['raw_description'])
# Добавляем фичи с длинами массивов
feat = ['len_description_responsibilities', 'len_description_requirements', 'len_description_conditions']
features_test = pd.DataFrame(features_test.toarray())
for feature in feat:
  features_test[feature] = X_test[feature]
  features_test[feature]= features_test[feature].fillna(features_test[feature].mode().values[0])
features_test = features_test.to_numpy()
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, 'raw_desc_tfidf_LR_prof_area_with_len.pkl', compress=8)
joblib.dump(y_pred, 'y_pred_raw_desc_tfidf_LR_prof_area_with_len.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report_lem = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report_lem.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
report = pd.DataFrame(regression_report)
report.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
report.index = salary['true'].unique()
report.style.format(formatter='{:,.1f}').background_gradient()
report.to_csv('regression_report_raw_desc_tfidf_LR_prof_area_with_len.csv')<jupyter_output><empty_output><jupyter_text>lem_desc_tfidf_LR_prof_area_14000[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>X = df[['lem_raw_desc', 'prof_area']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=14000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("lem_desc_tfidf.pkl","wb"))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features
features_test = tfidf.transform(X_test['lem_raw_desc'])
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, 'lem_desc_tfidf_LR_prof_area_14000.pkl', compress=8)
joblib.dump(y_pred, 'y_pred_lem_desc_tfidf_LR_prof_area_14000.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report_lem_LR = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report_lem_LR.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report_lem_LR = pd.DataFrame(regression_report)
regression_report_lem_LR.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report_lem_LR.index = salary['true'].unique()
regression_report_lem_LR.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_tfidf_LR_prof_area_14000_only_Russia[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['country_name'] == 'Россия']
utils.salary_process(df)
X = df[['lem_raw_desc', 'prof_area']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=14000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("lem_desc_tfidf.pkl","wb"))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features
features_test = tfidf.transform(X_test['lem_raw_desc'])
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, 'lem_desc_tfidf_LR_prof_area_14000_only_Russia.pkl', compress=8)
joblib.dump(y_pred, 'y_pred_lem_desc_tfidf_LR_prof_area_14000_only_Russia.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report_lem_LR = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report_lem_LR.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report_lem_LR = pd.DataFrame(regression_report)
regression_report_lem_LR.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report_lem_LR.index = salary['true'].unique()
regression_report_lem_LR.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_tfidf_LR_prof_area_14000_only_Russia[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['country_name'] == 'Россия']
utils.salary_process(df)
X = df[['lem_raw_desc', 'prof_area']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=14000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("lem_desc_tfidf.pkl","wb"))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features
features_test = tfidf.transform(X_test['lem_raw_desc'])
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, 'lem_desc_tfidf_LR_prof_area_14000_only_Russia.pkl', compress=8)
joblib.dump(y_pred, 'y_pred_lem_desc_tfidf_LR_prof_area_14000_only_Russia.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report_lem_LR = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report_lem_LR.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report_lem_LR = pd.DataFrame(regression_report)
regression_report_lem_LR.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report_lem_LR.index = salary['true'].unique()
regression_report_lem_LR.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_resp_tfidf_LR_prof_area_10000_only_Russia[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['country_name'] == 'Россия']
utils.salary_process(df)
X = df[['lem_raw_desc', 'prof_area', 'lem_desc_resp']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc', 'lem_desc_resp']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
tfidf2 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf2.fit(X_train['lem_desc_resp'])
features2 = tfidf2.transform(X_train['lem_desc_resp'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("../models/lem_desc_resp_tfidf_LR_prof_area_10000_only_Russia/lem_desc_tfidf.pkl","wb"))
pickle.dump(tfidf2.vocabulary_,open("../models/lem_desc_resp_tfidf_LR_prof_area_10000_only_Russia/lem_desc_resp_tfidf.pkl","wb"))
# Объединяем фичи
features = sp.hstack((features, features2))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features
features_test = tfidf.transform(X_test['lem_raw_desc'])
X_test['lem_desc_resp'][X_test['lem_desc_resp'].isnull()] = ''
features_test2 = tfidf2.transform(X_test['lem_desc_resp'])
features_test = sp.hstack((features_test, features_test2))
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, '../models/lem_desc_resp_tfidf_LR_prof_area_10000_only_Russia/lem_desc_resp_tfidf_LR_prof_area_10000_only_Russia.pkl', compress=8)
joblib.dump(y_pred, '../models/lem_desc_resp_tfidf_LR_prof_area_10000_only_Russia/y_pred_lem_desc_resp_tfidf_LR_prof_area_10000_only_Russia.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report_lem_LR = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report_lem_LR.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report_lem_LR = pd.DataFrame(regression_report)
regression_report_lem_LR.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report_lem_LR.index = salary['true'].unique()
regression_report_lem_LR.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_req_tfidf_LR_prof_area_10000_only_Russia[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['country_name'] == 'Россия']
utils.salary_process(df)
X = df[['lem_raw_desc', 'prof_area', 'lem_desc_req']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc', 'lem_desc_req']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
tfidf2 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf2.fit(X_train['lem_desc_req'])
features2 = tfidf2.transform(X_train['lem_desc_req'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("../models/lem_desc_req_tfidf_LR_prof_area_10000_only_Russia/lem_desc_tfidf.pkl","wb"))
pickle.dump(tfidf2.vocabulary_,open("../models/lem_desc_req_tfidf_LR_prof_area_10000_only_Russia/lem_desc_req_tfidf.pkl","wb"))
# Объединяем фичи
features = sp.hstack((features, features2))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features
del features2
features_test = tfidf.transform(X_test['lem_raw_desc'])
X_test['lem_desc_req'][X_test['lem_desc_req'].isnull()] = ''
features_test2 = tfidf2.transform(X_test['lem_desc_req'])
features_test = sp.hstack((features_test, features_test2))
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, '../models/lem_desc_req_tfidf_LR_prof_area_10000_only_Russia/lem_desc_req_tfidf_LR_prof_area_10000_only_Russia.pkl', compress=8)
joblib.dump(y_pred, '../models/lem_desc_req_tfidf_LR_prof_area_10000_only_Russia/y_pred_lem_desc_req_tfidf_LR_prof_area_10000_only_Russia.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report_lem_LR = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report_lem_LR.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report_lem_LR = pd.DataFrame(regression_report)
regression_report_lem_LR.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report_lem_LR.index = salary['true'].unique()
regression_report_lem_LR.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_con_tfidf_LR_prof_area_10000_only_Russia[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['country_name'] == 'Россия']
utils.salary_process(df)
X = df[['lem_raw_desc', 'prof_area', 'lem_desc_con']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc', 'lem_desc_con']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
tfidf2 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf2.fit(X_train['lem_desc_con'])
features2 = tfidf2.transform(X_train['lem_desc_con'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("../models/lem_desc_con_tfidf_LR_prof_area_10000_only_Russia/lem_desc_tfidf.pkl","wb"))
pickle.dump(tfidf2.vocabulary_,open("../models/lem_desc_con_tfidf_LR_prof_area_10000_only_Russia/lem_desc_con_tfidf.pkl","wb"))
# Объединяем фичи
features = sp.hstack((features, features2))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features, features2
features_test = tfidf.transform(X_test['lem_raw_desc'])
X_test['lem_desc_con'][X_test['lem_desc_con'].isnull()] = ''
features_test2 = tfidf2.transform(X_test['lem_desc_con'])
features_test = sp.hstack((features_test, features_test2))
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, '../models/lem_desc_con_tfidf_LR_prof_area_10000_only_Russia/lem_desc_con_tfidf_LR_prof_area_10000_only_Russia.pkl', compress=8)
joblib.dump(y_pred, '../models/lem_desc_con_tfidf_LR_prof_area_10000_only_Russia/y_pred_lem_desc_con_tfidf_LR_prof_area_10000_only_Russia.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report = pd.DataFrame(regression_report)
regression_report.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report.index = salary['true'].unique()
regression_report.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_resp_req_tfidf_LR_prof_area_10_Russia[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['country_name'] == 'Россия']
utils.salary_process(df)
X = df[['lem_raw_desc', 'prof_area', 'lem_desc_resp', 'lem_desc_req']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc', 'lem_desc_resp', 'lem_desc_req']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
tfidf2 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf2.fit(X_train['lem_desc_resp'])
features2 = tfidf2.transform(X_train['lem_desc_resp'])
tfidf3 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf3.fit(X_train['lem_desc_req'])
features3 = tfidf3.transform(X_train['lem_desc_req'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("../models/lem_desc_resp_req_tfidf_LR_prof_area_10_Russia/lem_desc_tfidf.pkl","wb"))
pickle.dump(tfidf2.vocabulary_,open("../models/lem_desc_resp_req_tfidf_LR_prof_area_10_Russia/lem_desc_resp_tfidf.pkl","wb"))
pickle.dump(tfidf3.vocabulary_,open("../models/lem_desc_resp_req_tfidf_LR_prof_area_10_Russia/lem_desc_req_tfidf.pkl","wb"))
# Объединяем фичи
features = sp.hstack((features, features2, features3))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features, features2, features3
features_test = tfidf.transform(X_test['lem_raw_desc'])
X_test['lem_desc_resp'][X_test['lem_desc_resp'].isnull()] = ''
X_test['lem_desc_req'][X_test['lem_desc_req'].isnull()] = ''
features_test2 = tfidf2.transform(X_test['lem_desc_resp'])
features_test3 = tfidf3.transform(X_test['lem_desc_req'])
features_test = sp.hstack((features_test, features_test2, features_test3))
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, '../models/lem_desc_resp_req_tfidf_LR_prof_area_10_Russia/lem_desc_resp_req_tfidf_LR_prof_area_10_Russia.pkl', compress=8)
joblib.dump(y_pred, '../models/lem_desc_resp_req_tfidf_LR_prof_area_10_Russia/y_pred_lem_desc_resp_req_tfidf_LR_prof_area_10_Russia.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report = pd.DataFrame(regression_report)
regression_report.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report.index = salary['true'].unique()
regression_report.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_resp_con_tfidf_LR_prof_area_10_Russia[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['country_name'] == 'Россия']
utils.salary_process(df)
X = df[['lem_raw_desc', 'prof_area', 'lem_desc_resp', 'lem_desc_con']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc', 'lem_desc_resp', 'lem_desc_con']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
tfidf2 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf2.fit(X_train['lem_desc_resp'])
features2 = tfidf2.transform(X_train['lem_desc_resp'])
tfidf3 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf3.fit(X_train['lem_desc_con'])
features3 = tfidf3.transform(X_train['lem_desc_con'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("../models/lem_desc_resp_con_tfidf_LR_prof_area_10_Russia/lem_desc_tfidf.pkl","wb"))
pickle.dump(tfidf2.vocabulary_,open("../models/lem_desc_resp_con_tfidf_LR_prof_area_10_Russia/lem_desc_resp_tfidf.pkl","wb"))
pickle.dump(tfidf3.vocabulary_,open("../models/lem_desc_resp_con_tfidf_LR_prof_area_10_Russia/lem_desc_con_tfidf.pkl","wb"))
# Объединяем фичи
features = sp.hstack((features, features2, features3))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features, features2, features3
features_test = tfidf.transform(X_test['lem_raw_desc'])
X_test['lem_desc_resp'][X_test['lem_desc_resp'].isnull()] = ''
X_test['lem_desc_con'][X_test['lem_desc_con'].isnull()] = ''
features_test2 = tfidf2.transform(X_test['lem_desc_resp'])
features_test3 = tfidf3.transform(X_test['lem_desc_con'])
features_test = sp.hstack((features_test, features_test2, features_test3))
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, '../models/lem_desc_resp_con_tfidf_LR_prof_area_10_Russia/lem_desc_resp_con_tfidf_LR_prof_area_10_Russia.pkl', compress=8)
joblib.dump(y_pred, '../models/lem_desc_resp_con_tfidf_LR_prof_area_10_Russia/y_pred_lem_desc_resp_con_tfidf_LR_prof_area_10_Russia.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report = pd.DataFrame(regression_report)
regression_report.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report.index = salary['true'].unique()
regression_report.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_req_con_tfidf_LR_prof_area_10_Russia[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['country_name'] == 'Россия']
utils.salary_process(df)
X = df[['lem_raw_desc', 'prof_area', 'lem_desc_req', 'lem_desc_con']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc', 'lem_desc_req', 'lem_desc_con']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
tfidf2 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf2.fit(X_train['lem_desc_req'])
features2 = tfidf2.transform(X_train['lem_desc_req'])
tfidf3 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=10000)
tfidf3.fit(X_train['lem_desc_con'])
features3 = tfidf3.transform(X_train['lem_desc_con'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("../models/lem_desc_req_con_tfidf_LR_prof_area_10_Russia/lem_desc_tfidf.pkl","wb"))
pickle.dump(tfidf2.vocabulary_,open("../models/lem_desc_req_con_tfidf_LR_prof_area_10_Russia/lem_desc_req_tfidf.pkl","wb"))
pickle.dump(tfidf3.vocabulary_,open("../models/lem_desc_req_con_tfidf_LR_prof_area_10_Russia/lem_desc_con_tfidf.pkl","wb"))
# Объединяем фичи
features = sp.hstack((features, features2, features3))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features, features2, features3
features_test = tfidf.transform(X_test['lem_raw_desc'])
X_test['lem_desc_req'][X_test['lem_desc_req'].isnull()] = ''
X_test['lem_desc_con'][X_test['lem_desc_con'].isnull()] = ''
features_test2 = tfidf2.transform(X_test['lem_desc_req'])
features_test3 = tfidf3.transform(X_test['lem_desc_con'])
features_test = sp.hstack((features_test, features_test2, features_test3))
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, '../models/lem_desc_req_con_tfidf_LR_prof_area_10_Russia/lem_desc_req_con_tfidf_LR_prof_area_10_Russia.pkl', compress=8)
joblib.dump(y_pred, '../models/lem_desc_req_con_tfidf_LR_prof_area_10_Russia/y_pred_lem_desc_req_con_tfidf_LR_prof_area_10_Russia.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report = pd.DataFrame(regression_report)
regression_report.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report.index = salary['true'].unique()
regression_report.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['country_name'] == 'Россия']
utils.salary_process(df)
X = df[['lem_raw_desc', 'prof_area', 'lem_desc_resp', 'lem_desc_req', 'lem_desc_con']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_raw_desc', 'lem_desc_resp', 'lem_desc_req', 'lem_desc_con']], X['prof_area'], test_size=0.2, random_state=42)
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=5000)
tfidf.fit(X_train['lem_raw_desc'])
features = tfidf.transform(X_train['lem_raw_desc'])
tfidf2 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=5000)
tfidf2.fit(X_train['lem_desc_req'])
features2 = tfidf2.transform(X_train['lem_desc_req'])
tfidf3 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=5000)
tfidf3.fit(X_train['lem_desc_con'])
features3 = tfidf3.transform(X_train['lem_desc_con'])
tfidf4 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=5000)
tfidf4.fit(X_train['lem_desc_resp'])
features4 = tfidf4.transform(X_train['lem_desc_resp'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("../models/lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/lem_desc_tfidf.pkl","wb"))
pickle.dump(tfidf2.vocabulary_,open("../models/lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/lem_desc_req_tfidf.pkl","wb"))
pickle.dump(tfidf3.vocabulary_,open("../models/lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/lem_desc_con_tfidf.pkl","wb"))
pickle.dump(tfidf4.vocabulary_,open("../models/lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/lem_desc_resp_tfidf.pkl","wb"))
# Объединяем фичи
features = sp.hstack((features, features2, features3, features4))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features, features2, features3, features4
features_test = tfidf.transform(X_test['lem_raw_desc'])
X_test['lem_desc_resp'][X_test['lem_desc_resp'].isnull()] = ''
X_test['lem_desc_req'][X_test['lem_desc_req'].isnull()] = ''
X_test['lem_desc_con'][X_test['lem_desc_con'].isnull()] = ''
features_test2 = tfidf2.transform(X_test['lem_desc_req'])
features_test3 = tfidf3.transform(X_test['lem_desc_con'])
features_test4 = tfidf4.transform(X_test['lem_desc_resp'])
features_test = sp.hstack((features_test, features_test2, features_test3, features_test4))
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, '../models/lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia.pkl', compress=8)
joblib.dump(y_pred, '../models/lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/y_pred_lem_desc_resp_req_con_tfidf_LR_prof_area_5_Russia.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report = pd.DataFrame(regression_report)
regression_report.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report.index = salary['true'].unique()
regression_report.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output><jupyter_text>lem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia[к содержанию](ксодержанию)  Моделируем[к содержанию](ксодержанию)<jupyter_code>df = pd.read_csv('vacancies_lemmatized.csv')
df = df[df['country_name'] == 'Россия']
utils.salary_process(df)
X = df[['lem_desc', 'prof_area', 'lem_desc_resp', 'lem_desc_req', 'lem_desc_con']]
X_train, X_test, y_train, y_test = train_test_split(X[['lem_desc', 'lem_desc_resp', 'lem_desc_req', 'lem_desc_con']], X['prof_area'], test_size=0.2, random_state=42)
X_train['lem_desc'][X_train['lem_desc'].isnull()] = ''
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=5000)
tfidf.fit(X_train['lem_desc'])
features = tfidf.transform(X_train['lem_desc'])
tfidf2 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=5000)
tfidf2.fit(X_train['lem_desc_req'])
features2 = tfidf2.transform(X_train['lem_desc_req'])
tfidf3 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=5000)
tfidf3.fit(X_train['lem_desc_con'])
features3 = tfidf3.transform(X_train['lem_desc_con'])
tfidf4 = TfidfVectorizer(sublinear_tf=True, min_df=5,
                        ngram_range=(1, 2), 
                        stop_words={'russian', 'english'}, max_features=5000)
tfidf4.fit(X_train['lem_desc_resp'])
features4 = tfidf4.transform(X_train['lem_desc_resp'])
# Сохраняем tfidf.vocabulary_
pickle.dump(tfidf.vocabulary_,open("../models/lem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/lem_notraw_desc_tfidf.pkl","wb"))
pickle.dump(tfidf2.vocabulary_,open("../models/lem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/lem_desc_req_tfidf.pkl","wb"))
pickle.dump(tfidf3.vocabulary_,open("../models/lem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/lem_desc_con_tfidf.pkl","wb"))
pickle.dump(tfidf4.vocabulary_,open("../models/lem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/lem_desc_resp_tfidf.pkl","wb"))
# Объединяем фичи
features = sp.hstack((features, features2, features3, features4))
del X_train
model = LogisticRegression()
model.fit(features, y_train)
del features, features2, features3, features4
X_test['lem_desc'][X_test['lem_desc'].isnull()] = ''
features_test = tfidf.transform(X_test['lem_desc'])
X_test['lem_desc_resp'][X_test['lem_desc_resp'].isnull()] = ''
X_test['lem_desc_req'][X_test['lem_desc_req'].isnull()] = ''
X_test['lem_desc_con'][X_test['lem_desc_con'].isnull()] = ''
features_test2 = tfidf2.transform(X_test['lem_desc_req'])
features_test3 = tfidf3.transform(X_test['lem_desc_con'])
features_test4 = tfidf4.transform(X_test['lem_desc_resp'])
features_test = sp.hstack((features_test, features_test2, features_test3, features_test4))
y_pred = model.predict(features_test)
# Сохраняем модель и y_pred
joblib.dump(model, '../models/lem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/model.pkl', compress=8)
joblib.dump(y_pred, '../models/lem_notraw_desc_resp_req_con_tfidf_LR_prof_area_5_Russia/y_pred.pkl')<jupyter_output><empty_output><jupyter_text>Отчёт[к содержанию](ксодержанию)<jupyter_code>report = pd.DataFrame(metrics.classification_report(y_test, y_pred, output_dict=True)).T
report.style.format(formatter='{:,.2f}').background_gradient()
regression_report, salary = utils.report(df, y_test, y_pred, 'prof_area')
regression_report = pd.DataFrame(regression_report)
regression_report.columns = ['MAE', 'salary_from_mean', 'salary_from_std']
regression_report.index = salary['true'].unique()
regression_report.style.format(formatter='{:,.1f}').background_gradient()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13472595289364442, 'fraction_numerical': 0.019002375296912115, 'mean_word_length': 8.369384359400998, 'pattern_counts': {'":': 0, '<': 137, '<?xml version=': 0, '>': 137, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '544011', 'license': 'FTL', 'n_tokens_mistral': 18521, 'n_tokens_neox': 18296, 'n_words': 1747, 'path': 'notebooks/prof_area_specialization_classification.ipynb', 'repo_name': 'alievaliza/diploma', 'size': '45091'}