<jupyter_start><jupyter_text>We have a very good explored dataset to this step. In this notebook, we are going to apply feature selection and feature engineering to get our dataset ready to use it on modeling.  Imports<jupyter_code>import pandas as pd
import numpy as np

import sys
sys.path.append('../source/')

import helpers
movies_df = pd.read_csv("../data/processed/transformation/movies_explored_list.csv", index_col=0)
data = movies_df.copy()
data = data.reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Data Transformation First thing? Separate our `release_date` and create new `year` and `month` column.<jupyter_code>data['year'] = pd.DatetimeIndex(data['release_date']).year
data['month'] = pd.DatetimeIndex(data['release_date']).month
data.drop("release_date", axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>We need to filter out by status. We need only the ones with `Released` one.<jupyter_code>data = data[data["status"] == "Released"]<jupyter_output><empty_output><jupyter_text>We decided to go only for movies from after 1980 to until now, we will reduce the dataset to only those movies.<jupyter_code>data = data[data["year"] >= 1980]<jupyter_output><empty_output><jupyter_text>We should not lose our focus and keeping in mind why we are doing all these transformations. It's because of these problems we want to solve.- Is a movie going to succeed?- Revenue our movie is going to make. As you may know on both problems we can provide a target as we face a supervised learning system. We need to use two different types of algorithms per separate - classification vs regression - so we need to be cautious to don't interfere with one to another.  Feature Selection On Classification model our **Target** is `vote_average`.We discovered it on the previous step, but as we are working on classification model, we are going to drop values equal to 0.<jupyter_code>data = data[data.vote_average > 0]
data.shape<jupyter_output><empty_output><jupyter_text>There are still columns that are not needed anymore. Time to drop them :)<jupyter_code>columns_to_remove = ["original_title", "video", "popularity"]
data.drop(columns_to_remove, axis=1, inplace=True)
data.head()<jupyter_output><empty_output><jupyter_text>Final feature selection: - adult - budget - genres - original_language - revenue - runtime - spoken_languages - status - title - vote_average - vote_count - cast - crew - year - month   Feature Engineering Our dataset is mainly composed by categorical columns. In order to help our models to predict what we want, we need to encode them.  Label Encoding  Genres<jupyter_code>helpers.create_new_columns(data, "genres")
data.drop("genres", axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Spoken Languages<jupyter_code>helpers.create_new_columns(data, "spoken_languages")
data.drop("spoken_languages", axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Cast<jupyter_code>data = data.reset_index(drop=True)
helpers.get_average_people(data, data.cast, data.year)
data.head()<jupyter_output><empty_output><jupyter_text>We finalize our work here. We selected the best features we can and encoded our categorical variables to be ready for modeling.Please, go to the next notebooks called `5.1.Model_Classification.ipynb` and `5.2.Model_Regression.ipynb`.<jupyter_code>data.to_csv("../data/processed/modeling/data_df_modeling.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09587449157466589, 'fraction_numerical': 0.0055200464846019755, 'mean_word_length': 6.420258620689655, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '248572', 'license': 'MIT', 'n_tokens_mistral': 1071, 'n_tokens_neox': 1035, 'n_words': 340, 'path': 'notebooks/4.Data_Wrangling.ipynb', 'repo_name': 'sideround/predict-revenue-new-releases', 'size': '3442'}