<jupyter_start><jupyter_text>EDA  Belongs To Collection<jupyter_code>for i in range(10):
    print(train['belongs_to_collection'][i])
train.isnull().sum()
test.isnull().sum()
train.drop(['belongs_to_collection'], axis=1, inplace=True)
test.drop(['belongs_to_collection'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Genres<jupyter_code>for i in range(10):
    print(train['genres'][i])<jupyter_output>[{'id': 35, 'name': 'Comedy'}]
[{'id': 35, 'name': 'Comedy'}, {'id': 18, 'name': 'Drama'}, {'id': 10751, 'name': 'Family'}, {'id': 10749, 'name': 'Romance'}]
[{'id': 18, 'name': 'Drama'}]
[{'id': 53, 'name': 'Thriller'}, {'id': 18, 'name': 'Drama'}]
[{'id': 28, 'name': 'Action'}, {'id': 53, 'name': 'Thriller'}]
[{'id': 16, 'name': 'Animation'}, {'id': 12, 'name': 'Adventure'}, {'id': 10751, 'name': 'Family'}]
[{'id': 27, 'name': 'Horror'}, {'id': 53, 'name': 'Thriller'}]
[{'id': 99, 'name': 'Documentary'}]
[{'id': 28, 'name': 'Action'}, {'id': 35, 'name': 'Comedy'}, {'id': 10402, 'name': 'Music'}, {'id': 10751, 'name': 'Family'}, {'id': 12, 'name': 'Adventure'}]
[{'id': 35, 'name': 'Comedy'}, {'id': 10402, 'name': 'Music'}]<jupyter_text>We Should just extract the useful info (in this case we just need the genre of the movie)<jupyter_code>print(train.iloc[1, :])
def get_dict(s):
    try:
        d = eval(s)
    except:
        d = {}
    return d
gn = pd.DataFrame(columns=['genre_name'])
gn
train = pd.concat([train, gn], axis=1)
train.head()
test = pd.concat([test, gn], axis=1)
test.head()
j = 0
for i in train['genres']:
    d = get_dict(i)
    if d != {}:
        train['genre_name'][j] = d[0]['name'] 
    else:
         train['genre_name'][j] = np.NaN
    j += 1

j = 0
for i in test['genres']:
    d = get_dict(i)
    if d != {}:
        test['genre_name'][j] = d[0]['name'] 
    else:
         test['genre_name'][j] = np.NaN
    j += 1
train.drop(['genres'], axis=1, inplace=True)
test.drop(['genres'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Budget<jupyter_code>train['budget'].unique()<jupyter_output><empty_output><jupyter_text>Zeros in budget mean unknown, we will treat it as missing values later<jupyter_code>train[train['budget'] == 0].shape[0]
plt.subplots(figsize=(12, 9))
plt.scatter(x=train['budget'], y=train['revenue'])<jupyter_output><empty_output><jupyter_text>Homepage<jupyter_code>train['homepage'].value_counts(dropna=False)
has_hompage = pd.DataFrame(columns=['has_homepage'])
train = pd.concat([train, has_hompage], axis=1)
test = pd.concat([test, has_hompage], axis=1)
j = 0
for i in train['homepage']:
    if str(train['homepage'][j]) == 'nan':
        train['has_homepage'][j] = 0
    else:
        train['has_homepage'][j] = 1
    j += 1

j = 0
for i in test['homepage']:
    if str(test['homepage'][j]) == 'nan':
        test['has_homepage'][j] = 0
    else:
        test['has_homepage'][j] = 1
    j += 1
sns.catplot(x='has_homepage', y='revenue', data=train)<jupyter_output><empty_output><jupyter_text>The scatter plot above seems interesting, this really shows that having a wepage for the movie is really affects its revenue<jupyter_code>train.drop(['homepage'], axis=1, inplace=True)
test.drop(['homepage'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Imdb Id<jupyter_code>train['imdb_id']<jupyter_output><empty_output><jupyter_text>There is no useful info in the imdb id<jupyter_code>train.drop(['imdb_id'], axis=1, inplace=True)
test.drop(['imdb_id'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Original Language<jupyter_code>print(len(train['original_language'].value_counts(dropna=False)))
train['original_language'].value_counts(dropna=False)
print(len(test['original_language'].value_counts(dropna=False)))
test['original_language'].value_counts(dropna=False)
plt.subplots(figsize=(12, 9))
sns.boxplot(x=train['original_language'], y=train['revenue'])<jupyter_output><empty_output><jupyter_text>It would be great to make a new feature (Is Original Language is English)<jupyter_code>isOrgEn = pd.DataFrame(columns=['is_en_original_language'])
train = pd.concat([train, isOrgEn], axis=1)
test = pd.concat([test, isOrgEn], axis=1)

print(train['original_language'][0])
j = 0
for i in train['original_language']:
    if i == 'en':
        train['is_en_original_language'][j] = 1
    else:
        train['is_en_original_language'][j] = 0
    j += 1
    
j = 0
for i in test['original_language']:
    if i == 'en':
        test['is_en_original_language'][j] = 1
    else:
        test['is_en_original_language'][j] = 0
    j += 1
sns.catplot(x='is_en_original_language', y='revenue', data=train)<jupyter_output><empty_output><jupyter_text>The feature we generated is great indeed  Original Title<jupyter_code>len(train['original_title'].value_counts())
train.drop(['original_title'], axis=1, inplace=True)
test.drop(['original_title'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Overview<jupyter_code>for i in range(5): 
    print(train['overview'][i])
    print("--------------------")<jupyter_output>When Lou, who has become the "father of the Internet," is shot by an unknown assailant, Jacob and Nick fire up the time machine again to save their friend.
--------------------
<NAME> is now a college graduate and on her way to Genovia to take up her duties as princess. Her best friend Lilly also joins her for the summer. Mia continues her 'princess lessons'- riding horses side-saddle, archery, and other royal. But her complicated life is turned upside down once again when she not only learns that she is to take the crown as queen earlier than expected...
--------------------
Under the direction of a ruthless instructor, a talented young drummer begins to pursue perfection at any cost, even his humanity.
--------------------
<NAME> (Vidya Balan) arrives in Kolkata from London to find her missing husband <NAME>. Seven months pregnant and alone in a festive city, she begins a relentless search for her husband. With nothing to rely on except fragments from her memorie[...]<jupyter_text>I don't think using overview is gonna help us<jupyter_code>train.drop(['overview'], axis=1, inplace=True)
test.drop(['overview'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Popularity<jupyter_code>train['popularity'].unique()
train['popularity'].isnull().sum()
plt.subplots(figsize=(12, 9))
# sns.catplot(x='revenue', y='popularity', data=train)
plt.scatter(x=train['popularity'], y=train['revenue'])
plt.xlabel('popularity')
plt.ylabel('revenue')
plt.show()<jupyter_output><empty_output><jupyter_text>This would be useful  Poster Path<jupyter_code>train.drop(['poster_path'], axis=1, inplace=True)
test.drop(['poster_path'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Production Companies<jupyter_code>for i in range(5):
    print(train['production_companies'][i])
    print("-------")<jupyter_output>[{'name': 'Paramount Pictures', 'id': 4}, {'name': 'United Artists', 'id': 60}, {'name': 'Metro-Goldwyn-Mayer (MGM)', 'id': 8411}]
-------
[{'name': 'Walt Disney Pictures', 'id': 2}]
-------
[{'name': 'Bold Films', 'id': 2266}, {'name': 'Blumhouse Productions', 'id': 3172}, {'name': 'Right of Way Films', 'id': 32157}]
-------
nan
-------
nan
-------<jupyter_text>Most of films have 1-2 production companies, cometimes 3-4. But there are films with 10+ companies! Let's have a look at some of them. Not sure yet what i'm gonna do with these data  I guess my best move for now to get the number of production companies, The most 50 common production companies  Number of prod companies<jupyter_code>num_prod_com = pd.DataFrame(columns=['num_production_companies'])
train = pd.concat([train, num_prod_com], axis=1)
test = pd.concat([test, num_prod_com], axis=1)
d = get_dict(train['production_companies'][0])
# This has 3 production companies
print(len(d))
d
j = 0
for i in train['production_companies']:
    d = get_dict(i)
    if len(d) != 0:
        train['num_production_companies'][j] = len(d)
    else:
        train['num_production_companies'][j] = np.NaN
    j += 1

j = 0
for i in test['production_companies']:
    d = get_dict(i)
    if len(d) != 0:
        test['num_production_companies'][j] = len(d)
    else:
        test['num_production_companies'][j] = np.NaN
    j += 1
sns.catplot(x='num_production_companies', y='revenue', data=train)<jupyter_output><empty_output><jupyter_text>Production Company<jupyter_code>list_of_companies = []
for i in train['production_companies']:
    d = get_dict(i)
    if d != {}:
        for j in range(len(d)):
            list_of_companies.append(d[j]['name'])
list_of_companies
# Top 50 production companies
top_companies_cnt = list(Counter(list_of_companies).most_common(50))
top_companies = []
for i in top_companies_cnt:
    top_companies.append(i[0])
temp = 0
for i in top_companies_cnt:
    temp += i[1]
temp
print(top_companies_cnt)
print(top_companies)
prod_company = pd.DataFrame(columns=['production_company'])
train = pd.concat([train, prod_company], axis=1)
test = pd.concat([test, prod_company], axis=1)
j = 0
cnt = 0
chk = False
for i in train['production_companies']:
    d = get_dict(i)
#     print(len(d))
    if d != {}:
        if len(d) > 1:
            for k in range(len(d)):
                company_name = d[k]['name']
                if company_name in top_companies:
                    train['production_company'][j] = company_name
                    cnt += 1
                    chk = True
                    break
            
            if chk is False:
                    train['production_company'][j] = d[0]['name']
                
        else:
            train['production_company'][j] = d[0]['name']
    else:
        train['production_company'][j] = np.NaN
    j += 1

print("THE NUMBER OF TOP COMPANIES AT TRAIN:", cnt)

j = 0
cnt = 0
chk = False
for i in test['production_companies']:
    d = get_dict(i)
#     print(len(d))
    if d != {}:
        if len(d) > 1:
            for k in range(len(d)):
                company_name = d[k]['name']
                if company_name in top_companies:
                    test['production_company'][j] = company_name
                    cnt += 1
                    chk = True
                    break
            
            if chk is False:
                    test['production_company'][j] = d[0]['name']
                
        else:
            test['production_company'][j] = d[0]['name']
    else:
        test['production_company'][j] = np.NaN
    j += 1

print("THE NUMBER OF TOP COMPANIES AT TEST:", cnt)
train['production_company'].value_counts(dropna=False)<jupyter_output><empty_output><jupyter_text>I guess using this column is just useless, we are gonna drop it<jupyter_code>train.drop(['production_companies', 'production_company'], axis=1, inplace=True)
test.drop(['production_companies', 'production_company'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Production Countries<jupyter_code>for i in range(10):
    print(train['production_countries'][i])
    print("---")
print(train['production_countries'].isnull().sum())
print(test['production_countries'].isnull().sum())<jupyter_output>55
102<jupyter_text>Number of production Countries<jupyter_code>num_of_production_countries = pd.DataFrame(columns=['num_of_production_countries'])
train = pd.concat([train, num_of_production_countries], axis=1)
test = pd.concat([test, num_of_production_countries], axis=1)
j = 0
for i in train['production_countries']:
    d = get_dict(i)
    if d != {}:
        train['num_of_production_countries'][j] = len(d)
    else:
        train['num_of_production_countries'][j] = np.NaN
    j += 1

j = 0
for i in test['production_countries']:
    d = get_dict(i)
    if d != {}:
        test['num_of_production_countries'][j] = len(d)
    else:
        test['num_of_production_countries'][j] = np.NaN
    j += 1
# Let's see how useful it is
sns.catplot(x='num_of_production_countries', y='revenue', data=train)<jupyter_output><empty_output><jupyter_text>Production Country<jupyter_code>prod_country = pd.DataFrame(columns=['production_country'])
train = pd.concat([train, prod_country], axis=1)
test = pd.concat([test, prod_country], axis=1)
j = 0
for i in train['production_countries']:
    d = get_dict(i)
    if d != {}:
        if len(d) > 1:
            countires = []
            for k in range(len(d)):
                countires.append(d[k]['name'])
            if 'United States of America' in countires:
                train['production_country'][j] = 'United States of America'
        else:
            train['production_country'][j] = d[0]['name']
    else:
        train['production_country'][j] = np.NaN
    j += 1
    
j = 0
for i in test['production_countries']:
    d = get_dict(i)
    if d != {}:
        if len(d) > 1:
            countires = []
            for k in range(len(d)):
                countires.append(d[k]['name'])
            if 'United States of America' in countires:
                test['production_country'][j] = 'United States of America'
        else:
            test['production_country'][j] = d[0]['name']
    else:
        test['production_country'][j] = np.NaN
    j += 1
sns.catplot(x='production_country', y='revenue', data=train)<jupyter_output><empty_output><jupyter_text>Some feature like is USA Production sould be useful here, let's see<jupyter_code>is_usa_production = pd.DataFrame(columns=['is_usa_production'])
train = pd.concat([train, is_usa_production], axis=1)
test = pd.concat([test, is_usa_production], axis=1)
j = 0
for i in train['production_country']:
    if i == 'United States of America':
        train['is_usa_production'][j] = 1
    elif str(i) == 'nan':
        train['is_usa_production'][j] = np.NaN
    else:
        train['is_usa_production'][j] = 0
    j += 1
    
j = 0
for i in test['production_country']:
    if i == 'United States of America':
        test['is_usa_production'][j] = 1
    elif str(i) == 'nan':
        test['is_usa_production'][j] = np.NaN
    else:
        test['is_usa_production'][j] = 0
    
    j += 1
sns.catplot(x='is_usa_production', y='revenue', data=train)
train.drop(['production_countries'], axis=1, inplace=True)
test.drop(['production_countries'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Release Date<jupyter_code>train['release_date'][:5]
print(list(train['release_date'][1]))
print(train['release_date'][1])<jupyter_output>['8', '/', '6', '/', '0', '4']
8/6/04<jupyter_text>Split the Release date to (Release Day, Release Month, Release Year)<jupyter_code>train[['release_month', 'release_day', 'release_year']] = train['release_date'].str.split('/',expand=True).replace(np.nan, -1).astype(int)
test[['release_month', 'release_day', 'release_year']] = test['release_date'].str.split('/',expand=True).replace(np.nan, -1).astype(int)
train.drop(['release_date'], axis=1, inplace=True)
test.drop(['release_date'], axis=1, inplace=True)
train.shape, test.shape
sns.catplot(x='release_month', y='revenue', data=train)
sns.catplot(x='release_day', y='revenue', data=train)
plt.subplots(figsize=(12, 9))
plt.scatter(x=train['release_year'], y=train['revenue'])<jupyter_output><empty_output><jupyter_text>RunTime<jupyter_code>print(train['runtime'].isnull().sum())
print(test['runtime'].isnull().sum())
plt.subplots(figsize=(12, 9))
plt.scatter(x=train['runtime'], y=train['revenue'])
# Zero running time means null value
print(len(train[train['runtime'] == 0]))
print(len(train[train['runtime'] == 0]))<jupyter_output>12
12<jupyter_text>Spoken Languages<jupyter_code>train['spoken_languages'][:5]<jupyter_output><empty_output><jupyter_text>Let's generate some features like (how many languages have been spoken in the movie, is English has been spoken)  Number of Spoken Languages<jupyter_code>number_of_spoken_languages = pd.DataFrame(columns=['number_of_spoken_languages'])
train = pd.concat([train, number_of_spoken_languages], axis=1)
test = pd.concat([test, number_of_spoken_languages], axis=1)
j = 0
for i in train['spoken_languages']:
    d = get_dict(i)
    if d != {}:
        train['number_of_spoken_languages'][j] = len(d)
    else:
        train['number_of_spoken_languages'][j] = np.NaN
    j += 1
    
j = 0
for i in test['spoken_languages']:
    d = get_dict(i)
    if d != {}:
        test['number_of_spoken_languages'][j] = len(d)
    else:
        test['number_of_spoken_languages'][j] = np.NaN
    j += 1
sns.catplot(x='number_of_spoken_languages', y='revenue', data=train)<jupyter_output><empty_output><jupyter_text>Clear pattern in the graph above  Is English a Spoken Language<jupyter_code>is_en_spoken = pd.DataFrame(columns=['is_en_spoken'])
train = pd.concat([train, is_en_spoken], axis=1)
test = pd.concat([test, is_en_spoken], axis=1)
j = 0
for i in train['spoken_languages']:
    d = get_dict(i)
    if d != {}:
        langs = []
        for k in range(len(d)):
            lang = d[k]['name']
            langs.append(lang)
        if 'English' in langs:
            train['is_en_spoken'][j] = 1
        else:
            train['is_en_spoken'][j] = 0
    else:
        train['is_en_spoken'][j] = np.NaN
    j += 1
        
j = 0
for i in test['spoken_languages']:
    d = get_dict(i)
    if d != {}:
        langs = []
        for k in range(len(d)):
            lang = d[k]['name']
            langs.append(lang)
        if 'English' in langs:
            test['is_en_spoken'][j] = 1
        else:
            test['is_en_spoken'][j] = 0
    else:
        test['is_en_spoken'][j] = np.NaN
    j += 1
sns.catplot(x='is_en_spoken', y='revenue', data=train)<jupyter_output><empty_output><jupyter_text>This feature is really useful  Spoken Language<jupyter_code>spoken_language = pd.DataFrame(columns=['spoken_language'])
train = pd.concat([train, spoken_language], axis=1)
test = pd.concat([test, spoken_language], axis=1)
j = 0
for i in train['spoken_languages']:
    d = get_dict(i)
    if d != {}:
        langs = []
        for k in range(len(d)):
            lang = d[k]['name']
            langs.append(lang)
        if 'English' in langs:
            train['spoken_language'][j] = 'English'
        else:
            train['spoken_language'][j] = langs[0]
    else:
        train['spoken_language'][j] = np.NaN
    j += 1
    
j = 0
for i in test['spoken_languages']:
    d = get_dict(i)
    if d != {}:
        langs = []
        for k in range(len(d)):
            lang = d[k]['name']
            langs.append(lang)
        if 'English' in langs:
            test['spoken_language'][j] = 'English'
        else:
            test['spoken_language'][j] = langs[0]
    else:
        test['spoken_language'][j] = np.NaN
    j += 1
print(len(train['spoken_language'].value_counts()))
print(len(test['spoken_language'].value_counts()))
sns.catplot(x='spoken_language', y='revenue', data=train)
train.drop(['spoken_languages'], axis=1, inplace=True)
test.drop(['spoken_languages'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Status<jupyter_code>print(train['status'].value_counts(dropna=False))
print(test['status'].value_counts(dropna=False))<jupyter_output>Released    2996
Rumored        4
Name: status, dtype: int64
Released           4389
Post Production       5
Rumored               2
NaN                   2
Name: status, dtype: int64<jupyter_text>Is released<jupyter_code>is_released = pd.DataFrame(columns=['is_released'])
train = pd.concat([train, is_released], axis=1)
test = pd.concat([test, is_released], axis=1)
j = 0
for i in train['status']:
    if i == 'Released':
        train['is_released'][j] = 1
    else:
        train['is_released'][j] = 0
    j += 1
    
j = 0
for i in test['status']:
    if i == 'Released':
        test['is_released'][j] = 1
    else:
        test['is_released'][j] = 0
    j += 1
print(train['is_released'].value_counts(dropna=False))
print(test['is_released'].value_counts(dropna=False))
sns.catplot(x='is_released', y='revenue', data=train)
train.drop(['status'], axis=1, inplace=True)
test.drop(['status'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Tagline<jupyter_code>train['tagline'][:10]
train.drop(['tagline'], axis=1, inplace=True)
test.drop(['tagline'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>We Will not delete Title now, We are gonne need it later to fill sum null values manually  Keywords<jupyter_code>for i in range(10): 
    print(train['Keywords'][i])
    print("-------")
keyword = pd.DataFrame(columns=['keyword'])
train = pd.concat([train, keyword], axis=1)
test = pd.concat([test, keyword], axis=1)
j = 0
for i in train['Keywords']:
    d = get_dict(i)
    if d != {}:
        train['keyword'][j] = d[0]['name']
    else:
        train['keyword'][j] = np.NaN
    j += 1
    
j = 0
for i in test['Keywords']:
    d = get_dict(i)
    if d != {}:
        test['keyword'][j] = d[0]['name']
    else:
        test['keyword'][j] = np.NaN
    j += 1
train['keyword'].value_counts(dropna=False)
# We can not handle this much of keywords
train.drop(['Keywords', 'keyword'], axis=1, inplace=True)
test.drop(['Keywords', 'keyword'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Cast<jupyter_code>for i in range(1):
    print(train['cast'][i])
    print("----------")<jupyter_output>[{'cast_id': 4, 'character': 'Lou', 'credit_id': '52fe4ee7c3a36847f82afae7', 'gender': 2, 'id': 52997, 'name': '<NAME>', 'order': 0, 'profile_path': '/k2zJL0V1nEZuFT08xUdOd3ucfXz.jpg'}, {'cast_id': 5, 'character': 'Nick', 'credit_id': '52fe4ee7c3a36847f82afaeb', 'gender': 2, 'id': 64342, 'name': '<NAME>', 'order': 1, 'profile_path': '/tVaRMkJXOEVhYxtnnFuhqW0Rjzz.jpg'}, {'cast_id': 6, 'character': 'Jacob', 'credit_id': '52fe4ee7c3a36847f82afaef', 'gender': 2, 'id': 54729, 'name': '<NAME>', 'order': 2, 'profile_path': '/oNzK0umwm5Wn0wyEbOy6TVJCSBn.jpg'}, {'cast_id': 7, 'character': '<NAME>.', 'credit_id': '52fe4ee7c3a36847f82afaf3', 'gender': 2, 'id': 36801, 'name': '<NAME>', 'order': 3, 'profile_path': '/5gb65xz8bzd42yjMAl4zwo4cvKw.jpg'}, {'cast_id': 8, 'character': 'Hot Tub Repairman', 'credit_id': '52fe4ee7c3a36847f82afaf7', 'gender': 2, 'id': 54812, 'name': '<NAME>', 'order': 4, 'profile_path': '/svjpyYtPwtjvRxX9IZnOmOkhDOt.jpg'}, {'cast_id': 9, 'character'[...]<jupyter_text>Size of crew (number of actors)<jupyter_code>size_of_cast = pd.DataFrame(columns=['size_of_cast'])
train = pd.concat([train, size_of_cast], axis=1)
test = pd.concat([test, size_of_cast], axis=1)
j = 0
for i in train['cast']:
    d = get_dict(i)
    if d != {}:
        train['size_of_cast'][j] = len(d)
    else:
        train['size_of_cast'][j] = np.NaN
    j += 1
    
j = 0
for i in test['cast']:
    d = get_dict(i)
    if d != {}:
        test['size_of_cast'][j] = len(d)
    else:
        test['size_of_cast'][j] = np.NaN
    j += 1
plt.subplots(figsize=(12, 9))
plt.scatter(x=train['size_of_cast'], y=train['revenue'])
train.drop(['cast'], axis=1, inplace=True)
test.drop(['cast'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Title, Crew<jupyter_code>train.drop(['id', 'crew', 'title'], axis=1, inplace=True)
test.drop(['id', 'crew', 'title'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Null Values  Budget<jupyter_code>print(train['budget'].isnull().sum())
print(test['budget'].isnull().sum())
print(len(train[train['budget'] == 0]))
print(len(test[test['budget'] == 0]))
plt.scatter(x=train['budget'], y=train['revenue'])
ntrain = train.shape[0]
ntest = test.shape[0]
all_data = pd.concat([train, test], axis=0)
train.shape, test.shape, all_data.shape
all_data['budget'].dtype
all_data['budget'] = all_data['budget'].replace(0, all_data['budget'].mean())<jupyter_output><empty_output><jupyter_text>Original Language<jupyter_code>print(all_data['original_language'].isnull().sum())
print(len(all_data[all_data['original_language'] == 0]))<jupyter_output>0
0<jupyter_text>Popularity<jupyter_code>print(all_data['popularity'].isnull().sum())
print(len(all_data[all_data['popularity'] == 0]))<jupyter_output>0
0<jupyter_text>Runtime<jupyter_code>print(all_data['runtime'].isnull().sum())
print(len(all_data[all_data['runtime'] == 0]))
all_data['runtime'].mean().round()
all_data['runtime'] = all_data['runtime'].replace(0, all_data['runtime'].mean().round())
all_data['runtime'] = all_data['runtime'].fillna(all_data['runtime'].mean().round())<jupyter_output><empty_output><jupyter_text>Genre Name<jupyter_code>print(all_data['genre_name'].isnull().sum())
print(len(all_data[all_data['genre_name'] == 0]))
print(all_data['genre_name'].value_counts())
all_data['genre_name'] = all_data['genre_name'].fillna(all_data['genre_name'].mode()[0])<jupyter_output><empty_output><jupyter_text>has homepage<jupyter_code>print(all_data['has_homepage'].isnull().sum())<jupyter_output>0<jupyter_text>Is En Original Language<jupyter_code>print(all_data['is_en_original_language'].isnull().sum())<jupyter_output>0<jupyter_text>Number of production companies<jupyter_code>print(all_data['num_production_companies'].isnull().sum())
print(len(all_data[all_data['num_production_companies'] == 0]))
all_data['num_production_companies'] = all_data['num_production_companies'].fillna(all_data['num_production_companies'].mean().round())<jupyter_output><empty_output><jupyter_text>Number of production countries<jupyter_code>print(all_data['num_of_production_countries'].isnull().sum())
print(len(all_data[all_data['num_of_production_countries'] == 0]))
print(all_data['num_of_production_countries'].mode()[0])
print(all_data['num_of_production_countries'].mean().round())
all_data['num_of_production_countries']= all_data['num_of_production_countries'].fillna(all_data['num_of_production_countries'].mode()[0])<jupyter_output><empty_output><jupyter_text>Production Country, Is USA production<jupyter_code>print(all_data['production_country'].isnull().sum())
print(len(all_data[all_data['production_country'] == 0]))
print(all_data['is_usa_production'].isnull().sum())
print(all_data['production_country'].value_counts())
all_data['production_country'] = all_data['production_country'].fillna(all_data['production_country'].mode()[0])
all_data['is_usa_production'] = all_data['is_usa_production'].fillna(1)<jupyter_output><empty_output><jupyter_text>Release day, month, year<jupyter_code>print(all_data['release_day'].isnull().sum())
print(len(all_data[all_data['release_day'] == 0]))
print(all_data['release_month'].isnull().sum())
print(len(all_data[all_data['release_month'] == 0]))
print(all_data['release_year'].isnull().sum())
print(len(all_data[all_data['release_year'] == 0]))<jupyter_output>0
0
0
0
0
155<jupyter_text>Number of spoken languages<jupyter_code>print(all_data['number_of_spoken_languages'].isnull().sum())
print(len(all_data[all_data['number_of_spoken_languages'] == 0]))
print(all_data['number_of_spoken_languages'].value_counts())
all_data['number_of_spoken_languages'].mean().round()
all_data['number_of_spoken_languages'] = all_data['number_of_spoken_languages'].fillna(1)<jupyter_output><empty_output><jupyter_text>Is En spoken<jupyter_code>print(all_data['is_en_spoken'].isnull().sum())
# print(len(all_data[all_data['is_en_spoken'] == 0]))
all_data['is_en_spoken'] = all_data['is_en_spoken'].fillna(1)<jupyter_output><empty_output><jupyter_text>Spoken Language<jupyter_code>print(all_data['spoken_language'].isnull().sum())
print(len(all_data[all_data['spoken_language'] == 0]))
all_data['spoken_language'] = all_data['spoken_language'].fillna('English')<jupyter_output><empty_output><jupyter_text>Is released<jupyter_code>print(all_data['is_released'].isnull().sum())
# print(all_data[all_data['spoken_language'] == 0])<jupyter_output>0<jupyter_text>Size of cast<jupyter_code>print(all_data['size_of_cast'].isnull().sum())
print(len(all_data[all_data['size_of_cast'] == 0]))
print(all_data['size_of_cast'].mean().round())
print(all_data['size_of_cast'].mode()[0])
all_data['size_of_cast'].value_counts(dropna=False)
all_data['size_of_cast'] = all_data['size_of_cast'].fillna(all_data['size_of_cast'].mean().round())
all_data['size_of_cast'] = all_data['size_of_cast'].replace(0, all_data['size_of_cast'].mode()[0])
# Split the data
train = all_data[:ntrain]
test = all_data[ntrain:]
test.drop(['revenue'], axis=1, inplace=True)
train.shape, test.shape<jupyter_output><empty_output><jupyter_text>Skewness<jupyter_code>train.skew()
sns.distplot(train['revenue'], fit=norm)
sns.distplot(train['popularity'], fit=norm)
train['revenue'] = np.log1p(train['revenue'])
train.skew()
sns.distplot(train['revenue'], fit=norm)<jupyter_output><empty_output><jupyter_text>Encoding Categorical Data<jupyter_code>target = train['revenue']
train.drop(['revenue'], axis=1, inplace=True)
ntrain = train.shape[0]
ntest = test.shape[0]
all_data = pd.concat([train, test], axis=0)
le = LabelEncoder()
cat_features = all_data.select_dtypes(include='object')
for i in cat_features.columns:
    all_data[i] = le.fit_transform(all_data[i])
    
train = all_data[:ntrain]
test = all_data[ntrain:]
train.shape, test.shape
cat_features.columns<jupyter_output><empty_output><jupyter_text>Scaling<jupyter_code>sc = StandardScaler()
train = sc.fit_transform(train)
test = sc.transform(test)<jupyter_output><empty_output><jupyter_text>Modelling<jupyter_code># XGBOOST
model_xgb = xgb.XGBRegressor(colsample_bytree=0.4603, gamma=0.0468, 
                             learning_rate=0.1, max_depth=10, n_estimators=200,
                             reg_alpha=0.4640, reg_lambda=0.8571)
model_xgb.fit(train, target)
y_pred_train_xgb = model_xgb.predict(train)
print(np.sqrt(mean_squared_log_error(target, y_pred_train_xgb)))
np.expm1(y_pred_train_xgb)
np.expm1(target)
y_pred = model_xgb.predict(test)
sub = pd.DataFrame()
sub['id'] = test_id
sub['revenue'] = np.expm1(y_pred)
sub.to_csv('sub7.csv', index=False)
# LightGBM<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17572957521770924, 'fraction_numerical': 0.018081499700857543, 'mean_word_length': 5.065927419354839, 'pattern_counts': {'":': 0, '<': 228, '<?xml version=': 0, '>': 232, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '58', 'dir': 'jupyter-structured-clean-dedup', 'id': '464929', 'license': 'MIT', 'n_tokens_mistral': 11308, 'n_tokens_neox': 10855, 'n_words': 2006, 'path': 'classical-ml-competitions/regression-competitions/TMDB-the-box-office/.ipynb_checkpoints/BoxOfficePrediction-checkpoint.ipynb', 'repo_name': 'KickItLikeShika/cracking-kaggle-competitions', 'size': '30134'}