<jupyter_start><jupyter_text>By reading data there is some observations realised:There is null valuesIt is better to partition data according to age range to handle one of the questionsThere is some columns that has no use (eg. country-year, country code, Unnamed: 5)<jupyter_code>#Assesing data & building intuitions & Cleansing(Step 2 & 3)

# Getting null values
df.isna().sum()
df['country'].isna().tail(20)

df[df['year']==2016]<jupyter_output><empty_output><jupyter_text>Note: before dealing with NaN values. It is realised that the first dataset with suicidal info. has years limited by year 2016 but the second dataset disturbanced that as it is shown on data.info() that the only non NAN column is that on which join was made which is the country.so depending on that year label & all of the related features with it will be the measure as our main dataset is the one related to suicide. so all of the labels with same no. of NAN values with year label will be deleted.<jupyter_code>df.isna().sum()
#We can make drop NANs for only one label of these and by the default these all labels will not be null but we will do all of
#it
df.dropna(subset=['year', 'sex', 'age', 'suicides_no', 'population', 'suicides/100k pop', 'country-year', 
                  'gdp_per_capita ($)', 'generation','IncomeGroup'],inplace=True)

df.info()
df['HDI for year'].value_counts()

from sklearn.impute import KNNImputer
imputer = KNNImputer()
df['HDI for year']=imputer.fit_transform(df[['HDI for year']])
df['HDI for year'].isna().sum()

df.isna().sum()

df.drop(columns=['Country Code',
                 'SpecialNotes','Unnamed: 5',' gdp_for_year ($) '],inplace=True)
df.isna().sum()
df.info()

#Dealing with datatypes
df.dtypes

df = df.astype({'year':'int64','suicides_no':'int64','population':'int64','suicides/100k pop':'int64'})
df.info()
# There are no duplicate values
# Dealing with duplicates
df.duplicated().sum()
df['generation'].unique()
df['IncomeGroup'].unique()
df['Region'].unique()
df['country'].unique()
df['age'].unique()<jupyter_output><empty_output><jupyter_text>unique categorical values that is important is found that there are:101 countries7 regions3 income groups<jupyter_code>df.head()
df[df['HDI for year']<0.550]
#GI generation: Born(1900-1927)
#Silent: Born(1928-1945)
#Boomers: Born(1946-1964)
#Generation x: Born (1965-1980)
#Millenials: Born(1981-1996)
#Generation z: Born(1997-)
# sns.jointplot(x='suicides_no',y='gdp_per_capita ($)',data=df)
# sns.pairplot(df,hue='IncomeGroup',vars=['suicides_no','gdp_per_capita ($)'],height=5, aspect=1.3)<jupyter_output><empty_output><jupyter_text>the upper middle income group(avg. group) has the highest suicide no.<jupyter_code># #count plot of income group
# sns.countplot(x='IncomeGroup',data=df)<jupyter_output><empty_output><jupyter_text>It is shown that the high income group is with the highest count so the plots should be biased to it<jupyter_code># sns.jointplot(x='gdp_per_capita ($)',y='suicides_no',data=df)
# sns.barplot(x='IncomeGroup',y='suicides_no',data=df)
# sns.stripplot(x='IncomeGroup',y='suicides_no',data=df)<jupyter_output><empty_output><jupyter_text>Being biased to upper middle and high income group is obvious above.SoThe highest number of suicides relative to its count is the low middle income groupupper middle income group shows a relatively high suicide real numbers(Not ratio)As the gdp increases, the suicide number decreases and that is acceptable as income group depends on gdpSuicide number is the most at hight HDI (0.700-0.799) (upper-middel income group) and this agrees with point two as it shows real values.<jupyter_code># #age range and suicide_numbers
# sns.stripplot(x='age',y='suicides_no',data=df)
# sns.barplot(x='age',y='suicides_no',data=df)<jupyter_output><empty_output><jupyter_text>It is seen that the suicide number increases in the age range(35-54) Which is at mostly the age of work and it is low in age ranges beyond 75 and lower than 14<jupyter_code># #The correlation function shows that there is a high correlation between population and suicide no.
# sns.jointplot(x='population',y='suicides_no',data=df,kind='reg')
# sns.lmplot(x='population',y='suicides_no',data=df,hue='sex')
# sns.barplot(y='suicides_no',x='sex',data=df)

# sns.set(rc={'figure.figsize': [15, 15]}, font_scale=1.2)
# sns.barplot(y='country',x='population',hue='sex',data=df)
# sns.barplot(x='sex',y='population',data=df)
pivot = pd.pivot_table(values='suicides_no', index='country', columns='sex',data=df)

print(pivot)<jupyter_output>sex                        female         male
country                                       
Albania                  5.250000     9.674242
Antigua and Barbuda      0.006173     0.061728
Argentina               98.526882   343.510753
Armenia                  3.322148     9.463087
Aruba                    0.202381     1.000000
...                           ...          ...
United Arab Emirates     2.722222    14.555556
United Kingdom         175.736559   559.774194
United States         1149.446237  4409.763441
Uruguay                 16.785714    61.416667
Uzbekistan              70.151515   193.507576

[93 rows x 2 columns]<jupyter_text>Drawing conclusionsAs we has gone through the analysis process we have dealt with data in order to answer the questions mentioned above. Of course there is more questions to be asked and can be got from data but that is good for now as our focus on data analysis is not as important as Showing the ability to draw conclusions using AI (Which is ML in our file).So First I will get to the conclusions that is got out of this analysis that answers the first two questions:What is the age range on which suicidal actions are the most? and for this question the answer was: It is seen that the suicide number increases in the age range(35-54) Which is at mostly the age of work and it is low in age ranges beyond 75 and lower than 14.What is the most correlated feature with the increase of suicidal actions? And the answer of these question was that: The correlation function shows that there is a high correlation between population and suicide number so we had gone into regression (Implot) to see the correlation visualized and to know its type and it was found that the regression line on the scatter plot shows that there is a positive correlation between suicide number and population. And by looking to the population two different genders it was found that: The suicides among males is more than that among females. But as sex is dependant on the population of male and female among the community som we need to make more plots to make sure of this result so we had gone into a barplot that shows the gender values relative to the population and it was found that: The sex label values are nearly equal in population so this indicates that the previous result that males commit more suicides than females is true.There is some historical features that was neglected as it has no purpose in our work. As an example of that the the political era for each age range in a each country, and the years that had the most suicidal actions(Which could be analyzed more to find global actions and national actions in this years that caused many suicidal actions). As an example of my talk is the huge increase of suicidal actions in russian federations at the years of the cold-war.Going into the third question: What is the expected number of suicides given other labels that is related to it?The answer of this question would be given through a ML model that needs from us to go further into data_preprocessing to get the best of these features and deal with problems that the ML model may face (E.g: Categorical data values) to get the most precise model<jupyter_code>!pip3 install geopy
from  geopy.geocoders import Nominatim
geolocator = Nominatim(user_agent='Essam')
#Getting two lists for lat and lang
lat=[]
lang=[]
counter=0
for i in df['country'].value_counts():
    country =df['country'].value_counts().index[counter]
    loc = geolocator.geocode(country)
    lat.append(loc.latitude)
    lang.append(loc.longitude)
    counter+=1

data={'country':df['country'].value_counts().index, 'lat':lat, 'lang': lang}
lat_lang=pd.DataFrame(data)
lat_lang
df=pd.merge(df,lat_lang,how='outer',on='country')

df.head()
df.info()
# sns.pairplot(df, hue='sex',vars=['lat','lang','suicides_no'])
#Checking weird data values or types
df.dtypes
#Is there any weird data types? (Negative or zero values)
list_5=['year','suicides_no','population','suicides/100k pop','HDI for year','gdp_per_capita ($)']
counter_0=0
counter_1=0
for i in list_5:
    for j in df:
        #As year has an upper limit& lower limit so it will be dealt alone, unlike other features which have only lower limit.
        if (counter_0 == 0):
            if (df['year'][counter_1]<=0) | (df['year'][counter_1]>2020):
                print(counter_1,'\n')
        else:
            if df[i][counter_1]<=0:
                print(counter_1,'\n') 
        counter_1+=1
    counter_0+=1
i=0
list_6=[22,23,45,46,47,57,58,59]
while(i<8):
    print(df.iloc[list_6[i]],'\n')
    i+=1
df.drop(columns=['suicides/100k pop','country-year'],inplace=True)
df['age'].value_counts().index

dict_age={'5-14 years':1,'15-24 years':2,'25-34 years':3,'35-54 years':4,'55-74 years':5,'75+ years':6}
#Mapping dict_age to age label
df['age']=df['age'].map(dict_age)
df['age']

from sklearn.preprocessing import LabelEncoder
le=LabelEncoder()
le1=LabelEncoder()

df['IncomeGroup']=le.fit_transform(list(df['IncomeGroup']))
df['IncomeGroup']
le.inverse_transform(df['IncomeGroup'])
df['generation']=le1.fit_transform(list(df['generation']))

le.inverse_transform(df['IncomeGroup'])

#list for the nominal categorical features
pd.get_dummies(df,columns=['sex','country','Region'],drop_first=True)

df=pd.get_dummies(df,columns=['sex'],drop_first=True)<jupyter_output><empty_output><jupyter_text>since region and country features may cause alot of features and dummy data existance in model training if it was encode to get_dummies so I will be using binary encoding as it will be more dimensionaly reduced.<jupyter_code>!pip3 install category_encoders
from category_encoders import BinaryEncoder
encoder1= BinaryEncoder(cols=['Region','country'])
df=encoder1.fit_transform(df)
df
encoder1.inverse_transform(df)<jupyter_output><empty_output><jupyter_text>Splitting data into train/test split<jupyter_code>df.info()
#Knowing & dealing with outliers
from scipy import stats
df[(np.abs(stats.zscore(df))<3).all(axis=1)]
# country	year	age	suicides_no	population	HDI for year	gdp_per_capita ($)	generation	Region	IncomeGroup	

# x=df[['country','region', 'generation', 'sex_male' ]]

x=df.drop(columns=['suicides_no', 'population', 'gdp_per_capita ($)', 'HDI for year', 'lat', 'lang', 'Region_0', 'Region_1', 'Region_2', 'Region_3'])
y=df['suicides_no']
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test= train_test_split(x,y,test_size=0.2,random_state=0)

#Checking if there is imbalanced classes in the categorical features that may hold like that.
list_7=['generation','sex_male','IncomeGroup','age','Region_0','Region_1','Region_2','Region_3']
for i in list_7:
    sns.catplot(x=i,data=df,kind='count')<jupyter_output><empty_output><jupyter_text>We needn’t deal with imbalanced data as for ['generation','sex_male','IncomeGroup','age'] the variation between values is not huge to deal with it. And for ['Region_0','Region_1','Region_2','Region_3'] This is natural that 0&1s values go like that as they are a result for one hot encoding 4 one feature so on sampling the ratios will remain the same as the other regions will be sampled to.<jupyter_code>from sklearn.preprocessing import RobustScaler
rb=RobustScaler()

training_data=pd.concat([x_train,y_train],axis=1)
column_names=training_data.columns
training_data=rb.fit_transform(training_data)
training_data=pd.DataFrame(training_data,columns=column_names)
training_data<jupyter_output><empty_output><jupyter_text>3. Feature SelectionSince feature selection is done for all x columns so we need to merge x_train & x_test before going in feature selection. Ans since X from above represents x_train & x_test So this method will be done on X before applying train_test_split once again.<jupyter_code>from sklearn.feature_selection import f_regression,SelectPercentile
selector = SelectPercentile (percentile=80, score_func=f_regression)

all_feat=x.columns
selector.fit(x,y)

selected_features_idx = selector.get_support(indices=True)

selected_features = all_feat[selected_features_idx]

selected_features

x=x[selected_features]

x_train, x_test, y_train, y_test= train_test_split(x,y,test_size=0.2,random_state=27)<jupyter_output><empty_output><jupyter_text>After finishing data preprocessing step, we shall head to choosing a ML algorithm to train data on.Since we want to get a predictive model for 'Y' with 'x' labeled data so we will deal with Supervised ML & since we want to predict suicides No. so it will be a regression problem.Knowing that we are dealing with many features not only one so we will apply Multiple linear regression.<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
model=RandomForestRegressor(random_state=27)
model.fit(x_train,y_train)
y_predict=model.predict(x_test)
y_predict
model.score(x_test,y_test)

import joblib
joblib.dump(model,'model.pkl')
model = joblib.load('model.pkl')
x_test.iloc[6]
model.predict([list(x_test.iloc[155].values)])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.104449027695934, 'fraction_numerical': 0.029316440777843254, 'mean_word_length': 5.344392523364486, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '460235', 'license': 'MIT', 'n_tokens_mistral': 4493, 'n_tokens_neox': 4164, 'n_words': 1437, 'path': 'project_3/main.ipynb', 'repo_name': 'royjoyita/Project_3', 'size': '13576'}