<jupyter_start><jupyter_text>Casting string to float<jupyter_code>df['bachelorsAvg']=df['bachelorsAvg'].apply(pd.to_numeric, errors='coerce')
df['masterAvg']=df['masterAvg'].apply(pd.to_numeric, errors='coerce')<jupyter_output><empty_output><jupyter_text>Doc about pandas.to_numerichttp://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_numeric.htmlerrors =‘coerce’,  invalid parsing will be set as NaN  Overall view of situation  Bachelors<jupyter_code>df['bachelorsAvg'].value_counts().sort_index().plot(xlim=(0,20))
plt.show()
myBins=[df['bachelorsAvg'].min(),0,4,5,10,20,df['bachelorsAvg'].max()]
avg = pd.cut(df['bachelorsAvg'], myBins) 
avg.value_counts()
df[df['bachelorsAvg']<=6].country.value_counts()<jupyter_output><empty_output><jupyter_text>Masters<jupyter_code>df['masterAvg'].value_counts().sort_index().plot(xlim=(0,20))
plt.show()
df['masterAvg'].describe()
myBins2=[0,4,6,10,20,df['masterAvg'].max()]
avg = pd.cut(df['masterAvg'], myBins2)
avg.value_counts()<jupyter_output><empty_output><jupyter_text>Converting GermanyGPA to Iranian Gpa based on https://www.visamondial.com/تبدیل-معدل-یا-نمره-به-سیستم-آلمانی/<jupyter_code>df[((df.masterAvg<=6)|(df.bachelorsAvg<=6)) & (df.country==u"آلمان")].count()[0]
def germanGPA_to_iranGPA(g):
    return 10*(g-1/3)-20
df[(df.masterAvg<=6) & (df.country==u"آلمان")].masterAvg=germanGPA_to_iranGPA(df[(df.masterAvg<=6) & (df.country==u"آلمان")].masterAvg)
df[(df.bachelorsAvg<=6) & (df.country==u"آلمان")].bachelorsAvg=germanGPA_to_iranGPA(df[(df.bachelorsAvg<=6) & (df.country==u"آلمان")].bachelorsAvg)<jupyter_output><empty_output><jupyter_text>Converting to GPA 0-4  About the GPA CalculatorBased of http://www.foreigncredits.com/resources/gpa-calculator/iran<jupyter_code>#methods
import math
def round_to(number,roundto):
    return math.floor(number / roundto) * roundto
def toGPA(iscore): #to GPA [0,4]
    return round_to((float)(iscore-10)/2,0.5)
x=df['bachelorsAvg'].copy()
for i in x.index:
    if (x[i]>10) & (x[i]<=20):x[i]=toGPA(x[i])
df['bachelorsAvg']=x
pd.cut(x, myBins).value_counts()
x=df['masterAvg'].copy()
for i in x.index:
    if (x[i]>10) & (x[i]<=20):x[i]=toGPA(x[i])
df['masterAvg']=x
pd.cut(df['masterAvg'], myBins2).value_counts()<jupyter_output><empty_output><jupyter_text>Outliers to NAN  bachelorsAvg<jupyter_code>(df[df['bachelorsAvg']==0]['bachelorsAvg']).count()
df.ix[df.bachelorsAvg>5, 'bachelorsAvg']<jupyter_output><empty_output><jupyter_text>** These data don't make sense so we are going to fill them with Null value**<jupyter_code>df.ix[df.bachelorsAvg==0, 'bachelorsAvg'] = np.nan
df.ix[df.bachelorsAvg>5, 'bachelorsAvg'] = np.nan
pd.cut(df['bachelorsAvg'], myBins).value_counts()
df.ix[df.bachelorsAvg>4, 'bachelorsAvg'].value_counts()
# GPA scale 0-5
# in this scale 5==4 in 0-4 scale
# and 4.5==3.6 in 0-4 scale
df.ix[df.bachelorsAvg==5, 'bachelorsAvg'] = 4
df.ix[df.bachelorsAvg>=4.5, 'bachelorsAvg'] = 3.6
pd.cut(df['bachelorsAvg'], myBins).value_counts()<jupyter_output><empty_output><jupyter_text>masterAvg<jupyter_code>pd.cut(df['masterAvg'], myBins2).value_counts()
df[df['masterAvg']>20]['masterAvg']
df.ix[[3384,3262,1332],'masterAvg']=np.nan #remove out of range records<jupyter_output><empty_output><jupyter_text>Convert 100 scale gpa to 4.0based of : http://www.collegeboard.com/html/academicTracker-howtoconvert.html<jupyter_code>def togpa(iscore,binss):
    return pd.cut(iscore, bins=binss,labels=np.arange(1,5))
avg=df[(df['masterAvg']>50) & (df['masterAvg']<=100)]['masterAvg']
x=togpa(avg,[60,72,79,89,100])
df.ix[(df['masterAvg']>50) & (df['masterAvg']<=100),'masterAvg']=x
pd.concat([x,avg],axis=1).head(7)
df.ix[df.masterAvg>5, 'masterAvg'] = np.nan
df.ix[df.masterAvg==5, 'masterAvg'] = 4
df[df['masterAvg']>4]['masterAvg'].value_counts()
df.ix[df.masterAvg>4, 'masterAvg'] = 3.5
pd.cut(df['masterAvg'], myBins2).value_counts()<jupyter_output><empty_output><jupyter_text>PLOT<jupyter_code>df.bachelorsAvg.value_counts().sort_index().plot(xlim=(0,4))
df.masterAvg.value_counts().sort_index().plot(xlim=(0,4),c='r')
plt.legend(loc = 1)
plt.legend(['Bachelors GPA', 'Masters GPA'], loc = 2)
plt.xlabel('GPA')
plt.ylabel('Count')
plt.show()<jupyter_output><empty_output><jupyter_text>Saving data<jupyter_code>df.rename(columns={'masterAvg': 'gpaMasters','bachelorsAvg':'gpaBachelors'}, inplace=True)
df.to_json('Data_FixedGPA.json',date_format='utf8')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.2036281179138322, 'fraction_numerical': 0.03106575963718821, 'mean_word_length': 9.502380952380953, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 57, 'https://': 1, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '592014', 'license': 'MIT', 'n_tokens_mistral': 2046, 'n_tokens_neox': 1968, 'n_words': 205, 'path': '03_DataPreprocessing/07_GPA/GPA.ipynb', 'repo_name': 'yazdipour/DM17', 'size': '4410'}