<jupyter_start><jupyter_text>Analysing H1B Visa data Trends H1B visa is a nonimmigrant visa issued to gradaute level applicants allowing them to work in the United States. The employer sponsors the H1B visa for workers with theoretical or technical expertise in specialized fields such as in IT, finance, accounting etc. An interesting fact about immigrant workers is that about 52 percent of new Silicon valley companies were founded by such workers during 1995 and 2005. Some famous CEOs like <NAME> (Pepsico), <NAME> (Tesla), <NAME> (Google),<NAME> (Microsoft) once arrived to the US on a H1B visa.\**Motivation**: Our team consists of five international graduate students, in the future we will be applying for H1B visa. The visa application process seems very long, complicated and uncertain. So we decided to understand this process and use Machine learning algorithms to predict the acceptance rate and trends of H1B visa.<jupyter_code>from google.colab import drive
drive.mount('/content/gdrive')
!pip install autocorrect
import string,nltk,warnings,sklearn
import pandas as pd
import numpy as np
import read_files as rd
import visualizations as vz
import plotly.graph_objects as go
import plotly.tools as tls
import cleaning as cw
import baseline as blc
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
from statistics import mean
from autocorrect import Speller 
from imblearn.over_sampling import ADASYN
from imblearn.under_sampling import EditedNearestNeighbours
from collections import Counter
from sklearn import metrics
from sklearn import model_selection
from sklearn.preprocessing import OneHotEncoder #ONE HOT ENCODING
from sklearn.ensemble import RandomForestClassifier #Build model - Random Forest Classifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import gensim.downloader as api
from sklearn.metrics import precision_recall_curve
nltk.download('wordnet')
nltk.download('words')
from sklearn.tree import DecisionTreeClassifier<jupyter_output><empty_output><jupyter_text>Data The data used in the project has been collected from the Office of Foreign Labor Certification (OFLC).The Data provides insight into each petition with information such as the Job title, Wage, Employer, Worksite location etc. To get the dataset click on the above link-> click on Disclosure data -> scroll down to H1B data.\<jupyter_code>link2015='/content/gdrive/My Drive/H1B_project/H-1B_Disclosure_Data_FY15_Q4.csv'
link2016='/content/gdrive/My Drive/H1B_project/H-1B_Disclosure_Data_FY16.csv'
link2017='/content/gdrive/My Drive/H1B_project/H-1B_Disclosure_Data_FY17.csv'
link2018='/content/gdrive/My Drive/H1B_project/H-1B_Disclosure_Data_FY2018_EOY.csv'
link2019='/content/gdrive/My Drive/H1B_project/H-1B_Disclosure_Data_FY2019.csv'
#read csv fuction to read and concatenate all the datasets
cleaned=rd.read_csv(pd,link2015,link2016,link2017,link2018,link2019)
original=cleaned.copy()
cleaned.shape<jupyter_output><empty_output><jupyter_text>Exploratory Data AnalysisBefore we begin working on our data we need to understand the traits of our data which we accomplish using EDA. We see that we have about 260 columns , not all 260 columns have essential information that contributes to our analysis. Hence we pick out the columns such as case status( Accepted/ Denied) ,Employer, Job title etc.We can look at all the colums and the types of object in each column using cleaned.info(). We also use cleaned['column_name'].value_counts() to find the classes in the column along with the count. For feature engineering we are converting quantitative data to categorical.The steps for data cleaning start from EDA till the baseline classifier, follow the comments to understand the cleaning steps.  Granularity:Each row in the dataframe represents an Electronically filles application that was filled by the employer sponsoring the H1b Visa and the application is processed by OFLC and data columns such as DECISION_DATE, CASE_STATUS are added. We can see the dataframe for this H1B data below.<jupyter_code>cleaned.dropna(inplace=True)# Drop all null values from the dataframe
cleaned.head(3)
cleaned['VISA_CLASS'].value_counts()
# the dataset has categories for Visa which are not of use , we require only H1B visa type , hence we drop all records with other visa types
cleaned.drop(labels=cleaned.loc[cleaned['VISA_CLASS']!='H-1B'].index , inplace=True)
cleaned['CASE_STATUS'].value_counts() # similarly we have 4 classes of case status which we do not require
#In case status column we can drop withdrawn records and certified-withdrawn records
cleaned.drop(labels=cleaned.loc[cleaned['CASE_STATUS']=='WITHDRAWN'].index , inplace=True)
cleaned.drop(labels=cleaned.loc[cleaned['CASE_STATUS']=='CERTIFIED-WITHDRAWN'].index , inplace=True)
cleaned.info() #Gives data about the columns in the dataframe and the object type pf each dataframe, we get the same number of non-null count for all columns<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1648780 entries, 143 to 664612
Data columns (total 15 columns):
 #   Column                 Non-Null Count    Dtype 
---  ------                 --------------    ----- 
 0   CASE_NUMBER            1648780 non-null  object
 1   CASE_STATUS            1648780 non-null  object
 2   CASE_SUBMITTED         1648780 non-null  object
 3   DECISION_DATE          1648780 non-null  object
 4   VISA_CLASS             1648780 non-null  object
 5   FULL_TIME_POSITION     1648780 non-null  object
 6   JOB_TITLE              1648780 non-null  object
 7   SOC_CODE               1648780 non-null  object
 8   SOC_TITLE              1648780 non-null  object
 9   EMPLOYER_NAME          1648780 non-null  object
 10  WAGE_RATE_OF_PAY_FROM  1648780 non-null  object
 11  WAGE_UNIT_OF_PAY       1648780 non-null  object
 12  WORKSITE_CITY          1648780 non-null  object
 13  WORKSITE_STATE         1648780 non-null  object
 14  H1B_DEPENDENT          1648780 no[...]<jupyter_text>TemporalityThe data is available from 1/1/2015 to  9/9/2019<jupyter_code>max_date=cleaned.CASE_SUBMITTED.max()	
min_date=cleaned.CASE_SUBMITTED.min()
print("The date in the dataset ranges from "+str(max_date)+" to",min_date)
#the column wages has a mix of both string and float value types and some record have the symbol '$' which we want to remove
cleaned['WAGES']=cleaned['WAGE_RATE_OF_PAY_FROM'].apply(cw.clean_wages).astype('float')
# the wage information that we have available has different unit of pay
cleaned['WAGE_UNIT_OF_PAY'].value_counts()
cleaned= cw.clean_wageUnit(np,cleaned)# we convert the different units of pay to the type 'Year'
cleaned.drop(columns=['WAGE_RATE_OF_PAY_FROM','WAGE_UNIT_OF_PAY'],axis=1,inplace=True)
print(cleaned['WAGES'].describe())#checking if the WAGES column have outliers
sns.boxplot(y=cleaned['WAGES'])
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that we have large outliers , we can remove them from the data.<jupyter_code># Remove the outliers.
lowerBound = 0.001
upperBound = 0.95
result = cleaned.WAGES.quantile([lowerBound, upperBound])
cleaned = cleaned[(result.loc[lowerBound] < cleaned.WAGES.values) &\
                  (cleaned.WAGES.values < result.loc[upperBound])]

print(cleaned['WAGES'].describe())
sns.boxplot(y=cleaned['WAGES'])
plt.show()
#Plot for top 10 companies that have the most applications for H1B visa
plt=vz.top_employer(plt,sns,cleaned)
plt.show()
cleaned= cw.clean_states(cleaned)<jupyter_output><empty_output><jupyter_text>Data ScopeThe data available contains applications submitted by Employers from 2015 till 2019. \Geographical Scope: The WORKSITE_STATE gives information about the state where the primary worksite location. The data set covers all the states of US. The Data set also cover different Industries majority of the applications are for IT industry but not limited to IT, it also covers Medical, Pharmacy, Law, Carpentery etc.<jupyter_code>df=cleaned.SOC_TITLE.value_counts()
df.head(5)
df=cleaned.WORKSITE_STATE.value_counts()
df.head(5)<jupyter_output><empty_output><jupyter_text>Faithfulness:We do not have any inconsistensies in the data apart from some spelling errors in texual data which we are fixing by lemmatizing and using spellcheckers.<jupyter_code>#The Job_title and Soc_title has inconsistencies such as numeric characters and symbols so it has to be cleaned
cleaned=cw.text_clean(cleaned) 
#Every column has some values that appear less than 15 times , we are eleminating such rows using the function below
cleaned= cw.drop_less_significant(cleaned)
cleaned=cw.cat_to_num(cleaned) #Function to convert categorical values to numerical values 
# Function to get data science domain jobs
data_scnt,data_anlst,data_eng,mach_learn=cw.data_jobs(cleaned)<jupyter_output><empty_output><jupyter_text>Are you a international student? Are you looking for jobs in Data Science domain?The below visualization shows median salary for four data science related jobs and the number of visa applications that are submitted for each job. This gives us insights about the jobs available for international students/workers about the job market for Data Science domain. We get to know the number of jobs available for each caegory as well as the median wage for each category.<jupyter_code>f,ax=plt.subplots(figsize=(8,6))
bplot1=plt.boxplot([data_scnt[data_scnt['WAGES']<200000].WAGES,data_anlst[data_anlst['WAGES']<200000].WAGES,data_eng[data_eng['WAGES']<200000].WAGES,mach_learn[mach_learn['WAGES']<200000].WAGES],patch_artist="True")
ax.set_xticklabels(['Data Scientists','Data Analysts','Data Engineer','Machine Learning'],fontsize=15)
ax.set_title('Salary Distribution for jobs in Data Science field in 2019', fontsize=15)
ax.tick_params(labelsize=10)
colors = ['blue','orange', 'green', 'red'] 
for patch, color in zip(bplot1['boxes'], colors): 
  patch.set_facecolor(color)
plt.show()
datajobs=cw.data_concat(pd,data_scnt,data_anlst,data_eng,mach_learn)
ax2=sns.countplot(x="data", data=datajobs)
ax2.set_title('Number of petitions for each jobs in Data Science field in 2019', fontsize=15)
warnings.simplefilter("ignore")<jupyter_output><empty_output><jupyter_text>Job Distribustion in the US for H1b applicants.The below visualizations gives us insights into the jobs available in USA for every state. H1B applicants can aim for jobs in states with high job openings.As the majority of applicants are from IT domain and software domain we can see that California has the highest number of jobs.<jupyter_code>cleaned=cw.states_vis(cleaned)
df=cleaned.groupby(['CODE']).size().reset_index(name='counts')
fig=vz.USA_map(go,tls,df)
fig.show()<jupyter_output><empty_output><jupyter_text>Comparison of Education levels for some common courses![picture](https://drive.google.com/uc?id=1J2SP3r-B17JZSRwkCE34O87iBBqx5CMc) **Features considered for the classifier**<jupyter_code># Cleaning SOC_TITLE(job_positions) column by removing punctuations
X, Y = cw.fun_preprocessing(cleaned)<jupyter_output><empty_output><jupyter_text>The baseline classifier is done with a basic model. In this case we are taking the mean of the labels ('certified' and 'denied' for H1B visa approvals). It will give us the base accuracy to which we will compare our classifier's accuracy. Our classifier should have a better accuracy than the baseline classifier accuracy.<jupyter_code># This step assigns a binary class label (0 or 1) to each label for H1B visa approval. 
y = Y.to_numpy().astype(int) # Groundtruth labels for the dataset
baseline_clasifier_accuracy,mean_AUC=blc.Acceptance_baseline(X,y)
print('Baseline accuracy (K-fold): ', baseline_clasifier_accuracy)
print('AUC K-fold: ', mean_AUC)<jupyter_output>Baseline accuracy (K-fold):  0.9925938830376568
AUC K-fold:  0.5<jupyter_text>The initial accuracy results of the baseline classifier was 0.99. This result is due to the highly imbalanced data with respect to the Case Status, where there are 624682 CERTIFIED applications and 5158 DENIED applications. The proportion of it is 134.02 to 1. Therefore, a performance measure based on the accuracy is not a good one. A better performance measure in imbalanced data is the Area under the ROC Curve (AUC). It meassures the likelihood that given two random points (one from the positive and one from the negative class) the classifier will rank the point from the positive class higher than the one from the negative one.  Balancing Data:Our Dataset has imbalanced data in the ratio of acceptance to rejection was 134:1 . We are using ADASYN to oversameple the minority and ENN to undersample the Majority class. After balancing the ratio is 2:1.<jupyter_code># Oversampling the minority class
ada = ADASYN(sampling_strategy=0.5, random_state=0)
X_ada, y_ada = ada.fit_resample(X, Y)
# Undersampling the majority class
enn = EditedNearestNeighbours()
X, Y = enn.fit_resample(X_ada, y_ada)
count_y_ada_enn=Counter(Y)
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25)# Testing with regular split
prediction_bl = blc.run_clasifier(X_train, y_train, X_test, np)
split_accuracy = blc.compute_accuracy(y_test, prediction_bl)
split_AUC = blc.compute_AUC(y_test, prediction_bl)
print('Baseline accuracy (split): ', split_accuracy)  
print('AUC split (80-20): ', split_AUC)
# Confusion matrix
tn, fp, fn, tp = metrics.confusion_matrix(y_test, prediction_bl).ravel()
print("tn, fp, fn, tp", (tn, fp, fn, tp))
# classification report
report = metrics.classification_report(y_test, prediction_bl)
print(report)
# Plot precision recall curve
plt=vz.plot_precision_recall(plt,y_test, prediction_bl,precision_recall_curve)<jupyter_output><empty_output><jupyter_text>**ML Task 1- Predicting H1B visa application Status using Random Forest**We are using Random forset classifier to predict if a visa aplication is accepted or rejected.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25, random_state=20)
model_clf=RandomForestClassifier(n_jobs=2,random_state=0)
model_clf.fit(X_train,y_train)#train the model
prediction_test = model_clf.predict(X_test)#test the model (predict with our test data)
prediction_test
#compare with original value, Y_test
print("Accuracy = ", metrics.accuracy_score(y_test, prediction_test))
# Confusion matrix
metrics.confusion_matrix(y_test, prediction_test)
# classification report
report = metrics.classification_report(y_test, prediction_test)
report
y_probs_rand_forest = model_clf.predict_proba(X_test)
vz.plot_precision_recall(plt, y_test, y_probs_rand_forest,precision_recall_curve)<jupyter_output><empty_output><jupyter_text>**President Trump's "Buy American and Hire American" Executive order's impact on H1B visa**The "Buy American and Hire American" order was passed on April 18, 2017, to Order to protect American workers and strengthen the American economy. The after-effects of which caused a gradual decrease in the request and acceptance of H1B visas have been creating more job opportunities for immigrants which in turn contributes to the American workforce.It was also observed that just before this order was passed there was a surge in the applications submitted. As the graph demonstrates, this type of order, after it has been approved, take some months (around 6 months in this case)  to show the after-effects. It's in 2018 when we see the number of applications and their acceptance is falling so far, as shown in the below visualization.<jupyter_code>plt=vz.plot_buy_american_order(plt, original, pd)
plt.show()<jupyter_output><empty_output><jupyter_text>From the above visualization made on a random sample of about 20000 records from 2015 to 2019, we can infer that the visa acceptance is gradually seen as a downfall, which is one the after-effects of the executive order passed earlier 2017.  **ML Task 2-WAGE RANGE RANGE PREDICTION:**Using the data from our dataset we divide the wages into 4 categories which is LOW, AVERAGE, HIGH, VERY_HIGH.And this predictor will predict the expected salary/wage range for any H1B visa holder or an H1B aspirant based on various factors like location , occupation ,employer etc. **BASELINE CLASSIFIER FOR WAGE RANGE PREDICTOR:**<jupyter_code>#assigning X, y for baseline classifier wage classifier.
cleaned,df_temp = cw.map_wage(cleaned)
X = cleaned['WC_NUM'].to_numpy()
y = df_temp["WC_NUM"].to_numpy().astype(int)
baseline_clasifier_accuracy=blc.baseline_wage(X,y)
print('Baseline accuracy (K-fold): ', baseline_clasifier_accuracy)
split_accuracy=blc.baseline_wage_kfold(X,y)
print('Baseline accuracy (split): ', split_accuracy)<jupyter_output>Baseline accuracy (K-fold):  0.7351642569256824
Baseline accuracy (split):  0.7355527960104863<jupyter_text>**Features considered for the wage classifier**<jupyter_code>Dataset_wage,df_wage=cw.get_df_wage(cleaned)
X,Y = cw.get_XY_wage(Dataset_wage,df_wage)
del Dataset_wage
#Split the date into training and test set
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25, random_state=20)
dec_clf = DecisionTreeClassifier(max_depth=15,criterion='gini')
dec_clf.fit(X_train,y_train)#train the model
prediction_test = dec_clf.predict(X_test)#test the model (predict with our test data)
print("Accuracy = ", metrics.accuracy_score(y_test, prediction_test))#compare with original value, Y_test
metrics.confusion_matrix(y_test, prediction_test)
report = metrics.classification_report(y_test, prediction_test)
print(report)<jupyter_output>precision    recall  f1-score   support

         0.0       0.64      0.01      0.02      5287
         1.0       0.80      0.99      0.89    129410
         2.0       0.70      0.15      0.24     27776
         3.0       0.00      0.00      0.00      3249

    accuracy                           0.80    165722
   macro avg       0.54      0.29      0.29    165722
weighted avg       0.77      0.80      0.74    165722
{'fraction_non_alphanumeric': 0.07573043770289936, 'fraction_numerical': 0.032351953431098174, 'mean_word_length': 5.420050305425799, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 68, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '218406', 'license': 'MIT', 'n_tokens_mistral': 5933, 'n_tokens_neox': 5371, 'n_words': 1896, 'path': 'H1B-Analysis.ipynb', 'repo_name': 'ginogustavo/h1bvisa', 'size': '17889'}