<jupyter_start><jupyter_text>filtering data from json given requirements<jupyter_code>config["DataFrameFilters"]
filterdata= df1[(df1["EntrySeconds"] <= int(config["DataFrameFilters"][0]["Maximum"])) &
               (df1["PositionType"] == config["DataFrameFilters"][1]["Equals"]) &
               (df1["Float"].between(int(config["DataFrameFilters"][2]["Minimum"]) ,int(config["DataFrameFilters"][2]["Maximum"]), inclusive = False)) &
               (df1["EntryPrice"].between(int(config["DataFrameFilters"][3]["Minimum"]) ,int(config["DataFrameFilters"][3]["Maximum"]), inclusive = False))].reset_index(drop=True)
filterdata.head()<jupyter_output><empty_output><jupyter_text>replace nan value from Fund52WeekChange which is given as ('∞%') to 0% as change in a stock cannot be in infinity<jupyter_code>#counting to numner nan value is present
print(f"the total no of nan value in Fund52WeekChange  is :  {filterdata['Fund52WeekChange'].str.count('∞%').sum()}.")
# As every data point is important here we cant drop the rows so its been replaced by 0% as specified in requirement documnet
filterdata['Fund52WeekChange']=filterdata['Fund52WeekChange'].replace('∞%',"0%")
filterdata['Fund52WeekChange']=filterdata['Fund52WeekChange'].str[:-1].astype(float)<jupyter_output><empty_output><jupyter_text>filtering columns as specified in the requirement document<jupyter_code>filterdata=filterdata.loc[:,['EntrySeconds','Float','FundShortRatio','FundShortPercofFloat','Vol1MinRatioMaxPD123','Vol1MinRatioMaxPD123_TotalVolume','AllExchangesVolume','IntradayCurrentMarketGapPerc','Fund52WeekChange','PercentageProfit','WinTrade']]
filterdata.head(2)<jupyter_output><empty_output><jupyter_text>Finding corelation Matrix<jupyter_code>filterdata.corr()
corrmat =filterdata.corr()
f, ax = plt.subplots(figsize=(12, 9))
sns.heatmap(corrmat,vmin=-1, vmax=1, square=True,cmap="YlGnBu",annot=True);<jupyter_output><empty_output><jupyter_text>we can clearly see the correlation between the two var i.e. AllExchange vs Vol1MinRatioMaPD123_TotalVolume. Actually, this correlation is so strong that it can indicate a situation of multicollinearity. If we think about these two variables, we can conclude that they give almost the same information so multicollinearity really occurs.<jupyter_code>sns.regplot(data=filterdata,x='AllExchangesVolume',y='Vol1MinRatioMaxPD123_TotalVolume')<jupyter_output><empty_output><jupyter_text>there is no difference between the two varibale ,One Variable would be suffice to explain other that makes other Variable redundant for our model<jupyter_code>#dropping a variable to avoid multicollinearity
filterdata.drop(['AllExchangesVolume'],1,inplace=True)<jupyter_output><empty_output><jupyter_text>looking at the collinearity between the independent variable VS two dependent variable we have i.e. [ WINTRADE , PERCENTPROFIT]<jupyter_code>filterdata[filterdata.columns[:]].corr()['WinTrade'][:].sort_values(ascending=False)
filterdata[filterdata.columns[:]].corr()['PercentageProfit'][:].sort_values(ascending=False)
filterdata.describe()<jupyter_output><empty_output><jupyter_text>from this Descriptive analysis we can clearly see that the scales of the varible are differing a long so it would be a problem non tree based model and for dimentionality reduction techniques<jupyter_code>from sklearn.preprocessing import RobustScaler
rs=RobustScaler()
filterdata.loc[:,:'Fund52WeekChange']=rs.fit_transform(filterdata.loc[:,:'Fund52WeekChange'])
filterdata.tail()
corrmat =filterdata.corr()
f, ax = plt.subplots(figsize=(12, 9))
sns.heatmap(corrmat,vmin=-1, vmax=1, square=True,cmap="YlGnBu",annot=True);
corrmat =filterdata.iloc[:,:-2].corr()
f, ax = plt.subplots(figsize=(12, 9))
sns.heatmap(corrmat,vmin=-1, vmax=1, square=True,cmap="YlGnBu",annot=True);
filterdata=filterdata.reset_index(drop=True)
train=filterdata.iloc[:926,:]
test=filterdata.iloc[926:,:]
train.head(5)
test.tail(5)<jupyter_output><empty_output><jupyter_text>Checking for imbalance data<jupyter_code>f,ax=plt.subplots(1,2,figsize=(18,8))
train['WinTrade'].value_counts().plot.pie(explode=[0,0.1],autopct='%1.1f%%',ax=ax[0],shadow=True)
ax[0].set_title('WinTrade')
ax[0].set_ylabel('')
sns.countplot('WinTrade',data=train,ax=ax[1])
ax[1].set_title('WinTrade')
plt.show()<jupyter_output><empty_output><jupyter_text>by lookking at the stats our target var are somewhat fairly distributed<jupyter_code>sns.pairplot(data=filterdata,y_vars=['PercentageProfit'],x_vars=['EntrySeconds','Float','FundShortRatio','FundShortPercofFloat','Vol1MinRatioMaxPD123','Vol1MinRatioMaxPD123_TotalVolume','IntradayCurrentMarketGapPerc','Fund52WeekChange'])
sns.set(rc={"figure.figsize": (20, 8)})
filterdata.drop(['PercentageProfit','WinTrade'],1).plot.kde()<jupyter_output><empty_output><jupyter_text>now we have to Find feature importance using various tree based and non tree based models<jupyter_code>#finding feature importance using Rf
train_copy=train.drop(['PercentageProfit'],1)
from sklearn.ensemble import RandomForestClassifier
rf=RandomForestClassifier(n_estimators=150, max_depth=3,min_samples_leaf=6,max_features=0.3,n_jobs=-1,random_state=2)
rf.fit(train_copy.drop(['WinTrade'],1),train_copy['WinTrade'])
feature=train_copy.drop(['WinTrade'],1).columns.values
print('--traing done___')
x,y=(list(x) for x in zip(*sorted(zip(rf.feature_importances_,feature)
                                   ,reverse=False)))

trace2 = go.Bar(
    x=x,
    y=y,
    marker=dict(
          color=x,
           colorscale='Viridis',
          reversescale= True),
    name='RF feature imp',
    orientation='h'
)

layout=dict(
      title='Barplot of fI',
    width=900, height=600,
     yaxis=dict(
          showgrid=False,
           showline=False,
          showticklabels=True))
data=[trace2]

fig1=go.Figure(data=data)
fig1['layout'].update(layout)
py.iplot(fig1,filename='bars')
#using xgboost
import xgboost as xgbp
xgb=xgbp.XGBClassifier(objective='binary:logistic')
xgb.fit(train_copy.drop(['WinTrade'],1),train_copy['WinTrade'])
feature=train_copy.drop(['WinTrade'],1).columns.values
print('--traing done___')
x,y=(list(x) for x in zip(*sorted(zip(xgb.feature_importances_,feature)
                                   ,reverse=False)))

trace2 = go.Bar(
    x=x,
    y=y,
    marker=dict(
          color=x,
           colorscale='Viridis',
          reversescale= True),
    name='XGBOOST feature imp',
    orientation='h'
)

layout=dict(
      title='Barplot of fI',
    width=900, height=600,
     yaxis=dict(
          showgrid=False,
           showline=False,
          showticklabels=True))
data=[trace2]

fig1=go.Figure(data=data)
fig1['layout'].update(layout)
py.iplot(fig1,filename='bars')
#using GbM
from sklearn.ensemble import GradientBoostingClassifier
gb=GradientBoostingClassifier(n_estimators=150,max_depth=3,min_samples_leaf=3,max_features=0.3,learning_rate=0.05,subsample=0.4,random_state=2)
gb.fit(train_copy.drop(['WinTrade'],1),train_copy['WinTrade'])
feature=train_copy.drop(['WinTrade'],1).columns.values
print('--traing done___')
x,y=(list(x) for x in zip(*sorted(zip(gb.feature_importances_,feature)
                                   ,reverse=False)))

trace2 = go.Bar(
    x=x,
    y=y,
    marker=dict(
          color=x,
           colorscale='Viridis',
          reversescale= True),
    name='GB feature imp',
    orientation='h'
)

layout=dict(
      title='Barplot of GB',
    width=900, height=600,
     yaxis=dict(
          showgrid=False,
           showline=False,
          showticklabels=True))
data=[trace2]

fig1=go.Figure(data=data)
fig1['layout'].update(layout)
py.iplot(fig1,filename='bars')
#using gini
from sklearn.model_selection import RandomizedSearchCV
from sklearn.tree import DecisionTreeClassifier
params={ 'class_weight':['balanced'], 
        'criterion':['gini'],
        'max_depth':[None,2,3,5,],
            'min_samples_leaf':[1,2,5,], 
            'min_samples_split':[2,5,]
       }
clf=DecisionTreeClassifier()
random_search=RandomizedSearchCV(clf,cv=10,
                                 param_distributions=params,
                                 scoring='roc_auc',
                                 n_iter=10
                                    )
random_search.fit(train_copy.drop(['WinTrade'],1),train_copy['WinTrade'])
dtree=random_search.best_estimator_
dtree.fit(train_copy.drop(['WinTrade'],1),train_copy['WinTrade'])
from sklearn import tree

dotfile = open("mytree.dot", 'w')

tree.export_graphviz(dtree,out_file=dotfile,
                     feature_names=train_copy.drop(['WinTrade'],1).columns,
                    class_names=["0","1"],
                     proportion=True)
dotfile.close()<jupyter_output><empty_output><jupyter_text>top 3 for dt is Vol1MinRatioMaxPD123 ,Fund52WeekChange ,Float   mytree.dot file might have been generated in to working directory ( copy and paste in the link to see the tree diagram)  dt can be viewed on : http://webgraphviz.com<jupyter_code>#by using pca
from sklearn.decomposition import PCA
from sklearn.preprocessing import scale
X=train_copy.iloc[:,:-1].copy()
X=scale(X)
pca = PCA(n_components=8)
pca.fit(X)
pca.components_.shape
var= pca.explained_variance_ratio_

print(var)
var1=np.cumsum(np.round(pca.explained_variance_ratio_, decimals=4)*100)

print(var1)
#showing imp feature of  pca1
pca.components_[0]
print(abs( pca.components_ ))
loadings=pca.components_[0]
loadings
list(zip(train_copy.iloc[:,:-1].columns,loadings))<jupyter_output><empty_output><jupyter_text>by observing all the following classifiers manually  'Vol1MinRatioMaxPD123','FundshortRatio' and a tie between 'float' and 'FundshortPercofFloat'  came out to be the top3 varible  trainig data using ensemble method<jupyter_code>from sklearn.svm import SVC
classifier={'RandomForest':RandomForestClassifier(),
            'GradientBoosting':GradientBoostingClassifier(),
            'XGBClassifier':xgbp.XGBClassifier(objective='binary:logistic'),
            'Decisiontree':DecisionTreeClassifier(),
            
    }
from sklearn.model_selection import cross_val_score
for key, classifier in classifier.items():
    classifier.fit(train_copy.drop(['WinTrade'],1),train_copy['WinTrade'])
    training_score = cross_val_score(classifier,train_copy.drop(['WinTrade'],1),train_copy['WinTrade'], cv=5)
    feature=train_copy.drop(['WinTrade'],1).columns.values
    print("Classifiers: ", classifier.__class__.__name__, "Has a training score of",round(training_score.mean(), 2) * 100, "% accuracy score")
    print('--traing done___')

x,y=(list(x) for x in zip(*sorted(zip(classifier.feature_importances_,feature)
                                   ,reverse=False)))

trace2 = go.Bar(
    x=x,
    y=y,
    marker=dict(
          color=x,
           colorscale='Viridis',
          reversescale= True),
    name='GB feature imp',
    orientation='h'
)

layout=dict(
      title='Barplot of FI using Ensemble',
    width=900, height=600,
     yaxis=dict(
          showgrid=False,
           showline=False,
          showticklabels=True))
data=[trace2]

fig1=go.Figure(data=data)
fig1['layout'].update(layout)
py.iplot(fig1,filename='bars')<jupyter_output><empty_output><jupyter_text>the top 3 factors using ensemble are 'Vol1MinRatioMaxPD123','EntrySeconds',''Vol1MinRatioMaxPD123_totalvolume'<jupyter_code>from sklearn.manifold import TSNE
import time
X=train_copy.drop(['WinTrade'],1)
y=train_copy['WinTrade']

t0 = time.time()
X_reduced_tsne = TSNE(n_components=2, random_state=42).fit_transform(X.values)
t1 = time.time()
print("T-SNE took {:.2} s".format(t1 - t0))<jupyter_output>T-SNE took 3.8 s<jupyter_text>To check if we have a good cluster of win and loss trade or not?<jupyter_code>import matplotlib.patches as mpatches
f, (ax1) = plt.subplots(ncols=1, figsize=(24,6))
f.suptitle('Clusters using Dimensionality Reduction', fontsize=14)

blue_patch = mpatches.Patch(color='#0A0AFF', label='Loss trade')
red_patch = mpatches.Patch(color='#AF0000', label='Win Trade')
ax1.scatter(X_reduced_tsne[:,0], X_reduced_tsne[:,1], c=(y == 0), cmap='coolwarm', label='Loss trade', linewidths=2)
ax1.scatter(X_reduced_tsne[:,0], X_reduced_tsne[:,1], c=(y == 1), cmap='coolwarm', label='Win Trade', linewidths=2)
ax1.set_title('t-SNE', fontsize=14)

ax1.grid(True)

ax1.legend(handles=[blue_patch, red_patch])<jupyter_output><empty_output><jupyter_text>as we got top 3 variables we will train our model<jupyter_code>top3=filterdata.loc[:,['Vol1MinRatioMaxPD123','EntrySeconds','Vol1MinRatioMaxPD123_TotalVolume','PercentageProfit','WinTrade']]
top3.head()
train=top3.iloc[:926,:]
test=top3.iloc[926:,:]
test.shape
train_r=train.drop(['WinTrade'],axis=1)
test_r=test.drop(['WinTrade'],axis=1)
train_l=train.drop(['PercentageProfit'],1)
test_l=test.drop(['PercentageProfit'],1)
test_l.shape
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_auc_score
from sklearn.metrics import accuracy_score
params={ 'class_weight':[None,'balanced'], 
        'criterion':['entropy','gini'],
        'max_depth':[None,3,5,7,10,15,20],
            'min_samples_leaf':[1,2,5,10,15], 
            'min_samples_split':[1.0,2,5,10,15]
       }
from sklearn.model_selection import GridSearchCV
clf=DecisionTreeClassifier()
grid_search=GridSearchCV(clf,cv=10,
                        param_grid=params,
                        scoring='roc_auc',
                         n_jobs=-1
                        )
                                    
grid_search.fit(train_l.drop(['WinTrade'],1),train_l['WinTrade'])
grid_search.best_estimator_
dt_bst=grid_search.best_estimator_
dtpred=dt_bst.predict(test_l.drop(['WinTrade'],1))
accuracy_score(dtpred,test_l['WinTrade'])
dt_mat=pd.DataFrame(list(zip(test_l['WinTrade'],dtpred)),columns=['real','predicted'])
pd.crosstab(dt_mat['real'],dt_mat['predicted'])
## GRID SEARCH TALKING ALOT OF TIME  IN MY SYSTEM SO I am  USING RANDOM SEARCH
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
rf = RandomForestClassifier()
param_dist = {"n_estimators":[100,200,300,500,700,1000],
              "bootstrap": [True, False],
              'class_weight':[None,'balanced'], 
                'criterion':['entropy','gini'],
                'max_depth':[None,2,4,5,10,15,20,30],
                'min_samples_leaf':[1,2,5,10,15,20], 
                'min_samples_split':[2,5,10,15,20]
                  }
random=RandomizedSearchCV(rf,cv=10,param_distributions=param_dist,
                        scoring='roc_auc',n_iter=10
                        )
                                    
random.fit(train_l.drop(['WinTrade'],1),train_l['WinTrade'])
random.best_estimator_
rf_bst=random.best_estimator_
rf_pred=rf_bst.predict(test_l.drop(['WinTrade'],1))
accuracy_score(rf_pred,test_l['WinTrade'])
rf_mat=pd.DataFrame(list(zip(test_l['WinTrade'],rf_pred)),columns=['real','predicted'])
pd.crosstab(rf_mat['real'],rf_mat['predicted'])
from sklearn.ensemble import GradientBoostingClassifier
gbm_params={'n_estimators':[40,50,100,200,300,500,700],            
           'learning_rate': [0.005,0.01,.05,0.1,0.4,0.8,1],
            'max_depth':[1,2,3,4,5,6],
            'subsample':[0.2,0.4,0.5,0.8,1],
            'max_features':[1,2,3]
           }
gbm=GradientBoostingClassifier()
random_gbm=RandomizedSearchCV(gbm,scoring='roc_auc',param_distributions=gbm_params,
                                 cv=10,n_iter=15,
                                 n_jobs=-1)
random_gbm.fit(train_l.drop(['WinTrade'],1),train_l['WinTrade'])
random_gbm.best_estimator_
rn_gbbst=random_gbm.best_estimator_
gb_pred=rn_gbbst.predict(test_l.drop(['WinTrade'],1))
accuracy_score(gb_pred,test_l['WinTrade'])
gb_mat=pd.DataFrame(list(zip(test_l['WinTrade'],gb_pred)),columns=['real','predicted'])
pd.crosstab(gb_mat['real'],gb_mat['predicted'])
import xgboost as xgbp
xgb=xgbp.XGBClassifier(objective='binary:logistic')
xgb_params = {  
                "learning_rate":[0.005,0.01,0.05,0.1,0.3,0.5,0.7],
                'max_delta_step':[0,1,3,6,10],
                "max_depth": [2,3,4,5,6,7,8],
                "min_child_weight":[1,2,5,7,10,12],
                "max_delta_step":[0,1,2,5,7,10],
                "subsample":[i/10.0 for i in range(5,10)],
                "colsample_bytree":[i/10.0 for i in range(5,10)],
                "reg_lambda":[1e-5, 1e-2, 0.1, 1, 100], 
                'reg_alpha':[i/10 for i in range(0,50)],
                "scale_pos_weight":[1,2,3,4,5,6,7,8,9],
                "n_estimators":[100,300,500,700,1000]
             }
random_search=RandomizedSearchCV(xgb,n_jobs=-1,cv=10,n_iter=10,scoring='roc_auc',
                                 param_distributions=xgb_params)

random_search.fit(train_l.drop(['WinTrade'],1),train_l['WinTrade'])
random_search.best_estimator_
xg_bst=random_search.best_estimator_
xg_pred=xg_bst.predict(test_l.drop(['WinTrade'],1))
accuracy_score(xg_pred,test_l['WinTrade'])
xg_mat=pd.DataFrame(list(zip(test_l['WinTrade'],xg_pred)),columns=['real','predicted'])
pd.crosstab(xg_mat['real'],xg_mat['predicted'])
#combination of all
classifiers={'Decisiontree':DecisionTreeClassifier(),
            'RandomForest':RandomForestClassifier(),
            'GradientBoosting':GradientBoostingClassifier(),
            'XGBClassifier':xgbp.XGBClassifier(objective='binary:logistic')
            }
for key, classifier in classifiers.items():
    classifier.fit(train_l.drop(['WinTrade'],1),train_l['WinTrade'])
    training_score = cross_val_score(classifier,train_l.drop(['WinTrade'],1),train_l['WinTrade'], cv=5)
    print("Classifiers: ", classifier.__class__.__name__, "Has a training score of",round(training_score.mean(), 2) * 100, "% accuracy score")
    
from sklearn.ensemble import VotingClassifier
model1=DecisionTreeClassifier(random_state=1)
model2=RandomForestClassifier(random_state=1)
model3=GradientBoostingClassifier(random_state=1)
model4=xgbp.XGBClassifier(objective='binary:logistic',random_state=1)
model = VotingClassifier(estimators=[('dt', model1), ('rf', model2),('gbm',model3),('xg',model4)], voting='hard')
model.fit(train_l.drop(['WinTrade'],1),train_l['WinTrade'])
cb=model.predict(test_l.drop(['WinTrade'],1))
accuracy_score(cb,test_l['WinTrade'])
xc_mat=pd.DataFrame(list(zip(test_l['WinTrade'],cb)),columns=['real','predicted'])
pd.crosstab(xc_mat['real'],xc_mat['predicted'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14994270748076607, 'fraction_numerical': 0.030937960386315272, 'mean_word_length': 5.109333333333334, 'pattern_counts': {'":': 12, '<': 65, '<?xml version=': 0, '>': 64, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '462820', 'license': 'MIT', 'n_tokens_mistral': 6712, 'n_tokens_neox': 6381, 'n_words': 921, 'path': 'Code/Nyse_Stock_prediction.ipynb', 'repo_name': 'amitthakur4/Nyse-Stock-Prediction-and-EDA-', 'size': '18327'}