<jupyter_start><jupyter_text>Fundamental Analysis for Growth Stocks Using the screener provided by Zacks.com, we segment "growth" companies by setting dividend to 0% and optionable to Yes to avoid OTC stocks. We also remove financial companies. Our target variable is Relative Price Change which compares the stock returns to the year to date returns of the S&P 500. Several features will be used for building the initial model. Later we also scrape insider and insitutional transactions from Yahoo Finance.<jupyter_code>import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler,OneHotEncoder
from sklearn.ensemble import IsolationForest
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read in created CSV file showing screener results from Zacks.<jupyter_code>df = pd.read_csv('zacks3.csv')<jupyter_output><empty_output><jupyter_text>View the different features along with the target in their respective column orders.<jupyter_code>features = {name: i for i,name in enumerate(df.columns)}
features<jupyter_output><empty_output><jupyter_text>Only keep certain columns for now for preliminary analysis (variable conversion, outlier detection, visualization).<jupyter_code>keep = df.columns[[1,4,7,10,11,13,6]]
df2 = df[keep]
df2.head()
df2['Sector'].unique()<jupyter_output><empty_output><jupyter_text>Note all the variables are percentages except for sector which is categorical.<jupyter_code># Check if any missing values are present in the dataframe.
df2.isnull().values.any()<jupyter_output><empty_output><jupyter_text>"The relative price is calculated by comparing the calendar year-to-date price change of each stock to the price change of the S&P 500 over the year-to-date time period. A value of 1.0 indicates a company's stock price performance was equal to that of the S&P 500.A negative value indicates that the stock underperformed the S&P.A positive value indicates that the stock outperformed the S&P."Let's see the relative price change by sector. The red vertical line represents performance equal to that of the S&P.<jupyter_code>df2.groupby('Sector')['Relative Price Change'].median().plot.barh(figsize=(10,10),)
plt.axvline(x=1, color='r', linestyle='-')
plt.title('Relative Price Change by Sector')
plt.show()<jupyter_output><empty_output><jupyter_text>As there are no missing values, we will separate out the numeric variables for now into a separate data frame to search for any outliers.<jupyter_code>df2_numeric = df2.select_dtypes(include='float64')
df2_numeric.head()
df2_numeric.hist(bins=30, figsize=(20, 15))
plt.show()<jupyter_output><empty_output><jupyter_text>We will now employ an Isolation forest to remove any potential outliers from the data. An isolation forest recursively generates trees by selecting a feature and split value. Outliers will have shorter paths to reaching a leaf.<jupyter_code>iso_for = IsolationForest(contamination=0.05,random_state=42,n_estimators=200)
out = iso_for.fit_predict(df2_numeric)
df2.loc[:,'iso'] = out
df2.head()<jupyter_output><empty_output><jupyter_text>We remove companies with iso value of -1 indicating an outlier.<jupyter_code>outliers = df2.loc[df2['iso']==-1]
df2_noOutliers = df2.loc[df2.index.difference(outliers.index), ]
outliers
df2_noOutliers['Relative Price Change'].describe()<jupyter_output><empty_output><jupyter_text>Not surprisingly, GameStop (GME) appears as an outlier above. To build our response variable, we create two bins for relative price change: everything above 1 and everything below 1 indicating performing better or worse than the S&P YTD.<jupyter_code>bins = [-100,1,200]
df2_noOutliers['binned'] = pd.cut(df2_noOutliers['Relative Price Change'], bins)
df2_noOutliers['binned'].value_counts()<jupyter_output><empty_output><jupyter_text>Factorize the binned variable to 1 or 0 corresponding to invest or not invest respectively. Then we remove unnecessary columns.<jupyter_code>invest = pd.factorize(df2_noOutliers['binned'])
df2_noOutliers['invest'] = invest[0].tolist()
df3 = df2_noOutliers.drop(['Ticker','iso','binned','Relative Price Change'],axis=1)<jupyter_output><empty_output><jupyter_text>Shuffle rows.<jupyter_code>df3 = df3.sample(frac=1)
def onehot_combine(data,column):
    
    """Function to one hot encode categorical feature
       and put back in place in the data frame."""
    
    one_hot = pd.get_dummies(data[[column]])
    data = data.drop([column],axis=1)
    res = pd.concat([data,one_hot],axis=1)
    return(res)<jupyter_output><empty_output><jupyter_text>Will use one hot encoder from sklearn later.<jupyter_code>from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder()
enc.fit(df3[['Sector']])
df3_final = onehot_combine(df3,column='Sector')
X = df3_final.loc[:, df3_final.columns != 'invest']
y = df3_final.loc[:, 'invest']
X.dtypes<jupyter_output><empty_output><jupyter_text>Separate train and test. Then extract numerical and categorical columns to perform standardization on numerical columns only.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42,stratify=y)
num_cols = df3_final.columns[df3_final.dtypes.apply(lambda f: np.issubdtype(f, np.float64))]
cat_cols = df3_final.columns[df3_final.dtypes.apply(lambda f: np.issubdtype(f, np.uint8))]
sc = StandardScaler()
X_train_num = sc.fit_transform(X_train[num_cols])
X_test_num = sc.transform(X_test[num_cols])
X_train_cat = X_train[cat_cols].values
X_test_cat = X_test[cat_cols].values<jupyter_output><empty_output><jupyter_text>Merge transformed numerical variables to categorical.<jupyter_code>X_train = np.hstack((X_train_num, X_train_cat))
X_test = np.hstack((X_test_num,X_test_cat))<jupyter_output><empty_output><jupyter_text>Build model and train using cross val score. We focus on precision as false positives can lead to investments that lose money.<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, recall_score, roc_auc_score, precision_score

rf = RandomForestClassifier(n_estimators=200,max_depth=3,min_samples_leaf = 20,random_state=42,class_weight = 'balanced_subsample')

clf = rf
cross_val_score(clf,X_train,y_train,cv=3,scoring='precision')
clf.fit(X_train, y_train)
preds = np.where(clf.predict_proba(X_test)[:,1]>.5,1,0)
pd.DataFrame(data=[accuracy_score(y_test, preds), recall_score(y_test, preds),
                   precision_score(y_test, preds), roc_auc_score(y_test, preds)], 
             index=["accuracy", "recall", "precision", "roc_auc_score"])
from sklearn.metrics import confusion_matrix
cf = confusion_matrix(np.array(y_test), preds)
cf<jupyter_output><empty_output><jupyter_text>We now train on the full data.<jupyter_code>num_cols = df3_final.columns[df3_final.dtypes.apply(lambda f: np.issubdtype(f, np.float64))]

cat_cols = df3_final.columns[df3_final.dtypes.apply(lambda f: np.issubdtype(f, np.uint8))]

sc = StandardScaler()
X_num = sc.fit_transform(X[num_cols])
X_cat = X[cat_cols].values

X = np.hstack((X_num, X_cat))
clf.fit(X,y)
# import pickle
# data = {'model':clf,'num_scale':sc,'cat_enc':enc}
# with open('saved_steps.pkl','wb') as file:
#     pickle.dump(data,file)
# with open('saved_steps.pkl','rb') as file:
#     data = pickle.load(file)
# rf = data['model']
# sc = data['num_scale']
# enc = data['cat_enc']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10778044231281642, 'fraction_numerical': 0.016919797495337063, 'mean_word_length': 7.159782608695652, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '621175', 'license': 'MIT', 'n_tokens_mistral': 2583, 'n_tokens_neox': 2476, 'n_words': 637, 'path': 'Fundamental_Analysis.ipynb', 'repo_name': 'aterzy42/StockAnalysis', 'size': '7506'}