<jupyter_start><jupyter_text>"Helping VCs target the next acquisition with better than 90% confidence"> "After analyzing start-up funding data from over 49,000 startups, I can predict which companies are going to be aquired and which are going to be defunct.">>This [dataset](https://www.kaggle.com/arindam235/startup-investments-crunchbase) originated from Crunchbase and was downloaded from kaggle.- toc: true- branch: master- badges: false- comments: true- categories: [embeddings, venture capital, shallow neural nets]- image: images/city_embeddings.png- hide: false- search_exclude: true- metadata_key1: metadata_value1- metadata_key2: metadata_value2 <!--  Helping VCs target the next acquisition with better than 90% confidence    After analyzing start-up funding data from over 49,000 startups, I can predict which companies are going to be aquired and which are going to be defunct.This [dataset](https://www.kaggle.com/arindam235/startup-investments-crunchbase) originated from Crunchbase and was downloaded from kaggle. --><jupyter_code>#hide

%reload_ext autoreload
%autoreload 2
%matplotlib inline

!pip install fastai --upgrade --q

from pandas.api.types import is_string_dtype, is_numeric_dtype, is_categorical_dtype
from fastai.tabular.all import *

from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from xgboost import XGBClassifier
from sklearn import model_selection
from sklearn.utils import class_weight
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
import matplotlib.style as style
style.use('fivethirtyeight')

from IPython.display import Image, display_svg, SVG
from sklearn.model_selection import train_test_split

from google.colab import drive
mnt=drive.mount('/content/gdrive', force_remount=True)
root_dir = "/content/gdrive/My Drive/"<jupyter_output>Mounted at /content/gdrive<jupyter_text>**Overall Strategy**Using the Fast.AI library, I used shallow neural networks to encode all the categorical features of the funding data (company market space, city, region, amount per round) and then extracted the embedding layers for further analysis with Machine Learning models. Instead of a simple label encoding of each feature, I created different vector representations of multiple lengths for each network.  The sample below shows a vector of length 3.  The advantage is that we are able to transform discreet variables into vectors that are continous and meaningful.> >  --- In the Fast.AI book, [Deep Learning for Coders](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527), an excellent example of the benefits of categorical embedding is provided.> On the left is a plot of the embedding matrix for the possible values of the State category. For a categorical variable we call the possible values of the variable its "levels" (or "categories" or "classes"), so here one level is "Berlin," another is "Hamburg," etc. On the right is a map of Germany. The actual physical locations of the German states were not part of the provided data, yet the model itself learned where they must be, based only on the behavior of store sales!    Ensembling the embeddings What makes this code unique is the fact that I took the same data and ran it through 4 different shallow neural networks with a unique embedding size.For each neural net, I extracted the latent layers and trained a Random Forest model on each embedding space and then ensembled the predictions and probabilites. The end result was an improvement of > 6%.  The predictions improve because each model puts an emphasis on different features of the input data.The two charts shown below show the top 10 features based on the importance to the model.  Each model analyzes the data differently based on the relative importance of the features.  Remember, the underlying data is identical, but I'm analyzing the latent space with different embedding sizes.For example, the Random Forest model training on a fixed embedding size of 2, ranks a member of the 'City' vector(City_0) as the most important feature.  Where as the Random Forest using the default embeddings, ranks City_0 8th from the top.  -->       Steps taken to predict the acquisition of the startup was the following:1.   Import and clean the data2.   Create additional features3.   Run data through multiple shallow neural networks to convert categorical data to continous4.   Train ML models on the latent space of the neural nets5.   Ensemble the predictions from each model6.   Train ML model on the Ensemble7.   Select the most confident prediction<jupyter_code>#@title
#hide

def random_seed(seed_value, use_cuda):
    np.random.seed(seed_value) # cpu vars
    torch.manual_seed(seed_value) # cpu  vars
    random.seed(seed_value) # Python
    if use_cuda: 
        torch.cuda.manual_seed(seed_value)
        torch.cuda.manual_seed_all(seed_value) # gpu vars
        torch.backends.cudnn.deterministic = True  #needed
        torch.backends.cudnn.benchmark = False

def feat_importance(m, df):
  # get list of feature importance for given model
    return pd.DataFrame({'cols':df.columns, 'imp':m.feature_importances_}
                      ).sort_values('imp', ascending=False)

    
def save_predictions(model_name, m, df, target_names, note):
  col = []
  for target_name in target_names:
    tmp = "_".join([model_name, target_name, note])
    col.append(tmp)
  try:
    predictions = m.predict_log_proba(df)
  except:
    predictions = m.predict_proba(df)
  
  predict_df = pd.DataFrame(predictions, columns=col).astype("float")
  delta_name = "_".join(['delta', col[1], col[0]])
  predict_df[delta_name] = predict_df[col[1]] - predict_df[col[0]]
  #predict_df['company_index']=df.index
  return predict_df

# modified based on this blog: https://towardsdatascience.com/quickly-test-multiple-models-a98477476f0
def run_exps(models, X_train: pd.DataFrame , y_train: pd.DataFrame, X_test: pd.DataFrame, y_test: pd.DataFrame, note, seed=90210) -> pd.DataFrame:
    '''
    Lightweight script to test many models and find winners
    :param models: array of tuples containing (model_name, model)
    :param X_train: training split
    :param y_train: training target vector
    :param X_test: test split
    :param y_test: test target vector
    :note: 
    :seed: int for randomizer
    :return: DataFrame of predictions
    '''
    #print ( embeded_xs_1.shape, y_train.shape, xs_valid_1.shape, y_test.shape)
    dfs = []
    dfs_predictions = []

    results = []
    names = []
    scoring = ['accuracy', 'precision_weighted', 'recall_weighted', 'f1_weighted', 'roc_auc']
    target_names = ['closed', 'aquired']
    for model_name, model in models:
        kfold = model_selection.KFold(n_splits=5, shuffle=True,
                                      random_state=seed)
        
        cv_results = model_selection.cross_validate(model, X_train, y_train,
                                                    cv=kfold, scoring=scoring)
        clf = model.fit(X_train, y_train)
        y_pred = clf.predict(X_test)
        print((model_name, note))
        print(classification_report(y_test, y_pred, target_names=target_names))

        #fi=feat_importance(clf, X_test)
        #plot_fi(model_name, note, fi[:10])

        print ()
        predict_df=save_predictions(model_name, clf, X_test, target_names, note)
        names.append(model_name)

        dfs_predictions.append(predict_df)
    prediction_dfs = pd.concat(dfs_predictions, axis=1)
    return prediction_dfs 

def nn_embeddings(emb_szs=None):
  # use a neural network to create ebeddings of categorical data
  learn = tabular_learner(dls,layers=[200,100], metrics=accuracy,
                          emb_szs=emb_szs)
  lr_in, lr_steep = learn.lr_find(show_plot=False)
  learn.fit_one_cycle(20, lr_steep, cbs=EarlyStoppingCallback( 
      monitor='accuracy', min_delta=0.1, patience=2))
  
  # after fitting, get embeddings to be used by ML
  embeded_xs = embed_features(learn, learn.dls.train.xs)
  xs_valid = embed_features(learn, learn.dls.valid.xs)

  return (learn, embeded_xs, xs_valid)

def embed_features(learner, xs):
  # https://forums.fast.ai/t/using-embedding-from-the-neural-network-in-random-forests/80063/9
  # citation: danielwbn
    xs = xs.copy()
    for i, feature in enumerate(learner.dls.cat_names):
        emb = learner.model.embeds[i]
        new_feat = pd.DataFrame(emb(tensor(xs[feature], dtype=torch.int64)), index=xs.index, columns=[f'{feature}_{j}' for j in range(emb.embedding_dim)])
        xs.drop(columns=feature, inplace=True)
        xs = xs.join(new_feat)
    return xs

def plot_fi(name, note, fi):

    title = "_".join([name, note])
    fi_plot = fi.plot('cols', 'imp', 'barh', figsize=(12,7), legend=False)
    
    fi_plot.tick_params(axis = 'both', which = 'major', labelsize = 18)

    # Remove the label of the x-axis
    fi_plot.xaxis.label.set_visible(False)
     # Remove the label of the y-axis
    fi_plot.yaxis.label.set_visible(False)
    titles="        " + title
    fi_plot.text(x = .0, y = 10, s = titles,
               fontsize = 26, weight = 'bold', alpha = .75)
    
    fig=fi_plot.get_figure()
    img_path = Path(base_dir + '/' + title + '.png')
    fig.savefig(img_path, bbox_inches = 'tight')

    return fi_plot

# hide
# @title
# https://www.kaggle.com/grfiv4/plot-a-confusion-matrix
# <NAME>
def plot_confusion_matrix(cm,
                          target_names,
                          title='Confusion matrix',
                          cmap=None,
                          normalize=True):
    """
    given a sklearn confusion matrix (cm), make a nice plot

    Arguments
    ---------
    cm:           confusion matrix from sklearn.metrics.confusion_matrix

    target_names: given classification classes such as [0, 1, 2]
                  the class names, for example: ['high', 'medium', 'low']

    title:        the text to display at the top of the matrix

    cmap:         the gradient of the values displayed from matplotlib.pyplot.cm
                  see http://matplotlib.org/examples/color/colormaps_reference.html
                  plt.get_cmap('jet') or plt.cm.Blues

    normalize:    If False, plot the raw numbers
                  If True, plot the proportions

    Usage
    -----
    plot_confusion_matrix(cm           = cm,                  # confusion matrix created by
                                                              # sklearn.metrics.confusion_matrix
                          normalize    = True,                # show proportions
                          target_names = y_labels_vals,       # list of names of the classes
                          title        = best_estimator_name) # title of graph

    Citiation
    ---------
    http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html

    """
    import matplotlib.pyplot as plt
    import numpy as np
    import itertools

    accuracy = np.trace(cm) / float(np.sum(cm))
    misclass = 1 - accuracy

    if cmap is None:
        cmap = plt.get_cmap('Blues')

    plt.figure(figsize=(8, 6))
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()

    if target_names is not None:
        tick_marks = np.arange(len(target_names))
        plt.xticks(tick_marks, target_names, rotation=45)
        plt.yticks(tick_marks, target_names)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]


    thresh = cm.max() / 1.5 if normalize else cm.max() / 2
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        if normalize:
            plt.text(j, i, "{:0.4f}".format(cm[i, j]),
                     horizontalalignment="center",
                     color="white" if cm[i, j] > thresh else "black")
        else:
            plt.text(j, i, "{:,}".format(cm[i, j]),
                     horizontalalignment="center",
                     color="white" if cm[i, j] > thresh else "black")


    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label\naccuracy={:0.4f}; misclass={:0.4f}'.format(accuracy, misclass))
    plt.show()<jupyter_output><empty_output><jupyter_text>Import the data, split into Training and Test, Split into continous and categorical data<jupyter_code>#hide
# load pre-processed data
base_dir = root_dir + 'crunchbase'
path = Path(base_dir)
csv_path = Path(base_dir+'/crunchbase_data_cleaned_3.csv')
df = pd.read_csv(csv_path)
print(csv_path)

cleanup_nums = {"status":     {2: True, 1:False}}
df = df.replace(cleanup_nums)
company_names = df.name # For use later, but not necessary to use it for learning
#city_names=df.city
y=df.status
X=df.copy()
X.drop(['status','name'],axis=1,inplace=True)

dep_var = 'status'

test_size = 0.2
seed = 7
#random_seed(seed,use_cuda=False )
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=seed)
splits = (list(X_train.index),list(X_test.index))

df.drop(['name'], axis=1, inplace=True)

df = df.astype({'funding_total_usd': 'int64',
                                  'first_funding_year':'int64',
                                  'founded_year':'int64',
                                  'last_funding_year':'int64',
                                  'funding_rounds':'int64',
                                  'venture': 'int64'})

cont_nn, cat_nn = cont_cat_split(df, max_card=10, dep_var=dep_var)<jupyter_output>/content/gdrive/My Drive/crunchbase/crunchbase_data_cleaned_3.csv<jupyter_text>List of Categorical features I will encode<jupyter_code>#show
cat_nn<jupyter_output><empty_output><jupyter_text>Run data through multiple shallow neural networks to convert categorical to continous dataTrain ML models on the latent space of the neural nets<jupyter_code># prepare data for neural net
procs_nn = [Categorify, FillMissing, Normalize]
to_nn = TabularPandas(df, procs_nn, cat_nn, cont_nn,
                      splits=splits, y_names=dep_var)
dls = to_nn.dataloaders(512)

# override default embeddings

# ebedding set to 1
emb_szs={'city':1, 'market':1,'region':1,'founded_month':1, 
         'founded_quarter':1,'state_code':1}
_, embeded_xs_1, xs_valid_1 = nn_embeddings(emb_szs)
embed_1_df = run_exps(models, embeded_xs_1, y_train, xs_valid_1, y_test,
                      'embed_1',)

# ebedding set to 2
emb_szs={'city':2, 'market':2,'region':2,'founded_month':2, 
         'founded_quarter':2,'state_code':2}
_, embeded_xs_2, xs_valid_2 = nn_embeddings(emb_szs)
embed_2_df = run_exps(models, embeded_xs_2, y_train, xs_valid_2, y_test,
                      'embed_2')

# ebedding set to 3
emb_szs={'city':3, 'market':3,'region':3,'founded_month':3, 
         'founded_quarter':3,'state_code':3}
_, embeded_xs_3, xs_valid_3 = nn_embeddings(emb_szs)
embed_3_df = run_exps(models, embeded_xs_3, y_train, xs_valid_3, y_test,
                      'embed_3')

# default embedding based on cardinality of data
_, embeded_xs_default, xs_valid_default = nn_embeddings(None)
embed_default_df = run_exps(models, embeded_xs_default, y_train,
                            xs_valid_default, y_test, 'default_embedding')<jupyter_output><empty_output><jupyter_text>Ensemble the predictions from each model<jupyter_code>#show
del combined_predictions
# combine all predictions into single dataframe and train with RF
combined_predictions = pd.concat([embed_1_df, embed_2_df, embed_3_df, embed_default_df ], axis=1)
combined_predictions = combined_predictions.T.drop_duplicates().T

combined_predictions.drop(['company_index'],axis=1,inplace=True, errors='ignore')
combined_predictions.drop(['combined_confidence'],axis=1,inplace=True, errors='ignore')

# now need to split combined_predictions into train and valid sets
X_train_combined, X_test_combined, y_train_combined, y_test_combined = train_test_split(combined_predictions, y_test, test_size=test_size, random_state=seed)<jupyter_output><empty_output><jupyter_text>Train ML model on the Ensemble<jupyter_code>#show

clf = RandomForestClassifier(max_depth=200, random_state=0, oob_score=True, min_samples_split=30)
clf.fit(X_train_combined, y_train_combined)
print (clf.score(X_test_combined, y_test_combined))<jupyter_output>0.75<jupyter_text>The ensemble of the latent layers improves the model by > 6%.  To reach 95% accuracy, I focus on the top 20% of the companies based on the confidence of the model. Select the most confident prediction<jupyter_code>#show
predicted_val=clf.predict_proba(X_test_combined)
predicted=clf.predict(X_test_combined)

pred_df=pd.DataFrame(predicted_val, columns=['z','o'])
#pred_df['names']=(X_test_combined.company_index.values)

pred_df['target']=y_test_combined.values
pred_df['names']=y_test_combined.index
pred_df['predict_acquisition']=predicted


pred_df['diff'] = pred_df.apply(lambda x: x['o'] - x['z'], axis=1)
confident_pred_df=pred_df[(pred_df['diff']>.71) | (pred_df['diff']<-.65)]

confident_pred_df.set_index('names',inplace=True)

final_acquisition_predictions = pd.merge(company_names, confident_pred_df,
                                         how = 'right',left_index = True,
                                         right_index = True)

final_acquisition_predictions.drop(['o','z'], inplace=True, axis=1, 
                                  )

final_acquisition_predictions.index.name = 'idx_name'
final_acquisition_predictions.sort_values(by='diff')<jupyter_output><empty_output><jupyter_text>Interesting high confidence predictionsI predicted that Kickfire was aquired with a 79% confidence, however the data provided by Crunchbase listed it as a closed.  After investigation, Kickfire was actually aquired in 2010 by [Teradata](https://www.teradata.com/).  Details can be found in this [article](https://https://www.infoworld.com/article/2625533/teradata-buys-analytics-vendor-kickfire.html).The other company that I predicted would get aquired is Breaktime Studios.  This company went under in 2016.While I was able to make accurate predictions on the limited data of market type and the amount of funding in each round.  It would have been helpful to know who was making the investments.  Was it credible VCs with a known trackrecord or individuals.  Stay tuined while I include investor data to the next model.<jupyter_code>#collapse-show
cm=confusion_matrix(final_acquisition_predictions.target, final_acquisition_predictions.predict_acquisition,  labels=None, sample_weight=None, normalize=None)
plot_confusion_matrix(cm, ['closed','aquired'], normalize=None)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09753864383045188, 'fraction_numerical': 0.012710723326331663, 'mean_word_length': 3.6664991203820056, 'pattern_counts': {'":': 1, '<': 37, '<?xml version=': 0, '>': 49, 'https://': 9, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '359736', 'license': 'Apache-2.0', 'n_tokens_mistral': 5968, 'n_tokens_neox': 5693, 'n_words': 1595, 'path': '_notebooks/2021_02_13_neural_nets_crunchbase.ipynb', 'repo_name': 'patrick-hanley/thought_experiments', 'size': '18574'}