<jupyter_start><jupyter_text>MediZen Strain Recommendation API Model Version 1.1 - 2019-11-19 --- Imports and Config<jupyter_code># General imports
import pandas as pd
import janitor
import os
# NLP Imports
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.neighbors import NearestNeighbors
# Configure pandas to display entire text of column
pd.set_option('max_colwidth', 200)
pd.set_option('max_columns', 200)  # Display up to 200 columns<jupyter_output><empty_output><jupyter_text>--- Data Loading and First Looks<jupyter_code># Load the data into pd.DataFrame
filepath = "/Users/Tobias/workshop/buildbox/medizen_ds_api/data/"
data_filename = "cannabis.csv"
data_filepath = os.path.join(filepath, data_filename)

df1 = pd.read_csv(data_filepath)
df1.head()<jupyter_output><empty_output><jupyter_text>--- Data Wrangling and Feature EngineeringThe end result that should be passed into the model is a single long string.  Therefore, the new feature will simply be a concatenation of all three current features:- `type`- `effects`- `flavor`The cell below uses [pyjanitor](pyjanitor.readthedocs.io/) method-chaining to:1. Clean up the feature names, which in this case only makes them lowercase2. Concatenate the three features into one one, comma-separated feature3. Remove all of the features except the new one<jupyter_code># User pyjanitor to wrangle the data

df2 = (df1
        .clean_names()  # In this case, fixes Title Case
        .concatenate_columns(
            # Explanation above - create one feature for NLP analysis
            column_names=["type", "effects", "flavor"],
            new_column_name="type_effects_flavor",
            sep=",",  # Staying consistent with comma-separation
        )
        .remove_columns(column_names=[
            "rating",
            "description",
            "type",
            "effects",
            "flavor",
        ]))
# Look at the resulting dataframe
print(df2.shape)
df2.head()
# Look at null values - hint: there shouldn't be any
# because they all have values in at least one of the three columns
df2.isnull().sum()<jupyter_output><empty_output><jupyter_text>--- TF-IDFTF-IDF is a method of finding unique aspects of documents (strings).  The more common a word is across the documents the lower the score.  The result is the unique topics rising to the top. (They are called _top_-ics, after all)...<jupyter_code># Instantiate the vectorizer object
tfidf = TfidfVectorizer(stop_words="english")

# Create a vocabulary from the new feature
dtm = tfidf.fit(df2["type_effects_flavor"])

# Create vectorized version of wrangled dataframe
sparse = tfidf.transform(df2["type_effects_flavor"])

# The result is a sparse matrix, which can be converted back to a dataframe
vdtm = pd.DataFrame(sparse.todense(), columns=tfidf.get_feature_names())<jupyter_output><empty_output><jupyter_text>Get Similarities with Nearest Neighbor (K-NN)In order to get a list of strains that are similar to a given input string, the K-Nearest Neighbor model will be used. This model uses a tree-based approach to calculate the distances between points and recursively clusters them until to find the desired number (k) of neighboring data points.<jupyter_code># Instantiate the knn model
nn = NearestNeighbors(n_neighbors=10, algorithm='ball_tree')

# Fit (train) the model on the TF-IDF vector dataframe created above
nn.fit(vdtm)<jupyter_output><empty_output><jupyter_text>Running the KNN Model, Part 1<jupyter_code># Example string to demonstrate the model
# This is a realistic example of what will be passed 
# into the model once it is integrated into the API
ex_1_str = "indica,happy,relaxed,hungry,talkative,citrus,tangy,flowery"
# Create the input vector
ex_1_vec = tfidf.transform([ex_1_str])
# Pass that vector into the trained knn model, specifying the number of neighbors to return
# This returns a list of two arrays: one is a measure of each neighbors 'near-ness'
# the other (the one we want) contains the indexes for the neighbors 
rec_array = nn.kneighbors(ex_1_vec.todense(), n_neighbors=10)
rec_array
# Extract the second array - the list of strain ids (indexes) that are 'closest' to input
rec_id_list = rec_array[1][0]
rec_id_list
# Although the API will return only this list of indexes,
# for the purposes of this demo I'll hydrate that list with the rest 
# of the strain data from the original (pre-wrangled) dataframe
recommendations = df1.iloc[rec_id_list]
recommendations<jupyter_output><empty_output><jupyter_text>--- The FuncZoneIn order to more easily integrate this recommendation process into the Flask API,the steps can be grouped into a function that will take in a request and return the recommendations.<jupyter_code>def recommend(req, n=10):
    """Function to recommend top n strains given a request."""
    # Create vector from request
    req_vec = tfidf.transform([req])

    # Access the top n indexes
    top_id = nn.kneighbors(req_vec.todense(), n_neighbors=n)[1][0]

    # Index-locate the neighbors in original dataframe
    top_df = df1.iloc[top_id]

    return top_df
# Another example request to test out the above function
ex_2_str = "hybrid,euphoric,energetic,creative,woody,earthy"

# Run the function, this time asking for the top 5 recommendations
ex_2_recs = recommend(ex_2_str, 5)
ex_2_recs
# The API should return a JSON object with only the ids
# Here's a slightly modified version to accomplish that
def recommend_json(req, n=10):
    """Function to recommend top n strains given a request."""
    # Create vector from request
    req_vec = tfidf.transform([req])

    # Access the top n indexes
    rec_id = nn.kneighbors(req_vec.todense(), n_neighbors=n)[1][0]

    # Convert np.ndarray to pd.Series then to JSON
    rec_json = pd.Series(rec_id).to_json(orient="records")

    return rec_json<jupyter_output><empty_output><jupyter_text>PicklingIn order to use the model in the Flask app, it can be pickled. The pickle module, and the pickle file format, allows Python objectsto be serialized and de-serialized. In this case, the trained vectorizerand model can be made into pickle files, which are then loaded into theFlask app for use in the recommendation API.<jupyter_code># Create pickle func to make pickling (a little) easier
def picklizer(to_pickle, filename, path):
    """
    Creates a pickle file.
    
    Parameters
    ----------
    to_pickle : Python object
        The trained / fitted instance of the 
        transformer or model to be pickled.
    filename : string
        The desired name of the output file,
        not including the '.pkl' extension.
    path : string or path-like object
        The path to the desired output directory.
    """
    import os
    import pickle

    # Create the path to save location
    picklepath = os.path.join(path, filename)

    # Use context manager to open file
    with open(picklepath, "wb") as p:
        pickle.dump(to_pickle, p)
# Picklize!

# Export vectorizer as pickle
picklizer(dtm, "vect_02.pkl", filepath)

# Export knn model as pickle
picklizer(nn, "knn_02.pkl", filepath)<jupyter_output><empty_output><jupyter_text>Running the KNN Model, Part 2 (this time with pickles)<jupyter_code>import pickle

# Load the vdtm pickle into new object for testing
vv_path = os.path.join(filepath, "vect_02.pkl")

# Use context manager to open and load pickle file
with open(vv_path, "rb") as p:
    vv = pickle.load(p)
# Load the knn pickle into new object for testing
knn_path = os.path.join(filepath, "knn_02.pkl")

# Use context manager to open and load pickle file
with open(knn_path, "rb") as p:
    knn = pickle.load(p)
# Another slightly modified version that uses test objects

def rec_pickle(req, n=10):
    """Function to recommend top n strains given a request."""
    # Create vector from request
    req_vec = vv.transform([req])

    # Access the top n indexes
    rec_id = knn.kneighbors(req_vec.todense(), n_neighbors=n)[1][0]

    # Convert np.ndarray to pd.Series then to JSON
    rec_json = pd.Series(rec_id).to_json(orient="records")

    return rec_json
# Test out the modified function / pickle objects
ex_3_str = "sativa,energetic,talkative,euphoric,creative,focused,spicy,tangy,sweet"

# This time I'll pass in 8, because why not?
ex_3_rec = rec_pickle(ex_3_str, 8)
ex_3_rec<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08248166406156066, 'fraction_numerical': 0.009258145966093543, 'mean_word_length': 4.284625158831004, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '64834', 'license': 'MIT', 'n_tokens_mistral': 2676, 'n_tokens_neox': 2561, 'n_words': 937, 'path': 'docs/notebooks/03-medizen-recommendations.ipynb', 'repo_name': 'Med-Cabinet-LS-BW/medizen_ds_api', 'size': '8317'}