<jupyter_start><jupyter_text>Quick summary- Kaggle  -  2351 rows, 2264 with "None" effect dropped  -  Rating (out of 5)  -  Up to 5 effects (majority have 5)    -  most likely in order, based on what I saw on the website  -  Up to 4 flavors (majority have 3)- Leafly  -  3447 rows, 1254 with NAN effects reports dropped  -  Rating (out of 5)    - includes number of ratings  -  Feelings, helps with, and negative effects    -  top 5 reported results in each category    -  includes number of reports    -  includes percentage of reports for that result    -  5 feelings & helps with for all rows that are not nan  -  No flavors (possibly merge from kaggle?)   Data sourced and scraped from leafly.com by <NAME>, Data Engineer<jupyter_code># Import data

import pandas as pd

kaggle = pd.read_csv('cannabis.csv')
leafly = pd.read_json('leafly.json')
kaggle.shape, leafly.shape
# Kaggle df head

kaggle.head()
# Leafly df head

leafly.head()
# Kaggle null values count

kaggle.isna().sum()
# Leafly null values count

leafly.isna().sum()<jupyter_output><empty_output><jupyter_text>Without none|nulls in effects|feelings-  Leafly still has nulls for some negative effects<jupyter_code># Get Kaggle Effects column where they equal none shape

kaggle[~(kaggle['Effects'] == 'None')].shape
# Get leafly shape

leafly[leafly['num_effects_reports'].notna()].shape
# Check leafly for nulls

leafly[leafly['num_effects_reports'].notna()].isnull().sum()<jupyter_output><empty_output><jupyter_text>Effects- Leafly feelings are almost identical to Kaggle's effects  -  Kaggle includes 'dry' and 'mouth'- Leafly has conditions that each strain helps with- Leafly has negative conditions  - including 'dry mouth'<jupyter_code># Kaggle effects

kaggle_effects = set()
for row in kaggle['Effects'].str.lower().str.split(','):
  kaggle_effects.update(set(row))
len(kaggle_effects), kaggle_effects
# Kaggle number of effects value counts

kaggle['Effects'].str.lower().str.split(',').str.len().value_counts(dropna=False).sort_index()
# Leafly Feelings
leafly_feelings = set()
for i in range(1,6):
  leafly_feelings.update(leafly[f'feeling_{i}'].str.lower().unique())
len(leafly_feelings), leafly_feelings
# Leafly helps with
leafly_helps = set()
for i in range(1,6):
  leafly_helps.update(leafly[f'helps_{i}'].str.lower().unique())
len(leafly_helps), leafly_helps
# Leafly negative
leafly_negative = set()
for i in range(1,6):
  leafly_negative.update(leafly[f'negative_{i}'].str.lower().unique())
len(leafly_negative), leafly_negative<jupyter_output><empty_output><jupyter_text>Kaggle Flavors- could probably add to leafly dataframe<jupyter_code># Kaggle flavors

kaggle['Flavor'].str.lower().str.split(',').str.len().value_counts(dropna=False).sort_index()
# Kaggle flavors null count

kaggle['Flavor'].isna().sum()<jupyter_output><empty_output><jupyter_text>Uniqueness of names<jupyter_code># Kaggle unique names of strains count

kaggle['Strain'].nunique(), kaggle.shape[0]
# Leafly unique names of strains count

leafly['strain'].nunique(), leafly.shape[0]<jupyter_output><empty_output><jupyter_text>Drop nans<jupyter_code># Check kaggle shape

kaggle.shape
# Drop nans from kaggle

kaggle = kaggle.dropna()
# Check kaggle shape again

kaggle.shape
# Check kaggle for nans

kaggle.isna().sum()
# Check leafly shape

leafly.shape
# Replace nans in aka column with "unknown" so they don't get dropped

leafly['aka'] = leafly['aka'].fillna("unknown")
# Drop nans from leafly

leafly = leafly.dropna()
# Check Leafly Shape again

leafly.shape
# Check leafly for nans

leafly.isna().sum()
# Check leafly for anything missing/prepare to combine columns

leafly.head()
# Combine feelings into one column for one hot encoding

leafly['feelings'] = leafly['feeling_1'] + ',' + leafly['feeling_2'] + ',' + leafly['feeling_3'] + ',' + leafly['feeling_4'] + ',' + leafly['feeling_5']
# Check feelings

leafly['feelings']
# Combine helps into one column for one hot encoding

leafly['helps'] = leafly['helps_1'] + ',' + leafly['helps_2'] + ',' + leafly['helps_3'] + ',' + leafly['helps_4'] + ',' + leafly['helps_5']
# Check helps

leafly['helps']
# Combine feelings and helps columns

leafly["feelings_helps"] = leafly['feelings'] + ',' + leafly['helps'] 
# Check feelings_helps

leafly['feelings_helps']<jupyter_output><empty_output><jupyter_text>Machine Learning Baseline - Neural Network<jupyter_code># Imports

from sklearn.model_selection import train_test_split
import pandas as pd
import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Embedding, Input
from tensorflow.keras import layers
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from tensorflow.keras.layers import LSTM
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn import preprocessing
import pickle
# Instantiate feelings_helps column for training

X = leafly['feelings_helps']
# Check x shape

X.shape
# Instantiate strain for target

y = leafly['strain']
# Check y shape

y.shape
# Check y

print(y)
# Since the neural network cannot take in straight text, vectorize X data

# Create the transformer

vect = CountVectorizer()

# Build vocab

vect.fit(X)

# Transform text

dtm = vect.transform(X)
# Checking to see that the feature names are correct

print(vect.get_feature_names())
# Now I am going to use todense() to turn X into a matrix

X = dtm.todense()
# Make sure it turned into a matrix correctly

X
# Create label encoder for y

# Label encoder turns y into a numeric representation of the data

le = preprocessing.LabelEncoder()
y = le.fit_transform(y)
# Check y is an array

y
# Check X shape

X.shape
# Check y shape

y.shape
# Create Neural Network for baseline model

# Customized optimizer

# AdamamsGrad is using a super low learning_rate which means the changes between epochs will be smaller

AdamamsGrad = tf.keras.optimizers.Adam(
    learning_rate=0.0001,
    amsgrad=False,
)
# Instantiating Sequential model type and create model structure


model = Sequential()

model.add(Input(38))
model.add(Dense(64, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(256, activation='relu'))
model.add(Dense(2350, activation='softmax'))

model.compile(loss='sparse_categorical_crossentropy',
              optimizer=AdamamsGrad, 
              metrics=['accuracy'])

model.summary()
# Fit model and get baseline accuracy

#model.fit(X,y, epochs=1000)

#  0s 8ms/step - loss: 2.0142 - accuracy: 0.3505<jupyter_output><empty_output><jupyter_text>Machine Learning Recommendation Model - Nearest Neighbors<jupyter_code># Imports

import pandas as pd
import re
import string
import matplotlib.pyplot as plt
import numpy as np
import spacy
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.neighbors import NearestNeighbors
leafly.head()
# Remove commas and add spaces

leafly['feelings'] = leafly['feelings'].str.replace(',', ' ')
leafly['helps']= leafly['helps'].str.replace(',',' ')
# Combine all text columns into one

leafly['feelings_helps_description'] = leafly["feelings"] + ' ' + leafly["helps"] + ' ' + leafly['description']
# Verify new column

leafly.head()
# Instantiate Vectorizer 

tfidf = TfidfVectorizer(stop_words = 'english',ngram_range = (1,2),max_features = 2000)
# Instantiate Document-Term Matrix

dtm = tfidf.fit_transform(leafly['feelings_helps_description'])
dtm = pd.DataFrame(dtm.todense(), columns=tfidf.get_feature_names())
# Instantiate Nearest Neighbors Model

nn = NearestNeighbors(n_neighbors=4, algorithm='kd_tree')
# Fit Nearest Neighbors Model

nn.fit(dtm)
# View Feature Matrix as DataFrame

print(dtm.shape)
dtm.head()
import pickle

# Dump the trained classifier (nn) with Pickle

nn_pickle_filename = 'nn_model.pkl'
pickled_model = open(nn_pickle_filename, 'wb')  # Open the file to save as pkl file
pickle.dump(nn, pickled_model)
pickled_model.close() # Close the pickle instances
# Load the saved model

model_pkl = open(nn_pickle_filename, 'rb')
NN_model = pickle.load(model_pkl)
print ("Loaded model :: ", NN_model)  # print to verify
# Dump the trained classifier (tfidf) with Pickle

tfidf_pickle_filename = 'tfidf.pkl'
pickled_tfidf = open(tfidf_pickle_filename, 'wb')  # Open the file to save as pkl file
pickle.dump(tfidf, pickled_tfidf)
pickled_tfidf.close() # Close the pickle instances
# Loading the saved model

tfidf_pkl = open(tfidf_pickle_filename , 'rb')
tfidf_model = pickle.load(tfidf_pkl)
print ("Loaded model :: ", tfidf_model)  # print to verify
# Recommend Function



import json


try:

    def recommend(user_input):
        temp_df = NN_model.kneighbors(tfidf_model.transform([user_input]).todense())[1]
    

    #print(temp_df)
    
        for i in range(4):
            info = leafly.iloc[temp_df[0][i]]['strain']
            info_aka = leafly.iloc[temp_df[0][i]]['aka']
            info_type = leafly.iloc[temp_df[0][i]]['type']
            info_rating = leafly.iloc[temp_df[0][i]]['rating']
            info_num_reviews = leafly.iloc[temp_df[0][i]]['num_reviews']
            info_effects_reports = leafly.iloc[temp_df[0][i]]['num_effects_reports']
            info_feeling_1 = leafly.iloc[temp_df[0][i]]['feeling_1']
            info_feeling_1_pct = leafly.iloc[temp_df[0][i]]['feeling_1_pct']
            info_feeling_2 = leafly.iloc[temp_df[0][i]]['feeling_2']
            info_feeling_2_pct = leafly.iloc[temp_df[0][i]]['feeling_2_pct']
            info_feeling_3 = leafly.iloc[temp_df[0][i]]['feeling_3']
            info_feeling_3_pct = leafly.iloc[temp_df[0][i]]['feeling_3_pct']
            info_feeling_4 = leafly.iloc[temp_df[0][i]]['feeling_4']
            info_feeling_4_pct = leafly.iloc[temp_df[0][i]]['feeling_4_pct']
            info_feeling_5 = leafly.iloc[temp_df[0][i]]['feeling_5']
            info_feeling_5_pct = leafly.iloc[temp_df[0][i]]['feeling_5_pct']
            info_helps_1 = leafly.iloc[temp_df[0][i]]['helps_1']
            info_helps_1_pct = leafly.iloc[temp_df[0][i]]['helps_1_pct']
            info_helps_2 = leafly.iloc[temp_df[0][i]]['helps_2']
            info_helps_2_pct = leafly.iloc[temp_df[0][i]]['helps_2_pct']
            info_helps_3 = leafly.iloc[temp_df[0][i]]['helps_3']
            info_helps_3_pct = leafly.iloc[temp_df[0][i]]['helps_3_pct']
            info_helps_4 = leafly.iloc[temp_df[0][i]]['helps_4']
            info_helps_4_pct = leafly.iloc[temp_df[0][i]]['helps_4_pct']
            info_helps_5 = leafly.iloc[temp_df[0][i]]['helps_5']
            info_helps_5_pct = leafly.iloc[temp_df[0][i]]['helps_5_pct']
            info_negative_1 = leafly.iloc[temp_df[0][i]]['negative_1']
            info_negative_1_pct = leafly.iloc[temp_df[0][i]]['negative_1_pct']
            info_negative_2 = leafly.iloc[temp_df[0][i]]['negative_2']
            info_negative_2_pct = leafly.iloc[temp_df[0][i]]['negative_2_pct']
            info_negative_3 = leafly.iloc[temp_df[0][i]]['negative_3']
            info_negative_3_pct = leafly.iloc[temp_df[0][i]]['negative_3_pct']
            info_negative_4 = leafly.iloc[temp_df[0][i]]['negative_4']
            info_negative_4_pct = leafly.iloc[temp_df[0][i]]['negative_4_pct']
            info_negative_5 = leafly.iloc[temp_df[0][i]]['negative_5']
            info_negative_5_pct = leafly.iloc[temp_df[0][i]]['negative_5_pct']
            info_feelings = leafly.iloc[temp_df[0][i]]['feelings']
            info_helps = leafly.iloc[temp_df[0][i]]['helps']
            info_description = leafly.iloc[temp_df[0][i]]['description']
            info_feelings_helps = leafly.iloc[temp_df[0][i]]['feelings_helps']
            info_feelings_helps_description = leafly.iloc[temp_df[0][i]]['feelings_helps_description']
            
            print(json.dumps(info))
            print(json.dumps(info_aka))
            print(json.dumps(info_type))
            print(json.dumps(info_rating))
            print(json.dumps(info_num_reviews))
            print(json.dumps(info_feeling_1))
            print(json.dumps(info_feeling_1_pct)) 
            print(json.dumps(info_feeling_2))
            print(json.dumps(info_feeling_2_pct)) 
            print(json.dumps(info_feeling_3))
            print(json.dumps(info_feeling_3_pct)) 
            print(json.dumps(info_feeling_4))
            print(json.dumps(info_feeling_4_pct)) 
            print(json.dumps(info_feeling_5))
            print(json.dumps(info_feeling_5_pct)) 
            print(json.dumps(info_helps_1))
            print(json.dumps(info_helps_1_pct)) 
            print(json.dumps(info_helps_2))
            print(json.dumps(info_helps_2_pct)) 
            print(json.dumps(info_helps_3))
            print(json.dumps(info_helps_3_pct)) 
            print(json.dumps(info_helps_4))
            print(json.dumps(info_helps_4_pct)) 
            print(json.dumps(info_helps_5))
            print(json.dumps(info_helps_5_pct)) 
            print(json.dumps(info_negative_1))
            print(json.dumps(info_negative_1_pct))
            print(json.dumps(info_negative_2))
            print(json.dumps(info_negative_2_pct))
            print(json.dumps(info_negative_3))
            print(json.dumps(info_negative_3_pct))
            print(json.dumps(info_negative_4))
            print(json.dumps(info_negative_4_pct))
            print(json.dumps(info_negative_5))
            print(json.dumps(info_negative_5_pct))
            print(json.dumps(info_feelings))
            print(json.dumps(info_helps))
            print(json.dumps(info_description))
            print(json.dumps(info_feelings_helps))
            print(json.dumps(info_feelings_helps_description))
            
            # This is for the data engineers, the return does not work in jupyter lab.  
            # Should work in vsCode, or elsewhere for production.
    
            #return json.dumps(info)
            #return json.dumps(info_aka)
            #return json.dumps(info_type)
            #return json.dumps(info_rating)
            #return json.dumps(info_num_reviews)
            #return json.dumps(info_feeling_1)
            #return json.dumps(info_feeling_1_pct)
            #return json.dumps(info_feeling_2)
            #return json.dumps(info_feeling_2_pct)
            #return json.dumps(info_feeling_3)
            #return json.dumps(info_feeling_3_pct)
            #return json.dumps(info_feeling_4)
            #return json.dumps(info_feeling_4_pct)
            #return json.dumps(info_feeling_5)
            #return json.dumps(info_feeling_5_pct)
            #return json.dumps(info_helps_1)
            #return json.dumps(info_helps_1_pct)
            #return json.dumps(info_helps_2)
            #return json.dumps(info_helps_2_pct)
            #return json.dumps(info_helps_3)
            #return json.dumps(info_helps_3_pct)
            #return json.dumps(info_helps_4)
            #return json.dumps(info_helps_4_pct)
            #return json.dumps(info_helps_5)
            #return json.dumps(info_helps_5_pct))
            #return json.dumps(info_negative_1)
            #return json.dumps(info_negative_1_pct)
            #return json.dumps(info_negative_2)
            #return json.dumps(info_negative_2_pct)
            #return json.dumps(info_negative_3)
            #return json.dumps(info_negative_3_pct)
            #return json.dumps(info_negative_4)
            #return json.dumps(info_negative_4_pct)
            #return json.dumps(info_negative_5)
            #return json.dumps(info_negative_5_pct)
            #return json.dumps(info_feelings)
            #return json.dumps(info_helps)
            #return json.dumps(info_description)
            #return json.dumps(info_feelings_helps)
            #return json.dumps(info_feelings_helps_description)
    

        # All column names

        #strain	aka	type	rating	num_reviews	num_effects_reports	feeling_1	feeling_1_pct	feeling_2	feeling_2_pct	feeling_3	feeling_3_pct	feeling_4	feeling_4_pct	feeling_5	feeling_5_pct	helps_1	helps_1_pct	helps_2	helps_2_pct	helps_3	helps_3_pct	helps_4	helps_4_pct	helps_5	helps_5_pct	negative_1	negative_1_pct	negative_2	negative_2_pct	negative_3	negative_3_pct	negative_4	negative_4_pct	negative_5	negative_5_pct	description	feelings	helps	feelings_helps	feelings_helps_description

except Exception:
    pass
recommend('Relaxed Happy Euphoric Uplifted Hungry Stress Pain Anxiety Depression')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11333617383894333, 'fraction_numerical': 0.01686042972792014, 'mean_word_length': 3.8753709198813056, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '520901', 'license': 'MIT', 'n_tokens_mistral': 6034, 'n_tokens_neox': 5844, 'n_words': 1106, 'path': 'notebooks/Wrangle_Bline_NN_Pickles.ipynb', 'repo_name': 'johnjdailey/DS', 'size': '16435'}