<jupyter_start><jupyter_text>Lambda School Data Science Unit 4 Sprint Challenge 4 RNNs, CNNs, AutoML, and more...In this sprint challenge, you'll explore some of the cutting edge of Data Science.*Caution* - these approaches can be pretty heavy computationally. All problems were designed so that you should be able to achieve results within at most 5-10 minutes of runtime on Colab or a comparable environment. If something is running longer, doublecheck your approach!  Part 1 - RNNsUse an RNN to fit a simple classification model on tweets to distinguish from tweets from <NAME> and tweets from <NAME>.Following is code to scrape the needed data (no API auth needed, uses [twitterscraper](https://github.com/taspinar/twitterscraper)):<jupyter_code>!pip install category_encoders
import imageio
import matplotlib.pyplot as plt
from skimage import color
from skimage.exposure import rescale_intensity
import numpy as np

from keras.applications.resnet50 import ResNet50
from keras.preprocessing import image
from keras.applications.resnet50 import preprocess_input, decode_predictions
import category_encoders as ce
from sklearn.pipeline import make_pipeline, Pipeline
import pandas as pd
from datetime import datetime
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, roc_curve
from sklearn.metrics import mean_squared_error, r2_score
import pandas as pd
from sklearn.model_selection import train_test_split
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from mpl_toolkits.mplot3d import Axes3D
from datetime import datetime
import math
from scipy.stats import mode
from sklearn.preprocessing import StandardScaler
from mpl_toolkits.mplot3d import Axes3D
import seaborn as sns
import statsmodels.api as s
import time
from functools import reduce
import regex
import category_encoders as ce
from sklearn.pipeline import make_pipeline, Pipeline
import pandas as pd


from __future__ import print_function

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding
from keras.layers import LSTM
!pip install twitterscraper
from twitterscraper import query_tweets

austen_tweets = query_tweets('from:austen', 1000)
len(austen_tweets)
austen_tweets[0].text
al_tweets = query_tweets('from:AlYankovic', 1000)
len(al_tweets)
al_tweets[0].text
len(austen_tweets + al_tweets)
type(austen_tweets)
max_length = 100
austin_text = ' '.join([t.text for t in austen_tweets])
al_text = ' '.join([t.text for t in al_tweets])

text = austin_text + al_text

  
chars = list(set(text)) # split and remove duplicate characters. convert to list
# one hot encode
char_to_int = dict((c, i) for i, c in enumerate(chars)) # "enumerate" retruns index and value. Convert it to dictionary
int_to_char = dict((i, c) for i, c in enumerate(chars))

# max_length = 100
#    [t.text for t in austen_tweets])

# integer_encoded_austin = [char_to_int[i] for i in austin_text]
# integer_encoded_al = [char_to_int[i] for i in al_text] 

def text_to_nums(text,austen):
  d = {'y': austen}
  for i in range(max_length):
    d[f"{i}"] = char_to_int[text[i]] if i < len(text) else -1
  return d 
  
a = [text_to_nums(t.text, 0) for t in austen_tweets]
a.extend([text_to_nums(t.text, 1) for t in al_tweets])
df = pd.DataFrame(a)
df.head()
X = df.drop('y', axis=1)
y = df['y']
X_train, X_test, y_train, y_test = train_test_split(
    X, y.values, train_size=0.75, test_size=0.25)
max_features = max_length
model = Sequential()
model.add(Embedding(max_features, 128))
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))
batch_size = 32
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

print('Train...')
model.fit(X_train, y_train,
          batch_size=batch_size,
          epochs=15,
          validation_data=(X_test, y_test))
score, acc = model.evaluate(X_test, y_test,
                            batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)<jupyter_output>Train...
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Train on 855 samples, validate on 286 samples
Epoch 1/15
855/855 [==============================] - 41s 47ms/step - loss: 0.5397 - acc: 0.8222 - val_loss: 0.4473 - val_acc: 0.8357
Epoch 2/15
855/855 [==============================] - 6s 7ms/step - loss: 0.4329 - acc: 0.8433 - val_loss: 0.4287 - val_acc: 0.8357
Epoch 3/15
855/855 [==============================] - 6s 7ms/step - loss: 0.4179 - acc: 0.8433 - val_loss: 0.4354 - val_acc: 0.8357
Epoch 4/15
855/855 [==============================] - 6s 7ms/step - loss: 0.4158 - acc: 0.8433 - val_loss: 0.4245 - val_acc: 0.8357
Epoch 5/15
855/855 [==============================] - 6s 7ms/step - loss: 0.3868 - acc: 0.8444 - val_loss: 0.3786 - val_acc: 0.8322
Epoch 6/15
855/855 [==========[...]<jupyter_text>Your tasks:- Encode the characters to a sequence of integers for the model- Get the data into the appropriate shape/format, including labels and a train/test split- Use Keras to fit a predictive model, classifying tweets as being from Austen versus Weird Al- Report your overall score and accuracyFor reference, the [Keras IMDB sentiment classification example](https://github.com/keras-team/keras/blob/master/examples/imdb_lstm.py) will be useful, as well the RNN code we used in class.*Note* - focus on getting a running model, not on maxing accuracy with extreme data size or epoch numbers. Only revisit and push accuracy if you get everything else done!<jupyter_code>"""Prediction of Users based on Tweet embeddings."""
import numpy as np
from sklearn.linear_model import LogisticRegression
from .models import User
from .twitter import BASILICA


def predict_user(user1_name, user2_name, tweet_text):
    """Determine and return which user is more likely to say a given Tweet."""
    user1 = User.query.filter(User.name == user1_name).one()
    user2 = User.query.filter(User.name == user2_name).one()
    user1_embeddings = np.array([tweet.embedding for tweet in user1.tweets])
    user2_embeddings = np.array([tweet.embedding for tweet in user2.tweets])
    embeddings = np.vstack([user1_embeddings, user2_embeddings])
    labels = np.concatenate([np.ones(len(user1.tweets)),
                             np.zeros(len(user2.tweets))])
    log_reg = LogisticRegression().fit(embeddings, labels)
    tweet_embedding = BASILICA.embed_sentence(tweet_text, model='twitter')
    return log_reg.predict(np.array(tweet_embedding).reshape(1, -1))
  
 
predict_user('<NAME>', '<NAME>',
model = Sequential()
model.add(Embedding(max_features, 128))
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))<jupyter_output><empty_output><jupyter_text>Conclusion - RNN runs, and gives pretty decent improvement over a naive "It's Al!" model. To *really* improve the model, more playing with parameters, and just getting more data (particularly Austen tweets), would help. Also - RNN may well not be the best approach here, but it is at least a valid one.  Part 2- CNNsTime to play "find the frog!" Use Keras and ResNet50 to detect which of the following images contain frogs:<jupyter_code>!pip install google_images_download
from google_images_download import google_images_download

response = google_images_download.googleimagesdownload()
arguments = {"keywords": "animal pond", "limit": 15, "print_urls": True}
absolute_image_paths = response.download(arguments)<jupyter_output>Item no.: 1 --> Item name = animal pond
Evaluating...
Starting Download...
Image URL: https://www.enchantedlearning.com/pgifs/Pondanimals.GIF
Completed Image ====> 1. pondanimals.gif
Image URL: https://i.ytimg.com/vi/NCbu0TND9vE/hqdefault.jpg
Completed Image ====> 2. hqdefault.jpg
Image URL: https://pklifescience.com/staticfiles/articles/images/PKLS4116_inline.png
Completed Image ====> 3. pkls4116_inline.png
Image URL: https://vetstreet-brightspot.s3.amazonaws.com/8d/ac/377fecad46d8820697c26efacc32/koi-pond-thinkstock-153560141-335sm61313.jpg
Completed Image ====> 4. koi-pond-thinkstock-153560141-335sm61313.jpg
Image URL: https://pixnio.com/free-images/fauna-animals/reptiles-and-amphibians/alligators-and-crocodiles-pictures/alligator-animal-on-pond.jpg
Completed Image ====> 5. alligator-animal-on-pond.jpg
Image URL: https://www.nwf.org/-/media/NEW-WEBSITE/Programs/Garden-for-Wildlife/amphibian_bronze-frog_Julia-Bartosh_400x267.ashx
Completed Image ====> 6. amphibian_bronze-frog_julia-[...]<jupyter_text>At time of writing at least a few do, but since the Internet changes - it is possible your 5 won't. You can easily verify yourself, and (once you have working code) increase the number of images you pull to be more sure of getting a frog. Your goal is to validly run ResNet50 on the input images - don't worry about tuning or improving the model.*Hint* - ResNet 50 doesn't just return "frog". The three labels it has for frogs are: `bullfrog, tree frog, tailed frog`*Stretch goal* - also check for fish.<jupyter_code>absolute_image_paths

def process_img_path(img_path):
  return image.load_img(img_path, target_size=(224, 224))

def img_contains_frog(img):
  x = image.img_to_array(img)
  x = np.expand_dims(x, axis=0)
  x = preprocess_input(x)
  model = ResNet50(weights='imagenet')
  features = model.predict(x)
  results = decode_predictions(features, top=3)[0]
  print(results)
  for entry in results:
    if entry[1] in ['bullfrog', 'tree frog', 'tailed frog']:
      return entry[2]
  return 0.0
for p in absolute_image_paths['animal pond']:
    if img_contains_frog(process_img_path(p)) != 0.0:
      print(f"image {p} contains frog")<jupyter_output>[('n03598930', 'jigsaw_puzzle', 0.8680313), ('n06359193', 'web_site', 0.06410024), ('n02834397', 'bib', 0.021264324)]
[('n01443537', 'goldfish', 0.8495859), ('n01631663', 'eft', 0.06760218), ('n02536864', 'coho', 0.035163548)]
[('n04243546', 'slot', 0.8712449), ('n04476259', 'tray', 0.04993588), ('n03908618', 'pencil_box', 0.023072386)]
[('n01443537', 'goldfish', 0.98815376), ('n09256479', 'coral_reef', 0.006681344), ('n12985857', 'coral_fungus', 0.00260608)]
[('n01698640', 'American_alligator', 0.96394104), ('n01697457', 'African_crocodile', 0.026759902), ('n01737021', 'water_snake', 0.005964664)]
[('n01641577', 'bullfrog', 0.95048445), ('n01644900', 'tailed_frog', 0.04144713), ('n01667114', 'mud_turtle', 0.0026259099)]
image /content/downloads/animal pond/6. amphibian_bronze-frog_julia-bartosh_400x267.ash contains frog
[('n01860187', 'black_swan', 0.8796092), ('n01847000', 'drake', 0.033985015), ('n01855032', 'red-breasted_merganser', 0.028971087)]
[('n01641577', 'bullfrog', 0.922330[...]<jupyter_text>Part 3 - AutoMLUse [TPOT](https://github.com/EpistasisLab/tpot) to fit a predictive model for the King County housing data, with `price` as the target output variable.<jupyter_code>!pip install tpot
from tpot import TPOTRegressor
!wget https://raw.githubusercontent.com/ryanleeallred/datasets/master/kc_house_data.csv
!head kc_house_data.csv
!ls<jupyter_output>downloads  kc_house_data.csv  sample_data<jupyter_text>As with previous questions, your goal is to run TPOT and successfully run and report error at the end.  Also, in the interest of time, feel free to choose small `generation=1` and `population_size=10` parameters so your pipeline runs efficiently and you are able to iterate and test.*Hint* - you'll have to drop and/or type coerce at least a few variables to get things working. It's fine to err on the side of dropping to get things running, as long as you still get a valid model with reasonable predictive power.<jupyter_code>df = pd.read_csv('kc_house_data.csv')
df.head()
df.dtypes
df['date_timestamp'] = df.date. \
  apply(lambda d: datetime.strptime(d, "%Y%m%dT000000").timestamp() if str(type(d)) == "<class 'str'>" else 0)
X = df.drop(['date','price'], axis=1)
y = df.price
X_train, X_test, y_train, y_test = train_test_split(
    X, y.values, train_size=0.75, test_size=0.25)
%%time

tpot = TPOTRegressor(generations=5, population_size=20, verbosity=2)
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))
py = tpot.predict(X_test)
mean_squared_error(y_test,py), r2_score(y_test,py)
# encoders = Pipeline([
#                 ('category', ce.OrdinalEncoder(cols=['genre_top'])),
#                 ('binary', ce.BinaryEncoder(cols=['name']))
# #                 ('onehot', ce.OneHotEncoder(use_cat_names=True,cols=['type']))
#             ])
# df_ = encoders.fit_transform(dfbc)
# df_.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12609170305676856, 'fraction_numerical': 0.06043356207111666, 'mean_word_length': 5.1129647283126785, 'pattern_counts': {'":': 3, '<': 27, '<?xml version=': 0, '>': 33, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '65238', 'license': 'MIT', 'n_tokens_mistral': 4947, 'n_tokens_neox': 4348, 'n_words': 1240, 'path': 'LS_DS_Unit_4_Sprint_Challenge_4.ipynb', 'repo_name': 'invegat/DS-Unit-4-Sprint-4-Deep-Learning', 'size': '12860'}