<jupyter_start><jupyter_text>Feature engineering One of the most important aspects which influences performance of machine learning models is the features used to represent the problem. If your underlying representation is bad whatever fancy model you use is not going to help. With a better feature representation, a simple and a more interpretable model is likely to perform reasonably well. **Feature engineering** is the process of transforming raw data into features that better represent the underlying problem to the predictive models. The code below reads the data CSV.<jupyter_code>df = pd.read_csv("tweets.csv", usecols=["keyword", "text", "target", "location"])
train_df, test_df = train_test_split(df, test_size=0.2, random_state=2)
train_df.head()
X_train, y_train = train_df.drop(columns=["target"]), train_df["target"]
X_test, y_test = test_df.drop(columns=["target"]), test_df["target"]<jupyter_output><empty_output><jupyter_text>The challenge here is correctly classifying the real disaster tweets. The data set includes tweets about disasters as well as keyword that relate the disaster, such as crash, suicide bomb, and so on and also the location. The prediction problem we're attempting to tackle is determining whether a tweet is connected to a real disaster or is merely a joke/movie review in a disaster-related environment.<jupyter_code>train_df["target"].value_counts()<jupyter_output><empty_output><jupyter_text>Yes, there is a class imbalance in the given data set. As can be seen from the target value counts above, there are only 1701 tweets of genuine disaster, accounting for less than 20% of the whole data set. To cope with this, we'll need to employ a different scoring measure than accuracy, one that focuses on judging the model's performance based on actual disaster tweets.<jupyter_code>scoring = ["precision" , "f1", "recall", "roc_auc" , "average_precision"]<jupyter_output><empty_output><jupyter_text>As seen above there is a class imbalance in the given data set.The scoring metric `accuracy` cannot be used to judge the model performance. For the given use case i.e. classifying the actual disaster tweets correctly, we need to be confident that the tweet is actually a disaster, a suitable metric here would be `precision` but we also need to avoid the false negatives which means the `recall` is equally important. So in order to account for the trade off `f1` score sounds a better scoring metric for the given use case. To examine how successfully the model discriminate between the two classes, we utilize the `auc roc` score.   The location featureThe location feature seems quite messy.<jupyter_code>train_df["location"].unique()
len(train_df["location"].unique())
train_df.info()
X_train["location"].isnull().value_counts()
X_train["location"].value_counts()<jupyter_output><empty_output><jupyter_text>1) There are many null values in the feature Location i.e. 2726 which might be a challenge to impute.2) Throughout the location feature, there are countries, cities, regions, and other meaningless terms jumbled in. 3) There are a lot of instances in the feature location with special characters that aren't in an acceptable format. 4) Furthermore, the information is not standardized. 5) Because there are 3747 distinct values, applying one hot encoding is difficult. We may divide the location feature into four categories: country, city, region, and other, and then run OHE on each of them. The category "OTHER" can be allocated to null and non-meaningful words. But in given case we choose to drop the feature   Identifying feature types<jupyter_code>drop_features = ["location"]
text_feature = "text"
key_word = "keyword"

preprocessor = make_column_transformer(
    (CountVectorizer(stop_words="english"), text_feature),
    (CountVectorizer(stop_words="english"), key_word)
)


data=preprocessor.fit_transform(X_train,y_train)
data.shape<jupyter_output><empty_output><jupyter_text>Due to null values and values that will not contribute significance to the model, we are removing the feature `location.`We use the `Count Vectorizer` feature to turn the `text` feature into vectorized values for each meaningful word that the model can interpret.We use a separate `Count Vectorizer` on the `keyword` feature so that the model may take into account the most common disaster-related keywords when making predictions.   DummyClassifier<jupyter_code>results = {}
def mean_std_cross_val_scores(model, X_train, y_train, **kwargs):
    """
    Returns mean and std of cross validation

    Parameters
    ----------
    model :
        scikit-learn model
    X_train : numpy array or pandas DataFrame
        X in the training data
    y_train :
        y in the training data

    Returns
    ----------
        pandas Series with mean scores from cross_validation
    """

    scores = cross_validate(model, X_train, y_train, **kwargs)

    mean_scores = pd.DataFrame(scores).mean()
    std_scores = pd.DataFrame(scores).std()
    out_col = []

    for i in range(len(mean_scores)):
        out_col.append((f"%0.3f (+/- %0.3f)" % (mean_scores[i], std_scores[i])))

    return pd.Series(data=out_col, index=mean_scores.index)
dummy = make_pipeline(preprocessor, DummyClassifier())
results['Dummy Classifier']= mean_std_cross_val_scores(dummy, X_train, y_train, scoring =scoring)
pd.DataFrame(results)<jupyter_output>/opt/miniconda3/envs/573/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1308: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/opt/miniconda3/envs/573/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1308: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/opt/miniconda3/envs/573/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1308: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/opt/miniconda3/envs/573/lib/python3.9/site-packages/sklearn/metrics/_classification.py:[...]<jupyter_text>Logistic regression<jupyter_code>LR = make_pipeline(preprocessor, LogisticRegression(max_iter =2000))
LR
results['Logistic Regression']= mean_std_cross_val_scores(LR, X_train, y_train, scoring =scoring)
pd.DataFrame(results)<jupyter_output><empty_output><jupyter_text>Hyperparameter optimization<jupyter_code>param_grid_gamma_random = {"columntransformer__countvectorizer-1__max_features": [1000,5000,10000,15000,20000, 23000],
                           "columntransformer__countvectorizer-2__max_features": [50,100,200,219],
                           "logisticregression__C": 10.0 ** np.arange(-3, 4),
                           "logisticregression__class_weight": [None, "balanced"]}

random_search = RandomizedSearchCV(
    LR, param_distributions=param_grid_gamma_random, n_jobs=-1,scoring ="f1",random_state =123, n_iter =20
)
random_search.fit(X_train, y_train)

print("Best Hyper-Parameters are:", random_search.best_params_)
print("Best Score is:", random_search.best_score_)<jupyter_output>Best Score is: 0.6717594271881064<jupyter_text>The best hyper parameters are: {'logistic regression - class_weight': 'balanced', 'logistic regression- C': 1.0, 'Count Vectorizer-text-max_features': 50, 'Count Vectorizer-keyword-max_features': 15000} The best cross validation f1 score is 0.6717594271881064  Feature engineering<jupyter_code>import nltk
from nltk.corpus import stopwords

nltk.download("vader_lexicon")
nltk.download("punkt")
from nltk.sentiment.vader import SentimentIntensityAnalyzer

sid = SentimentIntensityAnalyzer()
def get_relative_length(text, TWITTER_ALLOWED_CHARS=280.0):
    """
    Returns the relative length of text.

    Parameters:
    ------
    text: (str)
    the input text

    Keyword arguments:
    ------
    TWITTER_ALLOWED_CHARS: (float)
    the denominator for finding relative length

    Returns:
    -------
    relative length of text: (float)

    """
    return len(text) / TWITTER_ALLOWED_CHARS


def get_length_in_words(text):
    """
    Returns the length of the text in words.

    Parameters:
    ------
    text: (str)
    the input text

    Returns:
    -------
    length of tokenized text: (int)

    """
    return len(nltk.word_tokenize(text))


def get_sentiment(text):
    """
    Returns the compound score representing the sentiment of the given text: -1 (most extreme negative) and +1 (most extreme positive)
    The compound score is a normalized score calculated by summing the valence scores of each word in the lexicon.

    Parameters:
    ------
    text: (str)
    the input text

    Returns:
    -------
    sentiment of the text: (str)
    """
    scores = sid.polarity_scores(text)
    return scores["compound"]

train_df = train_df.assign(n_words=train_df["text"].apply(get_length_in_words))
train_df = train_df.assign(vader_sentiment=train_df["text"].apply(get_sentiment))
train_df = train_df.assign(rel_char_len=train_df["text"].apply(get_relative_length))

test_df = test_df.assign(n_words=test_df["text"].apply(get_length_in_words))
test_df = test_df.assign(vader_sentiment=test_df["text"].apply(get_sentiment))
test_df = test_df.assign(rel_char_len=test_df["text"].apply(get_relative_length))
#mention_count
train_df['mention_count'] = train_df['text'].apply(lambda x: len([c for c in str(x) if c == '@']))
test_df['mention_count'] = test_df['text'].apply(lambda x: len([c for c in str(x) if c == '@']))

# punctuation_count
train_df['punctuation_count'] = train_df['text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]))
test_df['punctuation_count'] = test_df['text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]))

# unique_word_count
train_df['unique_word_count'] = train_df['text'].apply(lambda x: len(set(str(x).split())))
test_df['unique_word_count'] = test_df['text'].apply(lambda x: len(set(str(x).split())))

# stop_word_count
train_df['stop_word_count'] = train_df['text'].apply(lambda x: len([w for w in str(x).lower().split() if w in stopwords.words('english')]))
test_df['stop_word_count'] = test_df['text'].apply(lambda x: len([w for w in str(x).lower().split() if w in stopwords.words('english')]))

# url_count
train_df['url_count'] = train_df['text'].apply(lambda x: len([w for w in str(x).lower().split() if 'http' in w or 'https' in w]))
test_df['url_count'] = test_df['text'].apply(lambda x: len([w for w in str(x).lower().split() if 'http' in w or 'https' in w]))

# mean_word_length
train_df['mean_word_length'] = train_df['text'].apply(lambda x: np.mean([len(w) for w in str(x).split()]))
test_df['mean_word_length'] = test_df['text'].apply(lambda x: np.mean([len(w) for w in str(x).split()]))

# char_count
train_df['char_count'] = train_df['text'].apply(lambda x: len(str(x)))
test_df['char_count'] = test_df['text'].apply(lambda x: len(str(x)))

# hashtag_count
train_df['hashtag_count'] = train_df['text'].apply(lambda x: len([c for c in str(x) if c == '#']))
test_df['hashtag_count'] = test_df['text'].apply(lambda x: len([c for c in str(x) if c == '#']))<jupyter_output><empty_output><jupyter_text>Mention count reasoning:In the event of a crisis, people frequently mention their loved ones and other authorities in attempt to disseminate the message. Punctuation count reasoning:When there is an emergency or a calamity, people prefer to use exclamation marks and other punctuation.<jupyter_code>import nltk
from spacymoji import Emoji
import en_core_web_md  # pre-trained model
import spacy

nlp = en_core_web_md.load()
nlp.add_pipe("emoji", first=True);

def get_emoji_count(text):
    """
    Returns the count of emojis in specified text
    
    Parameters:
    ------
    text: (str)
    the input text
    
    Returns:
    ------
    count of emojis in specified text : int
    """
    doc = nlp(text)
    return len(doc._.emoji)

train_df['emoji_count']=train_df["text"].apply(get_emoji_count)
test_df['emoji_count']=test_df["text"].apply(get_emoji_count)<jupyter_output><empty_output><jupyter_text>Emoji count reasoning:When there is an emergency or a calamity, people prefer to use less emoji's while tweeting as compared to writing a joke or a movie review.<jupyter_code>train_df.head()<jupyter_output><empty_output><jupyter_text>Pipeline with all features<jupyter_code>drop_features = ['location']
text_feature = "text" 
key_word= "keyword"
target = "target"
numeric_features = list(
    set(train_df.columns)
    - set(drop_features)
    - set([text_feature])
    - set([key_word])
    -set([target])
)

preprocessor = make_column_transformer(
    (StandardScaler(), numeric_features),
    (CountVectorizer(stop_words="english", max_features= 15000), text_feature),
    (CountVectorizer(stop_words="english", max_features = 50), key_word)

)
train_df.info()
X_train, y_train = train_df.drop(columns=["target"]), train_df["target"]
X_test, y_test = test_df.drop(columns=["target"]), test_df["target"]
data= preprocessor.fit_transform(X_train, y_train)
data.shape
pipe_lr = make_pipeline(preprocessor, LogisticRegression(class_weight= 'balanced', C = 1.0, max_iter =2000))
pipe_lr
results['LR_feature-engineered']= mean_std_cross_val_scores(pipe_lr, X_train, y_train, scoring =scoring)
pd.DataFrame(results)<jupyter_output><empty_output><jupyter_text>Interpretation Yes, as observed above, there is an improvement following feature engineering, i.e. the f1 score has grown dramatically after adding the additional features. In addition, while recall has grown significantly, precision has dropped. However, the model has delivered a superior f1 score, which compensates for the precision-recall tradeoff.<jupyter_code>col_names = numeric_features + pipe_lr.named_steps["columntransformer"].named_transformers_["countvectorizer-1"].get_feature_names_out().tolist() + pipe_lr.named_steps["columntransformer"].named_transformers_["countvectorizer-2"].get_feature_names_out().tolist()

pipe_lr.fit(X_train, y_train)

lr_coeffs = pipe_lr.named_steps["logisticregression"].coef_

cefficients= pd.DataFrame(
    data=lr_coeffs.T,index= col_names, columns=["Coefficients"]
).sort_values(by="Coefficients", ascending=False)

cefficients.head(10)<jupyter_output><empty_output><jupyter_text>Yes, the coefficients match my intuitions; as can be seen above, the attributes windstorm, rescued, thunderstorm, died, and so on have the greatest coefficients; these features reflect a disaster.  Tree Based Model<jupyter_code>from catboost import CatBoostClassifier
pipe_catboost_all = make_pipeline(
    preprocessor, CatBoostClassifier(random_state=123, verbose = 0)
)
results['CatBoost_feature-engineered']= mean_std_cross_val_scores(pipe_catboost_all, X_train, y_train, scoring =scoring)
pd.DataFrame(results)<jupyter_output><empty_output><jupyter_text>Test results<jupyter_code>from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score, roc_auc_score,average_precision_score

print('F1 Score')
print(f1_score(y_test, pipe_lr.predict(X_test)))

print('Precision Score')
print(precision_score(y_test, pipe_lr.predict(X_test)))

print('Recall Score')
print(recall_score(y_test, pipe_lr.predict(X_test)))

print('ROC AUC Score')
print(roc_auc_score(y_test, pipe_lr.predict(X_test)))

print('Average Precision Score')
print(average_precision_score(y_test, pipe_lr.predict(X_test)))<jupyter_output>F1 Score
0.7031431897555296
Precision Score
0.6771300448430493
Recall Score
0.7312348668280871
ROC AUC Score
0.8269285564661661
Average Precision Score
0.5439537630737555
{'fraction_non_alphanumeric': 0.11047246083880748, 'fraction_numerical': 0.018254168772107126, 'mean_word_length': 5.6191471571906355, 'pattern_counts': {'":': 4, '<': 58, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '377254', 'license': 'MIT', 'n_tokens_mistral': 5182, 'n_tokens_neox': 4899, 'n_words': 1413, 'path': 'Disaster.ipynb', 'repo_name': 'VaradrajPoojari/Disaster_tweet', 'size': '15832'}