<jupyter_start><jupyter_text>Logistic Regression  Task 1Recall the data processing routines from the last lab course. The following excercises build on top of the extracted feature representations, but instead of the prebuilt classifier, we want to implement logistic regression by hand. To this end, make sure, that the variables `train`, `test`, `train_data_features` and `test_data_features` are loaded to your IPython shell.<jupyter_code>import pandas as pd
from bs4 import BeautifulSoup
import re
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import CountVectorizer

#%% download nltk stopwords
# import nltk
# ntlk.download('stopwords')

# load stopwords
stops = set(stopwords.words('english'))

# function for preprocessing the data
def review_prepro(data, remove_stopwords=False):
    # remove HTML tags
    review_text = BeautifulSoup(data, 'lxml').get_text()
    # remove non-letters and numbers
    letters_only = re.sub( '[^a-zA-Z]',
                          ' ',
                          review_text )
    # make all characters lower case and split the documents into single words
    words = letters_only.lower().split()
    
    if remove_stopwords:
        # remove stop words
        meaningful_words = [ w for w in words if not w in stops ]
        # return concatenated single string
        return ' '.join(meaningful_words)
    else:
        # or don't and concatenate to single string
        return ' '.join(words)

# load data as pandas dataframe
train = pd.read_csv('labeledTrainData.tsv', 
                    header=0,
                    delimiter="\t", 
                    quoting=3 )

test = pd.read_csv('labeledTestData.tsv', 
                   header=0,
                   delimiter="\t",
                   quoting=3 )


# preprocess train and test data
num_reviews = train['review'].size

clean_train_reviews = []
for i in range(num_reviews):
  #  if (i+1)%1000 == 0:
    #    print('Review {} of {}\n'.format(i+1, num_reviews))
    clean_train_reviews.append( review_prepro(train['review'][i], remove_stopwords=True) )
    
num_test_reviews = test['review'].size

clean_test_reviews = []
for i in range(num_test_reviews):
    #if (i+1)%1000 == 0:
     #   print('Review {} of {}\n'.format(i+1, num_test_reviews))
    clean_test_reviews.append( review_prepro(test['review'][i], remove_stopwords=True) )
    

#%% create BoW
# Documentation:
# http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
vectorizer = CountVectorizer(analyzer = 'word',
                             tokenizer = None,
                             preprocessor = None,
                             stop_words = stops,
                             max_features = 5000)

# fit the vectorizer and return transformed reviews
vectorizer.fit(clean_train_reviews)
train_data_features = vectorizer.transform(clean_train_reviews)
clean_train_reviews=None 
# convert to numpy array
train_data_features = train_data_features.toarray()

# create BoW representation of test data
test_data_features = vectorizer.transform(clean_test_reviews)
clean_test_reviews=None
test_data_features = test_data_features.toarray()<jupyter_output><empty_output><jupyter_text>a) Write a PYTHON function `logistic_gradient` that expects a training set matrix `X_train`, a ground truth label vector `y_train` and a current weight vector `w` as its input and returns the gradient `g` of the negative log-likelihood function of the logistic regression. Refer to the lecture notes for the mathematical definition.<jupyter_code>import numpy as np

def sigmoid(x):
    # https://timvieira.github.io/blog/post/2014/02/11/exp-normalize-trick/
    z = np.exp(-np.abs(x))
    return np.where(x>=0.0,1.0/(1.0+z),z/(1.0+z))

def logistic_gradient(X_train,y_train,w,reg=0.0):
    # Rows are variables, Columns are samples
    g=np.dot(X_train,sigmoid(np.dot(X_train.T, w))-y_train)+reg*w
    return g<jupyter_output><empty_output><jupyter_text>b )Write a PYTHON function `logistic_hessian` that expects a training set matrix `X_train` and a current weight vector `w` as its input and returns the Hessian `H` of the negative log-likelihood function of the logistic regression. Refer to the lecture notes for the mathematical definition.<jupyter_code>def logistic_hessian(X_train,w, reg=0.0):
    S=sigmoid(np.dot(X_train.T,w))
    diagB=S*(1.0-S)
    XS=X_train*np.expand_dims(diagB,axis=0)
    H=np.dot(XS, X_train.T)+reg*np.eye(w.shape[0])
    return H<jupyter_output><empty_output><jupyter_text>c) Write a PYTHON function `find_w` that expects a training set matrix `X_train`, a ground truth label vector `y_train`, a fixed step size `h` and a maximum iteration number `max_it` that determines the optimal logistic regression weight vector `w_star` by performing gradient descent via calling `logistic_gradient` in each iteration. Make sure to include the affine offset $w_0$ in your model.<jupyter_code>def find_w(X_train, y_train, max_it, reg=0.0,w_init=None):
    X_offs=np.ones((X_train.shape[0]+1,X_train.shape[1]))
    X_offs[1:,:]=X_train
    w=np.zeros((X_offs.shape[0]))
    if w_init is not None:
        w=w_init
    cost=np.inf
    for i in range(max_it):
        delta_w=np.linalg.lstsq(logistic_hessian(X_offs, w,reg=reg),logistic_gradient(X_offs, y_train, w,reg=reg))[0]
        w-=np.array(delta_w)
        S=sigmoid(np.dot(w.T,X_offs))
        oldcost=cost
        cost=-(np.dot(np.log(np.where(S<np.finfo('float32').eps,np.finfo('float32').eps,S)),y_train)
              +np.dot(np.log(np.where((1.0-S)<np.finfo('float32').eps,np.finfo('float32').eps,1.0-S)),1.0-y_train)
              -reg*np.sum(w**2))
        if cost>=oldcost:
            break
        w_star=w
        print('Iteration:', i+1, ' Current cost:', cost)
    return w_star<jupyter_output><empty_output><jupyter_text>d) Write a function `classify_log` that expects a weight vector `w` and a test set matrix `X_test` and classifies the samples in `X_test` via logistic regression, returning a label vector `y_test`. Test your implementation on `train_data_features` and `test_data_features` with one iteration and with 10 iterations. What do you observe?<jupyter_code>from sklearn.metrics import roc_auc_score as AUC

def classify_log(w, X_test):
    w0=w[0]
    y_test=sigmoid(w0+np.dot(X_test.T,w[1:]))
    return y_test

#Testing implementation
y_train=train['sentiment'].values
y_test=test['sentiment'].values
train=None
test=None
w=find_w(train_data_features.T,y_train,1,reg=0)
y_pred=classify_log(w,test_data_features.T)
auc = AUC( y_test, y_pred )
print('AUC score after 1 iteration:',auc)
w=find_w(train_data_features.T,y_train,10,reg=0)
y_pred=classify_log(w,test_data_features.T)
auc = AUC( y_test, y_pred )
print('AUC score after 10 iteration:',auc)<jupyter_output>Iteration: 1  Current cost: 6165.48143183
AUC score after 1 iteration: 0.906356178126
Iteration: 1  Current cost: 6165.48143183
Iteration: 2  Current cost: 3961.1406912
Iteration: 3  Current cost: 2634.09296401
Iteration: 4  Current cost: 1738.31616292
Iteration: 5  Current cost: 1054.12358097
Iteration: 6  Current cost: 514.98272427
Iteration: 7  Current cost: 196.676386247
Iteration: 8  Current cost: 71.7497222914
Iteration: 9  Current cost: 26.299656854
Iteration: 10  Current cost: 9.67265985876
AUC score after 10 iteration: 0.875910041024<jupyter_text>e) Logistic regression is prone to *overfitting*. To prevent this, regularizing parameters can be used. Adjust your implementation in such a way that instead of minimizing $L(\mathbf{w})$, it minimizes the term    	\begin{equation}    	L(\mathbf{w})+\alpha\|\mathbf{w}\|^2,    	\end{equation}    	where $\alpha$ is a non-negative regularization parameter. Test your implementation with $\alpha=1$ with one iteration and with 10 iterations.<jupyter_code>w=find_w(train_data_features.T,y_train,1,reg=1)
y_pred=classify_log(w,test_data_features.T)
auc = AUC( y_test, y_pred )
print('AUC score after 1 iteration:',auc)
w=find_w(train_data_features.T,y_train,10,reg=1)
y_pred=classify_log(w,test_data_features.T)
auc = AUC( y_test, y_pred )
print('AUC score after 10 iteration:',auc)<jupyter_output>Iteration: 1  Current cost: 6304.09530912
AUC score after 1 iteration: 0.910314236004
Iteration: 1  Current cost: 6304.09530912
Iteration: 2  Current cost: 4330.85401281
Iteration: 3  Current cost: 3437.94965462
Iteration: 4  Current cost: 3205.4784196
AUC score after 10 iteration: 0.92835827158
{'fraction_non_alphanumeric': 0.10706258133207146, 'fraction_numerical': 0.04270673133798651, 'mean_word_length': 4.139209726443769, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '597284', 'license': 'Apache-2.0', 'n_tokens_mistral': 3101, 'n_tokens_neox': 2815, 'n_words': 735, 'path': 'information-retrieval-exercises/solutions/lab04_solution.ipynb', 'repo_name': 'achmart/inforet', 'size': '8453'}