<jupyter_start><jupyter_text>using sample dataset since svm takes quit a long time<jupyter_code>negative = final[final['Score'] == 0][['Score', 'Date', 'CleanedText']]
positive = final[final['Score'] == 1][['Score', 'Date', 'CleanedText']]

data = pd.concat([negative.iloc[:5, :], positive.iloc[:5, :]], axis = 0, ignore_index = True)
print(data)<jupyter_output>Score                 Date  \
0      0  2012-09-07 00:00:00   
1      0  2011-06-13 00:00:00   
2      0  2012-06-13 00:00:00   
3      0  2012-09-20 00:00:00   
4      0  2012-03-25 00:00:00   
5      1  2011-04-27 00:00:00   
6      1  2008-08-18 00:00:00   
7      1  2012-10-21 00:00:00   
8      1  2012-07-12 00:00:00   
9      1  2012-06-20 00:00:00   

                                         CleanedText  
0  product arriv label jumbo salt peanut peanut a...  
1  look secret ingredi robitussin believ found go...  
2  cat happili eat felida platinum two year got n...  
3  love eat good watch look movi sweet like trans...  
4        candi red flavor plan chewi would never buy  
5  bought sever vital can dog food product found ...  
6  confect around centuri light pillowi citrus ge...  
7  great taffi great price wide assort yummi taff...  
8  got wild hair taffi order five pound bag taffi...  
9  saltwat taffi great flavor soft chewi candi in...<jupyter_text>Sorting data based on Date<jupyter_code>data = final[['Score', 'Date', 'CleanedText']].copy()
data.sort_values('Date', inplace = True)
data.head(10)<jupyter_output><empty_output><jupyter_text>Unbalanced dataset<jupyter_code>#can be balanced by upsampling the negative data points

data['Score'].value_counts()
print(final)<jupyter_output>Id   ProductId          UserId                      ProfileName  \
0            1  B001E4KFG0  A3SGXH7AUHU8GW                       delmartian   
1            2  B00813GRG4  A1D87F6ZCVE5NK                           dll pa   
2            3  B000LQOCH0   ABXLMWJIXXAIN  <NAME> "<NAME>"   
3            4  B000UA0QIQ  A395BORC6FGVXV                             Karl   
4            5  B006K2ZZ7K  A1UQRSCLF8GW1T    <NAME> "<NAME>"   
...        ...         ...             ...                              ...   
363852  568450  B001EO7N10  A28KG5XORO54AY                 <NAME>   
363853  568451  B003S1WTCU  A3I8AFVPEE8KI5                        <NAME>   
363854  568452  B004I613EE  A121AA1GQV751Z                    pksd "pk_007"   
363855  568453  B004I613EE   A3IBEVCTXKNOH          <NAME> "katwel"   
363856  568454  B001LR2CU2  A3LGQPJCZVL9UC                         srfell17   

        HelpfulnessNumerator  HelpfulnessDenominato[...]<jupyter_text>Deleting final variable in order to empty some space<jupyter_code>del final<jupyter_output><empty_output><jupyter_text>Seperating the target variable from the features<jupyter_code>X= data['CleanedText']
y= data['Score']<jupyter_output><empty_output><jupyter_text>Splitting the training and testing data<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X[:8000], y[:8000], test_size = 0.3, shuffle = False)
X_train.shape, y_train.shape, X_test.shape, y_test.shape
X_train.values<jupyter_output><empty_output><jupyter_text>Applying Count BOW text preprocessing technique on training data<jupyter_code>bow = CountVectorizer()
X_train = bow.fit_transform(X_train.values)
X_train.shape<jupyter_output><empty_output><jupyter_text>Applying BOW vectorizer text preprocessing technique on test data<jupyter_code>X_test = bow.transform(X_test.values)
X_test.shape<jupyter_output><empty_output><jupyter_text>Standardizing feature by removing the mean and scaling to unit variance<jupyter_code>s = StandardScaler(with_mean = False)

X_train= s.fit_transform(X_train)

X_test = s.transform(X_test)<jupyter_output><empty_output><jupyter_text>Performing cross validation to find the best hyperparameters(c,gamma), kernels and find accuracy on training data<jupyter_code>start = time.time()
clf = SVC()
param_grid = [{'kernel' : ['rbf'], 'gamma': [50, 5, 10, 0.5],
               'C' :[10,0.1, 0.001]} ]

gsv = GridSearchCV(clf, param_grid, cv=5, n_jobs = -1)

gsv.fit(X_train, y_train)
end = time.time()
print("Time required = "+str(end-start) +"s")

print("Best HyperParameter: ",gsv.best_params_)
print("Best Accuracy: %.2f%%"%(gsv.best_score_*100))<jupyter_output>Best HyperParameter:  {'C': 10, 'gamma': 50, 'kernel': 'rbf'}
Best Accuracy: 88.34%<jupyter_text>applying model on testing data and finding accuracy on it<jupyter_code>clf = SVC(C= 10, gamma= 50)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
acc=accuracy_score(y_test, y_pred) * 100
print("Accuracy =", acc)
confusion_matrix(y_test, y_pred)<jupyter_output>Accuracy = 89.41666666666667
{'fraction_non_alphanumeric': 0.10670926517571885, 'fraction_numerical': 0.08498402555910543, 'mean_word_length': 3.126537785588752, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '372764', 'license': 'Apache-2.0', 'n_tokens_mistral': 1951, 'n_tokens_neox': 1690, 'n_words': 400, 'path': 'Projects/SVMonamazonfinefoodreview.ipynb', 'repo_name': 'Lonakshi/Data-Analysis', 'size': '4746'}