<jupyter_start><jupyter_text>1. Prepare and clean data<jupyter_code>os.chdir(r'D:\NDSUGoogle\Pyntebook\nlp analysis\LasVegasResult')
data=pd.read_csv("complete_data.csv")
data.head()
dummies=data['season'].str.get_dummies()
dummies.head()
seasondata=pd.concat([data,dummies],axis=1).drop(['Unnamed: 0'],axis=1)
seasondata.head()
newdata=data.drop(['season','Unnamed: 0','year','month'],axis=1)
newdata.head()<jupyter_output><empty_output><jupyter_text>change 12345 to pos,neg<jupyter_code>## recode rate to 3 categories
newdata.loc[data['rate']=="A", 'rate2']=1
newdata.loc[data['rate']=="B", 'rate2']=1
newdata.loc[data['rate']=="C", 'rate2']=0
newdata.loc[data['rate']=="D", 'rate2']=0
newdata.loc[data['rate']=="E", 'rate2']=0
data2=newdata.drop(['rate'],axis=1)
data2.head()
## 五个精度低，特别少。。。  我们在意的是positive或者negtive。。而不是very positive或者very negtive。。
## sum the weight for pos or neg, to estimate the direction of each topic
topicDirection=data2.groupby(['rate2']).mean()
topicDirection.to_csv('topicDirection.csv')<jupyter_output><empty_output><jupyter_text>2. try random forest to explore data pre-processing<jupyter_code>label.describe()<jupyter_output><empty_output><jupyter_text>set value <0.01 to 0 and use smote. OA decrease, but recall increase further<jupyter_code>### smote通过增加 线性组合的sample把neg和postive的样本量变得差不多。。
##因为我们的数据大部分都是positive的。。
features=data2.drop(['rate2'],axis=1)
#features[features<0.02]=0
label=data2.rate2
X_train, X_test, y_train, y_test = train_test_split(features, label, test_size=0.05, random_state=333)
sm = SMOTE()
nm=NearMiss() #near missing is undersampling, which will reduce the number of training set
X_res, y_res = sm.fit_sample(X_train, y_train)

clf = RandomForestClassifier(random_state=42)
clf_fit=clf.fit(X_res, y_res)
y_predict=clf_fit.predict(X_test)
#evaluate y_predict performance
from sklearn.metrics import classification_report, confusion_matrix
print("=== Confusion Matrix ===")
print(confusion_matrix(y_test, y_predict))
print('\n')
print("=== Classification Report ===")
print(classification_report(y_test, y_predict))
print('\n')

importance=clf.feature_importances_
print(importance)
## this is the most important feature name
##用这个看重要不重要
largest5=importance.argsort()[-5:]
[list(features)[i] for i in largest5]<jupyter_output>[0.02195538 0.02076753 0.02520255 0.018688   0.04486941 0.02220212
 0.02608338 0.02200368 0.02108351 0.0200318  0.0253033  0.03793029
 0.09652744 0.03029267 0.04490259 0.02084376 0.02310686 0.02283947
 0.03294735 0.02228456 0.02341881 0.02013446 0.02114298 0.06590224
 0.02040148 0.07944753 0.02277688 0.07237289 0.02091537 0.05362171]<jupyter_text>try use logistic model<jupyter_code>from sklearn.linear_model import LogisticRegression

features=data2.drop(['rate2'],axis=1) 
features[features<0.1]=0
label=data2.rate2
X_train, X_test, y_train, y_test = train_test_split(features, label, test_size=0.05, random_state=4)
logmodel=LogisticRegression( class_weight='balanced')# the balanced option will assign inversed weights 
log_fit=logmodel.fit(X_train, y_train)
y_predict=log_fit.predict(X_test)

#evluate the model
from sklearn.metrics import classification_report
print (classification_report(y_test,y_predict))

#print coefficient
coef=logmodel.coef_
print (coef)

print (list(coef[0]))
from sklearn.metrics import r2_score
r2_score(y_test,y_predict)#the result is negative, meaning rsquare doesnot fit for classification problems.<jupyter_output>[0.9960385897362978, 0.9799785685122737, 0.3210941533912557, 1.7847402165371933, 1.2959109128427928, -0.46294696666005325, -0.3457890565022754, -0.2177421407496199, -0.10085974450332809, 0.8677688609411798, -0.8505214524038588, 0.15558912933759914, 1.350712319416233, 0.3452908204517974, -1.0226863227311669, 0.7475552764708149, 0.9042699550604921, 2.302125769109948, 0.4528453963124849, 0.8573563083076265, 0.5138099734852941, -0.07094892301320305, -0.47728943847950833, -1.0401361205380535, 2.561573775347796, 4.947901643580021, 2.2368421230283246, -2.095895345153303, 0.44532094384309723, 3.8400847783049126]<jupyter_text>3. Implement decision tree following C.1 for interpretation purpose<jupyter_code>features=data2.drop(['rate2'],axis=1)
features[features<0.01]=0
label=data2.rate2
features.head()
X_train, X_test, y_train, y_test = train_test_split(features, label, test_size=0.2, random_state=42)
sm = SMOTE(random_state=42)
X_res, y_res = sm.fit_sample(X_train, y_train)

clf=tree.DecisionTreeClassifier(max_depth=4)  ## this max_depth is after trials among [3,4,5]
clf=clf.fit(X_res, y_res)
y_predict=clf.predict(X_test)
print(accuracy_score(y_test, y_predict))
print(recall_score(y_test, y_predict, average=None))
print(precision_score(y_test, y_predict, average=None))
with open("decisionTree_classifier.txt", "w") as f:
    f = tree.export_graphviz(clf, out_file=f,feature_names=list(features),class_names=["pos",'neg'])
#Use http://webgraphviz.com/ to see the tree graphic
print (list(features))
"""the result gini impurity look at how well the group been splitted. If all the group are Pos, then GINI is 0. If the pos and neg 
is 50%,50%, then the gini is 0.5, which is not good.48.97% means the 48.97% chance of a new data point being incorrectly 
classified, based on the observed training data we have at our disposal.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13176248821866163, 'fraction_numerical': 0.16588124410933083, 'mean_word_length': 8.132530120481928, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '183617', 'license': 'MIT', 'n_tokens_mistral': 2591, 'n_tokens_neox': 2095, 'n_words': 390, 'path': 'LasVegasResult/topicPrediction.ipynb', 'repo_name': 'theansweryang/nlp-analysis', 'size': '5305'}