<jupyter_start><jupyter_text>Today's Coaches- [<NAME>](https://github.com/moskytw)- [<NAME>](https://github.com/crazyplum) The Code and Data- The code and the minimal data → https://github.com/moskytw/mining-news- The full data → ask from coaches. Machine Learning ProcessThis notebook will base on the process:1. Define Problem2. Prepare Data3. Train and Evaluate4. Deployment Define Problem究竟一篇新聞在 PTT 會不會被噓呢？↑ 太難用英文寫了，哈哈哈哈。We will keep using English in this notebook, since most of the data mining documentations are written in English. You have to be used to it. 😆 PTT Terms in English- 文章: article- 標題: title- 推文: push- 推: push score +1- 噓: push score -1- 被噓: sum(push scores) <= -1 Choosing Model(s)We are categorizing the articles. Simply pick some from the [cheat sheet](http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html). Prepare DataWe prepared for you. Here we will let you know the scripts and the flow. Crawl and PreprocessDownload the HTML, and process them into JSON (∵ more machine-friendly). 1. ptt_crawling.py(PTT web) → `cache/*.html` 2. ptt_preproc_to_json.py`cache/*.html` → `preprocessed/*.json` 3. ptt_preproc_filtering.pyIt removes articles whcih are out-of-range in `preprocessed/`. It configured to keep the articles during 2016/7–2016/11 (5 months). 4. ptt_preproc_sampling.pyIt keeps 10,000 JSONs in `preprocessed/` for faster experiment. 5. ptt_preproc_to_target.py`preprocessed/*.json` → `targets/*.txt`Each text file has a corresponding push score sum. 6. ptt_corpus_tokenizer.py- tokenize: `preprocessed/*.json` → `corpus.txt`- `targets/*.txt` → `targets.txt`The `corpus.txt` will be the input, and `target.txt` will be the expected output.Now we only take the title to tokenize. TokenizingWe have used the libs to tokenize.1. [jieba](https://github.com/fxsjy/jieba) and its [tranditional Chinese dictionary](https://github.com/fxsjy/jieba其他词典)2. [Zhon](https://github.com/tsroten/zhon) Command Line Usage Example    python3 ptt_corpus_tokenizer.py \        --preprocessed-dir preprocessed/ \        --targets-dir targets/ \        --corpus-output corpus.txt \        --target-output target.txt  Explore DataNow we get the well-formatted data. Before we start training a model, take a look to know them better.Here are two important theories to work with text: tf-idfIt's term frequency–inverse document frequency.- tf: term frequency = (a term count in a doc) / (all terms count in the doc)- df: document frequency = (docs count having the term) / (all docs count)- idf: inverse document frequency = log(1 / df)- tf-idf = tf × idf n-gramAn n-gram is a contiguous sequence of n items from a given sequence of text or speech.If the unit is a word,- 1-gram: to be or not to be → to, be, or, not, to, be (same)- 2-gram: to be or not to be → to be, be or, or not, not to, to be- etc.And the docs entries: Pandas, NumPy, and SciPyWe will use them a bit, but won't explain them too much, please refer to the docs:- [pandas API Reference](http://pandas.pydata.org/pandas-docs/stable/api.html)- [NumPy Reference](https://docs.scipy.org/doc/numpy/reference/)- [SciPy Reference Guide](https://docs.scipy.org/doc/scipy/reference/)Or the tutorials:- [10 Minutes to pandas](http://pandas.pydata.org/pandas-docs/stable/10min.html)- [NumPy Quickstart Tutorial](https://docs.scipy.org/doc/numpy-dev/user/quickstart.html)- [SciPy Tutorial](https://docs.scipy.org/doc/scipy/reference/tutorial/index.html) See Also1. [EDA, Exploratory data analysis – Wikipedia](https://en.wikipedia.org/wiki/Exploratory_data_analysis)2. [Tf-idf – Example – Wikipedia](https://en.wikipedia.org/wiki/Tf%E2%80%93idfExample_of_tf.E2.80.93idf)3. [N-gram – Example – Wikipedia](https://en.wikipedia.org/wiki/N-gramExamples)4. [Text Feature Extraction – scikit-learn](http://scikit-learn.org/stable/modules/feature_extraction.htmltext-feature-extraction)<jupyter_code># init for plotting

%matplotlib inline
import matplotlib
matplotlib.style.use('ggplot')

# init the utils

import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Explore the TermsWhat are the important terms? The First Try<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

# doc: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
vectorizer = CountVectorizer(
    # the default regex select tokens of 2+ alphanumeric characters
    # it filters 催 停 到 from 催 太猛 停車 停 到 衝進 藥局,
    # but they are significant in Chinese, we rewrite a regex here
    token_pattern=r'(?u)\b\w+\b',
    # unigram and bigram
    # unigram: 催 / 太猛 / 衝進 -> 催 / 太猛 / 衝進
    # bigram: 催 / 太猛 / 衝進 -> 催 太猛 / 太猛 衝進
    ngram_range=(1, 2)
)
# smat: sparse matrix
input_term_count_smat = vectorizer.fit_transform([
    '新聞 中午 吃 什麼',
    '新聞 急症 沒錢醫',
])

print(vectorizer.get_feature_names())
# rows are inputs
# columns are counts of a feature
print(input_term_count_smat.toarray())
with open('corpus.txt') as f:
    print(next(f), end='')
    print(next(f), end='')
    print(next(f), end='')
with open('corpus.txt') as f:
    input_term_count_smat = vectorizer.fit_transform(f)
# mat: matrix
term_count_mat = input_term_count_smat.sum(axis=0)
term_count_arr = np.asarray(term_count_mat).reshape(-1)
sorted_term_count_arr = np.sort(term_count_arr)
# doc: http://pandas.pydata.org/pandas-docs/stable/visualization.html
sorted_term_count_s = pd.Series(sorted_term_count_arr)
sorted_term_count_s.plot()
sorted_term_count_s.describe()
vectorizer.stop_words_
feature_names = vectorizer.get_feature_names()
sorted_term_idx_arr = np.argsort(term_count_arr)

n = 10

print('Top', n, 'terms and counts:')
print()
for term_idx in sorted_term_idx_arr[:-n:-1]:
    print(feature_names[term_idx], term_count_arr[term_idx])<jupyter_output>Top 10 terms and counts:

新聞 10005
被 564
的 518
了 429
不 378
是 347
人 309
蔡 287
台灣 279<jupyter_text>The Second Try<jupyter_code>vectorizer_2 = CountVectorizer(
    min_df=1, max_df=0.5,
    token_pattern=r'(?u)\b\w+\b',
    ngram_range=(1, 2)
)

with open('corpus.txt') as f:
    input_term_count_smat_2 = vectorizer_2.fit_transform(f)
    
term_count_mat_2 = input_term_count_smat_2.sum(axis=0)
term_count_arr_2 = np.asarray(term_count_mat_2).reshape(-1)
sorted_term_count_arr_2 = np.sort(term_count_arr_2)

sorted_term_count_s_2 = pd.Series(sorted_term_count_arr_2)
sorted_term_count_s_2.plot()
sorted_term_count_s_2.describe()
vectorizer_2.stop_words_
feature_names = vectorizer_2.get_feature_names()
sorted_term_idx_arr = np.argsort(term_count_arr_2)

n = 10

print('Top', n, 'terms and counts:')
print()
for term_idx in sorted_term_idx_arr[:-n:-1]:
    print(feature_names[term_idx], term_count_arr_2[term_idx])<jupyter_output>Top 10 terms and counts:

被 564
的 518
了 429
不 378
是 347
人 309
蔡 287
台灣 279
遭 270<jupyter_text>Explore the Push Score SumsWhat are the output we wanna predict?<jupyter_code>with open('target.txt') as f:
    print(next(f), end='')
    print(next(f), end='')
    print(next(f), end='')
with open('target.txt') as f:
    push_score_sums = [int(line) for line in f]
push_score_sum_s = pd.Series(push_score_sums)
push_score_sum_s.plot.hist(bins=100)
push_score_sum_s.describe()<jupyter_output><empty_output><jupyter_text>Train and Evaluate Load Input and Expected Output InUse the findings from the previous section.<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

# doc: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
vectorizer = TfidfVectorizer(
    min_df=1, max_df=0.5,
    token_pattern=r'(?u)\b\w+\b',
    ngram_range=(1, 2)
)

with open('corpus.txt') as f:
    X = vectorizer.fit_transform(f)
    
with open('target.txt') as f:
    # let's use -1 to represent 被噓, and 0 for other cases
    y = [-1 if int(line) <= -1 else 0 for line in f]<jupyter_output><empty_output><jupyter_text>Split Training Set and Testing SetWe split some for training model, some for validating.<jupyter_code>from sklearn.model_selection import train_test_split

# doc: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)<jupyter_output><empty_output><jupyter_text>First TrySVM: [Support Vector Machines – scikit-learn](http://scikit-learn.org/stable/modules/svm.htmlsupport-vector-machines) SVM's Hyper-Plane![SVM's Hyper-Plane](http://scikit-learn.org/stable/_images/sphx_glr_plot_separating_hyperplane_0011.png)<jupyter_code>from sklearn import svm
from sklearn.metrics import classification_report

# doc: http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html
svc = svm.SVC(kernel='linear', C=1, class_weight='balanced')

print('SVC')
print()

%time svc.fit(X_train, y_train)
print()

# doc: http://scikit-learn.org/stable/modules/model_evaluation.html#accuracy-score
print('Traning Set Accuracy:', svc.score(X_train, y_train))
print('Validation Accuracy:', svc.score(X_test, y_test))
print()

print(classification_report(y_test, svc.predict(X_test)))<jupyter_output>SVC

CPU times: user 8.59 s, sys: 228 ms, total: 8.82 s
Wall time: 8.93 s

Traning Set Accuracy: 0.993857142857
Validation Accuracy: 0.810333333333

             precision    recall  f1-score   support

         -1       0.56      0.25      0.35       603
          0       0.83      0.95      0.89      2397

avg / total       0.78      0.81      0.78      3000<jupyter_text>Precision and Recall![Precision and Recall](https://upload.wikimedia.org/wikipedia/commons/2/26/Precisionrecall.svg) F1 Score![F1Score](https://wikimedia.org/api/rest_v1/media/math/render/svg/7d63c1f5c659f95b5dfe5893213cc8ea7f8bea0a)A F1 score is the harmonic mean of precision and recall. See [F1 score](https://en.wikipedia.org/wiki/F1_score) for details.  Explore the Trained ModelKnow the data even deeper.<jupyter_code>class_i = 0
n = 10

feature_names = vectorizer.get_feature_names()
sorted_feature_idx_arr = np.argsort(svc.coef_.toarray())[class_i]
print('Top', n, 'positive features:')
print()

for fidx in sorted_feature_idx_arr[:-n:-1]:
    print(feature_names[fidx], svc.coef_[class_i, fidx])
    
print()
    
print('Top', n, 'negative features:')
print()

for fidx in sorted_feature_idx_arr[:n]:
    print(feature_names[fidx], svc.coef_[class_i, fidx])<jupyter_output>Top 10 positive features:

美 1.49081155042
20 1.19937534089
泰國 1.18790931177
美國 1.11804937332
抗議 1.01609857754
日本 0.992017621316
年 0.93827629264
杜特蒂 0.91317068476
反同 0.909715194003

Top 10 negative features:

歐陽 -2.61933393496
吳秉叡 -2.55979999573
小s -2.52602534262
洪秀柱 -2.5077794054
黑人 -2.33544331716
兩岸 -2.28616224413
柯建銘 -2.06086631084
小英 -2.00811527009
勞動部 -2.0025928574
蔡 -1.98853279842<jupyter_text>Feature SelectionImprove models' accuracy scores or to boost their performance – [Feature Selection - scikit-learn](http://scikit-learn.org/stable/modules/feature_selection.htmlfeature-selection).<jupyter_code># doc: http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html
lsvc = svm.LinearSVC(C=1, penalty='l1', dual=False)

print('Linear SVC')
print()

%time lsvc.fit(X_train, y_train)
print()

print('Training Set Shape:', X_train.shape)
print('Traning Set Accuracy:', lsvc.score(X_train, y_train))
print('Validation Accuracy:', lsvc.score(X_test, y_test))
print()

print(classification_report(y_test, lsvc.predict(X_test)))
from sklearn.feature_selection import SelectFromModel

# doc: http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html
sfm = SelectFromModel(lsvc, prefit=True)
X_train_2 = sfm.transform(X_train)
X_test_2 = sfm.transform(X_test)

lsvc_2 = svm.LinearSVC(C=1, penalty='l1', dual=False)

print('Linear SVC #2')
print()

%time lsvc_2.fit(X_train_2, y_train)
print()

print('Training Set Shape:', X_train_2.shape)
print('Traning Set Accuracy:', lsvc_2.score(X_train_2, y_train))
print('Validation Accuracy:', lsvc_2.score(X_test_2, y_test))
print()

print(classification_report(y_test, lsvc_2.predict(X_test_2)))<jupyter_output>Linear SVC #2

CPU times: user 26.5 ms, sys: 592 µs, total: 27.1 ms
Wall time: 27.7 ms

Training Set Shape: (7000, 1264)
Traning Set Accuracy: 0.877142857143
Validation Accuracy: 0.811

             precision    recall  f1-score   support

         -1       0.59      0.20      0.30       603
          0       0.83      0.96      0.89      2397

avg / total       0.78      0.81      0.77      3000<jupyter_text>Model SelectionSelect the best model. See [Model Selection – scikit-learn](http://scikit-learn.org/stable/model_selection.html) for the full doc.<jupyter_code>from sklearn.model_selection import GridSearchCV

# CV: Cross-Validation, http://scikit-learn.org/stable/modules/cross_validation.html
# Grid Search: http://scikit-learn.org/stable/modules/grid_search.html
# doc: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html
gscv = GridSearchCV(
    svm.LinearSVC(),
    {'C': [1, 10, 100, 1000]},
    cv=5
)

print('Grid Search CV on LinearSVC')
print()

%time gscv.fit(X_train, y_train)
print()

print('Best Parameters:', gscv.best_params_)
print('Best Traning Set Accuracy:', gscv.best_score_)
best_lsvc = svm.LinearSVC(**gscv.best_params_)

print('Best Linear SVC')
print()

%time best_lsvc.fit(X_train, y_train)
print()

print('Traning Set Accuracy:', best_lsvc.score(X_train, y_train))
print('Validation Accuracy:', best_lsvc.score(X_test, y_test))
print()

print(classification_report(y_test, best_lsvc.predict(X_test)))<jupyter_output>Best Linear SVC

CPU times: user 1.34 s, sys: 6.16 ms, total: 1.35 s
Wall time: 1.35 s

Traning Set Accuracy: 0.999
Validation Accuracy: 0.813666666667

             precision    recall  f1-score   support

         -1       0.60      0.21      0.31       603
          0       0.83      0.96      0.89      2397

avg / total       0.78      0.81      0.78      3000<jupyter_text>Your Turn1. Try to use [.predict](http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.htmlsklearn.svm.LinearSVC.predict) to predict an unknown news.2. Add more features? e.g., adding `author_id` by [Dict Vectorizer](http://scikit-learn.org/stable/modules/feature_extraction.htmldict-feature-extraction).3. Predict more categories? e.g., sum(push scores) >= 10.4. Try other kernel of SVM? e.g., [RBF](http://scikit-learn.org/stable/modules/svm.htmlsvm-kernels). References1. Supervised Machine Learning – why & how to apply by 朱伯憲2. [Data mining – process – Wikipedia](https://en.wikipedia.org/wiki/Data_miningProcess)<jupyter_code>from ptt_corpus_tokenizer import preprocess_and_tokenizie
best_lsvc.predict(vectorizer.transform([
    preprocess_and_tokenizie('[新聞] 爭風吃醋！國中生臉書約談判　持西瓜刀'),
    preprocess_and_tokenizie('[新聞] 「炫妻狂魔」黑人狂PO范范美照　引古詞讚'),
    preprocess_and_tokenizie('[新聞] 分手後驚覺懷孕！前女友轉生氣　怒告男：'),
]))<jupyter_output>Building prefix dict from the default dictionary ...
Loading model from cache /var/<KEY>T/jieba.cache
Loading model cost 1.163 seconds.
Prefix dict has been built succesfully.
{'fraction_non_alphanumeric': 0.12103688933200399, 'fraction_numerical': 0.046992356264539716, 'mean_word_length': 4.597470238095238, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 44, 'https://': 17, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '448212', 'license': 'MIT', 'n_tokens_mistral': 6166, 'n_tokens_neox': 5707, 'n_words': 1322, 'path': 'minig_ptt_news.ipynb', 'repo_name': 'moskytw/mining-ptt-news', 'size': '15090'}