<commit_before>## Stock Index Prediction Based on Sentiment Analysis(基于情感分析的股指预测)

### Data Preprocessing(数据集预处理)
- stocktime.py is to provide the basic open time and close time of A stock(Shanghai Composite Index) in China
- snspy.py(supported by liaoxuefeng) is to get microblog from sina
- weibo.py is to process every blog clean and save it to txt every 5 minutes
- stockindex.py is to get newest market data from sina js server and save list to pickle(shindex_seq.pkl), list contains every 5-min close index

### Reviews Preprocessing(评论的手动标注)
- reviews_preprocessing.py is to seperate all the reviews into two part : positive and negative part(with some handful work)
- save the negative set to ./Reviews/neg_reviews.pkl and the positive set to ./Reviews/pos_reviews.pkl

### Classifier Score(分类器的比较)
- tools : using the package nltk and scikit-learn to implement machine learning method
- using the hand-tag positive and negative reviews to implement machine learning classifier(Logistic Regression, Naive Bayes and Support Vector Machine)

### Sentiment Index Computation(情感系数的计算)
- sentiment.py is to compute index from Comments/20160000/****.txt and save list to pickle(saindex_seq.pkl), list contains every 5 minutes sentiment index

### Correlation Analysis(相关性分析)
- correlation.py is to correlate the sequence of sentiment index and shanghai composite index at a specified day

### Linear Regression(多元线性回归)
- use historical 5-min close index to predict next close index(multi-linear regression)
- plus 5-min sentiment-index into multi-factor and predict again

### Other Infos(配置及文件夹信息)
- config.ini is to write information(App Key, App Secret and Callback URL) to connect sina open api
- Directory Comments is to save all the online comments from sina weibo
- Directory Lexicons is to save lexicon dict
- Directory Reviews is to save tagged positive and negtive comments.
- Directory Sentiment Index is to save everyday 5-min sentiment index time Series
- Directory Stock Index is to save everyday 5-min stock close index time series

### The Classifying Result(分类结果)
- not use bigrams
The Lexicons's average precision recall accuracy score is repectively : 0.95 0.93 0.94
The LR's average precision recall accuracy score is repectively : 0.91 0.90 0.90
The BernoulliNB's average precision recall accuracy score is repectively : 0.87 0.82 0.85
The MultinomialNB's average precision recall accuracy score is repectively : 0.87 0.80 0.84
The LinearSVC's average precision recall accuracy score is repectively : 0.90 0.89 0.90
The NuSVC's average precision recall accuracy score is repectively : 0.89 0.88 0.89
The SVC's average precision recall accuracy score is repectively : 0.85 0.85 0.85

- use bigrams
The Lexicon's average precision recall accuracy score is repectively : 0.95 0.93 0.94
The LR's average precision recall accuracy score is repectively : 0.91 0.89 0.90
The BernoulliNB's average precision recall accuracy score is repectively : 0.87 0.81 0.85
The MultinomialNB's average precision recall accuracy score is repectively : 0.87 0.80 0.84
The LinearSVC's average precision recall accuracy score is repectively : 0.89 0.88 0.89
The NuSVC's average precision recall accuracy score is repectively : 0.90 0.88 0.89
The SVC's average precision recall accuracy score is repectively : 0.85 0.85 0.85

### The Prediction Result(预测结果)
- contains historical data and plus sentiment index(based on lexicons and best-classifier:LR)
- from left to right, the metrics name respectively is : MA, RMSE, MAPE
#### Historical Regression
Test-Metrics :   0.970355  5.105886  0.001043

#### Lexicons-POS
- AVG_UP_DOWN_NUM:50.21%	COEF_AVG:0.55	COEF_MAX:0.89
- use emotion with forward_unit 0<commit_msg>change readme<commit_after>## Stock Index Prediction Based on Sentiment Analysis(基于情感分析的股指预测)

### Data Preprocessing(数据集预处理)
- stocktime.py is to provide the basic open time and close time of A stock(Shanghai Composite Index) in China
- snspy.py(supported by liaoxuefeng) is to get microblog from sina
- weibo.py is to process every blog clean and save it to txt every 5 minutes
- stockindex.py is to get newest market data from sina js server and save list to pickle(shindex_seq.pkl), list contains every 5-min close index

### Reviews Preprocessing(评论的手动标注)
- reviews_preprocessing.py is to seperate all the reviews into two part : positive and negative part(with some handful work)
- save the negative set to ./Reviews/neg_reviews.pkl and the positive set to ./Reviews/pos_reviews.pkl

### Classifier Score(分类器的比较)
- tools : using the package nltk and scikit-learn to implement machine learning method
- using the hand-tag positive and negative reviews to implement machine learning classifier(Logistic Regression, Naive Bayes and Support Vector Machine)

### Sentiment Index Computation(情感系数的计算)
- sentiment.py is to compute index from Comments/20160000/****.txt and save list to pickle(saindex_seq.pkl), list contains every 5 minutes sentiment index

### Correlation Analysis(相关性分析)
- correlation.py is to correlate the sequence of sentiment index and shanghai composite index at a specified day

### Linear Regression(多元线性回归)
- use historical 5-min close index to predict next close index(multi-linear regression)
- plus 5-min sentiment-index into multi-factor and predict again

### Other Infos(配置及文件夹信息)
- config.ini is to write information(App Key, App Secret and Callback URL) to connect sina open api
- Directory Comments is to save all the online comments from sina weibo
- Directory Lexicons is to save lexicon dict
- Directory Reviews is to save tagged positive and negtive comments.
- Directory Sentiment Index is to save everyday 5-min sentiment index time Series
- Directory Stock Index is to save everyday 5-min stock close index time series

### The Classifying Result(分类结果)
- not use bigrams
- The Lexicons's average precision recall accuracy score is repectively : 0.95 0.93 0.94
- The LR's average precision recall accuracy score is repectively : 0.91 0.90 0.90
- The BernoulliNB's average precision recall accuracy score is repectively : 0.87 0.82 0.85
- The MultinomialNB's average precision recall accuracy score is repectively : 0.87 0.80 0.84
- The LinearSVC's average precision recall accuracy score is repectively : 0.90 0.89 0.90
- The NuSVC's average precision recall accuracy score is repectively : 0.89 0.88 0.89
- The SVC's average precision recall accuracy score is repectively : 0.85 0.85 0.85

- use bigrams
- The Lexicon's average precision recall accuracy score is repectively : 0.95 0.93 0.94
- The LR's average precision recall accuracy score is repectively : 0.91 0.89 0.90
- The BernoulliNB's average precision recall accuracy score is repectively : 0.87 0.81 0.85
- The MultinomialNB's average precision recall accuracy score is repectively : 0.87 0.80 0.84
- The LinearSVC's average precision recall accuracy score is repectively : 0.89 0.88 0.89
- The NuSVC's average precision recall accuracy score is repectively : 0.90 0.88 0.89
- The SVC's average precision recall accuracy score is repectively : 0.85 0.85 0.85

### The Prediction Result(预测结果)
- contains historical data and plus sentiment index(based on lexicons and best-classifier:LR)
- from left to right, the metrics name respectively is : MA, RMSE, MAPE
#### Historical Regression
Test-Metrics :   0.970355  5.105886  0.001043

#### Lexicons-POS
- AVG_UP_DOWN_NUM:50.21%	COEF_AVG:0.55	COEF_MAX:0.89
- use emotion with forward_unit 0
{'fraction_non_alphanumeric': 0.0644122383252818, 'fraction_numerical': 0.046430488459473965, 'mean_word_length': 4.4963126843657815, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1935237', 'n_tokens_mistral': 2467, 'n_tokens_neox': 2172, 'n_words': 978}