<jupyter_start><jupyter_text>1. 加载数据<jupyter_code># 原始语料，3个文本
strs_train =[
'God is love',
'OpenGL on the GPU is fast',
'<NAME> is PHD']<jupyter_output><empty_output><jupyter_text>2. 提取Bags of words特征<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(strs_train)
X_train_counts.shape
X_train_counts.todense()#转换成数值特征Bags of words
count_vect.vocabulary_#特征名<jupyter_output><empty_output><jupyter_text>3. 提取TF特征<jupyter_code>from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
# 先提取 Bags of words特征
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(strs_train)
# 再基于Bags of words特征，变换为TF特征
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)
X_train_tf.shape
X_train_tf.todense()# 查看TF特征<jupyter_output><empty_output><jupyter_text>**sklearn的TF计算并不是用 该词出现次数/文档词数，而是直接将Bags of words做Normalization**  4. 提取TF-IDF特征<jupyter_code>from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
# 先提取 Bags of words特征
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(strs_train)
# 再基于Bags of words特征，变换为TF-IDF特征
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
X_train_tfidf.todense()<jupyter_output><empty_output><jupyter_text>5. FeatureHasher<jupyter_code>from sklearn.feature_extraction import FeatureHasher

# 将原始数据Hash到10维Feature
h = FeatureHasher(n_features=10, non_negative=True)

# 将数据组成D格式
D=[]
for s in strs_train:
    tmp = {}
    for w in s.split(' '):
        tmp[w] = 1
    D.append(tmp)

f = h.transform(D)
f.toarray()

D<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>import numpy as np
import scipy.sparse as sp

X = X_train_counts.data
hasattr(X, 'dtype')
    if sp.isspmatrix_csr(X):
        return bincount(X.indices, minlength=X.shape[1])
    else:
        return np.diff(sp.csc_matrix(X, copy=False).indptr)

np.log(X.data, X.data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10036663611365719, 'fraction_numerical': 0.005499541704857928, 'mean_word_length': 7.594488188976378, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '221160', 'license': 'MIT', 'n_tokens_mistral': 942, 'n_tokens_neox': 964, 'n_words': 133, 'path': 'text_feature/extract_mine_strings_feature.ipynb', 'repo_name': 'ybdesire/machinelearning', 'size': '2188'}