<jupyter_start><jupyter_text>机器学习工作流完成下面的步骤，实现该分类器的机器学习管道。<jupyter_code>import nltk
nltk.download(['punkt', 'wordnet'])
import re
import pandas as pd
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
url_regex = 'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+'

def load_data():
    df = pd.read_csv('corporate_messaging.csv', encoding='latin-1')
    df = df[(df["category:confidence"] == 1) & (df['category'] != 'Exclude')]
    X = df.text.values
    y = df.category.values
    return X, y

def tokenize(text):
    detected_urls = re.findall(url_regex, text)
    for url in detected_urls:
        text = text.replace(url, "urlplaceholder")

    tokens = word_tokenize(text)
    lemmatizer = WordNetLemmatizer()

    clean_tokens = []
    for tok in tokens:
        clean_tok = lemmatizer.lemmatize(tok).lower().strip()
        clean_tokens.append(clean_tok)

    return clean_tokens<jupyter_output><empty_output><jupyter_text>第一步：数据加载和训练集分割<jupyter_code># load data
X, y =

# perform train test split<jupyter_output><empty_output><jupyter_text>第二步：训练分类器* 使用 `CountVectorizer` 拟合 (fit) 和转换 (transform) 训练数据集。提示：你可以在 `tokenizer` 关键词参数中传入你的分词 (tokenize) 函数。* 用 `TfidfTransformer` 拟合 (fit) 和转换 (transform) 这些单词计数。* 用分类器来拟合这些 tfidf 值。<jupyter_code># Instantiate transformers and classifier
vect = 
tfidf = 
clf = 

# Fit and/or transform each to the data<jupyter_output><empty_output><jupyter_text>第三步：预测* 用上述的 CountVectorizer 和 TfidfTransformer 转换（注意，不是拟合）训练数据。* 用  tfidf 值预测标签<jupyter_code># Transform test data


# Predict test labels
y_pred =<jupyter_output><empty_output><jupyter_text>第四步：显示结果根据模型的预测显示混淆矩阵 (confusion matrix) 和准确度得分 (accuracy score)。<jupyter_code>labels = 
confusion_mat = 
accuracy = 

print("Labels:", labels)
print("Confusion Matrix:\n", confusion_mat)
print("Accuracy:", accuracy)<jupyter_output><empty_output><jupyter_text>最后一步：重构将上述步骤整理成下面的函数形式。<jupyter_code>def display_results():
    # insert step 4 here
    pass


def main():
    # insert steps 1 through 3 here
    pass
# run program
main()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13832306963524396, 'fraction_numerical': 0.005210800568450971, 'mean_word_length': 5.559006211180124, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '537967', 'license': 'MIT', 'n_tokens_mistral': 949, 'n_tokens_neox': 980, 'n_words': 154, 'path': 'lessons/ML Pipelines/ml_workflow-zh.ipynb', 'repo_name': 'callezenwaka/DSND_Term2', 'size': '2111'}