<jupyter_start><jupyter_text>普通模型<jupyter_code>from transformers import BertTokenizer, RobertaForTokenClassification


tokenizer = BertTokenizer.from_pretrained(model_name)
model = RobertaForTokenClassification.from_pretrained(model_name)

tokens = tokenizer(text, return_tensors='pt')
logits = model(**tokens).logits
logit_result = logits.argmax(axis=2).tolist()
print(logit_result)<jupyter_output>Some weights of the model checkpoint at ./models/guwen-quote were not used when initializing RobertaForTokenClassification: ['crf.start_transitions', 'crf.end_transitions', 'crf.transitions']
- This IS expected if you are initializing RobertaForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[[0, 0, 0, 1, 2, 0, 2, 2, 0, 0, 2, 2, 1, 2, 0, 0, 2, 2, 0, 0, 2, 2, 0, 0, 2, 0, 0, 2, 0, 0, 0, 2, 2, 0, 0, 0, 1, 0, 0, 0, 0, 2, 0, 0, 0, 2, 0, 2, 2, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 2, 2, 0, 0, 0, 2, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 0, 0, 1, 2, 2, 2, 0, 2, 2, 0, 0, 0, 1, 0, 0, 0, 0, 2, 0, 0, 2, 0, 0, 2[...]<jupyter_text>CRF模型<jupyter_code>from transformers import BertTokenizer
from crf.crf_roberta import CRFRobertaForTokenClassification

tokenizer = BertTokenizer.from_pretrained(model_name)
model = CRFRobertaForTokenClassification.from_pretrained(model_name)

tokens = tokenizer(text, return_tensors='pt')
logits = model(**tokens).logits
crf_result = model.decode(logits, mask=None)
print(crf_result)<jupyter_output>[[0, 0, 0, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 2, 2, 2, 2, 2, 0, 0, 0, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0]]<jupyter_text>贪心法将 `1 2 0 0 2 0` 转换为 `1 2 2 2 2 0`<jupyter_code>def greedy(logits):
    in_span = False
    result = []
    for tag in reversed(logits):
        if tag == 1:
            in_span = False
        if tag == 2:
            in_span = True
        if in_span:
            tag = 2
        result.append(tag)
    result.reverse()
    return result

greedy_result = greedy(logits.argmax(axis=2).tolist()[0])<jupyter_output><empty_output><jupyter_text>可视化<jupyter_code>def visualization(tags):
    in_span = False
    result = []
    for i, (tag, token) in enumerate(zip(tags, text + " ")):
        if tag in (0, 1) and in_span:
            result.append("」")
            in_span = False
        if tag == 1:
            result.append("「")
            in_span = True
        result.append(token)
    print(''.join(result))
visualization(logit_result[0][1:])
visualization(crf_result[0][1:])
visualization(greedy_result[1:])<jupyter_output>子曰「学而时习之不亦说乎」「有朋自远方来不亦乐乎人不知而不愠不亦君子乎」有子曰「其为人也孝弟而好犯上者鲜矣不好犯上而好作乱者未之有也君子务本本立而道生孝弟也者其为仁之本与」子曰「巧言令色鲜矣仁」曾子曰「吾日三省吾身为人谋而不忠乎与朋友交而不信乎传不习乎」子曰「道千乘之国敬事而信节用而爱人使民以时」
{'fraction_non_alphanumeric': 0.14860402281597118, 'fraction_numerical': 0.08105673971780246, 'mean_word_length': 3.9289940828402368, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '421102', 'license': 'Apache-2.0', 'n_tokens_mistral': 1735, 'n_tokens_neox': 1471, 'n_words': 430, 'path': 'crf_example.ipynb', 'repo_name': 'Ethan-yt/guwen-models', 'size': '3331'}