TCP 2023 for NTU students

Fine tuning pre-trained language models for text generation.

Pretrained model on Chinese language using a GPT2 for Large Language Head Model objective(GPT2LMHeadModel).

Model description

TCP 2023 is a transformers model that has undergone fine-tuning using the GPT-2 architecture. It was initially pretrained on an extensive corpus of Chinese data in a self-supervised manner. This implies that the pretraining process involved using raw text data without any human annotations, allowing the model to make use of a wide range of publicly available data. The model leveraged an automatic process to derive inputs and corresponding labels from these texts. To be more specific, the pretraining aimed at predicting the subsequent word in sentences. it was trained to guess the next word in sentences.

How to use

You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we set a seed for reproducibility:

>>> from transformers import GPT2LMHeadModel, AutoTokenizer, pipeline

>>> model_name = "DavidLanz/tcp2023"

>>> model = GPT2LMHeadModel.from_pretrained(model_name)
>>> tokenizer = AutoTokenizer.from_pretrained(model_name)

>>> text_generator = pipeline('text-generation', model=model, tokenizer=tokenizer)
>>> generated_text = text_generator(input_text, max_length=max_len, num_return_sequences=1)
>>> print(generated_text[0]['generated_text'])