<jupyter_start><jupyter_text>**Generate the program based on the question using codeTrans single task training model**You can make free prediction online through this Link (When using the prediction online, you need to parse and tokenize the code first.) **1. Load necessry libraries including huggingface transformers**<jupyter_code>!pip install -q transformers sentencepiece
from transformers import AutoTokenizer, AutoModelWithLMHead, SummarizationPipeline<jupyter_output><empty_output><jupyter_text>**2. Load the token classification pipeline and load it into the GPU if avilabile**<jupyter_code>pipeline = SummarizationPipeline(
    model=AutoModelWithLMHead.from_pretrained("SEBIS/code_trans_t5_small_program_synthese"),
    tokenizer=AutoTokenizer.from_pretrained("SEBIS/code_trans_t5_small_program_synthese", skip_special_tokens=True),
    device=0
)<jupyter_output>/usr/local/lib/python3.6/dist-packages/transformers/models/auto/modeling_auto.py:852: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
  FutureWarning,<jupyter_text>**3 Give the question for generating the code, parse and tokenize it**<jupyter_code>question = "you are given an array of numbers a and a number b, compute the difference of elements in a and b" #@param {type:"raw"}
import nltk
nltk.download('punkt')
from nltk.tokenize import word_tokenize
def englishTokenizer(sentence):
    result = []
    tokens = word_tokenize(sentence)
    for t in tokens:
        if( not len(t)>50):
            result.append(t)
    return ' '.join(result)

tokenized_question = englishTokenizer(question)
print("tokenized question: " + tokenized_question)<jupyter_output>tokenized question: you are given an array of numbers a and a number b , compute the difference of elements in a and b<jupyter_text>**4. Make Prediction**<jupyter_code>pipeline([tokenized_question])<jupyter_output>Your max_length is set to 512, but you input_length is only 28. You might consider decreasing max_length manually, e.g. summarizer('...', max_length=50)
{'fraction_non_alphanumeric': 0.08097165991902834, 'fraction_numerical': 0.009896536212325686, 'mean_word_length': 6.33993399339934, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '144045', 'license': 'MIT', 'n_tokens_mistral': 671, 'n_tokens_neox': 646, 'n_words': 208, 'path': 'prediction/single task/program synthesis/small_model.ipynb', 'repo_name': 'victory-hash/CodeTrans', 'size': '2223'}