<jupyter_start><jupyter_text>环境配置<jupyter_code>!pip install -r examples/pytorch/summarization/requirements.txt transformers==4.6<jupyter_output>Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com
Collecting transformers==4.6
  Downloading transformers-4.6.0-py3-none-any.whl (2.3 MB)
     |████████████████████████████████| 2.3 MB 19.8 MB/s            
[?25hCollecting datasets>=1.1.3
  Downloading datasets-1.15.1-py3-none-any.whl (290 kB)
     |████████████████████████████████| 290 kB 53.8 MB/s            
[?25hCollecting sentencepiece!=0.1.92
  Downloading sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 40.9 MB/s            
[?25hRequirement already satisfied: protobuf in /home/ec2-user/anaconda3/envs/pytorch_p37/lib/python3.7/site-packages (from -r examples/pytorch/summarization/requirements.txt (line 3)) (3.19.0)
Collecting rouge-score
  Downloading rouge_score-0.0.4-py2.py3-none-any.whl (22 kB)
Requirement already satisfied: nltk in /home/ec2-user/anaconda3/envs/pytorch_p37/lib/python3.7/site-packages (fr[...]<jupyter_text>预训练模型下载<jupyter_code>!mkdir -p models/pretrain/pegasus
!mkdir -p models/pretrain/bart

!mkdir -p ./models/local_train/pegasus-hp
!mkdir -p ./models/local_train/bart-hp

!aws s3 cp s3://datalab2021/hupo_nlp/models/pegasus/checkpoint-46314.zip models/pretrain/pegasus
!aws s3 cp s3://datalab2021/hupo_nlp/models/bart/checkpoint-46314.zip models/pretrain/bart
    
!unzip models/pretrain/pegasus/checkpoint-46314.zip -d models/pretrain/pegasus > /dev/null 2>&1
!unzip models/pretrain/bart/checkpoint-46314.zip -d models/pretrain/bart > /dev/null 2>&1<jupyter_output><empty_output><jupyter_text>模型训练<jupyter_code>!python -u examples/pytorch/summarization/run_summarization.py \
--model_name_or_path google/pegasus-large \
--do_train \
--do_eval \
--per_device_train_batch_size=2 \
--per_device_eval_batch_size=1 \
--save_strategy epoch \
--evaluation_strategy epoch \
--overwrite_output_dir \
--predict_with_generate \
--train_file './data/hp/summary/news_summary_cleaned_small_train.csv' \
--validation_file './data/hp/summary/news_summary_cleaned_small_test.csv' \
--text_column 'text' \
--summary_column 'headlines' \
--output_dir='./models/local_train/pegasus-hp' \
--num_train_epochs=1.0 \
--eval_steps=500 \
--save_total_limit=3 \
--source_prefix "summarize: " > train_pegasus.log
!python -u examples/pytorch/summarization/run_summarization.py \
--model_name_or_path facebook/bart-large-cnn \
--do_train \
--do_eval \
--per_device_train_batch_size=1 \
--per_device_eval_batch_size=1 \
--save_strategy epoch \
--evaluation_strategy epoch \
--overwrite_output_dir \
--predict_with_generate \
--train_file './data/hp/summary/news_summary_cleaned_small_train.csv' \
--validation_file './data/hp/summary/news_summary_cleaned_small_test.csv' \
--text_column 'text' \
--summary_column 'headlines' \
--output_dir='./models/local_train/bart-hp' \
--num_train_epochs=1.0 \
--eval_steps=1000 \
--save_total_limit=3 \
--source_prefix "summarize: " > train_bart.log<jupyter_output><empty_output><jupyter_text>本地推理<jupyter_code>import pandas as pd
df=pd.read_csv('./data/hp/summary/news_summary_cleaned_small_test.csv')
print('原文:',df.loc[0,'text'])
print('真实标签:',df.loc[0,'headlines'])
from transformers import pipeline
summarizer = pipeline("summarization", model="./models/local_train/pegasus-hp/checkpoint-500")
print('模型预测:',summarizer(df.loc[0,'text'], max_length=50)[0]['summary_text'])<jupyter_output>原文: Hybrid electric aircraft startup Zunum Aero, founded by Indian-origin entrepreneur <NAME>, has received investments from aerospace companies Boeing and JetBlue. The startup intends to make regional aircrafts with space for 10 to 50 passengers for flights up to 1,600 km. "Our goal is to be part of a disruptive force rather than the one being disrupted," said JetBlue.
真实标签: Boeing, JetBlue back Indian-origin man's aircraft startup
模型预测: Hybrid electric aircraft startup Zunum Aero, founded by Indian-origin entrepreneur <NAME>umar, has received investments from aerospace companies Boeing and JetBlue.<jupyter_text>增强训练<jupyter_code>!python -u examples/pytorch/summarization/run_summarization.py \
--model_name_or_path models/pretrain/pegasus/checkpoint-46314 \
--do_train \
--do_eval \
--per_device_train_batch_size=2 \
--per_device_eval_batch_size=1 \
--save_strategy epoch \
--evaluation_strategy epoch \
--overwrite_output_dir \
--predict_with_generate \
--train_file './data/hp/summary/news_summary_cleaned_small_train.csv' \
--validation_file './data/hp/summary/news_summary_cleaned_small_test.csv' \
--text_column 'text' \
--summary_column 'headlines' \
--output_dir='./models/local_train/pegasus-hp' \
--num_train_epochs=1.0 \
--eval_steps=500 \
--save_total_limit=3 \
--source_prefix "summarize: " > train_pegasus_2.log<jupyter_output>huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
loading configuration file models/pretrain/pegasus/checkpoint-46314/config.json
Model config PegasusConfig {
  "_name_or_path": "google/pegasus-large",
  "activation_dropout": 0.1,
  "activation_function": "relu",
  "add_bias_logits": false,
  "add_final_layer_norm": true,
  "architectures": [
    "PegasusForConditionalGeneration"
  ],
  "attention_dropout": 0.1,
  "bos_token_id": 0,
  "classif_dropout": 0.0,
  "classifier_dropout": 0.0,
  "d_model": 1024,
  "decoder_attention_heads": 16,
  "decoder_ffn_dim": 4096,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 16,
  "decoder_start_token_id": 0,
  "dropout": 0.1,
  "encoder_attention_heads": 16,
  "encoder_ffn_dim": [...]<jupyter_text>本地推理<jupyter_code>import pandas as pd
df=pd.read_csv('./data/hp/summary/news_summary_cleaned_small_test.csv')
print('原文:',df.loc[0,'text'])
print('真实标签:',df.loc[0,'headlines'])
from transformers import pipeline
summarizer = pipeline("summarization", model="./models/local_train/pegasus-hp/checkpoint-500")
print('模型预测:',summarizer(df.loc[0,'text'], max_length=50)[0]['summary_text'])<jupyter_output>原文: Hybrid electric aircraft startup Zunum Aero, founded by Indian-origin entrepreneur <NAME>, has received investments from aerospace companies Boeing and JetBlue. The startup intends to make regional aircrafts with space for 10 to 50 passengers for flights up to 1,600 km. "Our goal is to be part of a disruptive force rather than the one being disrupted," said JetBlue.
真实标签: Boeing, JetBlue back Indian-origin man's aircraft startup
模型预测: Boeing, JetBlue back Indian-origin man's aircraft startup<jupyter_text>模型部署<jupyter_code>!cp -r ./models/local_train/pegasus-hp/checkpoint-500 ./endpoint/pegasus
!sh build_and_push.sh pegasus-hp
#注意修改：847380964353.dkr.ecr.ap-northeast-1.amazonaws.com/pegasus-hp为自己对应的
%cd endpoint

!python create_endpoint.py \
--endpoint_ecr_image_path "847380964353.dkr.ecr.ap-northeast-1.amazonaws.com/pegasus-hp" \
--endpoint_name 'pegasus' \
--instance_type "ml.p3.2xlarge"

%cd ..
from boto3.session import Session
import json
df=pd.read_csv('./data/hp/summary/news_summary_cleaned_small_test.csv')
print('原文:',df.loc[0,'text'])
print('真实标签:',df.loc[0,'headlines'])
data={"data": df.loc[0,'text']}
session = Session()
    
runtime = session.client("runtime.sagemaker")
response = runtime.invoke_endpoint(
    EndpointName='pegasus',
    ContentType="application/json",
    Body=json.dumps(data),
)

result = json.loads(response["Body"].read())
print (result)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14558595766649457, 'fraction_numerical': 0.034202374806401654, 'mean_word_length': 5.224096385542168, 'pattern_counts': {'":': 20, '<': 28, '<?xml version=': 0, '>': 36, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '648749', 'license': 'Apache-2.0', 'n_tokens_mistral': 3077, 'n_tokens_neox': 2870, 'n_words': 505, 'path': 'hp_main.ipynb', 'repo_name': 'HaoranLv/nlp_transformer', 'size': '7762'}