first

ptuning/README.md

@@ -0,0 +1,248 @@
+# ChatGLM-6B-PT
+本仓库实现了对于 ChatGLM-6B 模型基于 [P-Tuning v2](https://github.com/THUDM/P-tuning-v2) 的微调。P-Tuning v2 将需要微调的参数量减少到原来的 0.1%，再通过模型量化、Gradient Checkpoint 等方法，最低只需要 7GB 显存即可运行。
+
+下面以 [ADGEN](https://aclanthology.org/D19-1321.pdf) (广告生成) 数据集为例介绍代码的使用方法。
+
+*Read this in [English](README_en.md).*
+
+## 软件依赖
+运行微调需要4.27.1版本的`transformers`。除 ChatGLM-6B 的依赖之外，还需要安装以下依赖
+```
+pip install rouge_chinese nltk jieba datasets
+```
+## 使用方法
+
+### 下载数据集
+ADGEN 数据集任务为根据输入（content）生成一段广告词（summary）。
+
+```json
+{
+    "content": "类型#上衣*版型#宽松*版型#显瘦*图案#线条*衣样式#衬衫*衣袖型#泡泡袖*衣款式#抽绳",
+    "summary": "这件衬衫的款式非常的宽松，利落的线条可以很好的隐藏身材上的小缺点，穿在身上有着很好的显瘦效果。领口装饰了一个可爱的抽绳，漂亮的绳结展现出了十足的个性，配合时尚的泡泡袖型，尽显女性甜美可爱的气息。"
+}
+```
+
+从 [Google Drive](https://drive.google.com/file/d/13_vf0xRTQsyneRKdD1bZIr93vBGOczrk/view?usp=sharing) 或者 [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b3f119a008264b1cabd1/?dl=1) 下载处理好的 ADGEN 数据集，将解压后的 `AdvertiseGen` 目录放到本目录下。
+
+### 训练
+
+#### P-tuning v2
+
+运行以下指令进行训练：
+```shell
+bash train.sh
+```
+`train.sh` 中的 `PRE_SEQ_LEN` 和 `LR` 分别是 soft prompt 长度和训练的学习率，可以进行调节以取得最佳的效果。P-Tuning-v2 方法会冻结全部的模型参数，可通过调整 `quantization_bit` 来被原始模型的量化等级，不加此选项则为 FP16 精度加载。
+
+在默认配置 `quantization_bit=4`、`per_device_train_batch_size=1`、`gradient_accumulation_steps=16` 下，INT4 的模型参数被冻结，一次训练迭代会以 1 的批处理大小进行 16 次累加的前后向传播，等效为 16 的总批处理大小，此时最低只需 6.7G 显存。若想在同等批处理大小下提升训练效率，可在二者乘积不变的情况下，加大 `per_device_train_batch_size` 的值，但也会带来更多的显存消耗，请根据实际情况酌情调整。
+
+如果你想要[从本地加载模型](https://github.com/THUDM/ChatGLM-6B#%E4%BB%8E%E6%9C%AC%E5%9C%B0%E5%8A%A0%E8%BD%BD%E6%A8%A1%E5%9E%8B)，可以将 `train.sh` 中的 `THUDM/chatglm-6b` 改为你本地的模型路径。
+
+#### Finetune
+
+如果需要进行全参数的 Finetune，需要安装 [Deepspeed](https://github.com/microsoft/DeepSpeed)，然后运行以下指令：
+
+```shell
+bash ds_train_finetune.sh
+```
+
+### 推理
+
+将 `evaluate.sh` 中的 `CHECKPOINT` 更改为训练时保存的 checkpoint 名称，运行以下指令进行模型推理和评测：
+```shell
+bash evaluate.sh
+```
+**[2023/04/10更新]** 在 P-tuning v2 训练时模型只保存 PrefixEncoder 部分的参数，所以在推理时需要同时加载原 ChatGLM-6B 模型以及 PrefixEncoder 的权重，因此需要指定参数（已更新 `evaluate.sh`） ：
+
+```shell
+--model_name_or_path THUDM/chatglm-6b
+--ptuning_checkpoint $CHECKPOINT_PATH
+```
+
+仍然兼容旧版全参保存的 Checkpoint，只需要跟之前一样设定 `model_name_or_path`：
+
+```shell
+--model_name_or_path $CHECKPOINT_PATH
+```
+
+评测指标为中文 Rouge score 和 BLEU-4。生成的结果保存在
+`./output/adgen-chatglm-6b-pt-8-1e-2/generated_predictions.txt`。
+
+### 例子
+#### 示例1
+* Input: 类型#上衣\*材质#牛仔布\*颜色#白色\*风格#简约\*图案#刺绣\*衣样式#外套\*衣款式#破洞
+* Label: 简约而不简单的牛仔外套,白色的衣身十分百搭。衣身多处有做旧破洞设计,打破单调乏味,增加一丝造型看点。衣身后背处有趣味刺绣装饰,丰富层次感,彰显别样时尚。
+* Output[微调前]: 这件上衣的材质是牛仔布,颜色是白色,风格是简约,图案是刺绣,衣样式是外套,衣款式是破洞。
+* Output[微调后]: 这是一款简约的牛仔外套,破洞设计,将牛仔布破洞,带来一种随意与个性。破洞的牛仔外套,展现出时尚气息,带来一种休闲感。同时,刺绣图案,让整件外套更加立体。
+
+#### 示例2
+
+* Input: 类型#裙\*版型#显瘦\*风格#文艺\*风格#简约\*图案#印花\*图案#撞色\*裙下摆#压褶\*裙长#连衣裙\*裙领型#圆领
+* Label: 文艺个性的印花连衣裙,藏青色底蕴,低调又大气,撞色太阳花分布整个裙身,绚丽而美好,带来时尚减龄的气质。基础款的舒适圆领,简约不失大方,勾勒精致脸庞。领后是一粒包布扣固定,穿脱十分方便。前片立体的打褶设计,搭配后片压褶的做工,增添层次和空间感,显瘦又有型。
+* Output[微调前]: 类型#裙*版型#显瘦*风格#文艺*风格#简约*图案#印花*图案#撞色*裙下摆#压褶*裙长#连衣裙*裙领型#圆领 1\. 连衣裙:简约风格,裙长为膝盖以上,裙领型为圆领。2\. 裙下摆:压褶设计,使裙摆呈现出流畅的褶皱效果。3\. 裙领型:裙领型为圆领,使穿上连衣裙后更加有型。4\. 版型:采用显瘦设计,让连衣裙看起来更加苗条。5\. 风格:文艺风格,让连衣裙更加有内涵和品味。6\. 图案:印花设计,在连衣裙上印有独特的图案。7\. 撞色:采用撞色设计,让连衣裙在色彩上更加鲜明、富有层次感。
+* Output[微调后]: 这是一款文艺范的连衣裙,以印花为元素,采用简约的印花,既能够突出文艺气质,又能够展现简约风。在印花的同时又有领子和裙摆的压褶设计,更加凸显文艺气质。简约而不会过于单调,搭配出街,穿着十分舒适。
+
+### 评估结果
+
+|               | Finetune | P-tuning v2 | LoRA |
+| ------------- | ----------- | ----- | ------------- |
+| BLEU-4        | 8.01    | 8.10 | 7.62 |
+| Rouge-1       | 31.23  | 31.12 | 30.60 |
+| Rouge-2       | 7.36    | 7.11 | 6.96 |
+| Rouge-l       | 25.08  | 24.97 | 24.80 |
+| Training Loss | 3.00 | 3.74 | 3.32 |
+
+
+
+#### 实验设置
+
+ ```
+max_source_length=64
+max_target_length=64
+max_steps=3000
+ ```
+
+##### P-tuning v2
+
+```
+pre_seq_len=128
+learning_rate=2e-2
+quantization_bit=4
+per_device_train_batch_size=16
+gradient_accumulation_steps=1
+```
+
+##### Finetune
+
+```
+learning_rate=1e-4
+fp16
+num_gpus=4
+per_device_train_batch_size=4
+gradient_accumulation_steps=1
+```
+
+##### LoRA
+
+实现采用的是 [simple_thu_chatglm6b](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/simple_thu_chatglm6b)
+
+```
+learning_rate=5e-4
+per_device_train_batch_size=16
+gradient_accumulation_steps=1
+```
+
+
+
+## 模型部署
+首先载入Tokenizer：
+
+```python
+import os
+import torch
+from transformers import AutoConfig, AutoModel, AutoTokenizer
+
+# 载入Tokenizer
+tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
+```
+
+1. 如果需要加载的是新 Checkpoint（只包含 PrefixEncoder 参数）：
+
+```python
+config = AutoConfig.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, pre_seq_len=128)
+model = AutoModel.from_pretrained("THUDM/chatglm-6b", config=config, trust_remote_code=True)
+prefix_state_dict = torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin"))
+new_prefix_state_dict = {}
+for k, v in prefix_state_dict.items():
+    if k.startswith("transformer.prefix_encoder."):
+        new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v
+model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)
+```
+注意你可能需要将 `pre_seq_len` 改成你训练时的实际值。如果你是[从本地加载模型](https://github.com/THUDM/ChatGLM-6B#%E4%BB%8E%E6%9C%AC%E5%9C%B0%E5%8A%A0%E8%BD%BD%E6%A8%A1%E5%9E%8B)的话，需要将 `THUDM/chatglm-6b` 改成本地的模型路径（注意不是checkpoint路径）。
+
+2. 如果需要加载的是旧 Checkpoint（包含 ChatGLM-6B 以及 PrefixEncoder 参数），或者进行的是全参数微调，则直接加载整个 Checkpoint：
+
+```python
+model = AutoModel.from_pretrained(CHECKPOINT_PATH, trust_remote_code=True)
+```
+
+之后根据需求可以进行量化，也可以直接使用：
+
+```python
+# Comment out the following line if you don't use quantization
+model = model.quantize(4)
+model = model.half().cuda()
+model.transformer.prefix_encoder.float()
+model = model.eval()
+
+response, history = model.chat(tokenizer, "你好", history=[])
+```
+
+## 使用自己的数据集
+修改 `train.sh` 和 `evaluate.sh` 中的 `train_file`、`validation_file`和`test_file`为你自己的 JSON 格式数据集路径，并将 `prompt_column` 和 `response_column` 改为 JSON 文件中输入文本和输出文本对应的 KEY。可能还需要增大 `max_source_length` 和 `max_target_length` 来匹配你自己的数据集中的最大输入输出长度。
+
+## 对话数据集
+
+如需要使用多轮对话数据对模型进行微调，可以提供聊天历史，例如
+
+```json
+{
+    "prompt": "是的。上下水管都好的",
+    "response": "那就要检查线路了，一般风扇继电器是由电脑控制吸合的，如果电路存在断路，或者电脑坏了的话会出现继电器不吸合的情况！",
+    "history": [
+        [
+            "长城h3风扇不转。继电器好的。保险丝好的传感器新的风扇也新的这是为什么。就是继电器缺一个信号线",
+            "用电脑能读数据流吗？水温多少"
+        ],
+        [
+            "95",
+            "上下水管温差怎么样啊？空气是不是都排干净了呢？"
+        ]
+    ]
+}
+```
+
+训练时需要指定 `--history_column` 为数据中聊天历史的 key（在此例子中是 `history`），将自动把聊天历史拼接，例如：
+
+- Input
+
+  ```
+  [Round 0]
+  问:长城h3风扇不转。继电器好的。保险丝好的传感器新的风扇也新的这是为什么。就是继电器缺一个信号线
+  ��:用电脑能读数据流吗?水温多少
+  [Round 1]
+  问:95
+  答:上下水管温差怎么样啊?空气是不是都排干净了呢?
+  [Round 2]
+  问:是的。上下水管都好的
+  答:
+  ```
+
+- Label
+
+  ```
+  那就要检查线路了,一般风扇继电器是由电脑控制吸合的,如果电路存在断路,或者电脑坏了的话会出现继电器不吸合的情况!
+  ```
+
+要注意超过输入长度 `max_source_length` 的内容会被截。
+
+可以参考以下指令：
+
+```shell
+bash train_chat.sh
+```
+
+## 引用
+
+```
+@inproceedings{liu2022p,
+  title={P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks},
+  author={Liu, Xiao and Ji, Kaixuan and Fu, Yicheng and Tam, Weng and Du, Zhengxiao and Yang, Zhilin and Tang, Jie},
+  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
+  pages={61--68},
+  year={2022}
+}
+```
+
+
+

ptuning/README_en.md

@@ -0,0 +1,115 @@
+# ChatGLM-6B-PT
+This repository implements tuning of the ChatGLM-6B model based on [P-Tuning v2](https://github.com/THUDM/P-tuning-v2). P-Tuning v2 reduces the amount of parameters that need to be optimized to 0.1% of the full fine-tuning, and then through model quantization, Gradient Checkpoint and other methods, it only needs a minimum of 7GB of video memory to run.
+
+The following uses the [ADGEN](https://aclanthology.org/D19-1321.pdf) (advertising generation) dataset as an example to introduce how to use the code.
+
+## Software dependencies
+Running p-tuning requires version 4.27.1 of `transformers`. In addition to the dependencies of ChatGLM-6B, the following dependencies are required
+```
+pip install rouge_chinese nltk jieba datasets
+```
+## Instructions
+
+### Download the dataset
+The task of the ADGEN dataset is to generate an advertisement word (summary) based on the input (content).
+
+```json
+{
+    "content": "类型#上衣*版型#宽松*版型#显瘦*图案#线条*衣样式#衬衫*衣袖型#泡泡袖*衣款式#抽绳",
+    "summary": "这件衬衫的款式非常的宽松，利落的线条可以很好的隐藏身材上的小缺点，穿在身上有着很好的显瘦效果。领口装饰了一个可爱的抽绳，漂亮的绳结展现出了十足的个性，配合时尚的泡泡袖型，尽显女性甜美可爱的气息。"
+}
+```
+
+From [Google Drive](https://drive.google.com/file/d/13_vf0xRTQsyneRKdD1bZIr93vBGOczrk/view?usp=sharing) or [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b3f119a008264b1cabd1/?dl=1) Download the processed ADGEN dataset, and put the decompressed `AdvertiseGen` directory into this directory.
+
+### Training
+Run the following commands for training:
+```shell
+bash train.sh
+```
+`PRE_SEQ_LEN` and `LR` in `train.sh` are soft prompt length and training learning rate respectively, which can be adjusted to achieve the best results. The P-Tuning-v2 method will freeze all model parameters, and the quantization level of the original model can be adjusted by adjusting `quantization_bit`. If this option is not added, it will be loaded with FP16 precision.
+
+Under the default configuration of `per_device_train_batch_size=1`, `gradient_accumulation_steps=16`, the model parameters of INT4 are frozen, and a training iteration will perform 16 cumulative forward and backward propagations with a batch size of 1, which is equivalent to the total batch size of 16, and only 6.7G GPU memory is required at this time with `quantization_bit=4`. If you want to improve the training efficiency under the same batch size, you can increase the value of `per_device_train_batch_size` while keeping the product of the two unchanged, but it will also bring more GPU memory consumption, please adjust it according to the actual situation.
+
+### Inference
+
+Change `CHECKPOINT` in `evaluate.sh` to the checkpoint name saved during training, and run the following commands for model inference and evaluation:
+```shell
+bash evaluate.sh
+```
+
+The evaluation indicators are Chinese Rouge score and BLEU-4. The generated results are saved in
+`./output/adgen-chatglm-6b-pt-8-1e-2/generated_predictions.txt`.
+
+### Example
+#### Example 1
+* Input: 类型#上衣\*材质#牛仔布\*颜色#白色\*风格#简约\*图案#刺绣\*衣样式#外套\*衣款式#破洞
+* Label: 简约而不简单的牛仔外套,白色的衣身十分百搭。衣身多处有做旧破洞设计,打破单调乏味,增加一丝造型看点。衣身后背处有趣味刺绣装饰,丰富层次感,彰显别样时尚。
+* Output[微调前]: 这件上衣的材质是牛仔布,颜色是白色,风格是简约,图案是刺绣,衣样式是外套,衣款式是破洞。
+* Output[微调后]: 这是一款简约的牛仔外套,破洞设计,将牛仔布破洞,带来一种随意与个性。破洞的牛仔外套,展现出时尚气息,带来一种休闲感。同时,刺绣图案,让整件外套更加立体。
+
+#### Example 2
+
+* Input: 类型#裙\*版型#显瘦\*风格#文艺\*风格#简约\*图案#印花\*图案#撞色\*裙下摆#压褶\*裙长#连衣裙\*裙领型#圆领
+* Label: 文艺个性的印花连衣裙,藏青色底蕴,低调又大气,撞色太阳花分布整个裙身,绚丽而美好,带来时尚减龄的气质。基础款的舒适圆领,简约不失大方,勾勒精致脸庞。领后是一粒包布扣固定,穿脱十分方便。前片立体的打褶设计,搭配后片压褶的做工,增添层次和空间感,显瘦又有型。
+* Output[微调前]: 类型#裙*版型#显瘦*风格#文艺*风格#简约*图案#印花*图案#撞色*裙下摆#压褶*裙长#连衣裙*裙领型#圆领 1\. 连衣裙:简约风格,裙长为膝盖以上,裙领型为圆领。2\. 裙下摆:压褶设计,使裙摆呈现出流畅的褶皱效果。3\. 裙领型:裙领型为圆领,使穿上连衣裙后更加有型。4\. 版型:采用显瘦设计,让连衣裙看起来更加苗条。5\. 风格:文艺风格,让连衣裙更加有内涵和品味。6\. 图案:印花设计,在连衣裙上印有独特的图案。7\. 撞色:采用撞色设计,让连衣裙在色彩上更加鲜明、���有层次感。
+* Output[微调后]: 这是一款文艺范的连衣裙,以印花为元素,采用简约的印花,既能够突出文艺气质,又能够展现简约风。在印花的同时又有领子和裙摆的压褶设计,更加凸显文艺气质。简约而不会过于单调,搭配出街,穿着十分舒适。
+
+### evaluation result
+
+| | P-tuning v2 | LoRA |
+| ------- | ----------- | ----- |
+| BLEU-4 | 7.71 | 6.13 |
+| Rouge-1 | 31.35 | 28.36 |
+| Rouge-2 | 7.19 | 4.38 |
+| Rouge-l | 25.17 | 17.54 |
+
+#### Experiment Settings
+
+  ```
+max_source_length=64
+max_target_length=64
+per_device_train_batch_size=1
+gradient_accumulation_steps=16
+max_steps=3000
+  ```
+
+##### P-tuning v2
+
+```
+pre_seq_len=128
+learning_rate=2e-2
+quantization_bit=4
+```
+
+##### LoRA
+
+```
+learning_rate=5e-4
+```
+
+The implementation uses [simple_thu_chatglm6b](https://github.com/yuanzhoulvpi2017/zero_nlp/tree/main/simple_thu_chatglm6b)
+
+
+
+## Model Deployment
+Replace `THUDM/chatglm-6b` in the corresponding demo or code with the path of the checkpoint after P-Tuning(in the example, `./output/adgen-chatglm-6b-pt-8-1e-2/ checkpoint-3000`). Note that the current fine-tuning does not support multiple rounds of data, so only the responses from the first round of the conversation are fine-tuned.
+
+## Use your own dataset
+Modify `train_file`, `validation_file` and `test_file` in `train.sh` and `evaluate.sh` to your own JSON format dataset paths, and change `prompt_column` and `response_column` to the keys in the JSON file corresponding to input text and output text.
+
+## TODO
+* [ ] Support for chat data
+* [ ] Support for full finetuning
+
+## quoting
+
+```
+@inproceedings{liu2022p,
+   title={P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks},
+   author={Liu, Xiao and Ji, Kaixuan and Fu, Yicheng and Tam, Weng and Du, Zhengxiao and Yang, Zhilin and Tang, Jie},
+   booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
+   pages={61--68},
+   year={2022}
+}
+```

ptuning/arguments.py

@@ -0,0 +1,224 @@
+from dataclasses import dataclass, field
+from typing import Optional
+
+
+@dataclass
+class ModelArguments:
+    """
+    Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.
+    """
+
+    model_name_or_path: str = field(
+        metadata={"help": "Path to pretrained model or model identifier from huggingface.co/models"}
+    )
+    ptuning_checkpoint: str = field(
+        default=None, metadata={"help": "Path to p-tuning v2 checkpoints"}
+    )
+    config_name: Optional[str] = field(
+        default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"}
+    )
+    tokenizer_name: Optional[str] = field(
+        default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}
+    )
+    cache_dir: Optional[str] = field(
+        default=None,
+        metadata={"help": "Where to store the pretrained models downloaded from huggingface.co"},
+    )
+    use_fast_tokenizer: bool = field(
+        default=True,
+        metadata={"help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."},
+    )
+    model_revision: str = field(
+        default="main",
+        metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."},
+    )
+    use_auth_token: bool = field(
+        default=False,
+        metadata={
+            "help": (
+                "Will use the token generated when running `huggingface-cli login` (necessary to use this script "
+                "with private models)."
+            )
+        },
+    )
+    resize_position_embeddings: Optional[bool] = field(
+        default=None,
+        metadata={
+            "help": (
+                "Whether to automatically resize the position embeddings if `max_source_length` exceeds "
+                "the model's position embeddings."
+            )
+        },
+    )
+    quantization_bit: Optional[int] = field(
+        default=None
+    )
+    pre_seq_len: Optional[int] = field(
+        default=None
+    )
+    prefix_projection: bool = field(
+        default=False
+    )
+
+
+@dataclass
+class DataTrainingArguments:
+    """
+    Arguments pertaining to what data we are going to input our model for training and eval.
+    """
+
+    lang: Optional[str] = field(default=None, metadata={"help": "Language id for summarization."})
+
+    dataset_name: Optional[str] = field(
+        default=None, metadata={"help": "The name of the dataset to use (via the datasets library)."}
+    )
+    dataset_config_name: Optional[str] = field(
+        default=None, metadata={"help": "The configuration name of the dataset to use (via the datasets library)."}
+    )
+    prompt_column: Optional[str] = field(
+        default=None,
+        metadata={"help": "The name of the column in the datasets containing the full texts (for summarization)."},
+    )
+    response_column: Optional[str] = field(
+        default=None,
+        metadata={"help": "The name of the column in the datasets containing the summaries (for summarization)."},
+    )
+    history_column: Optional[str] = field(
+        default=None,
+        metadata={"help": "The name of the column in the datasets containing the history of chat."},
+    )
+    train_file: Optional[str] = field(
+        default=None, metadata={"help": "The input training data file (a jsonlines or csv file)."}
+    )
+    validation_file: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": (
+                "An optional input evaluation data file to evaluate the metrics (rouge) on (a jsonlines or csv file)."
+            )
+        },
+    )
+    test_file: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": "An optional input test data file to evaluate the metrics (rouge) on (a jsonlines or csv file)."
+        },
+    )
+    overwrite_cache: bool = field(
+        default=False, metadata={"help": "Overwrite the cached training and evaluation sets"}
+    )
+    preprocessing_num_workers: Optional[int] = field(
+        default=None,
+        metadata={"help": "The number of processes to use for the preprocessing."},
+    )
+    max_source_length: Optional[int] = field(
+        default=1024,
+        metadata={
+            "help": (
+                "The maximum total input sequence length after tokenization. Sequences longer "
+                "than this will be truncated, sequences shorter will be padded."
+            )
+        },
+    )
+    max_target_length: Optional[int] = field(
+        default=128,
+        metadata={
+            "help": (
+                "The maximum total sequence length for target text after tokenization. Sequences longer "
+                "than this will be truncated, sequences shorter will be padded."
+            )
+        },
+    )
+    val_max_target_length: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": (
+                "The maximum total sequence length for validation target text after tokenization. Sequences longer "
+                "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`."
+                "This argument is also used to override the ``max_length`` param of ``model.generate``, which is used "
+                "during ``evaluate`` and ``predict``."
+            )
+        },
+    )
+    pad_to_max_length: bool = field(
+        default=False,
+        metadata={
+            "help": (
+                "Whether to pad all samples to model maximum sentence length. "
+                "If False, will pad the samples dynamically when batching to the maximum length in the batch. More "
+                "efficient on GPU but very bad for TPU."
+            )
+        },
+    )
+    max_train_samples: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": (
+                "For debugging purposes or quicker training, truncate the number of training examples to this "
+                "value if set."
+            )
+        },
+    )
+    max_eval_samples: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": (
+                "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
+                "value if set."
+            )
+        },
+    )
+    max_predict_samples: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": (
+                "For debugging purposes or quicker training, truncate the number of prediction examples to this "
+                "value if set."
+            )
+        },
+    )
+    num_beams: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": (
+                "Number of beams to use for evaluation. This argument will be passed to ``model.generate``, "
+                "which is used during ``evaluate`` and ``predict``."
+            )
+        },
+    )
+    ignore_pad_token_for_loss: bool = field(
+        default=True,
+        metadata={
+            "help": "Whether to ignore the tokens corresponding to padded labels in the loss computation or not."
+        },
+    )
+    source_prefix: Optional[str] = field(
+        default="", metadata={"help": "A prefix to add before every source text (useful for T5 models)."}
+    )
+
+    forced_bos_token: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": (
+                "The token to force as the first generated token after the decoder_start_token_id."
+                "Useful for multilingual models like mBART where the first generated token"
+                "needs to be the target language token (Usually it is the target language token)"
+            )
+        },
+    )
+
+    
+
+    def __post_init__(self):
+        if self.dataset_name is None and self.train_file is None and self.validation_file is None and self.test_file is None:
+            raise ValueError("Need either a dataset name or a training/validation/test file.")
+        else:
+            if self.train_file is not None:
+                extension = self.train_file.split(".")[-1]
+                assert extension in ["csv", "json"], "`train_file` should be a csv or a json file."
+            if self.validation_file is not None:
+                extension = self.validation_file.split(".")[-1]
+                assert extension in ["csv", "json"], "`validation_file` should be a csv or a json file."
+        if self.val_max_target_length is None:
+            self.val_max_target_length = self.max_target_length
+

ptuning/datasets/Zettels/dev.json

@@ -0,0 +1,21 @@
+{"content": "语料描述-为什么要进行低光增强？", "summary": "可见光图像的成像过程受光照强度及环境影响，低照度环境下采集的可见光图像存在信噪比、对比度、分辨率均较低等特点，给进一步图像处理，图像识别、目标检测等任务带来更严峻的挑战．"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "当前，人类在水下和低光环境中的活动越来越频繁，比如：目标检测、智能识别、水下探测等。因水下和低光图像具有很多的相似情况，因此，对于如何解决两者共同存在的问题成为了比较热门的话题"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "随着国家科技水平的发展和国民经济水平的提高，人们对于出行的要求也让汽车行业得以迅猛的发展.但是，汽车行业的迅猛发展，在方便人们的出行的同时，也导致了交通事故率的显著提高."}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "在有雾的天气下，大气中存在着很多悬浮的小液滴，导致所获取图像的可视性显著下降，会出现场景细节模糊、图像色彩衰减等问题。随着社会的发展，机器视觉被广泛应用于安全监控、目标识别、遥感成像、图像分类等领域，机器视觉的有效性通常建立在具有较好可视性的输入图像之上。因此，消除有雾天气产生的不良视觉效果，获得可视性良好的图像具有重要的意义。"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "随着智能算法的发展，图像识别技术越来越受到人们的关注，科研人员在训练识别模型的时候也希望得到更准确、更稳定的识别模型。但是，图像数据库的图像质量可能会受到各种各样外在因素的影响，如光照不均、相对运动和色彩失真等。对图像进行增强处理，可提高图像品质，丰富图像的信息量，便于计算机进一步理解图像。因此，彩色图像的增强处理技术在交通智能化、军事国防和医学等方面能够发挥相当重要的作用。"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "日常生活中经常需要在低光条件下捕捉图像，例如在夜间或昏暗的室内房间。在此环境下拍摄的图像往往会出现能见度差、对比度低、噪声大等多种问题。虽然自动曝光机制(如ISO、快门、闪光灯等)可以增强图像亮度，但同时也会产生其他的影响(如模糊、过饱和度等)。"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "受光照强度的影响，在夜间和背光条件下采集的图像往往含有较低的对比度、大面积的暗区域和明显的噪声污染。这些降质图像往往导致人们无法正确地辨识场景内容，也常常给图像检索、多媒体信息安全等后续计算机视觉任务带来严峻的挑战。因此，低照度图像增强具有重要的理论价值和现实意义，受到学界广泛关注。"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "在光线昏暗的环境下，摄影师拍摄出正常成像非常困难。在更极端的黑暗情况下，如光照严重受限(月光)或短时间的曝光，成像就更为困难。低光下拍摄的图像和极低光下拍摄的图像对比如图1所示，明显看出极低光下拍摄的图像相比低光下拍摄的图像被隐藏的信息更多。"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "在图像采集过程中，所在场景的光照条件往往是影响图像质量的重要因素之一，在现代社会生产生活中，人们采集图像变的更为方便和快捷，由于光照条件不足产生的低照度图像识别度不高，导致缺乏可用性，并对后续的图像处理、目标识别、语义分割等仸务造成了困难"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "可见光图像的成像过程受光照强度及环境影响，低照度环境下采集的可见光图像存在信噪比、对比度、分辨率均较低等特点，给进一步图像处理，图像识别、目标检测等任务带来更严峻的挑战"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "在很多计算机视觉任务中，如目标检测、图像检索、图像分割等，都要求输入图像亮度合适、细节清晰。然而，在弱光照或者曝光不足的情况下，采集到的图像存在亮度低、色彩不饱和、细节模糊等缺点，这些缺点将影响到来操作来都不像像后续的计算机视觉任务。因此，研究弱光照图像的增强很有必要。"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "随着计算机视觉领域的収展，携带着丰富信息的高质量图像无论是在日常生活还是科学研究中都有着巨大的研究潜力。但是，由于不同的光照条件、周围的噪声等原因，图像质量高低不一，严重影响了人们判别图片中的信息，从而引起不必要的冲突和结果。尤其是在黑暗条件下，人们难以识别摄像头捕捉到的图像信息，而且智能系统很大程度上也依赖于高质量的输入图像，为了解决这个问题，本文就低照度图像问题迚行了研究。"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "在光线昏暗的环境下，摄影师拍摄出正常成像非常困难。在更极端的黑暗情况下，如光照严重受限(月光)或短时间的曝光，成像就更为困难。低光下拍摄的图像和极低光下拍摄的图像对比如图1所示，明显看出极低光下拍摄的图像相比低光下拍摄的图像被隐藏的信息更多。"}
+{"content": "语料描述-为什么要进行低光增强？", "summary": "部分图像增强在计算机断层成像、工业产品质量检验、交通监控及卫星图像处理中有着广泛的应用。其中低照度图像是一种常见的图像种类,它的主要特点是微光、暗色区域占图像主要部分。造成图像低照度的原因有很多,例如光线不足,摄影设备性能有限以及设备配置不正确等,这类图像可见性偏低,不便于观察与分析,且会对相关应用产生负面影响,尤其是在图像的匹配、融合、分析、检测以及分割方面,给数字图像处理带来极大挑战。"}
+{"content": "结论   ", "summary": "本文提出了一种可以对夜间的油茶果果实图像进行快速、精准识别的目标检测网络YOLON，（1）可以对夜间的油茶果果实目标进行进行检测，mAP可以达到98.17%，高于YOLO v4、EfficientNet、YOLOX等对比算法的mAP，表明YOLON可以较好地对夜间油茶果的进行检测。（2）对不同遮挡程度下的（3）但是YOLON对的运行速度尚有待提高，此外运行时对GPU的依赖程度较大，无法在CPU上完成快速推理，对于嵌入式设备的部署尚不理想，这也是本文未来重点的研究方向。"}
+{"content": "单果、双果、多果、整树的识别效果 ", "summary": "结果表明，随着果实数量的增加，成功率呈下降趋势。在2个果实中，成功率为100%。三果、四果、五果串的成功率分别为91.6%、88.9%和85.3%。"}
+{"content": "检测效果对比  ", "summary": "使用YOLON、YOLOR、和YOLO V4在COA数据集上进行了进一步的比较， 本文提出的YOLON其在COA数据集上的表现优于YOLO V4 和YOLOR。"}
+{"content": "算法的运行时间 ", "summary": "随着不同分辨率的图像尺寸的增加，时间与mAP的变化，选取最适合的分辨率，可以花上去两个曲线，找两条曲线相交的那个点，类似于下面的效果"}
+{"content": "结果 ", "summary": "为了验证该方法的有效性，对COA数据集中216张测试图像进行了测试，结果如表1所示，该方法的准备率和召回率分别为90.00%和90.00%，平均IOU和mAP分别为90.00%和90.00%，检测速度可达30f/s，能够满足油茶果采摘机器人对果实进行实时检测的需求，可以为算法部署在油茶果采摘机器人的开发上提供技术支持。"}
+{"content": "本文要研究的重点问题 ", "summary": "夜间油茶果的果实目标识别"}
+{"content": "本文提出的基于多分支结构和U-net结合的低照度图像增强算法有以下三点贡献： ", "summary": "1)本文提出了一种新颖的端到端的低照度图像增强网络，可以应用于多种场景，计算速度和准确率也都有所提升。此网络还结合了多分支网络和U-net迚行特征提取，取得了不错的结果。2)本文的方法在噪声抑制、对比度增强等方面有着较好的效果，能够有效地减少噪声的影响。3)本文使用大量的实验来证明所提出模型的有效性，且使用了很多优秀的方法来进行对比，并且从定性和定量的角度分析对比结果，取得了满意的效果。总体来说，本文提出的斱法在各方面都很大程度上优于现有的算法。"}

ptuning/datasets/Zettels/train.json

@@ -0,0 +1,140 @@
+{"content": "目前国内外常见的低照度图像增强方法主要分为四种： ", "summary": "(1)基于直方图增强法(HistogramEqualization，HE).该方法通过整体调整图像直方图分布来增强图像亮度和对比度，此类方法精简快捷，但常出现颜色失真、细节丢失等问题."}
+{"content": "目前国内外常见的低照度图像增强方法主要分为四种： ", "summary": "(2)基于Retinex增强法.Land提出人眼视觉亮度与颜色感知由实际物体自身的反射率决定，与环境光强度无关.根据Retinex理论提出带色彩恢复的多尺度Retinex(multi-scaleRetinexwithcolorrestora-tion，MSRCR)和LIME等多种经典算法.此类方法容易出现颜色失真，虽然有学者增加颜色校正模块，但仍无法完全克服颜色失真问题."}
+{"content": "目前国内外常见的低照度图像增强方法主要分为四种： ", "summary": "(3)基于伪雾图增强法.该方法利用低照度图像的反转图像通过去雾算法进行增强.如Dong等提出增强方法取得较好的照度增强效果，但在应对复杂场景增强时容易出现块效应和噪声."}
+{"content": "目前国内外常见的低照度图像增强方法主要分为四种： ", "summary": "(4)基于神经网络的增强法.此类方法利用神经网络学习低照度图像到正常照度图像的映射，如刘超等提出利用卷积自编码网络从低照度图像训练集中学习图像特征.此类方法能够有效的对低照度图像进行照度增强，但增强的图像在细节及色彩方面有所欠缺."}
+{"content": "单纯地增加曝光量的问题 ", "summary": "相机的动态范围有限，如果增加相机曝光度来揭示曝光不足区域的信息，那么曝光良好的区域可能会出现过度曝光甚至饱和的现象。"}
+{"content": "单纯地增加曝光量的问题 ", "summary": "在图像拍摄过程中可以通过提高感光度(ISO)增加亮度，但不可避免地会放大噪音。采用缩放或直方图拉伸等一系列后期处理方法可以减弱噪音的影响，但并不能有效解决信噪比低 的问题。在拍摄过程中采用打开光圈、延长曝光时间、使用闪光灯等物理方法虽然能达到在低光环境下增加信噪比的效果，但图像会因为相机抖动或物体的移动而变得模糊。"}
+{"content": "说明别人方法不好 ", "summary": "为了解决这一问题，研究人员提出了一些有效的图像增强技术。采用这些方法虽然可以获到良好的主观质量，却不能准确地反映场景图像的真实亮度和对比度。"}
+{"content": "说明别人方法不好 ", "summary": "LIME虽整体图片偏亮，但是却与原色彩模式相比，存在失真的问题"}
+{"content": "深度学习算法的好处 ", "summary": "深度学习方法具有较高的 能力，同时可以通过训练样本的控制来提供较多的先验知识 来降低低照度图像重建问题的病态程度，可以针对特定的低照度数据集实现较好的增强效果。"}
+{"content": "油茶果描述 ", "summary": "油茶Camelliaoleifera是我国最主要的经济树种之一，与油棕、油橄榄和椰子并称为世界四大木本食用油料树种。因其良好的适应性，油茶适宜种植在我国南方广大的红壤丘陵地区，主要分布在湖南、江西、浙江、海南、广东、广西、重庆、四川、贵州、湖北、安徽、福建等地。其中湖南、江西和广西三省(区)合计栽培面积占全国总面积的75%以上，常年产茶油量占全国总产量的80%以上。据初步统计，截至2011年，全国已选育出的油茶新品种共有365个，其中通过国家审定的油茶良种有73个，省级油茶良种292个。品种繁多、良莠不齐，优良品种经过长期嫁接繁殖逐步退化等问题给油茶产业的发展带来了严重的消极影响。"}
+{"content": "油茶果描述 ", "summary": "油茶林主要分布于我国南方地区，其油茶果实经加工后，可广泛用于食品、工业、医药等重要领域，是重要的经济林树种之一。油茶树树叶繁茂，枝桠交叠，油茶果成熟时呈球形或椭圆形，颜色呈淡黄色或暗红色、或呈绿色，与树叶颜色相近。油茶果目标与背景成多元信息叠加，且受光照、摇晃等不稳定因素的影响，以上因素致使快速、准确地识别油茶果极其困难。在对林果采摘机器人视觉识别技术的相关研究中，目前主要集中于对目标的颜色。形态、纹理、光谱等单特征识别。在多特征识别研究中，Hayashi等人提出了基于形态学特征及颜色特征的茄子图像分割法，并采用网格模板获取完整的茄子目标;Zhao等人利用纹理特征和颜色特征识别树上苹果;Blasco等人结合多光谱特征和形态学特征，利用贝叶斯分类器检测柑橘表面;王津京等人利用支持向量机算法对待识别的目标苹果的颜色和形状特征进行了综合分类。"}
+{"content": "油茶果描述 ", "summary": "���积神经网络可以自动提取特征，并分类检测，精度高，实时性强，成为果蔬目标检测的主流框架。而Faster R-CNN 卷积神经网络经过RCNN、Fast-RCNN的不断改进，精度和检测效率都得到了进一步的提高。在卷积神经网络果蔬识别方面已经有大量的研究。西北农林科技大学冯亚利使用改进的DY3TNet模型实现了田间猕猴桃果实的检测。闫建伟等为了快速准确识别自然环境下刺梨果实，提出了一种基于改进的Faster-RCNN的刺梨果实识别方法。傅隆生等为了实现田间条件下快速、准确地识别多簇猕猴桃果实，提出了一种基于LeNet卷积神经网络的深度学习模型进行多簇猕猴桃果实图像的识别。程鸿芳等针对传统的基于内容的识别方法在特征提取方面存在的计算复杂、特征不可迁移等问题，对LeNet卷积神经网络结构进行改进，设计了一种基于改进LeNet卷积神经网络的苹果目标识别模型，并利用该模型对不同场景的苹果图像进行了识别与验证，综合识别率达到93.79%;与其他方法相比，该算法具有较强的抗干扰能力，图像识别速度快，识别率更高。中南林业科技大学张习之等提出了一种基于改进卷积自编码机神经网络的油茶果图像自动识别方法，该改进算法100次迭代所需时间为166s，平均识别准确率为90.4%。"}
+{"content": "低光增强深度学习算法使用的数据集 ", "summary": "CHEN等在2018年首次采用SID(see in the dark)数据集，基于数据驱动方法训练一个端到端的网络，实现了极端低光情况下图像的增强，取得了良好的效果，但该方法设计的网络对物体细节的还原仍然有着很大的不足，增强之后的图像中物体的边缘存在模糊现象，且该方法只在其构建的数据集上表现良好。"}
+{"content": "说明目前用的人很少 ", "summary": "目前在卷积神经网络识别油茶果方面的研究较少，尚无文献用Faster-RCNN卷积网络的方法识别油茶果。本文选用Faster-RCNN交替优化训练方法，并使用Faster-RCNN卷积神经网络对油茶果进行了识别。"}
+{"content": "最后一段说明自己要干啥 ", "summary": "为了提高低照度图像的图像质量，本文利用生成对抗网络的学习能力，提出一种基于U-Net生成对抗网络的低照度图像增强方法.该方法利用U-Net结构和深度卷积结构构造生成对抗网络，实现利用低照度图像生成照度与正常照度图像拟合的增强图像.实验表明，该方法能够有效提高低照度图像的亮度、对比度."}
+{"content": "低照度图像产生的硬件原因 ", "summary": "在低照度条件下拍摄的图像通常会出现不同程度的质量退化，虽然专业设备和先进的摄影技术可以在一定程度上缓解这些退化，但固有原因产生的噪声是不可避免的。由于没有足够的光，相机传感器的输出易受系统的固有噪声干扰，因此输出的图像可能会在曝光不足的区域丢失部分重要信息，从而加大了计算机视觉任务的难度。"}
+{"content": "深度学习发展很快 ", "summary": "近年来，深度学习发展迅速，在高层次视觉任务中应用非常广泛，如图像识别、语义分割等。与此同时，也有一些研究人员尝试用深度学习算法去解决低层次图像领域问题，如图像去噪、图像去雾、图像超分辨率等，这些算法也取得了较好的成绩。相对于传统算法，深度学习算法具有不需要人工设计特征提取方法，可直接端到端地训练和输出结果等优势"}
+{"content": "本文的贡献 ", "summary": "本文提出的基于多分支结构和U-net结合的低照度图像增强算法有以下三点贡献： ∑"}
+{"content": "引出我们研究的问题  ", "summary": "针对低照度条件下的图像对比度不高、颜色失衡和存在噪声的问题，提出了一种基于卷积神经网络的低照度图像增强模型。"}
+{"content": "引言中引出本研究的表述  ", "summary": "为了对低照度图像进行增强，本文利用卷积卷积神经网络的非线性映射能力，提出了一种基于U-Net网络的低照度图像增强方法。"}
+{"content": "引言中引出本研究的表述  ", "summary": "利用数据驱动的方式，实现了对低照度的图像进行增强，实验表明，该方法能够有效提高低照度油茶果图像的亮度、对比度，且细节可以保持完整"}
+{"content": "Dong的方法", "summary": "基于Dong的去雾算法增强。Dong等发现，低照度图像经过翻转后图像与雾天图像具有一定的相似性，因此可以采用图像去雾的思想对低照度图像进行增强，可以取得一定的增强效果。但由于Dong未考虑雾图中存在的白色区域所导致的暗通道理论失效的问题，在应对复杂场景增强时容易出现块效应和噪声等问题。基于伪雾图增强法。该方法利用低照度图像的反转图像通过��雾算法进行增强，如Dong等提出的增强方法取得了较好的照度增强效果，但在应对复杂场景增强时容易出现块效应和噪声等问题。"}
+{"content": "将注意力机制引入CNN结构中有什么好处  ", "summary": "将注意力机制应用于UNet分割网络中，可以比较好的实现对显著性区域的关注，以及对无关背景区域的抑制。"}
+{"content": "将注意力机制引入CNN结构中有什么好处  ", "summary": "注意力模型可以很好的嵌入到CNN框架中，而且不增加计算量的同时提高模型性能。"}
+{"content": "将注意力机制引入CNN结构中有什么好处  ", "summary": "提出了Attention-U-Net算法，该算法将通道注意力及空间注意力机制加入低光增强网络之中"}
+{"content": "CIL数据集的采集条件", "summary": "用于网络训练的低照度/正常图像对，于2020年10月25日-11月5日白天在湖南省天沃科技有限公司雷叔叔油茶果种植基地拍摄，拍摄的油茶果的品种为华顺203，205。图像采集设备为尼康D90数码相机，相机的光圈为F22，快门模式为M模式，ISO为200，拍摄设备如图1所示，尼康相机固定在三脚架上，三角架轴心到树干中心的距离为200cm，相机镜头到油茶树最外侧的距离为95cm，相机距离地面的高度为150cm；在对同一场景同一位置的油茶果采集不同曝光度的图像时，若采用手动按快门进行拍摄，会不可控地引起相机镜头的抖动，从而造成拍摄的不同曝光度的图像，无法达到像素级的对应，对后续网络的训练造成较大干扰，因此本研究在制作油茶果的低照度/正常光照图像的数据集CIL（Camellia oleifera Abel Image in low light）时，采用红外遥控的方式的进行图像的采集2021.01.25，其中红外遥控器的发射端由采集人员手持，红外遥控器的接收端置于单反相机的上方，两者通过热靴连接。"}
+{"content": "注意力机制引入的区域  ", "summary": "attention模块用在了skip connection上，原始U-Net只是单纯的把同层的下采样层的特征直接concate到上采样层中，改进后的使用attention模块对下采样层同层和上采样层上一层的特征图进行处理后再和上采样后的特征图进行concate"}
+{"content": "注意力机制引入的区域  ", "summary": "Attention coefficients（取值0~1）与feature map相乘，会让不相关的区域的值变小（抑制），target区域的值变大（Attention）。"}
+{"content": "注意力机制引入的区域  ", "summary": "利用下采样层的结构化信息和当前层纹理信息的融合，利用sigmoid归一化，得到关联性强的区域，和当前层做乘积，从而强调本层的显著性区域的特征。"}
+{"content": "注意力机制引入的区域  ", "summary": "在基础的UNet的基础上增加了attention 的机制，通过自动学习参数来调整激活值，attention的可视化效果还是主要部分，不像non-local的方式每一个像素点都要和其他像素点进行关联，可以视作一种隐式的注意力机制。"}
+{"content": "我们的方法如何好 ", "summary": "通过大量的实验表明，运用深度残差网络和U-net，可以更好地进行特征提取，低照度图像增强的效果也更好，很大程度上优于现有的技术。提出的方法不仅在视觉上提高了亮度和对比度，色彩更真实，更加符合人眼视觉系统特性，而且PSNR、SSIM等七项客观图像质量指标在几种算法中都是最优的。"}
+{"content": "我们的方法如何好 ", "summary": "3)本文使用大量的实验来证明所提出模型的有效性，且使用了很多"}
+{"content": "我们的方法如何好 ", "summary": "其可对不同尺度的低照度图像进行增强"}
+{"content": "我们的方法如何好 ", "summary": "通过在公开数据集（LOL，SID）上验证表明，RUNet方法在效果上有所改进，尤其是整体视觉效果。"}
+{"content": "我们的方法如何好 ", "summary": "不仅在客观评价指标上的评分更高，PSNR、SSIM等七项客观图像质量指标在几种算法中都是最优的，而且在视觉上提高了亮度和对比度，色彩更真实，更符合人眼的视觉原理。"}
+{"content": "合成数据集 ", "summary": "我们从RAISE收集了780张原始图像，700张用于生成训练对，80张用于网络的验证。使用Adobe Photoshop Lightroom调整对应的参数，包括曝光度、振动和对比度，具体参数为：曝光参数E设置为了[-5,0]，振动参数V设置为[-100,0]，对比度参数C设置为[-100,0]来合成低照度图像。为了防止颜色偏差的问题，在训练数据集中加入了700对灰度图像对。这些灰度图像对被转换为彩色图像对。为了使增强前后的黑白区域保持一致，我们添加了五对白-白训练对和五对黑-黑训练对。最后，将所有图像的大小调整为400x600，并转换为JPG格式。"}
+{"content": "合成数据集 ", "summary": "通过伽玛校正来合成数据集，但对于伽玛校正，可能无法反映不同曝光级别之间的关系。"}
+{"content": "CIL数据集的明细  ", "summary": "CIL数据集，分别拍摄了单果、多果、整树、花朵、花和果实、无果等多种油茶果图像场景的低照度/正常图像对，以确保算法对不同的场景的低照度图像的增强效果。CIL数据集用较短的快门时间拍摄的油茶果图像做为低照度图像，较长的快门时间拍摄的图像做为对应的参考图像，一共包含1200组不同曝光度的图像，各个场景下的具体图像数量如表1所示，曝光时间分别为1/1000s、1/500s、1/250s、1/60s、1/10s，其中做为低照度图像的曝光时间分别为1/1000s、1/500s、1/250s、1/60s，每组图像对应的参考图像的曝光时间为1/10s。"}
+{"content": "为什么要使用U-Net", "summary": "在数据集规模较小的情况，使用U-Net进行烟尘分割。相比于其他语义分割网络，U-Net的优点在于能针对小型数据集进行端到端的快速、有效训练。"}
+{"content": "存在的问题 ", "summary": "油茶果处于高山丘陵地区，因此山上多风，所以对在非同一时刻采集到的油茶果图像可能会存在轻微的晃动，"}
+{"content": "U-Net的结构描述", "summary": "U-net网络结构因其对称结构与英文字母“U”相似而得此名，主要由下采样、上采样以及“桥”连接三部分组成，其网络结构见图2。左边称为下采样，也叫压缩路径或编码器(encoder)，主要作用是提取图像的浅层特征，如图像的位置信息;右边称为上采样，也叫扩展路径或解码器(decoder)，主要作用是提取图像的深层特征，如图像中像素的类别信息;中间的箭头表示“桥”连接，也叫跳跃连接(skipconnection，SC)，主要作用是把下采样得到的特征图与上采样得到的特征图进行复制拼接，形成一个同时具有深层次和浅层次信息的特征图，实现更为有效的分割。除此之外，上、下采样和“桥”连接部分由卷积层、池化层、反卷积层以及激励层组成。其中卷积层用于提取甲状腺图像的特征;池化层用于下采样部分，其将获取的特征数据和参数进行压缩，减少模型过拟合;反卷积层用来还原特征图的尺寸大小，使得最后输出大小与原图大小一致;激励层是将卷积所得的输出进行一个非线性映射，来保证模型能够更好地拟合图像。本研究所使用的激活函数为ReLU，最后输出层使用Sigmoid函数对像素点进行分类"}
+{"content": "对U-Net增强效果的描述", "summary": "对单通道的图像进行伪彩色处理，可以直观地看到，经过U-Net增强后的图像，其整体照度得到了增强，而且，各个通道的轮廓和细节信息也得到了增强，变得更为突出。例如R通道，增强前的树叶、树枝等信息耦合在一起，无法区分，而增强后，两者的边界则变得非常显著，有利于后续图像的进一步分析。"}
+{"content": "对U-Net增强效果的描述", "summary": "自己的U-Net代码是对图像进行了端对端 的处理，不仅仅是可以对图像进行低照度增强，还进行去噪，"}
+{"content": "我们提出的网络", "summary": "受SID算法的启发，本文验证了利用神经网络对低照度图像进行端到端的增强，并提出了一个有连续的自网络组成的新型网络RV-UNet。"}
+{"content": "我们提出的网络", "summary": "通常原始输入图像的比编码-解码网络的输出包含更多细节，因为可以为恢复细节提供信息。采用级联代替跳跃连接，将上一个上采样块的特征映射与输入图像相结合，使原始信息和照明估计都能完全保留并传输到下一步。级联层之后是带有ReLU的三个激活层，它将输入的图像信息与估计的全局光照信息想结合，最终生成细节更好的增强效果。"}
+{"content": "多级", "summary": "在训练过程中，首先针对不同曝光时间的图像乘上对应的放大系数进行粗增强，，该系数不参与网络训练，由研究人员指定；而在网络测试时，模型先由输入图像的直方图进行亮度模式判定，根据先验知识分别乘上对应的放大因子，对图像进行粗调整后，再输入到A-UNet网络中进行端到端处理 。"}
+{"content": "油茶果图像的采集", "summary": "本研究采集油茶果的试验地点为湖南省永州市雷叔叔油茶果种植基地，拍摄时间为2020年10月28日-2020年11月6日的白天，，"}
+{"content": "油茶果图像的采集", "summary": "江西省林业科学院国家油茶林基地。拍摄相机为索尼相机，像素为640×480。于2019年9月14~23日的晴天拍摄，采集了典青、赣兴46、赣抚20等34个品种的油茶果图像并保存成JPG的格式。图1为拍摄的部分油茶果图像。环境下的油茶果图像，将训练集图像数量扩充到3820张，使卷积神经网络学习到各种情况下油茶果图像的特征(若训练数据集中没有包含多样化的样本，则会导致机器学习不足，识别结果置信度降低)。"}
+{"content": "试验环境", "summary": "本试验在Windows图形工作站计算机上进行，处理器为Intel-i5-9400F，内存为32GB，操作系统Windows10(64位)。考虑到GPU算力的需要，选用显卡为RTX 2080Ti，显存11GB。Python的版本是3.6.4，在pycharmIDE上编译。深度学习框架选择PyTorch。同时为了提高训练速度，采用GPU加速方法，cuda版本是8.1，cudnn版本为7.6.0。"}
+{"content": "U-Net的优点", "summary": "由于U型网络可以较好地保留图像的细节，为了学习低光图像和正常光照图像之间的映射关系，本文采用U型结构的卷积神经网络。如图3所示，该网络由编码器、解码器和跳跃连接构成。"}
+{"content": "实验结果的图表", "summary": "各个算法对应多个评价指标，使用多个图表"}
+{"content": "实验结果的图表", "summary": "在一个图表上对应上所有的方法和评价指标"}
+{"content": "说明本文的做法好", "summary": "以上数据表明本文算法在低照度图像的增强效果以及运行效率上均有一个较好的表现，为自然环境下油茶果的检测和识别提供了保障。"}
+{"content": "说明本文的做法好", "summary": "大量的实验表明，该方法不仅可以获得视觉上令人满意的低光增强效果，而且能够很好地表示图像的分解过程"}
+{"content": "说明本文的做法好", "summary": "我们进行了大量的实验，以证明我们设计算法的有效性及相对于常见低照度增强算法的优越性。此外，我们算法在2080Ti GPU上花费不高50ms来处理VGA分辨率的图像，这些特性使得我们的产品具有是有价值。"}
+{"content": "针对于本文不利的指标进行申辩", "summary": "本文所提的增强算法，尽管在时间效率上低于LIME和HE等传统算法，但是增强效果上，则远远高于这些算法。表明本文算法具有良好的实时性和准确性，为后续的计算机视觉的准确识别，有助于提高油茶果采摘机器人采摘作业的准确性和效率。"}
+{"content": "我们算法的增强效果", "summary": "实验结果表明，与传统的低照度增强算法相比，本文所以提出的模型，不仅可以最大程度地还原图像的真实亮度，而且能够有效提高图像的对比度、调整颜色失衡已经去除噪声，客观图像质量评价指标也高手同类算法。"}
+{"content": "我们算法的增强效果", "summary": "实验结果表明，本文所提算法不仅在主客观评价上有较好的表现，而且利用增强后的图像训练出来的YOLO v4网络比未经过增强处理的图像训练出来的YOLO  v4网络识别准确率更高。"}
+{"content": "我们算法的增强效果", "summary": "该增强算法是利用神经网络进行端到端的训练，参数调整过程中没有任何人工参与，大大减少了人工设计的复杂性。"}
+{"content": "我们算法的增强效果", "summary": "实验表明，本文算法，不仅在合成图像中表现出色，而且"}
+{"content": "我们算法的增强效果", "summary": "本文分别从定性和定量的角度将RV-UNet与流行的低照度增强方法进行了比较，结果表明，本文算法不仅在各个增强指标上表现出了，而且也最符合人眼特性，xxx效果最好。"}
+{"content": "我们算法的增强效果", "summary": "综上所述，RV-UNet的增强效果优于LIME等传统增强算法，而且与Retinex-Net等深度学习增强算法相比，图像的主客观增强效果均表现出色。但本文方法也存在一些问题，例如对一些区域进行处理时存在过度增强，从而引起亮斑问题。"}
+{"content": "我们算法的增强效果", "summary": "与其他算法相比，本文提出的算法可以有更生动和自然的结果，由于RV-UNet对输入图像具有全局感知能力，并且可以按照语义信息对整个图像进行增强，因此可以避免在正常或者较亮区域的过度曝光或者在较暗区域的曝光不足。此外，低照度图像的细节了增强后仍然可以保持不变，得益于细节重构步骤。"}
+{"content": "损失函数", "summary": "本文实验选择低照度增强公开基准数据集LOL进行训练的，该数据集包含1500组低光/亮光图像对。具体来说，有500对大小为400×600的真实场景图像和1000对大小为384×384的原始数据合成图像。本文使用1000对合成图像对和485对真实图像对进行训练，其余15对进行测试。由于LOL测试集的图像是在极弱光条件下拍摄的，图像的黑暗区域充满了强烈的噪声，因此通过这个数据集的结果可以很好地显示了本文算法在低光照条件下的增强性能。"}
+{"content": "常见的图像去噪算法", "summary": "最流行的方法可以归纳为BM3D 和WNNM 。由于测试中优化过程的高度复杂性，以及参数需要人工进行精心设计，这些传统去噪算法���实际应用中往往表现不是很出色。"}
+{"content": "人们对于图像去噪的态度", "summary": "在图像处理、多媒体、计算机视觉等领域，图像去噪一直是人们关注的热点，近几十年来提出了很多经典算法。"}
+{"content": "数据驱动", "summary": "Retinex分解图像的方法是直接在输入图像上估计反射率和照度，但设计适合各种场景的适当约束函数并不容易。因此，尝试以数据驱动的方式解决此问题。DeNet每次都会获取成对的弱光/正常光图像，并在弱光和正常光图像的指导下学习弱光及其对应的正常光图像的分解。"}
+{"content": "基于深度学习的去噪算法", "summary": "基于深度学习的去噪算法显示出了优越性，有代表性的工作，如使用堆叠稀疏去噪自动编码器的SSDA 、可训练非线性反应扩散的TNRD、具有残差学习和批处理归一化的DnCNN，由于"}
+{"content": "编码-解码描述", "summary": "在编码-解码网络中，首先是由编码网络对图像进行编码，然后将图像的全部信息压缩为瓶颈层的细长向量，最后由解码网络解码进行解码。"}
+{"content": "时间复杂度", "summary": "评价一个算法的优劣，一般主要从算法的执行时间（计算量）和所需要占用的存储空间（访问量）两方面进行衡量，但时间复杂度计算的不是程序具体运行的时间，而是算法执行语句的次数。"}
+{"content": "评价增强效果", "summary": "客观评价方式是通过峰值信噪比、信息熵、标准差等参数对低照度图像的增强效果进行定量分析。"}
+{"content": "评价增强效果", "summary": "使用PSNR评价指标，偶尔会出现评价结果与主观评价不一致的情况，这是由于该指标"}
+{"content": "评价增强效果", "summary": "PSNR在对图像进行评价的时候，只是对图像的绝对质量进行数字打分，而忽略了人眼的主观感受，例如人眼对一个区域的感知结果会受到其周围邻近区域的影响，因此常常会出现PSNR分数与主观评价不一致的情况。"}
+{"content": "评价增强效果", "summary": "，使用PSNR对图像进行评价，有时候会出现评价分数与主观评价不一致的情况，"}
+{"content": "评价指标", "summary": "本文在测试数据集上对所提出的网络性能进行了评估，并分别将其与文献、文献和文献三种主流低照度图像增强的方法进行了定性和定量的比较。为了公平起见，本文应用了作者提供的带有推荐参数设置的代码。为了评价这些算法的性能，本文采用峰值信噪比(PSNR)、结构相似性指数(SSIM)和自然图像质量评估器(NIQE)来量化增强后图像的恢复质量。"}
+{"content": "评价指标", "summary": "PSNR是一个绝对误差，使用像素相对于其最大可能值的均方误差来计算。在假设人类的视觉系统高度协调以提取结构信息的情况下，SSIM试图通过更紧密地与人类的感知保持一致来改进绝对误差度量。这两个客观评价指标的值越大，表明图像处理效果越好，而NIQE值越大说明与自然图像差距越大，质量越差。可以从表1中看出本文网络表现出最优的性能。"}
+{"content": "空间复杂度", "summary": "空间复杂度是对一个算法在运行过程中临时占用存储空间大小的量度。在深度学习中，空间复杂度决定了模型的参数数量，模型的参数越多，训练模型所需的数据量就越大，而现实生活中的数据集通常不会太大，这会导致模型的训练容易过拟合"}
+{"content": "使用Google Cloud Vision 进行检测", "summary": "这个东西相对来说会客观一点儿，大家都是认这个东西的"}
+{"content": "使用Google Cloud Vision 进行检测", "summary": "图3为谷歌云识别低照度/正常光照图像对，原始图像来自于MEF数据集。由于光照较低，谷歌云视觉只能将图像标记为“天空”、“云”和“尖顶”，经过增强后，前景埃菲尔铁塔被成功地检测到，并用一个绿色的boundingbox精确地标记出来，表明了我们方法的有效性"}
+{"content": "缺乏数据集", "summary": "由于在现实世界中，获取可以做到像素级匹配的低照度/正常图像对的难度较大，早期采用深度学习算法对低照度图像进行增强的研究中，多是采用人工合成过的低照度图像数据集。然而，这些低照度图像通常是由研究者们通过一些已知参数的具体算法，对正常光照条件下的图像进行随机变暗处理，以及增加随机噪声等方式获得。与真实场景下采集到低照度图像相比，这类人工合成的图像，往往过于简化，缺乏真实场景下的复杂的噪声与图像失真。"}
+{"content": "基于深度学习的低光增强方法", "summary": "由于深度学习网络对非线性映射有较好的拟合作用，近年来研究者相继尝试了使用深度学习来对低照度图像进行增强的方式。"}
+{"content": "基于深度学习的低光增强方法", "summary": "这类算法能够利用神经网络学习低照度图像到正常照度图像之间的非线性映射，例如Lore等人最早提出了将LLNet网络用于处理低照度图像的"}
+{"content": "基于深度学习的低光增强方法", "summary": "基于伪雾图增强法。该方法利用低照度图像的反转图像通过去雾算法进行增强，如Dong等提出的增强方法取得了较好的照度增强效果，但在应对复杂场景增强时容易出现块效应和噪声等问题。"}
+{"content": "基于深度学习的低光增强方法", "summary": "随着深度学习的出现，许多低级视觉任务都从中受益，例如[14]、[15]用于去噪，[16]用于超分辨率。"}
+{"content": "基于深度学习的低光增强方法", "summary": "RetinexNet"}
+{"content": "基于深度学习的低光增强方法", "summary": "最近出现的做法"}
+{"content": "防止卷积后图像变小", "summary": "为了避免卷积时对图像边缘造成影响，丢失图像边缘，每个卷积层在进行卷积操作前都会对图像的边缘进行0填充，是的图像在卷积前与卷积后的大小可以保持一致。"}
+{"content": "为什么不能直接对低照度图像进行亮度增强", "summary": "在低照度条件下拍摄的图像通常xxx质量很差，这是因为除了不理想的光照条件外，还存在着多种类型的图像退化，例如噪声和颜色失真，都一起被隐藏在图像中，因此仅仅提高低照度图像的亮度将不可避免地放到这些噪声，使图像产生伪影和失真。"}
+{"content": "为什么不能直接对低照度图像进行亮度增强", "summary": "不同低照度区域具有不同程度的噪声，如果直接对低照度图像进行亮度增强的话，也会不可避免的放大这些噪声，使图像产生伪影和失真"}
+{"content": "闪光灯", "summary": "闪光灯是点状光源，而自然光是平行光源，因此使用闪光灯辅助成像有可能会产生不自然的曝光，而且使用闪光灯可以在一定程度上使环境变得明亮，但也会了成像中引入高光和不平衡的照明，使图像在视觉上缺乏真实感。"}
+{"content": "拍出低照度图像的场景", "summary": "由于油茶果生长环境的特殊性，油茶果采收机器人在作业时，视觉系统并不是在所有时刻和地点都可以获得良好的光照条件，例如黄昏、阴天、雾天等光照不足的情景下，甚至在晴天的一些背光处也会有欠曝光的现象。在这种光照条件下直接捕获的图像，往往具有较低的信噪比，容易对后续机器视觉的目标检测等工作造成较大干扰。"}
+{"content": "拍出低照度图像的场景", "summary": "造成低照度图像的原因有很多，例如拍摄环境光照条件较差、摄影设备性能有限以及设备配置不当等。"}
+{"content": "必要性", "summary": "油茶果采摘机器人的工作环境复杂，尤其是在阴天、傍晚，采集到的果实图像存在整体偏暗、模糊、对比度不高、细节不清晰、动态范围压缩有限等问题，给后续油茶果的自动化采收带来了较大困难。"}
+{"content": "必要性", "summary": "从低光环境下采集的油茶果图像，经常存在对比度低、细节丢失、噪声污染严重等问题，不利于人眼的观察和计算机视觉的检测。"}
+{"content": "必要性", "summary": "图像质量与很多计算机视觉相关的相关技术的效果息息相关，高质量的图像可以带来更多信息，方便后续的增强任务。油茶果机器人作业时的很多因素都会直接或者间接地影响图像质量，低光照便是其中之一。"}
+{"content": "必要性", "summary": "低质量的图像会降低很多计算机视觉的性能，因为这些算法，通常是针对高质量输入图像设计的，因此对低照度图像进行增强，不仅可能图像的视觉效果，还可以提高相关视觉算法的作业效果。"}
+{"content": "必要性", "summary": "在拍照成像中，光照不足，会非常明显的影响成像质量，使其对比度降低并且丢失细节信息，不仅影响视觉效果，而且会给后续为自然光照图像设计的计算机视觉系统的性能造成较大影响。为了使这些隐藏在低照度区域的细节清晰可见，提高计算机视觉系统的准确性，需要对低照度图像进行增强。"}
+{"content": "延时摄影", "summary": "长时间曝光延长了拍摄时的曝光时间，可以让更多的光子到达成像设备的感光元件上，但是仅限于静态摄影，若是物体在拍摄过程中发生了位移，很可能导致成像结果模糊"}
+{"content": "目的", "summary": "研究低照度增强的主要目的在于提升图像亮度的同时，降低图像噪声、减小色彩偏差、增强图像整体与局部的对比度，以增强图像的视觉效果与质量，例如锐化图像特征，使图像具有更高的视觉质量。低照度图像增强的目的是"}
+{"content": "目的", "summary": "低照度图像增强的目的是改善低光照条件下的成像图像质量与视觉效果，同时减小对光照以及拍摄设备的依赖程度，具有广大的应用前景。"}
+{"content": "发展水平", "summary": "国内外很多学者，对低照度图像增强技术进行了相关研究，取得了不错的效果，但大多是针对通用场景的增强，针对农业领域里面的低照度图像进行增强的研究相对较少。"}
+{"content": "发展水平", "summary": "近年来，很多学者对低照度增强领域进行了研究，使低照度增强技术有了较大的发展，但是开发一种可以用于实际领域的低光增强策略仍然面临着较大挑战，因为常见的低光增强算法往往受限于特定的场景，增强效果只是限制在这个特定领域里面，换另外一个场景有可能就不再适用了。XXX"}
+{"content": "目标", "summary": "理想的低照度图像增强算法，不仅应该可以对低照度图像进行增强，而且还可以有效地去除隐藏在暗区域中的图像，并且灵活地调整图像的增强等级。"}
+{"content": "应用", "summary": "低照度图像增强的目的是改善低光照条件下的成像图像质量与视觉效果，同时减小对光照以及拍摄设备的依赖程度，具有广大的应用前景。"}
+{"content": "应用", "summary": "RV-UNet主要应用场景为帮助提高其他计算机视觉任务的性能，比如物体检测和识别。由于大部分视觉识别模型都基于自然"}
+{"content": "难度所在", "summary": "对于大部分低照度图像，对图像进行简单的调整，并不能同时提升图像的亮度与质量。xxxx（中间省略很多具体算法的做法）这些难以调和之处正是低照度增强算法的难点所在。"}
+{"content": "难度所在", "summary": "很难确定精准的Ground Truth∑"}
+{"content": "难度所在", "summary": "深度学习应用于低照度图像增强的难度所在∑"}
+{"content": "深度学习应用于低照度图像增强的难度所在", "summary": "如何从单个图像中有效地估计出照明分量，并灵活地调整光照水平在对低照度区域进行增强后，如何消除之前隐藏在暗区域中的噪声和颜色失真等变换如何只通过少量的数据集训练一个没有Ground Truth的低照度增强网络？"}
+{"content": "很难确定精准的Ground Truth", "summary": "从用户的角度考虑，不同的人、不同的需求场景可能需要不同的图像亮度值，对于研究人员而言，很难精准确定一个适用于所有人的Ground Truth。"}
+{"content": "直方图均衡化 ", "summary": "在常规的图像增强任务中，直方图均衡化被广泛地应用到了各种图像增强任务中，通过对图像的直方图进行变换，得到从当前像素值。"}
+{"content": "HSV颜色空间", "summary": "HSV色彩空间的H通道采用环形数据表示，在该通道上设计增强网络的损失函数难度较大，"}
+{"content": "为什么要将低照度图像转换到不同的颜色空间中进行处理", "summary": "低光照图像与自然光照图像之间的主要区别为：图像的亮度和色彩的偏移，而亮度和色度则可以很好地反映出两者之间的差别。 因此将图像转换到YCbCr 颜色空间中，从而得到一个比在RGB颜色空间更适合改变图像亮度和色度的调整模型。"}
+{"content": "YCbCr颜色空间", "summary": "图像在YCbCr颜色空间的数字矩阵，相比于在RGB颜色空间的数字矩阵，更适合进行低照度增强处理，可以帮助提升模型的性能。"}
+{"content": "直方图的作用  ", "summary": "直方图可以统计数字图像中具有不同像素值的像素数量，图像直方图描绘了图像中像素值的分布情况。直方图被广泛应用于各种图像增强任务中，通过对图像的直方图进行变换，得到从当前像素值到新的像素值的直接映射。这类方法中最经典最常用的是直方图均衡化，但是直方图均衡化不会对图像的内容进行判断，只是在单纯地对所有的像素值进行计算映射，容易放大原始图像中的噪声。同时直方图均衡会使图像的平均亮度保持在像素值的动态范围中间，这会破坏一些场景的整体亮度。"}
+{"content": "局部直方图  ", "summary": "一些方法使用局部直方图均衡来避免破坏图像整体的平均亮度，但容易导致一些边界问题，而使得图像出现Checkerboard效应 等伪像。"}
+{"content": "传统算法", "summary": "LIME的色彩保持度较好，但是存在过度增强的问题，细节处会有损失。"}
+{"content": "传统算法", "summary": "Wang等人提出了一种称为NPE的算法，这种算法可以在增强对比度的同时保持照度的自然性。"}
+{"content": "传统算法", "summary": "FU等人提出了一种算法，该算法通过融合最初估计的光照图的多个导数来调整图像亮度，但这种方法往往会牺牲包含丰富纹理区域的真实感。"}
+{"content": "传统算法", "summary": "Guo等考虑从最初的结构光照图估计结构光照图。这些方法通常假定图像无噪声和颜色失真，并且不明确考虑退化。"}
+{"content": "SID ", "summary": "chen等人提出了一种可以用于处理低照度图像的算法，该算法是基于全卷积网络的端到端训练，可以同时处理噪声和颜色失真。但该算法受限于特定格式的数据集，如果修改网络以接收JPEG格式的数据（这个说法是原文自己说的），性能将会显著下降。"}
+{"content": "伽马变换 ", "summary": "伽玛以非线性的方式对每个像素都进行非线性映射，虽然可以提高亮度，尤其是较暗区域的亮度，但是却没有考虑单个像素与其相邻像素之间的关系，因此增强效果往往会失真。"}
+{"content": "Retinex", "summary": "Retinex理论的关键假设是图像可以分解为两个分量，即反射和照明。早期的算法包括单尺度的Retinex（SSR）和多尺度的Retinex（MSR），但是其结果通常看起来不自然，并且在某些地方存在过度增强。"}
+{"content": "Retinex", "summary": "基于深度学习的算法∑"}
+{"content": "基于深度学习的算法", "summary": "shen等人认为多尺度Retinex等价于具有不同高斯卷积核的前馈卷积神经网络，受此启发，他们构建了一个卷积神经网络（MSR网络）来学习低照度图像和正常光照图像之间的端到端映射。ΞWei等人设计了一个深度网络，称为RetinexNet，Retinex-NetΞKindling the Darkness"}
+{"content": "传统Retinex算法的限制所在", "summary": "虽然这些算法在特定情况下会有较好的增强效果，但是对于不同场景的低照度图像增强，每个场景都需要进行人工进行精心的参数设计，费时费力，实际应用价值相对较弱。"}
+{"content": "传统Retinex算法的限制所在", "summary": "现在大多数基于Retinex的方法都为这种高度不适定分解精心设计了约束和参数，而这些约束和参数在应用于其他场景时可能受到模型容量的限制，从而无法取得良好的效果。"}
+{"content": "Retinex-Net", "summary": "在Retinex理论的指导下，我们设计了一个Deep Retinex-Net网络来一起完成反射/照明分解和弱光增强。网络由三个步骤组成：分解、调整和重建。在分解步骤中，Retinex网络通过Decom网络将输入图像分解为R和I。在训练阶段，同时将低光/正常光照图像输入到网络，而在测试阶段，则仅将低照度图像作为输入。在低/正常光图像具有相同反射率和照明平滑度的约束下，Decom网络通过数据驱动的方式，对反射图的分解参数进行学习，使得不同照明图像分解的反射图R保持一致。在调整步骤中，使用Enhance-Net来使照明变亮。Enhance-Net采用了编码-解码网络的总体框架。采用多尺度级联保持当对图像进行亮度调整时，使全局与局部之间保持语义上下文的连贯性。此外，如果需要，通常会将在弱光条件下产生的放大噪声从反射图中去除。然后，在重建阶段，我们将调整后的照明图和反射图通过元素的乘法结合起来，从而得到最终的低照度增强效果图。"}
+{"content": "shen等人认为多尺度Retinex等价于具有不同高斯卷积核的前馈卷积神经网络，受此启发，他们构建了一个卷积神经网络（MSR网络）来学习低照度图像和正常光照图像之间的端到端映射。", "summary": "L. Shen, Z. Yue, F. Feng, Q. Chen, S. Liu, and J. Ma, “Msr-net:low-light image enhancement using deep convolutional network,”p. arXiv , 11 2017."}
+{"content": "未来的研究方向", "summary": "本文未来的研究方向是：1.增加算法对不同场景的鲁棒性，将算法的应用场景扩展到雾天、雨天等背景中 2.对算法的增强效果进行合理化约束，避免对非低照度区域进行增强，从而导致增强效果过亮的问题。"}
+{"content": "未来的研究方向", "summary": "反光部分，也会被放大"}
+{"content": "批标准化", "summary": "当前实验实验使用的增强网络包含了6个卷积层，我们在其前5个卷积层的激活层后再加入批标准化层，使得目前的CNN结构由Conv-Relu变为Conv-Relu-BN结构。"}
+{"content": "对网络图的描述", "summary": "图中的K表示卷积核的尺寸，K3即表示卷积核尺寸为3x3，Conv表示卷积层，Conv前的数字表示卷积层中卷积核的个数，"}
+{"content": "低光照图像与正常光照图像的区别所在", "summary": "低光照图像与自然光照图像之间的主要区别为：图像的亮度和色彩的偏移，而亮度和色度则可以很好地反映出两者之间的差别。"}
+{"content": "对颜色的影响", "summary": "在低光照条件下，亮度和色度受到的损耗程度是不同的，亮度通道相比于色度通道受到了更多的全局损耗。"}
+{"content": "实验条件及参数设置", "summary": "本实验使用PyTorch 深度学习框架实现网络，在显存为11G的RTX 2080Ti上训练。使用的SID 数据集包含5094个原始的短曝光极低光图像，每个极低光图像均有对应的长曝光参考图像。长曝光参考图像I^{Hgt}高斯滤波 缩小4倍得到低分辨率图像I^{Lgt}，图像宽度W和高度H均为512，每批次输入1张图片。预训练中，转换子网络T-1采用Adam 优化算法，beta1设为0.900，beta2设为0.999，训练总批次为4000，学习率为10^–4，在批次大于2000后，学习率为10^–5。转换子网络T-2采用Adam优化算法，beta1设为0.500，beta2设为0.999，损失函数中λr设为1.000，λp设为0.006，λg设为0.001。训练总批次为2104，学习率为10–4，在104批次后将学习率慢慢衰减至10–6。转换网络T采用Adam优化算法，beta1设为0.500，beta2设为0.999，损失函数中λr设为1.000，λp设为0.006，λg设为0.001。训练总批次为2104，学习率为10^–4，在前100个批次将学习率线性衰减到10^–5，在10^4批次内将学习率线性衰减到10–6，之后再以学习率10–6训练104次。"}
+{"content": "实验结果与分析", "summary": "为了评估本方法的性能，与近期已有的几种方法包括多通道融合的方法(BIMEF)、带色彩恢复多尺度Retinex算法(MSRCR)、自然保留增强算法(NPE)、基于光照估计的方法(LIME)、多偏差融合方法(MF)、反射光照估计方法(SRIE)进行比较。本文在两个公共数据集(LIME数据和DICM数据)的低照度图像上对上述方法进行了性能评估。"}
+{"content": "结论", "summary": "本文提出了一种面向低照度图像增强的双曝光融合处理算法。首先，利用照度估计技术得到用于图像融合的权重矩阵;然后，通过摄像机响应模型合成双曝光图像。鉴于不同曝光量下图像颜色基本相同，定义低亮度像素和亮度分量找到最佳曝光率，使合成图像在原始图像曝光不足的区域得到更好的曝光;最后，根据权重矩阵将输入图像与合成图像进行融合，得到增强结果。和已有算法相比，本文方法能够获得较小的亮度失真，且具有合理的时间开销。由于实际环境的复杂性，对过度曝光进行优化建模仍是一个充满挑战性的问题，未来将对此展开进一步研究。"}
+{"content": "结论", "summary": "本文针对极端低光情况下的图像增强问题，提出一种新的增强模型，引入残差网络和感知损失重构图片的高频信息，更好地还原了图像的细节，得到了更好的视觉效果，在PSNR和SSIM这2个定量指标上也有所提升。"}
+{"content": "结论", "summary": "另一方面目前亮度放大倍数为人为输入，未来可以根据极低光图像的信息估算出亮度放大倍数。如何在进一步地提升增强后图像视觉效果的同时提高PSNR和SSIM定量指标的值，以及如何估算光度放大倍数，将是未来研究的方向。"}

ptuning/datasets/chat/dev.json

@@ -0,0 +1,20 @@
+{"content": "101", "summary": "10201"}
+{"content": "102", "summary": "10404"}
+{"content": "103", "summary": "10609"}
+{"content": "104", "summary": "10816"}
+{"content": "105", "summary": "11025"}
+{"content": "106", "summary": "11236"}
+{"content": "107", "summary": "11449"}
+{"content": "108", "summary": "11664"}
+{"content": "109", "summary": "11881"}
+{"content": "110", "summary": "12100"}
+{"content": "111", "summary": "12321"}
+{"content": "112", "summary": "12544"}
+{"content": "113", "summary": "12769"}
+{"content": "114", "summary": "12996"}
+{"content": "115", "summary": "13225"}
+{"content": "116", "summary": "13456"}
+{"content": "117", "summary": "13689"}
+{"content": "118", "summary": "13924"}
+{"content": "119", "summary": "14161"}
+{"content": "120", "summary": "14400"}

ptuning/datasets/chat/train.json

@@ -0,0 +1,102 @@
+{"content": "0", "summary": "0"}
+{"content": "1", "summary": "1"}
+{"content": "2", "summary": "4"}
+{"content": "3", "summary": "9"}
+{"content": "4", "summary": "16"}
+{"content": "5", "summary": "25"}
+{"content": "6", "summary": "36"}
+{"content": "7", "summary": "49"}
+{"content": "8", "summary": "64"}
+{"content": "9", "summary": "81"}
+{"content": "10", "summary": "100"}
+{"content": "11", "summary": "121"}
+{"content": "12", "summary": "144"}
+{"content": "13", "summary": "169"}
+{"content": "14", "summary": "196"}
+{"content": "15", "summary": "225"}
+{"content": "16", "summary": "256"}
+{"content": "17", "summary": "289"}
+{"content": "18", "summary": "324"}
+{"content": "19", "summary": "361"}
+{"content": "20", "summary": "400"}
+{"content": "21", "summary": "441"}
+{"content": "22", "summary": "484"}
+{"content": "23", "summary": "529"}
+{"content": "24", "summary": "576"}
+{"content": "25", "summary": "625"}
+{"content": "26", "summary": "676"}
+{"content": "27", "summary": "729"}
+{"content": "28", "summary": "784"}
+{"content": "29", "summary": "841"}
+{"content": "30", "summary": "900"}
+{"content": "31", "summary": "961"}
+{"content": "32", "summary": "1024"}
+{"content": "33", "summary": "1089"}
+{"content": "34", "summary": "1156"}
+{"content": "35", "summary": "1225"}
+{"content": "36", "summary": "1296"}
+{"content": "37", "summary": "1369"}
+{"content": "38", "summary": "1444"}
+{"content": "39", "summary": "1521"}
+{"content": "40", "summary": "1600"}
+{"content": "41", "summary": "1681"}
+{"content": "42", "summary": "1764"}
+{"content": "43", "summary": "1849"}
+{"content": "44", "summary": "1936"}
+{"content": "45", "summary": "2025"}
+{"content": "46", "summary": "2116"}
+{"content": "47", "summary": "2209"}
+{"content": "48", "summary": "2304"}
+{"content": "49", "summary": "2401"}
+{"content": "50", "summary": "2500"}
+{"content": "51", "summary": "2601"}
+{"content": "52", "summary": "2704"}
+{"content": "53", "summary": "2809"}
+{"content": "54", "summary": "2916"}
+{"content": "55", "summary": "3025"}
+{"content": "56", "summary": "3136"}
+{"content": "57", "summary": "3249"}
+{"content": "58", "summary": "3364"}
+{"content": "59", "summary": "3481"}
+{"content": "60", "summary": "3600"}
+{"content": "61", "summary": "3721"}
+{"content": "62", "summary": "3844"}
+{"content": "63", "summary": "3969"}
+{"content": "64", "summary": "4096"}
+{"content": "65", "summary": "4225"}
+{"content": "66", "summary": "4356"}
+{"content": "67", "summary": "4489"}
+{"content": "68", "summary": "4624"}
+{"content": "69", "summary": "4761"}
+{"content": "70", "summary": "4900"}
+{"content": "71", "summary": "5041"}
+{"content": "72", "summary": "5184"}
+{"content": "73", "summary": "5329"}
+{"content": "74", "summary": "5476"}
+{"content": "75", "summary": "5625"}
+{"content": "76", "summary": "5776"}
+{"content": "77", "summary": "5929"}
+{"content": "78", "summary": "6084"}
+{"content": "79", "summary": "6241"}
+{"content": "80", "summary": "6400"}
+{"content": "81", "summary": "6561"}
+{"content": "82", "summary": "6724"}
+{"content": "83", "summary": "6889"}
+{"content": "84", "summary": "7056"}
+{"content": "85", "summary": "7225"}
+{"content": "86", "summary": "7396"}
+{"content": "87", "summary": "7569"}
+{"content": "88", "summary": "7744"}
+{"content": "89", "summary": "7921"}
+{"content": "90", "summary": "8100"}
+{"content": "91", "summary": "8281"}
+{"content": "92", "summary": "8464"}
+{"content": "93", "summary": "8649"}
+{"content": "94", "summary": "8836"}
+{"content": "95", "summary": "9025"}
+{"content": "96", "summary": "9216"}
+{"content": "97", "summary": "9409"}
+{"content": "98", "summary": "9604"}
+{"content": "99", "summary": "9801"}
+{"content": "100", "summary": "10000"}
+

ptuning/deepspeed.json

@@ -0,0 +1,21 @@
+{
+  "train_micro_batch_size_per_gpu": "auto",
+  "zero_allow_untested_optimizer": true,
+  "fp16": {
+    "enabled": "auto",
+    "loss_scale": 0,
+    "initial_scale_power": 16,
+    "loss_scale_window": 1000,
+    "hysteresis": 2,
+    "min_loss_scale": 1
+  },
+  "zero_optimization": {
+    "stage": 2,
+    "allgather_partitions": true,
+    "allgather_bucket_size": 5e8,
+    "overlap_comm": false,
+    "reduce_scatter": true,
+    "reduce_bucket_size": 5e8,
+    "contiguous_gradients" : true
+  }
+}

ptuning/ds_train_finetune.sh

@@ -0,0 +1,28 @@
+
+LR=1e-4
+
+MASTER_PORT=$(shuf -n 1 -i 10000-65535)
+
+deepspeed --num_gpus=4 --master_port $MASTER_PORT main.py \
+    --deepspeed deepspeed.json \
+    --do_train \
+    --train_file AdvertiseGen/train.json \
+    --test_file AdvertiseGen/dev.json \
+    --prompt_column content \
+    --response_column summary \
+    --overwrite_cache \
+    --model_name_or_path THUDM/chatglm-6b \
+    --output_dir ./output/adgen-chatglm-6b-ft-$LR \
+    --overwrite_output_dir \
+    --max_source_length 64 \
+    --max_target_length 64 \
+    --per_device_train_batch_size 4 \
+    --per_device_eval_batch_size 1 \
+    --gradient_accumulation_steps 1 \
+    --predict_with_generate \
+    --max_steps 5000 \
+    --logging_steps 10 \
+    --save_steps 1000 \
+    --learning_rate $LR \
+    --fp16
+

ptuning/evaluate.sh

@@ -0,0 +1,22 @@
+PRE_SEQ_LEN=128
+CHECKPOINT=adgen-chatglm-6b-pt-128-2e-2
+STEP=1000
+
+CUDA_VISIBLE_DEVICES=0 python main.py \
+    --do_predict \
+    --train_file .\\datasets\\AdvertiseGen\\train.json \
+    --validation_file .\\datasets\\AdvertiseGen\\dev.json \
+    --test_file .\\datasets\\AdvertiseGen\\dev.json \
+    --overwrite_cache \
+    --prompt_column content \
+    --response_column summary \
+    --model_name_or_path ..\\models\\chatglm-6b-int4 \
+    --ptuning_checkpoint .\\output\\$CHECKPOINT\\checkpoint-$STEP \
+    --output_dir .\\output\\$CHECKPOINT \
+    --overwrite_output_dir \
+    --max_source_length 64 \
+    --max_target_length 64 \
+    --per_device_eval_batch_size 1 \
+    --predict_with_generate \
+    --pre_seq_len $PRE_SEQ_LEN \
+    --quantization_bit 4

ptuning/evaluate_finetune.sh

@@ -0,0 +1,18 @@
+CHECKPOINT=adgen-chatglm-6b-ft-1e-4
+STEP=3000
+
+CUDA_VISIBLE_DEVICES=0 python3 main.py \
+    --do_predict \
+    --validation_file AdvertiseGen/dev.json \
+    --test_file AdvertiseGen/dev.json \
+    --overwrite_cache \
+    --prompt_column content \
+    --response_column summary \
+    --model_name_or_path ./output/$CHECKPOINT/checkpoint-$STEP  \
+    --output_dir ./output/$CHECKPOINT \
+    --overwrite_output_dir \
+    --max_source_length 256 \
+    --max_target_length 256 \
+    --per_device_eval_batch_size 1 \
+    --predict_with_generate \
+    --fp16_full_eval

ptuning/main.py

@@ -0,0 +1,445 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright 2021 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Fine-tuning the library models for sequence to sequence.
+"""
+# You can also adapt this script on your own sequence to sequence task. Pointers for this are left as comments.
+
+import logging
+import os
+import sys
+import json
+
+import numpy as np
+from datasets import load_dataset
+import jieba 
+from rouge_chinese import Rouge
+from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
+import torch
+
+import transformers
+from transformers import (
+    AutoConfig,
+    AutoModel,
+    AutoTokenizer,
+    AutoTokenizer,
+    DataCollatorForSeq2Seq,
+    HfArgumentParser,
+    Seq2SeqTrainingArguments,
+    set_seed,
+)
+from trainer_seq2seq import Seq2SeqTrainer
+
+from arguments import ModelArguments, DataTrainingArguments
+
+logger = logging.getLogger(__name__)
+
+def main():
+
+    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, Seq2SeqTrainingArguments))
+    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
+        # If we pass only one argument to the script and it's the path to a json file,
+        # let's parse it to get our arguments.
+        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
+    else:
+        model_args, data_args, training_args = parser.parse_args_into_dataclasses()
+
+    # Setup logging
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
+        datefmt="%m/%d/%Y %H:%M:%S",
+        handlers=[logging.StreamHandler(sys.stdout)],
+    )
+
+    if training_args.should_log:
+        # The default of training_args.log_level is passive, so we set log level at info here to have that default.
+        transformers.utils.logging.set_verbosity_info()
+
+    log_level = training_args.get_process_log_level()
+    logger.setLevel(log_level)
+    # datasets.utils.logging.set_verbosity(log_level)
+    transformers.utils.logging.set_verbosity(log_level)
+    transformers.utils.logging.enable_default_handler()
+    transformers.utils.logging.enable_explicit_format()
+
+    # Log on each process the small summary:
+    logger.warning(
+        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
+        + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
+    )
+    logger.info(f"Training/evaluation parameters {training_args}")
+
+    # Set seed before initializing model.
+    set_seed(training_args.seed)
+
+    # Load dataset
+    data_files = {}
+    if data_args.train_file is not None:
+        data_files["train"] = data_args.train_file
+        extension = data_args.train_file.split(".")[-1]
+    if data_args.validation_file is not None:
+        data_files["validation"] = data_args.validation_file
+        extension = data_args.validation_file.split(".")[-1]
+    if data_args.test_file is not None:
+        data_files["test"] = data_args.test_file
+        extension = data_args.test_file.split(".")[-1]
+
+    raw_datasets = load_dataset(
+        extension,
+        data_files=data_files,
+        cache_dir=model_args.cache_dir,
+        use_auth_token=True if model_args.use_auth_token else None,
+    )
+    print('---------------------------------------------------')
+
+    print("raw_datasets:", raw_datasets)
+
+
+    # Load pretrained model and tokenizer
+    config = AutoConfig.from_pretrained(model_args.model_name_or_path, trust_remote_code=True)
+    config.pre_seq_len = model_args.pre_seq_len
+    config.prefix_projection = model_args.prefix_projection
+
+    tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, trust_remote_code=True)
+
+    if model_args.ptuning_checkpoint is not None:
+        # Evaluation
+        # Loading extra state dict of prefix encoder
+        model = AutoModel.from_pretrained(model_args.model_name_or_path, config=config, trust_remote_code=True)
+        prefix_state_dict = torch.load(os.path.join(model_args.ptuning_checkpoint, "pytorch_model.bin"))
+        new_prefix_state_dict = {}
+        for k, v in prefix_state_dict.items():
+            if k.startswith("transformer.prefix_encoder."):
+                new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v
+        model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)
+    else:
+        model = AutoModel.from_pretrained(model_args.model_name_or_path, config=config, trust_remote_code=True)
+
+    if model_args.quantization_bit is not None:
+        print(f"Quantized to {model_args.quantization_bit} bit")
+
+        try:
+            # kernel_file = "{}\\quantization_kernels.so".format(model_args.model_name_or_path)
+            kernel_file = "{}/quantization_kernels.so".format(model_args.model_name_or_path)
+            model = model.quantize(bits=model_args.quantization_bit, kernel_file=kernel_file)
+
+        except:
+            model = model.quantize(bits=model_args.quantization_bit)
+
+
+    if model_args.pre_seq_len is not None:
+        # P-tuning v2
+        model = model.half()
+        model.transformer.prefix_encoder.float()
+    else:
+        # Finetune
+        model = model.float()
+
+    prefix = data_args.source_prefix if data_args.source_prefix is not None else ""
+
+    # Preprocessing the datasets.
+    # We need to tokenize inputs and targets.
+    if training_args.do_train:
+        column_names = raw_datasets["train"].column_names
+    elif training_args.do_eval:
+        column_names = raw_datasets["validation"].column_names
+    elif training_args.do_predict:
+        column_names = raw_datasets["test"].column_names
+
+    else:
+        logger.info("There is nothing to do. Please pass `do_train`, `do_eval` and/or `do_predict`.")
+        return
+
+    # Get the column names for input/target.
+    prompt_column = data_args.prompt_column
+    response_column = data_args.response_column
+    history_column = data_args.history_column
+    
+    # Temporarily set max_target_length for training.
+    max_target_length = data_args.max_target_length
+
+    def preprocess_function_eval(examples):
+        inputs, targets = [], []
+        for i in range(len(examples[prompt_column])):
+            if examples[prompt_column][i] and examples[response_column][i]:
+                query = examples[prompt_column][i]
+                if history_column is None or len(examples[history_column][i]) == 0:
+                    prompt = query
+                else:
+                    prompt = ""
+                    history = examples[history_column][i]
+                    for turn_idx, (old_query, response) in enumerate(history):
+                        prompt += "[Round {}]\n问：{}\n答：{}\n".format(turn_idx, old_query, response)
+                    prompt += "[Round {}]\n问：{}\n答：".format(len(history), query)
+                inputs.append(prompt)
+                targets.append(examples[response_column][i])
+
+        inputs = [prefix + inp for inp in inputs]
+        model_inputs = tokenizer(inputs, max_length=data_args.max_source_length, truncation=True, padding=True)
+        labels = tokenizer(text_target=targets, max_length=max_target_length, truncation=True)
+
+        if data_args.ignore_pad_token_for_loss:
+            labels["input_ids"] = [
+                [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
+            ]
+        model_inputs["labels"] = labels["input_ids"]
+
+        return model_inputs
+
+    def preprocess_function_train(examples):
+        max_seq_length = data_args.max_source_length + data_args.max_target_length
+
+        model_inputs = {
+            "input_ids": [],
+            "labels": [],
+        }
+        for i in range(len(examples[prompt_column])):
+            if examples[prompt_column][i] and examples[response_column][i]:
+                query, answer = examples[prompt_column][i], examples[response_column][i]
+
+                if history_column is None:
+                    prompt = query
+                else:
+                    prompt = ""
+                    history = examples[history_column][i]
+                    for turn_idx, (old_query, response) in enumerate(history):
+                        prompt += "[Round {}]\n问：{}\n答：{}\n".format(turn_idx, old_query, response)
+                    prompt += "[Round {}]\n问：{}\n答：".format(len(history), query)
+
+                prompt = prefix + prompt
+                a_ids = tokenizer.encode(text=prompt, add_special_tokens=False)
+                b_ids = tokenizer.encode(text=answer, add_special_tokens=False)
+
+                if len(a_ids) > data_args.max_source_length - 1:
+                    a_ids = a_ids[: data_args.max_source_length - 1]
+
+                if len(b_ids) > data_args.max_target_length - 2:
+                    b_ids = b_ids[: data_args.max_target_length - 2]
+
+                input_ids = tokenizer.build_inputs_with_special_tokens(a_ids, b_ids)
+
+                context_length = input_ids.index(tokenizer.bos_token_id)
+                mask_position = context_length - 1
+                labels = [-100] * context_length + input_ids[mask_position+1:]
+                
+                pad_len = max_seq_length - len(input_ids)
+                input_ids = input_ids + [tokenizer.pad_token_id] * pad_len
+                labels = labels + [tokenizer.pad_token_id] * pad_len
+                if data_args.ignore_pad_token_for_loss:
+                    labels = [(l if l != tokenizer.pad_token_id else -100) for l in labels]
+
+                model_inputs["input_ids"].append(input_ids)
+                model_inputs["labels"].append(labels)
+
+        return model_inputs
+    
+    def print_dataset_example(example):
+        print("input_ids",example["input_ids"])
+        print("inputs", tokenizer.decode(example["input_ids"]))
+        print("label_ids", example["labels"])
+        print("labels", tokenizer.decode(example["labels"]))
+
+    if training_args.do_train:
+        if "train" not in raw_datasets:
+            raise ValueError("--do_train requires a train dataset")
+        train_dataset = raw_datasets["train"]
+        if data_args.max_train_samples is not None:
+            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
+            train_dataset = train_dataset.select(range(max_train_samples))
+        with training_args.main_process_first(desc="train dataset map pre-processing"):
+            train_dataset = train_dataset.map(
+                preprocess_function_train,
+                batched=True,
+                num_proc=data_args.preprocessing_num_workers,
+                remove_columns=column_names,
+                load_from_cache_file=not data_args.overwrite_cache,
+                desc="Running tokenizer on train dataset",
+            )
+        print_dataset_example(train_dataset[0])
+
+    if training_args.do_eval:
+        max_target_length = data_args.val_max_target_length
+        if "validation" not in raw_datasets:
+            raise ValueError("--do_eval requires a validation dataset")
+        eval_dataset = raw_datasets["validation"]
+        if data_args.max_eval_samples is not None:
+            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
+            eval_dataset = eval_dataset.select(range(max_eval_samples))
+        with training_args.main_process_first(desc="validation dataset map pre-processing"):
+            eval_dataset = eval_dataset.map(
+                preprocess_function_eval,
+                batched=True,
+                num_proc=data_args.preprocessing_num_workers,
+                remove_columns=column_names,
+                load_from_cache_file=not data_args.overwrite_cache,
+                desc="Running tokenizer on validation dataset",
+            )
+        print_dataset_example(eval_dataset[0])
+
+    if training_args.do_predict:
+        max_target_length = data_args.val_max_target_length
+        if "test" not in raw_datasets:
+            raise ValueError("--do_predict requires a test dataset")
+        predict_dataset = raw_datasets["test"]
+        if data_args.max_predict_samples is not None:
+            max_predict_samples = min(len(predict_dataset), data_args.max_predict_samples)
+            predict_dataset = predict_dataset.select(range(max_predict_samples))
+        with training_args.main_process_first(desc="prediction dataset map pre-processing"):
+            predict_dataset = predict_dataset.map(
+                preprocess_function_eval,
+                batched=True,
+                num_proc=data_args.preprocessing_num_workers,
+                remove_columns=column_names,
+                load_from_cache_file=not data_args.overwrite_cache,
+                desc="Running tokenizer on prediction dataset",
+            )
+        print_dataset_example(predict_dataset[0])
+
+    # Data collator
+    label_pad_token_id = -100 if data_args.ignore_pad_token_for_loss else tokenizer.pad_token_id
+    data_collator = DataCollatorForSeq2Seq(
+        tokenizer,
+        model=model,
+        label_pad_token_id=label_pad_token_id,
+        pad_to_multiple_of=None,
+        padding=False
+    )
+
+    # Metric
+    def compute_metrics(eval_preds):
+        preds, labels = eval_preds
+        if isinstance(preds, tuple):
+            preds = preds[0]
+        decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
+        if data_args.ignore_pad_token_for_loss:
+            # Replace -100 in the labels as we can't decode them.
+            labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
+        decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
+
+        score_dict = {
+            "rouge-1": [],
+            "rouge-2": [],
+            "rouge-l": [],
+            "bleu-4": []
+        }
+        for pred, label in zip(decoded_preds, decoded_labels):
+            hypothesis = list(jieba.cut(pred))
+            reference = list(jieba.cut(label))
+            rouge = Rouge()
+            scores = rouge.get_scores(' '.join(hypothesis) , ' '.join(reference))
+            result = scores[0]
+            
+            for k, v in result.items():
+                score_dict[k].append(round(v["f"] * 100, 4))
+            bleu_score = sentence_bleu([list(label)], list(pred), smoothing_function=SmoothingFunction().method3)
+            score_dict["bleu-4"].append(round(bleu_score * 100, 4))
+
+        for k, v in score_dict.items():
+            score_dict[k] = float(np.mean(v))
+        return score_dict
+
+    # Override the decoding parameters of Seq2SeqTrainer
+    training_args.generation_max_length = (
+        training_args.generation_max_length
+        if training_args.generation_max_length is not None
+        else data_args.val_max_target_length
+    )
+    training_args.generation_num_beams = (
+        data_args.num_beams if data_args.num_beams is not None else training_args.generation_num_beams
+    )
+    # Initialize our Trainer
+    trainer = Seq2SeqTrainer(
+        model=model,
+        args=training_args,
+        train_dataset=train_dataset if training_args.do_train else None,
+        eval_dataset=eval_dataset if training_args.do_eval else None,
+        tokenizer=tokenizer,
+        data_collator=data_collator,
+        compute_metrics=compute_metrics if training_args.predict_with_generate else None,
+        save_prefixencoder=model_args.pre_seq_len is not None
+    )
+
+    # Training
+    if training_args.do_train:
+        checkpoint = None
+        if training_args.resume_from_checkpoint is not None:
+            checkpoint = training_args.resume_from_checkpoint
+        # elif last_checkpoint is not None:
+        #     checkpoint = last_checkpoint
+        model.gradient_checkpointing_enable()
+        model.enable_input_require_grads()
+        train_result = trainer.train(resume_from_checkpoint=checkpoint)
+        # trainer.save_model()  # Saves the tokenizer too for easy upload
+
+        metrics = train_result.metrics
+        max_train_samples = (
+            data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)
+        )
+        metrics["train_samples"] = min(max_train_samples, len(train_dataset))
+
+        trainer.log_metrics("train", metrics)
+        trainer.save_metrics("train", metrics)
+        trainer.save_state()
+
+    # Evaluation
+    results = {}
+    if training_args.do_eval:
+        logger.info("*** Evaluate ***")
+        metrics = trainer.evaluate(metric_key_prefix="eval", do_sample=True, top_p=0.7, max_length=512, temperature=0.95)
+        max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+        metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
+
+        trainer.log_metrics("eval", metrics)
+        trainer.save_metrics("eval", metrics)
+
+    if training_args.do_predict:
+        logger.info("*** Predict ***")
+
+        predict_results = trainer.predict(predict_dataset, metric_key_prefix="predict", max_length=512, do_sample=True, top_p=0.7, temperature=0.95)
+        metrics = predict_results.metrics
+        max_predict_samples = (
+            data_args.max_predict_samples if data_args.max_predict_samples is not None else len(predict_dataset)
+        )
+        metrics["predict_samples"] = min(max_predict_samples, len(predict_dataset))
+
+        trainer.log_metrics("predict", metrics)
+        trainer.save_metrics("predict", metrics)
+
+        if trainer.is_world_process_zero():
+            if training_args.predict_with_generate:
+                predictions = tokenizer.batch_decode(
+                    predict_results.predictions, skip_special_tokens=True, clean_up_tokenization_spaces=True
+                )
+                predictions = [pred.strip() for pred in predictions]
+                labels = tokenizer.batch_decode(
+                    predict_results.label_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
+                )
+                labels = [label.strip() for label in labels]
+                output_prediction_file = os.path.join(training_args.output_dir, "generated_predictions.txt")
+                with open(output_prediction_file, "w", encoding="utf-8") as writer:
+                    for p, l in zip(predictions, labels):
+                        res = json.dumps({"labels": l, "predict": p}, ensure_ascii=False)
+                        writer.write(f"{res}\n")
+    return results
+
+
+def _mp_fn(index):
+    # For xla_spawn (TPUs)
+    main()
+
+
+if __name__ == "__main__":
+    main()

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/all_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 114.29,
+    "train_loss": 0.2596614052057266,
+    "train_runtime": 12879.4173,
+    "train_samples": 140,
+    "train_samples_per_second": 1.242,
+    "train_steps_per_second": 0.078
+}

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "..\\models\\chatglm-6b-int4",
+  "architectures": [
+    "ChatGLMForConditionalGeneration"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_chatglm.ChatGLMConfig",
+    "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration",
+    "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration"
+  },
+  "bos_token_id": 130004,
+  "eos_token_id": 130005,
+  "gmask_token_id": 130001,
+  "hidden_size": 4096,
+  "inner_hidden_size": 16384,
+  "layernorm_epsilon": 1e-05,
+  "mask_token_id": 130000,
+  "max_sequence_length": 2048,
+  "model_type": "chatglm",
+  "num_attention_heads": 32,
+  "num_layers": 28,
+  "pad_token_id": 3,
+  "position_encoding_2d": true,
+  "pre_seq_len": 128,
+  "prefix_projection": false,
+  "quantization_bit": 4,
+  "quantization_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.27.1",
+  "use_cache": true,
+  "vocab_size": 130528
+}

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/configuration_chatglm.py

@@ -0,0 +1,105 @@
+""" ChatGLM model configuration """
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class ChatGLMConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`~ChatGLMModel`].
+    It is used to instantiate an ChatGLM model according to the specified arguments, defining the model
+    architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
+    the ChatGLM-6B [THUDM/ChatGLM-6B](https://huggingface.co/THUDM/chatglm-6b) architecture.
+
+    Configuration objects inherit from  [`PretrainedConfig`] and can be used
+    to control the model outputs. Read the documentation from  [`PretrainedConfig`]
+    for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 150528):
+            Vocabulary size of the ChatGLM-6B model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`~ChatGLMModel`] or
+            [`~TFChatGLMModel`].
+        hidden_size (`int`, *optional*, defaults to 4096):
+            Dimension of the encoder layers and the pooler layer.
+        num_hidden_layers (`int`, *optional*, defaults to 28):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 32):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        inner_hidden_size (`int`, *optional*, defaults to 16384):
+            Dimension of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
+        max_sequence_length (`int`, *optional*, defaults to 512):
+            The maximum sequence length that this model might ever be used with.
+            Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
+        layernorm_epsilon (`float`, *optional*, defaults to 1e-5):
+            The epsilon used by the layer normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether the model should return the last key/values attentions (not used by all models).
+        Example:
+
+    ```python
+    >>> from configuration_chatglm import ChatGLMConfig
+    >>> from modeling_chatglm import ChatGLMModel
+
+    >>> # Initializing a ChatGLM-6B THUDM/ChatGLM-6B style configuration
+    >>> configuration = ChatGLMConfig()
+
+    >>> # Initializing a model from the THUDM/ChatGLM-6B style configuration
+    >>> model = ChatGLMModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```
+"""
+    model_type = "chatglm"
+
+    def __init__(
+            self,
+            vocab_size=150528,
+            hidden_size=4096,
+            num_layers=28,
+            num_attention_heads=32,
+            layernorm_epsilon=1e-5,
+            use_cache=False,
+            bos_token_id=150004,
+            eos_token_id=150005,
+            mask_token_id=150000,
+            gmask_token_id=150001,
+            pad_token_id=0,
+            max_sequence_length=2048,
+            inner_hidden_size=16384,
+            position_encoding_2d=True,
+            quantization_bit=0,
+            quantization_embeddings=False,
+            pre_seq_len=None,
+            prefix_projection=False,
+            **kwargs
+    ):
+        self.num_layers = num_layers
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_attention_heads = num_attention_heads
+        self.max_sequence_length = max_sequence_length
+        self.layernorm_epsilon = layernorm_epsilon
+        self.inner_hidden_size = inner_hidden_size
+        self.use_cache = use_cache
+        self.bos_token_id = bos_token_id
+        self.eos_token_id = eos_token_id
+        self.pad_token_id = pad_token_id
+        self.mask_token_id = mask_token_id
+        self.gmask_token_id = gmask_token_id
+        self.position_encoding_2d = position_encoding_2d
+        self.quantization_bit = quantization_bit
+        self.quantization_embeddings = quantization_embeddings
+        self.pre_seq_len = pre_seq_len
+        self.prefix_projection = prefix_projection
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            **kwargs
+        )

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 130004,
+  "eos_token_id": 130005,
+  "pad_token_id": 3,
+  "transformers_version": "4.27.1"
+}

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/ice_text.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e974d9a69c242ce014c88c2b26089270f6198f3c0b700a887666cd3e816f17e
+size 2706249

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/modeling_chatglm.py

@@ -0,0 +1,1472 @@
+""" PyTorch ChatGLM model. """
+
+import math
+import copy
+import os
+import warnings
+import re
+import sys
+
+import torch
+import torch.utils.checkpoint
+import torch.nn.functional as F
+from torch import nn
+from torch.nn import CrossEntropyLoss, LayerNorm
+from torch.nn.utils import skip_init
+from typing import Optional, Tuple, Union, List, Callable, Dict, Any
+
+from transformers.utils import (
+    add_code_sample_docstrings,
+    add_start_docstrings,
+    add_start_docstrings_to_model_forward,
+)
+from transformers.modeling_outputs import (
+    BaseModelOutputWithPast,
+    CausalLMOutputWithPast,
+    BaseModelOutputWithPastAndCrossAttentions,
+)
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import logging
+from transformers.generation.logits_process import LogitsProcessor
+from transformers.generation.utils import LogitsProcessorList, StoppingCriteriaList, GenerationConfig, ModelOutput
+
+from .configuration_chatglm import ChatGLMConfig
+
+
+# flags required to enable jit fusion kernels
+
+if sys.platform != 'darwin':
+    torch._C._jit_set_profiling_mode(False)
+    torch._C._jit_set_profiling_executor(False)
+    torch._C._jit_override_can_fuse_on_cpu(True)
+    torch._C._jit_override_can_fuse_on_gpu(True)
+
+logger = logging.get_logger(__name__)
+
+_CHECKPOINT_FOR_DOC = "THUDM/ChatGLM-6B"
+_CONFIG_FOR_DOC = "ChatGLM6BConfig"
+
+CHATGLM_6B_PRETRAINED_MODEL_ARCHIVE_LIST = [
+    "THUDM/chatglm-6b",
+    # See all ChatGLM-6B models at https://huggingface.co/models?filter=chatglm
+]
+
+
+class InvalidScoreLogitsProcessor(LogitsProcessor):
+    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
+        if torch.isnan(scores).any() or torch.isinf(scores).any():
+            scores.zero_()
+            scores[..., 5] = 5e4
+        return scores
+
+
+def load_tf_weights_in_chatglm_6b(model, config, tf_checkpoint_path):
+    """Load tf checkpoints in a pytorch model."""
+    try:
+        import re
+
+        import numpy as np
+        import tensorflow as tf
+    except ImportError:
+        logger.error(
+            "Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see "
+            "https://www.tensorflow.org/install/ for installation instructions."
+        )
+        raise
+    tf_path = os.path.abspath(tf_checkpoint_path)
+    logger.info(f"Converting TensorFlow checkpoint from {tf_path}")
+    # Load weights from TF model
+    init_vars = tf.train.list_variables(tf_path)
+    names = []
+    arrays = []
+    for name, shape in init_vars:
+        logger.info(f"Loading TF weight {name} with shape {shape}")
+        array = tf.train.load_variable(tf_path, name)
+        names.append(name)
+        arrays.append(array)
+
+    for name, array in zip(names, arrays):
+        name = name.split("/")
+        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
+        # which are not required for using pretrained model
+        if any(
+                n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
+                for n in name
+        ):
+            logger.info(f"Skipping {'/'.join(name)}")
+            continue
+        pointer = model
+        for m_name in name:
+            if re.fullmatch(r"[A-Za-z]+_\d+", m_name):
+                scope_names = re.split(r"_(\d+)", m_name)
+            else:
+                scope_names = [m_name]
+            if scope_names[0] == "kernel" or scope_names[0] == "gamma":
+                pointer = getattr(pointer, "weight")
+            elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
+                pointer = getattr(pointer, "bias")
+            elif scope_names[0] == "output_weights":
+                pointer = getattr(pointer, "weight")
+            elif scope_names[0] == "squad":
+                pointer = getattr(pointer, "classifier")
+            else:
+                try:
+                    pointer = getattr(pointer, scope_names[0])
+                except AttributeError:
+                    logger.info(f"Skipping {'/'.join(name)}")
+                    continue
+            if len(scope_names) >= 2:
+                num = int(scope_names[1])
+                pointer = pointer[num]
+        if m_name[-11:] == "_embeddings":
+            pointer = getattr(pointer, "weight")
+        elif m_name == "kernel":
+            array = np.transpose(array)
+        try:
+            assert (
+                    pointer.shape == array.shape
+            ), f"Pointer shape {pointer.shape} and array shape {array.shape} mismatched"
+        except AssertionError as e:
+            e.args += (pointer.shape, array.shape)
+            raise
+        logger.info(f"Initialize PyTorch weight {name}")
+        pointer.data = torch.from_numpy(array)
+    return model
+
+
+class PrefixEncoder(torch.nn.Module):
+    """
+    The torch.nn model to encode the prefix
+    Input shape: (batch-size, prefix-length)
+    Output shape: (batch-size, prefix-length, 2*layers*hidden)
+    """
+
+    def __init__(self, config):
+        super().__init__()
+        self.prefix_projection = config.prefix_projection
+        if self.prefix_projection:
+            # Use a two-layer MLP to encode the prefix
+            self.embedding = torch.nn.Embedding(config.pre_seq_len, config.hidden_size)
+            self.trans = torch.nn.Sequential(
+                torch.nn.Linear(config.hidden_size, config.hidden_size),
+                torch.nn.Tanh(),
+                torch.nn.Linear(config.hidden_size, config.num_layers * config.hidden_size * 2)
+            )
+        else:
+            self.embedding = torch.nn.Embedding(config.pre_seq_len, config.num_layers * config.hidden_size * 2)
+
+    def forward(self, prefix: torch.Tensor):
+        if self.prefix_projection:
+            prefix_tokens = self.embedding(prefix)
+            past_key_values = self.trans(prefix_tokens)
+        else:
+            past_key_values = self.embedding(prefix)
+        return past_key_values
+
+
+@torch.jit.script
+def gelu_impl(x):
+    """OpenAI's gelu implementation."""
+    return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * x *
+                                       (1.0 + 0.044715 * x * x)))
+
+
+def gelu(x):
+    return gelu_impl(x)
+
+
+class RotaryEmbedding(torch.nn.Module):
+    def __init__(self, dim, base=10000, precision=torch.half, learnable=False):
+        super().__init__()
+        inv_freq = 1. / (base ** (torch.arange(0, dim, 2).float() / dim))
+        inv_freq = inv_freq.half()
+        self.learnable = learnable
+        if learnable:
+            self.inv_freq = torch.nn.Parameter(inv_freq)
+            self.max_seq_len_cached = None
+        else:
+            self.register_buffer('inv_freq', inv_freq)
+            self.max_seq_len_cached = None
+            self.cos_cached = None
+            self.sin_cached = None
+        self.precision = precision
+
+    def _load_from_state_dict(self, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys,
+                              error_msgs):
+        pass
+
+    def forward(self, x, seq_dim=1, seq_len=None):
+        if seq_len is None:
+            seq_len = x.shape[seq_dim]
+        if self.max_seq_len_cached is None or (seq_len > self.max_seq_len_cached):
+            self.max_seq_len_cached = None if self.learnable else seq_len
+            t = torch.arange(seq_len, device=x.device, dtype=self.inv_freq.dtype)
+            freqs = torch.einsum('i,j->ij', t, self.inv_freq)
+            # Different from paper, but it uses a different permutation in order to obtain the same calculation
+            emb = torch.cat((freqs, freqs), dim=-1).to(x.device)
+            if self.precision == torch.bfloat16:
+                emb = emb.float()
+
+            # [sx, 1 (b * np), hn]
+            cos_cached = emb.cos()[:, None, :]
+            sin_cached = emb.sin()[:, None, :]
+            if self.precision == torch.bfloat16:
+                cos_cached = cos_cached.bfloat16()
+                sin_cached = sin_cached.bfloat16()
+            if self.learnable:
+                return cos_cached, sin_cached
+            self.cos_cached, self.sin_cached = cos_cached, sin_cached
+        return self.cos_cached[:seq_len, ...], self.sin_cached[:seq_len, ...]
+
+    def _apply(self, fn):
+        if self.cos_cached is not None:
+            self.cos_cached = fn(self.cos_cached)
+        if self.sin_cached is not None:
+            self.sin_cached = fn(self.sin_cached)
+        return super()._apply(fn)
+
+def rotate_half(x):
+    x1, x2 = x[..., :x.shape[-1] // 2], x[..., x.shape[-1] // 2:]
+    return torch.cat((-x2, x1), dim=x1.ndim - 1)  # dim=-1 triggers a bug in earlier torch versions
+
+
+@torch.jit.script
+def apply_rotary_pos_emb_index(q, k, cos, sin, position_id):
+    # position_id: [sq, b], q, k: [sq, b, np, hn], cos: [sq, 1, hn] -> [sq, b, 1, hn]
+    cos, sin = F.embedding(position_id, cos.squeeze(1)).unsqueeze(2), \
+        F.embedding(position_id, sin.squeeze(1)).unsqueeze(2)
+    q, k = (q * cos) + (rotate_half(q) * sin), (k * cos) + (rotate_half(k) * sin)
+    return q, k
+
+
+def attention_fn(
+        self,
+        query_layer,
+        key_layer,
+        value_layer,
+        attention_mask,
+        hidden_size_per_partition,
+        layer_id,
+        layer_past=None,
+        scaling_attention_score=True,
+        use_cache=False,
+):
+    if layer_past is not None:
+        past_key, past_value = layer_past[0], layer_past[1]
+        key_layer = torch.cat((past_key, key_layer), dim=0)
+        value_layer = torch.cat((past_value, value_layer), dim=0)
+
+    # seqlen, batch, num_attention_heads, hidden_size_per_attention_head
+    seq_len, b, nh, hidden_size = key_layer.shape
+
+    if use_cache:
+        present = (key_layer, value_layer)
+    else:
+        present = None
+
+    query_key_layer_scaling_coeff = float(layer_id + 1)
+    if scaling_attention_score:
+        query_layer = query_layer / (math.sqrt(hidden_size) * query_key_layer_scaling_coeff)
+
+    # ===================================
+    # Raw attention scores. [b, np, s, s]
+    # ===================================
+
+    # [b, np, sq, sk]
+    output_size = (query_layer.size(1), query_layer.size(2), query_layer.size(0), key_layer.size(0))
+
+    # [sq, b, np, hn] -> [sq, b * np, hn]
+    query_layer = query_layer.view(output_size[2], output_size[0] * output_size[1], -1)
+    # [sk, b, np, hn] -> [sk, b * np, hn]
+    key_layer = key_layer.view(output_size[3], output_size[0] * output_size[1], -1)
+
+    matmul_result = torch.zeros(
+        1, 1, 1,
+        dtype=query_layer.dtype,
+        device=query_layer.device,
+    )
+
+    matmul_result = torch.baddbmm(
+        matmul_result,
+        query_layer.transpose(0, 1),  # [b * np, sq, hn]
+        key_layer.transpose(0, 1).transpose(1, 2),  # [b * np, hn, sk]
+        beta=0.0,
+        alpha=1.0,
+    )
+
+    # change view to [b, np, sq, sk]
+    attention_scores = matmul_result.view(*output_size)
+
+    if self.scale_mask_softmax:
+        self.scale_mask_softmax.scale = query_key_layer_scaling_coeff
+        attention_probs = self.scale_mask_softmax(attention_scores, attention_mask.contiguous())
+    else:
+        if not (attention_mask == 0).all():
+            # if auto-regressive, skip
+            attention_scores.masked_fill_(attention_mask, -10000.0)
+        dtype = attention_scores.dtype
+        attention_scores = attention_scores.float()
+        attention_scores = attention_scores * query_key_layer_scaling_coeff
+
+        attention_probs = F.softmax(attention_scores, dim=-1)
+
+        attention_probs = attention_probs.type(dtype)
+
+    # =========================
+    # Context layer. [sq, b, hp]
+    # =========================
+
+    # value_layer -> context layer.
+    # [sk, b, np, hn] --> [b, np, sq, hn]
+
+    # context layer shape: [b, np, sq, hn]
+    output_size = (value_layer.size(1), value_layer.size(2), query_layer.size(0), value_layer.size(3))
+
+    # change view [sk, b * np, hn]
+    value_layer = value_layer.view(value_layer.size(0), output_size[0] * output_size[1], -1)
+
+    # change view [b * np, sq, sk]
+    attention_probs = attention_probs.view(output_size[0] * output_size[1], output_size[2], -1)
+
+    # matmul: [b * np, sq, hn]
+    context_layer = torch.bmm(attention_probs, value_layer.transpose(0, 1))
+
+    # change view [b, np, sq, hn]
+    context_layer = context_layer.view(*output_size)
+
+    # [b, np, sq, hn] --> [sq, b, np, hn]
+    context_layer = context_layer.permute(2, 0, 1, 3).contiguous()
+
+    # [sq, b, np, hn] --> [sq, b, hp]
+    new_context_layer_shape = context_layer.size()[:-2] + (hidden_size_per_partition,)
+    context_layer = context_layer.view(*new_context_layer_shape)
+
+    outputs = (context_layer, present, attention_probs)
+
+    return outputs
+
+
+def default_init(cls, *args, **kwargs):
+    return cls(*args, **kwargs)
+
+
+class SelfAttention(torch.nn.Module):
+    def __init__(self, hidden_size, num_attention_heads,
+                 layer_id, hidden_size_per_attention_head=None, bias=True,
+                 params_dtype=torch.float, position_encoding_2d=True, empty_init=True):
+        if empty_init:
+            init_method = skip_init
+        else:
+            init_method = default_init
+        super(SelfAttention, self).__init__()
+
+        self.layer_id = layer_id
+        self.hidden_size = hidden_size
+        self.hidden_size_per_partition = hidden_size
+        self.num_attention_heads = num_attention_heads
+        self.num_attention_heads_per_partition = num_attention_heads
+        self.position_encoding_2d = position_encoding_2d
+        self.rotary_emb = RotaryEmbedding(
+            self.hidden_size // (self.num_attention_heads * 2)
+            if position_encoding_2d
+            else self.hidden_size // self.num_attention_heads,
+            base=10000,
+            precision=torch.half,
+            learnable=False,
+        )
+
+        self.scale_mask_softmax = None
+
+        if hidden_size_per_attention_head is None:
+            self.hidden_size_per_attention_head = hidden_size // num_attention_heads
+        else:
+            self.hidden_size_per_attention_head = hidden_size_per_attention_head
+
+        self.inner_hidden_size = num_attention_heads * self.hidden_size_per_attention_head
+
+        # Strided linear layer.
+        self.query_key_value = init_method(
+            torch.nn.Linear,
+            hidden_size,
+            3 * self.inner_hidden_size,
+            bias=bias,
+            dtype=params_dtype,
+        )
+
+        self.dense = init_method(
+            torch.nn.Linear,
+            self.inner_hidden_size,
+            hidden_size,
+            bias=bias,
+            dtype=params_dtype,
+        )
+
+    @staticmethod
+    def attention_mask_func(attention_scores, attention_mask):
+        attention_scores.masked_fill_(attention_mask, -10000.0)
+        return attention_scores
+
+    def split_tensor_along_last_dim(self, tensor, num_partitions,
+                                    contiguous_split_chunks=False):
+        """Split a tensor along its last dimension.
+        Arguments:
+            tensor: input tensor.
+            num_partitions: number of partitions to split the tensor
+            contiguous_split_chunks: If True, make each chunk contiguous
+                                    in memory.
+        """
+        # Get the size and dimension.
+        last_dim = tensor.dim() - 1
+        last_dim_size = tensor.size()[last_dim] // num_partitions
+        # Split.
+        tensor_list = torch.split(tensor, last_dim_size, dim=last_dim)
+        # Note: torch.split does not create contiguous tensors by default.
+        if contiguous_split_chunks:
+            return tuple(chunk.contiguous() for chunk in tensor_list)
+
+        return tensor_list
+
+    def forward(
+            self,
+            hidden_states: torch.Tensor,
+            position_ids,
+            attention_mask: torch.Tensor,
+            layer_id,
+            layer_past: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
+            use_cache: bool = False,
+            output_attentions: bool = False,
+    ):
+        """
+        hidden_states: [seq_len, batch, hidden_size]
+        attention_mask: [(1, 1), seq_len, seq_len]
+        """
+
+        # [seq_len, batch, 3 * hidden_size]
+        mixed_raw_layer = self.query_key_value(hidden_states)
+
+        # [seq_len, batch, 3 * hidden_size] --> [seq_len, batch, num_attention_heads, 3 * hidden_size_per_attention_head]
+        new_tensor_shape = mixed_raw_layer.size()[:-1] + (
+            self.num_attention_heads_per_partition,
+            3 * self.hidden_size_per_attention_head,
+        )
+        mixed_raw_layer = mixed_raw_layer.view(*new_tensor_shape)
+
+        # [seq_len, batch, num_attention_heads, hidden_size_per_attention_head]
+        (query_layer, key_layer, value_layer) = self.split_tensor_along_last_dim(mixed_raw_layer, 3)
+
+        if self.position_encoding_2d:
+            q1, q2 = query_layer.chunk(2, dim=(query_layer.ndim - 1))
+            k1, k2 = key_layer.chunk(2, dim=(key_layer.ndim - 1))
+            cos, sin = self.rotary_emb(q1, seq_len=position_ids.max() + 1)
+            position_ids, block_position_ids = position_ids[:, 0, :].transpose(0, 1).contiguous(), \
+                position_ids[:, 1, :].transpose(0, 1).contiguous()
+            q1, k1 = apply_rotary_pos_emb_index(q1, k1, cos, sin, position_ids)
+            q2, k2 = apply_rotary_pos_emb_index(q2, k2, cos, sin, block_position_ids)
+            query_layer = torch.concat([q1, q2], dim=(q1.ndim - 1))
+            key_layer = torch.concat([k1, k2], dim=(k1.ndim - 1))
+        else:
+            position_ids = position_ids.transpose(0, 1)
+            cos, sin = self.rotary_emb(value_layer, seq_len=position_ids.max() + 1)
+            # [seq_len, batch, num_attention_heads, hidden_size_per_attention_head]
+            query_layer, key_layer = apply_rotary_pos_emb_index(query_layer, key_layer, cos, sin, position_ids)
+
+        # [seq_len, batch, hidden_size]
+        context_layer, present, attention_probs = attention_fn(
+            self=self,
+            query_layer=query_layer,
+            key_layer=key_layer,
+            value_layer=value_layer,
+            attention_mask=attention_mask,
+            hidden_size_per_partition=self.hidden_size_per_partition,
+            layer_id=layer_id,
+            layer_past=layer_past,
+            use_cache=use_cache
+        )
+
+        output = self.dense(context_layer)
+
+        outputs = (output, present)
+
+        if output_attentions:
+            outputs += (attention_probs,)
+
+        return outputs  # output, present, attention_probs
+
+
+class GEGLU(torch.nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.activation_fn = F.gelu
+
+    def forward(self, x):
+        # dim=-1 breaks in jit for pt<1.10
+        x1, x2 = x.chunk(2, dim=(x.ndim - 1))
+        return x1 * self.activation_fn(x2)
+
+
+class GLU(torch.nn.Module):
+    def __init__(self, hidden_size, inner_hidden_size=None,
+                 layer_id=None, bias=True, activation_func=gelu, params_dtype=torch.float, empty_init=True):
+        super(GLU, self).__init__()
+        if empty_init:
+            init_method = skip_init
+        else:
+            init_method = default_init
+        self.layer_id = layer_id
+        self.activation_func = activation_func
+
+        # Project to 4h.
+        self.hidden_size = hidden_size
+        if inner_hidden_size is None:
+            inner_hidden_size = 4 * hidden_size
+        self.inner_hidden_size = inner_hidden_size
+        self.dense_h_to_4h = init_method(
+            torch.nn.Linear,
+            self.hidden_size,
+            self.inner_hidden_size,
+            bias=bias,
+            dtype=params_dtype,
+        )
+        # Project back to h.
+        self.dense_4h_to_h = init_method(
+            torch.nn.Linear,
+            self.inner_hidden_size,
+            self.hidden_size,
+            bias=bias,
+            dtype=params_dtype,
+        )
+
+    def forward(self, hidden_states):
+        """
+        hidden_states: [seq_len, batch, hidden_size]
+        """
+
+        # [seq_len, batch, inner_hidden_size]
+        intermediate_parallel = self.dense_h_to_4h(hidden_states)
+
+        intermediate_parallel = self.activation_func(intermediate_parallel)
+
+        output = self.dense_4h_to_h(intermediate_parallel)
+
+        return output
+
+
+class GLMBlock(torch.nn.Module):
+    def __init__(
+            self,
+            hidden_size,
+            num_attention_heads,
+            layernorm_epsilon,
+            layer_id,
+            inner_hidden_size=None,
+            hidden_size_per_attention_head=None,
+            layernorm=LayerNorm,
+            use_bias=True,
+            params_dtype=torch.float,
+            num_layers=28,
+            position_encoding_2d=True,
+            empty_init=True
+    ):
+        super(GLMBlock, self).__init__()
+        # Set output layer initialization if not provided.
+
+        self.layer_id = layer_id
+
+        # Layernorm on the input data.
+        self.input_layernorm = layernorm(hidden_size, eps=layernorm_epsilon)
+
+        self.position_encoding_2d = position_encoding_2d
+
+        # Self attention.
+        self.attention = SelfAttention(
+            hidden_size,
+            num_attention_heads,
+            layer_id,
+            hidden_size_per_attention_head=hidden_size_per_attention_head,
+            bias=use_bias,
+            params_dtype=params_dtype,
+            position_encoding_2d=self.position_encoding_2d,
+            empty_init=empty_init
+        )
+
+        # Layernorm on the input data.
+        self.post_attention_layernorm = layernorm(hidden_size, eps=layernorm_epsilon)
+
+        self.num_layers = num_layers
+
+        # GLU
+        self.mlp = GLU(
+            hidden_size,
+            inner_hidden_size=inner_hidden_size,
+            bias=use_bias,
+            layer_id=layer_id,
+            params_dtype=params_dtype,
+            empty_init=empty_init
+        )
+
+    def forward(
+            self,
+            hidden_states: torch.Tensor,
+            position_ids,
+            attention_mask: torch.Tensor,
+            layer_id,
+            layer_past: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
+            use_cache: bool = False,
+            output_attentions: bool = False,
+    ):
+        """
+        hidden_states: [seq_len, batch, hidden_size]
+        attention_mask: [(1, 1), seq_len, seq_len]
+        """
+
+        # Layer norm at the begining of the transformer layer.
+        # [seq_len, batch, hidden_size]
+        attention_input = self.input_layernorm(hidden_states)
+
+        # Self attention.
+        attention_outputs = self.attention(
+            attention_input,
+            position_ids,
+            attention_mask=attention_mask,
+            layer_id=layer_id,
+            layer_past=layer_past,
+            use_cache=use_cache,
+            output_attentions=output_attentions
+        )
+
+        attention_output = attention_outputs[0]
+
+        outputs = attention_outputs[1:]
+
+        # Residual connection.
+        alpha = (2 * self.num_layers) ** 0.5
+        hidden_states = attention_input * alpha + attention_output
+
+        mlp_input = self.post_attention_layernorm(hidden_states)
+
+        # MLP.
+        mlp_output = self.mlp(mlp_input)
+
+        # Second residual connection.
+        output = mlp_input * alpha + mlp_output
+
+        if use_cache:
+            outputs = (output,) + outputs
+        else:
+            outputs = (output,) + outputs[1:]
+
+        return outputs  # hidden_states, present, attentions
+
+
+class ChatGLMPreTrainedModel(PreTrainedModel):
+    """
+    An abstract class to handle weights initialization and
+    a simple interface for downloading and loading pretrained models.
+    """
+
+    is_parallelizable = False
+    supports_gradient_checkpointing = True
+    config_class = ChatGLMConfig
+    base_model_prefix = "transformer"
+    _no_split_modules = ["GLMBlock"]
+
+    def __init__(self, *inputs, **kwargs):
+        super().__init__(*inputs, **kwargs)
+
+    def _init_weights(self, module: nn.Module):
+        """Initialize the weights."""
+        return
+
+    def get_masks(self, input_ids, device):
+        batch_size, seq_length = input_ids.shape
+        context_lengths = [seq.tolist().index(self.config.bos_token_id) for seq in input_ids]
+        attention_mask = torch.ones((batch_size, seq_length, seq_length), device=device)
+        attention_mask.tril_()
+        for i, context_length in enumerate(context_lengths):
+            attention_mask[i, :, :context_length] = 1
+        attention_mask.unsqueeze_(1)
+        attention_mask = (attention_mask < 0.5).bool()
+
+        return attention_mask
+
+    def get_position_ids(self, input_ids, mask_positions, device, use_gmasks=None):
+        batch_size, seq_length = input_ids.shape
+        if use_gmasks is None:
+            use_gmasks = [False] * batch_size
+        context_lengths = [seq.tolist().index(self.config.bos_token_id) for seq in input_ids]
+        if self.position_encoding_2d:
+            position_ids = torch.arange(seq_length, dtype=torch.long, device=device).unsqueeze(0).repeat(batch_size, 1)
+            for i, context_length in enumerate(context_lengths):
+                position_ids[i, context_length:] = mask_positions[i]
+            block_position_ids = [torch.cat((
+                torch.zeros(context_length, dtype=torch.long, device=device),
+                torch.arange(seq_length - context_length, dtype=torch.long, device=device) + 1
+            )) for context_length in context_lengths]
+            block_position_ids = torch.stack(block_position_ids, dim=0)
+            position_ids = torch.stack((position_ids, block_position_ids), dim=1)
+        else:
+            position_ids = torch.arange(seq_length, dtype=torch.long, device=device).unsqueeze(0).repeat(batch_size, 1)
+            for i, context_length in enumerate(context_lengths):
+                if not use_gmasks[i]:
+                    position_ids[context_length:] = mask_positions[i]
+
+        return position_ids
+
+    def _set_gradient_checkpointing(self, module, value=False):
+        if isinstance(module, ChatGLMModel):
+            module.gradient_checkpointing = value
+
+
+CHATGLM_6B_START_DOCSTRING = r"""
+    This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class.
+    Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general
+    usage and behavior.
+
+    Parameters:
+        config ([`~ChatGLM6BConfig`]): Model configuration class with all the parameters of the model.
+            Initializing with a config file does not load the weights associated with the model, only the configuration.
+            Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.
+"""
+
+CHATGLM_6B_INPUTS_DOCSTRING = r"""
+    Args:
+        input_ids (`torch.LongTensor` of shape `({0})`):
+            Indices of input sequence tokens in the vocabulary.
+
+            Indices can be obtained using [`ChatGLM6BTokenizer`].
+            See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            [What are input IDs?](../glossary#input-ids)
+        attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*):
+            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
+
+            - 1 for tokens that are **not masked**,
+            - 0 for tokens that are **masked**.
+
+            [What are attention masks?](../glossary#attention-mask)
+        token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*):
+            Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
+
+            - 0 corresponds to a *sentence A* token,
+            - 1 corresponds to a *sentence B* token.
+
+            [What are token type IDs?](../glossary#token-type-ids)
+        position_ids (`torch.LongTensor` of shape `({0})`, *optional*):
+            Indices of positions of each input sequence tokens in the position embeddings.
+            Selected in the range `[0, config.max_position_embeddings - 1]`.
+
+            [What are position IDs?](../glossary#position-ids)
+        head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*):
+            Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
+
+            - 1 indicates the head is **not masked**,
+            - 0 indicates the head is **masked**.
+
+        inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*):
+            Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation.
+            This is useful if you want more control over how to convert *input_ids* indices into associated vectors
+            than the model's internal embedding lookup matrix.
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+"""
+
+
+@add_start_docstrings(
+    "The bare ChatGLM-6B Model transformer outputting raw hidden-states without any specific head on top.",
+    CHATGLM_6B_START_DOCSTRING,
+)
+class ChatGLMModel(ChatGLMPreTrainedModel):
+    """
+
+    The model can behave as an encoder (with only self-attention) as well
+    as a decoder, in which case a layer of cross-attention is added between
+    the self-attention layers, following the architecture described in [Attention is
+    all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani,
+    Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.
+
+    To behave as an decoder the model needs to be initialized with the
+    `is_decoder` argument of the configuration set to `True`.
+    To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder`
+    argument and `add_cross_attention` set to `True`; an
+    `encoder_hidden_states` is then expected as an input to the forward pass.
+    """
+
+    def __init__(self, config: ChatGLMConfig, empty_init=True):
+        super().__init__(config)
+        if empty_init:
+            init_method = skip_init
+        else:
+            init_method = default_init
+        # recording parameters
+        self.max_sequence_length = config.max_sequence_length
+        self.hidden_size = config.hidden_size
+        self.params_dtype = torch.half
+        self.num_attention_heads = config.num_attention_heads
+        self.vocab_size = config.vocab_size
+        self.num_layers = config.num_layers
+        self.layernorm_epsilon = config.layernorm_epsilon
+        self.inner_hidden_size = config.inner_hidden_size
+        self.hidden_size_per_attention_head = self.hidden_size // self.num_attention_heads
+        self.position_encoding_2d = config.position_encoding_2d
+        self.pre_seq_len = config.pre_seq_len
+        self.prefix_projection = config.prefix_projection
+
+        self.word_embeddings = init_method(
+            torch.nn.Embedding,
+            num_embeddings=self.vocab_size, embedding_dim=self.hidden_size,
+            dtype=self.params_dtype
+        )
+        self.gradient_checkpointing = False
+
+        def get_layer(layer_id):
+            return GLMBlock(
+                self.hidden_size,
+                self.num_attention_heads,
+                self.layernorm_epsilon,
+                layer_id,
+                inner_hidden_size=self.inner_hidden_size,
+                hidden_size_per_attention_head=self.hidden_size_per_attention_head,
+                layernorm=LayerNorm,
+                use_bias=True,
+                params_dtype=self.params_dtype,
+                position_encoding_2d=self.position_encoding_2d,
+                empty_init=empty_init
+            )
+
+        self.layers = torch.nn.ModuleList(
+            [get_layer(layer_id) for layer_id in range(self.num_layers)]
+        )
+
+        # Final layer norm before output.
+        self.final_layernorm = LayerNorm(self.hidden_size, eps=self.layernorm_epsilon)
+
+        if self.pre_seq_len is not None:
+            for param in self.parameters():
+                param.requires_grad = False
+            self.prefix_tokens = torch.arange(self.pre_seq_len).long()
+            self.prefix_encoder = PrefixEncoder(config)
+            self.dropout = torch.nn.Dropout(0.1)
+
+            # total_params = sum(p.numel() for p in self.parameters())
+            # trainable_params = sum(p.numel() for p in self.parameters() if p.requires_grad)
+            # print("Using p-tuning v2: # trainable_params = {} / {}".format(trainable_params, total_params))
+
+    def get_input_embeddings(self):
+        return self.word_embeddings
+
+    def set_input_embeddings(self, new_embeddings: torch.Tensor):
+        self.word_embeddings = new_embeddings
+
+    def get_prompt(self, batch_size, device, dtype=torch.half):
+        prefix_tokens = self.prefix_tokens.unsqueeze(0).expand(batch_size, -1).to(device)
+        past_key_values = self.prefix_encoder(prefix_tokens).type(dtype)
+        past_key_values = past_key_values.view(
+            batch_size,
+            self.pre_seq_len,
+            self.num_layers * 2,
+            self.num_attention_heads,
+            self.hidden_size // self.num_attention_heads
+        )
+        # seq_len, b, nh, hidden_size
+        past_key_values = self.dropout(past_key_values)
+        past_key_values = past_key_values.permute([2, 1, 0, 3, 4]).split(2)
+        # past_key_values = [(v[0], v[1]) for v in past_key_values]
+        return past_key_values
+
+    @add_start_docstrings_to_model_forward(CHATGLM_6B_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
+    @add_code_sample_docstrings(
+        checkpoint=_CHECKPOINT_FOR_DOC,
+        output_type=BaseModelOutputWithPastAndCrossAttentions,
+        config_class=_CONFIG_FOR_DOC,
+    )
+    def forward(
+            self,
+            input_ids: Optional[torch.LongTensor] = None,
+            position_ids: Optional[torch.LongTensor] = None,
+            attention_mask: Optional[torch.Tensor] = None,
+            past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
+            inputs_embeds: Optional[torch.LongTensor] = None,
+            use_cache: Optional[bool] = None,
+            output_attentions: Optional[bool] = None,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor, ...], BaseModelOutputWithPast]:
+
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if self.gradient_checkpointing and self.training:
+            if use_cache:
+                logger.warning_once(
+                    "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
+                )
+                use_cache = False
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
+        elif input_ids is not None:
+            batch_size, seq_length = input_ids.shape[:2]
+        elif inputs_embeds is not None:
+            batch_size, seq_length, _ = inputs_embeds.shape[:2]
+        else:
+            raise ValueError("You have to specify either input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds = self.word_embeddings(input_ids)
+
+        if past_key_values is None:
+            if self.pre_seq_len is not None:
+                past_key_values = self.get_prompt(batch_size=input_ids.shape[0], device=input_ids.device,
+                                                  dtype=inputs_embeds.dtype)
+            else:
+                past_key_values = tuple([None] * len(self.layers))
+
+            if attention_mask is None:
+                attention_mask = self.get_masks(
+                    input_ids,
+                    device=input_ids.device
+                )
+
+
+            if position_ids is None:
+                MASK, gMASK = self.config.mask_token_id, self.config.gmask_token_id
+                seqs = input_ids.tolist()
+
+                mask_positions, use_gmasks = [], []
+                for seq in seqs:
+                    mask_token = gMASK if gMASK in seq else MASK
+                    use_gmask = mask_token == gMASK
+                    mask_positions.append(seq.index(mask_token))
+                    use_gmasks.append(use_gmask)
+
+                position_ids = self.get_position_ids(
+                    input_ids,
+                    mask_positions=mask_positions,
+                    device=input_ids.device,
+                    use_gmasks=use_gmasks
+                )
+
+        if self.pre_seq_len is not None and attention_mask is not None:
+            prefix_attention_mask = torch.ones(batch_size, 1, input_ids.size(-1), self.pre_seq_len).to(
+                attention_mask.device)
+            prefix_attention_mask = (prefix_attention_mask < 0.5).bool()
+            attention_mask = torch.cat((prefix_attention_mask, attention_mask), dim=3)
+
+        # [seq_len, batch, hidden_size]
+        hidden_states = inputs_embeds.transpose(0, 1)
+
+        presents = () if use_cache else None
+        all_self_attentions = () if output_attentions else None
+        all_hidden_states = () if output_hidden_states else None
+
+        if attention_mask is None:
+            attention_mask = torch.zeros(1, 1, device=input_ids.device).bool()
+
+        else:
+            attention_mask = attention_mask.to(input_ids.device)
+
+        for i, layer in enumerate(self.layers):
+
+            if output_hidden_states:
+                all_hidden_states = all_hidden_states + (hidden_states,)
+            layer_past = past_key_values[i]
+
+            if self.gradient_checkpointing and self.training:
+                layer_ret = torch.utils.checkpoint.checkpoint(
+                    layer,
+                    hidden_states,
+                    position_ids,
+                    attention_mask,
+                    torch.tensor(i),
+                    layer_past,
+                    use_cache,
+                    output_attentions
+                )
+            else:
+                layer_ret = layer(
+                    hidden_states,
+                    position_ids=position_ids,
+                    attention_mask=attention_mask,
+                    layer_id=torch.tensor(i),
+                    layer_past=layer_past,
+                    use_cache=use_cache,
+                    output_attentions=output_attentions
+                )
+
+            hidden_states = layer_ret[0]
+
+            if use_cache:
+                presents = presents + (layer_ret[1],)
+
+            if output_attentions:
+                all_self_attentions = all_self_attentions + (layer_ret[2 if use_cache else 1],)
+
+        # Final layer norm.
+        hidden_states = self.final_layernorm(hidden_states)
+
+        if output_hidden_states:
+            all_hidden_states = all_hidden_states + (hidden_states,)
+
+        if not return_dict:
+            return tuple(v for v in [hidden_states, presents, all_hidden_states, all_self_attentions] if v is not None)
+
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=presents,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attentions,
+        )
+
+
+class ChatGLMForConditionalGeneration(ChatGLMPreTrainedModel):
+    def __init__(self, config: ChatGLMConfig, empty_init=True):
+        super().__init__(config)
+        if empty_init:
+            init_method = skip_init
+        else:
+            init_method = default_init
+
+        # self.hidden_size = config.hidden_size
+        # self.params_dtype = torch.half
+        # self.vocab_size = config.vocab_size
+        self.max_sequence_length = config.max_sequence_length
+
+        self.position_encoding_2d = config.position_encoding_2d
+
+        self.transformer = ChatGLMModel(config, empty_init=empty_init)
+
+        self.lm_head = init_method(
+            nn.Linear,
+            config.hidden_size,
+            config.vocab_size,
+            bias=False,
+            dtype=torch.half
+        )
+
+        self.config = config
+
+        self.quantized = False
+
+        if self.config.quantization_bit:
+            self.quantize(self.config.quantization_bit, self.config.quantization_embeddings, use_quantization_cache=True, empty_init=True)
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def _update_model_kwargs_for_generation(
+        self,
+        outputs: ModelOutput,
+        model_kwargs: Dict[str, Any],
+        is_encoder_decoder: bool = False,
+        standardize_cache_format: bool = False,
+    ) -> Dict[str, Any]:
+        # update past_key_values
+        model_kwargs["past_key_values"] = self._extract_past_from_model_output(
+            outputs, standardize_cache_format=standardize_cache_format
+        )
+
+        # update attention mask
+        if "attention_mask" in model_kwargs:
+            attention_mask = model_kwargs["attention_mask"]
+            if attention_mask is not None and attention_mask.dtype == torch.bool:
+                attention_mask = torch.cat(
+                    [attention_mask, attention_mask.new_ones((*attention_mask.shape[:3], 1))], dim=3)
+                new_attention_mask = attention_mask[:, :, -1:].clone()
+                new_attention_mask[..., -1] = False
+                model_kwargs["attention_mask"] = torch.cat(
+                    [attention_mask, new_attention_mask], dim=2
+                )
+
+        # update position ids
+        if "position_ids" in model_kwargs:
+            position_ids = model_kwargs["position_ids"]
+            new_position_id = position_ids[..., -1:].clone()
+            new_position_id[:, 1, :] += 1
+            model_kwargs["position_ids"] = torch.cat(
+                [position_ids, new_position_id], dim=-1
+            )
+
+        return model_kwargs
+
+    def prepare_inputs_for_generation(
+            self,
+            input_ids: torch.LongTensor,
+            past: Optional[torch.Tensor] = None,
+            past_key_values: Optional[torch.Tensor] = None,
+            attention_mask: Optional[torch.Tensor] = None,
+            position_ids: Optional[torch.Tensor] = None,
+            **kwargs
+    ) -> dict:
+        batch_size, seq_length = input_ids.shape
+        MASK, gMASK = self.config.mask_token_id, self.config.gmask_token_id
+        seqs = input_ids.tolist()
+        mask_positions, use_gmasks = [], []
+        for seq in seqs:
+            mask_token = gMASK if gMASK in seq else MASK
+            use_gmask = mask_token == gMASK
+            mask_positions.append(seq.index(mask_token))
+            use_gmasks.append(use_gmask)
+
+        # only last token for input_ids if past is not None
+        if past is not None or past_key_values is not None:
+            last_token = input_ids[:, -1].unsqueeze(-1)
+            if attention_mask is not None and attention_mask.dtype == torch.bool:
+                attention_mask = attention_mask[:, :, -1:]
+            else:
+                attention_mask = None
+            if position_ids is not None:
+                position_ids = position_ids[..., -1:]
+            else:
+                context_lengths = [seq.index(self.config.bos_token_id) for seq in seqs]
+                if self.position_encoding_2d:
+                    position_ids = torch.tensor(
+                        [[mask_position, seq_length - context_length] for mask_position, context_length in
+                         zip(mask_positions, context_lengths)], dtype=torch.long, device=input_ids.device).unsqueeze(-1)
+                else:
+                    position_ids = torch.tensor([mask_position for mask_position in mask_positions], dtype=torch.long,
+                                                device=input_ids.device).unsqueeze(-1)
+
+            if past is None:
+                past = past_key_values
+            return {
+                "input_ids": last_token,
+                "past_key_values": past,
+                "position_ids": position_ids,
+                "attention_mask": attention_mask
+            }
+        else:
+            if attention_mask is not None and attention_mask.dtype != torch.bool:
+                logger.warning_once(f"The dtype of attention mask ({attention_mask.dtype}) is not bool")
+                attention_mask = None
+            if attention_mask is None:
+                attention_mask = self.get_masks(
+                    input_ids,
+                    device=input_ids.device
+                )
+            if position_ids is None:
+                position_ids = self.get_position_ids(
+                    input_ids,
+                    device=input_ids.device,
+                    mask_positions=mask_positions,
+                    use_gmasks=use_gmasks
+                )
+
+            return {
+                "input_ids": input_ids,
+                "past_key_values": past,
+                "position_ids": position_ids,
+                "attention_mask": attention_mask
+            }
+
+    def forward(
+            self,
+            input_ids: Optional[torch.Tensor] = None,
+            position_ids: Optional[torch.Tensor] = None,
+            attention_mask: Optional[torch.Tensor] = None,
+            past_key_values: Optional[Tuple[torch.FloatTensor]] = None,
+            inputs_embeds: Optional[torch.Tensor] = None,
+            labels: Optional[torch.Tensor] = None,
+            use_cache: Optional[bool] = None,
+            output_attentions: Optional[bool] = None,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+    ):
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        transformer_outputs = self.transformer(
+            input_ids=input_ids,
+            position_ids=position_ids,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        hidden_states = transformer_outputs[0]
+
+        lm_logits = self.lm_head(hidden_states).permute(1, 0, 2).contiguous()
+
+        loss = None
+        if labels is not None:
+            lm_logits = lm_logits.to(torch.float32)
+
+            # Shift so that tokens < n predict n
+            shift_logits = lm_logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = CrossEntropyLoss(ignore_index=-100)
+            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
+
+            lm_logits = lm_logits.to(hidden_states.dtype)
+            loss = loss.to(hidden_states.dtype)
+
+        if not return_dict:
+            output = (lm_logits,) + transformer_outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=lm_logits,
+            past_key_values=transformer_outputs.past_key_values,
+            hidden_states=transformer_outputs.hidden_states,
+            attentions=transformer_outputs.attentions,
+        )
+
+    @staticmethod
+    def _reorder_cache(
+            past: Tuple[Tuple[torch.Tensor, torch.Tensor], ...], beam_idx: torch.LongTensor
+    ) -> Tuple[Tuple[torch.Tensor, torch.Tensor], ...]:
+        """
+        This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or
+        [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct
+        beam_idx at every generation step.
+
+        Output shares the same memory storage as `past`.
+        """
+        return tuple(
+            (
+                layer_past[0].index_select(1, beam_idx.to(layer_past[0].device)),
+                layer_past[1].index_select(1, beam_idx.to(layer_past[1].device)),
+            )
+            for layer_past in past
+        )
+
+    def process_response(self, response):
+        response = response.strip()
+        response = response.replace("[[训练时间]]", "2023年")
+        punkts = [
+            [",", "，"],
+            ["!", "！"],
+            [":", "："],
+            [";", "；"],
+            ["\?", "？"],
+        ]
+        for item in punkts:
+            response = re.sub(r"([\u4e00-\u9fff])%s" % item[0], r"\1%s" % item[1], response)
+            response = re.sub(r"%s([\u4e00-\u9fff])" % item[0], r"%s\1" % item[1], response)
+        return response
+
+    @torch.no_grad()
+    def chat(self, tokenizer, query: str, history: List[Tuple[str, str]] = None, max_length: int = 2048, num_beams=1,
+             do_sample=True, top_p=0.7, temperature=0.95, logits_processor=None, **kwargs):
+        if history is None:
+            history = []
+        if logits_processor is None:
+            logits_processor = LogitsProcessorList()
+        logits_processor.append(InvalidScoreLogitsProcessor())
+        gen_kwargs = {"max_length": max_length, "num_beams": num_beams, "do_sample": do_sample, "top_p": top_p,
+                      "temperature": temperature, "logits_processor": logits_processor, **kwargs}
+        if not history:
+            prompt = query
+        else:
+            prompt = ""
+            for i, (old_query, response) in enumerate(history):
+                prompt += "[Round {}]\n问：{}\n答：{}\n".format(i, old_query, response)
+            prompt += "[Round {}]\n问：{}\n答：".format(len(history), query)
+        inputs = tokenizer([prompt], return_tensors="pt")
+        inputs = inputs.to(self.device)
+        outputs = self.generate(**inputs, **gen_kwargs)
+        outputs = outputs.tolist()[0][len(inputs["input_ids"][0]):]
+        response = tokenizer.decode(outputs)
+        response = self.process_response(response)
+        history = history + [(query, response)]
+        return response, history
+
+    @torch.no_grad()
+    def stream_chat(self, tokenizer, query: str, history: List[Tuple[str, str]] = None, max_length: int = 2048,
+                    do_sample=True, top_p=0.7, temperature=0.95, logits_processor=None, **kwargs):
+        if history is None:
+            history = []
+        if logits_processor is None:
+            logits_processor = LogitsProcessorList()
+        logits_processor.append(InvalidScoreLogitsProcessor())
+        gen_kwargs = {"max_length": max_length, "do_sample": do_sample, "top_p": top_p,
+                      "temperature": temperature, "logits_processor": logits_processor, **kwargs}
+        if not history:
+            prompt = query
+        else:
+            prompt = ""
+            for i, (old_query, response) in enumerate(history):
+                prompt += "[Round {}]\n问：{}\n答：{}\n".format(i, old_query, response)
+            prompt += "[Round {}]\n问：{}\n答：".format(len(history), query)
+        inputs = tokenizer([prompt], return_tensors="pt")
+        inputs = inputs.to(self.device)
+        for outputs in self.stream_generate(**inputs, **gen_kwargs):
+            outputs = outputs.tolist()[0][len(inputs["input_ids"][0]):]
+            response = tokenizer.decode(outputs)
+            response = self.process_response(response)
+            new_history = history + [(query, response)]
+            yield response, new_history
+
+    @torch.no_grad()
+    def stream_generate(
+            self,
+            input_ids,
+            generation_config: Optional[GenerationConfig] = None,
+            logits_processor: Optional[LogitsProcessorList] = None,
+            stopping_criteria: Optional[StoppingCriteriaList] = None,
+            prefix_allowed_tokens_fn: Optional[Callable[[int, torch.Tensor], List[int]]] = None,
+            **kwargs,
+    ):
+        batch_size, input_ids_seq_length = input_ids.shape[0], input_ids.shape[-1]
+
+        if generation_config is None:
+            generation_config = self.generation_config
+        generation_config = copy.deepcopy(generation_config)
+        model_kwargs = generation_config.update(**kwargs)
+        bos_token_id, eos_token_id = generation_config.bos_token_id, generation_config.eos_token_id
+
+        if isinstance(eos_token_id, int):
+            eos_token_id = [eos_token_id]
+
+        has_default_max_length = kwargs.get("max_length") is None and generation_config.max_length is not None
+        if has_default_max_length and generation_config.max_new_tokens is None:
+            warnings.warn(
+                f"Using `max_length`'s default ({generation_config.max_length}) to control the generation length. "
+                "This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we"
+                " recommend using `max_new_tokens` to control the maximum length of the generation.",
+                UserWarning,
+            )
+        elif generation_config.max_new_tokens is not None:
+            generation_config.max_length = generation_config.max_new_tokens + input_ids_seq_length
+            if not has_default_max_length:
+                logger.warn(
+                    f"Both `max_new_tokens` (={generation_config.max_new_tokens}) and `max_length`(="
+                    f"{generation_config.max_length}) seem to have been set. `max_new_tokens` will take precedence. "
+                    "Please refer to the documentation for more information. "
+                    "(https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)",
+                    UserWarning,
+                )
+
+        if input_ids_seq_length >= generation_config.max_length:
+            input_ids_string = "decoder_input_ids" if self.config.is_encoder_decoder else "input_ids"
+            logger.warning(
+                f"Input length of {input_ids_string} is {input_ids_seq_length}, but `max_length` is set to"
+                f" {generation_config.max_length}. This can lead to unexpected behavior. You should consider"
+                " increasing `max_new_tokens`."
+            )
+
+        # 2. Set generation parameters if not already defined
+        logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()
+        stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()
+
+        logits_processor = self._get_logits_processor(
+            generation_config=generation_config,
+            input_ids_seq_length=input_ids_seq_length,
+            encoder_input_ids=input_ids,
+            prefix_allowed_tokens_fn=prefix_allowed_tokens_fn,
+            logits_processor=logits_processor,
+        )
+
+        stopping_criteria = self._get_stopping_criteria(
+            generation_config=generation_config, stopping_criteria=stopping_criteria
+        )
+        logits_warper = self._get_logits_warper(generation_config)
+
+        unfinished_sequences = input_ids.new(input_ids.shape[0]).fill_(1)
+        scores = None
+        while True:
+            model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
+            # forward pass to get next token
+            outputs = self(
+                **model_inputs,
+                return_dict=True,
+                output_attentions=False,
+                output_hidden_states=False,
+            )
+
+            next_token_logits = outputs.logits[:, -1, :]
+
+            # pre-process distribution
+            next_token_scores = logits_processor(input_ids, next_token_logits)
+            next_token_scores = logits_warper(input_ids, next_token_scores)
+
+            # sample
+            probs = nn.functional.softmax(next_token_scores, dim=-1)
+            if generation_config.do_sample:
+                next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
+            else:
+                next_tokens = torch.argmax(probs, dim=-1)
+
+            # update generated ids, model inputs, and length for next step
+            input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)
+            model_kwargs = self._update_model_kwargs_for_generation(
+                outputs, model_kwargs, is_encoder_decoder=self.config.is_encoder_decoder
+            )
+            unfinished_sequences = unfinished_sequences.mul((sum(next_tokens != i for i in eos_token_id)).long())
+
+            # stop when each sentence is finished, or if we exceed the maximum length
+            if unfinished_sequences.max() == 0 or stopping_criteria(input_ids, scores):
+                break
+            yield input_ids
+
+    def quantize(self, bits: int, quantize_embeddings=False, use_quantization_cache=False, empty_init=False, **kwargs):
+        if bits == 0:
+            return
+
+        from .quantization import quantize, QuantizedEmbedding, QuantizedLinear, load_cpu_kernel
+
+        if self.quantized:
+            if self.device == torch.device("cpu"):
+                logger.info("Already quantized, reloading cpu kernel.")
+                load_cpu_kernel(**kwargs)
+            else:
+                logger.info("Already quantized.")
+            return self
+
+        self.quantized = True
+
+        self.config.quantization_bit = bits
+        self.config.quantization_embeddings = quantize_embeddings
+
+        self.transformer = quantize(self.transformer, bits, use_quantization_cache=use_quantization_cache, empty_init=empty_init, **kwargs)
+
+        if self.device == torch.device("cpu"):
+            dtype = torch.float32
+        else:
+            dtype = torch.half
+
+        if quantize_embeddings:
+            logger.info("Applying quantization to embeddings")
+            self.transformer.word_embeddings = QuantizedEmbedding(
+                weight_bit_width=bits,
+                weight_tensor=self.transformer.word_embeddings.weight.to(self.device),
+                num_embeddings=self.transformer.word_embeddings.num_embeddings,
+                embedding_dim=self.transformer.word_embeddings.embedding_dim,
+                dtype=dtype,
+                empty_init=empty_init,
+                device=self.transformer.word_embeddings.weight.device,
+            )
+            self.lm_head = QuantizedLinear(
+                weight_bit_width=bits,
+                weight_tensor=self.lm_head.weight.to(self.device),
+                bias_tensor=None,
+                in_features=self.lm_head.in_features,
+                out_features=self.lm_head.out_features,
+                bias=False,
+                quantized_weight=self.transformer.word_embeddings.weight,
+                quantized_weight_scale=self.transformer.word_embeddings.weight_scale,
+                dtype=dtype,
+                empty_init=empty_init,
+                device=self.lm_head.weight.device,
+            )
+
+        return self

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c1bae103ea1bfc1bd47213378dc044d55554c61d5da8d3771230013f4300f11
+size 234882351

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f00304c7adb89a5779f15a408af0d674fd60b3c341455bb7ca8d1d88b519dbe
+size 117441341

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/quantization.py

@@ -0,0 +1,515 @@
+from torch.nn import Linear, Embedding
+from torch.nn.parameter import Parameter
+import torch.nn.functional as F
+
+import os
+import bz2
+import torch
+import base64
+import ctypes
+from transformers.utils import logging
+
+from typing import List
+from functools import partial
+
+logger = logging.get_logger(__name__)
+
+try:
+    from cpm_kernels.kernels.base import LazyKernelCModule, KernelFunction, round_up
+
+    class Kernel:
+        def __init__(self, code: bytes, function_names: List[str]):
+            self.code = code
+            self._function_names = function_names
+            self._cmodule = LazyKernelCModule(self.code)
+
+            for name in self._function_names:
+                setattr(self, name, KernelFunction(self._cmodule, name))
+
+    quantization_code = "$QlpoOTFBWSZTWU9yuJUAQHN//////////f/n/8/n///n//bt4dTidcVx8X3V9FV/92/v4B7/AD5FBQFAAAChSgKpFCFAFVSigUAAAEKhSgUUqgFBKigqVREQAABQBQIANDTTIGI00BkZBkNGE0A0BkBkGQGRkaNAaAGQNBoGgDIAAYIGTI0DQAQAaGmmQMRpoDIyDIaMJoBoDIDIMgMjI0aA0AMgaDQNAGQAAwQMmRoGgAgA0NNMgYjTQGRkGQ0YTQDQGQGQZAZGRo0BoAZA0GgaAMgABggZMjQNABABoaaZAxGmgMjIMhowmgGgMgMgyAyMjRoDQAyBoNA0AZAADBAyZGgaAAmqU1NEgJqnptU/Sn4jRR6J6epk2pqb1Q/SgAPUGgyNNGjQ2SBpoAZAAGg0NB6mgDIAAAAA2oaApSREBNAARhGiYEaEwU8pvImlP0k2aam1GaGqbFNM1MHpTwmkepmyU9R6nqPKekHqNNPUxNGhp6n6p6QaZ6o9TG1GMqcoV9ly6nRanHlq6zPNbnGZNi6HSug+2nPiZ13XcnFYZW+45W11CumhzYhchOJ2GLLV1OBjBjGf4TptOddTSOcVxhqYZMYwZXZZY00zI1paX5X9J+b+f4e+x43RXSxXPOdquiGpduatGyXneN696M9t4HU2eR5XX/kPhP261NTx3JO1Ow7LyuDmeo9a7d351T1ZxnvnrvYnrXv/hXxPCeuYx2XsNmO003eg9J3Z6U7b23meJ4ri01OdzTk9BNO96brz+qT5nuvvH3ds/G+m/JcG/F2XYuhXlvO+jP7U3XgrzPN/lr8Sf1n6j4j7jZs+s/T0tNaNNYzTs12rxjwztHlnire3Nzc3N1wuBwOBwXBvZfoHpD7rFmR99V5vj3aXza3xdBbXMalubTg/jIv5dfAi54Pdc75j4z412n3Npj3Ld/ENm7a3b/Cod6h/ret1/5vn/C+l+gdslMvgPSLJ8d8q+U66fevYn/tW1chleEtNTGlcHCbLRlq0tHzF5tsbbZZfHjjLgZu42XCuC3NrdjTasZGNzgxPIrGqp7r3p7L2p5XjnpPSmTd5XtzqnB6U87zzg1Ol0zd0zsLszxR6lkxp35u6/teL0L0W922cR7Lu1lpL9CsHirzuM2T+BgsyViT6LHcm0/Vr6U/7LGGyJeqTEjt0PHWhF5mCT7R9mtlDwriYv0Tyr/OxYt6qp5r0mPVT0608TqnqMZaarU2nFwrTzzlrs1ed7z1ux60wyr4ydCaTi3enW8x68x0zU7tXSlcmPSW1mGpWJMg4zmPC2lK96tp0OE80y4MfEvnZj8zGluR6b22ki1Ou9V2nCd9xovcPvcYMZYy0lvN60ScZ45vN6yeCeeXFb1lVjnnCar5fwXwE2bzJ4HI1XVPXfXZMm44GUsMpYsmLB65TuVdm0cl0b+i/wGNN66XjeV7zuPpHcnK/juhhjdfId5jMdE5nN0dGmmm2zZs2cexD5n9p/dY352XsvXHaZNWWsmmS1atjR452nYudzvqv2HMRyvNNnlMcDl3R2+yx2uVrBubTW9icHDVtbNXlZm7jma1rM4VurZZd2y6nUau7ZXZ7bVU+mnoOVxZGMrVmvX60605JwmzGZhhhjTWtaaaMaaGTGmNMZasY0iX8VMUl8eepaIrzGSpemWOQyZORk2bNpjUybMmxqYmknCGCFynutfksaZpjTNMaaatM0xsxcGR0sociNqxNSmhhR1ZJPbsn8qyF0t2qH6iYBclclalbtTTcHTDsPaX6rlnElph2Jyumumtynv2Kk8GI7rsvXbIcJgHJOSaSXnnGaI3m87RtVXJOZ/YtgdTE6Wpha6ZlE8ayXkef1fh602r2WwvfMXtMdLlkfnLFdYYwYso+bWqm7yJqHXZGw2nrS5ZanSYnWlxBxMF1V940K2wdrI7R6OYf7DGGamMmTSbRhlS45xmVOumF1EyPCmHrrN8wwZOOrdNtLeMtzFzDlWnfTBxMk2NaXIZHBYxYLD4w8yju0ao65Vz1OIXoS9dLanwCe1PWrYuWMqf1if1z2k2yYfKJ741PDgno1ZQ8DRqvUny3mNoWTzGO6m1DkrJI8JiR5cSd+vZdGOO8nrMoc5+NDUFsMSXaZJeNlMmGLtJsovOsUp7I9S5VojKxF6bTVEelXqlfJobQr3LozSh2Jk7VcrVMfhXqszGWMzNqGhqZY0OadxkyyMssKugZR0KNFXBHlqwmJgTE/BNVMk6ItJXZMR0H47GpXv/DMOvNkmVuaV1PRfEdxuqc7Hcd+ZV/zTLaRxWk0nl9CdCeM6mn5rstHIBcpiuwmUZXeq81DacHI2rmrZ5SuE5mOZd6LQrZg9mx32TprA8BMo5jKN6yLTCi3WzQaZSuhzTtM1fUTGVpG8Tw+KXI0tjEpiWxtLYynOlktSbVlaI5kxP8TDH8kx50xoxi5KcA4pcja8KWLRlO/Ks6q06ergnvm1ca3Tq8Uw7LTUsmWyctXPWmpitl/uvGcWTGXGuAXDfhqazGmjkxcJW5hMMMMpYsXl2TZYtVOddG3XCarUt6Ptq9CZXSNzyuRzqRZOjsxdBbFVz6OA5HI43r1jityVlVpVkxmOsyaYWE1NTGq1sOVh36mHMcxtSvcy70edG0ZGR3I1Go1GRlV7mWWo1G0ZGRqlvH40l7o4m5xMWLLLYyNjnqc8556mdPqLJ31n/1nWOncxzG1tizrHs/Z+d2vP/B/l8wdJ6rHUn2nbbDq4p6htFtYzMMMTaZis1K5GKzGNmxhmUx2DDlZ/qNnIx41xnaMfCZWYaZWtNLTNW8ND4Fw1MyZOCdM428suKG1ehW8TesOydg7J+YYcD4cYR+8dFK6M4E3HM9ZfRNNL+Sn6rsl4DsrDl2HpPCnfxjGXtbZtYys1ttlyJ4T+BvexjGWRjMszK4Jpc77D3GyuVD7q0+G8m9G+2+rGm7cOR2y7FdtY2XUYx/oNlfRYxhMYyYZkyyg55enna9Kt/FFi6GMMwYwdwxWgxGMLKYmUyGExTKMZkMFhkymKuh0NOBNnBu+23LdwDoZYYzGGMxtORaTU1pjTGWTTGGtMrNWUsyyTTLLG1qy2ZjbK2DBllWqxMtBMaYZQmcE7zvvRcTkclUwdkxTaSdyySt/7fpL+T1v516Ji97fwr5JbLu305zMn5+GMTTZ9F+y7ExwmGVfG44yxn3dLv6l5i+Wth1jCrDq21nW9LqvvDzz3Vf3LLH/O/32TJ/erx3bXftO4eF+G956D952K/An4NfvOpjFjExjevP/UmE0fIoZXx6/w6lX/no3D0bLt+ixjieBM6ksRd0yB4Lt2SwYNE+gd1detlZWUnpiZfGfFaK+4PyCa/v18V8X75pe9fLXzp7l3VjF76vWZmHwGz1IZNWT7b8yddJ4q5kyrVdfru6atWc7bVYztL9Jf4GXvT+Y8m9/YsXP6H018a8D4XVOqvfzqeR+6yZOD8dPv0+U7/q5Pl+2dNb0MjzGVH5p6MNQ7cOWvw62U9aHE8DprDek+McLyvDz+te+9Zhq5+YTruufMcWMabqysTmZVWjKPfnK0wyVcrsuhjZRdLkHNvD72b9abriOSGIxiLixMOoalNPXzy+wT/tf+U6HHONfsz+xe8ufHBdQWWGWLA9if0rsnmrxK5LvRZQeWsTCsrmOYy8VteVfuRfcVTtDLItLIsMYxZLdU/DbtSemxF6Z6Zo5WBXE4tFdCyVMMXMTEMZXVlS6Xec2T4e0tHsRcEuWshcJ2YsNF5rUx1E8ifCq6Z+ZP7qdCeu/aTwFd53l16/o0NOw6O3dLavP4Hbi4RdmuDk6DoYaninC0+o4uZjbJ7Rxeu0/FbuFg+q7DVS6fQe0rZ6NDGUNNU6DEqOaLTicKnYZMnBWruljQxoaS3dZhocDge0bSTyOvdAbG5hxe2xji7E/L55xX13wWNDi6HCekcFxfCPGxY0MXC+s7afWaMdDyjyr+o8Rudm/NabOZvdl274zH4f5XK9z6On1Pe/K5TdPAslg77BjuO6Y3eO7GqvOPG/stknp1leyvLL0Z7bl9I4noMvLkzytLhWYzrOZzLXCORe028rORzOg4N/L0HlMOQ3Pgmnbb6KczlabORpu980q37TBqRu0/p3PO6234Bl03Ynuz+9W7gnsEcmvYaYY3aMYY0wx3pYd+ujsXauWdaY5Xkbtl23fPzFHiDB/QMo0yFjBllYxTQYYyxkrwn7JufwJ/PfgJ+C83X69ni6zvXcnyXabv0ncbLwsceS+RNlyN2mnneJtX0ngYO0+e+0+UnA+Wch3ji8hj5an4h+i6XBySU4n+R0roVcbw5yvHrmr4Yw8Y7x6c+9POPYHI5HI5HI5HI5HGXGww4nE4nrVyOR8XeqPEO7PLOiukYa3Novk5hV4cdtYZLI93e+uxff2jRo0aNGjRo0aNG1bVtW1dy3m83m8+tQ5ZzHw3nObwOu8La9Rc1dtkdS8A3eTk823tnktXWlxN6Oixe06zrN70Isd9jiOgZFq9yfkPqP/SLhN2Myl8jDM43bl1nbcb4cO57jlh8Jow6pzXZdL4dyODTuuhu77FyO27DdwdRxmvO+O+3N2+BdqyTwLHVczDVY4UPE4O66/ZO2cx1LFzVdSXtF7G4HMbrauOHRw6c8FdZ5m9fHZHYZXfTlZquyynSyTTKke6vcffSD9pzPA/G7n7jxPmuhc1DHMynPMrGL6AdewYmwu5ko+UUyTwrMv27rPH1v1nGqd87+p6N6LU8k3NEng53xXyHS97+44OSg/sy/hn+Se6yfYNjW0/uTgP+PvWYzLMmjhcLB/gGpri6H83/84eUXWT6T9Hsv7785z/7z4icpW+zfXypuR7rx/gMdZb1/wC678pcs8/2a3mDitGHxl9mfPlll5MafWWqxk/eYuTDgcNMzDGWLWvsuglNxs53GtN6uWpktlW1tZZYcuinMMWmnNnJydze3b2Y1McBxrBkXw799izLMZZYyy0TkbsGM4p03S2uVu5s/XXUdSdec6smVxZYYGpVmT8A+8ajuEyV5FatkvVru2x6uxGXXbH4A+jvgP4GMYy3iPLXzq/6z65+E005ey+cwMZD3fZcqc6xpjTFjQ0P3U+e++cPYmTIwj0nrK5NPTfl3WvpfLtXDcb2HQMudYOxFXQBor4L4T6vrOauFctYXJQ++NUWmJe5bmx1jDiZS1dTqWxo4GR8jm3fttpmPHppk9PEyv4/y8/sO07XacOmcqc0x2Vi9BvNJvN5oW8x4mOsydpidRxMYJPx06m1bqPzq9KtK8sxXNXFodD/+MYYaJTLwOhc9brCsV18oOR1i4tXChyTkq4lf4y1Ke+9axjDHqs1mfBbMXuP4Hzi+X7t8vzv7bHerrUPgPCxhjre4fXdfLNtNM+Jd+Zdh8xd8wP87uNPoPgv4W7/5P2BuxfsMabNnMnza+54Pdi5U671GPZY8CehX8Voeoo7FHpkeEc6715FwHZrIrUrHaviPUbPZHND+IhczrP6FcYvhOZ0Di/ETt0OI+YwNWR9r7tpf6WDeZKZDB1+z2IthOl1mPyb5FluvEx9h9d0NnM0Y1XPFkWIsk1WotJ0PBMmkvjvQTd0e71tfeV+8r8lQ/tpzpsmxJ+InrI/dj2UajUajVTUajatRqNRtGo1Go1Go4wjeMpZFMVV9CHbofPraLsJ3JpWV2XOoanCuFky4y3PPNxucK2uKC1Lbdb1eo+m5XomN6HfeZsabHLHRX/K+offtNGGmHWctcVcG44MdSqsOLY9VzX+Zxfxn2HPdWTpzWvkrtJ8M5zorrKcquRytJ5N5DZmcaW02l76nWO+BqPXm1A2Ry/0q71dH/mqrqeFjkYxjEXtsX8qubTk67rGycyqsdm4tZx5D6D5hhi0waaWmiaMP81Yjii5qxPlPuU/GfTL1Y5E6Jyfiq63qTa39A4J0sOGDgO9WF9bOXl0XfPRbsY2bPNKPy1YrFYrFYmRhhlTIyMjJWJYZHXuCXI8OoXsvfljGLFicNifpp2XunoPiG1wtx3p1Tah+/DD66OnVtVXP9rKbVxOnL0tR/rHtqB5UDErUVcl11D4qqvjpOcxX7armUNJB3LpW6bxVvD08e8h3odKKvyCFZBdSh2FVcST9xV3n3T8t1j7Kr9qgrqXg+13Pt5U7JCvFXVIV1YG5lRhkVYZJYYDDD4KOIMoHCp26WS8GB7uBh2zIdgq/PKyInjV2STShuoapUdCpX1yTwqq/z1VvET7Kh5nVPkO8YyxjLt2MaaMmWTLQvx3qnzltnXW0p2jxgbEtSny/Osv8Y9pLMXYoHVPAhkVdWVeODhR6q9/Sxe2liwwZWMVvFXfRkeIDxAePUPIrdJ4ey6yquzH+PD/bUOWAu05qVHtFd8rrKHSoeNIOUqrYr3FXyToqfYJgwmJdKpXXOwYYegNNGMzfZPp/t3t/DVs4zjNTN61rRqaWaa4NYbRjTa0tWwy2Y2tGN8ZO8ofNKq4j9SL7I+cSm4/6ovLV5HNXLI0jJidwrtk6ynCaP6Z++GjRlWS3tLeW129Mi9evxU9mtz6s5J3Z7M2ngTgnKvmpomxpaLCzPfmx0JWE+m3NLDDGOX47RctdYYNK5jakdqLkRlI39n590T5zctGSwwZZDJj6kW8XSi6ot2MmWWJ0DUT3nuvebBudScjZ79g8cWJ8av0k+/bE5WKd5MdbFpbDVMxu1DVMmtNZGJvq1mtRbn6M+g/kP0FwDwr7quZs7xosNGpbscyxhhd9TyJyFwbLcxlTasg75vW7TsV5K7ji44XPMMrdoj+Y3rT0Hie62nlYV/pwczzOmdLqLhYkzGMzCZWGMQzGMSsZYY6Di1t4nlJ+Em63mJxrVLxPbYxNEdgc1dU2iOKyoYYWjNrEeHTYybVk0atSa7ehuwsWMWTqn1TrnS6hYsi71d1+s+k+ic70e20fzE/VaTdxT9ZtU4GIXdeNx3X77guYYfpHeTQjaMX6brOu4OY4K7Y2d9mbHarI5ox3p4GpJ2Vd/Tst60f7j999pppjR+Q/Qf8J/VaORs3cji7FfFuN61+ui9s8hix1OCh5KGVV23BPXvZfz3CLyHpix+exi8z/KnCnosY2eunor+cxyPO/xJ0vKey9OvE9VjqaYu0x3Z3jd6o2b1T12D+F8l232lwaaacD5LE8LBxu7WTlbWraWpew8Xexjel3E+wWD4APITdNqR8F3R3T0lunCQ4GaE9R37DxeCYfcHi4xci5ovKfxVs55y2hf+65E/Xdp6jR5nrebTmi5incpkyOjs50JvrZwstbbW6kfuuQw+2mykf/EXNFzxfKTrxew929TR6bWnGL//F3JFOFCQT3K4lQ"
+
+    kernels = Kernel(
+        bz2.decompress(base64.b64decode(quantization_code)),
+        [
+            "int4WeightCompression",
+            "int4WeightExtractionFloat",
+            "int4WeightExtractionHalf",
+            "int8WeightExtractionFloat",
+            "int8WeightExtractionHalf",
+        ],
+    )
+except Exception as exception:
+    kernels = None
+    logger.warning("Failed to load cpm_kernels:", exception)
+
+
+class W8A16Linear(torch.autograd.Function):
+    @staticmethod
+    def forward(ctx, inp: torch.Tensor, quant_w: torch.Tensor, scale_w: torch.Tensor, weight_bit_width):
+        ctx.inp_shape = inp.size()
+        ctx.weight_bit_width = weight_bit_width
+        out_features = quant_w.size(0)
+        inp = inp.contiguous().view(-1, inp.size(-1))
+        weight = extract_weight_to_half(quant_w, scale_w, weight_bit_width)
+        ctx.weight_shape = weight.size()
+        output = inp.mm(weight.t())
+        ctx.save_for_backward(inp, quant_w, scale_w)
+        return output.view(*(ctx.inp_shape[:-1] + (out_features,)))
+
+    @staticmethod
+    def backward(ctx, grad_output: torch.Tensor):
+        inp, quant_w, scale_w = ctx.saved_tensors
+        weight = extract_weight_to_half(quant_w, scale_w, ctx.weight_bit_width)
+        grad_output = grad_output.contiguous().view(-1, weight.size(0))
+        grad_input = grad_output.mm(weight)
+        grad_weight = grad_output.t().mm(inp)
+        return grad_input.view(ctx.inp_shape), grad_weight.view(ctx.weight_shape), None, None
+
+
+class W8A16LinearCPU(torch.autograd.Function):
+    @staticmethod
+    def forward(ctx, inp: torch.Tensor, quant_w: torch.Tensor, scale_w: torch.Tensor, weight_bit_width, quantization_cache=None):
+        ctx.inp_shape = inp.size()
+        ctx.weight_bit_width = weight_bit_width
+        out_features = quant_w.size(0)
+        inp = inp.contiguous().view(-1, inp.size(-1))
+        weight = extract_weight_to_float(quant_w, scale_w, weight_bit_width, quantization_cache=quantization_cache)
+        ctx.weight_shape = weight.size()
+        output = inp.mm(weight.t())
+        ctx.save_for_backward(inp, quant_w, scale_w)
+        return output.view(*(ctx.inp_shape[:-1] + (out_features,)))
+
+    @staticmethod
+    def backward(ctx, grad_output: torch.Tensor):
+        inp, quant_w, scale_w = ctx.saved_tensors
+        weight = extract_weight_to_float(quant_w, scale_w, ctx.weight_bit_width)
+        grad_output = grad_output.contiguous().view(-1, weight.size(0))
+        grad_input = grad_output.mm(weight)
+        grad_weight = grad_output.t().mm(inp)
+        return grad_input.view(ctx.inp_shape), grad_weight.view(ctx.weight_shape), None, None
+
+
+default_cpu_kernel_code_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "quantization_kernels.c")
+default_cpu_kernel_code = "QlpoOTFBWSZTWXLbSoQAAgzbgERwQXxmTwAAr/ff3kABt0Q2oRVT0hpo9RtEAAAAyBEiSQ9EGjQGQAAAwANGhowjJoNGmgMEUplMTNSMJ5TQaDJpsoMyRMj8P4mZzFSVVwqSXG8GG7MlVwiToYEQwVD7noBxMhNfkeZYtYFtbgOBUSIGtIQjhNHCEnPJsadhb3yBmRIOD3TeAtNLSaU5GgvKUBWSNuuOIHmVt0YhW6rsmDMDUjeUJGJ64R1Jm5lrh0Aa0tKjhFwPdWcGogxLDSXPWQUWTM8Sd3Qz1HMYNxx3HMeiNqNo4jeRDEfZ3gUSHIcU/heomq0vEzL1Msz5KKGxH8FrNOYw3KaxdqaEmNHYMxJFgQbR0DyRknL2L4kwUSxKRdhjRpEtUqilVfggFL1klaMS3PPRDfNqbBOPWO7m4JTVGhS9QTBDDJaEbLbrUQNB+IpJSKQbG5SZZ5gkwJEhJ3aYKJipZ/i7kinChIOW2lQg"
+default_cpu_parallel_kernel_code_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "quantization_kernels_parallel.c")
+default_cpu_parallel_kernel_code = "QlpoOTFBWSZTWUzax5EAALXbgERwSX1mTwAAr/ff3kACNyXUbZYwBpoaNGIyAaADQwRSaVP9QoMg0A2oAPU0AEUkU9GaaKMaQB6gA09T1ARRKnpk0niaJkaaNDJ6g0DTIKVKfZ/g6v1Kem5LJLa0WmkukkuCIHUqWbtJGJMsCSQFiPEIYHgBIZDzR8R6REbYxIqD2Cu7lMkFoPu6LmHeOAy0GF83Tc40jgmTs4HnCe60QfJa2bDBZ0Y1lhgbiZjW8SNsAKCk42UOEdjWN3KoiCIYeQUCCKWIyHewhtSoInLKSG22l4jKM2ZDCVKtBm3OTYBl3jsVqMImtj7PQw7xKxLXQzwgJaPPgW1fRhrvPJICl4YFDYfNbkbBh5JDgrazFml50xEQQwQUjxNwE0IDSofLzSg7UNVKn+Rr1KErzBHUxBqdHRlXzqYsIa5K9Y0UuE2ugw3g5KYofm7AaGNTzJSMhcchhxdaU4JZ0F1UNgQ8XcGDguypqYza8yFaEoGgNRcLej+g2t0feGKFE5OY2PFluQ3q4HgycxlfvzHqo0KcM0JI8OKXtzayJFgsqC1NdUQVu8rChnA6FO3MFyGOoC9KO8ITPpYM5pRqTlczFkLES/4u5IpwoSCZtY8i"
+
+cpu_kernels = None
+
+
+class CPUKernel:
+    def __init__(self, kernel_file="", source_code=default_cpu_kernel_code_path, compile_parallel_kernel=None, parallel_num=None):
+        self.load =False
+        self.int8WeightExtractionFloat = None
+        self.int4WeightExtractionFloat = None
+        self.int4WeightCompression = None
+        self.SetNumThreads = lambda x: x
+
+        try:
+            if not os.path.exists(default_cpu_kernel_code_path):
+                with open(default_cpu_kernel_code_path, "w", encoding="utf-8") as file:
+                    code = default_cpu_kernel_code
+                    cpu_quantization_code = bz2.decompress(base64.b64decode(code)).decode()
+                    file.write(cpu_quantization_code)
+
+            if not os.path.exists(default_cpu_parallel_kernel_code_path):
+                with open(default_cpu_parallel_kernel_code_path, "w", encoding="utf-8") as file:
+                    code = default_cpu_parallel_kernel_code
+                    cpu_quantization_code = bz2.decompress(base64.b64decode(code)).decode()
+                    file.write(cpu_quantization_code)
+
+        except Exception as ex:
+            print("Error when generating default cpu kernel code(can be ignored when using custom kernels).")
+
+        if compile_parallel_kernel is None:
+            compile_parallel_kernel = bool(int(os.cpu_count()) >= 4)
+
+        if compile_parallel_kernel and source_code == default_cpu_kernel_code_path:
+            source_code = default_cpu_parallel_kernel_code_path
+
+        kernels = None
+
+        if (not kernel_file) or (not os.path.exists(kernel_file)):
+            print("No compiled kernel found.")
+            try:
+                if os.path.exists(source_code):
+                    print("Compiling kernels :", source_code)
+                    kernel_file = source_code[:-2] + ".so"
+
+                    if compile_parallel_kernel:
+                        compile_command = "gcc -O3 -fPIC -pthread -fopenmp -std=c99 {} -shared -o {}".format(source_code, kernel_file)
+                        print("Compiling", compile_command)
+                        exit_state = os.system(compile_command)
+                        if not exit_state:
+                            try:
+                                kernels = ctypes.cdll.LoadLibrary(kernel_file)
+                                print("Load kernel :", kernel_file)
+                            except:
+                                kernels = None
+                                print("Load parallel cpu kernel failed, using default cpu kernel code:")
+                                import traceback
+                                exception = traceback.format_exc()
+                                print(exception)
+                        else:
+                            print("Compile default cpu kernel failed, using default cpu kernel code.")
+                            
+                        if kernels is None:  # adjust config, use default cpu kernel
+                            compile_parallel_kernel = False
+                            source_code = default_cpu_kernel_code_path
+                            kernel_file = source_code[:-2] + ".so"
+
+                    if kernels is None:
+                        compile_command = "gcc -O3 -fPIC -std=c99 {} -shared -o {}".format(source_code, kernel_file)
+                        print("Compiling", compile_command)
+                        exit_state = os.system(compile_command)
+                        if not exit_state:
+                            try:
+                                kernels = ctypes.cdll.LoadLibrary(kernel_file)
+                                print("Load kernel :", kernel_file)
+                            except:
+                                kernels = None
+                                print("Load default cpu kernel failed:")
+                                import traceback
+                                exception = traceback.format_exc()
+                                print(exception)
+                        else:
+                            print("Compile default cpu kernel failed.")
+                else:
+                    print("Kernel source code not found.")
+                    return
+            except:
+                print("Failed to build cpu kernel:")
+                import traceback
+                exception = traceback.format_exc()
+                print(exception)
+                return
+        else:
+            try:
+                kernels = ctypes.cdll.LoadLibrary(kernel_file)
+                print("Load kernel :", kernel_file)
+            except:
+                kernels = None
+                print("Load custom cpu kernel failed:")
+                import traceback
+                exception = traceback.format_exc()
+                print(exception)
+
+        if kernels is not None:
+            self.int8WeightExtractionFloat = kernels.extract_int8_weight_to_float
+            self.int4WeightExtractionFloat = kernels.extract_int4_weight_to_float
+            self.int4WeightCompression = kernels.compress_int4_weight
+            if compile_parallel_kernel:
+                try:
+                    self.SetNumThreads = kernels.set_num_threads
+                except:
+                    print("No set_num_threads() found in kernel.")
+            self.load = True
+        else:
+            print("Failed to load kernel.")
+            return
+
+        if compile_parallel_kernel:
+            if parallel_num is None:
+                parallel_num = max(os.cpu_count() // 2, 1)
+            print("Setting CPU quantization kernel threads to", parallel_num)
+            if parallel_num < 4:
+                print("Parallel kernel is not recommended when parallel num < 4.")
+            self.SetNumThreads(parallel_num)
+
+        self.parallel_num = parallel_num
+
+
+def compress_int4_weight(weight: torch.Tensor):  # (n, m)
+    """compress weight on cpu or cuda to int4"""
+    if weight.device == torch.device("cpu"):
+        assert isinstance(cpu_kernels, CPUKernel)
+        n, m = weight.size(0), weight.size(1)
+        assert m % 2 == 0
+        m = m // 2
+        out = torch.empty(n, m, dtype=torch.int8, device="cpu")
+        cpu_kernels.int4WeightCompression(
+            ctypes.c_void_p(weight.data_ptr()),
+            ctypes.c_void_p(out.data_ptr()),
+            ctypes.c_int32(n),
+            ctypes.c_int32(m)
+        )
+        return out
+    else:
+        with torch.cuda.device(weight.device):
+            n, m = weight.size(0), weight.size(1)
+            assert m % 2 == 0
+            m = m // 2
+            out = torch.empty(n, m, dtype=torch.int8, device="cuda")
+            stream = torch.cuda.current_stream()
+
+            gridDim = (n, 1, 1)
+            blockDim = (min(round_up(m, 32), 1024), 1, 1)
+
+            kernels.int4WeightCompression(
+                gridDim,
+                blockDim,
+                0,
+                stream,
+                [ctypes.c_void_p(weight.data_ptr()), ctypes.c_void_p(out.data_ptr()), ctypes.c_int32(n), ctypes.c_int32(m)],
+            )
+            return out
+
+
+def extract_weight_to_half(weight: torch.Tensor, scale_list: torch.Tensor, source_bit_width: int):
+    if source_bit_width == 8:
+        func = kernels.int8WeightExtractionHalf
+    elif source_bit_width == 4:
+        func = kernels.int4WeightExtractionHalf
+    else:
+        assert False, "Unsupported bit-width"
+
+    with torch.cuda.device(weight.device):
+        n, m = weight.size(0), weight.size(1)
+        out = torch.empty(n, m * (8 // source_bit_width), dtype=torch.half, device="cuda")
+        stream = torch.cuda.current_stream()
+
+        gridDim = (n, 1, 1)
+        blockDim = (min(round_up(m, 32), 1024), 1, 1)
+
+        func(
+            gridDim,
+            blockDim,
+            0,
+            stream,
+            [
+                ctypes.c_void_p(weight.data_ptr()),
+                ctypes.c_void_p(scale_list.data_ptr()),
+                ctypes.c_void_p(out.data_ptr()),
+                ctypes.c_int32(n),
+                ctypes.c_int32(m),
+            ],
+        )
+        return out
+
+
+def extract_weight_to_float(weight: torch.Tensor, scale_list: torch.Tensor, source_bit_width: int, quantization_cache=None):
+    """extract weight on cpu to float32"""
+    if source_bit_width == 8:
+        func = cpu_kernels.int8WeightExtractionFloat
+    elif source_bit_width == 4:
+        func = cpu_kernels.int4WeightExtractionFloat
+    else:
+        assert False, "Unsupported bit-width"
+
+    n, m = weight.size(0), weight.size(1)
+
+    if quantization_cache is not None:
+        out = quantization_cache
+        func(
+            ctypes.c_void_p(weight.data_ptr()),
+            ctypes.c_void_p(scale_list.data_ptr()),
+            ctypes.c_void_p(out.data_ptr()),
+            ctypes.c_int32(n),
+            ctypes.c_int32(m)
+        )
+        return out.tensor
+    else:
+        out = torch.empty(n, m * (8 // source_bit_width), dtype=torch.float, device="cpu")
+        func(
+            ctypes.c_void_p(weight.data_ptr()),
+            ctypes.c_void_p(scale_list.data_ptr()),
+            ctypes.c_void_p(out.data_ptr()),
+            ctypes.c_int32(n),
+            ctypes.c_int32(m)
+        )
+        return out
+
+
+class CacheTensor():
+    def __init__(self, *args, **kwargs):
+        self.tensor = torch.empty(*args, **kwargs)
+
+    def to(self, *args, **kwargs):
+        self.tensor = self.tensor.to(*args, **kwargs)
+
+    def data_ptr(self):
+        return self.tensor.data_ptr()
+
+
+class QuantizedLinear(Linear):
+    def __init__(self, weight_bit_width: int, weight_tensor=None, bias_tensor=None, quantized_weight=None, quantized_weight_scale=None, quantization_cache=None, empty_init=False, *args, **kwargs):
+        super(QuantizedLinear, self).__init__(*args, **kwargs)
+        self.weight_bit_width = weight_bit_width
+        self.quantization_cache = quantization_cache
+
+        if (quantized_weight is not None) and (quantized_weight_scale is not None):
+            del self.weight
+            self.weight = Parameter(quantized_weight.to(kwargs["device"]), requires_grad=False)
+            self.weight_scale = Parameter(quantized_weight_scale.to(kwargs["device"]), requires_grad=False)
+        else:
+            shape = self.weight.shape
+            del self.weight
+
+            if weight_tensor is None or empty_init:
+                self.weight = torch.empty(
+                    shape[0], shape[1] * weight_bit_width // 8, dtype=torch.int8, device=kwargs["device"]
+                )
+                self.weight_scale = torch.empty(shape[0], dtype=kwargs["dtype"], device=kwargs["device"])
+            else:
+                self.weight_scale = (weight_tensor.abs().max(dim=-1).values / ((2 ** (weight_bit_width - 1)) - 1)).to(kwargs["dtype"])
+                self.weight = torch.round(weight_tensor / self.weight_scale[:, None]).to(torch.int8)
+                if weight_bit_width == 4:
+                    self.weight = compress_int4_weight(self.weight)
+
+            self.weight = Parameter(self.weight.to(kwargs["device"]), requires_grad=False)
+            self.weight_scale = Parameter(self.weight_scale.to(kwargs["device"]), requires_grad=False)
+
+        if bias_tensor is not None:
+            self.bias = Parameter(bias_tensor.to(kwargs["device"]), requires_grad=False)
+        else:
+            self.bias = None
+
+    def reset_parameters(self):
+        """To accelerate initialization"""
+        pass
+
+    def forward(self, input):
+        if self.weight.device == torch.device("cpu"):
+            output = W8A16LinearCPU.apply(input, self.weight, self.weight_scale, self.weight_bit_width, self.quantization_cache)
+        else:
+            output = W8A16Linear.apply(input, self.weight, self.weight_scale, self.weight_bit_width)
+        if self.bias is not None:
+            output = output + self.bias
+        return output
+
+    def _apply(self, fn):
+        self_obj = super()._apply(fn)
+        if self.quantization_cache is not None:
+            self.quantization_cache.to(self_obj.weight.device)
+            self.quantization_cache.to(self_obj.weight_scale.dtype)
+        return self_obj
+
+
+class QuantizedEmbedding(Embedding):  # TODO: backward, check empty_init
+    def __init__(self, weight_bit_width: int, weight_tensor=None, quantized_weight=None, quantized_weight_scale=None, empty_init=False, *args, **kwargs):
+        super(QuantizedEmbedding, self).__init__(*args, **kwargs)
+        self.weight_bit_width = weight_bit_width
+
+        if (quantized_weight is not None) and (quantized_weight_scale is not None):
+            del self.weight
+            self.weight = Parameter(quantized_weight.to(kwargs["device"]), requires_grad=False)
+            self.weight_scale = Parameter(quantized_weight_scale.to(kwargs["device"]), requires_grad=False)
+        else:
+            shape = self.weight.shape
+            del self.weight
+
+            if weight_tensor is None or empty_init:
+                self.weight = torch.empty(
+                    shape[0], shape[1] * weight_bit_width // 8, dtype=torch.int8, device=kwargs["device"]
+                )
+                self.weight_scale = torch.empty(shape[0], dtype=kwargs["dtype"], device=kwargs["device"])
+            else:
+                self.weight_scale = (weight_tensor.abs().max(dim=-1).values / ((2 ** (weight_bit_width - 1)) - 1)).half()
+                self.weight = torch.round(weight_tensor / self.weight_scale[:, None]).to(torch.int8)
+                if weight_bit_width == 4:
+                    self.weight = compress_int4_weight(self.weight)
+
+            self.weight = Parameter(self.weight.to(kwargs["device"]), requires_grad=False)
+            self.weight_scale = Parameter(self.weight_scale.to(kwargs["device"]), requires_grad=False)
+
+    def forward(self, input):
+        if self.weight.device == torch.device("cpu"):
+            original_weight = extract_weight_to_float(weight=self.weight, scale_list=self.weight_scale, source_bit_width=self.weight_bit_width)
+        else:
+            original_weight = extract_weight_to_half(weight=self.weight, scale_list=self.weight_scale, source_bit_width=self.weight_bit_width)
+        output = F.embedding(
+            input, original_weight, self.padding_idx, self.max_norm,
+            self.norm_type, self.scale_grad_by_freq, self.sparse
+        )
+        return output
+
+
+def load_cpu_kernel(**kwargs):
+    global cpu_kernels
+    cpu_kernels = CPUKernel(**kwargs)
+    assert cpu_kernels.load
+
+
+def quantize(model, weight_bit_width, use_quantization_cache=False, empty_init=False, **kwargs):
+    """Replace fp16 linear with quantized linear"""
+
+    query_key_value_quantization_cache = None
+    dense_quantization_cache = None
+    dense_h_to_4h_quantization_cache = None
+    dense_4h_to_h_quantization_cache = None
+
+    try:
+        load_cpu_kernel(**kwargs)
+    except:
+        if kernels is None:  # CUDA kernels failed
+            print("Cannot load cpu or cuda kernel, quantization failed:")
+            assert kernels is not None
+        print("Cannot load cpu kernel, don't use quantized model on cpu.")
+
+    current_device = model.device
+
+    if model.device == torch.device("cpu"):
+        dtype=torch.float32
+    else:
+        dtype = torch.half
+
+    QuantizedLinearWithPara = partial(
+        QuantizedLinear,
+        weight_bit_width=weight_bit_width,
+        bias=True,
+        dtype=dtype,
+        empty_init=empty_init
+    )
+
+    if use_quantization_cache:
+        print("Using quantization cache")
+        layer = model.layers[0]
+        weight = layer.attention.query_key_value.weight
+        n, m = weight.size(0), weight.size(1)
+        query_key_value_quantization_cache = CacheTensor(n, m, dtype=dtype, device=current_device, requires_grad=False)
+        weight = layer.attention.dense.weight
+        n, m = weight.size(0), weight.size(1)
+        dense_quantization_cache = CacheTensor(n, m, dtype=dtype, device=current_device, requires_grad=False)
+        weight = layer.mlp.dense_h_to_4h.weight
+        n, m = weight.size(0), weight.size(1)
+        dense_h_to_4h_quantization_cache = CacheTensor(n, m, dtype=dtype, device=current_device, requires_grad=False)
+        weight = layer.mlp.dense_4h_to_h.weight
+        n, m = weight.size(0), weight.size(1)
+        dense_4h_to_h_quantization_cache = CacheTensor(n, m, dtype=dtype, device=current_device, requires_grad=False)
+
+    print("Applying quantization to glm layers")
+
+    for layer in model.layers:
+        layer.attention.query_key_value = QuantizedLinearWithPara(
+            weight_tensor=layer.attention.query_key_value.weight.to(current_device),
+            bias_tensor=layer.attention.query_key_value.bias,
+            in_features=layer.attention.query_key_value.in_features,
+            out_features=layer.attention.query_key_value.out_features,
+            device=layer.attention.query_key_value.weight.device,
+            quantization_cache=query_key_value_quantization_cache
+        )
+        layer.attention.dense = QuantizedLinearWithPara(
+            weight_tensor=layer.attention.dense.weight.to(current_device),
+            bias_tensor=layer.attention.dense.bias,
+            in_features=layer.attention.dense.in_features,
+            out_features=layer.attention.dense.out_features,
+            device=layer.attention.dense.weight.device,
+            quantization_cache=dense_quantization_cache
+        )
+        layer.mlp.dense_h_to_4h = QuantizedLinearWithPara(
+            weight_tensor=layer.mlp.dense_h_to_4h.weight.to(current_device),
+            bias_tensor=layer.mlp.dense_h_to_4h.bias,
+            in_features=layer.mlp.dense_h_to_4h.in_features,
+            out_features=layer.mlp.dense_h_to_4h.out_features,
+            device=layer.mlp.dense_h_to_4h.weight.device,
+            quantization_cache=dense_h_to_4h_quantization_cache
+        )
+        layer.mlp.dense_4h_to_h = QuantizedLinearWithPara(
+            weight_tensor=layer.mlp.dense_4h_to_h.weight.to(current_device),
+            bias_tensor=layer.mlp.dense_4h_to_h.bias,
+            in_features=layer.mlp.dense_4h_to_h.in_features,
+            out_features=layer.mlp.dense_4h_to_h.out_features,
+            device=layer.mlp.dense_4h_to_h.weight.device,
+            quantization_cache=dense_4h_to_h_quantization_cache
+        )
+    return model

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ced23e0e417f2dfcf6eed7e0bfc1d56c4c6ed8e2c602d433d06066af88ab2e7
+size 14575

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce21225b3a04f92b1556e137b9e4f00a59dda32eda3027243d8cfd9edab1acb7
+size 627

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "bos_token": "<sop>",
+  "eos_token": "<eop>",
+  "mask_token": "[MASK]",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/tokenization_chatglm.py

@@ -0,0 +1,430 @@
+"""Tokenization classes for ChatGLM."""
+from typing import List, Optional, Union
+import os
+
+from transformers.tokenization_utils import PreTrainedTokenizer
+from transformers.utils import logging, PaddingStrategy
+from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
+from typing import Dict
+import sentencepiece as spm
+import numpy as np
+
+logger = logging.get_logger(__name__)
+
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
+    "THUDM/chatglm-6b": 2048,
+}
+
+
+class TextTokenizer:
+    def __init__(self, model_path):
+        self.sp = spm.SentencePieceProcessor()
+        self.sp.Load(model_path)
+        self.num_tokens = self.sp.vocab_size()
+
+    def encode(self, text):
+        return self.sp.EncodeAsIds(text)
+
+    def decode(self, ids: List[int]):
+        return self.sp.DecodeIds(ids)
+
+    def tokenize(self, text):
+        return self.sp.EncodeAsPieces(text)
+
+    def convert_tokens_to_ids(self, tokens):
+        return [self.sp.PieceToId(token) for token in tokens]
+
+    def convert_token_to_id(self, token):
+        return self.sp.PieceToId(token)
+
+    def convert_id_to_token(self, idx):
+        return self.sp.IdToPiece(idx)
+
+    def __len__(self):
+        return self.num_tokens
+
+
+class SPTokenizer:
+    def __init__(
+            self,
+            vocab_file,
+            num_image_tokens=20000,
+            max_blank_length=80,
+            byte_fallback=True,
+    ):
+        assert vocab_file is not None
+        self.vocab_file = vocab_file
+        self.num_image_tokens = num_image_tokens
+        self.special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "<unused_0>", "<sop>", "<eop>", "<ENC>", "<dBLOCK>"]
+        self.max_blank_length = max_blank_length
+        self.byte_fallback = byte_fallback
+        self.text_tokenizer = TextTokenizer(vocab_file)
+
+    def _get_text_tokenizer(self):
+        return self.text_tokenizer
+
+    @staticmethod
+    def get_blank_token(length: int):
+        assert length >= 2
+        return f"<|blank_{length}|>"
+
+    @staticmethod
+    def get_tab_token():
+        return f"<|tab|>"
+
+    @property
+    def num_text_tokens(self):
+        return self.text_tokenizer.num_tokens
+
+    @property
+    def num_tokens(self):
+        return self.num_image_tokens + self.num_text_tokens
+
+    @staticmethod
+    def _encode_whitespaces(text: str, max_len: int = 80):
+        text = text.replace("\t", SPTokenizer.get_tab_token())
+        for i in range(max_len, 1, -1):
+            text = text.replace(" " * i, SPTokenizer.get_blank_token(i))
+        return text
+
+    def _preprocess(self, text: str, linebreak=True, whitespaces=True):
+        if linebreak:
+            text = text.replace("\n", "<n>")
+        if whitespaces:
+            text = self._encode_whitespaces(text, max_len=self.max_blank_length)
+        return text
+
+    def encode(
+            self, text: str, linebreak=True, whitespaces=True, add_dummy_prefix=True
+    ) -> List[int]:
+        """
+        @param text: Text to encode.
+        @param linebreak: Whether to encode newline (\n) in text.
+        @param whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding.
+        @param special_tokens: Whether to encode special token ([MASK], [gMASK], etc.) in text.
+        @param add_dummy_prefix: Whether to add dummy blank space in the beginning.
+        """
+        text = self._preprocess(text, linebreak, whitespaces)
+        if not add_dummy_prefix:
+            text = "<n>" + text
+        tmp = self._get_text_tokenizer().encode(text)
+        tokens = [x + self.num_image_tokens for x in tmp]
+        return tokens if add_dummy_prefix else tokens[2:]
+
+    def decode(self, text_ids: List[int]) -> str:
+        ids = [int(_id) - self.num_image_tokens for _id in text_ids]
+        ids = [_id for _id in ids if _id >= 0]
+        text = self._get_text_tokenizer().decode(ids)
+        text = text.replace("<n>", "\n")
+        text = text.replace(SPTokenizer.get_tab_token(), "\t")
+        for i in range(2, self.max_blank_length + 1):
+            text = text.replace(self.get_blank_token(i), " " * i)
+        return text
+
+    def tokenize(
+            self, text: str, linebreak=True, whitespaces=True, add_dummy_prefix=True
+    ) -> List[str]:
+        """
+        @param text: Text to encode.
+        @param linebreak: Whether to encode newline (\n) in text.
+        @param whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding.
+        @param special_tokens: Whether to encode special token ([MASK], [gMASK], etc.) in text.
+        @param add_dummy_prefix: Whether to add dummy blank space in the beginning.
+        """
+        text = self._preprocess(text, linebreak, whitespaces)
+        if not add_dummy_prefix:
+            text = "<n>" + text
+        tokens = self._get_text_tokenizer().tokenize(text)
+        return tokens if add_dummy_prefix else tokens[2:]
+
+    def __getitem__(self, x: Union[int, str]):
+        if isinstance(x, int):
+            if x < self.num_image_tokens:
+                return "<image_{}>".format(x)
+            else:
+                return self.text_tokenizer.convert_id_to_token(x - self.num_image_tokens)
+        elif isinstance(x, str):
+            if x.startswith("<image_") and x.endswith(">") and x[7:-1].isdigit():
+                return int(x[7:-1])
+            else:
+                return self.text_tokenizer.convert_token_to_id(x) + self.num_image_tokens
+        else:
+            raise ValueError("The key should be str or int.")
+
+
+class ChatGLMTokenizer(PreTrainedTokenizer):
+    """
+    Construct a ChatGLM tokenizer. Based on byte-level Byte-Pair-Encoding.
+
+    Args:
+        vocab_file (`str`):
+            Path to the vocabulary file.
+    """
+
+    vocab_files_names = {"vocab_file": "ice_text.model"}
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    model_input_names = ["input_ids", "attention_mask", "position_ids"]
+
+    def __init__(
+            self,
+            vocab_file,
+            do_lower_case=False,
+            remove_space=False,
+            bos_token='<sop>',
+            eos_token='<eop>',
+            end_token='</s>',
+            mask_token='[MASK]',
+            gmask_token='[gMASK]',
+            padding_side="left",
+            pad_token="<pad>",
+            unk_token="<unk>",
+            num_image_tokens=20000,
+            **kwargs
+    ) -> None:
+        super().__init__(
+            do_lower_case=do_lower_case,
+            remove_space=remove_space,
+            padding_side=padding_side,
+            bos_token=bos_token,
+            eos_token=eos_token,
+            end_token=end_token,
+            mask_token=mask_token,
+            gmask_token=gmask_token,
+            pad_token=pad_token,
+            unk_token=unk_token,
+            num_image_tokens=num_image_tokens,
+            **kwargs
+        )
+
+        self.do_lower_case = do_lower_case
+        self.remove_space = remove_space
+        self.vocab_file = vocab_file
+
+        self.bos_token = bos_token
+        self.eos_token = eos_token
+        self.end_token = end_token
+        self.mask_token = mask_token
+        self.gmask_token = gmask_token
+
+        self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens)
+
+        """ Initialisation """
+
+    @property
+    def gmask_token_id(self) -> Optional[int]:
+        if self.gmask_token is None:
+            return None
+        return self.convert_tokens_to_ids(self.gmask_token)
+
+    @property
+    def end_token_id(self) -> Optional[int]:
+        """
+        `Optional[int]`: Id of the end of context token in the vocabulary. Returns `None` if the token has not been
+        set.
+        """
+        if self.end_token is None:
+            return None
+        return self.convert_tokens_to_ids(self.end_token)
+
+    @property
+    def vocab_size(self):
+        """ Returns vocab size """
+        return self.sp_tokenizer.num_tokens
+
+    def get_vocab(self):
+        """ Returns vocab as a dict """
+        vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
+        vocab.update(self.added_tokens_encoder)
+        return vocab
+
+    def preprocess_text(self, inputs):
+        if self.remove_space:
+            outputs = " ".join(inputs.strip().split())
+        else:
+            outputs = inputs
+
+        if self.do_lower_case:
+            outputs = outputs.lower()
+
+        return outputs
+
+    def _tokenize(self, text, **kwargs):
+        """ Returns a tokenized string. """
+        text = self.preprocess_text(text)
+
+        seq = self.sp_tokenizer.tokenize(text)
+
+        return seq
+
+    def _decode(
+            self,
+            token_ids: Union[int, List[int]],
+            skip_special_tokens: bool = False,
+            clean_up_tokenization_spaces: bool = True,
+            **kwargs
+    ) -> str:
+        if isinstance(token_ids, int):
+            token_ids = [token_ids]
+        if len(token_ids) == 0:
+            return ""
+        if self.pad_token_id in token_ids:  # remove pad
+            token_ids = list(filter((self.pad_token_id).__ne__, token_ids))
+        return self.sp_tokenizer.decode(token_ids)
+
+    def _convert_token_to_id(self, token):
+        """ Converts a token (str) in an id using the vocab. """
+        return self.sp_tokenizer[token]
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        return self.sp_tokenizer[index]
+
+    def save_vocabulary(self, save_directory, filename_prefix=None):
+        """
+        Save the vocabulary and special tokens file to a directory.
+
+        Args:
+            save_directory (`str`):
+                The directory in which to save the vocabulary.
+            filename_prefix (`str`, *optional*):
+                An optional prefix to add to the named of the saved files.
+
+        Returns:
+            `Tuple(str)`: Paths to the files saved.
+        """
+        if os.path.isdir(save_directory):
+            vocab_file = os.path.join(
+                save_directory, self.vocab_files_names["vocab_file"]
+            )
+        else:
+            vocab_file = save_directory
+
+        with open(self.vocab_file, 'rb') as fin:
+            proto_str = fin.read()
+
+        with open(vocab_file, "wb") as writer:
+            writer.write(proto_str)
+
+        return (vocab_file,)
+
+    def build_inputs_with_special_tokens(
+            self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
+        adding special tokens. A BERT sequence has the following format:
+
+        - single sequence: `[CLS] X [SEP]`
+        - pair of sequences: `[CLS] A [SEP] B [SEP]`
+
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs to which the special tokens will be added.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
+        """
+        gmask_id = self.sp_tokenizer[self.gmask_token]
+        eos_id = self.sp_tokenizer[self.eos_token]
+        token_ids_0 = token_ids_0 + [gmask_id, self.sp_tokenizer[self.bos_token]]
+        if token_ids_1 is not None:
+            token_ids_0 = token_ids_0 + token_ids_1 + [eos_id]
+        return token_ids_0
+
+    def _pad(
+            self,
+            encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
+            max_length: Optional[int] = None,
+            padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
+            pad_to_multiple_of: Optional[int] = None,
+            return_attention_mask: Optional[bool] = None,
+    ) -> dict:
+        """
+        Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
+
+        Args:
+            encoded_inputs:
+                Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
+            max_length: maximum length of the returned list and optionally padding length (see below).
+                Will truncate by taking into account the special tokens.
+            padding_strategy: PaddingStrategy to use for padding.
+
+                - PaddingStrategy.LONGEST Pad to the longest sequence in the batch
+                - PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
+                - PaddingStrategy.DO_NOT_PAD: Do not pad
+                The tokenizer padding sides are defined in self.padding_side:
+
+                    - 'left': pads on the left of the sequences
+                    - 'right': pads on the right of the sequences
+            pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
+                This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
+                `>= 7.5` (Volta).
+            return_attention_mask:
+                (optional) Set to False to avoid returning attention mask (default: set to model specifics)
+        """
+        # Load from model defaults
+        bos_token_id = self.sp_tokenizer[self.bos_token]
+        mask_token_id = self.sp_tokenizer[self.mask_token]
+        gmask_token_id = self.sp_tokenizer[self.gmask_token]
+        assert self.padding_side == "left"
+
+        required_input = encoded_inputs[self.model_input_names[0]]
+        seq_length = len(required_input)
+
+        if padding_strategy == PaddingStrategy.LONGEST:
+            max_length = len(required_input)
+
+        if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
+            max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
+
+        needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length
+
+        # Initialize attention mask if not present.
+        if max_length is not None:
+            if "attention_mask" not in encoded_inputs:
+                if bos_token_id in required_input:
+                    context_length = required_input.index(bos_token_id)
+                else:
+                    context_length = seq_length
+                attention_mask = np.ones((1, seq_length, seq_length))
+                attention_mask = np.tril(attention_mask)
+                attention_mask[:, :, :context_length] = 1
+                attention_mask = np.bool_(attention_mask < 0.5)
+                encoded_inputs["attention_mask"] = attention_mask
+
+            if "position_ids" not in encoded_inputs:
+                if bos_token_id in required_input:
+                    context_length = required_input.index(bos_token_id)
+                else:
+                    context_length = seq_length
+                position_ids = np.arange(seq_length, dtype=np.int64)
+                mask_token = mask_token_id if mask_token_id in required_input else gmask_token_id
+                if mask_token in required_input:
+                    mask_position = required_input.index(mask_token)
+                    position_ids[context_length:] = mask_position
+                block_position_ids = np.concatenate(
+                    [np.zeros(context_length, dtype=np.int64),
+                     np.arange(1, seq_length - context_length + 1, dtype=np.int64)])
+                encoded_inputs["position_ids"] = np.stack([position_ids, block_position_ids], axis=0)
+
+        if needs_to_be_padded:
+            difference = max_length - len(required_input)
+
+            if "attention_mask" in encoded_inputs:
+                encoded_inputs["attention_mask"] = np.pad(encoded_inputs["attention_mask"],
+                                                          pad_width=[(0, 0), (difference, 0), (difference, 0)],
+                                                          mode='constant', constant_values=True)
+            if "token_type_ids" in encoded_inputs:
+                encoded_inputs["token_type_ids"] = [self.pad_token_type_id] * difference + encoded_inputs[
+                    "token_type_ids"
+                ]
+            if "special_tokens_mask" in encoded_inputs:
+                encoded_inputs["special_tokens_mask"] = [1] * difference + encoded_inputs["special_tokens_mask"]
+            if "position_ids" in encoded_inputs:
+                encoded_inputs["position_ids"] = np.pad(encoded_inputs["position_ids"],
+                                                        pad_width=[(0, 0), (difference, 0)])
+            encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input
+
+        return encoded_inputs

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenization_chatglm.ChatGLMTokenizer",
+      null
+    ]
+  },
+  "bos_token": "<sop>",
+  "do_lower_case": false,
+  "end_token": "</s>",
+  "eos_token": "<eop>",
+  "gmask_token": "[gMASK]",
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "num_image_tokens": 0,
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "remove_space": false,
+  "special_tokens_map_file": null,
+  "tokenizer_class": "ChatGLMTokenizer",
+  "unk_token": "<unk>"
+}

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/trainer_state.json

@@ -0,0 +1,616 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 114.28571428571429,
+  "global_step": 1000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.14,
+      "learning_rate": 0.0198,
+      "loss": 4.7949,
+      "step": 10
+    },
+    {
+      "epoch": 2.29,
+      "learning_rate": 0.0196,
+      "loss": 3.7519,
+      "step": 20
+    },
+    {
+      "epoch": 3.43,
+      "learning_rate": 0.0194,
+      "loss": 3.3049,
+      "step": 30
+    },
+    {
+      "epoch": 4.57,
+      "learning_rate": 0.0192,
+      "loss": 2.8868,
+      "step": 40
+    },
+    {
+      "epoch": 5.71,
+      "learning_rate": 0.019,
+      "loss": 2.4806,
+      "step": 50
+    },
+    {
+      "epoch": 6.86,
+      "learning_rate": 0.0188,
+      "loss": 1.865,
+      "step": 60
+    },
+    {
+      "epoch": 8.0,
+      "learning_rate": 0.018600000000000002,
+      "loss": 1.4186,
+      "step": 70
+    },
+    {
+      "epoch": 9.14,
+      "learning_rate": 0.0184,
+      "loss": 0.9316,
+      "step": 80
+    },
+    {
+      "epoch": 10.29,
+      "learning_rate": 0.0182,
+      "loss": 0.5643,
+      "step": 90
+    },
+    {
+      "epoch": 11.43,
+      "learning_rate": 0.018000000000000002,
+      "loss": 0.3509,
+      "step": 100
+    },
+    {
+      "epoch": 12.57,
+      "learning_rate": 0.0178,
+      "loss": 0.2172,
+      "step": 110
+    },
+    {
+      "epoch": 13.71,
+      "learning_rate": 0.0176,
+      "loss": 0.1486,
+      "step": 120
+    },
+    {
+      "epoch": 14.86,
+      "learning_rate": 0.0174,
+      "loss": 0.1196,
+      "step": 130
+    },
+    {
+      "epoch": 16.0,
+      "learning_rate": 0.0172,
+      "loss": 0.0802,
+      "step": 140
+    },
+    {
+      "epoch": 17.14,
+      "learning_rate": 0.017,
+      "loss": 0.0739,
+      "step": 150
+    },
+    {
+      "epoch": 18.29,
+      "learning_rate": 0.0168,
+      "loss": 0.0619,
+      "step": 160
+    },
+    {
+      "epoch": 19.43,
+      "learning_rate": 0.0166,
+      "loss": 0.062,
+      "step": 170
+    },
+    {
+      "epoch": 20.57,
+      "learning_rate": 0.016399999999999998,
+      "loss": 0.0595,
+      "step": 180
+    },
+    {
+      "epoch": 21.71,
+      "learning_rate": 0.016200000000000003,
+      "loss": 0.0569,
+      "step": 190
+    },
+    {
+      "epoch": 22.86,
+      "learning_rate": 0.016,
+      "loss": 0.0533,
+      "step": 200
+    },
+    {
+      "epoch": 24.0,
+      "learning_rate": 0.0158,
+      "loss": 0.052,
+      "step": 210
+    },
+    {
+      "epoch": 25.14,
+      "learning_rate": 0.015600000000000001,
+      "loss": 0.0471,
+      "step": 220
+    },
+    {
+      "epoch": 26.29,
+      "learning_rate": 0.0154,
+      "loss": 0.0478,
+      "step": 230
+    },
+    {
+      "epoch": 27.43,
+      "learning_rate": 0.0152,
+      "loss": 0.0415,
+      "step": 240
+    },
+    {
+      "epoch": 28.57,
+      "learning_rate": 0.015,
+      "loss": 0.0462,
+      "step": 250
+    },
+    {
+      "epoch": 29.71,
+      "learning_rate": 0.0148,
+      "loss": 0.0437,
+      "step": 260
+    },
+    {
+      "epoch": 30.86,
+      "learning_rate": 0.0146,
+      "loss": 0.0452,
+      "step": 270
+    },
+    {
+      "epoch": 32.0,
+      "learning_rate": 0.0144,
+      "loss": 0.041,
+      "step": 280
+    },
+    {
+      "epoch": 33.14,
+      "learning_rate": 0.014199999999999999,
+      "loss": 0.0418,
+      "step": 290
+    },
+    {
+      "epoch": 34.29,
+      "learning_rate": 0.013999999999999999,
+      "loss": 0.0469,
+      "step": 300
+    },
+    {
+      "epoch": 35.43,
+      "learning_rate": 0.0138,
+      "loss": 0.0394,
+      "step": 310
+    },
+    {
+      "epoch": 36.57,
+      "learning_rate": 0.013600000000000001,
+      "loss": 0.0444,
+      "step": 320
+    },
+    {
+      "epoch": 37.71,
+      "learning_rate": 0.0134,
+      "loss": 0.0393,
+      "step": 330
+    },
+    {
+      "epoch": 38.86,
+      "learning_rate": 0.013200000000000002,
+      "loss": 0.045,
+      "step": 340
+    },
+    {
+      "epoch": 40.0,
+      "learning_rate": 0.013000000000000001,
+      "loss": 0.0392,
+      "step": 350
+    },
+    {
+      "epoch": 41.14,
+      "learning_rate": 0.0128,
+      "loss": 0.0351,
+      "step": 360
+    },
+    {
+      "epoch": 42.29,
+      "learning_rate": 0.0126,
+      "loss": 0.0389,
+      "step": 370
+    },
+    {
+      "epoch": 43.43,
+      "learning_rate": 0.0124,
+      "loss": 0.0374,
+      "step": 380
+    },
+    {
+      "epoch": 44.57,
+      "learning_rate": 0.0122,
+      "loss": 0.035,
+      "step": 390
+    },
+    {
+      "epoch": 45.71,
+      "learning_rate": 0.012,
+      "loss": 0.0349,
+      "step": 400
+    },
+    {
+      "epoch": 46.86,
+      "learning_rate": 0.0118,
+      "loss": 0.0361,
+      "step": 410
+    },
+    {
+      "epoch": 48.0,
+      "learning_rate": 0.0116,
+      "loss": 0.0368,
+      "step": 420
+    },
+    {
+      "epoch": 49.14,
+      "learning_rate": 0.011399999999999999,
+      "loss": 0.0352,
+      "step": 430
+    },
+    {
+      "epoch": 50.29,
+      "learning_rate": 0.011200000000000002,
+      "loss": 0.0353,
+      "step": 440
+    },
+    {
+      "epoch": 51.43,
+      "learning_rate": 0.011000000000000001,
+      "loss": 0.0337,
+      "step": 450
+    },
+    {
+      "epoch": 52.57,
+      "learning_rate": 0.0108,
+      "loss": 0.032,
+      "step": 460
+    },
+    {
+      "epoch": 53.71,
+      "learning_rate": 0.0106,
+      "loss": 0.0366,
+      "step": 470
+    },
+    {
+      "epoch": 54.86,
+      "learning_rate": 0.010400000000000001,
+      "loss": 0.0317,
+      "step": 480
+    },
+    {
+      "epoch": 56.0,
+      "learning_rate": 0.0102,
+      "loss": 0.0332,
+      "step": 490
+    },
+    {
+      "epoch": 57.14,
+      "learning_rate": 0.01,
+      "loss": 0.0328,
+      "step": 500
+    },
+    {
+      "epoch": 58.29,
+      "learning_rate": 0.0098,
+      "loss": 0.0325,
+      "step": 510
+    },
+    {
+      "epoch": 59.43,
+      "learning_rate": 0.0096,
+      "loss": 0.0327,
+      "step": 520
+    },
+    {
+      "epoch": 60.57,
+      "learning_rate": 0.0094,
+      "loss": 0.0344,
+      "step": 530
+    },
+    {
+      "epoch": 61.71,
+      "learning_rate": 0.0092,
+      "loss": 0.0351,
+      "step": 540
+    },
+    {
+      "epoch": 62.86,
+      "learning_rate": 0.009000000000000001,
+      "loss": 0.0325,
+      "step": 550
+    },
+    {
+      "epoch": 64.0,
+      "learning_rate": 0.0088,
+      "loss": 0.0329,
+      "step": 560
+    },
+    {
+      "epoch": 65.14,
+      "learning_rate": 0.0086,
+      "loss": 0.031,
+      "step": 570
+    },
+    {
+      "epoch": 66.29,
+      "learning_rate": 0.0084,
+      "loss": 0.0326,
+      "step": 580
+    },
+    {
+      "epoch": 67.43,
+      "learning_rate": 0.008199999999999999,
+      "loss": 0.0311,
+      "step": 590
+    },
+    {
+      "epoch": 68.57,
+      "learning_rate": 0.008,
+      "loss": 0.033,
+      "step": 600
+    },
+    {
+      "epoch": 69.71,
+      "learning_rate": 0.0078000000000000005,
+      "loss": 0.0297,
+      "step": 610
+    },
+    {
+      "epoch": 70.86,
+      "learning_rate": 0.0076,
+      "loss": 0.0331,
+      "step": 620
+    },
+    {
+      "epoch": 72.0,
+      "learning_rate": 0.0074,
+      "loss": 0.0318,
+      "step": 630
+    },
+    {
+      "epoch": 73.14,
+      "learning_rate": 0.0072,
+      "loss": 0.0307,
+      "step": 640
+    },
+    {
+      "epoch": 74.29,
+      "learning_rate": 0.006999999999999999,
+      "loss": 0.03,
+      "step": 650
+    },
+    {
+      "epoch": 75.43,
+      "learning_rate": 0.0068000000000000005,
+      "loss": 0.0307,
+      "step": 660
+    },
+    {
+      "epoch": 76.57,
+      "learning_rate": 0.006600000000000001,
+      "loss": 0.0334,
+      "step": 670
+    },
+    {
+      "epoch": 77.71,
+      "learning_rate": 0.0064,
+      "loss": 0.0321,
+      "step": 680
+    },
+    {
+      "epoch": 78.86,
+      "learning_rate": 0.0062,
+      "loss": 0.029,
+      "step": 690
+    },
+    {
+      "epoch": 80.0,
+      "learning_rate": 0.006,
+      "loss": 0.0315,
+      "step": 700
+    },
+    {
+      "epoch": 81.14,
+      "learning_rate": 0.0058,
+      "loss": 0.0294,
+      "step": 710
+    },
+    {
+      "epoch": 82.29,
+      "learning_rate": 0.005600000000000001,
+      "loss": 0.0323,
+      "step": 720
+    },
+    {
+      "epoch": 83.43,
+      "learning_rate": 0.0054,
+      "loss": 0.0274,
+      "step": 730
+    },
+    {
+      "epoch": 84.57,
+      "learning_rate": 0.005200000000000001,
+      "loss": 0.0305,
+      "step": 740
+    },
+    {
+      "epoch": 85.71,
+      "learning_rate": 0.005,
+      "loss": 0.0316,
+      "step": 750
+    },
+    {
+      "epoch": 86.86,
+      "learning_rate": 0.0048,
+      "loss": 0.0262,
+      "step": 760
+    },
+    {
+      "epoch": 88.0,
+      "learning_rate": 0.0046,
+      "loss": 0.0305,
+      "step": 770
+    },
+    {
+      "epoch": 89.14,
+      "learning_rate": 0.0044,
+      "loss": 0.0294,
+      "step": 780
+    },
+    {
+      "epoch": 90.29,
+      "learning_rate": 0.0042,
+      "loss": 0.0291,
+      "step": 790
+    },
+    {
+      "epoch": 91.43,
+      "learning_rate": 0.004,
+      "loss": 0.0274,
+      "step": 800
+    },
+    {
+      "epoch": 92.57,
+      "learning_rate": 0.0038,
+      "loss": 0.032,
+      "step": 810
+    },
+    {
+      "epoch": 93.71,
+      "learning_rate": 0.0036,
+      "loss": 0.0262,
+      "step": 820
+    },
+    {
+      "epoch": 94.86,
+      "learning_rate": 0.0034000000000000002,
+      "loss": 0.0325,
+      "step": 830
+    },
+    {
+      "epoch": 96.0,
+      "learning_rate": 0.0032,
+      "loss": 0.0276,
+      "step": 840
+    },
+    {
+      "epoch": 97.14,
+      "learning_rate": 0.003,
+      "loss": 0.029,
+      "step": 850
+    },
+    {
+      "epoch": 98.29,
+      "learning_rate": 0.0028000000000000004,
+      "loss": 0.0256,
+      "step": 860
+    },
+    {
+      "epoch": 99.43,
+      "learning_rate": 0.0026000000000000003,
+      "loss": 0.0305,
+      "step": 870
+    },
+    {
+      "epoch": 100.57,
+      "learning_rate": 0.0024,
+      "loss": 0.0271,
+      "step": 880
+    },
+    {
+      "epoch": 101.71,
+      "learning_rate": 0.0022,
+      "loss": 0.0302,
+      "step": 890
+    },
+    {
+      "epoch": 102.86,
+      "learning_rate": 0.002,
+      "loss": 0.0288,
+      "step": 900
+    },
+    {
+      "epoch": 104.0,
+      "learning_rate": 0.0018,
+      "loss": 0.0261,
+      "step": 910
+    },
+    {
+      "epoch": 105.14,
+      "learning_rate": 0.0016,
+      "loss": 0.0272,
+      "step": 920
+    },
+    {
+      "epoch": 106.29,
+      "learning_rate": 0.0014000000000000002,
+      "loss": 0.0285,
+      "step": 930
+    },
+    {
+      "epoch": 107.43,
+      "learning_rate": 0.0012,
+      "loss": 0.0271,
+      "step": 940
+    },
+    {
+      "epoch": 108.57,
+      "learning_rate": 0.001,
+      "loss": 0.0289,
+      "step": 950
+    },
+    {
+      "epoch": 109.71,
+      "learning_rate": 0.0008,
+      "loss": 0.028,
+      "step": 960
+    },
+    {
+      "epoch": 110.86,
+      "learning_rate": 0.0006,
+      "loss": 0.0263,
+      "step": 970
+    },
+    {
+      "epoch": 112.0,
+      "learning_rate": 0.0004,
+      "loss": 0.0279,
+      "step": 980
+    },
+    {
+      "epoch": 113.14,
+      "learning_rate": 0.0002,
+      "loss": 0.0272,
+      "step": 990
+    },
+    {
+      "epoch": 114.29,
+      "learning_rate": 0.0,
+      "loss": 0.0285,
+      "step": 1000
+    }
+  ],
+  "max_steps": 1000,
+  "num_train_epochs": 125,
+  "total_flos": 3.4665721233408e+16,
+  "trial_name": null,
+  "trial_params": null
+}

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-1000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:800d88035249fde5c3f1ce80f96f4ccad529f4cf9101a6a495f31e43927109c4
+size 3771

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_03-38-54_LAPTOP-U8KCJD82/1682019627.5574055/events.out.tfevents.1682019627.LAPTOP-U8KCJD82.39620.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9dbba98e909e8f3a837bb9936dd583c354d8b83d6864e2e8b01328489a808369
+size 6115

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_03-38-54_LAPTOP-U8KCJD82/events.out.tfevents.1682019627.LAPTOP-U8KCJD82.39620.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b561736c3552194be6ec935d7a465bfb336fbd58bb8ab1d68a78ac97eb833cd7
+size 4497

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_03-57-00_LAPTOP-U8KCJD82/1682020719.8067539/events.out.tfevents.1682020719.LAPTOP-U8KCJD82.34144.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bc26469b12b39eefbfb19bf9be5eac4ad0c68d6e1ddd33e905d3059a06eee64
+size 6115

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_03-57-00_LAPTOP-U8KCJD82/events.out.tfevents.1682020719.LAPTOP-U8KCJD82.34144.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be50f347728891e15e260d563eb2c32822e3ad0a5ab4e4d5c88689d517c633db
+size 4999

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_04-03-18_LAPTOP-U8KCJD82/1682021099.0769536/events.out.tfevents.1682021099.LAPTOP-U8KCJD82.4528.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c09d6a7d9ba37d37dfae6a9fc383664b7b345ed5b19930da85a2da075a85c8e
+size 6115

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_04-03-18_LAPTOP-U8KCJD82/events.out.tfevents.1682021099.LAPTOP-U8KCJD82.4528.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44de312738e1f73fdf252da896b82969f54b288a16abdec15d9bccf5ead2ec89
+size 4497

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_04-07-46_LAPTOP-U8KCJD82/1682021363.5070107/events.out.tfevents.1682021363.LAPTOP-U8KCJD82.34384.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c47bc2a2092ec54c278c7f331de46b05c3da8951fc57b79b59353f46d92040cd
+size 6115

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_04-07-46_LAPTOP-U8KCJD82/events.out.tfevents.1682021363.LAPTOP-U8KCJD82.34384.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95acacb51277b736cd043af6c3df8b0949d37c1932f8dd39775db731d552b037
+size 20519

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_11-45-46_LAPTOP-U8KCJD82/1682048840.1281629/events.out.tfevents.1682048840.LAPTOP-U8KCJD82.30268.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8ac8fe57b89689c202e9c8543749e69915e6670e59659f288602c5fc361779b
+size 6115

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_11-45-46_LAPTOP-U8KCJD82/events.out.tfevents.1682048840.LAPTOP-U8KCJD82.30268.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4662640a42ea57ab2152db47ef3f50d721d5509c11c69b51651f99e19411e95
+size 4501

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_11-48-35_LAPTOP-U8KCJD82/events.out.tfevents.1682049010.LAPTOP-U8KCJD82.40476.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43ba45944f035296405454ac17ff21e2da2fdbc697e34273d7dc45c7eb310ff9
+size 88

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_11-53-35_LAPTOP-U8KCJD82/1682049309.5723379/events.out.tfevents.1682049309.LAPTOP-U8KCJD82.19992.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c0d37ef4a5778b1a30ca92ff219aa6c92fb662e0df7d88cbf52f0c9d1787d81b
+size 6115

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_11-53-35_LAPTOP-U8KCJD82/events.out.tfevents.1682049309.LAPTOP-U8KCJD82.19992.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fff7d6993d790a4606be9bebec5478bd2bba7ee0b58fe9c3bfc89edb004c28c8
+size 4501

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_11-59-08_LAPTOP-U8KCJD82/1682049642.960882/events.out.tfevents.1682049642.LAPTOP-U8KCJD82.21296.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2f300310a299cfb4131842c1f4a0b0741d07142c5ad82c82086a08ffb1d0716
+size 6115

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_11-59-08_LAPTOP-U8KCJD82/events.out.tfevents.1682049642.LAPTOP-U8KCJD82.21296.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:adb237b2fa30205061e1d9ba3f2423f8f065aba5b3f0cdaad6908760cc69ff0c
+size 4501

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_12-04-16_LAPTOP-U8KCJD82/1682049952.9553204/events.out.tfevents.1682049952.LAPTOP-U8KCJD82.3092.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:854624c3499913de971099060c16894b8ff3cc8fb8ba1f912148ac43a812330f
+size 6115

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_12-04-16_LAPTOP-U8KCJD82/events.out.tfevents.1682049952.LAPTOP-U8KCJD82.3092.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d9e876c62e1fa37df30e6a2d151671f2370546fd7a5f0803bd120affdd0404c
+size 4414

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_15-57-52_LAPTOP-U8KCJD82/1682063879.4572506/events.out.tfevents.1682063879.LAPTOP-U8KCJD82.41476.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a56b44c1ba6b2d18be7f70271867146e964eba79a175486dd6b55b702a82892b
+size 6115

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_15-57-52_LAPTOP-U8KCJD82/events.out.tfevents.1682063879.LAPTOP-U8KCJD82.41476.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17547f8ac5cc000b35a1116cf4b79eed7c52ef17447e1e0e6842c7a6bee5bb34
+size 6262

ptuning/output/adgen-chatglm-6b-pt-128-2e-2/runs/Apr21_16-51-52_LAPTOP-U8KCJD82/1682067121.1639612/events.out.tfevents.1682067121.LAPTOP-U8KCJD82.14196.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f9a4229035dff3bd94f3f4bd94f88c1db0bbd318e5d085a98addc2fe12e9084
+size 6115