language: zh
tags:
- VAE
- CVAE
- NER
- Data Augmentation
inference: false
Randeng-DELLA-226M-Chinese
- Github: Fengshenbang-LM
- Docs: Fengshenbang-Docs
简介 Brief Introduction
在悟道数据集上进行通用预训练的Deep VAE模型后又在命名实体识别任务的数据上微调的CVAE模型。其中编码器和解码器都是GPT-2架构。可以在给定命名实体和其类别后生成包含该命名实体的句子。
A deep VAE model pretrained on Wudao dataset and finetinued on NER dataset. Both encoder and decoder are based on GPT-2 architecture. This model is capable of generating a sentence given the named entity and its entity type.
模型分类 Model Taxonomy
| 需求 Demand | 任务 Task | 系列 Series | 模型 Model | 参数 Parameter | 额外 Extra |
|---|---|---|---|---|---|
| 通用 General | 受控文本生成 CTG | 燃灯 Randeng | DELLA | 226M | 条件变分自编码器-中文 CVAE-Chinese |
模型信息 Model Information
参考论文 Reference Paper：Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation
本模型使用了Della论文里的循环潜在向量架构，但对于解码器生成并未采用原论文的low-rank-tensor-product来进行信息融合，而是使用了简单的线性变换后逐位逐词添加的方式。该方式对于开放域数据集的预训练稳定性有较大正向作用。
Note that although we adopted the layer-wise recurrent latent variables structure as the paper, we did not use the low-rank-tensor-product to fuse the latent vectors to the decoder hidden states. Instead we applied a simple linear transformation on the latent vectors and then add them to the hidden states independently.
使用 Usage
# Checkout the latest Fengshenbang-LM directory and run following script under Fengshenbang-LM root directory
import torch
from torch.nn.utils.rnn import pad_sequence
from fengshen.models.deepVAE.deep_vae import Della
from transformers.models.bert.tokenization_bert import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("IDEA-CCNL/Randeng-DELLA-226M-CVAE-NER-Chinese")
vae_model = Della.from_pretrained("IDEA-CCNL/Randeng-DELLA-226M-CVAE-NER-Chinese")
引用 Citation
如果您在您的工作中使用了我们的模型，可以引用我们的论文：
If you are using the resource for your work, please cite the our paper:
@article{fengshenbang,
author = {Junjie Wang and Yuxiang Zhang and Lin Zhang and Ping Yang and Xinyu Gao and Ziwei Wu and Xiaoqun Dong and Junqing He and Jianheng Zhuo and Qi Yang and Yongfeng Huang and Xiayu Li and Yanghan Wu and Junyu Lu and Xinyu Zhu and Weifeng Chen and Ting Han and Kunhao Pan and Rui Wang and Hao Wang and Xiaojun Wu and Zhongshen Zeng and Chongpei Chen and Ruyi Gan and Jiaxing Zhang},
title = {Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence},
journal = {CoRR},
volume = {abs/2209.02970},
year = {2022}
}
也可以引用我们的网站:
You can also cite our website:
@misc{Fengshenbang-LM,
title={Fengshenbang-LM},
author={IDEA-CCNL},
year={2021},
howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}},
}