
```console
$ source .etc/Script/activate
```

## BERT
[자체 데이터로 Bert Training 하기](https://cryptosalamander.tistory.com/141)
[RoBERTa/BERT/DistilBERT and masked language modeling](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling)
[pretrainging](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling)


아래 경로에 model 생성됨
`D:\tmp\test-mlm`

```bash
python run_mlm.py \
    --model_name_or_path roberta-base \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-mlm
```


## ref
* [pytorch](https://pytorch.kr/get-started/locally/)
* [[HuggingFace 튜토리얼] 1. Quick Tour - Pipeline & AutoClass](https://bo-10000.tistory.com/130)
* [BERT](https://huggingface.co/docs/transformers/v4.19.2/en/model_doc/bert#bert)
* [python virtualenv](https://bskyvision.com/entry/python-%EA%B0%80%EC%83%81%ED%99%98%EA%B2%BD-virtualenv-%EC%82%AC%EC%9A%A9-%EB%B0%A9%EB%B2%95-venv%EC%99%80-%EB%B9%84%EA%B5%90)
* [CUDA/cuDNN](https://afsdzvcx123.tistory.com/entry/%EC%9D%B8%EA%B3%B5%EC%A7%80%EB%8A%A5-Windows%EC%9C%88%EB%8F%84%EC%9A%B0-CUDA-cuDNN-%EC%84%A4%EC%B9%98%EB%B0%A9%EB%B2%95)
* [CUDA Tookit 12.1(latest](https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exe_local)
* [cuDDN](https://developer.nvidia.com/rdp/cudnn-download)

## repo
* [Seonu](https://huggingface.co/oysu/Seonu)
{"license": "apache-2.0"}
license:apache-2.0 region:us
TAGS
#license-apache-2.0 #region-us 




## BERT
자체 데이터로 Bert Training 하기
RoBERTa/BERT/DistilBERT and masked language modeling
pretrainging


아래 경로에 model 생성됨
'D:\tmp\test-mlm'




## ref
* pytorch
* [[HuggingFace 튜토리얼] 1. Quick Tour - Pipeline & AutoClass](URL
* BERT
* python virtualenv
* CUDA/cuDNN
* CUDA Tookit 12.1(latest
* cuDDN

## repo
* Seonu
## BERT
자체 데이터로 Bert Training 하기
RoBERTa/BERT/DistilBERT and masked language modeling
pretrainging


아래 경로에 model 생성됨
'D:\tmp\test-mlm' ## ref
* pytorch
* [[HuggingFace 튜토리얼] 1. Quick Tour - Pipeline & AutoClass](URL
* BERT
* python virtualenv
* CUDA/cuDNN
* CUDA Tookit 12.1(latest
* cuDDN ## repo
* Seonu
TAGS
#license-apache-2.0 #region-us 
 ## BERT
자체 데이터로 Bert Training 하기
RoBERTa/BERT/DistilBERT and masked language modeling
pretrainging


아래 경로에 model 생성됨
'D:\tmp\test-mlm' ## ref
* pytorch
* [[HuggingFace 튜토리얼] 1. Quick Tour - Pipeline & AutoClass](URL
* BERT
* python virtualenv
* CUDA/cuDNN
* CUDA Tookit 12.1(latest
* cuDDN ## repo
* Seonu