<jupyter_start><jupyter_text>Install requirements 0 (optional). install apex for mixed presicion support<jupyter_code>!cd ../input/apex-master/apex-master/apex-master/ && pip install --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" .<jupyter_output>/opt/conda/lib/python3.6/site-packages/pip/_internal/commands/install.py:207: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Processing /kaggle/input/apex-master/apex-master/apex-master
Installing collected packages: apex
  Running setup.py install for apex ... [?25l- \ | / - \ | / - \ | / - \ | / - \ done
[?25hSuccessfully installed apex-0.1<jupyter_text>1. pip install pytorch-pretrained-bert without internet<jupyter_code>os.system('pip install --no-index --find-links="../input/pytorchpretrainedbert/" pytorch_pretrained_bert')<jupyter_output><empty_output><jupyter_text>Import Bert<jupyter_code>from pytorch_pretrained_bert import BertTokenizer
from pytorch_pretrained_bert.modeling import BertModel
BERT_FP = '../input/torch-bert-weights/bert-base-uncased/bert-base-uncased/'<jupyter_output><empty_output><jupyter_text>2. create BERT model and put on GPU<jupyter_code>bert = BertModel.from_pretrained(BERT_FP).cuda()
bert.eval()<jupyter_output><empty_output><jupyter_text>Setup tokenizer<jupyter_code>tokenizer = BertTokenizer(vocab_file='../input/torch-bert-weights/bert-base-uncased-vocab.txt')<jupyter_output><empty_output><jupyter_text>Make prediction<jupyter_code># lets tokenize some text (I intentionally mispelled 'plastic' to check berts subword information handling)
text = 'hi my name is Dieter and I like wearing my yellow pglastic hat while coding.'
tokens = tokenizer.tokenize(text)
tokens
# added start and end token and convert to ids
tokens = ["[CLS]"] + tokens + ["[SEP]"]
input_ids = tokenizer.convert_tokens_to_ids(tokens)
input_ids
# put input on gpu and make prediction
bert_output = bert(torch.tensor([input_ids]).cuda())
bert_output<jupyter_output><empty_output><jupyter_text>(Optional) Convert model to fp16<jupyter_code>import apex
bert.half()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14068780705672176, 'fraction_numerical': 0.007146047342563644, 'mean_word_length': 6.111111111111111, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '377192', 'license': 'MIT', 'n_tokens_mistral': 829, 'n_tokens_neox': 811, 'n_words': 144, 'path': '4 jigsaw/loading-bert-using-pytorch-with-tokenizer-apex.ipynb', 'repo_name': 'MLVPRASAD/KaggleProjects', 'size': '2239'}