<jupyter_start><jupyter_text>BERT Demo  References https://www.tensorflow.org/hubhttps://tfhub.dev/google/collections/bert/1https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4https://www.youtube.com/c/codebasics  Loading Libraries<jupyter_code>import tensorflow_hub as hub
import tensorflow_text as text<jupyter_output><empty_output><jupyter_text>Loading Urlsencoder and preprocessing url<jupyter_code>encoder_url = "https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4"
preprocessing_url = "https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3"<jupyter_output><empty_output><jupyter_text>Loading BERT Preprocessor<jupyter_code>preprocessor = hub.KerasLayer(preprocessing_url)<jupyter_output><empty_output><jupyter_text>testing preprocessor<jupyter_code>test_text = [
    "this is first document",
    "I love pasta"
]

test_dict = preprocessor(test_text)
test_dict.keys()<jupyter_output><empty_output><jupyter_text>Preprocessor returns a dictionary with 3 values. - input_word_ids : has the token ids of the input sequences.- input_mask : has value 1 at the position of all input tokens present before padding and value 0 for the padding tokens.- input_type_ids : has the index of the input segment that gave rise to the input token at the respective position. The first input segment (index 0) includes the start-of-sequence token and its end-of-segment token. The second segment (index 1, if present) includes its end-of-segment token. Padding tokens get index 0 again.every sentence/row has prefix `CLS` and suffix `SEP`.and total size of vector for 1 row will be `128`.<jupyter_code>test_dict['input_word_ids'].shape<jupyter_output><empty_output><jupyter_text>`CLS this is first document SEP`<jupyter_code>test_dict['input_word_ids'][0]
test_dict['input_mask'].shape

test_dict['input_mask'][0]

test_dict['input_type_ids'].shape
test_dict['input_type_ids'][0]<jupyter_output><empty_output><jupyter_text>Loading Encoder<jupyter_code>encoder_model = hub.KerasLayer(encoder_url)<jupyter_output><empty_output><jupyter_text>testing encoder<jupyter_code>encoded_dict = encoder_model(test_dict)
encoded_dict.keys()
encoded_dict['pooled_output']<jupyter_output><empty_output><jupyter_text>Pooled output gives embedding for the sentences/ rows. we have 2 sentences in input text and two pooled outputs representing the embedding.<jupyter_code>encoded_dict['sequence_output'].shape
enc_output = encoded_dict['encoder_outputs']
len(enc_output)<jupyter_output><empty_output><jupyter_text>As we are using BERT small and it has 12 encoder layers. and the outputs mentioned in this is output of each individual encoder layer.<jupyter_code>enc_output[0].shape<jupyter_output><empty_output><jupyter_text>as we have 2 sentences, so first shape is 2, then 128 is padding and 768 sized embedding.<jupyter_code>enc_output[-1] == encoded_dict['sequence_output']<jupyter_output><empty_output><jupyter_text>so the last encoder output is actually sequence output.. Obviously.<jupyter_code>encoded_dict['default'] == encoded_dict['pooled_output']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0963659961064244, 'fraction_numerical': 0.01427644386761843, 'mean_word_length': 8.148367952522255, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '333639', 'license': 'MIT', 'n_tokens_mistral': 1036, 'n_tokens_neox': 995, 'n_words': 228, 'path': 'Concepts/bert_demo.ipynb', 'repo_name': 'NishantBaheti/NatualLanguageProcessing', 'size': '3082'}