<jupyter_start><jupyter_text>start exporting model to one that can be loaded with tensorflow.js<jupyter_code>callable = tf.function(model.call)
max_sequence_length = 512
concrete_function = callable.get_concrete_function([tf.TensorSpec([None, max_sequence_length], tf.int32, name="input_ids"), tf.TensorSpec([None, max_sequence_length], tf.int32, name="attention_mask")])
print(concrete_function)
tf.saved_model.save(model, 'distilbert_nela', signatures=concrete_function)<jupyter_output>WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
WARNING:tensorflow:Skipping full serialization of Keras layer <tensorflow.python.keras.layers.core.Dropout object at 0x000001C9528EFEB0>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <tensorflow.python.keras.layers.core.Dropout object at 0x000001C952B354F0>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <tensorflow.python.keras.layers.core.Dropout object at 0x000001C958D63A00>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <tensorflow.python.keras.layers.core.Dropout object at 0x000001C958D6DF10>, be[...]<jupyter_text>run tensorflowjs_converter --input_format=tf_saved_model --saved_model_tags=serve distilbert_nela distilbert_nela_js<jupyter_code># from tensorflowjs.converters import convert_tf_saved_model
# tests for output sizes
# a = tf.constant(1, shape=[1,3])
# b = tf.constant(1, shape=[1,8])

# def pad_up_to(t, max_in_dims, constant_values):
#     s = tf.shape(t)
#     paddings = [[0, m-s[i]] for (i,m) in enumerate(max_in_dims)]
#     return tf.pad(t, paddings, 'CONSTANT', constant_values=constant_values)

# tf.stack([pad_up_to(a, b.shape, 0), b])
# tf.concat(a, b, 0)
# convert tokenizer vocab
import json
vocab = []
with open('model/vocab.txt', 'r', encoding='utf-8') as f:
    for line in f:
        line = line.rstrip()
        if line.startswith('##'):  # idk why but the tokenizer library we're using does it the opposite of huggingface
            vocab.append(line.strip('##'))
        elif line.startswith('['):
            vocab.append(line)
        else:
            vocab.append(u'\u2581' + line)

# jsonString = json.dumps(vocab)
# print(jsonString[:50])
with open('../src/detection/tokenizer/vocab.json', 'w+', encoding='utf-8') as f:
    json.dump(vocab, f, ensure_ascii=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1210564930300807, 'fraction_numerical': 0.028246515040352166, 'mean_word_length': 5.571084337349397, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '31437', 'license': 'Unlicense', 'n_tokens_mistral': 973, 'n_tokens_neox': 925, 'n_words': 223, 'path': 'ml/convert_checkpoints.ipynb', 'repo_name': 'sirmammingtonham/FakeBlock', 'size': '2726'}