Upload README.md

README.md

@@ -1,109 +1,79 @@
 ---
-license: mit
+license: apache-2.0
 datasets:
 - dmitva/human_ai_generated_text
-language:
-- en
-widget:
-  - text: "This model trains on a diverse dataset and serves functions in applications requiring a mechanism for distinguishing between human and AI-generated text."
-tags:
-- nlp
-- code
-inference: false
 ---
 
-# 0xnu/AGTD-v0.1
+## 0xnu/AGTD-v0.2
 
-The **0xnu/AGTD-v0.1** model represents a significant breakthrough in distinguishing between text written by humans and one generated by Artificial Intelligence (AI). It is rooted in sophisticated algorithms and offers exceptional accuracy and efficiency in text analysis and classification. I detailed the findings in a study, and it is accessible [here](https://finns.cc/oxdlai).
+The "0xnu/AGTD-v0.2" model represents a significant breakthrough in distinguishing between text written by humans and one generated by Artificial Intelligence (AI). It is rooted in sophisticated algorithms and offers exceptional accuracy and efficiency in text analysis and classification. Everything is detailed in the study and accessible [here](https://arxiv.org/abs/2311.15565).
 
-## Instruction Format
+### Training Details
 
-```
-<BOS> [CLS] [INST] Instruction [/INST] Model answer [SEP] [INST] Follow-up instruction [/INST] [SEP] [EOS]
+```sh
+Precision: 0.6269
+Recall: 1.0000
+F1-score: 0.7707
+Accuracy: 0.7028
+Confusion Matrix:
+[[197 288]
+ [  0 484]]
 ```
 
-Pseudo-code for tokenizing instructions with the new format:
+### Run the model
 
 ```Python
-def tokenize(text):
-    return tok.encode(text, add_special_tokens=False)
-
-[BOS_ID] + 
-tokenize("[CLS]") + tokenize("[INST]") + tokenize(USER_MESSAGE_1) + tokenize("[/INST]") +
-tokenize(BOT_MESSAGE_1) + tokenize("[SEP]") +
-…
-tokenize("[INST]") + tokenize(USER_MESSAGE_N) + tokenize("[/INST]") +
-tokenize(BOT_MESSAGE_N) + tokenize("[SEP]") + [EOS_ID]
-```
-
-Notes:
+import os
+os.environ["KERAS_BACKEND"] = "tensorflow"
 
-- `[CLS]`, `[SEP]`, `[PAD]`, `[UNK]`, and `[MASK]` tokens are integrated based on their definitions in the tokenizer configuration.
-- `[INST]` and `[/INST]` are utilized to encapsulate instructions.
-- The tokenize method should not automatically add BOS or EOS tokens but should add a prefix space.
-- The `do_lower_case` parameter indicates that text should be in lowercase for consistent tokenization.
-- `clean_up_tokenization_spaces` remove unnecessary spaces in the tokenization process.
-- The `tokenize_chinese_chars` parameter indicates special handling for Chinese characters.
-- The maximum model length is set to 512 tokens.
-
-## Installing Libraries
-
-```sh
-pip install torch transformers
-```
+import keras
+import tensorflow as tf
+import pickle
+import numpy as np
+from huggingface_hub import hf_hub_download
 
-## Run the model
+# Hugging Face repository details
+REPO_ID = "0xnu/AGTD-v0.1"
+MODEL_FILENAME = "human_ai_text_classification_model.keras"
+TOKENIZER_FILENAME = "tokenizer.pkl"
 
-```Python
-import torch
-from transformers import AutoTokenizer, AutoModelForSequenceClassification
+# Download the model and tokenizer
+model_path = hf_hub_download(repo_id=REPO_ID, filename=MODEL_FILENAME)
+tokenizer_path = hf_hub_download(repo_id=REPO_ID, filename=TOKENIZER_FILENAME)
 
-model_id = "0xnu/AGTD-v0.1"
+# Load the model
+model = keras.models.load_model(model_path)
 
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForSequenceClassification.from_pretrained(model_id)
+# Load the tokenizer
+with open(tokenizer_path, 'rb') as tokenizer_file:
+    tokenizer = pickle.load(tokenizer_file)
 
 # Input text
 text = "This model trains on a diverse dataset and serves functions in applications requiring a mechanism for distinguishing between human and AI-generated text."
 
-# Preprocess the text
-inputs = tokenizer(text, return_tensors='pt')
-
-# Run the model
-outputs = model(**inputs)
-
-# Interpret the output
-logits = outputs.logits
+# Parameters (these should match the training parameters)
+MAX_LENGTH = 100000
 
-# Apply softmax to convert logits to probabilities
-probabilities = torch.softmax(logits, dim=1)
+# Tokenization function
+def tokenize_text(text, tokenizer, max_length):
+    sequences = tokenizer.texts_to_sequences([text])
+    padded_sequence = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=max_length, padding='post', truncating='post')
+    return padded_sequence
 
-# Assuming the first class is 'human' and the second class is 'ai'
-human_prob, ai_prob = probabilities.detach().numpy()[0]
+# Prediction function
+def predict_text(text, model, tokenizer, max_length):
+    processed_text = tokenize_text(text, tokenizer, max_length)
+    prediction = model.predict(processed_text)[0][0]
+    return prediction
 
-# Print probabilities
-print(f"Human Probability: {human_prob:.4f}")
-print(f"AI Probability: {ai_prob:.4f}")
+# Make prediction
+prediction = predict_text(text, model, tokenizer, MAX_LENGTH)
 
-# Determine if the text is human or AI-generated
-if human_prob > ai_prob:
-    print("The text is likely human-generated.")
+# Interpret results
+if prediction >= 0.5:
+    print(f"The text is likely AI-generated (confidence: {prediction:.2f})")
 else:
-    print("The text is likely AI-generated.")
-```
-
-## Citation
-
-Please cite the paper if you are using the resource for your work.
+    print(f"The text is likely human-written (confidence: {1-prediction:.2f})")
 
-```bibtex
-@misc{abiodunfinbarrsoketunji-agtd2023,
-  doi = {10.48550/arXiv.2311.15565},
-  url = {https://arxiv.org/abs/2311.15565},
-  author = {Abiodun Finbarrs Oketunji},
-  title = {Evaluating the Efficacy of Hybrid Deep Learning Models in Distinguishing AI-Generated Text},
-  publisher = {arXiv},
-  year = {2023},
-  copyright = {arXiv.org perpetual, non-exclusive license}
-}
+print(f"Raw prediction value: {prediction}")
 ```