Create README.md

README.md

@@ -0,0 +1,92 @@
+# Gemma 2 model card
+
+-  DatafoundryAI/OpenVino_Ratnamuladve.Q8_K_M-GGUF
+
+# Authors
+
+- DatafoundryAI
+# Model Information
+
+- Summary description and brief definition of inputs and outputs.
+
+## Description
+
+- Gemma-2-2b-it builds on the technological advancements of the Gemini models, offering high-quality language generation capabilities. We have enhanced this model by applying INT8 quantization using the Intel OpenVINO Toolkit. This process optimizes the model for deployment in resource-constrained environments.
+
+# About Openvino
+
+## Model Conversion and Quantization with Intel OpenVINO::
+
+- ### Model Optimizer:
+ 
+  - OpenVINO includes a tool called the Model Optimizer that converts pre-trained models from popular frameworks (such as TensorFlow, PyTorch, and ONNX) into an intermediate representation (IR). This IR consists of two files: a .xml file describing the model's structure and a .bin file containing the weights.
+- ### Quantization:
+ 
+  -  During the conversion process, you can apply quantization techniques to reduce model size and improve inference speed. OpenVINO supports INT8 quantization, which reduces floating-point precision to 8-bit integers.
+
+ # Benefits:
+   - INT8 quantization improves computational efficiency and reduces memory footprint, making the model more suitable for deployment on devices with limited hardware resources. The OpenVINO toolkit facilitates this process by providing tools and optimizations that ensure the model's performance remains high while being more resource-efficient.
+
+### Resources and Technical Documentation
+- [Responsible Generative AI Toolkit for Qunatisation](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html)
+
+
+#### Installing the Transformers Library and Inference
+
+To use this model, first install the Transformers library:
+```bash
+!pip install transformers openvino openvino-dev
+`````
+# Inference code 
+
+Below is an example of how to perform inference with the quantized Gemma-2-2b-it model:
+
+```bash
+import openvino_genai 
+def streamer(subword): 
+    print(subword, end='', flush=True)  
+    return False 
+`````
+```bash
+model_dir = "your path "  
+`````
+```bash
+device = 'CPU'  # GPU can be used as well 
+pipe = openvino_genai.LLMPipeline(model_dir, device)
+`````
+```bash
+import time
+config = openvino_genai.GenerationConfig()
+config.max_new_tokens = 100
+pipe.start_chat()
+
+total_tokens = 0
+total_time = 0
+
+while True:
+    prompt = input('question:\n')
+    if 'Stop!' == prompt:
+        break
+    
+    start_time = time.time()
+    output = pipe.generate(prompt, config, streamer)
+    end_time = time.time()
+
+    elapsed_time = end_time - start_time
+    num_tokens = len(output.split())  # Adjust this based on how tokens are represented
+
+    total_tokens += num_tokens
+    total_time += elapsed_time
+
+    print(f'Generated tokens: {num_tokens}')
+    print(f'Time taken: {elapsed_time:.2f} seconds')
+    print('\n----------')
+
+pipe.finish_chat()
+
+if total_time > 0:
+    tok_per_s = total_tokens / total_time
+    print(f'Tokens per second: {tok_per_s:.2f}')
+else:
+    print('No tokens generated.')
+'''''