update

README.md

@@ -4,7 +4,10 @@ license: mit
 📢 [[Project Page](https://microsoft.github.io/OmniParser/)] [[Blog Post](https://www.microsoft.com/en-us/research/articles/omniparser-for-pure-vision-based-gui-agent/)]
 
 # Model Summary
-OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent. It includes a finetuned version of YOLOv8 and a finetuned BLIP-2 model. For more details of the models used and finetuning, please refer to the [paper](https://arxiv.org/abs/2408.00203).
+OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent. 
+Training Datasets include: 1) an interactable icon detection dataset, which was curated from 67k popular web pages and automatically annotated to highlight clickable and actionable regions, and 2) an icon description dataset, designed to associate each UI element with its corresponding function. This dataset serves as a key component for training models to understand the semantics of detected elements.
+
+It includes a finetuned version of YOLOv8 and a finetuned BLIP-2 model on the above dataset respectively. For more details of the models used and finetuning, please refer to the [paper](https://arxiv.org/abs/2408.00203).
 
 # Responsible AI Considerations
 ## Intended Use

config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "Salesforce/blip2-opt-2.7b",
+  "architectures": [
+    "Blip2ForConditionalGeneration"
+  ],
+  "initializer_factor": 1.0,
+  "initializer_range": 0.02,
+  "model_type": "blip-2",
+  "num_query_tokens": 32,
+  "qformer_config": {
+    "classifier_dropout": null,
+    "model_type": "blip_2_qformer"
+  },
+  "text_config": {
+    "_name_or_path": "facebook/opt-2.7b",
+    "activation_dropout": 0.0,
+    "architectures": [
+      "OPTForCausalLM"
+    ],
+    "eos_token_id": 50118,
+    "ffn_dim": 10240,
+    "hidden_size": 2560,
+    "model_type": "opt",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 32,
+    "prefix": "</s>",
+    "torch_dtype": "float16",
+    "word_embed_proj_dim": 2560
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.40.2",
+  "use_decoder_only_language_model": true,
+  "vision_config": {
+    "dropout": 0.0,
+    "initializer_factor": 1.0,
+    "model_type": "blip_2_vision_model",
+    "num_channels": 3,
+    "projection_dim": 512
+  }
+}