simplify the usage example in the README, point to other rad-dino version. update link to license
Browse files
README.md
CHANGED
@@ -1,6 +1,8 @@
|
|
1 |
---
|
2 |
-
library_name: transformers
|
3 |
license: other
|
|
|
|
|
|
|
4 |
---
|
5 |
|
6 |
# Model card for RAD-DINO
|
@@ -17,7 +19,7 @@ RAD-DINO-MAIRA-2 is the version of RAD-DINO used in [MAIRA-2: Grounded Radiology
|
|
17 |
|
18 |
- **Developed by:** Microsoft Health Futures
|
19 |
- **Model type:** Vision transformer
|
20 |
-
- **License:** MSRLA
|
21 |
- **Finetuned from model:** [`dinov2-base`](https://huggingface.co/facebook/dinov2-base)
|
22 |
|
23 |
## Uses
|
@@ -55,71 +57,13 @@ Underlying biases of the training datasets may not be well characterized.
|
|
55 |
|
56 |
## Getting started
|
57 |
|
58 |
-
Let us first write an auxiliary function to download a chest X-ray.
|
59 |
-
|
60 |
-
```python
|
61 |
-
>>> import requests
|
62 |
-
>>> from PIL import Image
|
63 |
-
>>> def download_sample_image() -> Image.Image:
|
64 |
-
... """Download chest X-ray with CC license."""
|
65 |
-
... base_url = "https://upload.wikimedia.org/wikipedia/commons"
|
66 |
-
... image_url = f"{base_url}/2/20/Chest_X-ray_in_influenza_and_Haemophilus_influenzae.jpg"
|
67 |
-
... headers = {"User-Agent": "RAD-DINO"}
|
68 |
-
... response = requests.get(image_url, headers=headers, stream=True)
|
69 |
-
... return Image.open(response.raw)
|
70 |
-
...
|
71 |
```
|
72 |
-
|
73 |
-
|
74 |
-
|
75 |
-
```python
|
76 |
-
>>> import torch
|
77 |
-
>>> from transformers import AutoModel
|
78 |
-
>>> from transformers import AutoImageProcessor
|
79 |
-
>>>
|
80 |
-
>>> # Download the model
|
81 |
-
>>> repo = "microsoft/rad-dino-maira-2"
|
82 |
-
>>> model = AutoModel.from_pretrained(repo)
|
83 |
-
>>>
|
84 |
-
>>> # The processor takes a PIL image, performs resizing, center-cropping, and
|
85 |
-
>>> # intensity normalization using stats from MIMIC-CXR, and returns a
|
86 |
-
>>> # dictionary with a PyTorch tensor ready for the encoder
|
87 |
-
>>> processor = AutoImageProcessor.from_pretrained(repo)
|
88 |
-
>>>
|
89 |
-
>>> # Download and preprocess a chest X-ray
|
90 |
-
>>> image = download_sample_image()
|
91 |
-
>>> image.size # (width, height)
|
92 |
-
(2765, 2505)
|
93 |
-
>>> inputs = processor(images=image, return_tensors="pt")
|
94 |
-
>>>
|
95 |
-
>>> # Encode the image!
|
96 |
-
>>> with torch.inference_mode():
|
97 |
-
>>> outputs = model(**inputs)
|
98 |
-
>>>
|
99 |
-
>>> # Look at the CLS embeddings
|
100 |
-
>>> cls_embeddings = outputs.pooler_output
|
101 |
-
>>> cls_embeddings.shape # (batch_size, num_channels)
|
102 |
-
torch.Size([1, 768])
|
103 |
```
|
104 |
|
105 |
-
|
106 |
-
We will use [`einops`](https://einops.rocks/) (install with `pip install einops`) for this.
|
107 |
-
|
108 |
-
```python
|
109 |
-
>>> def reshape_patch_embeddings(flat_tokens: torch.Tensor) -> torch.Tensor:
|
110 |
-
... """Reshape flat list of patch tokens into a nice grid."""
|
111 |
-
... from einops import rearrange
|
112 |
-
... image_size = processor.crop_size["height"]
|
113 |
-
... patch_size = model.config.patch_size
|
114 |
-
... embeddings_size = image_size // patch_size
|
115 |
-
... patches_grid = rearrange(flat_tokens, "b (h w) c -> b c h w", h=embeddings_size)
|
116 |
-
... return patches_grid
|
117 |
-
...
|
118 |
-
>>> flat_patch_embeddings = outputs.last_hidden_state[:, 1:] # first token is CLS
|
119 |
-
>>> reshaped_patch_embeddings = reshape_patch_embeddings(flat_patch_embeddings)
|
120 |
-
>>> reshaped_patch_embeddings.shape # (batch_size, num_channels, height, width)
|
121 |
-
torch.Size([1, 768, 37, 37])
|
122 |
-
```
|
123 |
|
124 |
## Training details
|
125 |
|
|
|
1 |
---
|
|
|
2 |
license: other
|
3 |
+
license_name: msrla
|
4 |
+
license_link: https://huggingface.co/microsoft/rad-dino-maira-2/blob/main/LICENSE
|
5 |
+
library_name: transformers
|
6 |
---
|
7 |
|
8 |
# Model card for RAD-DINO
|
|
|
19 |
|
20 |
- **Developed by:** Microsoft Health Futures
|
21 |
- **Model type:** Vision transformer
|
22 |
+
- **License:** [MSRLA](./LICENSE)
|
23 |
- **Finetuned from model:** [`dinov2-base`](https://huggingface.co/facebook/dinov2-base)
|
24 |
|
25 |
## Uses
|
|
|
57 |
|
58 |
## Getting started
|
59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
60 |
```
|
61 |
+
from transformers import pipeline
|
62 |
+
pipe = pipeline(task="image-feature-extraction", model="microsoft/rad-dino-maira-2", pool=False)
|
63 |
+
patch_features = pipe("https://www.bhf.org.uk/-/media/images/information-support/tests/chest-x-ray/normal-chest-x-ray-620x400.jpg")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
64 |
```
|
65 |
|
66 |
+
Refer to [RAD-DINO](https://huggingface.co/microsoft/rad-dino) for a more detailed example.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
67 |
|
68 |
## Training details
|
69 |
|