Image Feature Extraction
Transformers
Safetensors
dinov2
Inference Endpoints
sthyland commited on
Commit
6dd5e74
1 Parent(s): ecc3c49

simplify the usage example in the README, point to other rad-dino version. update link to license

Browse files
Files changed (1) hide show
  1. README.md +8 -64
README.md CHANGED
@@ -1,6 +1,8 @@
1
  ---
2
- library_name: transformers
3
  license: other
 
 
 
4
  ---
5
 
6
  # Model card for RAD-DINO
@@ -17,7 +19,7 @@ RAD-DINO-MAIRA-2 is the version of RAD-DINO used in [MAIRA-2: Grounded Radiology
17
 
18
  - **Developed by:** Microsoft Health Futures
19
  - **Model type:** Vision transformer
20
- - **License:** MSRLA
21
  - **Finetuned from model:** [`dinov2-base`](https://huggingface.co/facebook/dinov2-base)
22
 
23
  ## Uses
@@ -55,71 +57,13 @@ Underlying biases of the training datasets may not be well characterized.
55
 
56
  ## Getting started
57
 
58
- Let us first write an auxiliary function to download a chest X-ray.
59
-
60
- ```python
61
- >>> import requests
62
- >>> from PIL import Image
63
- >>> def download_sample_image() -> Image.Image:
64
- ... """Download chest X-ray with CC license."""
65
- ... base_url = "https://upload.wikimedia.org/wikipedia/commons"
66
- ... image_url = f"{base_url}/2/20/Chest_X-ray_in_influenza_and_Haemophilus_influenzae.jpg"
67
- ... headers = {"User-Agent": "RAD-DINO"}
68
- ... response = requests.get(image_url, headers=headers, stream=True)
69
- ... return Image.open(response.raw)
70
- ...
71
  ```
72
-
73
- Now let us download the model and encode an image.
74
-
75
- ```python
76
- >>> import torch
77
- >>> from transformers import AutoModel
78
- >>> from transformers import AutoImageProcessor
79
- >>>
80
- >>> # Download the model
81
- >>> repo = "microsoft/rad-dino-maira-2"
82
- >>> model = AutoModel.from_pretrained(repo)
83
- >>>
84
- >>> # The processor takes a PIL image, performs resizing, center-cropping, and
85
- >>> # intensity normalization using stats from MIMIC-CXR, and returns a
86
- >>> # dictionary with a PyTorch tensor ready for the encoder
87
- >>> processor = AutoImageProcessor.from_pretrained(repo)
88
- >>>
89
- >>> # Download and preprocess a chest X-ray
90
- >>> image = download_sample_image()
91
- >>> image.size # (width, height)
92
- (2765, 2505)
93
- >>> inputs = processor(images=image, return_tensors="pt")
94
- >>>
95
- >>> # Encode the image!
96
- >>> with torch.inference_mode():
97
- >>> outputs = model(**inputs)
98
- >>>
99
- >>> # Look at the CLS embeddings
100
- >>> cls_embeddings = outputs.pooler_output
101
- >>> cls_embeddings.shape # (batch_size, num_channels)
102
- torch.Size([1, 768])
103
  ```
104
 
105
- If we are interested in the feature maps, we can reshape the patch embeddings into a grid.
106
- We will use [`einops`](https://einops.rocks/) (install with `pip install einops`) for this.
107
-
108
- ```python
109
- >>> def reshape_patch_embeddings(flat_tokens: torch.Tensor) -> torch.Tensor:
110
- ... """Reshape flat list of patch tokens into a nice grid."""
111
- ... from einops import rearrange
112
- ... image_size = processor.crop_size["height"]
113
- ... patch_size = model.config.patch_size
114
- ... embeddings_size = image_size // patch_size
115
- ... patches_grid = rearrange(flat_tokens, "b (h w) c -> b c h w", h=embeddings_size)
116
- ... return patches_grid
117
- ...
118
- >>> flat_patch_embeddings = outputs.last_hidden_state[:, 1:] # first token is CLS
119
- >>> reshaped_patch_embeddings = reshape_patch_embeddings(flat_patch_embeddings)
120
- >>> reshaped_patch_embeddings.shape # (batch_size, num_channels, height, width)
121
- torch.Size([1, 768, 37, 37])
122
- ```
123
 
124
  ## Training details
125
 
 
1
  ---
 
2
  license: other
3
+ license_name: msrla
4
+ license_link: https://huggingface.co/microsoft/rad-dino-maira-2/blob/main/LICENSE
5
+ library_name: transformers
6
  ---
7
 
8
  # Model card for RAD-DINO
 
19
 
20
  - **Developed by:** Microsoft Health Futures
21
  - **Model type:** Vision transformer
22
+ - **License:** [MSRLA](./LICENSE)
23
  - **Finetuned from model:** [`dinov2-base`](https://huggingface.co/facebook/dinov2-base)
24
 
25
  ## Uses
 
57
 
58
  ## Getting started
59
 
 
 
 
 
 
 
 
 
 
 
 
 
 
60
  ```
61
+ from transformers import pipeline
62
+ pipe = pipeline(task="image-feature-extraction", model="microsoft/rad-dino-maira-2", pool=False)
63
+ patch_features = pipe("https://www.bhf.org.uk/-/media/images/information-support/tests/chest-x-ray/normal-chest-x-ray-620x400.jpg")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
64
  ```
65
 
66
+ Refer to [RAD-DINO](https://huggingface.co/microsoft/rad-dino) for a more detailed example.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
67
 
68
  ## Training details
69