Update README.md (#2)
Browse files- Update README.md (c188d745fbb431654ec4007bed424fbc78351a0b)
README.md
CHANGED
@@ -10,24 +10,98 @@ widget:
|
|
10 |
example_title: Teapot
|
11 |
- src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/palace.jpg
|
12 |
example_title: Palace
|
13 |
-
---
|
14 |
-
|
15 |
-
# DPT (large-sized model)
|
16 |
-
|
17 |
-
Dense Prediction Transformer (DPT) model trained on 1.4 million images for monocular depth estimation. It was introduced in the paper [Vision Transformers for Dense Prediction](https://arxiv.org/abs/2103.13413) by Ranftl et al. and first released in [this repository](https://github.com/isl-org/DPT).
|
18 |
|
19 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
20 |
|
21 |
-
## Model
|
22 |
|
|
|
|
|
23 |
DPT uses the Vision Transformer (ViT) as backbone and adds a neck + head on top for monocular depth estimation.
|
24 |
-
|
25 |
![model image](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dpt_architecture.jpg)
|
26 |
|
27 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
28 |
|
29 |
-
You can use the raw model for zero-shot monocular depth estimation. See the [model hub](https://huggingface.co/models?search=dpt) to look for
|
30 |
-
fine-tuned versions on a task that interests you.
|
31 |
|
32 |
### How to use
|
33 |
|
|
|
10 |
example_title: Teapot
|
11 |
- src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/palace.jpg
|
12 |
example_title: Palace
|
|
|
|
|
|
|
|
|
|
|
13 |
|
14 |
+
model-index:
|
15 |
+
- name: dpt-large
|
16 |
+
results:
|
17 |
+
- task:
|
18 |
+
type: monocular-depth-estimation
|
19 |
+
name: Monocular Depth Estimation
|
20 |
+
dataset:
|
21 |
+
type: MIX 6
|
22 |
+
name: MIX 6
|
23 |
+
metrics:
|
24 |
+
- type: Zero-shot transfer
|
25 |
+
value: 10.82
|
26 |
+
name: Zero-shot transfer
|
27 |
+
config: Zero-shot transfer
|
28 |
+
verified: false
|
29 |
+
---
|
30 |
|
31 |
+
## Model Details: DPT (large-sized model)
|
32 |
|
33 |
+
Dense Prediction Transformer (DPT) model trained on 1.4 million images for monocular depth estimation.
|
34 |
+
It was introduced in the paper [Vision Transformers for Dense Prediction](https://arxiv.org/abs/2103.13413) by Ranftl et al. (2021) and first released in [this repository](https://github.com/isl-org/DPT).
|
35 |
DPT uses the Vision Transformer (ViT) as backbone and adds a neck + head on top for monocular depth estimation.
|
|
|
36 |
![model image](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dpt_architecture.jpg)
|
37 |
|
38 |
+
The model card has been written in combination by the Hugging Face team and Intel.
|
39 |
+
|
40 |
+
| Model Detail | Description |
|
41 |
+
| ----------- | ----------- |
|
42 |
+
| Model Authors - Company | Intel |
|
43 |
+
| Date | March 22, 2022 |
|
44 |
+
| Version | 1 |
|
45 |
+
| Type | Computer Vision - Monocular Depth Estimation |
|
46 |
+
| Paper or Other Resources | [Vision Transformers for Dense Prediction](https://arxiv.org/abs/2103.13413) and [GitHub Repo](https://github.com/isl-org/DPT) |
|
47 |
+
| License | [MIT](https://github.com/isl-org/DPT/blob/main/LICENSE) |
|
48 |
+
| Questions or Comments | [Community Tab](https://huggingface.co/Intel/dpt-large/discussions) and [Intel Developers Discord](https://discord.gg/rv2Gp55UJQ)|
|
49 |
+
|
50 |
+
| Intended Use | Description |
|
51 |
+
| ----------- | ----------- |
|
52 |
+
| Primary intended uses | You can use the raw model for zero-shot monocular depth estimation. See the [model hub](https://huggingface.co/models?search=dpt) to look for
|
53 |
+
fine-tuned versions on a task that interests you. |
|
54 |
+
| Primary intended users | Anyone doing monocular depth estimation |
|
55 |
+
| Out-of-scope uses | This model in most cases will need to be fine-tuned for your particular task. |
|
56 |
+
|
57 |
+
| Factors | Description |
|
58 |
+
| ----------- | ----------- |
|
59 |
+
| Groups | Multiple datasets compiled together |
|
60 |
+
| Instrumentation | - |
|
61 |
+
| Environment | Inference completed on Intel Xeon Platinum 8280 CPU @ 2.70GHz with 8 physical cores and an NVIDIA RTX 2080 GPU. |
|
62 |
+
| Card Prompts | Model deployment on alternate hardware and software will change model performance |
|
63 |
+
|
64 |
+
| Metrics | Description |
|
65 |
+
| ----------- | ----------- |
|
66 |
+
| Model performance measures | Zero-shot Transfer |
|
67 |
+
| Decision thresholds | - |
|
68 |
+
| Approaches to uncertainty and variability | - |
|
69 |
+
|
70 |
+
| Training and Evaluation Data | Description |
|
71 |
+
| ----------- | ----------- |
|
72 |
+
| Datasets | The dataset is called MIX 6, and contains around 1.4M images. The model was initialized with ImageNet-pretrained weights.|
|
73 |
+
| Motivation | To build a robust monocular depth prediction network |
|
74 |
+
| Preprocessing | "We resize the image such that the longer side is 384 pixels and train on random square crops of size 384. ... We perform random horizontal flips for data augmentation." See [Ranftl et al. (2021)](https://arxiv.org/abs/2103.13413) for more details. |
|
75 |
+
|
76 |
+
## Quantitative Analyses
|
77 |
+
| Model | Training set | DIW WHDR | ETH3D AbsRel | Sintel AbsRel | KITTI δ>1.25 | NYU δ>1.25 | TUM δ>1.25 |
|
78 |
+
| --- | --- | --- | --- | --- | --- | --- | --- |
|
79 |
+
| DPT - Large | MIX 6 | 10.82 (-13.2%) | 0.089 (-31.2%) | 0.270 (-17.5%) | 8.46 (-64.6%) | 8.32 (-12.9%) | 9.97 (-30.3%) |
|
80 |
+
| DPT - Hybrid | MIX 6 | 11.06 (-11.2%) | 0.093 (-27.6%) | 0.274 (-16.2%) | 11.56 (-51.6%) | 8.69 (-9.0%) | 10.89 (-23.2%) |
|
81 |
+
| MiDaS | MIX 6 | 12.95 (+3.9%) | 0.116 (-10.5%) | 0.329 (+0.5%) | 16.08 (-32.7%) | 8.71 (-8.8%) | 12.51 (-12.5%)
|
82 |
+
| MiDaS [30] | MIX 5 | 12.46 | 0.129 | 0.327 | 23.90 | 9.55 | 14.29 |
|
83 |
+
| Li [22] | MD [22] | 23.15 | 0.181 | 0.385 | 36.29 | 27.52 | 29.54 |
|
84 |
+
| Li [21] | MC [21] | 26.52 | 0.183 | 0.405 | 47.94 | 18.57 | 17.71 |
|
85 |
+
| Wang [40] | WS [40] | 19.09 | 0.205 | 0.390 | 31.92 | 29.57 | 20.18 |
|
86 |
+
| Xian [45] | RW [45] | 14.59 | 0.186 | 0.422 | 34.08 | 27.00 | 25.02 |
|
87 |
+
| Casser [5] | CS [8] | 32.80 | 0.235 | 0.422 | 21.15 | 39.58 | 37.18 |
|
88 |
+
|
89 |
+
Table 1. Comparison to the state of the art on monocular depth estimation. We evaluate zero-shot cross-dataset transfer according to the
|
90 |
+
protocol defined in [30]. Relative performance is computed with respect to the original MiDaS model [30]. Lower is better for all metrics. ([Ranftl et al., 2021](https://arxiv.org/abs/2103.13413))
|
91 |
+
|
92 |
+
|
93 |
+
| Ethical Considerations | Description |
|
94 |
+
| ----------- | ----------- |
|
95 |
+
| Data | The training data come from multiple image datasets compiled together. |
|
96 |
+
| Human life | The model is not intended to inform decisions central to human life or flourishing. It is an aggregated set of monocular depth image datasets. |
|
97 |
+
| Mitigations | No additional risk mitigation strategies were considered during model development. |
|
98 |
+
| Risks and harms | The extent of the risks involved by using the model remain unknown.|
|
99 |
+
| Use cases | - |
|
100 |
+
|
101 |
+
| Caveats and Recommendations |
|
102 |
+
| ----------- |
|
103 |
+
| There are no additional caveats or recommendations for this model. |
|
104 |
|
|
|
|
|
105 |
|
106 |
### How to use
|
107 |
|