---
license: apache-2.0
language:
- en
tags:
- Pytorch
- mmsegmentation
- segmentation
- Crop Classification
- Multi Temporal
- Geospatial
- Foundation model
datasets:
- ibm-nasa-geospatial/multi-temporal-crop-classification
metrics:
- accuracy
- IoU
---
### Model and Inputs
The pretrained [Prithvi-100m](https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M/blob/main/README.md) parameter model is finetuned to classify crop and other land cover types based off HLS data and CDL labels from the [multi_temporal_crop_classification dataset](https://huggingface.co/datasets/ibm-nasa-geospatial/multi-temporal-crop-classification). 

This dataset includes input chips of 224x224x18, where 224 is the height and width and 18 is combined with 6 bands of 3 time-steps. The bands are:
 
1. Blue
2. Green
3. Red
4. Narrow NIR
5. SWIR 1
6. SWIR 2

Labels are from CDL(Crop Data Layer) and classified into 13 classes.

![](multi_temporal_crop_classification.png)

The Prithvi-100m model was initially pretrained using a sequence length of 3 timesteps. For this task, we leverage the capacity for multi-temporal data input, which has been integrated from the foundational pretrained model. This adaptation allows us to achieve more generalized finetuning outcomes.

### Code
Code for Finetuning is available through [github](https://github.com/NASA-IMPACT/hls-foundation-os/)

Configuration used for finetuning is available through [config](https://github.com/NASA-IMPACT/hls-foundation-os/blob/main/fine-tuning-examples/configs/multi_temporal_crop_classification.py).

### Results
The experiment by running the mmseg stack for 80 epochs using the above config led to the following result:

|     **Classes**    | **IoU**| **Acc**|
|:------------------:|:------:|:------:|
| Natural Vegetation | 0.3362 | 39.06% |
|       Forest       | 0.4362 | 65.88% |
|        Corn        | 0.4574 | 54.53% |
|      Soybeans      | 0.4682 | 62.25% |
|      Wetlands      | 0.3246 | 45.62% |
|  Developed/Barren  | 0.3077 |  49.1% |
|     Open Water     | 0.6181 | 90.04% |
|    Winter Wheat    | 0.4497 | 66.75% |
|       Alfalfa      | 0.2518 | 63.97% |
|Fallow/Idle Cropland| 0.328  | 54.46% |
|       Cotton       | 0.2679 | 66.37% |
|       Sorghum      | 0.2741 | 75.91% |
|        Other       | 0.2803 | 39.76% |

|**aAcc**|**mIoU**|**mAcc**|
|:------:|:------:|:------:|
| 54.32% | 0.3692 | 59.51% |

It is important to acknowledge that the CDL (Crop Data Layer) labels employed in this process are known to contain noise and are not entirely precise, thereby influencing the model's performance. Fine-tuning the model with more accurate labels is expected to further enhance its overall effectiveness, leading to improved results.


### Inference
The github repo includes an inference script an inference script that allows to run the hls-cdl crop classification model for inference on HLS images. These input have to be geotiff format, including 18 bands for 3 time-step, and each time-step includes the channels described above (Blue, Green, Red, Narrow NIR, SWIR, SWIR 2) in order.