---
license: mit
language:
- en
base_model:
- openai/clip-vit-large-patch14
pipeline_tag: video-classification
tags:
- dance
- vision
- breaking
---
# CLIP-Based Break Dance Move Classifier

A deep learning model for classifying break dance moves using CLIP (Contrastive Language-Image Pre-Training) embeddings. The model is fine-tuned on break dance videos to classify different power moves including windmills, halos, swipes, and baby mills.

## Features

- Video-based classification using CLIP embeddings
- Multi-frame temporal analysis
- Configurable frame sampling and data augmentation
- Real-time inference using Cog
- Misclassification analysis tools
- Hyperparameter tuning support

## Setup

```bash
# Install dependencies
pip install -r requirements.txt

# Install Cog (if not already installed)
curl -o /usr/local/bin/cog -L https://github.com/replicate/cog/releases/latest/download/cog_`uname -s`_`uname -m`
chmod +x /usr/local/bin/cog
```

## Cog

download the weights

```bash
gdown https://drive.google.com/uc?id=1Gn3UdoKffKJwz84GnGx-WMFTwZuvDsuf -O ./checkpoints/
```

build the image

```bash
cog build --separate-weights
```

push a new image

```bash
cog push
```

## Training

download the training data

```bash
gdown https://drive.google.com/uc?id=11M6nSuSuvoU2wpcV_-6KFqCzEMGP75q6?usp=drive_link -O ./data/
```

```bash
# Run training with default configuration
python scripts/train.py

# Run hyperparameter tuning
python scripts/hyperparameter_tuning.py
```

## Inference

```bash
# Using Cog for inference
cog predict -i video=@path/to/your/video.mp4

# Using standard Python script
python scripts/inference.py --video path/to/your/video.mp4
```

## Analysis

```bash
# Generate misclassification report
python scripts/visualization/miscalculations_report.py

# Visualize model performance
python scripts/visualization/visualize.py
```

## Project Structure

```
clip/
├── src/                    # Source code
│   ├── data/              # Dataset and data processing
│   ├── models/            # Model architecture
│   └── utils/             # Utility functions
├── scripts/               # Training and inference scripts
│   └── visualization/     # Visualization tools
├── config/                # Configuration files
├── runs/                  # Training runs and checkpoints
├── cog.yaml              # Cog configuration
└── requirements.txt      # Python dependencies
```

## Training Data

To run training on your own, you can find the training data [here](https://drive.google.com/drive/folders/11M6nSuSuvoU2wpcV_-6KFqCzEMGP75q6?usp=drive_link) and put it in the a directory at the root of the project called `./data`.

## Checkpoints

To run predictions with cog or locally on an existing checkpoint, you can find a checkpoint and configuration files [here](https://drive.google.com/drive/folders/1Gn3UdoKffKJwz84GnGx-WMFTwZuvDsuf?usp=sharing) and put them in the a directory at the root of the project called `./checkpoints`.

## Model Architecture

- Base: CLIP ViT-Large/14
- Custom temporal pooling layer
- Fine-tuned vision encoder (last 3 layers)
- Output: 4-class classifier

## License

MIT License

Copyright (c) 2024 Bryant Wolf

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Citation

If you use this model in your research, please cite:

```bibtex
@misc{clip-breakdance-classifier,
  author = {Bryant Wolf},
  title = {CLIP-Based Break Dance Move Classifier},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{https://github.com/bawolf/breaking_vision_clip_cog}}
}
```