<jupyter_start><jupyter_text>Training and Scaling your Model Computer Vision model with Determined AI This notebook will walk through the benefits of building a Deep Learning model with Determined.  We will build an object detection model trained on the [Penn-Fudan Database for Pedestrian Detection and Segmentation](https://www.cis.upenn.edu/~jshi/ped_html/).  Prep We'll be creating a `checkpoints` directory later. This `.detignore` will ensure we don't upload that directory along with our model code when we start an experiment!<jupyter_code>!echo "checkpoints" >> .detignore<jupyter_output><empty_output><jupyter_text>-----  Imagine...  You're a **model developer** and you've been tasked with developing a **computer vision model**.  **MNIST**!<jupyter_code># Illustrative MNIST run...

#!git clone https://github.com/pytorch/examples
#%cd examples/mnistip install -qr requirements.txt
#!python main.py --epochs=50 --batch-size=64<jupyter_output><empty_output><jupyter_text>Clarification: An actually *useful* computer vision model We'll be training a Pedestrian Detection model with PyTorch!  **Object Detection with PyTorch**  Google an off-the-shelf Object Detection model<jupyter_code># Illustrative model training...

#!git clone <model_repo>
#%cd repo
#!python main.py<jupyter_output><empty_output><jupyter_text>You've got your model training, but now you're thinking: - [x] Find a model- [x] Training- [ ] Checkpointing?- [ ] Fault tolerance??- [ ] Visualization??? ------------  **Day 2** of Model Development with Determined AI If Day 1 was about finding your model and making sure it ran, Day 2 of model development is about making sure your model is manageable and scalable.   Here’s where Determined’s platform starts to come in handy:  1. Structure your code to adhere to the Determined APIs.  Example Determined model repository: *this one!*  2. Install Determined Locally  Example command to install Determined locally<jupyter_code># !pip install determined-cli
# !det deploy local cluster-up <--no-gpu if you do not have a GPU><jupyter_output><empty_output><jupyter_text>*Full instructions to install Determined: https://docs.determined.ai/latest/index.html  3. Launch your model and train it on your local cluster:<jupyter_code>!det experiment create standard.yaml .<jupyter_output><empty_output><jupyter_text>**Note**: *Experiments will take slightly longer the first time they are executed on any given Determined installation since Determined will pull the appropriate Docker containers to run the experiment. After the first run, the containers are cached and reused.*  So what's happening on the backend AKA Infrastructure you don't need to worry about   Local Inference  See how the trained model performs on a local image **Note**: *Check the Determined Web UI to make sure the experiment is `Complete` before trying to run predictions*<jupyter_code>from support.helper import predict
experiment_id = <experiment id>
predict(experiment_id, "test.jpg")<jupyter_output><empty_output><jupyter_text>**Note**: *Model training is a stochastic process so expect variable results, especially with the shortened training time used in this experiment. Feel free to update the configuration `.yaml` files to train for more epochs to ensure convergence.* ----  **Multi-GPU training**  Example of configuration change to enable multi-GPU training   Launch a multi-GPU training job You can install Determined on a local machine with multiple GPUs the same way as before, with:<jupyter_code># !pip install determined-cli
# !det deploy local cluster-up <--no-gpu if you do not have a GPU><jupyter_output><empty_output><jupyter_text>If you'd like to install in the cloud to get more GPUs, skip to the next section: **Cloud cluster** for installation instructions, then come back to run this experiment.<jupyter_code>DET_MASTER="your-remote-cluster-ip" # e.g. 172.16.17.32
!det -m $DET_MASTER experiment create multi.yaml .<jupyter_output><empty_output><jupyter_text>What's happening now?   Better Model performance<jupyter_code># Downloading checkpoints requires credentials to where the
# checkpoints are stored. If your remote cluster is using GCS,
# for example, you will need to launch a terminal in this Jupyter 
# instance and run the following:
#
# gcloud auth application-default login

# Then, run the following to set your default project in GCP
import os
os.environ["GCLOUD_PROJECT"] = "your-gcp-project-id"
experiment_id = <experiment id>
predict(experiment_id, "test.jpg", DET_MASTER)<jupyter_output><empty_output><jupyter_text>**Note**: *Model training is a stochastic process so expect variable results, especially with the shortened training time used in this experiment. Feel free to update the configuration `.yaml` files to train for more epochs to ensure convergence.* **Note**: *Experiments will take slightly longer the first time they are executed on any given Determined installation since Determined will pull the appropriate Docker containers to run the experiment. After the first run, the containers are cached and reused.*  **Cloud cluster**  Deploying a Determined Cluster to the cloud  Example command to start a Determined Cluster in the cloud<jupyter_code># For GCP
# !det-deploy gcp up --cluster-id <my-cluster> --project-id <your-gcp-project-id>

# For AWS
# !det-deploy aws up --cluster-id <my-cluster> --keypair <your-keypair-name><jupyter_output><empty_output><jupyter_text>* Full GCP installation instructions: https://docs.determined.ai/latest/how-to/installation/gcp.html* Full AWS installation instructions: https://docs.determined.ai/latest/how-to/installation/aws.html -----  Example configuration change to enable **Hyperparameter search**   Launch a Distributed Hyperparameter search job<jupyter_code># DET_MASTER="your-remote-cluster-ip" # if different than the one you set above
!det -m $DET_MASTER experiment create hyper.yaml .<jupyter_output><empty_output><jupyter_text>What's happening in the cloud?<jupyter_code>experiment_id = <experiment id>
predict(experiment_id, "test.jpg", DET_MASTER)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08787779237844941, 'fraction_numerical': 0.0031208935611038106, 'mean_word_length': 5.498399146211312, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 61, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '448370', 'license': 'Apache-2.0', 'n_tokens_mistral': 1763, 'n_tokens_neox': 1663, 'n_words': 696, 'path': 'examples/object_detection/experiments/GTC-ObjDet-PyTorch.ipynb', 'repo_name': 'rafaelrpinto/data-science-stack', 'size': '6088'}