<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation.Licensed under the MIT License. Train your own Model and Deploy to Device**NOTE*** Warning: copying *.pb, *.bin, or, *.blob using the web interface can corrupt the files. If needed download and use Azure storage explorer or the CL.* You can run all the cells (after you manually do the first one), but do note that the deployment ones will require some input specific to your device.This notebook shows how to create a Tensorflow object detection model, how to convert the model to the appropriate format for the Perception DevelopmentKit, and how to deploy the model to your kit.This notebook takes a transfer learning approach, using a pre-trained Tensorflow Mobilenet model with custom datasetand SSDLite layers that we will train to detect bowls.We use the [Tensorflow object detection API](https://github.com/tensorflow/models/tree/master/research/object_detection).The trained model will be deployed to the Azure Percept Devkit using the Module Twin Update method.  Setting up the GPU for Tensorflow 1.15We will be using Tensorflow 1.15 for this example, and Tensorflow 2.x examples should be coming soon.The datascience notebook VMs do not come with the appropriate libraries installed for this version of Tensorflow, and so we will need toinstall them. If your compute node/cluster is a GPU machine, you will want to make sure the appropriate version of CUDA is installedinto your VM so that Tensorflow can use it to accelerate training substantially.If you are using a GPU-enabled compute device, you will want to follow these steps. Otherwise, feel free to skip this cell.1. Start your compute instance if it is not already started.1. Select Open Terminal, which should be a button next to your compute and kernel selection boxes.1. Run `conda info -e` to list the available conda environments. There should be an azureml_py36 environment.1. Select azureml_py36 by running `conda activate azureml_py36`1. Uninstall Pytorch and Tensorflow (which depend on the wrong version of CUDA) and the wrong version of CUDA with `conda uninstall cudatoolkit pytorch`1. Install the correct version of CUDA with `conda install cudatoolkit=10.0`You may now close the terminal tab. We will install Tensorflow in a cell of this notebook later.  Collecting the dataIn this notebook, we use a custom dataset that will be used to train a model that detects bowls.To compile this dataset, we have a couple of options. First, we could use a labeling tool like [VoTT](https://github.com/microsoft/VoTT) or[LabelImg](https://github.com/tzutalin/labelImg). Just make sure to export the dataset to Pascal VOC format. There should be one XML file foreach image, and the XML files should be under annotations/xmls.The other option for compiling a dataset is to use an already existing dataset like COCO and then filter out all the images and annotationsthat we don't care about. This is the approach we take here.**NOTE:**The first cell in this notebook contains the code necessary to get the dataset and convert it to the format we need. This will involve downloadingthe entire COCO dataset (train and validation splits for 2017), which is tens of GBs in size. This may not fit in your Azure ML compute's storage.Our recommendation is to run this cell locally (i.e., copy and paste the commands into your own shell and run them), then uploadthe resulting dataset to Azure via [Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/).<jupyter_code># Install tqdm for Python:
# pip install --user tqdm

# First, get the Advanced Code Repository
#!wget https://github.com/microsoft/azure-percept-advanced-development/archive/main.zip -O azure-percept-advanced-development-main.zip
#!unzip azure-percept-advanced-development-main.zip
#!rm azure-percept-advanced-development-main.zip
#!cd azure-percept-advanced-development-main/machine-learning-notebooks/transfer-learning/scripts

# Get the data
#!mkdir -p coco/images

# Get the 2017 training split
#!wget http://images.cocodataset.org/zips/train2017.zip
#!unzip train2017.zip
#!rm train2017.zip
#!mv train2017 coco/images/

# Get the 2017 validation split
#!wget http://images.cocodataset.org/zips/val2017.zip
#!unzip val2017.zip
#!rm val2017.zip
#!mv val2017 coco/images/

# Get all the annotations
#!wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
#!unzip annotations_trainval2017.zip
#!rm annotations_trainval2017.zip
#!mv annotations coco/annotations

# Filter the annotations down to a JSON that only contains images that contain at least one bowl
#!python filter_coco.py -i coco/annotations/instances_train2017.json -o coco_train2017_bowls.json -c bowl
#!python filter_coco.py -i coco/annotations/instances_val2017.json -o coco_val2017_bowls.json -c bowl

# Now create a filtered version of the coco train and val split based on their annotation JSONs
#!python create_coco_subset.py -i coco_train2017_bowls.json coco_val2017_bowls.json -g coco/images/train2017 coco/images/val2017 -o coco_filtered

# Create Pascal VOC formatted dataset from the COCO subset
#!python convert_coco_to_voc.py coco_filtered --target bowls_voc

# You should now have a data folder called bowls_voc, which should look like this:
# bowls_voc/
#   - annotations/
#       - xmls/
#   - images/

# Now upload 'bowls_voc' to your workspace using the Azure Storage Explorer.
# To do so, please follow these instructions:
# 1. Install Azure Storage Explorer and set it up according to the instructions found here: https://azure.microsoft.com/en-us/features/storage-explorer/
# 2. Once your account is linked, you should be able to open Azure Storage Explorer, select the appropriate subscription, storage account, and file share.
# 3. Upload 'bowls_voc' to /Users/<your user>/percept-transfer-learning under the appropriate file share.
# 4. Make sure that this notebook is in the same folder (/Users/<your user>/percept-transfer-learning)
# Save current directory for later reference
modelroot = !pwd
modelroot = modelroot[0]
modelroot
# Setup workspace for Azure ML
!pip install azureml.core

import azureml.core
from azureml.core import Workspace
print(azureml.core.VERSION)
# Install tensorflow v1.15 required for OpenVINO model conversion
!pip install tensorflow-gpu==1.15
# Install Tensor flow models and scripts.
repository = '--depth 1 --branch v1.13.0 https://github.com/tensorflow/models.git'
!pip install tf-slim
!git clone $repository
# Install required TF packages
!sudo -s apt-get install -qq protobuf-compiler python-tk
!pip install Cython contextlib2 pillow lxml matplotlib PyDrive pycocotools build utils dataclasses install azure-iot-device azure-iot-hub numpy==1.17
# Setup python path for TF object detection API and TF-Slim
import os
import sys

cwd = os.getcwd()
sys.path.append(cwd)

research = cwd + '/models/research'
sys.path.append(research)

slim = cwd + '/models/research/slim'
sys.path.append(slim)

%set_env PYTHONPATH=''
os.environ['PYTHONPATH'] = research + ":" + slim +  ":" + cwd
os.environ['PYTHONPATH']
# Update protocol buffers for TF object detection API
!curl -OL 'https://github.com/google/protobuf/releases/download/v3.2.0/protoc-3.2.0-linux-x86_64.zip'

# Unzip it
unzipcmd = '-o protoc-3.2.0-linux-x86_64.zip -d protoc3'
!unzip $unzipcmd

# Put it into /usr/local/bin to put it in the system path
movecmd = 'mv protoc3/bin/* /usr/local/bin/'
!sudo $movecmd

# Add header files to the linker's include path
movecmd = 'mv protoc3/include/* /usr/local/include/'
!sudo $movecmd
# Jump into the tensorflow object detection API research directory
researchfolder =  modelroot + '/models/research'
%cd $researchfolder

# As per their installation instructions, compile everything in the protos folder using protoc
protoccmd = '/usr/local/bin/protoc ' + 'object_detection/protos/*.proto --python_out=.'
!$protoccmd
# Check Tensorflow version. OpenVINO currently requires TF 1.x, so that's what we use
import tensorflow.compat.v1 as tf
print(tf.__version__)
print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
# Run setup for TF object detection API
args = researchfolder + '/setup.py build'
!python $args

# Install TF object detection API
args = researchfolder + '/setup.py install'
!python $args
# Run tensorflow model builder test just to make sure we have TF object detection API set up correctly
args = modelroot + '/models/research/object_detection/builders/model_builder_test.py'
!python $args
# Create data folder for training dataset and model
%cd $modelroot
!mv $modelroot/bowls_voc $modelroot/data
%cd data
# This is a little wonky, but we are later on using a script from the TF object detection API,
# create_pet_tf_record.py to generate a TF record. This script requires certain things in the
# dataset that we don't have, and which we don't use. We create those things here.
from PIL import Image

image_files = os.listdir('images')
im_files = [os.path.splitext(x)[0] for x in image_files]
with open('annotations/trainval.txt', 'w') as text_file:
  for row in im_files:
    text_file.write(row + '\n')

%cd ./annotations
!mkdir trimaps

image = Image.new('RGB', (640, 480))
for fname in os.listdir("xmls"):
  fname, _ = os.path.splitext(fname)
  image.save(os.path.join("trimaps", fname + ".png"))
# Create category labels file for Tensorflow training and validation files (label map)
label_map_fpath = modelroot + '/models/research/object_detection/data/bowls.pbtxt'
print(label_map_fpath)
%%writefile $label_map_fpath
item {
  id: 1
  name: 'bowl'
}
# Fix to remove ^M nonprintable char from end of string lines in file created above
# to see issue run "!cat -v $label_map_fname" before and after fix
with open(label_map_fpath, 'r') as file:
    label_map_file = file.read()
update_file = open(label_map_fpath, "w")
update_file.writelines(label_map_file)
update_file.close() 
# Create Tensorflow training data record files by co-opting this create_pet_tf_record.py script for our purposes
# This will take a while (tens of minutes probably)
%cd $modelroot/data

script = modelroot + "/models/research/object_detection/dataset_tools/create_pet_tf_record.py"
args = f"{script} --label_map_path={label_map_fpath} --data_dir=./ --output_dir=./ --num_shards=1"
!python $args

# Now update the names of the output files
!mv pet_faces_train.record-00000-of-00001 tf_train.record
!mv pet_faces_val.record-00000-of-00001 tf_val.record
# Download pretrained model for transfer learning: SSD Lite MobileNet V2 COCO
!curl -OL 'http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz'
model_file = os.path.join(modelroot, "data", "ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz")
# Uncompress model
import os
import shutil
import glob
import urllib
import tarfile
import urllib.request

tar = tarfile.open(model_file)
tar.extractall()
tar.close()
# Prepare model config file paths and tensorflow configuration file for SSDLiteV2 Model
config_fname = modelroot + '/models/research/object_detection/samples/configs/ssdlite_mobilenet_retrained.config'
fine_tune_checkpoint = '"' + modelroot + '/data/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt' + '"'
input_path_train = '"' + modelroot + '/data/tf_train.record' + '"'
label_map_path = '"' + modelroot + '/models/research/object_detection/data/bowls.pbtxt' + '"'
input_path_eval = '"' + modelroot + '/data/tf_val.record' + '"'
%%writefile $config_fname
model {
  ssd {
    num_classes: 1
    box_coder {
      faster_rcnn_box_coder {
        y_scale: 10.0
        x_scale: 10.0
        height_scale: 5.0
        width_scale: 5.0
      }
    }
    matcher {
      argmax_matcher {
        matched_threshold: 0.5
        unmatched_threshold: 0.5
        ignore_thresholds: false
        negatives_lower_than_unmatched: true
        force_match_for_each_row: true
      }
    }
    similarity_calculator {
      iou_similarity {
      }
    }
    anchor_generator {
      ssd_anchor_generator {
        num_layers: 6
        min_scale: 0.2
        max_scale: 0.95
        aspect_ratios: 1.0
        aspect_ratios: 2.0
        aspect_ratios: 0.5
        aspect_ratios: 3.0
        aspect_ratios: 0.3333
      }
    }
    image_resizer {
      fixed_shape_resizer {
        height: 300
        width: 300
      }
    }
    box_predictor {
      convolutional_box_predictor {
        min_depth: 0
        max_depth: 0
        num_layers_before_predictor: 0
        use_dropout: false
        dropout_keep_probability: 0.8
        kernel_size: 3
        use_depthwise: true
        box_code_size: 4
        apply_sigmoid_to_scores: false
        conv_hyperparams {
          activation: RELU_6,
          regularizer {
            l2_regularizer {
              weight: 0.00004
            }
          }
          initializer {
            truncated_normal_initializer {
              stddev: 0.03
              mean: 0.0
            }
          }
          batch_norm {
            train: true,
            scale: true,
            center: true,
            decay: 0.9997,
            epsilon: 0.001,
          }
        }
      }
    }
    feature_extractor {
      type: 'ssd_mobilenet_v2'
      min_depth: 16
      depth_multiplier: 1.0
      use_depthwise: true
      conv_hyperparams {
        activation: RELU_6,
        regularizer {
          l2_regularizer {
            weight: 0.00004
          }
        }
        initializer {
          truncated_normal_initializer {
            stddev: 0.03
            mean: 0.0
          }
        }
        batch_norm {
          train: true,
          scale: true,
          center: true,
          decay: 0.9997,
          epsilon: 0.001,
        }
      }
    }
    loss {
      classification_loss {
        weighted_sigmoid {
        }
      }
      localization_loss {
        weighted_smooth_l1 {
        }
      }
      hard_example_miner {
        num_hard_examples: 3000
        iou_threshold: 0.99
        loss_type: CLASSIFICATION
        max_negatives_per_positive: 3
        min_negatives_per_image: 3
      }
      classification_weight: 1.0
      localization_weight: 1.0
    }
    normalize_loss_by_num_matches: true
    post_processing {
      batch_non_max_suppression {
        score_threshold: 1e-8
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SIGMOID
    }
  }
}

train_config: {
  batch_size: 24
  optimizer {
    rms_prop_optimizer: {
      learning_rate: {
        exponential_decay_learning_rate {
          initial_learning_rate: 0.004
          decay_steps: 800720
          decay_factor: 0.95
        }
      }
      momentum_optimizer_value: 0.9
      decay: 0.9
      epsilon: 1.0
    }
  }
  fine_tune_checkpoint: $fine_tune_checkpoint
  fine_tune_checkpoint_type:  "detection"
  num_steps: 200000
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  data_augmentation_options {
    ssd_random_crop {
    }
  }
}

train_input_reader: {
  tf_record_input_reader {
    input_path: $input_path_train
  }
  label_map_path: $label_map_path
}

eval_config: {
  num_examples: 8000
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 10
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: $input_path_eval
  }
  label_map_path: $label_map_path
  shuffle: false
  num_readers: 1
}
# Update config file paths
with open(config_fname, 'r') as f:
    config_file = f.read()
config_file = config_file.replace('$fine_tune_checkpoint', fine_tune_checkpoint)
config_file = config_file.replace('$label_map_path', label_map_path)
config_file = config_file.replace('$input_path_train', input_path_train)
config_file = config_file.replace('$input_path_eval', input_path_eval)
update_file = open(config_fname, 'w')
update_file.writelines(config_file)
update_file.close() 

# Run tensorflow fine tuning training.
# To train on CPU instead of GPU uncomment the 'CUDA_VISIBLE_DEVICES' line below
#
# Note that we train for 30k steps here. This is sufficient to get reasonable results, though you may prefer to train for longer.
# **This will take several hours on a GPU compute node, and days on a CPU machine. It would be preferable to run on GPU for faster training. **
# If your kernel disconnects, don't worry, the code checkpoints periodically, click rerun all cells and it will pick up the latest checkpoint.
%cd $modelroot

model_dir = os.path.join(modelroot, "data", "retrained")
!mkdir $model_dir

import os
# os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
!python ./models/research/object_detection/model_main.py --pipeline_config_path=$config_fname --model_dir=$model_dir --alsologtostderr --num_train_steps=30000 --num_eval_steps=30000
# Get the last checkpoint to use for exporting a frozen graph
import re

%cd $modelroot/data

lst = os.listdir('retrained')
lf = filter(lambda k: 'model.ckpt-' in k, lst)
fileList = str(list(lf))
checkPointNumbers = re.findall(r'[0-9]+', fileList)
checkPointNumbers = [int(i) for i in checkPointNumbers]  
last_model = 'model.ckpt-' + str(max(checkPointNumbers))
print(last_model)
# Export frozen graph
!python $modelroot/models/research/object_detection/export_inference_graph.py --input_type=image_tensor --pipeline_config_path=$config_fname --output_directory=fine_tuned_model --trained_checkpoint_prefix=retrained/$last_model
# Test inference on photo using frozen graph
import tensorflow.compat.v1 as tf
import numpy as np
import os
import cv2
from object_detection.utils import label_map_util
from object_detection.utils import visualization_utils as vis_util
from matplotlib import pyplot as plt
from PIL import Image

# Path to the frozen graph:
PATH_TO_FROZEN_GRAPH = modelroot + '/data/fine_tuned_model/frozen_inference_graph.pb'

# Path to the label map
PATH_TO_LABEL_MAP = modelroot + '/models/research/object_detection/data/bowls.pbtxt'

# Bowl image
# Note that because the train/eval split is randomized, it is possible this image was in the train split.
IMAGE_PATH = modelroot + '/data/images/bowl_000000084259.jpg'

# Number of classes 
NUM_CLASSES = 1

# Minimum confidence value needed to display the bounding box on the image. In range [0.0, 1.0].
MIN_THRESHOLD = 0.40

# Read the frozen graph
detection_graph = tf.Graph()
with detection_graph.as_default():
    od_graph_def = tf.GraphDef()
    with tf.gfile.GFile(PATH_TO_FROZEN_GRAPH, 'rb') as fid:
        serialized_graph = fid.read()
        od_graph_def.ParseFromString(serialized_graph)
        tf.import_graph_def(od_graph_def, name='')

label_map = label_map_util.load_labelmap(PATH_TO_LABEL_MAP)
categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
category_index = label_map_util.create_category_index(categories)

# Detection
with detection_graph.as_default():
    with tf.Session(graph=detection_graph) as sess:
        # Read in the image and convert BGR to RGB
        image_np = cv2.imread(IMAGE_PATH)[:,:,::-1]

        # Expand dimensions since the model expects images to have shape: [1, None, None, 3] 
        image_np_expanded = np.expand_dims(image_np, axis=0)

        # Extract image tensor
        image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
        
        # Extract detection boxes
        boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
        
        # Extract detection scores
        scores = detection_graph.get_tensor_by_name('detection_scores:0')
        
        # Extract detection classes
        classes = detection_graph.get_tensor_by_name('detection_classes:0')
        
        # Extract number of detections
        num_detections = detection_graph.get_tensor_by_name('num_detections:0')

        # Actual detection.
        boxes, scores, classes, num_detections = sess.run([boxes, scores, classes, num_detections], feed_dict={image_tensor: image_np_expanded})
        print(f"BOXES (shaped {boxes.shape}):\n{boxes}")
        print(f"SCORES (shaped {scores.shape}):\n{scores}")
        print(f"CLASSES (shaped {classes.shape}):\n{classes}")
        print(f"NDETECTIONS (shaped {num_detections.shape}):\n{num_detections}")

        # Visualization of the results of a detection.
        vis_util.visualize_boxes_and_labels_on_image_array(
            image_np,
            np.squeeze(boxes),
            np.squeeze(classes).astype(np.int32),
            np.squeeze(scores),
            category_index,
            use_normalized_coordinates=True,
            line_thickness=3,
            min_score_thresh=MIN_THRESHOLD
            )
# Display test inference photo
from matplotlib.pyplot import imshow
import numpy as np
from PIL import Image

%matplotlib inline
plt.figure(figsize = (12, 8))
imshow(image_np)<jupyter_output><empty_output><jupyter_text>OpenVINOAt this point, you should have a model exported from Tensorflow and it should be providing good bounding boxes on bowls.The rest of this notebook will show you how to convert the model into the format that the EyeSOM dev kit requires,and then how to download it to your device.<jupyter_code># Use OpenVINO to convert the frozen graph PB to IR format
%cd $modelroot

convertCMD = "docker run --rm -v `pwd`:/working -w /working openvino/ubuntu18_dev:2021.1 "
convertCMD += '/opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py '
convertCMD += '--input_model /working/data/fine_tuned_model/frozen_inference_graph.pb '
convertCMD += '--tensorflow_object_detection_api_pipeline_config /working/data/fine_tuned_model/pipeline.config '
convertCMD += '--tensorflow_use_custom_operations_config /opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/ssd_v2_support.json '
convertCMD += '--reverse_input_channels'

print(convertCMD)
!$convertCMD
%%writefile $modelroot/compile.sh
#!/bin/bash

# OpenVINO compilation script

source /opt/intel/openvino_2021/bin/setupvars.sh

/opt/intel/openvino_2021/deployment_tools/inference_engine/lib/intel64/myriad_compile \
    -m /working/frozen_inference_graph.xml -o /working/ssdlite_mobilenet_v2.blob -VPU_NUMBER_OF_SHAVES 8 -VPU_NUMBER_OF_CMX_SLICES 8 -ip U8 -op FP32
# Use OpenVINO to compile IR format to blob
%cd $modelroot

# Run compilation in the container (ignore all the XLink warnings)
!docker run --rm -v `pwd`:/working -w /working openvino/ubuntu18_dev:2021.1 /bin/bash compile.sh

print(convertCMD)
!$convertCMD
# Package up blob for delevery to devkit

# Clean recreate directory if it exists and create it if not 
!rm -rf blob
!mkdir blob

!cp $modelroot/ssdlite_mobilenet_v2.blob blob/ssdlite_mobilenet_v2.blob
%%writefile $modelroot/blob/labels.txt
zeroindex
bowl
%%writefile $modelroot/blob/config.json
{
    "DomainType": "ssd100",
    "LabelFileName": "labels.txt",
    "ModelFileName": "ssdlite_mobilenet_v2.blob"
}
# Package up model and support files for dev kit
!cd $modelroot/blob && zip -r model.zip ./*
%cd $modelroot
!mv $modelroot/blob/model.zip ./ 
# Use the default datasore associated with the current workspace
from azureml.core import Workspace
from azureml.core import Dataset

ws = Workspace.from_config()
ds = ws.get_default_datastore()

print(ds.name, ds.datastore_type, ds.account_name, ds.container_name)
# Set model as unencrypted by default
MODEL_ENCRYPTED = False
# Optional - protect the model with Azure Percept SDK

# install Azure Percept SDK
!pip install sczpy

# Declare the Azure account and model management variables to be used for model protection.
AZURE_CLIENT_ID='<ENTER YOUR SERVICE PRINCIPAL CLIENT ID>'
AZURE_CLIENT_SECRET='<ENTER YOUR SERVICE PRINCIPAL CLIENT SECRET>'
AZURE_TENANT_ID='<ENTER YOUR SERVICE PRINCIPAL TENANT ID>'
SERVER_URL='<ENTER YOUR AZUURE PERCEPT MM SERVER URL>'
MODEL_NAME='<ENTER YOUR MODEL NAME>'
MODEL_VERSION='<ENTER YOUR MODEL VERSION>'

# Protect the model
import sczpy
import os

os.environ["AZURE_CLIENT_ID"] = AZURE_CLIENT_ID
os.environ["AZURE_CLIENT_SECRET"] = AZURE_CLIENT_SECRET
os.environ["AZURE_TENANT_ID"] = AZURE_TENANT_ID

client = sczpy.SCZClient(SERVER_URL)
client.register_model(MODEL_NAME, MODEL_VERSION)
os.rename("model.zip", "model.unencrypted.zip")
client.encrypt(MODEL_NAME, MODEL_VERSION, "model.unencrypted.zip", "model.zip")
client.upload_model(MODEL_NAME, MODEL_VERSION, "model.zip")

# Set the model tag as encrypted
MODEL_ENCRYPTED = True
# Upload the model.zip file to the datastore
ds.upload_files(['model.zip'], target_path='transfer-learning-models', overwrite=True)
# Install Azure Storage tools 
%pip install azure-storage-blob==2.1.0 msrest
# Generate download SAS URL for model.zip
from datetime import datetime, timedelta
from azure.storage.blob import (
    BlockBlobService,
    ContainerPermissions,
    BlobPermissions,
    PublicAccess,
)
   
AZURE_ACC_NAME = ds.account_name
AZURE_PRIMARY_KEY = ds.account_key
AZURE_CONTAINER = ds.container_name
AZURE_BLOB=ds.name
AZURE_File='transfer-learning-models/model.zip' 

block_blob_service = BlockBlobService(account_name=AZURE_ACC_NAME, account_key=AZURE_PRIMARY_KEY)
sas_url = block_blob_service.generate_blob_shared_access_signature(AZURE_CONTAINER,
                                                                   AZURE_File,
                                                                   permission=BlobPermissions.READ,
                                                                   expiry= datetime.utcnow() + timedelta(hours=30*24))
downloadurl ='https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+AZURE_File+'?'+sas_url
print(downloadurl)
# Install Azure Iot Hub tools
%pip install azure-iot-hub
import sys
from azure.iot.hub import IoTHubRegistryManager
from azure.iot.hub.models import Twin, TwinProperties
# Incorporate the connection string, device_id and the module_id values from your IoT Hub
# Go to https://portal.azure.com
# Select your IoT Hub
# Click on Shared access policies
# Click 'service' policy on the right (or another policy having 'service connect' permission)
# Copy Connection string--primary key
CONNECTION_STRING = "<YOUR-CONNECTION-STRING>"

DEVICE_ID = "<YOUR-DEV-ID>"   # Change this to your device's name as found in the IoT Azure Portal
MODULE_ID = "azureeyemodule"  # Make sure this is what your azureeyemodule is called (check your IoT device in the Azure Portal)

iothub_registry_manager = IoTHubRegistryManager(CONNECTION_STRING)
module_twin = iothub_registry_manager.get_module_twin(DEVICE_ID, MODULE_ID)

print ( "" )
print ( "Module twin properties before update    :" )
print ( "{0}".format(module_twin.properties) )
# Update twin
twin_patch = Twin()
if MODEL_ENCRYPTED == True:
    twin_patch.properties = TwinProperties(desired={"ModelZipUrl": downloadurl, 
                                                    "SecureAILifecycleEnabled": True, 
                                                    "SCZ_MODEL_NAME": MODEL_NAME, 
                                                    "SCZ_MODEL_VERSION": MODEL_VERSION, 
                                                    "SCZ_MM_SERVER_URL": SERVER_URL, 
                                                    "DownloadSecuredModelFromMMServer": False})
else:
    twin_patch.properties = TwinProperties(desired={"ModelZipUrl": downloadurl, 
                                                    "SecureAILifecycleEnabled": False})
updated_module_twin = iothub_registry_manager.update_module_twin(DEVICE_ID, MODULE_ID, twin_patch, module_twin.etag)

print ( "" )
print ( "Module twin properties after update     :" )
print ( "{0}".format(updated_module_twin.properties) )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08333333333333333, 'fraction_numerical': 0.016695389918138733, 'mean_word_length': 3.5623259623259624, 'pattern_counts': {'":': 13, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 31, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '26041', 'license': 'MIT', 'n_tokens_mistral': 9196, 'n_tokens_neox': 8621, 'n_words': 2518, 'path': 'machine-learning-notebooks/transfer-learning/transfer-learning-using-ssd.ipynb', 'repo_name': 'microsoft/azure-percept-samples', 'size': '27852'}