<jupyter_start><jupyter_text>Run Blueoil on Amazon SageMaker Docker build and push (to Amazon ECR)<jupyter_code>!bash ./docker_push_ecr.sh blueoil-sagemaker blueoil/blueoil:v0.22.0<jupyter_output><empty_output><jupyter_text>Preparing data (upload a face images subset of OpenimagesV4 to Amazon S3) Create sagemaker session<jupyter_code>import sagemaker
sess = sagemaker.Session()
import os
import shutil

def upload_data(sess, path, key_prefix='data', compress=False):
    if compress:
        path = shutil.make_archive(path, 'gztar', '.', path)
    s3_data = sess.upload_data(path=path, key_prefix=key_prefix)
    return s3_data<jupyter_output><empty_output><jupyter_text>Download a face image subset of OpenimagesV4 dataset<jupyter_code>!curl -O https://s3-ap-northeast-1.amazonaws.com/leapmind-public-storage/datasets/openimages_face.tgz
!tar xf openimages_face.tgz<jupyter_output><empty_output><jupyter_text>Upload dataset<jupyter_code>%%time
train_data = upload_data(sess, 'openimages_face', compress=True)<jupyter_output><empty_output><jupyter_text>Preparing config (upload to Amazon S3)  Create config filehttps://docs.blueoil.org/tutorial/image_det.htmlcreate `openimages_face_sample.py` by ```blueoil init -o openimages_face_sample.py```with```dataset:  format: OpenImagesV4  train dataset path: /opt/ml/input/data/dataset/openimages_face/  validation dataset path: /opt/ml/input/data/dataset/openimages_face/```<jupyter_code>%%time
config_data = upload_data(sess, 'openimages_face_sample.py', key_prefix='config', compress=False)
# # Retrieve data location

# train_data  = 's3://' + sagemaker.Session().default_bucket() + '/data/openimages_face.tar.gz'
# config_data += 's3://' + sagemaker.Session().default_bucket() + '/config/openimages_face_sample.yml'
print(config_data)
print(train_data)<jupyter_output><empty_output><jupyter_text>Train  On Amazon Sagemaker on-demand instance Create session<jupyter_code>import boto3

algorithm_name = 'blueoil-sagemaker'

client = boto3.client('sts')
account = client.get_caller_identity()['Account']

my_session = boto3.session.Session()
region = my_session.region_name

ecr_image = '{}.dkr.ecr.{}.amazonaws.com/{}:latest'.format(account, region, algorithm_name)<jupyter_output><empty_output><jupyter_text>Run trainning<jupyter_code>import sagemaker
from sagemaker.estimator import Estimator

train_instance_type = 'ml.p3.2xlarge'

estimator = Estimator(
    image_name=ecr_image, 
    role=sagemaker.get_execution_role(), 
    train_instance_count=1, 
    train_instance_type=train_instance_type, 
    hyperparameters={
        'config': '/opt/ml/input/data/config/openimages_face_sample.py', 
        'experiment_id': 'objectdetection_face_sample'
    })
estimator.fit({'dataset': train_data, 'config': config_data})<jupyter_output><empty_output><jupyter_text>Convert On Amazon Sagemaker on-demand instance<jupyter_code>from sagemaker.processing import ScriptProcessor, ProcessingInput, ProcessingOutput

convert_instance_type = 'ml.m5.xlarge'

processor = ScriptProcessor(
    image_uri=ecr_image,
    role=sagemaker.get_execution_role(),
    command=['python3'],
    base_job_name="blueoil-convert",
    instance_count=1,
    instance_type=convert_instance_type)
trained_model = estimator.model_data
converted_model = os.path.join(os.path.dirname(trained_model), 'converted')
processor.run(code='script/main.py',
    inputs=[
        ProcessingInput(source=train_data, destination='/opt/ml/processing/input/data/dataset'),
        ProcessingInput(source=estimator.model_data, destination='/opt/ml/processing/input/data/model'),
    ],
    outputs=[
        ProcessingOutput(source='/opt/ml/processing/output/converted', destination=converted_model),
    ],
    arguments=['convert', '--experiment_id', 'objectdetection_face_sample'],
)

print(f"Converted models are saved to {converted_model}")<jupyter_output><empty_output><jupyter_text>Download converted model<jupyter_code>!aws s3 cp $converted_model ./ --recursive
!tar zxvf output.tar.gz<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12211466865227104, 'fraction_numerical': 0.0062050136510300325, 'mean_word_length': 6.705544933078394, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '531154', 'license': 'Apache-2.0', 'n_tokens_mistral': 1407, 'n_tokens_neox': 1356, 'n_words': 208, 'path': 'blueoil_sagemaker/blueoil_openimages_example.ipynb', 'repo_name': 'hadusam/bluegrass', 'size': '4029'}