<jupyter_start><jupyter_text>Register/Reference a Datastore<jupyter_code># workspace
ws = Workspace.from_config(
    path='./azureml-config.json')
print(ws.datastores)
#datastore = Datastore.register_azure_blob_container(workspace=ws, 
#                                             datastore_name='damoseerdata', 
#                                             container_name='seer-container',
#                                             account_name='damoaimlstorage', 
#                                             account_key='',
#                                             create_if_not_exists=True)
# data
datastore = ws.datastores['damoseerdata']

# compute target
compute = ws.compute_targets['damoseercompute']

print(datastore)
print(compute)<jupyter_output><azureml.data.azure_storage_datastore.AzureBlobDatastore object at 0x7f926c220278>
AmlCompute(workspace=Workspace.create(name='damo-mlworkspace', subscription_id='bc202ec2-54ef-4576-b7fb-a961c983398e', resource_group='damo-aiml'), name=damoseercompute, id=/subscriptions/bc202ec2-54ef-4576-b7fb-a961c983398e/resourceGroups/damo-aiml/providers/Microsoft.MachineLearningServices/workspaces/damo-mlworkspace/computes/damoseercompute, type=AmlCompute, provisioning_state=Succeeded, location=australiaeast, tags=None)<jupyter_text>Define Pipeline!The following will be created and then run:  1. Pipeline Parameters  2. Data Process Step  3. Training Step  4. Model Registration Step  5. Pipeline registration  6. Submit the pipeline for execution  Pipeline ParametersWe need to tell the Pipeline what it needs to learn to see!<jupyter_code>datapath = DataPath(datastore=datastore, path_on_datastore='hardware')
data_path_pipeline_param = (PipelineParameter(name="data", 
                                             default_value=datapath), 
                                             DataPathComputeBinding(mode='mount'))
print(data_path_pipeline_param)

# Configuration for data prep and training steps
dataprepEnvironment = Environment.from_pip_requirements('dataprepenv', 'requirements-dataprepandtraining.txt')
dataprepRunConfig = RunConfiguration()
dataprepRunConfig.environment = dataprepEnvironment<jupyter_output>(<azureml.pipeline.core.graph.PipelineParameter object at 0x7f926c23e320>, <azureml.data.datapath.DataPathComputeBinding object at 0x7f926c23e358>)<jupyter_text>Data Process Step<jupyter_code>seer_tfrecords = PipelineData(
    "tfrecords_set",
    datastore=datastore,
    is_directory=True
)

prepStep = PythonScriptStep(
    'parse.py',
    source_directory='.',
    name='Data Preparation',
    compute_target=compute,
    arguments=["--source_path", data_path_pipeline_param, "--target_path", seer_tfrecords],
    runconfig=dataprepRunConfig,
    inputs=[data_path_pipeline_param],
    outputs=[seer_tfrecords]
)

print(prepStep)<jupyter_output><azureml.pipeline.steps.python_script_step.PythonScriptStep object at 0x7f926c2205f8><jupyter_text>Training Step<jupyter_code>seer_training = PipelineData(
    "train",
    datastore=datastore,
    is_directory=True
)

train = Estimator(source_directory='.',
                    compute_target=compute,
                    entry_script='train.py',
                    pip_requirements_file='requirements-dataprepandtraining.txt')

trainStep = EstimatorStep(
    name='Model Training',
    estimator=train,
    estimator_entry_script_arguments=["--source_path", seer_tfrecords, 
                                    "--target_path", seer_training,
                                    "--epochs", 10,
                                    "--batch", 20,
                                    "--lr", 0.001],
    inputs=[seer_tfrecords],
    outputs=[seer_training],
    compute_target=compute
)

print(trainStep)<jupyter_output>WARNING - 'gpu_support' is no longer necessary; AzureML now automatically detects and uses nvidia docker extension when it is available. It will be removed in a future release.
WARNING - 'gpu_support' is no longer necessary; AzureML now automatically detects and uses nvidia docker extension when it is available. It will be removed in a future release.<jupyter_text>Register Model Step<jupyter_code>registerEnvironment = Environment.from_pip_requirements('registerenv', 'requirements-registration.txt')
registerRunConfig = RunConfiguration()
registerRunConfig.environment = registerEnvironment

seer_model = PipelineData(
    "model",
    datastore=datastore,
    is_directory=True
)

registerStep = PythonScriptStep(
    'register.py',
    source_directory='.',
    name='Model Registration',
    arguments=["--source_path", seer_training, 
               "--target_path", seer_model],
    inputs=[seer_training],
    outputs=[seer_model],
    compute_target=compute,
    runconfig=registerRunConfig
)

print(registerStep)<jupyter_output><azureml.pipeline.steps.python_script_step.PythonScriptStep object at 0x7f926c23ebe0><jupyter_text>Create and publish the Pipeline<jupyter_code>pipeline = Pipeline(workspace=ws, steps=[prepStep, trainStep, registerStep])

published_pipeline = pipeline.publish(
    name="Seer Pipeline", 
    description="Transfer learned image classifier. Uses folders as labels.")
# Submit the pipeline to be run
pipeline_run = Experiment(ws, 'seer',).submit(published_pipeline)
print('Run created with ID: ', pipeline_run.id)

RunDetails(pipeline_run).show()<jupyter_output>Run created with ID:  7103c1bc-807a-4e25-8874-e578511eb035
{'fraction_non_alphanumeric': 0.09322503230570427, 'fraction_numerical': 0.023444711094701864, 'mean_word_length': 3.7989371124889284, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '262727', 'license': 'MIT', 'n_tokens_mistral': 1686, 'n_tokens_neox': 1598, 'n_words': 285, 'path': 'seer_pipeline.ipynb', 'repo_name': 'Damovisa/seer', 'size': '5417'}