<jupyter_start><jupyter_text>Automating image buildingWe saw in the last notebook how we can build images of our funwave-tvd code and use Agave to make the process a bit easier. We can take some lessons learned from the devops community to automate the building of our images and implement basic benchmarking and testing. While the Agave fork app we created is handy, it doesn't provide particularly good visibility, let along security. We certainly do not want to share an app like that for others to use. So, let's start start by creating a new Agave app that will build our Docker container. First up, our updated app assets.  Creating more meaningful DockerfilesWhile functional, our previous Dockerfile didn't give us much info we could use for things like attribution, discovery, etc. Let's add in some additional fields to give our Dockerfile meaning. We will use a couple new Dockerfile directives to do thisARG a runtime argument supplied to the `docker build` commandLABEL one or more terms applied to the image as metadata<jupyter_code>writefile("funwave-tvd-docker-automation/Dockerfile","""
FROM stevenrbrandt/science-base
MAINTAINER <NAME> <<EMAIL>>

ARG BUILD_DATE
ARG VERSION

LABEL org.agaveplatform.ax.architecture="x86_64"                                \
      org.agaveplatform.ax.build-date="\$BUILD_DATE"                             \
      org.agaveplatform.ax.version="\$VERSION"                             \
      org.agaveplatform.ax.name="${AGAVE_USERNAME}/funwave-tvd"    \
      org.agaveplatform.ax.summary="Funwave-TVD is a code to simulate the shallow water and Boussinesq equations written by Dr. <NAME>." \
      org.agaveplatform.ax.vcs-type="git"                                       \
      org.agaveplatform.ax.vcs-url="https://github.com/fengyanshi/FUNWAVE-TVD" \
      org.agaveplatform.ax.license="BSD 3-clause"
      
USER root
RUN mkdir -p /home/install
RUN chown jovyan /home/install
USER jovyan

RUN cd /home/install && \
    git clone https://github.com/fengyanshi/FUNWAVE-TVD && \
    cd FUNWAVE-TVD/src && \
    perl -p -i -e 's/FLAG_8 = -DCOUPLING/#$&/' Makefile && \
    make

WORKDIR /home/install/FUNWAVE-TVD/src
RUN mkdir -p /home/jovyan/rundir
WORKDIR /home/jovyan/rundir
""")<jupyter_output>Writing file `funwave-tvd-docker-automation/Dockerfile'<jupyter_text>Single purpose wrapper scriptsIn our previous wrapper script, we simply took whatever was given to us an ran it. Here we will restrict the wrapper to run a specific build comman.   > Note that we mix an match a couple variable types. The `Agave_*` variable are template variables resolved by Agave at runtime with the vales from the job etails. The `version` variable is a parameter we will define in our app description.<jupyter_code>writefile("funwave-tvd-docker-automation/funwave-build-wrapper.txt","""

sudo docker build \
    --build-arg "BUILD_DATE=\${AGAVE_JOB_SUBMIT_TIME}" \
    --build-arg "VERSION=\${code_version}" \
    --rm -t funwave-tvd:\${code_version} .

docker inspect funwave-tvd:\${code_version}

""")<jupyter_output>Writing file `funwave-tvd-docker-automation/funwave-build-wrapper.txt'<jupyter_text>More descriptive apps  Now we need to create some JSON to tell Agave how to run and advertise our app. This app definition will look a lot like the fork app definition with a few changes. First, we are updating the app id so a new app will be created. Second we change the parameter.  * `code_version` is a string parameter describing the version of the code.   We have also removed the data file input from the previous app description. This is because our deployment folder contains the Dockerfile to build our image. No other info is needed to run our build app.<jupyter_code>writefile("funwave-tvd-docker-automation/funwave-build-app.txt","""
{  
   "name":"${AGAVE_USERNAME}-${MACHINE_NAME}-funwave-dbuild",
   "version":"1.0",
   "label":"Builds the funwave docker image",
   "shortDescription":"Funwave docker build",
   "longDescription":"",
   "deploymentSystem":"${AGAVE_STORAGE_SYSTEM_ID}",
   "deploymentPath":"automation/funwave-tvd-docker-automation",
   "templatePath":"funwave-build-wrapper.txt",
   "testPath":"test.txt",
   "executionSystem":"${AGAVE_EXECUTION_SYSTEM_ID}",
   "executionType":"CLI",
   "parallelism":"SERIAL",
   "modules":[],
   "inputs":[],
   "parameters":[{
     "id" : "code_version",
     "value" : {
       "visible":true,
       "required":true,
       "type":"string",
       "order":0,
       "enquote":false,
       "default":"latest"
     },
     "details":{
         "label": "Version of the code",
         "description": "If true, output will be packed and compressed",
         "argument": null,
         "showArgument": false,
         "repeatArgument": false
     },
     "semantics":{
         "argument": null,
         "showArgument": false,
         "repeatArgument": false
     }
   }],
   "outputs":[]
}
""")<jupyter_output>Writing file `funwave-tvd-docker-automation/funwave-build-app.txt'<jupyter_text>Here is our default test file<jupyter_code>writefile("funwave-tvd-docker-automation/test.txt","""
code_version=latest
""")
!files-mkdir -S ${AGAVE_STORAGE_SYSTEM_ID} -N automation
!files-upload -S ${AGAVE_STORAGE_SYSTEM_ID} -F funwave-tvd-docker-automation automation
!apps-addupdate -F funwave-tvd-docker-automation/funwave-build-app.txt<jupyter_output>[1;0m[1;0mSuccessfully added app training003-sandbox-funwave-dbuild-1.0[0m[0m<jupyter_text>Running the build<jupyter_code>requestbin_url = !requestbin-create
os.environ['REQUESTBIN_URL'] = requestbin_url[0]
setvar("""
WEBHOOK_URL=${REQUESTBIN_URL}
""")<jupyter_output>WEBHOOK_URL=https://requestbin.agaveapi.co/126tsd91<jupyter_text>Now we'll run our build using the following job request. This is very similar to before.<jupyter_code>writefile("funwave-tvd-docker-automation/job.json","""
 {
   "name":"funwave-build",
   "appId": "${AGAVE_USERNAME}-${MACHINE_NAME}-funwave-dbuild-1.0",
   "maxRunTime":"00:10:00",
   "archive": false,
   "notifications": [
    {
      "url":"${WEBHOOK_URL}",
      "event":"*",
      "persistent":"true"
    }
   ],
   "parameters": {
     "code_version":"latest"
   }
 }
""")<jupyter_output>Writing file `funwave-tvd-docker-automation/job.json'<jupyter_text>Because the setvar() command can evalute `$()` style bash shell substitutions, we will use it to submit our job. This will capture the output of the submit command, and allow us to parse it for the JOB_ID. We'll use the JOB_ID in several subsequent steps.<jupyter_code>setvar("""
# Capture the output of the job submit command
OUTPUT=$(jobs-submit -F funwave-tvd-docker-automation/job.json)
# Parse out the job id from the output
JOB_ID=$(echo $OUTPUT | cut -d' ' -f4)
""")
for iter in range(20):
    setvar("STAT=$(jobs-status $JOB_ID)")
    stat = os.environ["STAT"]
    sleep(5.0)
    if stat == "FINISHED" or stat == "FAILED":
        break
!jobs-output-get -P ${JOB_ID} funwave-build.out<jupyter_output>Sending build context to Docker daemon  15.87kB


Step 1/13 : FROM stevenrbrandt/science-base
 ---> e93db47f971c
Step 2/13 : MAINTAINER <NAME> <<EMAIL>>
 ---> Using cache
 ---> 13f7ea48e9cb
Step 3/13 : ARG BUILD_DATE
 ---> Using cache
 ---> 548f7a1528d7
Step 4/13 : ARG VERSION
 ---> Using cache
 ---> fdf915eadde3
Step 5/13 : LABEL org.agaveplatform.ax.architecture "x86_64" org.agaveplatform.ax.build-date "$BUILD_DATE" org.agaveplatform.ax.version "$VERSION" org.agaveplatform.ax.name "training003/funwave-tvd" org.agaveplatform.ax.summary "Funwave-TVD is a code to simulate the shallow water and Boussinesq equations written by Dr. <NAME>." org.agaveplatform.ax.vcs-type "git" org.agaveplatform.ax.vcs-url "https://github.com/fengyanshi/FUNWAVE-TVD" org.agaveplatform.ax.license "BSD 3-clause"
 ---> Running in 96e93ee95458
 ---> d68fc3ac3cb6
Removing intermediate container 96e93ee95458
Step 6/13 : USER root
 ---> Running in bbd4a5e10730
 ---> 70866[...]<jupyter_text>To rebuild our Docker image, we can rerun our job submission command, or simply resumbit the previous job.<jupyter_code>setvar("""
# Capture the output of the job submit command
OUTPUT=$(jobs-resubmit ${JOB_ID})
# Parse out the job id from the output
JOB_ID=$(echo $OUTPUT | cut -d' ' -f4)
""")<jupyter_output>OUTPUT=Successfully resubmitted job 7446628594819600871-242ac11b-0001-007 as 495140868949470745-242ac11b-0001-007
JOB_ID=7446628594819600871-242ac11b-0001-007
{'fraction_non_alphanumeric': 0.11952522255192878, 'fraction_numerical': 0.026468842729970327, 'mean_word_length': 3.890307603017992, 'pattern_counts': {'":': 43, '<': 33, '<?xml version=': 0, '>': 45, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '574240', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2943, 'n_tokens_neox': 2719, 'n_words': 770, 'path': '06 - Automation/Automation and Benchmarking.ipynb', 'repo_name': 'agaveplatform/notebooks', 'size': '8479'}