<jupyter_start><jupyter_text>1 - Beginning WorkflowsIn this lesson, we'll cover the basics of using atomate to run calculations. This will be a hands-on lesson where we dive into running a full workflows and break that down into components to understand how the various moving parts give us the ability to scale from 1 calculation to 10's of thousands.<jupyter_code>import mp_workshop.atomate<jupyter_output><empty_output><jupyter_text>Building a workflowTo begin, we'll start by grabbing a structure from materials project using pymatgen and the MPRester interface we learned about in a previous course<jupyter_code>from pymatgen import MPRester

mpr = MPRester()

struc = mpr.get_structure_by_material_id("mp-27")
print(struc)<jupyter_output><empty_output><jupyter_text>Now, let's construct a workflow using atomate to optimize this structure in DFT<jupyter_code>from atomate.vasp.workflows.presets.core import wf_structure_optimization
wf = wf_structure_optimization(struc,{"DB_FILE": None})
print(wf)<jupyter_output><empty_output><jupyter_text>Get some more information on the workflow<jupyter_code>wf.as_dict()<jupyter_output><empty_output><jupyter_text>Running with Fake VASP to simulate a DFT calculationDue to a combination of licensing issues and just not being able to run this quickly on the jupyter server, we're going to simulate VASP running with a magic function. You will later learn about powerups, which let you modify a workflow. For this exercise we're going to use a powerup that will replace the normal VASP running functionality with something that just copies files we've prepared for you<jupyter_code>from atomate.vasp.powerups import use_fake_vasp<jupyter_output><empty_output><jupyter_text>Lets do some work to get the path to fake VASP files<jupyter_code>from mp_workshop.atomate import si_struct_opt_path

print(si_struct_opt_path)
wf = use_fake_vasp(wf, ref_dirs={"Si-structure optimization": si_struct_opt_path})
wf.as_dict()<jupyter_output><empty_output><jupyter_text>Now we have to get ourself a LaunchPad so that we can submit this workflow to our databaseAtomate uses Fireworks as its workflow engine. Fireworks hides the database with an object called a LaunchPad. This allows you to submit and query workflows from anywhere you have database access. We need to get ourselves a LaunchPad object so we can submit our workflow<jupyter_code>from fireworks.core.launchpad import LaunchPad

lp = LaunchPad.auto_load()<jupyter_output><empty_output><jupyter_text>For this one time, we have to initialize the database. In everyday use, we'll only do this once. For this lesson, we'll use it a few times:<jupyter_code>lp.reset(None,require_password=False)<jupyter_output><empty_output><jupyter_text>We can use the launchpad to add a workkflow to our database:<jupyter_code>lp.add_wf(wf)<jupyter_output><empty_output><jupyter_text>Monitoring WorkflowsFireworks lets you monitor the status of workflows and fireworks using both python and the command line. Let's start off by looking at the status of our workflow. For each bit of python code, i'll include a cell with a command line command using jupyter-notebook's '!' functionality. In practice, we use the command line tools quite a bit and will be emphasized in this notebook.**Command Line Access in Jupyter**: Jupyter lets you running command line commands by prefacing them with the exclamation mark:<jupyter_code># Lets get workflows

def get_wflows():
    for wf_id in lp.get_wf_ids():
        for key,value in lp.get_wf_summary_dict(wf_id).items():
            print(key, ": ",value)
        print("\n")

get_wflows()<jupyter_output><empty_output><jupyter_text>This is how you get workflow information on the command line<jupyter_code>!lpad get_wflows
def get_fws():
    for fw_id in lp.get_fw_ids():
        fw = lp.get_fw_dict_by_id(fw_id)
        for prop in ["fw_id","updated_on","state","name"]:
            print(prop, ": ",fw[prop])

        print("\n")
        
get_fws()<jupyter_output><empty_output><jupyter_text>This command line gets you the same information<jupyter_code>!lpad get_fws
!lpad --help
# Let's look at what this command can do:
!lpad get_fws --help<jupyter_output><empty_output><jupyter_text>Now lets run this workflowThere are a few different ways to run a workflow. The first is to just run it within this notebook directly.<jupyter_code>from fireworks.core.rocket_launcher import launch_rocket
# Lets move into a temporary working directory
import os

os.mkdir("temp")
os.chdir("temp")
launch_rocket(lp)<jupyter_output><empty_output><jupyter_text>Now, lets see how that changed our fireworks<jupyter_code>!lpad get_fws<jupyter_output><empty_output><jupyter_text>This let me run a single firework in the notebook. What if I wanted to run multiple fireworks? First lets reset the old firework and add some more workflows to our database<jupyter_code># We can do the same thing using the command line:
!lpad rerun_fws 
!lpad get_fws
# Let's add the workflow a few more times to have multiple fireworks in database
lp.add_wf(wf)
lp.add_wf(wf)<jupyter_output><empty_output><jupyter_text>We can run all of the available fireworks using a 2 lines of python and a single command:<jupyter_code>from fireworks.core.rocket_launcher import rapidfire
rapidfire(lp)<jupyter_output><empty_output><jupyter_text>This let us run fireworks until we no longer had any to run. But we're still running fireworks in our jupyter notebook. If I want to run on this on another machine I need to do something else. Normally, we would want to launch these jobs to our supercomputing queue and let that run them as resources become available.   Using the queue launcher:Setting up the queue launcher unfortunately takes some work. There are configuration files to tell atomate how to submit jobs, where the database is and what special parameters to use for this supercomputer. This has all been setup for you in this workshop. Once setup, to use the queue, we simply launch the fireworks to the queue. Lets start off by clearing the database of fireworks<jupyter_code>lp.reset(None,require_password=False)
from atomate.vasp.workflows.presets.core import wf_bandstructure

wf = wf_bandstructure(struc,{"DB_FILE": None})
wf.as_dict()
from mp_workshop.atomate import si_static_path,si_nscf_line_path,si_nscf_uniform_path
wf = use_fake_vasp(wf,{"Si-structure optimization":si_struct_opt_path,
                       "Si-static": si_static_path,
                       "Si-nscf uniform" : si_nscf_uniform_path,
                       "Si-nscf line": si_nscf_line_path})

lp.add_wf(wf)
!lpad get_fws<jupyter_output><empty_output><jupyter_text>Fireworks has a commmand line method to submit jobs to the SLURM queue:<jupyter_code>!qlaunch -r rapidfire --nlaunches 1<jupyter_output><empty_output><jupyter_text>Now, the supercomputer will take care of running the jobs and eventually we can test to see that they are working<jupyter_code>!lpad get_fws<jupyter_output><empty_output><jupyter_text>Now lets have qlaunch submit fireworks until all are done.<jupyter_code>!qlaunch -r rapidfire
!lpad get_fws<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06654473832301631, 'fraction_numerical': 0.0011254924029262803, 'mean_word_length': 5.732007575757576, 'pattern_counts': {'":': 8, '<': 81, '<?xml version=': 0, '>': 81, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '147691', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2266, 'n_tokens_neox': 2146, 'n_words': 796, 'path': 'lessons/atomate/1 - Beginning Workflows.ipynb', 'repo_name': 'srshivani/workshop', 'size': '7108'}