<jupyter_start><jupyter_text>Exploring cosmological simulations with CosmoSim Introduction to job handling with uws-clientCosmoSim is a web application available at http://www.cosmosim.org/ where data from cosmological simulations is available. This includes catalogues of dark matter halos (clusters) and galaxies for different time steps during the evolution of the simulated universe, merging information, substructure data, density fields and more.In this tutorial, we will use the [uws-client](https://github.com/aipescience/uws-client) for connecting with CosmoSim's UWS-interface for seeing your list of jobs, sending jobs and retrieving results.  ImportsImport the necessary libraries and the UWS module from the uws-client:<jupyter_code># load astropy for reading VOTABLE format
from astropy.io.votable import parse_single_table

# import matplotlib for plotting results, mplot3d for 3D plots
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
# import sys
# sys.path.append('<your own path>/uws-client')

from uws import UWS<jupyter_output><empty_output><jupyter_text>Setup the connectionThe URL for connecting with CosmoSim's uws-client is 'https://www.cosmosim.org/uws/query/'. You also need to define your username and password, either by inserting it directly below or by saving your credentials in a local cosmosim-user.json file and reading it here. The credentials are the same as on the CosmoSim webpage. If you do not have an account yet, please register at [CosmoSim registration](https://www.cosmosim.org/auth/registration/register). Alternatively, you can use the user `uwstest` with password `<PASSWORD>` for testing purposes. (Be aware that anyone can use this user and delete your results at any time!)<jupyter_code># set credentials here:
# username = 'uwstest'
# password = '<PASSWORD>'

# or read your own username and password from a json-file,
# format: { "username": "<yourname>", "password": "<<PASSWORD>>" }
import json
with open('cosmosim-user.json') as credentials_file:    
    username, password = json.load(credentials_file).values()

url = 'https://www.cosmosim.org/uws/query/'
c = UWS.client.Client(url, username, password)<jupyter_output><empty_output><jupyter_text>List previous jobsOnce the connection is set up, you can retrieve the list of previously run jobs with `c.get_job_list()`. You can also provide extra filters for the job list, e.g. filtering by phase or creation time of the job or just output the most recent ones using the `last` keyword.<jupyter_code>filters = {'phases': ['PENDING', 'COMPLETED', 'ERROR'], 'last': 5}
jobs = c.get_job_list(filters)

# printing the returned resulting jobs-object gives a list of jobs
print jobs<jupyter_output><empty_output><jupyter_text>For each job, its unique id, ownerId, creationTime and phase are stored within this job list. At CosmoSim, we store the table name as the `runId` for each job. If no table name was given during job creation, the current timestamp is used.<jupyter_code>print "# jobId, ownerId, creationTime, phase, runId:"
for job in jobs.job_reference:
    print job.id, job.ownerId, job.creationTime, job.phase[0], job.runId<jupyter_output><empty_output><jupyter_text>Create, check and run a jobFor creating a new job, first define the necessary parameters. For CosmoSim this is `query`, which is the SQL string and the optional parameters `queue` (long or short) and `table` (a unique table name). We set here an SQL query that will select the 10 most massive clusters from the MDPL2 simulation, Rockstar catalog.<jupyter_code>parameters = {'query': 
              'SELECT rockstarId, x,y,z, Mvir FROM MDPL2.Rockstar'\
                + ' WHERE snapnum=125 ORDER BY Mvir DESC LIMIT 10',
              'queue': 'short'}<jupyter_output><empty_output><jupyter_text>Now **create a new job** with these parameters:<jupyter_code>job = c.new_job(parameters)<jupyter_output><empty_output><jupyter_text>And print the job's id and phase:<jupyter_code>print job.job_id, job.phase[0]<jupyter_output><empty_output><jupyter_text>The job is created now, but it is not started yet - you can still adjust its parameters with c.set_parameters_job. E.g. let's change the queue to long:<jupyter_code>update_params = {'queue': 'long'}
job = c.set_parameters_job(job.job_id, update_params)<jupyter_output><empty_output><jupyter_text>Print the parameters to check this:<jupyter_code>for p in job.parameters:
    print p.id, p.value<jupyter_output><empty_output><jupyter_text>Now **start the job**, i.e. put it into the job queue using `run_job`:<jupyter_code>run = c.run_job(job.job_id)
print run.job_id<jupyter_output><empty_output><jupyter_text>The job should now also be visible in the web interface, at the Query Interface, left side, under 'Jobs'.  Let's check the job's phase:<jupyter_code>job = c.get_job(run.job_id)
print job.phase[0]<jupyter_output><empty_output><jupyter_text>You can also use the `wait` parameter to wait at most the specified number of seconds until the phase has changed:<jupyter_code>job = c.get_job(run.job_id, '10', 'QUEUED')
print job.phase[0]<jupyter_output><empty_output><jupyter_text>Repeat the step above using "EXECUTING" as job phase until the job phase is "COMPLETED".<jupyter_code>job = c.get_job(run.job_id, '10', 'EXECUTING')
print job.phase[0]<jupyter_output><empty_output><jupyter_text>Get the results Once your job is in "COMPLETED" phase, you can retrieve the results.Print the job result entries:<jupyter_code>for r in job.results:
    print r<jupyter_output><empty_output><jupyter_text>With `r.reference` you can access the URL of each result. Let's download the results in VOTABLE format:<jupyter_code>fileurl = str(job.results[1].reference)
resultfilename = "result.xml"
success = c.connection.download_file(fileurl, username, password, file_name=resultfilename)
if not success:
    print "File could not be downloaded, please check the job phase and result urls."
else:
    print "File downloaded successfully."<jupyter_output><empty_output><jupyter_text>Since there is only one table, we can quickly read the VOTABLE into a numpy array using astropy:<jupyter_code>table = parse_single_table(resultfilename, pedantic=False)
data = table.array<jupyter_output><empty_output><jupyter_text>Print the results row by row:<jupyter_code>print data<jupyter_output><empty_output><jupyter_text>Or print only a column:<jupyter_code>print data['x']<jupyter_output><empty_output><jupyter_text>Get the units for x and y values:<jupyter_code>field = table.get_field_by_id('x')
unit_x = field.unit
field = table.get_field_by_id('y')
unit_y = field.unit

print "Units for x and y: ", unit_x, unit_y<jupyter_output><empty_output><jupyter_text>Plot results<jupyter_code>%matplotlib inline
ax = plt.subplot(111)

# set axis labels
ax.set_xlabel('x [' + str(unit_x) + ']')
ax.set_ylabel('y [' + str(unit_y) + ']')

# set axis range
ax.set_xlim(0, 1000)
ax.set_ylim(0, 1000)

# plot data,
# using decreasing point size,
# so the biggest point is the most massive object
s = range(20,0,-2)
ax.scatter(data['x'], data['y'], s, color='b');
plt.show()<jupyter_output><empty_output><jupyter_text>Delete jobDelete the job on the server, because we don't need it anymore:<jupyter_code>deleted = c.delete_job(job.job_id)
deleted<jupyter_output><empty_output><jupyter_text>Example: Retrieve progenitors of a haloLet's do a more elaborate example: for the most massive dark matter halo from the previous query, get its progenitors that merged into this halo and plot their positions over time. We restrict the progenitors to those with mass > 1.e12/h solar masses.<jupyter_code># store id of most massive dark matter halo from query before
most_massive_rockstarId = data[0]['rockstarId']
query = """
SELECT p.rockstarId, p.snapnum as snapnum, p.x as x, p.y as y, p.z as z, p.Mvir as Mvir, p.Rvir as Rvir
FROM MDPL2.Rockstar AS p,
  (SELECT depthFirstId, lastProg_depthFirstId FROM MDPL2.Rockstar
  WHERE rockstarId = """ + str(most_massive_rockstarId) + """) AS m
WHERE p.depthFirstId BETWEEN m.depthFirstId AND m.lastProg_depthFirstId
AND p.Mvir > 5.e11
ORDER BY snapnum
"""<jupyter_output><empty_output><jupyter_text>Create and start the job:<jupyter_code>job = c.new_job({'query': query, 'queue': 'long'})
if job.phase[0] != "PENDING":
    print "ERROR: not in pending phase!"
else:
    run = c.run_job(job.job_id)
print job.phase[0]<jupyter_output><empty_output><jupyter_text>Check the status and wait until it is finished (this can take a couple of minutes!!):<jupyter_code>job = c.get_job(run.job_id, '60', 'QUEUED')
print "Time out or job is not in QUEUED phase anymore."
job = c.get_job(run.job_id, '60', 'EXECUTING')
print "Time out or job is not in EXECUTING phase anymore."
print "Job phase: ", job.phase[0]
print job<jupyter_output><empty_output><jupyter_text>Retrieve the results:<jupyter_code>fileurl = str(job.results[1].reference)
resultfilename = "result.xml"
success = c.connection.download_file(fileurl, username, password, file_name=resultfilename)
if not success:
    print "File could not be downloaded, please check the job phase and result urls."
else:
    print "File downloaded successfully."<jupyter_output><empty_output><jupyter_text>Plot the positions of the progenitor halos, colored by snapshot number (increasing time):<jupyter_code>table = parse_single_table(resultfilename, pedantic=False)
data = table.array

field = table.get_field_by_id('x')
unit_x = field.unit
field = table.get_field_by_id('y')
unit_y = field.unit
field = table.get_field_by_id('z')
unit_z = field.unit

%matplotlib inline
ax = plt.subplot(111)

# set axis labels
ax.set_xlabel('x [' + str(unit_x) + ']')
ax.set_ylabel('y [' + str(unit_y) + ']')

# plot data,
# color by snapnum, i.e. snapshot number, i.e. increasing time
cm = plt.cm.get_cmap('viridis')
sc = ax.scatter(data['x'], data['y'], s=0.7, c=data['snapnum'], alpha=0.5, cmap=cm)
plt.colorbar(sc)
plt.show()<jupyter_output><empty_output><jupyter_text>Now use an interactive 3D plot instead:<jupyter_code># Make an interactive 3D plot
%matplotlib notebook
fig = plt.figure()

ax = fig.add_subplot(111, projection='3d')

# set axis labels
ax.set_xlabel('x [' + str(unit_x) + ']')
ax.set_ylabel('y [' + str(unit_y) + ']')
ax.set_zlabel('z [' + str(unit_z) + ']')

# plot the data
cm = plt.cm.get_cmap('plasma')

ax.scatter(data['x'], data['y'], data['z'], s=0.7, c=data['snapnum'], depthshade=True, cmap=cm)
plt.show()<jupyter_output><empty_output><jupyter_text>Delete your job on the server:<jupyter_code>deleted = c.delete_job(job.job_id)
deleted<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10961884044263691, 'fraction_numerical': 0.006904379078785586, 'mean_word_length': 5.938320209973753, 'pattern_counts': {'":': 3, '<': 119, '<?xml version=': 0, '>': 121, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '443332', 'license': 'Apache-2.0', 'n_tokens_mistral': 3586, 'n_tokens_neox': 3503, 'n_words': 1090, 'path': 'cosmosim-uws-intro.ipynb', 'repo_name': 'kristinriebe/cosmosim-uws-notebook', 'size': '10564'}