<jupyter_start><jupyter_text>Creating SED-ML and COMBINE archives from model files To make it easy for investigators to work with a broad range of model formats, modeling frameworks, simulation types, simulation algorithms, and simulation tools, BioSimulators uses the [Simulation Experiment Description Markup Language (SED-ML)](http://sed-ml.org/) and [COMBINE/OMEX archive](https://combinearchive.org/) formats. BioSimulators uses SED-ML to describe simulation experiments. This includes:* Which models to simulate* How to modify models to simulate variants such as alternative initial conditions* What type of simulations to execute (e.g., steady-state, time course)* Which algorithms to use (e.g., CVODE, SSA)* Which observables to record* How to reduce the recorded values of the observables* How to plot the observables* How to export the observables to reports (e.g., CSV, HDF5)BioSimulators uses COMBINE/OMEX archives to bundle the multiple files typically involved in modeling projects into a single archive.* Models (e.g., in CellML, SBML format)* Simulation experiments (SED-ML files)* Visualizations for visualizing simulation results (e.g., in [Vega format](https://vega.github.io/vega/))* Supplementary files, such as data used to calibrate the model* Metadata about the simulation project (RDF files that follow the [OMEX Metadata guidelines](http://co.mbine.org/standards/omex-metadata))This tutorial illustrates how to use the create COMBINE/OMEX archives from model files by illustrating the construction of a COMBINE/OMEX archive for a [flux balance model of the core metabolism of Escherichia coli](../_data/Escherichia-coli-core-metabolism.xml) encoded in SBML.  1. Create a SED-ML file and COMBINE archive online [runBioSimuations](https://run.biosimulations.org/utils/create-project) provides a simple web form for building COMBINE/OMEX archives with SED-ML files from model files (e.g., CellML, SBML). This tool support all of the modeling languages supported by BioSimulators.  2. Create a SED-ML file and COMBINE archive with the command-line program [BioSimulators-utils](https://github.com/biosimulators/Biosimulators_utils) provides a command-line application building COMBINE/OMEX archives with SED-ML files from model files. Like the online program, this tool support all of the modeling languages supported by BioSimulators. First, install [BioSimulators-utils](https://github.com/biosimulators/Biosimulators_utils). Installation instructions are available at [https://docs.biosimulators.org](https://docs.biosimulators.org/Biosimulators_utils). Note, BioSimulators-utils must be installed with the installation options for the model languages that you wish to generate simulations for. A Docker image with BioSimulators utils and all dependencies is also available ([`ghcr.io/biosimulators/biosimulators`](https://github.com/biosimulators/Biosimulators/pkgs/container/biosimulators)). Inline help for the `biosimulators-utils` command-line program is available by running the program with the `--help` option. Additional help is avilable at [https://docs.biosimulators.org](https://docs.biosimulators.org/Biosimulators_utils/).<jupyter_code>!biosimulators-utils build-project --help<jupyter_output>usage: biosimulators-utils build-project [-h]
                                         model-language model-filename
                                         simulation-type archive-filename

Build a COMBINE/OMEX archive with a SED-ML file for a model

positional arguments:
  model-language    Model language (`BNGL`, `CellML`, `RBA`, `SBML`,
                    `Smoldyn`, or `XPP`)
  model-filename    Path to model
  simulation-type   Type of simulation (`OneStep`, `SteadyState`, or
                    `UniformTimeCourse`)
  archive-filename  Path to save archive

optional arguments:
  -h, --help        show this help message and exit<jupyter_text>Next, create a temporary directory to store the created COMBINE/OMEX archive.<jupyter_code>import os
if not os.path.isdir('tmp'):
    os.mkdir('tmp')<jupyter_output><empty_output><jupyter_text>Next, use the command-line program to build a SED-ML file and COMBINE archive from a [model file](../_data/Ciliberto-J-Cell-Biol-2003-morphogenesis-checkpoint-continuous.xml).<jupyter_code>!biosimulators-utils build-project \
    SBML \
    ../_data/Ciliberto-J-Cell-Biol-2003-morphogenesis-checkpoint-continuous.xml \
    UniformTimeCourse \
    tmp/Ciliberto-J-Cell-Biol-2003-morphogenesis-checkpoint-continuous.omex<jupyter_output>/home/jonrkarr/.local/lib/python3.7/site-packages/biosimulators_utils/warnings.py:31: BioSimulatorsWarning: [33mThe SED document is potentially incorrect.
  - Model `model` may be invalid.
    - The model file `Ciliberto-J-Cell-Biol-2003-morphogenesis-checkpoint-continuous.xml` may be invalid.
      - The value of the 'sboTerm' attribute on a <species> is expected to be an SBO identifier (http://www.biomodels.net/SBO/). In SBML Level 2 prior to Version 4 it is expected to refer to a participant physical type (i.e., terms derived from SBO:0000236, "participant physical type"); in Versions 4 and above it is expected to refer to a material entity (i.e., terms derived from SBO:0000240, "material entity").
        Reference: L2V4 Section 5
         SBO term 'SBO:0000014' on the <species> is not in the appropriate branch.
        
      - The value of the 'sboTerm' attribute on a <species> is expected to be an SBO identifier (http://www.biomodels.net/SBO/). In SBML Level 2 prior to Version [...]<jupyter_text>3. Create a SED-ML file and COMBINE archive programmatically with Python These online tools are easy to use. However, they provide investigators little flexibility to customize the generation of COMBINE/OMEX archives and SED-ML files, such as to bundle simulations of multiple model files into a single COMBINE/OMEX archive. The BioSimulators-utils Python API provides additional flexibility to customize the creation of COMBINE/OMEX archives.  3.1. Use `get_parameters_variables_outputs_for_simulation` to introspect the model file<jupyter_code>from biosimulators_utils.sedml.data_model import ModelLanguage, SteadyStateSimulation
from biosimulators_utils.sedml.model_utils import get_parameters_variables_outputs_for_simulation
model_filename = '../_data/Escherichia-coli-core-metabolism.xml'
model_language = ModelLanguage.SBML
simulation_type = SteadyStateSimulation
params, simulations, vars, outputs = get_parameters_variables_outputs_for_simulation(
    model_filename, model_language, simulation_type, native_ids=True)<jupyter_output><empty_output><jupyter_text>3.2. Sort the outputs to the model into objectives and reaction fluxes<jupyter_code>obj_variables = list(filter(lambda var: var.target.startswith('/sbml:sbml/sbml:model/fbc:listOfObjectives/'), vars))
rxn_flux_variables = list(filter(lambda var: var.target.startswith('/sbml:sbml/sbml:model/sbml:listOfReactions/'), vars))<jupyter_output><empty_output><jupyter_text>3.3 Initialize a SED-ML document which describes a flux balance analysis simulation of the model<jupyter_code>from biosimulators_utils.sedml.data_model import SedDocument, Model, Task, Report
import os.path
# initialize SED document
sed_doc = SedDocument()

# create model
model = Model(
    id='model',
    source=os.path.basename(model_filename),
    language=ModelLanguage.SBML.value,
    changes=params,
)
sed_doc.models.append(model)

# attach simulations
sed_doc.simulations = simulations

# create task
task = Task(
    id='task',
    model=model,
    simulation=simulations[0],
)
sed_doc.tasks.append(task)<jupyter_output><empty_output><jupyter_text>3.4. Add reports to the model to export the objective and reaction fluxes predicted by the simulation<jupyter_code>from biosimulators_utils.sedml.data_model import Report, DataSet, DataGenerator
report = Report(
    id='objective',
    name='Objective',
)
sed_doc.outputs.append(report)
for var in obj_variables:
    var_id = var.id
    var_name = var.name

    var.id = 'variable_' + var_id
    var.name = None

    var.task = task
    data_gen = DataGenerator(
        id='data_generator_{}'.format(var_id),
        variables=[var],
        math=var.id,
    )
    sed_doc.data_generators.append(data_gen)
    report.data_sets.append(DataSet(
        id=var_id,
        label=var_id,
        name=var_name,
        data_generator=data_gen,
    ))
report = Report(
    id='reaction_fluxes',
    name='Reaction fluxes',
)
sed_doc.outputs.append(report)
for var in rxn_flux_variables:
    var_id = var.id
    var_name = var.name

    var.id = 'variable_' + var_id
    var.name = None

    var.task = task
    data_gen = DataGenerator(
        id='data_generator_{}'.format(var_id),
        variables=[var],
        math=var.id,
    )
    sed_doc.data_generators.append(data_gen)
    report.data_sets.append(DataSet(
        id=var_id,
        label=var_id,
        name=var_name if len(rxn_flux_variables) < 4000 else None,
        data_generator=data_gen,
    ))<jupyter_output><empty_output><jupyter_text>3.5. Create a temporary directory to collect the files for the COMBINE archive<jupyter_code>import os
import tempfile
if not os.path.isdir('tmp'):
    os.mkdir('tmp')
archive_dirname = tempfile.mkdtemp(dir='tmp/')<jupyter_output><empty_output><jupyter_text>3.6. Copy the model file to the temporary directory<jupyter_code>import shutil
shutil.copyfile(model_filename, os.path.join(archive_dirname, os.path.basename(model_filename)))<jupyter_output><empty_output><jupyter_text>3.7. Export the SED document to a file<jupyter_code>from biosimulators_utils.sedml.io import SedmlSimulationWriter
sedml_filename = os.path.join(archive_dirname, 'simulation.sedml')
SedmlSimulationWriter().run(sed_doc, sedml_filename)<jupyter_output>/home/jonrkarr/Documents/Biosimulators_utils/biosimulators_utils/warnings.py:31: BioSimulatorsWarning: [33mThe SED document is potentially incorrect.
  - Model `model` may be invalid.
    - The model file `Escherichia-coli-core-metabolism.xml` may be invalid.
      - In situations where a mathematical expression refers to a compartment, species or parameter, it is necessary to know the units of the object to establish unit consistency. In models where the units of an object have not been declared, libSBML does not yet have the functionality to accurately verify the consistency of the units in mathematical expressions referring to that object. 
         The units of the <compartment> 'c' cannot be fully checked. Unit consistency reported as either no errors or further unit errors related to this object may not be accurate.
        
      - If neither the attribute 'units' nor the attribute 'spatialDimensions' on a Compartment object is set, the unit associated with that compartment's s[...]<jupyter_text>3.8. Collect metadata about the model<jupyter_code>import datetime
citation_doi = '10.1186/1752-0509-7-74'
now = datetime.datetime.now()
metadata = {
    "combine_archive_uri": 'http://omex-library.org/Escherichia-coli-core-metabolism.omex',
    "uri": '.',
    'title': 'Escherichia coli core metabolism',
    'abstract': 'Flux balance analysis model of the metabolism of Escherichia coli',
    'keywords': [
        'metabolism',
        'BiGG',
    ],
    'description': None,
    'taxa': [
        {
            'uri': 'http://identifiers.org/taxonomy:83333',
            'label': 'Escherichia coli K-12',
        },
    ],
    'encodes': [
        {
            'uri': 'http://identifiers.org/GO:0008152',
            'label': 'metabolic process',
        },
    ],
    'thumbnails': [
        ''
    ],
    'sources': [],
    'predecessors': [],
    'successors': [],
    'see_also': [],
    'creators': [{
        'uri': 'https://identifiers.org/github:opencobra/cobrapy',
        'label': 'COBRApy Team',
    }],
    'contributors': [{
        'uri': 'https://identifiers.org/orcid:0000-0002-2605-5080',
        'label': '<NAME>',
    }],
    'identifiers': [
        {
            'uri': 'http://identifiers.org/bigg.model:e_coli_core',
            'label': 'bigg.model:e_coli_core',
        },
    ],
    'citations': [
        {
            'uri': 'http://identifiers.org/doi:' + citation_doi,
            'label': None,
        },
    ],
    'license': {
        'uri': 'http://bigg.ucsd.edu/license',
        'label': 'BiGG',
    },
    'funders': [],
    'created': '{}-{:02d}-{:02d}'.format(now.year, now.month, now.day),
    'modified': [
        '{}-{:02d}-{:02d}'.format(now.year, now.month, now.day),
    ],
    'other': [],
}<jupyter_output><empty_output><jupyter_text>3.9. Use `get_reference` to lookup citation information for the refernece<jupyter_code>from biosimulators_utils.ref.utils import get_reference
import Bio.Entrez
Bio.Entrez.email = '<EMAIL>'
citation = get_reference(doi=citation_doi)<jupyter_output><empty_output><jupyter_text>3.10. Add the citation to the metadata<jupyter_code>metadata['citations'][0]['label'] = citation.get_citation()<jupyter_output><empty_output><jupyter_text>3.11. Use `get_pubmed_central_open_access_graphics` to retrieve thumbnail images for the model from the [open-access subset of PubMed Central](https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/)<jupyter_code>from biosimulators_utils.ref.utils import get_pubmed_central_open_access_graphics
thumbnails = get_pubmed_central_open_access_graphics(
    citation.pubmed_central_id,
    archive_dirname,
)<jupyter_output><empty_output><jupyter_text>3.12. Add the thumbnails to the metadata<jupyter_code>metadata['thumbnails'] = [os.path.relpath(thumbnail.filename, archive_dirname) for thumbnail in thumbnails]<jupyter_output><empty_output><jupyter_text>3.13. Save metadata about the model to an RDF file<jupyter_code>from biosimulators_utils.omex_meta.io import BiosimulationsOmexMetaWriter
metadata_filename = os.path.join(archive_dirname, 'metadata.rdf')
BiosimulationsOmexMetaWriter().run([metadata], metadata_filename)<jupyter_output><empty_output><jupyter_text>3.14. Create a description of the desired COMBINE/OMEX archive<jupyter_code>from biosimulators_utils.combine.data_model import CombineArchive, CombineArchiveContent, CombineArchiveContentFormat
# initialize the archive
archive = CombineArchive()

# add the model file to the archive
archive.contents.append(CombineArchiveContent(
    location=os.path.basename(model_filename),
    format=CombineArchiveContentFormat.SBML.value,
))

# add the SED-ML file to the archive
archive.contents.append(CombineArchiveContent(
    location='simulation.sedml',
    format=CombineArchiveContentFormat.SED_ML.value,
    master=True,
))

# add the RDF metadata file to the archive
archive.contents.append(CombineArchiveContent(
    location='metadata.rdf',
    format=CombineArchiveContentFormat.OMEX_METADATA.value,
))

# add thumbnails to the archive
for thumbnail in thumbnails:
    archive.contents.append(CombineArchiveContent(
        location=os.path.relpath(thumbnail.filename, archive_dirname),
        format=CombineArchiveContentFormat.JPEG.value,
    ))<jupyter_output><empty_output><jupyter_text>3.15. Export the COMBINE/OMEX archive to a file<jupyter_code>from biosimulators_utils.combine.io import CombineArchiveWriter
_, archive_filename = tempfile.mkstemp(dir='tmp/', suffix='.omex')
CombineArchiveWriter().run(archive, archive_dirname, archive_filename)<jupyter_output><empty_output><jupyter_text>3.16. Inspect the COMBINE/OMEX archive<jupyter_code>from IPython.display import FileLink, FileLinks
FileLinks(archive_dirname)
FileLink(archive_filename)<jupyter_output><empty_output><jupyter_text>3.16. Verify the COMBINE/OMEX archive  Execute the archive with [CBMpy](https://biosimulators.org/simulators/cbmpy) and [COBRApy](https://biosimulators.org/simulators/cobrapy).<jupyter_code>from biosimulators_utils.config import Config
import biosimulators_cbmpy
import biosimulators_cobrapy
output_dirname = tempfile.mkdtemp(dir='tmp/')
config = Config(
    COLLECT_COMBINE_ARCHIVE_RESULTS=True,
    LOG=False,
)

results_cbmpy, _ = biosimulators_cbmpy.exec_sedml_docs_in_combine_archive(archive_filename, output_dirname, config=config)
results_cobrapy, _ = biosimulators_cobrapy.exec_sedml_docs_in_combine_archive(archive_filename, output_dirname, config=config)<jupyter_output>Archive contains 1 SED-ML documents with 1 models, 1 simulations, 1 tasks, 2 reports, and 0 plots:
  simulation.sedml:
    Tasks (1):
      task
    Reports (2):
      objective: 1 data sets
      reaction_fluxes: 95 data sets

Executing SED-ML file 1: simulation.sedml ...
  Found 1 tasks and 2 outputs:
    Tasks:
      `task`
    Outputs:
      `objective`
      `reaction_fluxes`
  Executing task 1: `task`
    Executing simulation ...<jupyter_text>Check that the predicted objectives are positive<jupyter_code>assert results_cbmpy['simulation.sedml']['objective']['obj'] > 0
assert results_cobrapy['simulation.sedml']['objective']['obj'] > 0<jupyter_output><empty_output><jupyter_text>Check that the simulators produced consistent objective values<jupyter_code>import numpy.testing
numpy.testing.assert_allclose(
    results_cbmpy['simulation.sedml']['objective']['obj'],
    results_cobrapy['simulation.sedml']['objective']['obj'],
)<jupyter_output><empty_output><jupyter_text>3.17. Submit model to runBioSimulations The following code can be used to submit the COMBINE/OMEX archive to runBioSimulations.<jupyter_code>from biosimulators_utils.biosimulations.utils import run_simulation_project
run_id = run_simulation_project(
    name=metadata['title'], 
    filename_or_url=archive_filename, 
    simulator='cobrapy')
print('https://run.biosimulations.org/simulations/' + run_id)<jupyter_output>https://run.biosimulations.org/simulations/61ba2c799ecb3476cd34c1d9
{'fraction_non_alphanumeric': 0.10121434622987857, 'fraction_numerical': 0.011013837898898616, 'mean_word_length': 4.658676893576223, 'pattern_counts': {'":': 2, '<': 95, '<?xml version=': 0, '>': 96, 'https://': 17, 'lorem ipsum': 0, 'www.': 3, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '267811', 'license': 'MIT', 'n_tokens_mistral': 5759, 'n_tokens_neox': 5348, 'n_words': 1397, 'path': 'tutorials/2. Creating simulations/Creating SED-ML and COMBINE archives from model files.ipynb', 'repo_name': 'biosimulators/Biosimulators_tutorials', 'size': '17728'}