<jupyter_start><jupyter_text>DKRZ data ingest information handlingThe submission_forms package provides a collection of components to support the management of information related to data ingest related activities (data transport, data checking, data publication and data archival):* data submission related information management  * who, when, what, for which project, data characteristics* data management related information collection  * ingest, quality assurance, publication, archiving  The information is stored in structured json files which are 1-to-1 mapped to Form objects to simplify information handling. In the following it is assumed that an initial structured json file was generated. For the different ways to generate initial structured json files see the **Workflow_Form_Generation.ipynb** notebook:    DKRZ ingest workflow systemApproach:* Data managment related information is managed in structured json files* To simplify interactive information updates etc. json files are converted to Form objects* There are multiple possibilities to populate the json files (and associated Form objects):  * DKRZ served jupyter notebooks (e.g. in DKRZ jupyterhub http://data-forms.dkrz.de:8080)  * Client side jupyter notebooks (submission via email, rt ticket, git commit)  * Client side excel sheets (submission via email, rt ticket)  * Unstructured email exchange (json population done by data managers)* A toolset to manage Form objects (specially structured json files) along a well defined workflow* A toolset to search and intercorrelate data submission information* Support for W3C prov standard exposure of the structured json files  1) Get a Form object for information stored in a json file<jupyter_code>## the following libraries are needed to interact with 
## json based form submissions

from dkrz_forms import form_handler, utils, checks,wflow_handler
from datetime import datetime
## info_file = "path to json file"
info_file = "../Forms/../xxx.json"

# load json file and convert to Form object for simple updating
my_form = utils.load_workflow_form(info_file)
# use "tab" completion to view the attributes
# every form has a project and has the above workflow steps associated
my_form.
# evalulate to see doc string of submission part
?my_form<jupyter_output><empty_output><jupyter_text>2) Explore the structure of a workflow Form object      (i.e submission workflow json file)      The workflow is structured according to the following workfow steps:* 'sub': data **submission** related information (client side: who, what, how, .., manager side: who, status,.. )* 'rev': data submission **review** information* 'ing': data **ingest** related information* 'qua': data **quality assurance** related information* 'pub': data **publication** related information* 'lta': data **long term archival** and data citation related information    information on the form objects can be retrieved interactively in ipython in jupyter notebooks - use again "tab" for completion and ? to retrievedocstring documentation. Examples:<jupyter_code># evaluate to view associated documentation string
?my_form.sub
# use "tab" completion
my_form.sub.<jupyter_output><empty_output><jupyter_text>each workflow step is structured acording to:* **agent:** step related person or software tool information* **activity**: step execution related information* **entity_in**: input information for this workflow step* **entity_out**: output information for this workflow stepthese parts have to be filled for each workflow step to characterize who (**agent**), did what (**activity**) using which input information (**entity_in**) to produce which output information (**entity_out**). These parts align with the WC3 Prov model allowing for a translation of all collected information based on the W3C prov standard (see the provenance.ipynb notebook for an example).<jupyter_code># example: "tab" completion to view attributes of agent 
# thus - agent has an email, first_name and last_name

my_form.sub.agent.<jupyter_output><empty_output><jupyter_text>agent related informationthis is generally defined in the *dkrz_forms.config.workflow_steps.py* templates (see source code on github: https://github.com/IS-ENES/submission_forms/dkrz_forms/config/workflow_steps.py)for example the agent responsible for data submission this is SUBMISSION_AGENT, which is defined as:    SUBMISSION_AGENT = {    '__doc__': """Attributes characterizing the person responsible for form completion and submission:       - last_name: Last name of the person responsible for the submission form content       - first_name: Corresponding first name       - email: Valid user email address: all follow up activities will use this email to contact end user       - keyword : user provided key word to remember and separate submission              """,    'i_name': 'submission_agent',    'last_name' : 'mandatory',    'first_name' : 'mandatory',    'keyword': 'mandatory',    'email': 'mandatory',    'responsible_person':'mandatory'  }All entries charactized as 'mandatory' have to be filled.<jupyter_code># e.g. set email of person responsible for data submission:
my_form.sub.agent.email = '<EMAIL>'<jupyter_output><empty_output><jupyter_text>activity related informationagain the generic definition is defined in the dkrz_forms.workflow_steps.py templates. for example the quality assurance (qua) related activity information is defined as:    QUA_ACTIVITY= {    '__doc__': """        Attributes characterizing the data quality assurance activity:        - status: status information        - start_time, end_time: data ingest timing information        - comment : free text comment        - ticket_id: related RT ticket number        - follow_up_ticket: in case new data has to be provided        - quality_report: dictionary with quality related information (tbd.)        """,      'i_name':'qua_activity',      'status':ACTIVITY_STATUS,      'error_status':ERROR_STATUS,      'qua_tool_version':"mandatory",      "start_time":"mandatory",      "end_time":"optional",      "comment":"optional",      "ticket_id": "mandatory",      "follow_up_ticket": 'optional',  qa feedback to users, follow up actions      }<jupyter_code>## back to example: submission related activity information
import pprint
pprint.pprint(my_form.sub.activity.__doc__)<jupyter_output><empty_output><jupyter_text>workflow step report documentseach workflow step produces an output associated to the **entity_out** keyword.To each output a user defined dictionary can be attached as **report** so e.g.   my_form.sub.entity_out.report contains all the user input provided e.g. by mail or in a excel   sheet or provided via a (jupyter notebook) form       my_form.qua.entity_out.report contains the quality_assurance tool json output as dictionary       etc.<jupyter_code># view the submission related information provided by the end user:

pprint.pprint(my_form.sub.entity_out.report.__dict__)
## Example for the quality assurance workflow step (qua):
my_form.qua.entity_out.report = {
    "QA_conclusion": "PASS",
    "project": "CORDEX",
    "institute": "CLMcom",
    "model": "CLMcom-CCLM4-8-17-CLM3-5",
    "domain": "AUS-44",
    "driving_experiment":  [ "ICHEC-EC-EARTH"],
    "experiment": [ "history", "rcp45", "rcp85"],
    "ensemble_member": [ "r12i1p1" ],
    "frequency": [ "day", "mon", "sem" ],
    "annotation":
    [
        {
            "scope": ["mon", "sem"],
            "variable": [ "tasmax", "tasmin", "sfcWindmax" ],
            "caption": "attribute <variable>:cell_methods for climatologies requires <time>:climatology instead of time_bnds",
            "comment": "due to the format of the data, climatology is equivalent to time_bnds",
            "severity": "note"
        }
    ]
}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08532204259686939, 'fraction_numerical': 0.003464203233256351, 'mean_word_length': 4.727406318883174, 'pattern_counts': {'":': 20, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '539494', 'license': 'Apache-2.0', 'n_tokens_mistral': 2078, 'n_tokens_neox': 2011, 'n_words': 852, 'path': 'dkrz_forms/Templates/Forms/Doc/Workflow_Form_Overview.ipynb', 'repo_name': 'IS-ENES-Data/submission_forms', 'size': '7810'}