<jupyter_start><jupyter_text>CRDS For DevelopersOverview========This notebook demos a basic command line submission. Setup CRDS Server and Cache=========================<jupyter_code>import os
PROJECT = 'jwst'   # Also hst
USECASE = 'bit'    # Also test, dev, cit   (ops is an exception)
os.environ["CRDS_SERVER_URL"] = "https://" + PROJECT + "-crds-" + USECASE + ".stsci.edu"
os.environ["CRDS_PATH"] = os.path.join(os.environ["HOME"], "crds_cache_" + USECASE)<jupyter_output><empty_output><jupyter_text>NOTE:  The OPS servers use an empty USECASE for CRDS_SERVER_URL, e.g. JWST OPS is https://jwst-crds.stsci.edu Setup auth.mast token==================**NOTE**:  This section illustrates setting up a MAST authentication token,  you must obtain your owntoken and replace the token shown below.Probably the best way to set your auth.mast token is to mimic Astroquery and define MAST_API_TOKEN.   If you've already defined MAST_API_TOKEN you can stop.Otherwise visit https://auth.mast.stsci.edu/token, fill in an identifying name for this token like "CRDS submit"and click the Create Token button. (If you already have a token you can skip this and use it.)Copy the resulting token string here:<jupyter_code>os.environ["MAST_API_TOKEN"] = " <KEY>"<jupyter_output><empty_output><jupyter_text>**NOTE:**  you must change the hex string above to one generated for you personally by visiting auth.mast.CRDS has two other mechanisms for conveying the auth.mast token.   You can set it in the .crds.ini file in your $HOME.   Or you can enter it in response to a prompt for a "password".  auth.mast tokens currently remain valid for 2 weeks.   Checking Configuration Status------------------------------------CRDS configuration status can be dumped out using:<jupyter_code>! crds list --status<jupyter_output>CRDS Version = '7.3.3, 7.3.3, 0739f7565259bc3e4946dbcda5e292149972b5ab'
CRDS_MODE = 'auto'
CRDS_PATH = '/Users/jmiller/crds_cache_bit'
CRDS_SERVER_URL = 'https://jwst-crds-bit.stsci.edu'
Cache Locking = 'enabled, multiprocessing'
Effective Context = 'jwst_0532.pmap'
Last Synced = '2019-06-14 11:45:18.941572'
Python Executable = '/Users/jmiller/miniconda3/envs/jwst_dev/bin/python'
Python Version = '3.7.3.final.0'
Readonly Cache = False<jupyter_text>Obtaining test data-----------------------For the sake of this demo,  you can download a demo reference from the CRDS server like this:<jupyter_code>old_path = os.environ["CRDS_PATH"]
os.environ["CRDS_PATH"] = "/tmp/crds_cache_demo"
!crds sync --output-dir=. --files jwst_miri_dark_0057.fits --organize=flat --repair-files
os.environ["CRDS_PATH"] = old_path
! crds_unique CRDSUNIQ *.fits
! crds sync --contexts jwst-edit --repair-files<jupyter_output>CRDS - INFO -  Symbolic context 'jwst-operational' resolves to 'jwst_0532.pmap'
CRDS - INFO -  Removing all context pickles.  Use --save-pickles to recreate for specified contexts.
CRDS - INFO -  Reorganizing 35 references from 'flat' to 'flat'
CRDS - INFO -  Syncing explicitly listed files.
CRDS - INFO -  0 errors
CRDS - INFO -  0 warnings
CRDS - INFO -  4 infos
('jwst_miri_dark_0057.fits', 'CRDSUNIQ', '=', '2019-06-14T11:59:48')
CRDS - INFO -  Symbolic context 'jwst-edit' resolves to 'jwst_0537.pmap'
CRDS - INFO -  Removing all context pickles.  Use --save-pickles to recreate for specified contexts.
CRDS - WARNING -  Failed listing mappings for 'jwst_0537.pmap' : CRDS jsonrpc failure 'get_mapping_names' UnavailableFile: File 'jwst_0537.pmap' is not yet available.
CRDS - INFO -  0 errors
CRDS - INFO -  1 warnings
CRDS - INFO -  2 infos<jupyter_text>Note that your output may vary depending on the content of your CRDS cache defined above with CRDS_PATH.   Once files have been synced they are generally not re-synced. CRDS Submit Programmatic API==========================Below is a basic run of the programmatic API for file submissions.   This is intended as a working examplemore than as a tutorial on the API.In the log output, >> indicates progress messages being emitted from the server in real time.The same >> messages show up on a web page for through-the-web submissions.Adding --verbose shows DEBUG log messages which show more detail, particularly web operations.  --verbosity=80 shows even more detail which can be helpful for debugging failed web interactions.The command line interface does a hand over to the web server for review and confirmation,  server actions result in STARTED, READY / BAD FILES / FAIL, and CONFIRMED / FORCED e-mails sent to the <EMAIL> mailing list (devs) and <EMAIL>.<jupyter_code># from crds import log
# log.set_verbose(80)   # 80 for debugging web activity, 0 for no debug, 50 for basic flow

from crds.submit import rc_submit
script = rc_submit.Submission(PROJECT, USECASE)
script["deliverer"] = "<NAME>"
script["other_email"] = ""
script["file_type"] = "DARK"
script["history_updated"] = True
script["descrip_updated"] = True
script["useafter_updated"] = True
script["useafter_matches"] = True
script["pedigree_updated"] = True
script["keywords_checked"] = True
script["compliance_verified"] = True
script["ingest_files"] = True
script["etc_delivery"] = False
script["jwst_etc"] = True
script["calpipe_version"] = "xxx, 1.0"
script["replacement_files"] = False
script["old_reference_files"] = "xxx"
script["replacing_badfiles"] = False
script["jira_issue"] = ""
script["table_rows_changed"] = "xxx"
script["modes_affected"] = "xxxx"
script["correctness_testing"] = "lots"
script["additional_considerations"] = "a few"
script["change_level"] = "SEVERE"
script["description"] = "something very wordy"
script["instrument"] = "MIRI"
script.add_file("jwst_miri_dark_0057.fits")
script.validate()
script.submit()<jupyter_output>2019-06-14 12:00:08,050 - CRDS - INFO -  =============================== setting up ===============================
2019-06-14 12:00:08,052 - CRDS - INFO -  Symbolic context 'jwst-edit' resolves to 'jwst_0534.pmap'
2019-06-14 12:00:08,053 - CRDS - INFO -  ########################################
2019-06-14 12:00:08,055 - CRDS - INFO -  Certifying './jwst_miri_dark_0057.fits' (1/1) as 'FITS' relative to context 'jwst_0534.pmap'
2019-06-14 12:00:09,141 - CRDS - INFO -  FITS file 'jwst_miri_dark_0057.fits' conforms to FITS standards.
2019-06-14 12:00:10,856 - CRDS - WARNING -  Missing suggested keyword 'META.MODEL_TYPE [DATAMODL]'
2019-06-14 12:00:10,969 - CRDS - INFO -  [0] DETECTOR MIRIFULONG 
2019-06-14 12:00:10,970 - CRDS - INFO -  [0] EXP_TYPE MIR_MRS 
2019-06-14 12:00:10,980 - CRDS - INFO -  META.AUTHOR [AUTHOR] = '<NAME>'
2019-06-14 12:00:10,981 - CRDS - INFO -  META.DESCRIPTION [DESCRIP] = 'MIRI Dark Correction for MIRI'
2019-06-14 12:00:10,982 - CRDS - INFO -  META.EXPOSUR[...]
{'fraction_non_alphanumeric': 0.14181111775254035, 'fraction_numerical': 0.04946204423191871, 'mean_word_length': 4.633838383838384, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 27, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '175689', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2462, 'n_tokens_neox': 2218, 'n_words': 690, 'path': 'crds/tests/simple_submission.ipynb', 'repo_name': 'stscieisenhamer/crds', 'size': '6755'}