<jupyter_start><jupyter_text>Compare spex with desi_extract_spectra This was run at jupyter.nersc.gov on a cori shared cpu node with the DESI master kernel.Generate sample extraction files for comparison on NERSC: ```- Setup environementsource /global/cfs/cdirs/desi/software/desi_environment.sh mastercd gpu_specterexport PATH=$(pwd)/bin:$PATHexport PYTHONPATH=$(pwd)/py:$PYTHONPATH- Get an interactive nodesalloc -N 1 -C haswell -t 60 -q interactivebasedir=/global/cfs/cdirs/desi/spectro/redux/daily/- spextime srun -n 32 -c 2 spex --mpi -w 6000.0,6400.0,0.8 \    -i $basedir/preproc/20200219/00051060/preproc-r0-00051060.fits \    -p $basedir/exposures/20200219/00051060/psf-r0-00051060.fits \    -o $SCRATCH/spex.fits- desi_extract_spectratime srun -n 32 -c 2 desi_extract_spectra --mpi -w 6000.0,6400.0,0.8 \    -i $basedir/preproc/20200219/00051060/preproc-r0-00051060.fits \    -p $basedir/exposures/20200219/00051060/psf-r0-00051060.fits \    -o $SCRATCH/desi_extract.fits    ```<jupyter_code>import os

import numpy as np
import matplotlib.pyplot as plt

import fitsio<jupyter_output><empty_output><jupyter_text>Load extraction samples<jupyter_code>spex_filename = os.path.join(os.environ.get("SCRATCH"), 'spex.fits')
desi_filename = os.path.join(os.environ.get("SCRATCH"), 'desi_extract.fits')
# TODO: use desi machinery for loading extraction output?
def read_extract(filename):
    with fitsio.FITS(filename) as fx:
        flux = fx['FLUX'].read().astype('f8')
        ivar = fx['IVAR'].read().astype('f8')
        wave = fx['WAVELENGTH'].read().astype('f8')
    extract = dict(
        flux=flux,
        ivar=ivar,
        wave=wave,
    )
    return extract
spex = read_extract(spex_filename)
desi = read_extract(desi_filename)<jupyter_output><empty_output><jupyter_text>Compare samples  Compare basics<jupyter_code>spex['wave'].shape, desi['wave'].shape
spex['flux'].shape, desi['flux'].shape
spex['ivar'].shape, desi['ivar'].shape
np.all(spex['wave'] == desi['wave'])<jupyter_output><empty_output><jupyter_text>Compare flux<jupyter_code>dw = 0.8
nwavestep = 50
nspec = spex['flux'].shape[0]

wave = spex['wave']
wmin, wmax = wave[0], wave[-1]
nwave = len(wave)
fig, ax = plt.subplots(figsize=(12, 12))
extent = [wmin, wmax, nspec, 0]
im = ax.imshow(
    np.isclose(desi['flux'], spex['flux']), 
    extent=extent,
    interpolation='none',
)
# Draw red vertical lines at patch wavelength boundary
ax.vlines(np.arange(wmin, wmax, nwavestep*dw), 0, nspec, color='red', lw=3)
ax.set_ylabel('ispec')
ax.set_xlabel('wave')
ax.set_title('np.isclose(desi, spex)')
;<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15195702225633154, 'fraction_numerical': 0.05755947812739831, 'mean_word_length': 4.884875846501129, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '352466', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1133, 'n_tokens_neox': 1019, 'n_words': 187, 'path': 'nb/CompareFullExtraction.ipynb', 'repo_name': 'sbailey/gpu_specter', 'size': '2606'}