<jupyter_start><jupyter_text>Calibrating bias images The purpose of calibrating bias images is three-fold:+ Subtract overscan if you have decided your science will be better if you subtract overscan. See [this discussion of overscan](01.08-Overscan.ipynb) for some guidance. + Trim the overscan region off of the image if it is present, regardless of whether you have chosen to subtract the overscan.+ Combine the bias images into a "combined" bias to be used in calibrating the rest of the images. The purpose of combining several images is to reduce as much as possible the read noise in the combined bias.The approach in this notebook will be to reduce a single image, look at the effects the reduction step had on that image and then demonstrate how to calibrate a folder containing several images of that type.  TODO: remove the reload stuff below<jupyter_code>%load_ext autoreload
%autoreload 2
from pathlib import Path
import os

from astropy.nddata import CCDData
from astropy.visualization import hist
import ccdproc as ccdp
import matplotlib.pyplot as plt
import numpy as np

from convenience_functions import show_image

download_base_url = 'http://physics.mnstate.edu/craig/ccd-guide/'<jupyter_output><empty_output><jupyter_text>Decide where to put your calibrated images Though it is possible to overwrite your raw data with calibrated images that is a bad idea. Here we create a folder called `reduced` that will contain the calibrated data and create it if it doesn't exist.<jupyter_code>calibrated_data = Path('.', 'reduced')
calibrated_data.mkdir(exist_ok=True)<jupyter_output><empty_output><jupyter_text>Example with overscan subtraction  Data for this exampleThe data for this example can be downloaded from http://www.stsci.edu/~etollerud/python_imred_data.tar<jupyter_code>files = ccdp.ImageFileCollection('python_imred_data')
files.summary['file', 'imagetyp', 'filter', 'exptime', 'naxis1', 'naxis2']
p = Path('.')
p
os.listdir(files.location)
darks_only = ccdp.ImageFileCollection('python_imred_data/darks/')
darks_only.summary['file', 'imagetyp', 'exptime']<jupyter_output><empty_output><jupyter_text>Determine overscan regionPlease see the discussion of this camera in [the overscan notebook](01.08-Overscan.ipynbCase-1:-Cryogenically-cooled-Large-Format-Camera-(LFC)-at-Palomar) for the appropriate overscan regioin to use for this camera. Note, in particular, that it differs from the the value given in the `BIASSEC` keyword in the header of the images. The astropy affiliated package [ccdproc](https://ccdproc.readthedocs.io) provides two useful functions here:+ `subtract_overscan` for subtracting the overscan from the image, and + `trim_image` for trimming off the overscan.First, let's see what the values of `BIASSEC` which sometimes (but not always) indicates that there is is overscan and which part of the chip is the overscan, and `CCDSEC`, which is sometimes, but not always present, and indicates which part of the chip light hit.Note that neither of these are standard; sometimes, for example, `trimsec` is used instead of `ccdsec`, and there are likely other variants. Some images may have neither keyword in the header. That does not necessary indicate that ovserscan isn't present. The best advice is to carefully check the documentation for the camer you are using.<jupyter_code>files.summary['file', 'imagetyp', 'biassec', 'ccdsec', 'datasec'][0]<jupyter_output><empty_output><jupyter_text>The fits header claims the overscan extends from the 2049th column to the end of the image (this is one-based indexing) and that the part of the image exposed to light extends over all rows and from the first column to the 2048$^{th}$ column (again, this is one-indexed).There are two differences between FITS and Python in terms of indexing:+ Python indexes are zero-based (i.e. numbering starts at zero), FITS indexes are one-based (i.e. numbering starts at 1).+ The *order* of the indexes is swapped.For example, the **FITS** representation of the part of the chip exposed to light is `[1:2048,1:4128]`. To access that part of the data from a numpy array in **Python**, switch the order so that the indexing looks like this: `[0:4128, 0:2048]` (or, more compactly `[:, :2048]`). Note that the *ending* indexes given here for python are correct because the second part of a range (after the colon) is *not included* in the array slice. For example, `0:2048` starts at 0 (the first pixel) and goes up to but not including 2048, so the last pixel included is `2047` (the 2048$^{th}$ pixel). As discussed in [the overscan notebook](01.08-Overscan.ipynbCase-1:-Cryogenically-cooled-Large-Format-Camera-(LFC)-at-Palomar), the useful overscan region for this camera starts at the 2055$^th$ column, not column 2049 as indicated by the `BIASSEC` keyword in the header. This situation is not unusual; column 2049 is the first of the columns masked by the manufacturer from light but there is some leakage into this region from the rest of the CCD.If you are going to overscan you need to carefully emaine the overscan in a few representative images to understand which part of the overscan to use.In what follow, we will use for the overscan the region (Python/numpy indexing) `[:, 2055:]`.  Subtract and then trim the overscan (one sample image) Using `subtract_overscan` is reasonably straightforward, as shown in the cell below.<jupyter_code>raw_biases = files.files_filtered(include_path=True, imagetyp='BIAS')
first_bias = CCDData.read(raw_biases[0], unit='adu')
bias_overscan_subtracted = ccdp.subtract_overscan(first_bias, overscan=first_bias[:, 2055:], median=True)<jupyter_output><empty_output><jupyter_text>Next, we trim off the full overscan region (not just the part we used for subtracting overscan)<jupyter_code>trimmed_bias = ccdp.trim_image(bias_overscan_subtracted[:, :2048])
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))

show_image(first_bias.data, cmap='gray', ax=ax1, fig=fig)
ax1.set_title('Raw bias')
show_image(trimmed_bias.data, cmap='gray', ax=ax2, fig=fig)
ax2.set_title('Bias, overscan subtracted and trimmed')<jupyter_output><empty_output><jupyter_text>DiscussionVisually, the images look nearly identical before and after calibration. The only prominent difference is a shift in the pixel values, as one would expect from subtracting the same value from each pixel in an image. It simply shifts the zero point. There is ones other important difference between the images: the input image uses 32MB of memory while the calibrated, overscan-subtracted image uses roughly 128MB. The input image is stored as unsigned 16-bit integers; the calibrated image is stored as floating point numbers, which default in python to 64-bit floats. The memory size is also the size the files will have when written to disk (ignoring any compression). One can reduce the memory and disk footprint by changing the `dtype` of the image: `trimmed_bias.dtype = 'float32'`. It is best to do this just before writing the image out because arithmetic operations on the image may convert its dtype back to `float64`.  Processing a folder of bias imagesProcessing each of the bias images individually would be tedious, at best. Instead, we can use the [`ImageFileCollection`](https://ccdproc.readthedocs.io/en/latest/ccdproc/image_management.html) we created above to loop over only the bias images, saving each in the folder `calibrated_data`. In this example the files are saved uncompressed because the Python library for compressing gzip files is extremely slow.<jupyter_code>for ccd, file_name in files.ccds(imagetyp='BIAS',            # Just get the bias frames
                                 ccd_kwargs={'unit': 'adu'}, # CCDData requires a unit for the image if 
                                                             # it is not in the header
                                 return_fname=True           # Provide the file name too.
                                ):
    save_name, _ = file_name.split('.gz')
    
    # Subtract the overscan
    ccd = ccdp.subtract_overscan(ccd, overscan=first_bias[:, 2055:], median=True)
    
    # Trim the overscan
    ccd = ccdp.trim_image(ccd[:, :2048])
    
    # Save the result
    ccd.write(calibrated_data / save_name)<jupyter_output><empty_output><jupyter_text>Let's check that we really did get the images we expect by creating an [`ImageFileCollection`](https://ccdproc.readthedocs.io/en/latest/ccdproc/image_management.html) for the reduced folder and displaying the size of each image. We are expecting the images to be 2048 × 4128, and that there will be the same number of reduced bias images as input bias images (six).<jupyter_code>reduced_images = ccdp.ImageFileCollection('reduced')
reduced_images.summary['file', 'imagetyp', 'naxis1', 'naxis2']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07566276026851747, 'fraction_numerical': 0.015587666401183297, 'mean_word_length': 4.840531561461794, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '251661', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2598, 'n_tokens_neox': 2430, 'n_words': 1091, 'path': 'notebooks/02.01-Calibrating-bias-images.ipynb', 'repo_name': 'aadlander/ccd-reduction-and-photometry-guide', 'size': '8789'}