<jupyter_start><jupyter_text>Real dark current: noise and other artifacts The dark current on modern CCDs, even relatively inexpensive ones, is verysmall. In a typical exposure the dark current is likely smaller, maybe muchsmaller, than the read noise. Despite that, it is worth examining some darkframes from your camera because not all pixels have the same dark current.Some, called hot pixels, have much higher dark current than the rest of thesensor. Some sensors have non-astronomical sources of counts whose value areproportional to exposure time, like dark current, but whose origin is nonthermaleffects.As we did in the [notebook about overscan](01-08-Overscan.ipynb), we will focus on a pair ofcameras to illustrate this point and demonstrate how to take similar images withyour camera and analyze them.<jupyter_code>from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np

from astropy.nddata import CCDData
from astropy.visualization import hist

from convenience_functions import show_image
# Use custom style for larger fonts and figures
plt.style.use('guide.mplstyle')<jupyter_output><empty_output><jupyter_text>Making an image to check for hot pixelsMeasuring the dark current in a camera requires taking images with fairly longexposure time, ideally long enough that the expected counts due to dark currentis larger than the expected noise in the image. If that isn't the case then thedark images will be measuring noise, not dark current, as we saw in the[previous section](03-01-Dark-current-The-ideal-case.ipynb).There are two ways to ensure that you are measuring dark current:+ Take some long-exposure dark frames. The exposure time should be long enoughthat the expected dark counts is at least as large as the noise expected in asingle image.+ Take several of these images and combine them to reduce the noise in theresult. The noise in the combined image is proportional to $1/\sqrt{N}$, where$N$ is the number of images that are combined. What if that isn't possible?Do your best. The darks from the Large Format Camera were taken as part of ascience run in which there was limited time for taking dark frames. The keything to keep in mind is the lower limit on the dark current you can measure,given the read noise and exposure time.The darks in Case 2 were taken with a new camera that was not yet mounted on thetelescope as part of commissioning the camera. That made taking over five hours ofdark frames more feasible!  Case 1: Cryogenically cooled  Large Format Camera (LFC) at PalomarThe images in this section are from chip 0 of the LFC at the Palomar 200-inchtelescope. Technical information about the camera is[here](http://www.astro.caltech.edu/palomar/observer/200inchResources/lfcspecs.html). The technical information says nothing aboutthe expected dark current of this camera. The expectation for cameras cooled byliquid nitrogen is that their dark current is essentially zero.What we will see with this camera is that+ not all of the pixels have negligible dark current, and+ not all of the counts in a dark image are from dark current.  Estimating exposure time and number of frames for measuring dark currentUnfortunately, the technical information for this camera does not include anyinformation about the expected dark current. For thermoelectrically-cooledcameras the dark current is no more than 0.1 $e^-$/sec/pixel, and for newercamers is more like 0.01$e^-$/sec/pixel. We will assume the smaller dark currenthere.The read noise for this camera is 11$e^-$/pixel, so the exposure time needed tojust reach the case where the dark counts is equal to the read noise is$$t_\text{dark, min} = \frac{11 e^-/\text{pixel}}{0.01 e^-/\text{sec/pixel}} = 1100~\text{sec}.$$Even that leaves us with an image dominated by read noise rather than darkcurrent. Combining 100 of those images would reduce the read noise by a factorof 10, which would be ideal for precisely measuring the dark current in eachpixel. Unfortunately, taking over 30 hours of dark images is not feasible.Instead, we will work with what we have for this camera: three dark images, each300 seconds. This places a lower limit on the dark current we can measure — wewill only be able to measure the dark current for fairly "hot" pixels — but itat least gives us some idea of the dark properties of the camera.Another way to phrase this is much more positive: for most of the pixels in theimage the dark current simply does not matter because it is completely swampedby the read noise. First, we read in the image.<jupyter_code>calibrated_images = Path('.')
combined_dark_lfc = CCDData.read(calibrated_images / 'combined_dark_300.000.fits.bz2')<jupyter_output><empty_output><jupyter_text>Examine the imageThe full image is on the left, and the lower left corner is shown on the right.The light area in the lower left corner is due to sensor glow, which is lightemitted by the electronics of the CCD. The counts in the image due to sensorglow should grow linearly with exposure time, like dark counts do, but the pixelvalues are much higher than for pixels whose counts are due to dark current.The top right panel in the figure below shows the part of the CCD near thesensor glow; pixel values are several thousand counts. The bottom right panelshows a portion of the CCD in the upper right part of the chip; pixel values are*much* lower, typically under 10.<jupyter_code>fig = plt.figure(figsize=(10, 10))
whole_im_ax = plt.subplot(121)
glow_ax = plt.subplot(222)
dark_ax = plt.subplot(224)

show_image(combined_dark_lfc, cmap='gray', ax=whole_im_ax, fig=fig)
whole_im_ax.set_title('Entire chip')

show_image(combined_dark_lfc[:200, :200], cmap='gray', ax=glow_ax, fig=fig)
glow_ax.set_title('Lower left corner of chip')

show_image(combined_dark_lfc[-200:, -200:], cmap='gray', ax=dark_ax, fig=fig)
dark_ax.set_title('Upper right corner of chip')

plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Dark current in this CCDThe histogram below shows dark current along the horizontal axis and the numberof pixels in the CCD with that dark current along the vertical axis. Since thereis obvious sensor glow in the darks, three distributions are shown:+ The full sensor, which at first glance has some pixels with dark current ashigh as 10 $e^-$/sec.+ The portion of the sensor *excluding* the sensor glow, in which the darkcurrent is, as expected, low, though there is a small fraction of "hot pixels"with dark current around 1 $e^-$/sec.+ The sensor glow region. That region is apparent in the clump of pixels withequivalent dark current of 0.5$e^-$/sec and higher.<jupyter_code>gain = 2.0
exposure_time = 300
dark_current_ln2 = gain * combined_dark_lfc.data / exposure_time

fig = plt.figure(figsize=(20, 10))
hist(dark_current_ln2.flatten(), bins=500, density=False, alpha=0.5, color='C0',
     label='Full sensor');
hist(dark_current_ln2[200:, 200:].flatten(), bins=500, density=False, alpha=0.5, color='C1',
     label='Sensor, excluding sensor glow region')
hist(dark_current_ln2[:200, :200].flatten(), bins=500, density=False, alpha=0.5, color='C9',
     label='Sensor glow region only');
#plt.semilogy()
plt.loglog()

plt.grid()
noise_limit = 2 * 11/exposure_time/np.sqrt(3)
plt.vlines(2 * 11/exposure_time/np.sqrt(3), 0.1, 8e6)
plt.ylim(0.1, 8e6)
plt.xlim(1e-3, 1e2)
x_min, x_max = plt.xlim()
y_min, y_max = plt.ylim()
noisy_region = plt.Rectangle((x_min, y_min), noise_limit - x_min, y_max - y_min, label='Noise-dominated region',
                             color='gray', alpha=0.2, hatch='/')
ax = plt.gca()
ax.add_patch(noisy_region)
plt.legend()
plt.xlabel('dark current, electrons per second')
plt.ylabel('Number of pixels with that dark current');<jupyter_output><empty_output><jupyter_text>Case 2: Thermoelectrically-cooled camera  Taking an image to measure dark current The image below is a combination of 20 dark frames, each a 1000 sec exposure.That exposure time was chosen so that the expected number of dark electrons wasat least somewhat larger than the read noise expected in a combination of 20images. As we saw in the previous notebook, unless that is the case the darkframes will measure read noise instead of dark current.This camera has a dark current of $0.01 e^-$/sec/pixel and read noise of roughly$10 e^-$/read/pixel/, so the exposure time at which the dark current is as largeas the read noise is$$t_\text{dark, min} = \frac{10 e^-/\text{pixel}}{0.01 e^-/\text{sec/pixel}} = 1000~\text{sec}.$$By combining 20 of the images, the expected noise in the combination is reducedby a factor of $\sqrt{20}$ to $2.2 e^-$/pixel. This puts us in the "low readnoise" limit in the previous notebook about ideal dark current.Measuring the dark current once you have the images is relativelydirect: subtract the bias, multiply the result by the gain, and thendivide by the exposure time to obtain the dark current in $e^-$/pixel/sec.<jupyter_code>dark_1000 = CCDData.read('combined_dark_exposure_1000.0.fit.bz2')
show_image(dark_1000, cmap='gray')<jupyter_output><empty_output><jupyter_text>Calculate the dark current for each pixelRecall that the dark current $d_c(T)$ is given by$$d_c(T) = d_e(t) / t,$$where $d_e(t)$ is the number of dark electrons in the images. That is related tothe dark counts, $n_{dark}(t)$, the image values displayed in the image above,by the gain of the camera, $d_e(t) = g n_{dark}(t)$, so that$$d_C(T) = g n_{dark}(t) / t.$$This particular camera has a gain of $g = 1.5 e^-$/ADU.<jupyter_code>gain = 1.5
read_noise = 10.0
exposure_time = 1000

dark_current = gain * dark_1000.data / exposure_time

noisy_region = 2 * read_noise / exposure_time / np.sqrt(20)

plt.figure(figsize=(20, 10))
hist(dark_current.flatten(), bins=5000, density=False, label='Dark current');

plt.ylim(0.1, 4e7)
plt.vlines(noisy_region, *plt.ylim())

plt.loglog()
plt.grid()


x_min, x_max = plt.xlim()
y_min, y_max = plt.ylim()
noisy_region = plt.Rectangle((x_min, y_min), noisy_region - x_min, y_max - y_min, label='Noise-dominated region',
                             color='gray', alpha=0.2, hatch='/')
ax = plt.gca()
ax.add_patch(noisy_region)

plt.legend()

plt.xlabel('dark current, electrons per second')
plt.ylabel('Number of pixels with that dark current');<jupyter_output><empty_output><jupyter_text>There is a large range in the dark currentWhile the vast majority of the pixels do, as expected, have very low darkcurrent, it is much higher for other pixels. These pixels, called hot pixels,can occur even in cryogenically-cooled cameras.The vast majority of the pixels in the image have a dark current around thevalue promised by the manufacturer, 0.01 $e^-$/sec; the highest nominally have adark current of 98 $e^-$/sec.However, there is  an upper limit to the dark current that can be measured in aparticular exposure because the CCD saturates once the counts become largeenough. In other words, there is an upper limit to the number of counts a CCDcan represent; for this camera the limit is $2^{16} -1$, or $65,563$ counts. Ifwe convert that to a dark current for a 1000 sec exposure, it is approximately$$\text{dark current for saturated pixel} =  \text{gain} \times \frac{\text{saturation counts}}{\text{exposure time}} =  1.5 \frac{e^-}{\text{count}}\frac{65,563~\text{count}}{1000 \text{sec}} = 98 \frac{e^-}{\text{sec}}.$$For the hottest pixels in this image, that is the *lower limit* of the darkcurrent for those pixels.The dark current is also not well estimated for pixels a bit below the maximumbecause CCDs stop responding linearly once they pass a pixel value that dependson the camera. For this camera, the linearity limit is around 55,000 counts,corresponding to a dark current of roughly  82 $e^-$/sec.You can, in principle, also check whether or not the hot pixels have a constant darkcurrent that does not change with time by creating a number of these darks atdifferent exposure times and plotting the dark current as a function of exposuretime for the hot pixels. If the dark current is constant then the dark countswill be properly removed when the dark is subtracted from the science image. Ifthe dark current is not constant, the pixel should be excluded from analysis.The fraction of very hot pixels, those above 1 $e^-$/sec, is relatively small,roughly 0.1% of the pixels.<jupyter_code>plt.figure(figsize=(20, 10))
hist(dark_current_ln2[200:, 200:].flatten(), bins=5000, density=True, alpha=0.5, label='Cryogenically cooled, excluding sensor glow');
hist(dark_current.flatten(), bins=5000, density=True, alpha=0.5, label='Thermo-electrically cooled');
plt.grid()
plt.loglog()
plt.xlabel('dark current, electrons per second')
plt.ylabel('Number of pixels with that dark current');
plt.legend();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08221331446985773, 'fraction_numerical': 0.022557572899473393, 'mean_word_length': 5.024621212121212, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '466136', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3899, 'n_tokens_neox': 3546, 'n_words': 1678, 'path': 'notebooks/03-02-Real-dark-current-noise-and-other-artifacts.ipynb', 'repo_name': 'programatt/ccd-reduction-and-photometry-guide', 'size': '12723'}