<jupyter_start><jupyter_text>Using Threshold - Based Source Detection and Confusion MatrixThis notebook provides an example of how to run the astropy - based source detection on a fits file. This also demonstrates how to generate a confusion matrix based on the results of the source detection.<jupyter_code>import astropy
from astropy.io import fits
from astropy.coordinates import SkyCoord
from astropy.wcs import WCS
from astropy.wcs.utils import skycoord_to_pixel
from astropy.stats import sigma_clipped_stats,gaussian_fwhm_to_sigma,gaussian_sigma_to_fwhm
from astropy.table import Table,Column,Row,vstack,setdiff,join
from astropy.nddata import Cutout2D,NDData
import astropy.units as u
import os 
import matplotlib.pyplot as plt
from astropy.visualization import ZScaleInterval,simple_norm
from astropy.wcs.utils import skycoord_to_pixel, pixel_to_skycoord

import photutils
from photutils.datasets import make_gaussian_sources_image

import numpy as np
import itertools
import copy

import matplotlib
from astropy.visualization import ZScaleInterval,simple_norm
zscale = ZScaleInterval()
import pickle
import numpy as np
import diffimageml
from diffimageml import fakeplanting, util<jupyter_output><empty_output><jupyter_text>Make a FitsImage class object with the desired image<jupyter_code>example_file = "../test_data/sky_image_1.fits.fz"
FitsImageExample = diffimageml.FitsImage(example_file)<jupyter_output><empty_output><jupyter_text>Run source detection<jupyter_code>sources = FitsImageExample.detect_sources()
print (sources[0])<jupyter_output>WARNING: FITSFixedWarning: 'obsfix' made the change 'Set OBSGEO-L to   149.070647 from OBSGEO-[XYZ].
Set OBSGEO-B to   -31.272933 from OBSGEO-[XYZ].
Set OBSGEO-H to     1164.993 from OBSGEO-[XYZ]'. [astropy.wcs.wcs]<jupyter_text>Confusion Matrix example   The confusion matrix requires a fakeplanter object, which needs to have images with fakes planted.<jupyter_code>#Load examle with fakes
diffim = "../test_data/diff_pydia_1_fakegrid.fits"
search_im = "../test_data/sky_image_1.fits.fz"
fake_sn = diffimageml.FakePlanter(diffim , searchim_fitsfilename = search_im)
matrix = fake_sn.confusion_matrix()<jupyter_output>Gaia catalog ../test_data/sky_image_1_GaiaCat.ecsv exists. 
Reading without fetching.<jupyter_text>The output matrix is a list of astropy tables containing the locations and magnitudes for the sources in the given categories. To simply display the resulting 2x2 matrix, use the following.<jupyter_code>print (len(matrix[0]) , len(matrix[1]))
print (len(matrix[2]) , 0) ##No true negatives<jupyter_output>440 1
588 0<jupyter_text>In this case we recover 440 True positives, one False negative and 588 false positives. We can also make a confusion matrix that imposes magnitude limits. For this example, the fake sources have magnitudes around 16.5, so if we limit the magnitudes to be between 18 and 22, we filter out all the true positives.<jupyter_code>matrix = fake_sn.confusion_matrix(low_mag_lim = 18 , high_mag_lim = 22)
print (len(matrix[0]) , len(matrix[1]))
print (len(matrix[2]) , 0) ##No true negatives<jupyter_output>0 0
1 0
{'fraction_non_alphanumeric': 0.07099261162865403, 'fraction_numerical': 0.022165114037905557, 'mean_word_length': 6.467625899280575, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '19727', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1045, 'n_tokens_neox': 984, 'n_words': 312, 'path': 'diffimageml/examples/example_source_detection.ipynb', 'repo_name': 'jpierel14/diffimageml', 'size': '3113'}