<jupyter_start><jupyter_text>Comparision between MultiMatch and NWayMatchMultiMatch is the current default Multi-catalog matcher in the Rubin DM stack.  It performs a seriesof two-way catalog matches, updating the reference catalog as it goes.NWayMatch is a simple proof-of-concept for multi-catalog matching.Rather than do a series of 2-way matches, it clusters sources and then resolvesthe cluster membership to ensure that each input catalog only contributes at mostone source to any given "object".This example assumes that you:    1. Have downloaded the data here:  https://lsst.ncsa.illinois.edu/~yusra/nway-matcher/    2. Are running from the directory you download the data into.    3. Have run MultiMatch on those data (using multiMatch.py to generate matchCat.fits)    4. Have run NWayMatch on those data (using nway.py to generate out.fits)<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.colors as colors
import numpy as np
from astropy import table
from astropy.table import Table<jupyter_output><empty_output><jupyter_text>Read the input tables<jupyter_code>cAssoc = Table.read('out.fits', hdu=1)
oAssoc = Table.read('out.fits', hdu=2)
cStats = Table.read('out.fits', hdu=3)
oStats = Table.read('out.fits', hdu=4)
mAssoc = Table.read('matchCat.fits')
(cStats['nObject'] != 1).sum()
print(len(oStats))
print((oStats['nSrcs'] == 1).sum())
print(len(cStats))
print((cStats['nSrcs'] == 1).sum())<jupyter_output>1050857
462058
910162
367735<jupyter_text>Quick check on the input tables.Note that there are 8636020 total sources in the input catalogs.Right away we see that MultiMatch is leaving some sources unassociated.  This isactually b/c of the way that it is removing 'ambiguities', i.e., cases were wehave multiple sources from a single catalog inside the matching radius.<jupyter_code>print("Total number of sources assocaited to NWayMatch Clusters: %i" % len(cAssoc))
print("Total number of sources assocaited to NWayMatch Objects: %i" % len(oAssoc))
print("Total number of sources assocaited to MutliMatch Objects: %i" % len(mAssoc))<jupyter_output>Total number of sources assocaited to NWayMatch Clusters: 8636020
Total number of sources assocaited to NWayMatch Objects: 8636020
Total number of sources assocaited to MutliMatch Objects: 7923128<jupyter_text>Lets look at the number of sources per cluster and per object in NWayMatch<jupyter_code># This is the number of source per cluster.   The high-end tail depends on the cell size
_ = plt.hist(cStats['nSrcs'], bins=np.linspace(0.5, 90.5, 91))
plt.yscale('log')
# This is the number of sources per object.  It can not exceed the number of input catalogs
_ = plt.hist(oStats['nSrcs'], bins=np.linspace(0.5, 35.5, 36))
plt.yscale('log')<jupyter_output><empty_output><jupyter_text>Now lets compare the number of sources per object from MultiMatch.We see that NWayMatch is giving slightly more associations.  Again, this is b/c of theassociations that have been removed.<jupyter_code># This is the number of sources per object
# We are using np.bincount to count the number of times a given object id occurs in the association catalog
_ = plt.hist(oStats['nSrcs'], bins=np.linspace(0.5, 35.5, 36))
_ = plt.hist(np.bincount(mAssoc['object']), bins=np.linspace(0.5, 35.5, 36))
plt.yscale('log')<jupyter_output><empty_output><jupyter_text>Now let's make vectors that keep track of how sources are in each object    srcCountC -> Number of sources in each cluster    srcCountO -> Number of sources in each NWayMatch object    srcCountM -> Number of sources in each MultiMatch object<jupyter_code>srcCountC = np.bincount(cAssoc['object'])
srcCountM = np.bincount(mAssoc['object'])
srcCountO = np.bincount(oAssoc['object'])<jupyter_output><empty_output><jupyter_text>Now let's join the tables, using the 'id' column as the matching keyThis could take about a minute<jupyter_code>jj = table.join(oAssoc, mAssoc, join_type='outer', keys='id')
jc = table.join(cAssoc, mAssoc, join_type='outer', keys='id')<jupyter_output><empty_output><jupyter_text>Now we can get parallel arrays, with the number of sources associated to each input source    mo -> number of sources in the associated NWayMatch Object    mc -> number of sources in the associated NWayMatch Cluster    mm -> number of sources in the associated MultiMatch Object<jupyter_code>mo = srcCountO[jj['object_1'].data]
mc = srcCountC[jc['object_1'].data]
mm = np.where(jj['object_2'].data.mask, 0, srcCountM[jj['object_2'].data])
mask = mo != mm
mask2 = mc != mm<jupyter_output><empty_output><jupyter_text>Make a density plot of the number of sources in the NWayMatch cluster versus the number of associated sources in MultiMatch.   Note that the Clusters can have many more sources than the MultiMatch objects.  If the same matching radius is used, and is the same as the cell size, there should never be fewer sources in a cluster than in a MultiMatch object.<jupyter_code>norm=colors.LogNorm(vmin=0.9, vmax=1e5)
oo = plt.hist2d(mm[mask2], mc[mask2], bins=[np.linspace(-0.5, 33.5, 35), np.linspace(-0.5, 100.5, 102)], norm=norm)
cb = plt.colorbar()
_ = plt.xlabel("Associated Sources (Multi-Match)")
_ = plt.ylabel("Assocaited Sources (Clustering)")<jupyter_output><empty_output><jupyter_text>Make a density plot of the number of sources in the NWayMatch objects versus the number of associated sources in MultiMatch.   Note that feature where nSrcMultiMatch == 0.  These are sources that have had all their associated removed b/c of ambiguities.<jupyter_code>norm=colors.LogNorm(vmin=0.9, vmax=1e5)
oo = plt.hist2d(mm[mask], mo[mask], bins=[np.linspace(-0.5, 33.5, 35), np.linspace(-0.5, 33.5, 35)], norm=norm)
cb = plt.colorbar()
_ = plt.xlabel("Associated Sources (MultiMatch)")
_ = plt.ylabel("Assocaited Sources (NWayMatch)")<jupyter_output><empty_output><jupyter_text>Now let's make a histogram of the difference in the number of sourcesassociated to any given source.  (I.e,. there is one entry per source, and thex-axis value is the difference in the number of associated sources as foundby MultiMatch and NWayMatch)<jupyter_code>plt.yscale('log')
_ = plt.hist(mm-mo, bins=np.linspace(-33.5, 33.5, 68))
_ = plt.xlabel(r"$n_{\rm assoc, mm} - n_{\rm assoc, nw}$")<jupyter_output><empty_output><jupyter_text>Now let's make a histogram of the difference in the number of sourcesassociated to a cluster as compared to the number associated to a MultiMatchobject.  Really this is just telling us about the cell size and the number ofsources inside the matching radius.<jupyter_code>plt.yscale('log')
_ = plt.hist(mm[mask2]-mc[mask2], bins=np.linspace(-66.5, 33.5, 101))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10217881292261458, 'fraction_numerical': 0.024192336589030805, 'mean_word_length': 5.636091724825524, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 53, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '598524', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2266, 'n_tokens_neox': 2132, 'n_words': 743, 'path': 'nb/AssocStats.ipynb', 'repo_name': 'KIPAC/NWayMatch', 'size': '6655'}