<jupyter_start><jupyter_text>Registry search and data collection organizationNote on Discovering Data Collections (<NAME> and <NAME>, 2018)http://www.ivoa.net/documents/discovercollections/20190322/PEN-discovercollections-1.1-20180322.htmlGist: In order to deal with the complex ecosystem of data collections and services that has emerged over the years from what was originally a 1-1 association of catalogs and cone searches, some normalization of resources needs done.Listing every table of every schema in TAP services for SEO makes for unweildy records. (To the point that the ST registry had to change timeouts and *string lengths* in databases to handle it.)Listing individual catalogs as resources with a reference back to the service helps."Auxiliary capabilities" is a way to do this. They don't seem to currently work with pyvo searching? I could be doing this wrong. First, with code from http://localhost:8888/tree/Documents/GitHub/aas_workshop_2020_winter<jupyter_code>import numpy as np
import astropy.units as u
## There are a number of relatively unimportant warnings that 
## show up, so for now, suppress them:
import warnings
warnings.filterwarnings("ignore", module="astropy.io.votable.*")
warnings.filterwarnings("ignore", module="pyvo.utils.xml.*")

## For simple astropy tables
import astropy, io

## For handling ordinary astropy Tables
from astropy.table import Table

## For handling VO table type objects
from astropy.io import votable as apvot

## Generic VO access routines
import pyvo as vo
heasarc_tap_services=vo.regsearch(servicetype='tap',keywords=['heasarc'])
heasarc_tap_services

heasarc_tables=heasarc_tap_services[0].service.tables
for tablename in heasarc_tables.keys():
    if "zcat" in tablename:  
        heasarc_tables[tablename].describe()
        print("Columns={}".format(sorted([k.name for k in heasarc_tables[tablename].columns ])))
        print("----")<jupyter_output>abellzcat
    Abell Clusters Measured Redshifts Catalog

Columns=['"__row"', '"__x_ra_dec"', '"__y_ra_dec"', '"__z_ra_dec"', 'abell_distance_class', 'abell_radius', 'abell_richness_class', 'bautz_morgan_class', 'bautz_morgan_flag', 'bii', 'class', 'dec', 'lii', 'log_redshift_ratio', 'mag_10', 'name', 'ra', 'redshift', 'redshift_flag', 'ref_redshift']
----
romabzcat
    Roma-BZCAT Multi-Frequency Catalog of Blazars

Columns=['"__row"', '"__x_ra_dec"', '"__y_ra_dec"', '"__z_ra_dec"', 'bii', 'dec', 'flux_143_ghz', 'flux_1p4_ghz', 'gr_photon_flux', 'lii', 'name', 'object_type', 'ra', 'redshift', 'redshift_flag', 'rmag', 'ro_spectral_index', 'source_number', 'xray_flux']
----
zcat
    CfA Redshift Catalog (June 1995 Version)

Columns=['"__row"', '"__x_ra_dec"', '"__y_ra_dec"', '"__z_ra_dec"', '"distance"', 'bar_type', 'bii', 'bmag', 'bt_mag', 'class', 'comments', 'dec', 'diameter_1', 'diameter_2', 'lii', 'luminosity_class', 'morph_type', 'name', 'notes', 'ra', 'radial_velocity', 'radial_vel[...]<jupyter_text>Now with pyvo.registry.regtap, and its connecting *ivoid2service* function<jupyter_code>regtap_search_results = vo.registry.regtap.search(keywords=['redshift'])
regtap_search_results[0].ivoid
related_service = vo.registry.regtap.ivoid2service(regtap_search_results[0].ivoid, servicetype="tap")
related_service
related_tables = related_service.tables
for tablename in related_tables.keys():
    related_tables[tablename].describe()
    print("Columns={}".format(sorted([k.name for k in related_tables[tablename].columns ])))
    print("----")<jupyter_output>bolshoi.dens256
    Densities for 256^3 grid cells; for some snapnums

Columns=['dens', 'ix', 'iy', 'iz', 'phkey', 'snapnum', 'webid']
----
bolshoi.fof
    Friends-of-Friends halo catalogue, level 0 with relative linking
    length 0.17

Columns=['angmom', 'angmom_x', 'angmom_y', 'angmom_z', 'axis1', 'axis1_x', 'axis1_y', 'axis1_z', 'axis2', 'axis2_x', 'axis2_y', 'axis2_z', 'axis3', 'axis3_x', 'axis3_y', 'axis3_z', 'delta', 'disp', 'disp_v', 'fofid', 'ix', 'iy', 'iz', 'level', 'mass', 'ninfile', 'np', 'phkey', 'size', 'snapnum', 'spin', 'vx', 'vy', 'vz', 'x', 'y', 'z']
----
bolshoi.redshifts
    Redshifts for snapshot numbers

Columns=['aexp', 'snapnum', 'zred']
----
glots.services
    A table of TAP services harvested from the registry (and some
    spoon-fed).

Columns=['accessurl', 'harvestinterval', 'ivoid', 'lastsuccess', 'nextharvest']
----
glots.tables
    A table of tables accesible through the TAP services known to
    glots.services.

Columns=['ivoid', 'table_desc', 'table_na[...]<jupyter_text>But why? If we want to organize our resources to normalize table/catalog metadata out from the same record as a service that serves MANY of them (and keep the service record itself shorter, by way of adding one level of indirection).We know VizieR has resources set up this way.Let's see what ivoid2service does: https://pyvo.readthedocs.io/en/latest/_modules/pyvo/registry/regtap.htmlivoid2service<jupyter_code>#regtap_search_results = vo.registry.regtap.search(keywords=['vizier'], servicetype='tap')
regtap_search_results = vo.registry.regtap.search(keywords=['vizier'])
for resource in regtap_search_results:
    if 'ivo://cds.vizier' in resource.ivoid:
        print(resource.ivoid)
related_services = vo.registry.regtap.ivoid2service('ivo://CDS.VizieR/J/A+AS/106/373')
related_services<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14183998501030542, 'fraction_numerical': 0.015364436949597151, 'mean_word_length': 5.977777777777778, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '656138', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1906, 'n_tokens_neox': 1793, 'n_words': 485, 'path': 'notebook_regtap.ipynb', 'repo_name': 'theresadower/workshop-dev', 'size': '5353'}