<jupyter_start><jupyter_text>Index file checkThis is a quick look at the first and last few rows in some of the index files, to check if all values seem OK.Let's look at "agn" as one example ...<jupyter_code>import numpy as np
import astropy.units as u
from gammapy.data import DataStore
ds = DataStore.from_dir('1dc/1dc/index/agn')
ds.obs_table.info()
ds.obs_table.info('stats')
ds.obs_table[:20]
ds.hdu_table.info()
ds.hdu_table.info('stats')
ds.hdu_table[:20]<jupyter_output><empty_output><jupyter_text>Check IRF valuesNow let's look at which values appear for IRF in the OBS and HDU index table ...<jupyter_code>ds = DataStore.from_dir('1dc/1dc/index/all')
irf = ds.obs_table.to_pandas()['IRF']
irf.value_counts()
irf2 = ds.hdu_table.to_pandas()
irf2 = irf2.query('HDU_TYPE == "aeff"')
irf2 = irf2['FILE_DIR'].str.split('/').apply(lambda _: _[-1])
irf2.value_counts()<jupyter_output><empty_output><jupyter_text>Check loadingTry to load all info for some of the runs ... should work for all runs.<jupyter_code>ds = DataStore.from_dir('1dc/1dc/index/all')
obs_ids = ds.obs_table['OBS_ID'].data[::500]
obs_ids
for obs_id in obs_ids:
    print('loading', obs_id)
    obs = ds.obs(obs_id)
    print('events', len(obs.events.table))
    print('aeff', obs.aeff.data.data.max())
    print('edisp', obs.edisp.data.data.max())
    print('psf', float(obs.psf.containment_radius(energy=1*u.TeV, theta=0*u.deg).deg))
    print('bkg', obs.bkg.data.data.max())
ds.info()<jupyter_output>Data store summary info:
name: noname

HDU index table:
BASE_DIR: 1dc/1dc/index/all
Rows: 48792
OBS_ID: 110000 -- 511919
HDU_TYPE: ['aeff', 'bkg', 'edisp', 'events', 'gti', 'psf']
HDU_CLASS: ['aeff_2d', 'bkg_3d', 'edisp_2d', 'events', 'gti', 'psf_3gauss']

Observation table:
Number of observations: 8132<jupyter_text>Sorted?Check if OBS_ID is sorted ...<jupyter_code>names = 'agn', 'gps', 'gc', 'egal', 'agn'
for name in names:
    print('checking', name)
    ds = DataStore.from_dir('1dc/1dc/index/' + name)
    obs_id = ds.obs_table['OBS_ID']
    print(np.all(np.diff(obs_id) > 0))
    
    obs_id = ds.hdu_table['OBS_ID']
    print(np.all(np.diff(obs_id) >= 0))<jupyter_output>checking agn
True
True
checking gps
True
True
checking gc
True
True
checking egal
True
True
checking agn
True
True
{'fraction_non_alphanumeric': 0.1568281938325991, 'fraction_numerical': 0.023348017621145373, 'mean_word_length': 5.121293800539084, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '170669', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 953, 'n_tokens_neox': 920, 'n_words': 204, 'path': 'data/index_file_check.ipynb', 'repo_name': 'gammasky/cta-dc', 'size': '2270'}