<jupyter_start><jupyter_text>I/OThis notebook demonstrates several examples of configuring/adding backends for PLINK dataset import as well as registering custom I/O extensions.<jupyter_code>from lib import api
import pandas as pd
import xarray as xr
%run nb/paths.py
xr.set_options(display_style='html');
# Path to PLINK dataset for demonstration
path = HAPMAP_PLINK_PATH_01
path<jupyter_output><empty_output><jupyter_text>Choosing BackendsLike pandas and xarray, settings for I/O backends are configured with a global options system (in this case, they are literally using Pandas options).An advantage of this is that the options themselves are callable python objects with docstrings, which makes interactive exploration easier.  This also provides a convenient way to scope settings globally, locally with a context manager, or on a per-call basis.<jupyter_code># Config properties are live attributes (autocomplete makes them easy to find)
api.config.io.plink.backend
# Most backends will default to 'auto' so that they can be chosen based on what's installed
api.config.describe('io.plink.backend')
# Set an option globally to use pysnptools for PLINK I/O (https://github.com/MicrosoftGenomics/PySnpTools)
api.config.set('io.plink.backend', 'pysnptools')
api.config.describe('io.plink.backend')
# Use global options set above to read a file
ds = api.read_plink(path, chunks='auto', fam_sep=' ', bim_sep='\t')

# OR Define context manager for more local scoping
with api.config.context('io.plink.backend', 'pysnptools'):
    ds = api.io.read_plink(path, chunks='auto', fam_sep=' ', bim_sep='\t')
    
# OR Set backend for each call
ds = api.io.read_plink(path, chunks='auto', backend='pysnptools', fam_sep=' ', bim_sep='\t')

ds<jupyter_output><empty_output><jupyter_text>Using Custom BackendsAdding unmanaged extension mechanisms for custom backends like those used in Pandas or exposed as a part of the [Astropy I/O Registry](https://docs.astropy.org/en/stable/io/registry.htmlio-registry) can be done at runtime.  These are "unmanaged" in the sense that it is entirely up to the user to install dependencies and make them compatible.<jupyter_code># Create a custom backend that just reads PLINK pedigree data
class CustomPLINKBackend(api.io.PLINKBackend):
    
    id = 'custom-backend'
    
    def read_plink(self, path):
        cols = names=['sample_id', 'fam_id', 'pat_id', 'mat_id', 'is_female', 'phenotype']
        return pd.read_csv(path + '.fam', sep=' ', names=cols)
# Register the backend and use it
api.io.register_backend(CustomPLINKBackend())

df = api.read_plink(path, backend='custom-backend')
df<jupyter_output><empty_output><jupyter_text>ExportExport functions are thin wrappers around `xarray.Dataset.to_*` that add default filters and compression options where appropriate.  These can easily be overriden or emulated with the Xarray API instead.  Here are a few examples with Zarr:<jupyter_code># Write using Xarray directly (note the resulting size on disk)
ds.to_zarr('/tmp/ds1.zarr', mode='w')
!du -sh /tmp/ds1.zarr
# Write via the API, which will add appropriate bitpacking filters for this 
# particular dataset type and make the size on disk much smaller
api.write_zarr(ds, '/tmp/ds2.zarr', mode='w')
!du -sh /tmp/ds2.zarr
# The codecs used to do this can also be accessed directly and used 
# as Xarray write function arguments
from lib.io import codecs
ds.to_zarr('/tmp/ds3.zarr', mode='w', encoding={'data': {'filters': [codecs.PackGeneticBits()]}})
!du -sh /tmp/ds3.zarr<jupyter_output>65M	/tmp/ds3.zarr
{'fraction_non_alphanumeric': 0.09966120835686053, 'fraction_numerical': 0.003105590062111801, 'mean_word_length': 5.270796460176991, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '251217', 'license': 'Apache-2.0', 'n_tokens_mistral': 1107, 'n_tokens_neox': 1087, 'n_words': 389, 'path': 'notebooks/platform/xarray/io.ipynb', 'repo_name': 'tomwhite/gwas-analysis', 'size': '3542'}