<jupyter_start><jupyter_text>How to use the datasourceFirst step is to create the datasource, which will load1. the memory-mapped genotype files (takes little memory)2. the phenotypes, with balanced case/control and with number of samples = `max_samples`<jupyter_code>test_datasource = chromosome_datasource(chromosomes = list(range(1, 23)), snp_filters= [snp_filter('I83', snp_filter.SORT_PVALUE, threshold= 1e-6)], max_samples=100_000, balance_pheno="I83")
test_datasource.get_simulated_data(slice(1, 3), split = "train", snp_pair = ["6:134911816_G_A", "6:26118570_T_C"])
test_datasource.snp_full_info
assert set(test_datasource.data_dict["test"].index) & set(test_datasource.data_dict["train"].index) == set()
test_datasource_not_upsampled.get_data(sample_slice = slice(0, None), split = "train_tune")[1]<jupyter_output><empty_output><jupyter_text>Since the data is sampled with replacement, the phenotype dataframe will have duplicated samples<jupyter_code>test_datasource.pheno_df_ordered.index.nunique() #how many unique samples in 100k samples?
test_datasource.pheno_df_ordered.apply(pd.Series.value_counts)<jupyter_output><empty_output><jupyter_text>The `datasource` object stores the SNPs on `chromosome 21` and `chromosome 22` that are associated with both of the traits with low pvalues<jupyter_code>test_datasource.snp_full_info[["full_id", "chr", "position", "trait", "pval"]]<jupyter_output><empty_output><jupyter_text>We now get the genotype data matrix by reading the genotype files. The `get_data` function takes a `slice` object, and returns all genotypes at the 80 SNPs above for the samples in that slice. Here we load `80 snps for the first 1000 samples` (row 1 - 1000 of `test_datasource.pheno_df_ordered`)<jupyter_code>data = test_datasource.get_data(slice(0,1000))
data.shape<jupyter_output>reading -- time=0:00:00.00, thread 1 of 11, part 1 of 1
reading -- time=0:00:00.03, thread 1 of 48, part 2 of 2
{'fraction_non_alphanumeric': 0.10643821391484942, 'fraction_numerical': 0.046209761163032194, 'mean_word_length': 7.719457013574661, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '532175', 'license': 'Apache-2.0', 'n_tokens_mistral': 741, 'n_tokens_neox': 643, 'n_words': 175, 'path': 'notebooks/package/high_level.ipynb', 'repo_name': 'hoangthienan95/combinatorial_GWAS', 'size': '1926'}