<jupyter_start><jupyter_text>Phenotype data formattingThis is the region extraction step for data processing pipeline for xqtl workflow, containing the generation of:1. Molecular_phenotype per chrom within selected regions in the format APEX and tensorQTL takes InputThe input for this workflow is the collection of data for 1 conditions as described in the readme of this git repo1. 1 complete residual molecular_phenotype data2. 1 region_listBoth of these input can be generated by the annotation module of this pipeline OutputFor each collection, the output is 1. 1 lists of phenotype file (bed+index) for each chrom, suitable to be fed into both apex and tensorQTL, annotated with chrom and pos2. 1 lists of phenotype file (bed+index) for each gene, annotated with chrom and tss<jupyter_code>sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/data_preprocessing/phenotype/phenotype_formatting.ipynb reformat \
--region_list /home/hs3163/GIT/ADSPFG-xQTL/MWE/mwe_region_long \
--phenoFile /mnt/mfs/statgen/xqtl_workflow_testing/success_example/testing_10/Data_Processing/Phenotype/AC.mol_phe.bed  \
--cwd ./  \
--name "Dry" --container "/mnt/mfs/statgen/containers/apex.sif" &
[global]
import os
# Work directory & output directory
parameter: cwd = path
# The filename namefor output data
parameter: container = 'gaow/twas'
# An index text file with 5 columns specifying the chr, start, end and names of regions to analyze
parameter: region_list = path
# For cluster jobs, number commands to run per job
parameter: job_size = 1
# Wall clock time expected
parameter: walltime = "5h"
# Memory expected
parameter: mem = "16G"
# Number of threads
parameter: numThreads = 20
# Path to the input molecular phenotype data.
parameter: phenoFile = path
# name for the analysis output
parameter: name= f'{phenoFile:bn}'
parameter: pop_file = "None"
regions = [x.strip().split() for x in open(region_list).readlines() if x.strip() and not x.strip().startswith('#')]
# Get the unique chormosome that have regions to be analyzed.
def extract(lst):
    return [item[0] for item in lst]
chrom = list(set(extract(regions)))
# Whether the input data is named by gene_id or gene_name. By default it is gene_id, if not, please change it to gene_name
parameter: gene_name_as_phenotype_id = False<jupyter_output><empty_output><jupyter_text>Process of molecular phenotype fileThis workflow produce a bed+tabix file for all the molecular pheno data that are included in the region list to feed into downstream analysis<jupyter_code>[reformat_1,partition_by_chrom_1]
# Path to the input molecular phenotype data.
input: phenoFile ,for_each = "chrom"
output: f'{cwd:a}/{name}.{_chrom}.mol_phe.bed.gz'
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime,  mem = mem, tags = f'{step_name}_{_output:bn}'  
bash: expand = "$[ ]", stderr = f'{_output[0]}.stderr', stdout = f'{_output[0]}.stdout',container = container
    zcat $[_input] | head -1 > $[_output:n]
    tabix $[_input] $[_chrom] >> $[_output:n] 
    bgzip -f $[_output:n]
    tabix -p bed $[_output] -f
[reformat_2,partition_by_chrom_2]
# Path to the input molecular phenotype data.
input: group_by = "all"
output: f'{cwd:a}/{name}.processed_phenotype.per_chrom.recipe'
import pandas as pd
chrom_df = pd.DataFrame({"#id" : chrom ,"#dir" : _input})
chrom_df.to_csv(_output,index = 0,sep = "\t")
[partition_by_gene_1]
# Path to the input molecular phenotype data.
input: phenoFile ,for_each = "regions"
output: f'{cwd:a}/{name}.{_regions[3]}.{_regions[4]}.mol_phe.bed.gz'
task: trunk_workers = 1, trunk_size = job_size, walltime = walltime,  mem = mem, tags = f'{step_name}_{_output:bn}'  
bash: expand = "$[ ]", stderr = f'{_output[0]}.stderr', stdout = f'{_output[0]}.stdout',container = container
    zcat $[_input] | head -1 > $[_output:n]
    zcat $[_input] | grep  $[_regions[3] if gene_name_as_phenotype_id else _regions[4]] >> $[_output:n]
    bgzip -f $[_output:n]
    tabix -p bed $[_output] -f
[partition_by_gene_2]
input: group_by = "all"
output: f'{cwd:a}/{name}.processed_phenotype.per_gene.recipe'
import pandas as pd
region_df = pd.DataFrame({"#id" : [x[3] for x in regions] ,"dir" : _input})
region_df.to_csv(_output,index = 0,sep = "\t")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11675726778539353, 'fraction_numerical': 0.011581186480737414, 'mean_word_length': 5.232695139911635, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '160160', 'license': 'MIT', 'n_tokens_mistral': 1490, 'n_tokens_neox': 1398, 'n_words': 452, 'path': 'pipeline/phenotype_formatting.ipynb', 'repo_name': 'seriousbamboo/xqtl-pipeline', 'size': '4231'}