<jupyter_start><jupyter_text>Covariate data preprocessing1. **This tutorial are for the factor analysis and PCA modules of the following flowchart**2. **Each commands in this tutorials should be run once per theme**![captain](../../images/Data_proc_flowchart.png)  Factor analysis1. Factor analysis can be done using one of the following method:PEER & APEX (BiCV).2. Both factor analysis method can run with or without covariate file input.3. No need to run both.  PEERFirst command is an example for where there is no more external covariate file to be input. ( i.e. the input molecular phenotype is already an residual phenotype )Input:1. a molecular phenotype file in bed format2. (optionally) a covariate file, with each rows a covariate, first column is id, and each subsquent column a sample<jupyter_code>nohup sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/data_preprocessing/covariate/PEER_factor.ipynb PEER \
        --molecular_pheno  /mnt/mfs/statgen/xqtl_workflow_testing/demo/annotation/AC.mol_phe.annotated.bed.gz \
        --wd ./ \
        --container /mnt/mfs/statgen/containers/xqtl_pipeline_sif/PEER.sif \
        --name "demo" &
nohup sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/data_preprocessing/covariate/PEER_factor.ipynb PEER \
        --molecular_pheno  /mnt/mfs/statgen/xqtl_workflow_testing/demo/PEER/Example_data/example_data.bed.gz \
        --covariate_file /mnt/mfs/statgen/xqtl_workflow_testing/demo/PEER/Example_data/example_cov.txt \
        --wd ./ \
        --container /mnt/mfs/statgen/containers/xqtl_pipeline_sif/PEER.sif \
        --name "demo" &<jupyter_output><empty_output><jupyter_text>APEXFirst command is an example for where there is no more external covariate file to be input. ( i.e. the input molecular phenotype is already an residual phenotype )<jupyter_code>nohup sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/data_preprocessing/covariate/BiCV_factor.ipynb APEX \
        --molecular_pheno  /mnt/mfs/statgen/xqtl_workflow_testing/demo/annotation/AC.mol_phe.annotated.bed.gz \
        --genotype_list  /mnt/mfs/statgen/xqtl_workflow_testing/demo/genotype_reformmating/demo_vcf_geno/demo.vcf_geno_list.txt \
        --covariate /mnt/mfs/statgen/xqtl_workflow_testing/success_example/testing_no_cov/Data_Processing/Factor_and_Covariate/AC.APEX.cov \
        --wd ./ \
        --container_apex /mnt/mfs/statgen/containers/xqtl_pipeline_sif/apex.sif \
        --name "demo" &
nohup sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/data_preprocessing/covariate/BiCV_factor.ipynb APEX \
        --molecular_pheno  /mnt/mfs/statgen/xqtl_workflow_testing/demo/annotation/AC.mol_phe.annotated.bed.gz \
        --genotype_list  /mnt/mfs/statgen/xqtl_workflow_testing/demo/genotype_reformmating/demo_vcf_geno/demo.vcf_geno_list.txt \
        --wd ./ \
        --container_apex /mnt/mfs/statgen/containers/xqtl_pipeline_sif/apex.sif \
        --name "demo" &<jupyter_output><empty_output><jupyter_text>PCAAfter the factor analysis of the previouse module, the residual expression shall be generated as outlined in the phenotype processing tutorials, and then used for the follwoing PCA analysis<jupyter_code>nohup sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/data_preprocessing/covariate/PCA.ipynb pca \
            --cwd /mnt/mfs/statgen/xqtl_workflow_testing/demo/PCA/ \
            --container_lmm "/mnt/mfs/statgen/containers/xqtl_pipeline_sif/flashpcaR.sif" \
            --name "demo" \
            --unrelated_genotype /mnt/mfs/statgen/xqtl_workflow_testing/demo/GWAS_QC/merge.mergrd.unrelated.bed \
            --related_genotype /mnt/mfs/statgen/xqtl_workflow_testing//demo/GWAS_QC/merge.mergrd.related.bed \
            --phenoFile /mnt/mfs/statgen/xqtl_workflow_testing/demo/phenotype_reformat/demo.for_pca.mol_phe.exp \
            --label_col "RACE" \
            --pop_col "RACE"  &<jupyter_output><empty_output><jupyter_text>Merge covariateThe factor_and_covariate file shall be the output from gou-29's peer and apex module. For the time being, please use the provided input<jupyter_code>nohup sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/data_preprocessing/covariate/merge_covariate.ipynb pca_factor_merge \
            --wd /mnt/mfs/statgen/xqtl_workflow_testing/demo/merge_factor/ \
            --name "demo" \
            --factor_and_covariate /mnt/mfs/statgen/xqtl_workflow_testing/success_example/testing_no_cov/Data_Processing/Factor_and_Covariate/AC.APEX.cov \
            --PC /mnt/mfs/statgen/xqtl_workflow_testing/demo/PCA/demo.for_pca.mol_phe.demo.pca.projected.rds  &<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1017467248908297, 'fraction_numerical': 0.007205240174672489, 'mean_word_length': 3.5222112537018755, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '170596', 'license': 'MIT', 'n_tokens_mistral': 1713, 'n_tokens_neox': 1646, 'n_words': 281, 'path': 'pipeline/data_preprocessing/covariate_preprocessing.ipynb', 'repo_name': 'floutt/xqtl-pipeline', 'size': '4580'}