<jupyter_start><jupyter_text>Compute LD and PCA via PLINKIn this notebook, we compute linkage disequilibrium and principal components analysis using [PLINK2](https://www.cog-genomics.org/plink/2.0/).Note that this work is part of a larger project to [Demonstrate the Potential for Pooled Analysis of All of Us and UK Biobank Genomic Data](https://github.com/all-of-us/ukb-cross-analysis-demo-project). Specifically this is for the portion of the project that is the **siloed** analysis.  Setup     Cloud Environment: This notebook was written for use on the All of Us Workbench.            Use "Recommended Environment" General Analysis which creates compute type Standard VM and change the resources to 8 CPUs and 30 GB RAM.        This notebook takes about an hour to run. Recommend that it is run in the background via run_notebook_in_the_background.<jupyter_code>from datetime import datetime
import os
import time<jupyter_output><empty_output><jupyter_text>Setup plink2https://www.cog-genomics.org/plink/2.0/<jupyter_code>%%bash

##### plink 2 install
PLINK_VERSION=2.3.Alpha
PLINK_ZIP_PATH=/tmp/plink-$PLINK_VERSION.zip
curl -L -o $PLINK_ZIP_PATH https://s3.amazonaws.com/plink2-assets/alpha2/plink2_linux_x86_64.zip
mkdir -p /tmp/plink2/
unzip -o $PLINK_ZIP_PATH -d /tmp/plink2/
!/tmp/plink2/plink2 --version # --help<jupyter_output><empty_output><jupyter_text>Define constants<jupyter_code># Papermill parameters. See https://papermill.readthedocs.io/en/latest/usage-parameterize.html

#---[ Inputs ]---
# The BGEN file was created via aou_workbench_siloed_analyses/02_aou_write_filtered_bgen.ipynb.
REMOTE_BGEN = 'gs://fc-secure-9169fc87-b35f-4b52-bf83-0f8e45994708/data/aou/geno/20220315/aou-alpha3-chr1-chr22.bgen'
# The sample file was created via aou_workbench_siloed_analyses/02_aou_write_filtered_bgen.ipynb.
REMOTE_BGEN_SAMPLE = 'gs://fc-secure-9169fc87-b35f-4b52-bf83-0f8e45994708/data/aou/geno/20220315/aou-alpha3-chr1-chr22.sample'
# These two files were created via notebook aou_workbench_siloed_analyses/03_aou_variant_qc.ipynb
# NOTE: use variant QC files created for regenie step 2, not step 1, here.
REMOTE_VARIANT_QC_ID = 'gs://fc-secure-9169fc87-b35f-4b52-bf83-0f8e45994708/data/aou/variant-qc/20220316/aou_alpha3_lipids_step2QC_plink.id'
REMOTE_VARIANT_QC_SNPLIST = 'gs://fc-secure-9169fc87-b35f-4b52-bf83-0f8e45994708/data/aou/variant-qc/20220316/aou_alpha3_lipids_step2QC_plink.snplist'

#---[ Outputs ]---
# Create a timestamp for a folder of results generated today.
DATESTAMP = time.strftime('%Y%m%d')

OUTPUT_FILENAME_PREFIX = 'aou_alpha3_lipids'
OUTPUT_FOLDER = f'{os.getenv("WORKSPACE_BUCKET")}/data/aou/ld-pca/{DATESTAMP}/'
LOCAL_BGEN = os.path.basename(REMOTE_BGEN)
LOCAL_BGEN_SAMPLE = os.path.basename(REMOTE_BGEN_SAMPLE)
LOCAL_VARIANT_QC_ID = os.path.basename(REMOTE_VARIANT_QC_ID)
LOCAL_VARIANT_QC_SNPLIST = os.path.basename(REMOTE_VARIANT_QC_SNPLIST)<jupyter_output><empty_output><jupyter_text>Copy data locally<jupyter_code>!gsutil cp -n {REMOTE_BGEN} {REMOTE_BGEN_SAMPLE} .    
!gsutil cp {REMOTE_VARIANT_QC_ID} {REMOTE_VARIANT_QC_SNPLIST} .<jupyter_output><empty_output><jupyter_text>Compute linkage disequilibrium via plink2<jupyter_code>!/tmp/plink2/plink2 \
  --bgen {LOCAL_BGEN} ref-first \
  --sample {LOCAL_BGEN_SAMPLE} \
  --chr 1-22 \
  --keep {LOCAL_VARIANT_QC_ID} \
  --extract {LOCAL_VARIANT_QC_SNPLIST} \
  --indep-pairwise 200 50 0.25 \
  --out {OUTPUT_FILENAME_PREFIX}_plink_ld
%%bash

ls -lat | head
%%bash

wc -l *prune*
!gsutil -m cp {OUTPUT_FILENAME_PREFIX}_plink_ld* {OUTPUT_FOLDER}
!gsutil ls {OUTPUT_FOLDER}<jupyter_output><empty_output><jupyter_text>Compute principal components analysis via plink2    Note: the --memory parameter below assumes the machine has 30 GB of RAM. Adjust this value if the machine has more or less than 30 GB of RAM.<jupyter_code>!/tmp/plink2/plink2 \
  --bgen {LOCAL_BGEN} ref-first \
  --sample {LOCAL_BGEN_SAMPLE} \
  --chr 1-22 \
  --keep {LOCAL_VARIANT_QC_ID} \
  --extract {OUTPUT_FILENAME_PREFIX}_plink_ld.prune.in \
  --pca 15 approx \
  --memory 27500 \
  --out {OUTPUT_FILENAME_PREFIX}_plink_pca
%%bash

ls -lat | head
!gsutil -m cp {OUTPUT_FILENAME_PREFIX}_plink_pca* {OUTPUT_FOLDER}
!gsutil ls {OUTPUT_FOLDER}<jupyter_output><empty_output><jupyter_text>Provenance<jupyter_code>%%bash

date
%%bash

pip3 freeze<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10934622467771639, 'fraction_numerical': 0.0462707182320442, 'mean_word_length': 5.278901734104046, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '255214', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1873, 'n_tokens_neox': 1682, 'n_words': 331, 'path': 'aou_workbench_siloed_analyses/04_aou_plink_ld_and_pca.ipynb', 'repo_name': 'all-of-us/ukb-cross-analysis-demo-project', 'size': '4344'}