<commit_before>{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "from typing import *\n",
    "from Bio import SeqIO\n",
    "from Bio.SeqRecord import SeqRecord\n",
    "from Bio.Seq import Seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a genome index for use with loompy and cytograph\n",
    "\n",
    "WARNING: these instruction will create a genome index that is *only* suitable for use with 10x Chromium data. We create a composite index, with separate sequence fragments representing unspliced and spliced transcripts. Unspliced fragments will be generated from genomic sequence upstream of every poly-A stretch located inside of a gene locus. Spliced fragments will be generated from transcriptomic sequence upstream of every polyadenylation site in every known spliced transcript. This reflects the locations of reads generated by Chromium, but may not be suitable for other methods.\n",
    "\n",
    "The instructions below show how to build a genome index for the human genome. For other species, the code below will have to be modified to accomodate different sources and formats of metadata for those species. Instructions for mouse are available in the loompy/kallisto subdirectory.\n",
    "\n",
    "### Install prerequisites\n",
    "\n",
    "Install the [BioPython](https://biopython.org/wiki/Packages) package in your python distribution:\n",
    "\n",
    "```\n",
    "pip install biopython  # Method recommended by the BioPython people\n",
    "conda install -c conda-forge biopython  # Alternative, for Anaconda\n",
    "```\n",
    "\n",
    "Install [gawk](https://www.gnu.org/software/gawk/) (on Linux, this is typically already installed), [bedtools](https://bedtools.readthedocs.io/en/latest/), and [kallisto](https://pachterlab.github.io/kallisto/) (instructions below are for macOS using [Brew](https://brew.s)):\n",
    "```\n",
    "brew install gawk\n",
    "brew install bedtools\n",
    "brew install kallisto\n",
    "```\n",
    "\n",
    "### Download genome data\n",
    "\n",
    "#### Download transcript sequences and metadata from [Gencode](https://www.gencodegenes.org/human/)\n",
    "\n",
    "Get the \"Genome sequence, primary assembly (GRCh38)” file named [GRCh38.primary_assembly.genome.fa.gz](ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_31/GRCh38.primary_assembly.genome.fa.gz)\n",
    "\n",
    "Download the comprehensive PRI gene annotation on the primary assembly GTF file named [gencode.v31.primary_assembly.annotation.gtf](ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_31/gencode.v31.primary_assembly.annotation.gtf.gz)\n",
    "\n",
    "#### Download additional gene metadata from [HGNC](https://www.genenames.org/download/statistics-and-files/)\n",
    "\n",
    "Get the “Complete HGNC dataset” as TXT, a file named [hgnc_complete_set.txt](ftp://ftp.ebi.ac.uk/pub/databases/genenames/new/json/hgnc_complete_set.txt)\n",
    "\n",
    "#### Download 10x Chromium barcode whitelists\n",
    "\n",
    "Chromium cell barcodes are generated from a set of known sequences, which differ by version of the Chromium kits. Get three files from the [cellranger GitHub repository](https://github.com/10XGenomics/cellranger/tree/master/lib/python/cellranger/barcodes) and rename them like this:\n",
    "\n",
    "```\n",
    "3M-february-2018.txt.gz -> (unzip) -> 10xv3_whitelist.txt\n",
    "737K-august-2016.txt -> 10xv2_whitelist.txt\n",
    "737K-april-2014_rc.txt -> 10xv1_whitelist.txt\n",
    "```\n",
    "\n",
    "#### Download the human transcription factor motif database\n",
    "\n",
    "Get the [human_tfs_consensus.tab](https://storage.googleapis.com/linnarsson-lab-www-blobs/human_tfs_consensus.tab) file from our Google Cloud. These metadata will be used to determine which genes are transcription factors, and to assign them to families.\n",
    "\n",
    "### Preprocess the genome data\n",
    "\n",
    "Create a BED file with all the genes:\n",
    "\n",
    "```\n",
    "cat gencode.v31.primary_assembly.annotation.gtf |  gawk 'OFS=\"\\t\" {if ($3==\"gene\") {print $1,$4-1,$5,$10,0,$7}}' | tr -d '\";' > gencode.v31.primary_assembly.annotation.bed\n",
    "```\n",
    "\n",
    "Create a fasta file of pre-mRNA (exons + introns) transcripts:\n",
    "\n",
    "```\n",
    "bedtools sort -i gencode.v31.primary_assembly.annotation.bed > gencode.v31.primary_assembly.annotation.sorted.bed\n",
    "\n",
    "bedtools merge -i gencode.v31.primary_assembly.annotation.sorted.bed -s -c 4 -o collapse > gencode.v31.primary_assembly.annotation.merged.bed\n",
    "\n",
    "bedtools getfasta -name -fo gencode.v31.unspliced.fa -fi GRCh38.primary_assembly.genome.fa -bed gencode.v31.primary_assembly.annotation.sorted.bed\n",
    "```\n",
    "\n",
    "### Create the index manifest file, and an index directory\n",
    "\n",
    "A *manifest* file is necessary to tell cytograph how to find the relevant files in the index. Create a file `manifest.json` with the following content:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"species\": \"Homo sapiens\",\n",
    "    \"index_file\": \"gencode.v31.fragments.idx\",\n",
    "    \"gene_metadata_file\": \"gencode.v31.metadata.tab\",\n",
    "    \"gene_metadata_key\": \"AccessionVersion\",\n",
    "    \"fragments_to_genes_file\": \"fragments2genes.txt\",\n",
    "    \"layers\": {\n",
    "        \"unspliced\": \"unspliced_fragments.txt\",\n",
    "        \"spliced\": \"spliced_fragments.txt\"\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "The `gene_metadata_key` indicated which column in the metadata file (`gencode.v31.metadata.tab`) contains the gene IDs (as used in `fragments2genes.txt`).\n",
    "\n",
    "Finally, organize your files in a directory and subdirectory as follows:\n",
    "\n",
    "```\n",
    "10xv1_whitelist.txt\n",
    "10xv2_whitelist.txt\n",
    "10xv3_whitelist.txt\n",
    "inputs/\n",
    "    GRCh38.primary_assembly.genome.fa\n",
    "    GRCh38.primary_assembly.genome.fa.fai\n",
    "    gencode.v31.unspliced.fa\n",
    "    gencode.v31.primary_assembly.annotation.bed\n",
    "    gencode.v31.primary_assembly.annotation.gtf\n",
    "    gencode.v31.primary_assembly.annotation.sorted.bed\n",
    "    gencode.v31.transcripts.fa\n",
    "    hgnc_complete_set.txt\n",
    "    human_tfs_consensus.txt\n",<commit_msg>Update build_index.ipynb<commit_after>{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "from typing import *\n",
    "from Bio import SeqIO\n",
    "from Bio.SeqRecord import SeqRecord\n",
    "from Bio.Seq import Seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a genome index for use with loompy and cytograph\n",
    "\n",
    "WARNING: these instruction will create a genome index that is *only* suitable for use with 10x Chromium data. We create a composite index, with separate sequence fragments representing unspliced and spliced transcripts. Unspliced fragments will be generated from genomic sequence upstream of every poly-A stretch located inside of a gene locus. Spliced fragments will be generated from transcriptomic sequence upstream of every polyadenylation site in every known spliced transcript. This reflects the locations of reads generated by Chromium, but may not be suitable for other methods.\n",
    "\n",
    "The instructions below show how to build a genome index for the human genome. For other species, the code below will have to be modified to accomodate different sources and formats of metadata for those species. Instructions for mouse are available in the loompy/kallisto subdirectory.\n",
    "\n",
    "### Install prerequisites\n",
    "\n",
    "Install the [BioPython](https://biopython.org/wiki/Packages) package in your python distribution:\n",
    "\n",
    "```\n",
    "pip install biopython  # Method recommended by the BioPython people\n",
    "conda install -c conda-forge biopython  # Alternative, for Anaconda\n",
    "```\n",
    "\n",
    "Install [gawk](https://www.gnu.org/software/gawk/) (on Linux, this is typically already installed), [bedtools](https://bedtools.readthedocs.io/en/latest/), and [kallisto](https://pachterlab.github.io/kallisto/) (instructions below are for macOS using [Brew](https://brew.s)):\n",
    "```\n",
    "brew install gawk\n",
    "brew install bedtools\n",
    "brew install kallisto\n",
    "```\n",
    "\n",
    "### Download genome data\n",
    "\n",
    "#### Download transcript sequences and metadata from [Gencode](https://www.gencodegenes.org/human/)\n",
    "\n",
    "Get the \"Genome sequence, primary assembly (GRCh38)” file named [GRCh38.primary_assembly.genome.fa.gz](ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_31/GRCh38.primary_assembly.genome.fa.gz)\n",
    "\n",
    "Download the comprehensive PRI gene annotation on the primary assembly GTF file named [gencode.v31.primary_assembly.annotation.gtf](ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_31/gencode.v31.primary_assembly.annotation.gtf.gz)\n",
    "\n",
    "#### Download additional gene metadata from [HGNC](https://www.genenames.org/download/statistics-and-files/)\n",
    "\n",
    "Get the “Complete HGNC dataset” as TXT, a file named [hgnc_complete_set.txt](ftp://ftp.ebi.ac.uk/pub/databases/genenames/new/json/hgnc_complete_set.txt)\n",
    "\n",
    "#### Download 10x Chromium barcode whitelists\n",
    "\n",
    "Chromium cell barcodes are generated from a set of known sequences, which differ by version of the Chromium kits. Get three files from the [cellranger GitHub repository](https://github.com/10XGenomics/cellranger/tree/master/lib/python/cellranger/barcodes) and rename them like this:\n",
    "\n",
    "```\n",
    "3M-february-2018.txt.gz -> (unzip) -> 10xv3_whitelist.txt\n",
    "737K-august-2016.txt -> 10xv2_whitelist.txt\n",
    "737K-april-2014_rc.txt -> 10xv1_whitelist.txt\n",
    "```\n",
    "\n",
    "#### Download the human transcription factor motif database\n",
    "\n",
    "Get the [human_tfs_consensus.tab](https://storage.googleapis.com/linnarsson-lab-www-blobs/human_tfs_consensus.tab) file from our Google Cloud. These metadata will be used to determine which genes are transcription factors, and to assign them to families.\n",
    "\n",
    "### Preprocess the genome data\n",
    "\n",
    "Unpack gzip:ed files:\n",
    "\n",
    "```\n",
    "gunzip gencode.v31.primary_assembly.annotation.gtf.gz\n",
    "\n",
    "gunzip GRCh38.primary_assembly.genome.fa.gz\n",
    "```\n",
    "Create a BED file with all the genes:\n",
    "\n",
    "```\n",
    "cat gencode.v31.primary_assembly.annotation.gtf |  gawk 'OFS=\"\\t\" {if ($3==\"gene\") {print $1,$4-1,$5,$10,0,$7}}' | tr -d '\";' > gencode.v31.primary_assembly.annotation.bed\n",
    "```\n",
    "\n",
    "Create a fasta file of pre-mRNA (exons + introns) transcripts:\n",
    "\n",
    "```\n",
    "bedtools sort -i gencode.v31.primary_assembly.annotation.bed > gencode.v31.primary_assembly.annotation.sorted.bed\n",
    "\n",
    "bedtools merge -i gencode.v31.primary_assembly.annotation.sorted.bed -s -c 4 -o collapse > gencode.v31.primary_assembly.annotation.merged.bed\n",
    "\n",
    "bedtools getfasta -name -fo gencode.v31.unspliced.fa -fi GRCh38.primary_assembly.genome.fa -bed gencode.v31.primary_assembly.annotation.sorted.bed\n",
    "```\n",
    "\n",
    "### Create the index manifest file, and an index directory\n",
    "\n",
    "A *manifest* file is necessary to tell cytograph how to find the relevant files in the index. Create a file `manifest.json` with the following content:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"species\": \"Homo sapiens\",\n",
    "    \"index_file\": \"gencode.v31.fragments.idx\",\n",
    "    \"gene_metadata_file\": \"gencode.v31.metadata.tab\",\n",
    "    \"gene_metadata_key\": \"AccessionVersion\",\n",
    "    \"fragments_to_genes_file\": \"fragments2genes.txt\",\n",
    "    \"layers\": {\n",
    "        \"unspliced\": \"unspliced_fragments.txt\",\n",
    "        \"spliced\": \"spliced_fragments.txt\"\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "The `gene_metadata_key` indicated which column in the metadata file (`gencode.v31.metadata.tab`) contains the gene IDs (as used in `fragments2genes.txt`).\n",
    "\n",
    "Finally, organize your files in a directory and subdirectory as follows:\n",
    "\n",
    "```\n",
    "10xv1_whitelist.txt\n",
    "10xv2_whitelist.txt\n",
    "10xv3_whitelist.txt\n",
    "inputs/\n",
    "    GRCh38.primary_assembly.genome.fa\n",
    "    GRCh38.primary_assembly.genome.fa.fai\n",
    "    gencode.v31.unspliced.fa\n",
    "    gencode.v31.primary_assembly.annotation.bed\n",
    "    gencode.v31.primary_assembly.annotation.gtf\n",
    "    gencode.v31.primary_assembly.annotation.sorted.bed\n",
    "    gencode.v31.transcripts.fa\n",
    "    hgnc_complete_set.txt\n",
    "    human_tfs_consensus.tab\n",
{'fraction_non_alphanumeric': 0.16131961259079902, 'fraction_numerical': 0.016646489104116223, 'mean_word_length': 3.2498392282958197, 'pattern_counts': {'":': 34, '<': 3, '<?xml version=': 0, '>': 17, 'https://': 18, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1524661', 'n_tokens_mistral': 4755, 'n_tokens_neox': 4453, 'n_words': 1115}