<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading Annotations from a _GO Association File_ (GAF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Download a GAF file\n",
    "2. Load the GAF file into the GafReader\n",
    "3. Get Annotations\n",
    "\n",
    "**Bonus: Each line in the GAF file is stored in a namedtuple**:\n",
    "  * Namedtuple fields\n",
    "  * Print a subset of the namedtuple fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Download a GAF file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-04-08 09:43:16--  http://current.geneontology.org/annotations/goa_human.gaf.gz\n",
      "Resolving current.geneontology.org (current.geneontology.org)... 172.16.31.10, 172.16.17.32, 172.16.31.10, ...\n",
      "Connecting to current.geneontology.org (current.geneontology.org)|172.16.31.10|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 8524279 (8.1M) [application/gzip]\n",
      "Saving to: ‘goa_human.gaf.gz’\n",
      "\n",
      "goa_human.gaf.gz    100%[===================>]   8.13M  2.05MB/s    in 3.9s    \n",
      "\n",
      "utime(goa_human.gaf.gz): Operation not permitted\n",
      "2019-04-08 09:43:20 (2.07 MB/s) - ‘goa_human.gaf.gz’ saved [8524279/8524279]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget http://current.geneontology.org/annotations/goa_human.gaf.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gunzip goa_human.gaf.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Load the GAF file into the GafReader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  READ      476,348 associations: goa_human.gaf\n"
     ]
    }
   ],
   "source": [
    "from goatools.anno.gaf_reader import GafReader\n",
    "\n",
    "ogaf = GafReader(\"goa_human.gaf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Get Annotations\n",
    "The annotations will be stored in a dict where:\n",
    "  * the key is the protein ID and \n",
    "  * the value is a list of GO IDs associated with the protein."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2gos = ogaf.read_gaf()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A0A024R161 : GO:0003924 GO:0005515 GO:0005834 GO:0007186\n",
      "\n",
      "A0A024RBG1 : GO:0003723 GO:0005829 GO:0008486 GO:0046872 GO:0052840 GO:0052842\n",
      "\n",
      "A0A075B6H7 : GO:0002377 GO:0005615 GO:0006955\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for protein_id, go_ids in sorted(id2gos.items())[:3]:\n",
    "    print(\"{PROT:7} : {GOs}\\n\".format(\n",
    "        PROT=protein_id,\n",
    "        GOs=' '.join(sorted(go_ids))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus: The GAF is stored as a list of named tuples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ntgafobj(DB='UniProtKB', DB_ID='A0A024R161', DB_Symbol='DNAJC25-GNG10', Qualifier=[], GO_ID='GO:0005834', DB_Reference={'PMID:21873635'}, Evidence_Code='IBA', With_From={'UniProtKB:P63212', 'FB:FBgn0004921', 'UniProtKB:O14610', 'RGD:621514', 'PANTHER:PTN001418483', 'RGD:69268', 'RGD:1595475'}, Aspect='C', DB_Name={'Guanine nucleotide-binding protein subunit gamma'}, DB_Synonym={'hCG_1994888', 'DNAJC25-GNG10'}, DB_Type='protein', Taxon=[9606], Date=datetime.date(2018, 4, 25), Assigned_By='GO_Central', Annotation_Extension=set(), Gene_Product_Form_ID=set())\n"
     ]
    }
   ],
   "source": [
    "nts = sorted(ogaf.associations, key=lambda nt:nt.DB_ID)\n",
    "for nt_line in nts[:1]:\n",
    "    print(nt_line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Namedtuple fields\n",
    "```\n",
    "DB             #  0 required 1              UniProtKB\n",
    "DB_ID          #  1 required 1              P12345\n",
    "DB_Symbol      #  2 required 1              PHO3\n",<commit_msg>Clarify named tuple contents and simplify code https://github.com/tanghaibao/goatools/issues/125<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading Annotations from a _GO Association File_ (GAF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Download a GAF file\n",
    "2. Load the GAF file into the GafReader\n",
    "3. Get Annotations\n",
    "\n",
    "**Bonus: Each line in the GAF file is stored in a namedtuple**:\n",
    "  * Namedtuple fields\n",
    "  * Print a subset of the namedtuple fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Download a GAF file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-04-08 09:43:16--  http://current.geneontology.org/annotations/goa_human.gaf.gz\n",
      "Resolving current.geneontology.org (current.geneontology.org)... 172.16.31.10, 172.16.17.32, 172.16.31.10, ...\n",
      "Connecting to current.geneontology.org (current.geneontology.org)|172.16.31.10|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 8524279 (8.1M) [application/gzip]\n",
      "Saving to: ‘goa_human.gaf.gz’\n",
      "\n",
      "goa_human.gaf.gz    100%[===================>]   8.13M  2.05MB/s    in 3.9s    \n",
      "\n",
      "utime(goa_human.gaf.gz): Operation not permitted\n",
      "2019-04-08 09:43:20 (2.07 MB/s) - ‘goa_human.gaf.gz’ saved [8524279/8524279]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget http://current.geneontology.org/annotations/goa_human.gaf.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gunzip goa_human.gaf.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Load the GAF file into the GafReader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  READ      476,348 associations: goa_human.gaf\n"
     ]
    }
   ],
   "source": [
    "from goatools.anno.gaf_reader import GafReader\n",
    "\n",
    "ogaf = GafReader(\"goa_human.gaf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Get Annotations\n",
    "The annotations will be stored in a dict where:\n",
    "  * the key is the protein ID and \n",
    "  * the value is a list of GO IDs associated with the protein."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2gos = ogaf.read_gaf()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A0A024R161 : GO:0003924 GO:0005515 GO:0005834 GO:0007186\n",
      "\n",
      "A0A024RBG1 : GO:0003723 GO:0005829 GO:0008486 GO:0046872 GO:0052840 GO:0052842\n",
      "\n",
      "A0A075B6H7 : GO:0002377 GO:0005615 GO:0006955\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for protein_id, go_ids in sorted(id2gos.items())[:3]:\n",
    "    print(\"{PROT:7} : {GOs}\\n\".format(\n",
    "        PROT=protein_id,\n",
    "        GOs=' '.join(sorted(go_ids))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus: The GAF is stored as a list of named tuples\n",
    "The list of namedtuples is stored in **GafReader** data member named **_associations_**.\n",
    "\n",
    "Each namedtuple stores data for one line in the GAF file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ntgafobj(DB='UniProtKB', DB_ID='A0A024R161', DB_Symbol='DNAJC25-GNG10', Qualifier=[], GO_ID='GO:0005834', DB_Reference={'PMID:21873635'}, Evidence_Code='IBA', With_From={'UniProtKB:P63212', 'FB:FBgn0004921', 'UniProtKB:O14610', 'RGD:621514', 'PANTHER:PTN001418483', 'RGD:69268', 'RGD:1595475'}, Aspect='C', DB_Name={'Guanine nucleotide-binding protein subunit gamma'}, DB_Synonym={'hCG_1994888', 'DNAJC25-GNG10'}, DB_Type='protein', Taxon=[9606], Date=datetime.date(2018, 4, 25), Assigned_By='GO_Central', Annotation_Extension=set(), Gene_Product_Form_ID=set())\n"
     ]
    }
   ],
   "source": [
    "# Sort the list of GAF namedtuples by ID\n",
    "nts = sorted(ogaf.associations, key=lambda nt:nt.DB_ID)\n",
    "\n",
    "# Print one namedtuple\n",
    "print(nts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Namedtuple fields\n",
    "```\n",
    "DB             #  0 required 1              UniProtKB\n",
    "DB_ID          #  1 required 1              P12345\n",
    "DB_Symbol      #  2 required 1              PHO3\n",
{'fraction_non_alphanumeric': 0.20016440608302508, 'fraction_numerical': 0.0707973695026716, 'mean_word_length': 2.549598832968636, 'pattern_counts': {'":': 128, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 8, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5378619', 'n_tokens_mistral': 4314, 'n_tokens_neox': 3813, 'n_words': 732}