<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Standardisation Pipeline\n",
    "\n",
    "In this notebook, I detail the process of importing data into `crowdastro`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input data sources\n",
    "\n",
    "The input data sources are:\n",
    "\n",
    "- Radio Galaxy Zoo\n",
    "    - Subjects (`radio_subjects.json`)\n",
    "    - Classifications (`radio_classifications.json`)\n",
    "- ATLAS\n",
    "    - Catalogue (`ATLASDR3_cmpcat_23July2015.dat`)\n",
    "    - FITS images of the radio sky (`cdfs` & `elais`)\n",
    "- SPITZER\n",
    "    - SWIRE Catalogue (`SWIRE3_CDFS_cat_IRAC24_21Dec05.tbl`)\n",
    "    - FITS images of the infrared sky (`cdfs` & `elais`)\n",
    "\n",
    "Paths to these should be specified in `crowdastro.json`. Radio Galaxy Zoo data should be imported into a database in MongoDB, specified by `radio_galaxy_zoo_db` in `crowdastro.json`. Following is an example `crowdastro.json`:\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"data_sources\": {\n",
    "        \"atlas_catalogue\": \"data/ATLASDR3_cmpcat_23July2015.dat\",\n",
    "        \"cdfs_fits\": \"data/cdfs\",\n",
    "        \"elais_s1_fits\": \"data/elais\",\n",
    "        \"radio_galaxy_zoo_db\": \"radio\",\n",
    "        \"swire_catalogue\": \"data/SWIRE3_CDFS_cat_IRAC24_21Dec05.tbl\"\n",
    "    },\n",
    "\n",
    "    \"mongo\": {\n",
    "        \"host\": \"localhost\",\n",
    "        \"port\": 27017\n",
    "    }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output data format\n",
    "\n",
    "The input data is converted into the output data by this pipeline. There are two output files for the training data and two output files for the testing data. The files are `rgz_atlas_data_{test,train}.h5` and `rgz_atlas_data_{test,train}.csv`. The partitioning ratio is specified in `crowdastro.json` as `test_size`.\n",
    "\n",
    "The `.h5` files contain numeric data, including all FITS images (both radio and infrared), classifications, and subject metadata. The structure is as follows, with datasets italicised:\n",
    "\n",
    "- `/`\n",
    "    - `atlas`\n",
    "        - `cdfs`\n",
    "            - *`images`*\n",
    "            - *`classifications`*\n",
    "            - *`positions`*\n",
    "        - `elais-s1`\n",
    "            - *`images`*\n",
    "            - *`classifications`*\n",
    "            - *`positions`*\n",
    "    - `swire`\n",
    "        - `cdfs`\n",
    "            - *`images`*\n",
    "            - *`classifications`*\n",
    "            - *`positions`*\n",
    "        - `elais-s1`\n",
    "            - *`images`*\n",
    "            - *`classifications`*\n",
    "            - *`positions`*\n",
    "\n",
    "I'm only using ATLAS and SWIRE for now, but this is easily generalised to FIRST and WISE or EMU and MIGHTEE.\n",
    "\n",
    "The `.csv` files contain textual data such as Zooniverse IDs. They are pretty much lookup tables; the reason for using CSV instead of HDF5 tables is partly for human readability and partly because dealing with textual data in HDF5 is unpleasant. The columns (examples parenthesised) are:\n",
    "\n",
    "- `survey` (atlas)\n",
    "- `field` (cdfs)\n",
    "- `zooniverse_id` (ARG0003r18)\n",
    "- `name` (ATLAS3_J033403.6-282423C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing ATLAS data\n",
    "\n",
    "The ATLAS dataset consists of images of the radio sky in two fields, CDFS and ELAIS-S1, as well as a catalogue of objects in these fields. Unfortunately, the identifiers of the images and the identifiers in the catalogue are *different*, so we cannot rely on them matching (in fact, they don't match *at all*). The ATLAS catalogue identifiers correspond to nothing I've found so far, but the CDFS/ELAIS-S1 identifiers correspond to the image names, the Galaxy Zoo: Radio Talk survey IDs, and the radio subjects' `source` field in the Radio Galaxy Zoo dataset. There's a catalogue associated with the images, but it doesn't include the ATLAS names.\n",
    "\n",
    "Each object in the dataset is called a \"radio component\" and the $2' \\times 2'$ patch of radio sky centred on each component is called a \"radio subject\". There are $2461$ CDFS component and $1935$ ELAIS-S1 component.\n",
    "\n",
    "The origin of each image is in the *top left* corner. Each image is $201$ pixels in width and height.\n",
    "\n",
    "The RA/DEC of each component is stored in the FITS images.\n",
    "\n",
    "Each component is sorted by RA (and by DEC in case of ties) in the output data. The images are stored in `/atlas/{cdfs,elais-s1}/images`. The RA/DEC of the components are stored in `/atlas/{cdfs,elais-s1}/positions` in decimal degrees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"<commit_msg>Documentation fixes for ATLAS import.<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Standardisation Pipeline\n",
    "\n",
    "In this notebook, I detail the process of importing data into `crowdastro`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input data sources\n",
    "\n",
    "The input data sources are:\n",
    "\n",
    "- Radio Galaxy Zoo\n",
    "    - Subjects (`radio_subjects.json`)\n",
    "    - Classifications (`radio_classifications.json`)\n",
    "- ATLAS\n",
    "    - Catalogue (`ATLASDR3_cmpcat_23July2015.dat`)\n",
    "    - FITS images of the radio sky (`cdfs` & `elais`)\n",
    "- SPITZER\n",
    "    - SWIRE Catalogue (`SWIRE3_CDFS_cat_IRAC24_21Dec05.tbl`)\n",
    "    - FITS images of the infrared sky (`cdfs` & `elais`)\n",
    "\n",
    "Paths to these should be specified in `crowdastro.json`. Radio Galaxy Zoo data should be imported into a database in MongoDB, specified by `radio_galaxy_zoo_db` in `crowdastro.json`. Following is an example `crowdastro.json`:\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"data_sources\": {\n",
    "        \"atlas_catalogue\": \"data/ATLASDR3_cmpcat_23July2015.dat\",\n",
    "        \"cdfs_fits\": \"data/cdfs\",\n",
    "        \"elais_s1_fits\": \"data/elais\",\n",
    "        \"radio_galaxy_zoo_db\": \"radio\",\n",
    "        \"swire_catalogue\": \"data/SWIRE3_CDFS_cat_IRAC24_21Dec05.tbl\"\n",
    "    },\n",
    "\n",
    "    \"mongo\": {\n",
    "        \"host\": \"localhost\",\n",
    "        \"port\": 27017\n",
    "    }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output data format\n",
    "\n",
    "The input data is converted into the output data by this pipeline. There are two output files for the training data and two output files for the testing data. The files are `rgz_atlas_data_{test,train}.h5` and `rgz_atlas_data_{test,train}.csv`. The partitioning ratio is specified in `crowdastro.json` as `test_size`.\n",
    "\n",
    "The `.h5` files contain numeric data, including all FITS images (both radio and infrared), classifications, and subject metadata. The structure is as follows, with datasets italicised:\n",
    "\n",
    "- `/`\n",
    "    - `atlas`\n",
    "        - `cdfs`\n",
    "            - *`images_2x2`*\n",
    "            - *`images_5x5`*\n",
    "            - *`classifications`*\n",
    "            - *`positions`*\n",
    "        - `elais-s1`\n",
    "            - *`images_2x2`*\n",
    "            - *`images_5x5`*\n",
    "            - *`classifications`*\n",
    "            - *`positions`*\n",
    "    - `swire`\n",
    "        - `cdfs`\n",
    "            - *`images_2x2`*\n",
    "            - *`images_5x5`*\n",
    "            - *`classifications`*\n",
    "            - *`positions`*\n",
    "        - `elais-s1`\n",
    "            - *`images_2x2`*\n",
    "            - *`images_5x5`*\n",
    "            - *`classifications`*\n",
    "            - *`positions`*\n",
    "\n",
    "I'm only using ATLAS and SWIRE for now, but this is easily generalised to FIRST and WISE or EMU and MIGHTEE.\n",
    "\n",
    "The `.csv` files contain textual data such as Zooniverse IDs. They are pretty much lookup tables; the reason for using CSV instead of HDF5 tables is partly for human readability and partly because dealing with textual data in HDF5 is unpleasant. The columns (examples parenthesised) are:\n",
    "\n",
    "- `survey` (atlas)\n",
    "- `field` (cdfs)\n",
    "- `zooniverse_id` (ARG0003r18)\n",
    "- `name` (ATLAS3_J033403.6-282423C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing ATLAS data\n",
    "\n",
    "The ATLAS dataset consists of images of the radio sky in two fields, CDFS and ELAIS-S1, as well as a catalogue of objects in these fields. Unfortunately, the identifiers of the images and the identifiers in the catalogue are *different*, so we cannot rely on them matching (in fact, they don't match *at all*). The ATLAS catalogue identifiers correspond to nothing I've found so far, but the CDFS/ELAIS-S1 identifiers correspond to the image names, the Galaxy Zoo: Radio Talk survey IDs, and the radio subjects' `source` field in the Radio Galaxy Zoo dataset. There's a catalogue associated with the images, but it doesn't include the ATLAS names.\n",
    "\n",
    "Each object in the dataset is called a \"radio component\" and the $2' \\times 2'$ patch of radio sky centred on each component is called a \"radio subject\". There are $2460$ CDFS components and $1935$ ELAIS-S1 components &mdash; but the ELAIS-S1 components do not appear in Radio Galaxy Zoo at all.\n",
    "\n",
    "The origin of each image is in the *top left* corner. Each image is $201$ pixels in width and height.\n",
    "\n",
    "The RA/DEC of each component is stored in the Radio Galaxy Zoo dataset.\n",
    "\n",
    "Each component is sorted by ATLAS name in all output data. The images are stored in `/atlas/{cdfs,elais-s1}/images`. The RA/DEC of the components are stored in `/atlas/{cdfs,elais-s1}/positions` in decimal degrees. The names and Zooniverse IDs of each component are stored directly in the CSV.\n",
    "\n",
    "ATLAS components are only imported if they have a corresponding Zooniverse ID (i.e., if they are in the Radio Galaxy Zoo dataset), and if they are within a threshold radius from a Zooniverse object. The threshold is set as `radio_location_threshold` in `crowdastro.json`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
{'fraction_non_alphanumeric': 0.15964643275908721, 'fraction_numerical': 0.015964643275908724, 'mean_word_length': 2.465, 'pattern_counts': {'":': 66, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3011007', 'n_tokens_mistral': 3904, 'n_tokens_neox': 3647, 'n_words': 1154}