<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1a72486f-ffbd-4800-bc03-5e6427964faa",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Step 4: Upload a Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0484a120-b651-434d-a439-039200c3e9b2",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setup Variables\n",
    "\n",
    "Before we start let's store some variables that will come in handy later in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8683929a-fbf8-4ade-a7e0-90d35c378675",
   "metadata": {},
   "outputs": [],
   "source": [
    "# autodetect the host_ip\n",
    "from utils import auto_detect_domain_host_ip\n",
    "\n",
    "DOMAIN_HOST_IP = auto_detect_domain_host_ip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a497bd88-53c9-4bd9-acba-e152b983d383",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set the email and password of your Domain node.\n",
    "# We will be using the default email and password that got created during Domain creation.\n",
    "\n",
    "ADMIN_EMAIL=\"<EMAIL>\"\n",
    "ADMIN_PASSWORD=\"<PASSWORD>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78ca2364",
   "metadata": {},
   "source": [
    "## Step 4a: Log into our Domain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b923206d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import syft as sy\n",
    "\n",
    "print(f\"You're running syft version: {sy.__version__}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f682a99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's log into the domain using the credentials\n",
    "try:\n",
    "    domain_client = sy.login(\n",
    "        url=DOMAIN_HOST_IP, email=ADMIN_EMAIL, password=<PASSWORD>",
    "    )\n",
    "    print()\n",
    "    print(\"🎉 You successfully connected to your domain!\")\n",
    "except Exception as e:\n",
    "    print(\"❌ Unable to connect, did you set the `DOMAIN_HOST_IP` variable above?\")\n",
    "    raise e"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "770bc6e1-8106-4079-a916-1f6fea2bec40",
   "metadata": {},
   "source": [
    "## Step 4b: Creating a Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55ad95a9",
   "metadata": {},
   "source": [
    "### MedNIST Dataset\n",
    "\n",
    "We will be using the MedNIST dataset. The MedNIST dataset was gathered from several sets from TCIA, the RSNA Bone Age Challenge, and the NIH Chest X-ray dataset.\n",
    "\n",
    "The dataset is kindly made available by Dr. <NAME>., Ph.D. (Department of Radiology, Mayo Clinic) under the Creative Commons CC BY-SA 4.0 license. If you use the MedNIST dataset, please acknowledge the source, e.g. https://colab.research.google.com/drive/1wy8XUSnNWlhDNazFdvGBHLfdkGvOHBKe#scrollTo=ZaHFhidyCBJa\n",
    "\n",
    "Let's move on to download and extract the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d5af725",
   "metadata": {},
   "source": [
    "The dataset has been stored pickle file. Lets download the dataset using the instructions below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0dcc170",
   "metadata": {},
   "source": [
    "Copy your variables `MY_DATASET_URL` from your session details."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d79ecf5",
   "metadata": {},
   "source": [
    "```\n",
    "Hi Person,\n",
    "These are your Session Details:\n",
    "-------------------------------\n",
    "Username: azureuser\n",
    "Password: <PASSWORD>",
    "VM IP Address: x.x.x.x\n",
    "    \n",
    "📎 MY_DATASET_URL:\n",
    "'https://media.githubusercontent.com/media/shubham3121/datasets/main/MedNIST/subsets/xxxxxxx.pkl'\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83ab0392",
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace these with your own from the session details\n",
    "# paste your dataset url here\n",
    "MY_DATASET_URL = \"https://media.githubusercontent.com/media/shubham3121/datasets/main/MedNIST/subsets/MedNIST-437467c744.pkl\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35e6a8b8-89f4-4849-a1dd-02fc166da76f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# download MedNIST.pkl\n",
    "from utils import download_mednist_dataset\n",
    "\n",
    "FILE_PATH = download_mednist_dataset(MY_DATASET_URL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a864da2e",
   "metadata": {},
   "source": [
    "Now, before we move forward, let's store some variables related to the dataset.\n",
    "\n",
    "We require your participant number and the total participant count in the session to allocate you a unique subset of the MedNIST data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7515311f",
   "metadata": {},<commit_msg>notebooks: remove hard coded dataset url<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1a72486f-ffbd-4800-bc03-5e6427964faa",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Step 4: Upload a Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0484a120-b651-434d-a439-039200c3e9b2",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setup Variables\n",
    "\n",
    "Before we start let's store some variables that will come in handy later in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8683929a-fbf8-4ade-a7e0-90d35c378675",
   "metadata": {},
   "outputs": [],
   "source": [
    "# autodetect the host_ip\n",
    "from utils import auto_detect_domain_host_ip\n",
    "\n",
    "DOMAIN_HOST_IP = auto_detect_domain_host_ip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a497bd88-53c9-4bd9-acba-e152b983d383",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set the email and password of your Domain node.\n",
    "# We will be using the default email and password that got created during Domain creation.\n",
    "\n",
    "ADMIN_EMAIL=\"<EMAIL>\"\n",
    "ADMIN_PASSWORD=\"<PASSWORD>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78ca2364",
   "metadata": {},
   "source": [
    "## Step 4a: Log into our Domain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b923206d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import syft as sy\n",
    "\n",
    "print(f\"You're running syft version: {sy.__version__}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f682a99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's log into the domain using the credentials\n",
    "try:\n",
    "    domain_client = sy.login(\n",
    "        url=DOMAIN_HOST_IP, email=ADMIN_EMAIL, password=<PASSWORD>\n",
    "    )\n",
    "    print()\n",
    "    print(\"🎉 You successfully connected to your domain!\")\n",
    "except Exception as e:\n",
    "    print(\"❌ Unable to connect, did you set the `DOMAIN_HOST_IP` variable above?\")\n",
    "    raise e"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "770bc6e1-8106-4079-a916-1f6fea2bec40",
   "metadata": {},
   "source": [
    "## Step 4b: Creating a Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55ad95a9",
   "metadata": {},
   "source": [
    "### MedNIST Dataset\n",
    "\n",
    "We will be using the MedNIST dataset. The MedNIST dataset was gathered from several sets from TCIA, the RSNA Bone Age Challenge, and the NIH Chest X-ray dataset.\n",
    "\n",
    "The dataset is kindly made available by Dr. <NAME>., Ph.D. (Department of Radiology, Mayo Clinic) under the Creative Commons CC BY-SA 4.0 license. If you use the MedNIST dataset, please acknowledge the source, e.g. https://colab.research.google.com/drive/1wy8XUSnNWlhDNazFdvGBHLfdkGvOHBKe#scrollTo=ZaHFhidyCBJa\n",
    "\n",
    "Let's move on to download and extract the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d5af725",
   "metadata": {},
   "source": [
    "The dataset has been stored pickle file. Lets download the dataset using the instructions below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0dcc170",
   "metadata": {},
   "source": [
    "Copy your variables `MY_DATASET_URL` from your session details."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d79ecf5",
   "metadata": {},
   "source": [
    "```\n",
    "Hi Person,\n",
    "These are your Session Details:\n",
    "-------------------------------\n",
    "Username: azureuser\n",
    "Password: <PASSWORD>",
    "VM IP Address: x.x.x.x\n",
    "    \n",
    "📎 MY_DATASET_URL:\n",
    "'https://media.githubusercontent.com/media/shubham3121/datasets/main/MedNIST/subsets/xxxxxxx.pkl'\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83ab0392",
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace these with your own from the session details\n",
    "# paste your dataset url here\n",
    "MY_DATASET_URL = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35e6a8b8-89f4-4849-a1dd-02fc166da76f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# download MedNIST.pkl\n",
    "from utils import download_mednist_dataset\n",
    "\n",
    "FILE_PATH = download_mednist_dataset(MY_DATASET_URL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a864da2e",
   "metadata": {},
   "source": [
    "Now, before we move forward, let's store some variables related to the dataset.\n",
    "\n",
    "We require your participant number and the total participant count in the session to allocate you a unique subset of the MedNIST data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7515311f",
   "metadata": {},
{'fraction_non_alphanumeric': 0.18814979307560312, 'fraction_numerical': 0.039668920964974264, 'mean_word_length': 2.7501892505677517, 'pattern_counts': {'":': 158, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6125716', 'n_tokens_mistral': 3999, 'n_tokens_neox': 3746, 'n_words': 852}