<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Here we go! So I still need to figure out some details on how to use these notebooks and what a good structure would be.  Let's keep this simple and look at the finished product when needed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## This is interesting "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Action Plan:\n",
    "1. Create a folder structure for images (already done)\n",
    "2. Create a validation and sample set\n",
    "3. Finetune and train model\n",
    "4. Generate Predictions \n",
    "5. Validate Predictions\n",
    "6. Submit Predictions to Kaggle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The folder structure as you created it in AWS:\n",
    "utils/\n",
    "    vgg16.py\n",
    "    utils.py\n",
    "lesson1/\n",
    "    valid/\n",
    "    results/\n",
    "    redux.ipynb\n",
    "    data/\n",
    "        redux/\n",
    "            train/\n",
    "                cat.437.jpg\n",
    "                dog.9924.jpg\n",
    "                cat.1029.jpg\n",
    "                dog.4374.jpg\n",
    "            test/\n",
    "                231.jpg\n",
    "                325.jpg\n",
    "                1235.jpg\n",
    "                9923.jpg"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}<commit_msg>Created folders for data<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Here we go! So I still need to figure out some details on how to use these notebooks and what a good structure would be.  Let's keep this simple and look at the finished product when needed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## This is interesting "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Action Plan:\n",
    "1. Create a folder structure for images (already done)\n",
    "2. Create a validation and sample set\n",
    "3. Finetune and train model\n",
    "4. Generate Predictions \n",
    "5. Validate Predictions\n",
    "6. Submit Predictions to Kaggle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The folder structure as you created it in AWS:\n",
    "utils/\n",
    "    vgg16.py\n",
    "    utils.py\n",
    "lesson1/\n",
    "    valid/\n",
    "    results/\n",
    "    redux.ipynb\n",
    "    data/\n",
    "        redux/\n",
    "            train/\n",
    "                cat.437.jpg\n",
    "                dog.9924.jpg\n",
    "                cat.1029.jpg\n",
    "                dog.4374.jpg\n",
    "            test/\n",
    "                231.jpg\n",
    "                325.jpg\n",
    "                1235.jpg\n",
    "                9923.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Here we create a homebase for directories\n",
    "import os, sys\n",
    "current_dir = os.getcwd()\n",
    "LESSON_HOME_DIR = current_dir\n",
    "DATA_HOME_DIR = current_dir+'/data/redux'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Allow relative imports to directories above Lesson1/\n",
    "sys.path.insert(1, os.path.join(sys.path[0], '..'))\n",
    "\n",
    "#import modules\n",
    "from utils import *\n",
    "from vgg16 import Vgg16\n",
    "\n",
    "#Instantiate plotting tool\n",
    "#In Jupyter notebooks you will need to do this before doing any plotting \n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Action Plan\n",
    "1. Create a Validation and Sample sets\n",
    "2. Rearrange image files into their respective directories\n",
    "3. Finetune and train model\n",
    "4. Generate predictions\n",
    "5. Validate predictions\n",
    "6. Submit predictions to Kaggle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Create directories (This is already done)\n",
    "%cd $DATA_HOME_DIR\n",
    "%mkdir valid\n",
    "%mkdir results\n",
    "%mkdir -p sample/train\n",
    "%mkdir -p sample/test\n",
    "%mkdir -p sample/valid\n",
    "%mkdir -p sample/results\n",
    "%mkdir -p sample/unknown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%cd %DATA_HOME_DIR/train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# glob will find any file, in this case with a .jpg\n",
    "g = glob('*.jpg')\n",
    "# np.random.permutation will randomly permutate a sequence (from docs ;))\n",
    "shuf = np.random.permutation(g)\n",
    "for i in range(2000): os.rename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from shutil import copyfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "g = glob('*.jpg')\n",
    "shuf = np.random.permutation(g)\n",
    "#from docs: copyfile(source, destination)\n",
    "for i in range of(200): copyfile(shuf[i], DATA_HOME_DIR+'sample/valid/' + shuf[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rearrange Image Files into their respective directories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
{'fraction_non_alphanumeric': 0.1860431654676259, 'fraction_numerical': 0.016402877697841725, 'mean_word_length': 2.098974587605885, 'pattern_counts': {'":': 122, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4105144', 'n_tokens_mistral': 2624, 'n_tokens_neox': 2537, 'n_words': 576}