<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step 1__: Launch a **t2.medium** instance (*redhat 7.x*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step 2__: login and run the following commands\n",
    "\n",
    "* ``` pushd /tmp ```\n",
    "* ``` sudo yum install wget -y # to get files we need ```\n",
    "\n",
    "* ``` wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm # the epel repo for dockr-compose```\n",
    "* ``` sudo yum install -y epel-release-latest-7.noarch.rpm ```\n",
    "* ``` popd ```\n",
    "\n",
    "* ``` sudo yum update -y ```\n",
    "* ``` sudo yum install vim-enhanced mlocate python-pip python-wheel python-wheel postgresql-devel wget tmux git -y ```\n",
    "\n",
    "* ``` sudo sh -c \"wget -qO- https://get.docker.com/ | sh\" # installs docker ```\n",
    "* ``` sudo groupadd -f docker && sudo usermod -aG docker $USER # creates docker user ```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step 3__: Logout and login back. (due to setting a new user group for docker)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step4__: install docker-compose \n",
    "* ``` sudo systemctl enable docker.service ```\n",
    "* ``` sudo systemctl start docker.service ```\n",
    "* ``` sudo pip install docker-compose # install docker compose now we have docker ```\n",
    "* ``` alias dc='docker-compose' ```\n",
    "* ``` echo \"alias dc='docker-compose'\" > ~/.bashrc ```\n",
    "* ``` echo \"alias dc='docker-compose'\" > ~/.bash_profile ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__step 5__: Create DART environment files to run using docker-compose.\n",
    "\n",
    "* ``` cd ~ ```\n",
    "* ``` git clone https://github.com/RetailMeNotSandbox/dart.git # the source code```\n",
    "* ``` mv dart/tools/vagrant . ```\n",
    "* ``` rm -rf dart ```\n",
    "* ``` cd vagrant  ```\n",
    "* ``` git clone https://github.com/RetailMeNotSandbox/dart.git # this source code will be referenced by d-compose```\n",
    "* ``` sudo pip install -r tools_requirements.txt # installs jinja so we can create d-compose files```\n",
    "* ``` python generate_deployment_files.py # generates files from templates, a set for each part (engine, web, ...) ```\n",
    "* ``` cd docker_files ```\n",
    "* ``` sed -i 's|/vagrant_data/docker_files|.|g' docker-compose.yml ```\n",
    "* ``` sed -i 's|/vagrant_data|.|g' docker-compose.yml # removing prefixes since we will use the files in cur dir```\n",
    "* ``` sed -i 's|/docker_files|.|g' docker-compose.yml ```\n",
    "* ``` docker-compose up -d --build web postgres elasticmq # launch dependent containers first so we can init dart ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step 6__: init DART database\n",
    "``` \n",
    "STOP=1;\n",
    "while [ $STOP -gt 0 ]; do\n",
    "  sh -c 'docker-compose logs web | tail -10'\n",
    "  docker-compose exec web curl -XPOST \"http://127.0.0.1:5000/admin/create_all\";\n",
    "  STOP=$?;\n",
    "  echo \"Waiting for web worker to complete. Trying again in 15 seconds ...\"\n",
    "  sleep 15;\n",
    "done \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__step 7__: Launch the remaining DART components.\n",
    "* ``` docker-compose up -d --build engine trigger subscription ```\n",
    "* ``` docker-compose ps # to verify ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step 8__: Add no-op engine so we can run unit tests.\n",
    "```\n",
    "STOP=1; \n",
    "while [ $STOP -gt 0 ]; do \n",
    "  docker-compose exec web python /tmp/src/python/dart/engine/no_op/add_engine.py; \n",
    "  STOP=$?; \n",
    "  sleep 1; \n",
    "done\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__step 9__: Run unit tests.\n",
    "* ``` docker-compose exec web python -m unittest discover /tmp/src/python/dart/test/schema/ \"test_*.py\" ```\n",
    "* ``` docker-compose exec web python -m unittest discover /tmp/src/python/dart/test/graph \"test_*.py\" ```\n",
    "* ``` docker-compose exec web python -m unittest discover /tmp/src/python/dart/test/crud \"test_*.py\" ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Step 10__: Misc.\n",
    "* Open port 5000 as a custom tcp rule (use `my ip`) then you can access this instance via browser\n",
    "* open 5432 for sql and so on ..(see docker-compose file for ports)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
<commit_msg>moving dart git clone to inside the docker_files directory<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step 1__: Launch a **t2.medium** instance (*redhat 7.x*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step 2__: login and run the following commands\n",
    "\n",
    "* ``` pushd /tmp ```\n",
    "* ``` sudo yum install wget -y # to get files we need ```\n",
    "\n",
    "* ``` wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm # the epel repo for dockr-compose```\n",
    "* ``` sudo yum install -y epel-release-latest-7.noarch.rpm ```\n",
    "* ``` popd ```\n",
    "\n",
    "* ``` sudo yum update -y ```\n",
    "* ``` sudo yum install vim-enhanced mlocate python-pip python-wheel python-wheel postgresql-devel wget tmux git -y ```\n",
    "\n",
    "* ``` sudo sh -c \"wget -qO- https://get.docker.com/ | sh\" # installs docker ```\n",
    "* ``` sudo groupadd -f docker && sudo usermod -aG docker $USER # creates docker user ```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step 3__: Logout and login back. (due to setting a new user group for docker)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step4__: install docker-compose \n",
    "* ``` sudo systemctl enable docker.service ```\n",
    "* ``` sudo systemctl start docker.service ```\n",
    "* ``` sudo pip install docker-compose # install docker compose now we have docker ```\n",
    "* ``` alias dc='docker-compose' ```\n",
    "* ``` echo \"alias dc='docker-compose'\" > ~/.bashrc ```\n",
    "* ``` echo \"alias dc='docker-compose'\" > ~/.bash_profile ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__step 5__: Create DART environment files to run using docker-compose.\n",
    "\n",
    "* ``` cd ~ ```\n",
    "* ``` mv dart/tools/vagrant . ```\n",
    "* ``` rm -rf dart ```\n",
    "* ``` cd vagrant  ```\n",
    "* ``` git clone https://github.com/RetailMeNotSandbox/dart.git # this source code will be referenced by d-compose```\n",
    "* ``` sudo pip install -r tools_requirements.txt # installs jinja so we can create d-compose files```\n",
    "* ``` python generate_deployment_files.py # generates files from templates, a set for each part (engine, web, ...) ```\n",
    "* ``` cd docker_files ```\n",
    "* ``` sed -i 's|/vagrant_data/docker_files|.|g' docker-compose.yml ```\n",
    "* ``` sed -i 's|/vagrant_data|.|g' docker-compose.yml # removing prefixes since we will use the files in cur dir```\n",
    "* ``` sed -i 's|/docker_files|.|g' docker-compose.yml ```\n",
    "* ``` git clone https://github.com/RetailMeNotSandbox/dart.git # the source code```\n",    
    "* ``` docker-compose up -d --build web postgres elasticmq # launch dependent containers first so we can init dart ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step 6__: init DART database\n",
    "``` \n",
    "STOP=1;\n",
    "while [ $STOP -gt 0 ]; do\n",
    "  sh -c 'docker-compose logs web | tail -10'\n",
    "  docker-compose exec web curl -XPOST \"http://127.0.0.1:5000/admin/create_all\";\n",
    "  STOP=$?;\n",
    "  echo \"Waiting for web worker to complete. Trying again in 15 seconds ...\"\n",
    "  sleep 15;\n",
    "done \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__step 7__: Launch the remaining DART components.\n",
    "* ``` docker-compose up -d --build engine trigger subscription ```\n",
    "* ``` docker-compose ps # to verify ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Step 8__: Add no-op engine so we can run unit tests.\n",
    "```\n",
    "STOP=1; \n",
    "while [ $STOP -gt 0 ]; do \n",
    "  docker-compose exec web python /tmp/src/python/dart/engine/no_op/add_engine.py; \n",
    "  STOP=$?; \n",
    "  sleep 1; \n",
    "done\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__step 9__: Run unit tests.\n",
    "* ``` docker-compose exec web python -m unittest discover /tmp/src/python/dart/test/schema/ \"test_*.py\" ```\n",
    "* ``` docker-compose exec web python -m unittest discover /tmp/src/python/dart/test/graph \"test_*.py\" ```\n",
    "* ``` docker-compose exec web python -m unittest discover /tmp/src/python/dart/test/crud \"test_*.py\" ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Step 10__: Misc.\n",
    "* Open port 5000 as a custom tcp rule (use `my ip`) then you can access this instance via browser\n",
    "* open 5432 for sql and so on ..(see docker-compose file for ports)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

{'fraction_non_alphanumeric': 0.21550023485204323, 'fraction_numerical': 0.0099577266322217, 'mean_word_length': 2.6773747841105355, 'pattern_counts': {'":': 108, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '682577', 'n_tokens_mistral': 4057, 'n_tokens_neox': 4005, 'n_words': 1006}