<commit_before>#+OPTIONS: ':nil *:t -:t ::t <:t H:3 \n:nil ^:nil arch:headline author:t c:nil
#+OPTIONS: creator:nil d:(not "LOGBOOK") date:t e:t email:nil f:t inline:t
#+OPTIONS: num:t p:nil pri:nil prop:nil stat:t tags:t tasks:t tex:t timestamp:t
#+OPTIONS: title:t toc:t todo:t |:t
#+TITLE: AMS 2016: UniCloud, Docker at Unidata
#+SUBTITLE: LDM, TDS, and RAMADDA on Microsoft Azure VM
#+DATE: <2015-12-08 Tue>
#+AUTHOR: <NAME>
#+EMAIL: <EMAIL>
#+LANGUAGE: en
#+SELECT_TAGS: export
#+EXCLUDE_TAGS: noexport
#+CREATOR: Emacs 24.5.1 (Org mode 8.3.2)

#+LATEX_HEADER: \usepackage{bigfoot}
#+LATEX_HEADER: \DeclareNewFootnote{URL}[arabic]
#+LATEX_HEADER: \renewcommand{\href}[2]{#2\footnoteURL{\url{#1}}}

#+SETUPFILE: theme-readtheorg.setup

#+PROPERTY: header-args :tangle no


* Org Export Set up (Internal Only)                                :noexport:

# org-mode stuff. Don't want confirmation for babel exec, nor should babel block be evaluated during export.

#+BEGIN_SRC emacs-lisp :results silent :exports none 
  (setq org-confirm-babel-evaluate nil)
  (setq org-export-babel-evaluate nil)
#+END_SRC

#+BEGIN_SRC emacs-lisp :results silent 
  (defun ams2016/org-save-and-export ()
    (interactive)
    (when (eq major-mode 'org-mode)
        (progn
          (org-html-export-to-html)
          (org-md-export-to-markdown)
          (org-latex-export-to-pdf)
          (org-ascii-export-to-ascii))))

  (add-hook 'after-save-hook 'ams2016/org-save-and-export nil t)
#+END_SRC

# Defining the VM we will be work with for the remainder of this org babel session.

#+BEGIN_SRC org :noweb-ref myvm :exports none 
unidata-server
#+END_SRC

#+NAME: azure-vm
#+BEGIN_SRC org :results silent :exports none :noweb yes 
<<myvm>>
#+END_SRC

# Setting up org babel default arguments for executing =sh= commands below. We will be using tramp for the remote execution. You should have something like this in your ssh-config:

#+BEGIN_SRC sh :eval no :exports none 
Host <<myvm>>
    User     ubuntu
    Port     22
    IdentityFile ~/.docker/machine/machines/<<myvm>>/id_rsa
    Hostname <<myvm>>.cloudapp.net
#+END_SRC

# Defaulting to using remote VM. Be careful to specify :dir ~ for the sh blocks where you do not want remote VM execution of commands.

#+BEGIN_SRC emacs-lisp :noweb yes :results silent :exports none 
  (setq-local org-babel-default-header-args:sh
              '((:dir . "/ubuntu@<<myvm>>:")))a
#+END_SRC

* Preamble

The following instructions describe how to configure a [[https://azure.microsoft.com][Microsoft Azure VM]] serving data with the [[http://www.unidata.ucar.edu/software/ldm/][LDM]], [[http://www.unidata.ucar.edu/software/thredds/current/tds/][TDS]], and [[http://sourceforge.net/projects/ramadda/][RAMADDA]]. This document assumes you have access to Azure resources though these instructions should be fairly similar on other cloud providers (e.g., Amazon). They also assume familiarity with Unix, Docker, and Unidata technology in general. You must have =sudo= priviliges on your Azure host which will hopfully be provided to you be default. You will have to be comfortable entering commands at the Unix command line. We will be using Docker images defined at the [[https://github.com/Unidata/Unidata-Dockerfiles][Unidata-Dockerfiles repository]] in addition to a configuration specifically planned for AMS 2016 demonstrations  project [[https://github.com/Unidata/Unidata-Dockerfiles/tree/master/ams2016][AMS 2016 demonstrations  project]].

:PROPERTIES:
:END:
* Preliminary Setup on Azure

The instructions assume we will create an Azure VM called =unidata-server.cloudapp.net= abbreviated to =unidata-server=. Tailor the VM name for your purposes when following this document. This VM will be our *Docker Host* from where we will run Docker containers for the LDM, TDS, and RAMADDA.

** =docker-machine=

[[https://docs.docker.com/machine/install-machine/][Install]] =docker-machine= on your local computer. =docker-machine= is a command line tool that gives users the ability to create Docker VMs on your local computer or on a cloud provider such as Azure.

** Create a VM on Azure.

The following =docker-machine= command will create a Docker VM on Azure in which you will run various Docker containers. It will take a few minutes to run (between 5 and 10 minutes). You will have to supply =azure-subscription-id= and =azure-subscription-cert= path. See these Azure =docker-machine= [[https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-docker-machine/][instructions]], if you have questions about this process.

# Don't execute the following code block in this buffer; it will take too long. Instead =org-babel-tangle= with a prefix argument; C-u C-c C-v t. Then execute from the command line =/tmp/azure.sh=.
<commit_msg>info on azure vm sizes<commit_after>#+OPTIONS: ':nil *:t -:t ::t <:t H:3 \n:nil ^:nil arch:headline author:t c:nil
#+OPTIONS: creator:nil d:(not "LOGBOOK") date:t e:t email:nil f:t inline:t
#+OPTIONS: num:t p:nil pri:nil prop:nil stat:t tags:t tasks:t tex:t timestamp:t
#+OPTIONS: title:t toc:t todo:t |:t
#+TITLE: AMS 2016: UniCloud, Docker at Unidata
#+SUBTITLE: LDM, TDS, and RAMADDA on Microsoft Azure VM
#+DATE: <2015-12-08 Tue>
#+AUTHOR: <NAME>
#+EMAIL: <EMAIL>
#+LANGUAGE: en
#+SELECT_TAGS: export
#+EXCLUDE_TAGS: noexport
#+CREATOR: Emacs 24.5.1 (Org mode 8.3.2)

#+LATEX_HEADER: \usepackage{bigfoot}
#+LATEX_HEADER: \DeclareNewFootnote{URL}[arabic]
#+LATEX_HEADER: \renewcommand{\href}[2]{#2\footnoteURL{\url{#1}}}

#+SETUPFILE: theme-readtheorg.setup

#+PROPERTY: header-args :tangle no


* Org Export Set up (Internal Only)                                :noexport:

# org-mode stuff. Don't want confirmation for babel exec, nor should babel block be evaluated during export.

#+BEGIN_SRC emacs-lisp :results silent :exports none 
  (setq org-confirm-babel-evaluate nil)
  (setq org-export-babel-evaluate nil)
#+END_SRC

#+BEGIN_SRC emacs-lisp :results silent 
  (defun ams2016/org-save-and-export ()
    (interactive)
    (when (eq major-mode 'org-mode)
        (progn
          (org-html-export-to-html)
          (org-md-export-to-markdown)
          (org-latex-export-to-pdf)
          (org-ascii-export-to-ascii))))

  (add-hook 'after-save-hook 'ams2016/org-save-and-export nil t)
#+END_SRC

# Defining the VM we will be work with for the remainder of this org babel session.

#+BEGIN_SRC org :noweb-ref myvm :exports none 
unidata-server
#+END_SRC

#+NAME: azure-vm
#+BEGIN_SRC org :results silent :exports none :noweb yes 
<<myvm>>
#+END_SRC

# Setting up org babel default arguments for executing =sh= commands below. We will be using tramp for the remote execution. You should have something like this in your ssh-config:

#+BEGIN_SRC sh :eval no :exports none 
Host <<myvm>>
    User     ubuntu
    Port     22
    IdentityFile ~/.docker/machine/machines/<<myvm>>/id_rsa
    Hostname <<myvm>>.cloudapp.net
#+END_SRC

# Defaulting to using remote VM. Be careful to specify :dir ~ for the sh blocks where you do not want remote VM execution of commands.

#+BEGIN_SRC emacs-lisp :noweb yes :results silent :exports none 
  (setq-local org-babel-default-header-args:sh
              '((:dir . "/ubuntu@<<myvm>>:")))a
#+END_SRC

* Preamble

The following instructions describe how to configure a [[https://azure.microsoft.com][Microsoft Azure VM]] serving data with the [[http://www.unidata.ucar.edu/software/ldm/][LDM]], [[http://www.unidata.ucar.edu/software/thredds/current/tds/][TDS]], and [[http://sourceforge.net/projects/ramadda/][RAMADDA]]. This document assumes you have access to Azure resources though these instructions should be fairly similar on other cloud providers (e.g., Amazon). They also assume familiarity with Unix, Docker, and Unidata technology in general. You must have =sudo= priviliges on your Azure host which will hopfully be provided to you be default. You will have to be comfortable entering commands at the Unix command line. We will be using Docker images defined at the [[https://github.com/Unidata/Unidata-Dockerfiles][Unidata-Dockerfiles repository]] in addition to a configuration specifically planned for AMS 2016 demonstrations  project [[https://github.com/Unidata/Unidata-Dockerfiles/tree/master/ams2016][AMS 2016 demonstrations  project]].

:PROPERTIES:
:END:
* Preliminary Setup on Azure

The instructions assume we will create an Azure VM called =unidata-server.cloudapp.net= abbreviated to =unidata-server=. Tailor the VM name for your purposes when following this document. This VM will be our *Docker Host* from where we will run Docker containers for the LDM, TDS, and RAMADDA.

** =docker-machine=

[[https://docs.docker.com/machine/install-machine/][Install]] =docker-machine= on your local computer. =docker-machine= is a command line tool that gives users the ability to create Docker VMs on your local computer or on a cloud provider such as Azure.

** Create a VM on Azure.

The following =docker-machine= command will create a Docker VM on Azure in which you will run various Docker containers. It will take a few minutes to run (between 5 and 10 minutes). You will have to supply =azure-subscription-id= and =azure-subscription-cert= path. See these Azure =docker-machine= [[https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-docker-machine/][instructions]], if you have questions about this process. Also the the size of the VM is currently set to =ExtraLarge=. See [[https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-size-specs/][here]] to learn more about sizes for virtual machines.

# Don't execute the following code block in this buffer; it will take too long. Instead =org-babel-tangle= with a prefix argument; C-u C-c C-v t. Then execute from the command line =/tmp/azure.sh=.

{'fraction_non_alphanumeric': 0.11373149566105156, 'fraction_numerical': 0.009800918836140888, 'mean_word_length': 4.227321237993596, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 29, 'https://': 11, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5771002', 'n_tokens_mistral': 3284, 'n_tokens_neox': 3066, 'n_words': 1118}