<commit_before>#+TITLE: mk-inventory
#+AUTHOR: <NAME>
#+EMAIL: <EMAIL>
#+MACRO: this-file  mk-inventory.org
#+MACRO: project-name: Big-Data-Stack
#+PROPERTY: header-args :tangle mk-inventory :comments both :padline true

* About

This is a literate python script written in Emacs Org-Mode.  If making
changes, make sure to edit {{{this-file}}}, not the tangled
(generated) file.

To update the code, open {{{this-file}}} in Emacs (with Org-Mode
installed) and execute =C-c C-v t=.  While reading the tangled file
you can jump back to the definition =C-u C-c C-v t=.

Note: if you are reading this on GitHub, be aware that the org-mode
renderer doesn't respect indentation. Thus if the definition of a
class is broken into several src blocks, the indentation (as shown in
GitHub) will be reset to 0.


* Purpose

Given the name of a virtual cluster and associated IP addresses,
generate an ansible inventory that works with {{{project-name}}}, and
any associated =host_vars= or =group_vars= file.


* Requirements

In order to run this code you need:

- Python 2.7
- a name of a virtual cluster (alphanumeric with underscores or dashes)
- IP addresses that you can successfully SSH into
- the nodes of the virtual cluster must have Python 2.7 installed


* Imports

We are using the =argparse= module that comes with Python.  While
others argument parsing libraries exists such as =docopts=, =argparse=
is easy to use and (importantly) adds to extra dependencies.

#+BEGIN_SRC python
import argparse
#+END_SRC

We also need the =defaultdict= from the =collections= module.
#+BEGIN_SRC python
from collections import defaultdict
#+END_SRC

We'll also be generating some strings so we'll use =StringIO= to do so effeciently.

#+BEGIN_SRC python
from StringIO import StringIO
#+END_SRC

We'll also need to be reading and writing yaml files, so we should import:

#+BEGIN_SRC python
import yaml
#+END_SRC

We also will be interfacing with the operating system and dealing with
file paths, so we need these modules

#+BEGIN_SRC python
import os
import os.path
#+END_SRC

We will be using =itertools= (namely =cycle=) when assigning nodes to groups

#+BEGIN_SRC python
import itertools
#+END_SRC

* Logging

We'll use the standard built-in =logging= module with a default level of INFO.
In order to not dump too much info we'll simplify the output format.

#+BEGIN_SRC python
import logging
logging.basicConfig(level=logging.INFO,
                    format='%(levelname)-9s %(message)s')
logger = logging.getLogger(__name__)
#+END_SRC

* Getting User Options

This is a convenient description of this program that will be shown to users.

#+BEGIN_SRC python
_DESCRIPTION = """
Generate a valid Ansible inventory file and associated host_vars and group_vars.
You need to provide a name for the virtual cluster, and SSH-able IP addresses.
"""
#+END_SRC

#+BEGIN_SRC python
def get_opts():
    p = argparse.ArgumentParser(
        description=_DESCRIPTION,
        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    p.add_argument('-n', '--name', metavar='TEXT', default='myvc',
                   help='The name for the virtual cluster')
    p.add_argument('addresses', nargs='+', metavar='IP',
                   help='IP addresses of the cluster')

    return p.parse_args()
#+END_SRC


* Classes

** Node

The =Node= represents everything that is known about a node: namely, name and IP address<commit_msg>provide example in help message<commit_after>#+TITLE: mk-inventory
#+AUTHOR: <NAME>
#+EMAIL: <EMAIL>
#+MACRO: this-file  mk-inventory.org
#+MACRO: project-name: Big-Data-Stack
#+PROPERTY: header-args :tangle mk-inventory :comments both :padline true

* About

This is a literate python script written in Emacs Org-Mode.  If making
changes, make sure to edit {{{this-file}}}, not the tangled
(generated) file.

To update the code, open {{{this-file}}} in Emacs (with Org-Mode
installed) and execute =C-c C-v t=.  While reading the tangled file
you can jump back to the definition =C-u C-c C-v t=.

Note: if you are reading this on GitHub, be aware that the org-mode
renderer doesn't respect indentation. Thus if the definition of a
class is broken into several src blocks, the indentation (as shown in
GitHub) will be reset to 0.


* Purpose

Given the name of a virtual cluster and associated IP addresses,
generate an ansible inventory that works with {{{project-name}}}, and
any associated =host_vars= or =group_vars= file.


* Requirements

In order to run this code you need:

- Python 2.7
- a name of a virtual cluster (alphanumeric with underscores or dashes)
- IP addresses that you can successfully SSH into
- the nodes of the virtual cluster must have Python 2.7 installed


* Imports

We are using the =argparse= module that comes with Python.  While
others argument parsing libraries exists such as =docopts=, =argparse=
is easy to use and (importantly) adds to extra dependencies.

#+BEGIN_SRC python
import argparse
#+END_SRC

We also need the =defaultdict= from the =collections= module.
#+BEGIN_SRC python
from collections import defaultdict
#+END_SRC

We'll also be generating some strings so we'll use =StringIO= to do so effeciently.

#+BEGIN_SRC python
from StringIO import StringIO
#+END_SRC

We'll also need to be reading and writing yaml files, so we should import:

#+BEGIN_SRC python
import yaml
#+END_SRC

We also will be interfacing with the operating system and dealing with
file paths, so we need these modules

#+BEGIN_SRC python
import os
import os.path
#+END_SRC

We will be using =itertools= (namely =cycle=) when assigning nodes to groups

#+BEGIN_SRC python
import itertools
#+END_SRC

* Logging

We'll use the standard built-in =logging= module with a default level of INFO.
In order to not dump too much info we'll simplify the output format.

#+BEGIN_SRC python
import logging
logging.basicConfig(level=logging.INFO,
                    format='%(levelname)-9s %(message)s')
logger = logging.getLogger(__name__)
#+END_SRC

* Getting User Options

This is a convenient description of this program that will be shown to users.

#+BEGIN_SRC python
_DESCRIPTION = """
Generate a valid Ansible inventory file and associated host_vars and group_vars.
You need to provide a name for the virtual cluster, and SSH-able IP addresses.

The inventory will be written to stdout and log messages to stderr.

Example:

  ./mk-inventory -n foo 192.168.1.10{1,2,3,4,5} >inventory.txt
"""
#+END_SRC

#+BEGIN_SRC python
def get_opts():
    p = argparse.ArgumentParser(
        description=_DESCRIPTION,
        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    p.add_argument('-n', '--name', metavar='TEXT', default='myvc',
                   help='The name for the virtual cluster')
    p.add_argument('addresses', nargs='+', metavar='IP',
                   help='IP addresses of the cluster')

    return p.parse_args()
#+END_SRC


* Classes

** Node

The =Node= represents everything that is known about a node: namely, name and IP address
{'fraction_non_alphanumeric': 0.08293940696175334, 'fraction_numerical': 0.0037243947858473, 'mean_word_length': 4.206562266964951, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4464378', 'n_tokens_mistral': 2153, 'n_tokens_neox': 2048, 'n_words': 919}