<commit_before>*Devstack v2* is a set of python scripts and utilities to quickly deploy an OpenStack cloud.

# Goals

* To quickly build dev OpenStack environments in a clean environment (as well as start, stop, and uninstall those environments) with as little baggage as possible.
* To describe working configurations of OpenStack.
    * Which code branches work together? 
    * What do config files look like for those branches? 
    * What packages are needed for installation for a given distribution?
* To make it easier for developers to dive into OpenStack so that they can productively contribute without having to understand every part of the system at once.
* To make it easy to prototype cross-project features.

Read more at <http://devstack.org> (TBD) or <https://github.com/yahoo/Openstack-Devstack2/wiki>

**IMPORTANT:** Be sure to carefully read *stack* and any other scripts you execute before you run them, as they install software and may alter your networking configuration.  We strongly recommend that you run stack in a clean and disposable vm when you are first getting started. (*TODO* dry-run mode would be great!).

# Help

In order to determine what *stack* can do for you run the following.

    ./stack --help
 
This will typically produce:
    
    Usage: stack [options]
    
    Options:
      --version             show program's version number and exit
      -h, --help            show this help message and exit
      -a ACTION, --action=ACTION
                            action to perform, ie (install, start, stop, uninstall)
      -d DIR, --directory=DIR
                            root DIR for new components or DIR with existing
                            components (ACTION dependent)
      -c COMPONENT, --component=COMPONENT
                            stack component, ie (db, glance, horizon, keystone,
                            keystone-client, nova, nova-client, openstack-x,
                            quantum, rabbit, swift)
      -f, --force           force ACTION even if no trace found (ACTION dependent)
      -i, --ignoredeps      ignore dependencies when performing ACTION
      -e, --ensuredeps      ensure dependencies occur when performing ACTION
      -s, --listdeps        show dependencies of COMPONENT

# Stack prerequisites

* linux (tested on ubuntu 11 (aka oneiric) and rhel 6.1)
* python 2.6 or 2.7 (not tested with python 3.0)
* git
    * In ubuntu oneiric *apt-get install git*
* easy_install termcolor (used for colored console logging)
    * In ubuntu oneiric *apt-get install python-pip*
* easy_install netifaces (used to determine host ip information)
    * In ubuntu oneiric *apt-get install python-pip* and *apt-get install python-dev*
 
# Actions

*Stack* can do the following:

* __install__ OpenStack components
* __uninstall__ OpenStack components (from a previous *stack* install)
* __start__ OpenStack components (from a previous *stack* install)
* __stop__ OpenStack components (from a previous *stack* start)

Typically the interaction would be that you install a set of components and then start them. 

# Config

If you want to change which git branches or other various *stack* configurations:

Check out *conf/stack.ini* for various configuration settings applied (branches, git repositories...).  
> When you see a configuration in *stack.ini* with the format *${NAME:-DEFAULT}* this means that the environment the *stack* script is running in while be referred to and if a environment value exists for *NAME* it will be used (otherwise the *DEFAULT* will be used).  

Also check out *conf/* for various component specific settings and files. 
> Note that some of these files are templates (*TODO* rename the files so that people can see this). 
> These files may have strings of the format *%NAME%* where *NAME* will most often be adjusted to a real value by the *stack* script.  

Also check *conf/pkgs* for package listings and *conf/pips* for python packages for various distributions. 
>Note that these files are in a modified json format which allows for simple comments (lines starting with #). These comments are useful for explanations of why a version was chosen or the like.

# Starting

**!Installing in a dedicated, disposable vm is safer than installing on your dev machine!**

1. Get ubuntu 11 (oneiric) or redhat 6 (or equivalent) and create a new machine/vm with that distribution
1. Install the above prerequisites
1. *git clone git://github.com/yahoo/Openstack-Devstack2.git*

## Glance

### Installing

1. Run: *./stack -a install -d $HOME/openstack -c glance*
    * *Note:* This will also install glances dependencies (to show dependencies run *./stack -s*)
1. When prompted for passwords either press enter (to have it generate one) or enter a password.
1. Wait for it to finish...
    * On finish you should see all configurations/passwords/database dsn's that have been fetched (for future reference). 

You will also see a set of directories that end with *traces*.  
These directories contain files with the actions of exactly what occurred (files touched, directories created, packages installed).  
They are used for knowing what occurred and also for *stack's* uninstalling process. 

An example of this end state is the following:

    INFO: @devstack : Finished install of glance - check /tmp/openstack/glance/traces for traces of what happened.
    INFO: @devstack : After install your config is:
    INFO: @devstack : Passwords:
    INFO: @devstack : 	horizon_keystone_admin@passwords=<PASSWORD>
    INFO: @devstack : 	service_token@passwords=<PASSWORD>
    INFO: @devstack : 	sql@passwords=<PASSWORD>
    INFO: @devstack : Configs:
    INFO: @devstack : 	glance_branch@git=master
    INFO: @devstack : 	glance_repo@git=https://github.com/openstack/glance.git
    INFO: @devstack : 	host_ip@default=
    INFO: @devstack : 	keystone_branch@git=stable/diablo
    INFO: @devstack : 	keystone_repo@git=https://github.com/openstack/keystone.git
    INFO: @devstack : 	port@db=3306
    INFO: @devstack : 	sql_host@db=localhost
    INFO: @devstack : 	sql_user@db=root
    INFO: @devstack : 	syslog@default=0
    INFO: @devstack : 	type@db=mysql
    INFO: @devstack : Data source names:
    INFO: @devstack : 	glance=mysql://root:c910be697958ccb7@localhost:3306/glance
    INFO: @devstack : 	keystone=mysql://root:c910be697958ccb7@localhost:3306/keystone
    INFO: @devstack : Finished action [install] on Fri, 20 Jan 2012 18:29:12
    INFO: @devstack : Check [/tmp/openstack/db/traces, /tmp/openstack/keystone/traces, /tmp/openstack/glance/traces] for traces of what happened.
   
### Starting

1. Run *./stack -a start -d $HOME/openstack -c glance*
    * *Note:* This will also start glances dependencies (to show dependencies run *./stack -s*)
    * *Note:* Currently forking is done instead of running screen (*TODO* get screen working)
1. On finish you should see a list of files which will have information about what is started
    * For forking mode this will be a file with information on where the PID is, where the STDERR/STDOUT files are.
    
An example of one of these files is the following:

    $ cat /tmp/openstack/glance/traces/glance-api.fork.trace
      RUN - FORK
      PID_FN - /tmp/openstack/glance/traces/glance-api.fork.pid
      STDERR_FN - /tmp/openstack/glance/traces/glance-api.fork.stderr
      STDOUT_FN - /tmp/openstack/glance/traces/glance-api.fork.stdout


### Stopping

1. Run *./stack -a stop -d $HOME/openstack -c glance*
    * *Note:* This will also stop glances dependencies (to show dependencies run *./stack -s*)

On finish you should see something like the following:

    INFO: @devstack.component : Stopping glance-registry
    INFO: @devstack.runners.fork : Attempting to kill pid 17282
    INFO: @devstack.runners.fork : Sleeping for 1 seconds before next attempt to kill pid 17282
    INFO: @devstack.runners.fork : Attempting to kill pid 17282
    INFO: @devstack.runners.fork : Killed pid 17282 after 2 attempts
    INFO: @devstack.runners.fork : Removing pid file /tmp/openstack/glance/traces/glance-registry.fork.pid
    INFO: @devstack.runners.fork : Removing stderr file /tmp/openstack/glance/traces/glance-registry.fork.stderr
    INFO: @devstack.runners.fork : Removing stdout file /tmp/openstack/glance/traces/glance-registry.fork.stdout
    INFO: @devstack.runners.fork : Removing glance-registry trace file /tmp/openstack/glance/traces/glance-registry.fork.trace
    INFO: @devstack.component : Deleting trace file /tmp/openstack/glance/traces/start.trace

# Customizing

You can override environment variables used in *stack* by editing *stack.ini* or by sourcing a file that contains your environment overrides before your run *stack*.

## Logging

To adjust logging edit the *conf/logging.ini* file which controls the logging levels and handlers. 
> You can also change which logging file name python will select ([format defined here](http://docs.python.org/dev/library/logging.config.html)) by setting the environment variable *LOG_FILE*.
<commit_msg>Adjusted to format better.<commit_after>*Devstack v2* is a set of python scripts and utilities to quickly deploy an OpenStack cloud.

# Goals

* To quickly build dev OpenStack environments in a clean environment (as well as start, stop, and uninstall those environments) with as little baggage as possible.
* To describe working configurations of OpenStack.
    * Which code branches work together? 
    * What do config files look like for those branches? 
    * What packages are needed for installation for a given distribution?
* To make it easier for developers to dive into OpenStack so that they can productively contribute without having to understand every part of the system at once.
* To make it easy to prototype cross-project features.

Read more at <http://devstack.org> (TBD) or <https://github.com/yahoo/Openstack-Devstack2/wiki>

**IMPORTANT:** Be sure to carefully read *stack* and any other scripts you execute before you run them, as they install software and may alter your networking configuration.  We strongly recommend that you run stack in a clean and disposable vm when you are first getting started. (*TODO* dry-run mode would be great!).

# Help

In order to determine what *stack* can do for you run the following.

    ./stack --help
 
This will typically produce:
    
    Usage: stack [options]
    
    Options:
      --version             show program's version number and exit
      -h, --help            show this help message and exit
      -a ACTION, --action=ACTION
                            action to perform, ie (install, start, stop, uninstall)
      -d DIR, --directory=DIR
                            root DIR for new components or DIR with existing
                            components (ACTION dependent)
      -c COMPONENT, --component=COMPONENT
                            stack component, ie (db, glance, horizon, keystone,
                            keystone-client, nova, nova-client, openstack-x,
                            quantum, rabbit, swift)
      -f, --force           force ACTION even if no trace found (ACTION dependent)
      -i, --ignoredeps      ignore dependencies when performing ACTION
      -e, --ensuredeps      ensure dependencies occur when performing ACTION
      -s, --listdeps        show dependencies of COMPONENT

# Stack prerequisites

* linux (tested on ubuntu 11 (aka oneiric) and rhel 6.1)
* python 2.6 or 2.7 (not tested with python 3.0)
* git
    * In ubuntu oneiric *apt-get install git*
* easy_install termcolor (used for colored console logging)
    * In ubuntu oneiric *apt-get install python-pip*
* easy_install netifaces (used to determine host ip information)
    * In ubuntu oneiric *apt-get install python-pip* and *apt-get install python-dev*
 
# Actions

*Stack* can do the following:

* __install__ OpenStack components
* __uninstall__ OpenStack components (from a previous *stack* install)
* __start__ OpenStack components (from a previous *stack* install)
* __stop__ OpenStack components (from a previous *stack* start)

Typically the interaction would be that you install a set of components and then start them. 

# Config

For those of you that are brave enough to change *stack* here are some starting points.

###  conf/stack.ini

Check out *conf/stack.ini* for various configuration settings applied (branches, git repositories...).  

When you see a configuration in *stack.ini* with the format *${NAME:-DEFAULT}* this means that the environment the *stack* script is running in while be referred to and if a environment value exists for *NAME* it will be used (otherwise the *DEFAULT* will be used).  

### conf/

Check out *conf/* for various component specific settings and files. 

Note that some of these files are templates (*TODO* rename the files so that people can see this). These files may have strings of the format *%NAME%* where *NAME* will most often be adjusted to a real value by the *stack* script.  

### conf/pkgs

Check out *conf/pkgs* for package listings and *conf/pips* for python packages for various distributions. 

Note that these files are in a modified json format which allows for simple comments (lines starting with #). These comments are useful for explanations of why a version was chosen or the like.

# Starting

**!Installing in a dedicated, disposable vm is safer than installing on your dev machine!**

1. Get ubuntu 11 (oneiric) or redhat 6 (or equivalent) and create a new machine/vm with that distribution
1. Install the above prerequisites
1. *git clone git://github.com/yahoo/Openstack-Devstack2.git*

## Glance

### Installing

1. Run: *./stack -a install -d $HOME/openstack -c glance*
    * *Note:* This will also install glances dependencies (to show dependencies run *./stack -s*)
1. When prompted for passwords either press enter (to have it generate one) or enter a password.
1. Wait for it to finish...
    * On finish you should see all configurations/passwords/database dsn's that have been fetched (for future reference). 

You will also see a set of directories that end with *traces*.  
These directories contain files with the actions of exactly what occurred (files touched, directories created, packages installed).  
They are used for knowing what occurred and also for *stack's* uninstalling process. 

An example of this end state is the following:

    INFO: @devstack : Finished install of glance - check /tmp/openstack/glance/traces for traces of what happened.
    INFO: @devstack : After install your config is:
    INFO: @devstack : Passwords:
    INFO: @devstack : 	horizon_keystone_admin@passwords=<PASSWORD>
    INFO: @devstack : 	service_token@passwords=<PASSWORD>
    INFO: @devstack : 	sql@passwords=<PASSWORD>
    INFO: @devstack : Configs:
    INFO: @devstack : 	glance_branch@git=master
    INFO: @devstack : 	glance_repo@git=https://github.com/openstack/glance.git
    INFO: @devstack : 	host_ip@default=
    INFO: @devstack : 	keystone_branch@git=stable/diablo
    INFO: @devstack : 	keystone_repo@git=https://github.com/openstack/keystone.git
    INFO: @devstack : 	port@db=3306
    INFO: @devstack : 	sql_host@db=localhost
    INFO: @devstack : 	sql_user@db=root
    INFO: @devstack : 	syslog@default=0
    INFO: @devstack : 	type@db=mysql
    INFO: @devstack : Data source names:
    INFO: @devstack : 	glance=mysql://root:<PASSWORD>@<EMAIL>:3306/glance
    INFO: @devstack : 	keystone=mysql://root:<PASSWORD>@<EMAIL>:3306/keystone
    INFO: @devstack : Finished action [install] on Fri, 20 Jan 2012 18:29:12
    INFO: @devstack : Check [/tmp/openstack/db/traces, /tmp/openstack/keystone/traces, /tmp/openstack/glance/traces] for traces of what happened.
   
### Starting

1. Run *./stack -a start -d $HOME/openstack -c glance*
    * *Note:* This will also start glances dependencies (to show dependencies run *./stack -s*)
    * *Note:* Currently forking is done instead of running screen (*TODO* get screen working)
1. On finish you should see a list of files which will have information about what is started
    * For forking mode this will be a file with information on where the PID is, where the STDERR/STDOUT files are.
    
An example of one of these files is the following:

    $ cat /tmp/openstack/glance/traces/glance-api.fork.trace
      RUN - FORK
      PID_FN - /tmp/openstack/glance/traces/glance-api.fork.pid
      STDERR_FN - /tmp/openstack/glance/traces/glance-api.fork.stderr
      STDOUT_FN - /tmp/openstack/glance/traces/glance-api.fork.stdout


### Stopping

1. Run *./stack -a stop -d $HOME/openstack -c glance*
    * *Note:* This will also stop glances dependencies (to show dependencies run *./stack -s*)

On finish you should see something like the following:

    INFO: @devstack.component : Stopping glance-registry
    INFO: @devstack.runners.fork : Attempting to kill pid 17282
    INFO: @devstack.runners.fork : Sleeping for 1 seconds before next attempt to kill pid 17282
    INFO: @devstack.runners.fork : Attempting to kill pid 17282
    INFO: @devstack.runners.fork : Killed pid 17282 after 2 attempts
    INFO: @devstack.runners.fork : Removing pid file /tmp/openstack/glance/traces/glance-registry.fork.pid
    INFO: @devstack.runners.fork : Removing stderr file /tmp/openstack/glance/traces/glance-registry.fork.stderr
    INFO: @devstack.runners.fork : Removing stdout file /tmp/openstack/glance/traces/glance-registry.fork.stdout
    INFO: @devstack.runners.fork : Removing glance-registry trace file /tmp/openstack/glance/traces/glance-registry.fork.trace
    INFO: @devstack.component : Deleting trace file /tmp/openstack/glance/traces/start.trace

# Customizing

You can override environment variables used in *stack* by editing *stack.ini* or by sourcing a file that contains your environment overrides before your run *stack*.

## Logging

To adjust logging edit the *conf/logging.ini* file which controls the logging levels and handlers. 

* You can also change which logging file name python will select ([format defined here](http://docs.python.org/dev/library/logging.config.html)) by setting the environment variable *LOG_FILE*.

{'fraction_non_alphanumeric': 0.08173183123481334, 'fraction_numerical': 0.00905677048818202, 'mean_word_length': 3.46474358974359, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 22, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5226807', 'n_tokens_mistral': 5294, 'n_tokens_neox': 5008, 'n_words': 2142}