<commit_before>#Topology Converter Documentation

##Table of Contents
* [Glossary](#glossary)
* [Features](#features)
* [Installation](#installation)
* [Using Topology Converter](#using-topology-converter)
  * [The Basic Workflow](#the-basic-workflow)
  * [What is it doing?](#what-is-happening-when-you-run-topology-converter)
* [Optional Features](#optional-features)
  * [Providers](#providers)
  * [Faked Devices](#faked-devices)
  * [Functional Defaults](#functional-defaults)
  * [Boot Ordering](#boot-ordering)
  * [MAC Address Handout](#mac-handout)
  * [Ansible Hostfile Generation](#ansible-hostfile-generation)
  * [Inter-Hypervisor Simulation](#inter-hypervisor-simulation)
  * [Custom Templates](#custom-templates)
  * [Passthrough Attributes](#passthrough-attributes)
  * [Provisioning Scripts](#provisioning-scripts)
  * [Specifying an Ansible Playbooks](#ansible-playbooks)
  * [Debugging Mode](#debugging-mode)
* [Miscellaneous Info](#miscellaneous-info)
* [Example Topologies](#example-topologies)
  * [The Reference Topology](#the-reference-topology)
  * [2 Switch](#2-switch-topology)
  * [1 Switch 1 Server](#1-switch-1-server-topology)
  * [2 Switch 1 Server](#2-switch-1-server-topology)
  * [3 Switch Circular](#3-switch-circular-topology)


```
                                                                       +---------+
                                                                  +--> | LibVirt |
                    +----------------------+                      |    +---------+
+--------------+    |                      |    +--------------+  |
| Topology.dot +--> |  Topology-Converter  +--> | Vagrantfile  +--+
+--------------+    |                      |    +--------------+  |
                    +----------------------+                      |    +------------+
                                                                  +--> | VirtualBox |
                                                                       +------------+
```

##Glossary:
* **Topology File** -- Usually a file ending in the ".dot" suffix. This file describes the network topology link-by-link. Written in https://en.wikipedia.org/wiki/DOT_(graph_description_language). This file can be the same one used as part of the [Perscriptive Topology Manager (PTM) feature](https://docs.cumulusnetworks.com/display/DOCS/Prescriptive+Topology+Manager+-+PTM) in Cumulus Linux.
* **Provider** -- similar to a hypervisor, providers are Vagrant's term for the device that is either directoy hosting the VM (virtualbox) or the subsystem that vagrant is communicating with to further orchestratrate the creation of the VM (libvirt)
* **Interface Remapping** -- Interface remapping is the process by which interfaces are renamed to match the interfaces specified in the topology file. Interface Remapping uses UDEV rules that are orchestrated in the Vagrantfile and applied by the apply_udev.py script on the machines under simulation. This process allows a device in a topologyfile to simulate ports like "swp49" without having to simulate ports swp1-48. See the "[Miscellaneous Info](#miscellaneous-info)" Section for additional information.


##Features
* Converts a topology file "topology.dot" into a Vagrantfile
* 1 file is modified by the user to create a suitable Vagrantfile (topology.dot)
* Handles interface remapping on Vx instances (and hosts) to match the interfaces used in the provided topology file
* Removes extra Ruby-based logic from the Vagrantfile making a simple human-readable output
* Can generate Vagrantfiles that contain servers and switches and anything else that can be found in a Vagrant Box image
* Does not require Vagrant/Virtualbox/libvirt to be installed to create the Vagrantfile
* Supports the Virtualbox and Libvirt Vagrant Providers (Hypervisors)

##Installation

###Ubuntu
Both 16.04 and 14.04.

```
sudo apt install python-pip
sudo pip install --upgrade pip
sudo pip install setuptools
sudo pip install pydotplus
sudo pip install jinja2
```

###Mac


```
sudo easy_install pip
sudo pip install --upgrade pip
sudo pip install setuptools
sudo pip install pydotplus
sudo pip install jinja2
```


##Using Topology Converter
To use Topology Converter [TC] you need to work with one file: topology.dot
 
The actual name of the topology file is irrelevant. Your "topology.dot" file could be named "HAPPYHAPPYJOYJOY.dot" and that would be just fine.
 
###The Basic Workflow
**1). Create a Topology File** 
Create a topology.dot file or borrow a provided ".dot" file from the "topology_converter/examples/" directory
In this example, we'll work with the topology_converter / examples / 2switch_1server.dot file shown below:

```
graph dc1 {
 "leaf1" [function="leaf" os="CumulusCommunity/cumulus-vx" memory="200" config="./helper_scripts/extra_switch_config.sh"]
 "leaf2" [function="leaf" os="CumulusCommunity/cumulus-vx" memory="200" config="./helper_scripts/extra_switch_config.sh"]
 "server1" [function="host" os="boxcutter/ubuntu1404" memory="400" ubuntu=True config="./helper_scripts/extra_server_config.sh"]
   "leaf1":"swp40" -- "leaf2":"swp40"
   "leaf1":"swp50" -- "leaf2":"swp50"
   "server1":"eth1" -- "leaf1":"swp1"
   "server1":"eth2" -- "leaf2":"swp1"
}
```

Place this topology.dot file in the same directory as topology_converter.py (or any subdirectory beneath the directory which contains topology_converter.py)

**2). Convert it to a Vagrantfile**

```
      $ python ./topology_converter.py ./topology.dot
```

or if using Libvirt:


```
      $ python ./topology_converter.py ./topology.dot -p libvirt
```

**3). Start the Simulation**

```
      $ vagrant up
```

or if using Libvirt:


```
      $ vagrant up --provider=libvirt
```

###What is happening when you run Topology Converter?
1. When topology_converter (TC) is called, TC reads the provided topology file line by line and learns information about each node and each link in the topology.
2. This information is stored in a variables datastructure. (View this datastructure using the "python ./topology_converter.py [topology_file] -dd" option)
3. A jinja2 template "Vagrantfile.j2" (stored in the /templates directory) is used to render a Vagrantfile based on the variables datastructure.


##Optional Features (Everything Else)

###Providers
Topology Converter supports the use of two providers, Virtualbox and Libvirt (/w KVM). Virtualbox is the default provider. 

To use Libvirt/KVM specify the "-p libvirt" option

```
-p PROVIDER, --provider PROVIDER
                      specifies the provider to be used in the Vagrantfile,
                      script supports "virtualbox" or "libvirt", default is
                      virtualbox.
```


###Faked Devices
In virtual environments it may not always be possible to simulate every single device due to memory restrictions, interest, proprietary OSes etc. Faked devices give Topology Converter a way to know that a device in a topology.dot file is not actually going to be simulated. However when a faked device is connected to a real device the real device MUST create an interface as if the faked device was actually present.
Creating the interface allows for the simulation of interface configuration that would face the faked device. If no interface was present, the interface configuration may fail which could inhibit automation tooling tests and ultimately provide a less accurate simulation. To specify that a device is to be faked, add it to the "function" attribute of the node definition in the topology file.

```
graph dc1 {
 "leaf1" [function="fake"]
 "leaf2" [function="leaf" os="CumulusCommunity/cumulus-vx" memory="200" config="./helper_scripts/extra_switch_config.sh"]
   "leaf1":"swp40" -- "leaf2":"swp40"
   "leaf1":"swp50" -- "leaf2":"swp50"
}
```

###Functional Defaults
Functional defaults provide basic options for memory and OS when using pre-defined functions. Presently the functional defaults are defined as follows but can be overwritten by manually specifying the associated attribute.

**For Functions:** "oob-switch" "exit" "spine" "leaf"

**Functional Defaults are:**
* os="CumulusCommunity/cumulus-vx"
* memory="300"

**For Functions:** "oob-server" and "host"

**Functional Defaults are:**
* os="boxcutter/ubuntu1604"
* memory="500"
<commit_msg>updated documentation, moved functional defaults out of Optional Features<commit_after>#Topology Converter Documentation

##Table of Contents
* [Glossary](#glossary)
* [Features](#features)
* [Installation](#installation)
* [Using Topology Converter](#using-topology-converter)
  * [The Basic Workflow](#the-basic-workflow)
  * [What is it doing?](#what-is-happening-when-you-run-topology-converter)
  * [Functional Defaults](#functional-defaults)
* [Optional Features](#optional-features)
  * [Providers](#providers)
  * [Faked Devices](#faked-devices)
  * [Boot Ordering](#boot-ordering)
  * [MAC Address Handout](#mac-handout)
  * [Ansible Hostfile Generation](#ansible-hostfile-generation)
  * [Inter-Hypervisor Simulation](#inter-hypervisor-simulation)
  * [Custom Templates](#custom-templates)
  * [Passthrough Attributes](#passthrough-attributes)
  * [Provisioning Scripts](#provisioning-scripts)
  * [Specifying an Ansible Playbooks](#ansible-playbooks)
  * [Debugging Mode](#debugging-mode)
* [Miscellaneous Info](#miscellaneous-info)
* [Example Topologies](#example-topologies)
  * [The Reference Topology](#the-reference-topology)
  * [2 Switch](#2-switch-topology)
  * [1 Switch 1 Server](#1-switch-1-server-topology)
  * [2 Switch 1 Server](#2-switch-1-server-topology)
  * [3 Switch Circular](#3-switch-circular-topology)


```
                                                                       +---------+
                                                                  +--> | LibVirt |
                    +----------------------+                      |    +---------+
+--------------+    |                      |    +--------------+  |
| Topology.dot +--> |  Topology-Converter  +--> | Vagrantfile  +--+
+--------------+    |                      |    +--------------+  |
                    +----------------------+                      |    +------------+
                                                                  +--> | VirtualBox |
                                                                       +------------+
```

##Glossary:
* **Topology File** -- Usually a file ending in the ".dot" suffix. This file describes the network topology link-by-link. Written in https://en.wikipedia.org/wiki/DOT_(graph_description_language). This file can be the same one used as part of the [Perscriptive Topology Manager (PTM) feature](https://docs.cumulusnetworks.com/display/DOCS/Prescriptive+Topology+Manager+-+PTM) in Cumulus Linux.
* **Provider** -- similar to a hypervisor, providers are Vagrant's term for the device that is either directoy hosting the VM (virtualbox) or the subsystem that vagrant is communicating with to further orchestratrate the creation of the VM (libvirt)
* **Interface Remapping** -- Interface remapping is the process by which interfaces are renamed to match the interfaces specified in the topology file. Interface Remapping uses UDEV rules that are orchestrated in the Vagrantfile and applied by the apply_udev.py script on the machines under simulation. This process allows a device in a topologyfile to simulate ports like "swp49" without having to simulate ports swp1-48. See the "[Miscellaneous Info](#miscellaneous-info)" Section for additional information.


##Features
* Converts a topology file "topology.dot" into a Vagrantfile
* 1 file is modified by the user to create a suitable Vagrantfile (topology.dot)
* Handles interface remapping on Vx instances (and hosts) to match the interfaces used in the provided topology file
* Removes extra Ruby-based logic from the Vagrantfile making a simple human-readable output
* Can generate Vagrantfiles that contain servers and switches and anything else that can be found in a Vagrant Box image
* Does not require Vagrant/Virtualbox/libvirt to be installed to create the Vagrantfile
* Supports the Virtualbox and Libvirt Vagrant Providers (Hypervisors)

##Installation

###Ubuntu
Both 16.04 and 14.04.

```
sudo apt install python-pip
sudo pip install --upgrade pip
sudo pip install setuptools
sudo pip install pydotplus
sudo pip install jinja2
```

###Mac


```
sudo easy_install pip
sudo pip install --upgrade pip
sudo pip install setuptools
sudo pip install pydotplus
sudo pip install jinja2
```


##Using Topology Converter
To use Topology Converter [TC] you need to work with one file: topology.dot
 
The actual name of the topology file is irrelevant. Your "topology.dot" file could be named "HAPPYHAPPYJOYJOY.dot" and that would be just fine.
 
###The Basic Workflow
**1). Create a Topology File** 
Create a topology.dot file or borrow a provided ".dot" file from the "topology_converter/examples/" directory
In this example, we'll work with the topology_converter / examples / 2switch_1server.dot file shown below:

```
graph dc1 {
 "leaf1" [function="leaf" os="CumulusCommunity/cumulus-vx" memory="200" config="./helper_scripts/extra_switch_config.sh"]
 "leaf2" [function="leaf" os="CumulusCommunity/cumulus-vx" memory="200" config="./helper_scripts/extra_switch_config.sh"]
 "server1" [function="host" os="boxcutter/ubuntu1404" memory="400" ubuntu=True config="./helper_scripts/extra_server_config.sh"]
   "leaf1":"swp40" -- "leaf2":"swp40"
   "leaf1":"swp50" -- "leaf2":"swp50"
   "server1":"eth1" -- "leaf1":"swp1"
   "server1":"eth2" -- "leaf2":"swp1"
}
```

Place this topology.dot file in the same directory as topology_converter.py (or any subdirectory beneath the directory which contains topology_converter.py)

**2). Convert it to a Vagrantfile**

```
      $ python ./topology_converter.py ./topology.dot
```

or if using Libvirt:


```
      $ python ./topology_converter.py ./topology.dot -p libvirt
```

**3). Start the Simulation**

```
      $ vagrant up
```

or if using Libvirt:


```
      $ vagrant up --provider=libvirt
```

###What is happening when you run Topology Converter?
1. When topology_converter (TC) is called, TC reads the provided topology file line by line and learns information about each node and each link in the topology.
2. This information is stored in a variables datastructure. (View this datastructure using the "python ./topology_converter.py [topology_file] -dd" option)
3. A jinja2 template "Vagrantfile.j2" (stored in the /templates directory) is used to render a Vagrantfile based on the variables datastructure.

###Functional Defaults
Functional defaults provide basic options for memory and OS when using pre-defined functions. Presently the functional defaults are defined as follows but can be overwritten by manually specifying the associated attribute.

**For Functions:** "oob-switch" "exit" "spine" "leaf"

**Functional Defaults are:**
* os="CumulusCommunity/cumulus-vx"
* memory="300"

**For Functions:** "oob-server" and "host"

**Functional Defaults are:**
* os="boxcutter/ubuntu1604"
* memory="500"

Note: See more information about what functions are used for in the [Faked Devices](#faked-devices) and [Boot Ordering](#boot-ordering) sections.


##Optional Features (Everything Else)

###Providers
Topology Converter supports the use of two providers, Virtualbox and Libvirt (/w KVM). Virtualbox is the default provider. 

To use Libvirt/KVM specify the "-p libvirt" option

```
-p PROVIDER, --provider PROVIDER
                      specifies the provider to be used in the Vagrantfile,
                      script supports "virtualbox" or "libvirt", default is
                      virtualbox.
```


###Faked Devices
In virtual environments it may not always be possible to simulate every single device due to memory restrictions, interest, proprietary OSes etc. Faked devices give Topology Converter a way to know that a device in a topology.dot file is not actually going to be simulated. However when a faked device is connected to a real device the real device MUST create an interface as if the faked device was actually present.
Creating the interface allows for the simulation of interface configuration that would face the faked device. If no interface was present, the interface configuration may fail which could inhibit automation tooling tests and ultimately provide a less accurate simulation. To specify that a device is to be faked, add it to the "function" attribute of the node definition in the topology file.

```
graph dc1 {
 "leaf1" [function="fake"]
 "leaf2" [function="leaf" os="CumulusCommunity/cumulus-vx" memory="200" config="./helper_scripts/extra_switch_config.sh"]
   "leaf1":"swp40" -- "leaf2":"swp40"
   "leaf1":"swp50" -- "leaf2":"swp50"
}
```
{'fraction_non_alphanumeric': 0.11637776449491931, 'fraction_numerical': 0.012313209802749551, 'mean_word_length': 3.3367029548989113, 'pattern_counts': {'":': 24, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3589765', 'n_tokens_mistral': 4918, 'n_tokens_neox': 4475, 'n_words': 1702}