<commit_before># Cloud Foundation Toolkit - User Guide

<!-- TOC -->

- [Overview](#overview)
- [CFT Configs](#cft-configs)
    - [Extra YAML Directives](#extra-yaml-directives)
        - [name](#name)
        - [project](#project)
        - [description](#description)
    - [Extra Features](#extra-features)
        - [Cross-deployment References with the `$(out)` Tag](#cross-deployment-references-with-the-out-tag)
        - [Jinja Templating](#jinja-templating)
    - [Samples](#samples)
        - [network.yaml](#networkyaml)
        - [firewall.yaml](#firewallyaml)
        - [instance.yaml](#instanceyaml)
- [Templates](#templates)
- [Toolkit Installation and Configuration](#toolkit-installation-and-configuration)
    - [Installing Prerequisites](#installing-prerequisites)
        - [Python 2.7 + pip](#python-27--pip)
        - [Google Cloud SDK](#google-cloud-sdk)
    - [Getting the CFT Code](#getting-the-cft-code)
    - [Installing the CFT](#installing-the-cft)
    - [Uninstalling the CFT](#uninstalling-the-cft)
    - [Updating the CFT](#updating-the-cft)
- [CLI Usage](#cli-usage)
    - [Syntax](#syntax)
    - [Actions](#actions)
        - [The "create" Action](#the-create-action)
        - [The "update" Action](#the-update-action)
        - [The "apply" Action](#the-apply-action)
        - [The "delete" Action](#the-delete-action)

<!-- /TOC -->

## Overview

The GCP Deployment Manager service does not support cross-deployment
references, and the `gcloud` utility does not support concurrent deployment of
multiple inter-dependent configs. The `Cloud Foundation toolkit` (henceforth,
`CFT`) expands the capabilities of Deployment Manager and `gcloud` to support
the following scenarios:

- Creation, update, and deletion of multiple deployments in a single operation
  which:
  - Accepts multiple config files as input
  - Automatically resolves dependencies between these configs
  - Creates/updates deployments in the dependency-stipulated order, or
    deletes deployments in a reverse dependency order
- Cross-deployment (including cross-project) referencing of deployment outputs,
  which removes the need for hard-coding many parameters in the configs

For example, if config file `A` contained all network resources, config file
`B` contained all instances, and config `C` contained firewall rules, router,
and VPN, in `gcloud` you would need to *manually* define the config deployment
order according to the resource dependencies. The VPN would depend on the cloud
router, both of them would depend on the network, etc. The `CFT` computes the
dependencies *automatically*, which eliminates the need for manual deployment
ordering.

`Note:` This User Guide assumes that you are familiar with the Google Cloud SDK
operations related to resource deployment and management. For additional
information, refer to the
[SDK documentation](https://cloud.google.com/sdk/docs/).

The CFT includes:

- A command-line interface (henceforth, CLI) that deploys resources defined in
  single or multiple CFT-compliant config files
- A comprehensive set of production-ready resource [templates](#templates) that follow
  Google's best practices, which can be used with the CFT or the `gcloud`
  utility. (`gcloud` is part of the Google Cloud SDK).

You can use the CFT "as is" or modify it to suit your specific needs. Instructions
and recommendations for the CFT code modifications are in the
[CFT Developer Guide](tool_dev_guide.md).

## CFT Configs

To use the CFT, you need to first create the config files for the desired
deployments. These configs are YAML structures very similar to, and compatible
with, the `gcloud` config files. The difference is that they contain extra YAML
directives and features to support the expanded capabilities of the CFT
(multi-config deployment and cross-deployment references).

### Extra YAML Directives

#### name

This directive is used to specify the name of the deployment; for example:

```yaml
name: my-network
```

If not specified, the name of the deployment is inferred from the config
file name. For example, if the path to the config file is
`path/to/configs/my-network.yaml`, and the config does not specify the `name`
directive, the deployment name is set to `my-network`. This is meant as a
workaround for maintaining compatibility between the `CFT` and `gcloud` configs.
However, **it is strongly recommended that the `name` directive is specified**.

#### project

This directive defines the project in which the resource is deployed; for
example:

```yaml
project: my-project
```

While this directive is optional, **its use in your configs is highly
recommended**. In addition to the project directive in the config file,
the project for a deployment to be created in can be specified by other means.
The order of precedence is as follows:

1. The `--project` command-line option. If a project is specified via this
   option, all configs in the run use that project. This is a way of
   quickly overriding the project specified in a config file, which should be
   used with caution.
2. The `project` directive in the config file.
3. The `CLOUD_FOUNDATION_PROJECT_ID` environment variable.
4. The "default project" configured with the GCP SDK.

`Note:` When deployments utilize cross-project resources, the `project`
directive becomes mandatory in at least one of the deployments.

#### description

This directive is the deployment description, which allows you
to add some documentation to your configs; for example:

```yaml
description: My firewall deployment for {{environment}} environment
```

### Extra Features

#### Cross-deployment References with the `$(out)` Tag

A config/deployment can specify a dependency on another deployment's output
without the need to create the dependent deployment in advance. This is the
mechanism the CFT uses to determine the order of execution of the deployments.

```yaml
$(out.<project>.<deployment>.<resource>.<output>)

# or

$(out.<deployment>.<resource>.<output>)
```

wherein:

- `$(out)` is the prefix that indicates that the value references an output
  from a resource defined in an external deployment (in another config file)
- `project` is the ID of the project in which the external deployment is
   created
- `deployment` is the he name of the external deployment (config) that
  defines the referenced resource
- `resource` is the DM name of the referenced resource
- `output` is the name of the output parameter to be referenced

The above construct works very similarly to Deployment Manager's
`$(ref.<resource>.<property>)`. However, it allows defining not only references
to resource properties not only *within* a deployment, but also
*inter-deployment/inter-project* references, using deployment outputs. The
value of output of a dependent deployment is only looked up during the current
deployment's execution, which allows you to create config files without knowing
in advance the actual values of the outputs in the dependent deployments, or
even having to create these deployments.

For example:

```yaml
network: $(out.my-network-prod.my-network-prod.name)
```

#### Jinja Templating

All configs submitted via the CFT CLI are rendered by the [Jinja Template
Engine](http://jinja.pocoo.org/). This supports compact code by using the DRY
pattern. For example, by using variable substitution and `for loops`:

```yaml
{% set environment = 'prod' %}
{% set applications = ['app1', 'app2', 'app3'] %}

name: my-network-{{environment}}
description: Network deployment for {{environment}} environment
project: sourced-gus-1
imports:
  - path: templates/network/network.py
resources:
{% for application in applications %}
  - type: templates/network/network.py
    name: {{application}}-{{environment}}-network
    properties:
      autoCreateSubnetworks: false
{% endfor %}
```

An alternative to using Jinja in your configs is to write wrapper DM Python
templates and reference these templates in your configs (see the
[Templates](#templates) section).

### Samples

Following are three sample config files that illustrate the above directives
and features. These will be used as examples in the action-specific sections of
this User Guide:

- [network.yaml](#network.yaml) - two networks that have no dependencies
- [firewall.yaml](#firewall.yaml) - two firewall rules, which depend on the
  corresponding networks
- [instance.yaml](#instance.yaml) - one VM instance, which depends on the
  network

#### network.yaml

```yaml
name: my-networks
description: my networks deployment

imports:
  - path: templates/network/network.py

resources:
  - type: templates/network/network.py
    name: my-network-prod
    properties:
      autoCreateSubnetworks: true

  - type: templates/network/network.py
    name: my-network-dev
    properties:
      autoCreateSubnetworks: false
```

#### firewall.yaml

```yaml
name: my-firewalls
description: My firewalls deployment

imports:
  - path: templates/firewall/firewall.py
resources:
  - type: templates/firewall/firewall.py
    name: my-firewall-prod
    properties:
      network: $(out.my-networks.my-network-prod.name)
      rules:
        - name: allow-proxy-from-inside-prod
      allowed:
            - IPProtocol: tcp
              ports:
                - "80"
                - "444"
          description: This rule allows connectivity to the HTTP proxies
          direction: INGRESS
          sourceRanges:
            - 10.0.0.0/8
        - name: allow-dns-from-inside-prod
          allowed:
            - IPProtocol: udp
              ports:
                - "53"
            - IPProtocol: tcp
              ports:
                - "53"
          description: this rule allows DNS queries to google's 8.8.8.8
          direction: EGRESS
          destinationRanges:
            - 8.8.8.8/32
  - type: templates/firewall/firewall.py
    name: my-firewall-dev
    properties:
      network: $(out.my-networks.my-network-dev.name)
      rules:
        - name: allow-proxy-from-inside-dev
          allowed:
            - IPProtocol: tcp
              ports:
                - "80"
                - "444"
          description: This rule allows connectivity to the HTTP proxies
          direction: INGRESS
          sourceRanges:
            - 10.0.0.0/8
```

#### instance.yaml

```yaml
name: my-instance-prod-1
description: My instance deployment for prod environment

imports:
  - path: templates/instance/instance.py
    name: instance.py

resources:
  - name: my-instance-prod-1
    type: instance.py
    properties:
      zone: us-central1-a
      diskImage: projects/ubuntu-os-cloud/global/images/family/ubuntu-1804-lts
      diskSizeGb: 100
      machineType: f1-micro
      diskType: pd-ssd
      network: $(out.my-networks.my-network-prod.name)
      metadata:
        items:
          - key: startup-script
            value: sudo apt-get update && sudo apt-get install -y nginx
```

## Templates

CFT-compliant configs can use templates written in Python or Jinja2. [Templates
included in the toolkit](../templates/README.md) are recommended (although not mandatory)
as they offer robust functionality, ease of use, and adherence to best
practices.

You can use the templates included in our library "as is," and/or modify them
to suit your needs, as well as develop your own templates. Instructions and
recommendations for template development are in the
[Template Developer Guide](template_dev_guide.md).  

## Toolkit Installation and Configuration

This toolkit was developed primarily on/for Linux. Therefore, the Linux platform
is expected to offer the most seamless user experience.

### Installing Prerequisites

#### Python 2.7 + pip

Follow your OS package manager instructions. For example, for Ubuntu:

```shell
sudo apt-get install python2.7
sudo apt-get install python-pip
```

#### Google Cloud SDK

1. Install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/quickstarts).
2. Ensure that the `gcloud` command is in the user's PATH:

```shell
which gcloud
```

### Getting the CFT Code

Proceed as follows:

```shell
git clone https://github.com/GoogleCloudPlatform/deploymentmanager-samples
cd deploymentmanager-samples/
git checkout cloud-foundation  # CFT is in a branch for now
```

### Installing the CFT
<commit_msg>Doc update - merged to master<commit_after># Cloud Foundation Toolkit - User Guide

<!-- TOC -->

- [Overview](#overview)
- [CFT Configs](#cft-configs)
    - [Extra YAML Directives](#extra-yaml-directives)
        - [name](#name)
        - [project](#project)
        - [description](#description)
    - [Extra Features](#extra-features)
        - [Cross-deployment References with the `$(out)` Tag](#cross-deployment-references-with-the-out-tag)
        - [Jinja Templating](#jinja-templating)
    - [Samples](#samples)
        - [network.yaml](#networkyaml)
        - [firewall.yaml](#firewallyaml)
        - [instance.yaml](#instanceyaml)
- [Templates](#templates)
- [Toolkit Installation and Configuration](#toolkit-installation-and-configuration)
    - [Installing Prerequisites](#installing-prerequisites)
        - [Python 2.7 + pip](#python-27--pip)
        - [Google Cloud SDK](#google-cloud-sdk)
    - [Getting the CFT Code](#getting-the-cft-code)
    - [Installing the CFT](#installing-the-cft)
    - [Uninstalling the CFT](#uninstalling-the-cft)
    - [Updating the CFT](#updating-the-cft)
- [CLI Usage](#cli-usage)
    - [Syntax](#syntax)
    - [Actions](#actions)
        - [The "create" Action](#the-create-action)
        - [The "update" Action](#the-update-action)
        - [The "apply" Action](#the-apply-action)
        - [The "delete" Action](#the-delete-action)

<!-- /TOC -->

## Overview

The GCP Deployment Manager service does not support cross-deployment
references, and the `gcloud` utility does not support concurrent deployment of
multiple inter-dependent configs. The `Cloud Foundation toolkit` (henceforth,
`CFT`) expands the capabilities of Deployment Manager and `gcloud` to support
the following scenarios:

- Creation, update, and deletion of multiple deployments in a single operation
  which:
  - Accepts multiple config files as input
  - Automatically resolves dependencies between these configs
  - Creates/updates deployments in the dependency-stipulated order, or
    deletes deployments in a reverse dependency order
- Cross-deployment (including cross-project) referencing of deployment outputs,
  which removes the need for hard-coding many parameters in the configs

For example, if config file `A` contained all network resources, config file
`B` contained all instances, and config `C` contained firewall rules, router,
and VPN, in `gcloud` you would need to *manually* define the config deployment
order according to the resource dependencies. The VPN would depend on the cloud
router, both of them would depend on the network, etc. The `CFT` computes the
dependencies *automatically*, which eliminates the need for manual deployment
ordering.

`Note:` This User Guide assumes that you are familiar with the Google Cloud SDK
operations related to resource deployment and management. For additional
information, refer to the
[SDK documentation](https://cloud.google.com/sdk/docs/).

The CFT includes:

- A command-line interface (henceforth, CLI) that deploys resources defined in
  single or multiple CFT-compliant config files
- A comprehensive set of production-ready resource [templates](#templates) that follow
  Google's best practices, which can be used with the CFT or the `gcloud`
  utility. (`gcloud` is part of the Google Cloud SDK).

You can use the CFT "as is" or modify it to suit your specific needs. Instructions
and recommendations for the CFT code modifications are in the
[CFT Developer Guide](tool_dev_guide.md).

## CFT Configs

To use the CFT, you need to first create the config files for the desired
deployments. These configs are YAML structures very similar to, and compatible
with, the `gcloud` config files. The difference is that they contain extra YAML
directives and features to support the expanded capabilities of the CFT
(multi-config deployment and cross-deployment references).

### Extra YAML Directives

#### name

This directive is used to specify the name of the deployment; for example:

```yaml
name: my-network
```

If not specified, the name of the deployment is inferred from the config
file name. For example, if the path to the config file is
`path/to/configs/my-network.yaml`, and the config does not specify the `name`
directive, the deployment name is set to `my-network`. This is meant as a
workaround for maintaining compatibility between the `CFT` and `gcloud` configs.
However, **it is strongly recommended that the `name` directive is specified**.

#### project

This directive defines the project in which the resource is deployed; for
example:

```yaml
project: my-project
```

While this directive is optional, **its use in your configs is highly
recommended**. In addition to the project directive in the config file,
the project for a deployment to be created in can be specified by other means.
The order of precedence is as follows:

1. The `--project` command-line option. If a project is specified via this
   option, all configs in the run use that project. This is a way of
   quickly overriding the project specified in a config file, which should be
   used with caution.
2. The `project` directive in the config file.
3. The `CLOUD_FOUNDATION_PROJECT_ID` environment variable.
4. The "default project" configured with the GCP SDK.

`Note:` When deployments utilize cross-project resources, the `project`
directive becomes mandatory in at least one of the deployments.

#### description

This directive is the deployment description, which allows you
to add some documentation to your configs; for example:

```yaml
description: My firewall deployment for {{environment}} environment
```

### Extra Features

#### Cross-deployment References with the `$(out)` Tag

A config/deployment can specify a dependency on another deployment's output
without the need to create the dependent deployment in advance. This is the
mechanism the CFT uses to determine the order of execution of the deployments.

```yaml
$(out.<project>.<deployment>.<resource>.<output>)

# or

$(out.<deployment>.<resource>.<output>)
```

wherein:

- `$(out)` is the prefix that indicates that the value references an output
  from a resource defined in an external deployment (in another config file)
- `project` is the ID of the project in which the external deployment is
   created
- `deployment` is the he name of the external deployment (config) that
  defines the referenced resource
- `resource` is the DM name of the referenced resource
- `output` is the name of the output parameter to be referenced

The above construct works very similarly to Deployment Manager's
`$(ref.<resource>.<property>)`. However, it allows defining not only references
to resource properties not only *within* a deployment, but also
*inter-deployment/inter-project* references, using deployment outputs. The
value of output of a dependent deployment is only looked up during the current
deployment's execution, which allows you to create config files without knowing
in advance the actual values of the outputs in the dependent deployments, or
even having to create these deployments.

For example:

```yaml
network: $(out.my-network-prod.my-network-prod.name)
```

#### Jinja Templating

All configs submitted via the CFT CLI are rendered by the [Jinja Template
Engine](http://jinja.pocoo.org/). This supports compact code by using the DRY
pattern. For example, by using variable substitution and `for loops`:

```yaml
{% set environment = 'prod' %}
{% set applications = ['app1', 'app2', 'app3'] %}

name: my-network-{{environment}}
description: Network deployment for {{environment}} environment
project: sourced-gus-1
imports:
  - path: templates/network/network.py
resources:
{% for application in applications %}
  - type: templates/network/network.py
    name: {{application}}-{{environment}}-network
    properties:
      autoCreateSubnetworks: false
{% endfor %}
```

An alternative to using Jinja in your configs is to write wrapper DM Python
templates and reference these templates in your configs (see the
[Templates](#templates) section).

### Samples

Following are three sample config files that illustrate the above directives
and features. These will be used as examples in the action-specific sections of
this User Guide:

- [network.yaml](#network.yaml) - two networks that have no dependencies
- [firewall.yaml](#firewall.yaml) - two firewall rules, which depend on the
  corresponding networks
- [instance.yaml](#instance.yaml) - one VM instance, which depends on the
  network

#### network.yaml

```yaml
name: my-networks
description: my networks deployment

imports:
  - path: templates/network/network.py

resources:
  - type: templates/network/network.py
    name: my-network-prod
    properties:
      autoCreateSubnetworks: true

  - type: templates/network/network.py
    name: my-network-dev
    properties:
      autoCreateSubnetworks: false
```

#### firewall.yaml

```yaml
name: my-firewalls
description: My firewalls deployment

imports:
  - path: templates/firewall/firewall.py
resources:
  - type: templates/firewall/firewall.py
    name: my-firewall-prod
    properties:
      network: $(out.my-networks.my-network-prod.name)
      rules:
        - name: allow-proxy-from-inside-prod
      allowed:
            - IPProtocol: tcp
              ports:
                - "80"
                - "444"
          description: This rule allows connectivity to the HTTP proxies
          direction: INGRESS
          sourceRanges:
            - 10.0.0.0/8
        - name: allow-dns-from-inside-prod
          allowed:
            - IPProtocol: udp
              ports:
                - "53"
            - IPProtocol: tcp
              ports:
                - "53"
          description: this rule allows DNS queries to google's 8.8.8.8
          direction: EGRESS
          destinationRanges:
            - 8.8.8.8/32
  - type: templates/firewall/firewall.py
    name: my-firewall-dev
    properties:
      network: $(out.my-networks.my-network-dev.name)
      rules:
        - name: allow-proxy-from-inside-dev
          allowed:
            - IPProtocol: tcp
              ports:
                - "80"
                - "444"
          description: This rule allows connectivity to the HTTP proxies
          direction: INGRESS
          sourceRanges:
            - 10.0.0.0/8
```

#### instance.yaml

```yaml
name: my-instance-prod-1
description: My instance deployment for prod environment

imports:
  - path: templates/instance/instance.py
    name: instance.py

resources:
  - name: my-instance-prod-1
    type: instance.py
    properties:
      zone: us-central1-a
      diskImage: projects/ubuntu-os-cloud/global/images/family/ubuntu-1804-lts
      diskSizeGb: 100
      machineType: f1-micro
      diskType: pd-ssd
      network: $(out.my-networks.my-network-prod.name)
      metadata:
        items:
          - key: startup-script
            value: sudo apt-get update && sudo apt-get install -y nginx
```

## Templates

CFT-compliant configs can use templates written in Python or Jinja2. [Templates
included in the toolkit](../templates/README.md) are recommended (although not mandatory)
as they offer robust functionality, ease of use, and adherence to best
practices.

You can use the templates included in our library "as is," and/or modify them
to suit your needs, as well as develop your own templates. Instructions and
recommendations for template development are in the
[Template Developer Guide](template_dev_guide.md).  

## Toolkit Installation and Configuration

This toolkit was developed primarily on/for Linux. Therefore, the Linux platform
is expected to offer the most seamless user experience.

### Installing Prerequisites

#### Python 2.7 + pip

Follow your OS package manager instructions. For example, for Ubuntu:

```shell
sudo apt-get install python2.7
sudo apt-get install python-pip
```

#### Google Cloud SDK

1. Install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/quickstarts).
2. Ensure that the `gcloud` command is in the user's PATH:

```shell
which gcloud
```

### Getting the CFT Code

Proceed as follows:

```shell
git clone https://github.com/GoogleCloudPlatform/deploymentmanager-samples
cd deploymentmanager-samples/
```

### Installing the CFT

{'fraction_non_alphanumeric': 0.0914997956681651, 'fraction_numerical': 0.005394360441356764, 'mean_word_length': 3.5367074527252504, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 8, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3510504', 'n_tokens_mistral': 6934, 'n_tokens_neox': 6500, 'n_words': 2706}