<commit_before># Magic Modules

<img src="mmv1/images/magic-modules.svg" alt="Magic Modules Logo" width="300" align="right" />

## Overview

Magic Modules is a tool used to autogenerate support in a variety of open source DevOps
tools for Google Cloud Platform. [GCP "resource"](https://cloud.google.com/docs/overview/#gcp_resources)
definitions are encoded in a shared data file, and that data is used to fill in
tool-specific templates across each of the tools Magic Modules
generates.

Magic Modules generates GCP support for:

* Terraform
* Ansible
* InSpec

In addition, Magic Modules generates support for several companion
features/tools:

* Terraform Google Inventory Mapper
* Terraform in Cloud Shell

Importantly, Magic Modules *isn't* full code generation. Every change is made
manually; more than a code generator, Magic Modules is a force multiplier for
development. While many Magic Modules resources are defined exactly based on the
GCP API, we use Magic Modules to preemptively solve issues across each tool by
encoding our field-tested learnings from other tools in those definitions. In
effect, an issue solved in one tool will be solved for each other tool.

## Getting Started with Magic Modules

You can try out Magic Modules immediately with Open in Cloud Shell below; if
you're getting set up on a local workstation, this guide serves as a reference
to help you get it set up.

[![Open in Cloud Shell](http://gstatic.com/cloudssh/images/open-btn.svg)](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/magic-modules&tutorial=mmv1/TUTORIAL.md)

### Requirements

To get started, you'll need:

* Ruby 2.6.0
  * You can use `rbenv` to manage your Ruby version(s)
* [`Bundler`](https://github.com/bundler/bundler)
  * This can be installed with `gem install bundler`
* If you are getting "Too many open files" ulimit needs to be raised.
  * Mac OSX: `ulimit -n 1000`

### Preparing Magic Modules / One-time setup

**Important:**
Compiling Magic Modules can be done directly from the `mmv1` directory within this repository.
In the future we will add hybrid generation with multiple generators. All the information below
pertains only to the contents of the `mmv1` directory, and commands should be executed from
that directory.


To get started right away, use the bootstrap script with:

```bash
./tools/bootstrap
```

---

Otherwise, follow the manual steps below:

If you're generating the Terraform providers (`google` and `google-beta`),
you'll need to check out the repo(s) you're generating in your GOPATH. For
example:

```bash
git clone https://github.com/hashicorp/terraform-provider-google.git $GOPATH/src/github.com/hashicorp/terraform-provider-google
git clone https://github.com/hashicorp/terraform-provider-google-beta.git $GOPATH/src/github.com/hashicorp/terraform-provider-google-beta
```

Magic Modules won't work with old versions of the Terraform provider repos. If
you're encountering issues with vendoring and paths, make sure both MM and the
Terraform provider are running on up to date copies of `master`.

Once you've prepared the target folders for the tools, run the following to
finish getting Magic Modules set up by installing the Ruby gems it needs to run:

```bash
cd mmv1
bundle install
```

Now, you can verify you're ready with:

```bash
./tools/doctor
```

### Generating the Terraform Providers

Before making any changes, you can compile the Terraform provider you're working
on by running the following command. If Magic Modules has been installed
correctly, you'll get no errors.

The following commands should be run from the root directory of the repository.
OUTPUT_PATH should be set to the location of your provider repository, which is
recommended to be inside your GOPATH.

```bash
make terraform VERSION=ga OUTPUT_PATH="$GOPATH/src/github.com/hashicorp/terraform-provider-google"
make terraform VERSION=beta OUTPUT_PATH="$GOPATH/src/github.com/hashicorp/terraform-provider-google-beta"

# Only generate a specific product (plus all common files)
make terraform VERSION=ga OUTPUT_PATH="$GOPATH/src/github.com/hashicorp/terraform-provider-google" PRODUCT=dataproc
```

It's worth noting that Magic Modules will only generate new files when run
locally. The "Magician"- the Magic Modules CI system- handles deletion of old
files when creating PRs.

#### Generating terraform-google-conversion

You can compile terraform-google-conversion by running the following command.
If Magic Modules has been installed correctly, you'll get no errors.

```bash
make validator OUTPUT_PATH="/path/to/your/terraform-google-conversion"
```

### Making changes to resources

Once again, see the Open in Cloud Shell example above for an interactive example
of making a Magic Modules change; this section will serve as a reference more
than a specific example.

Magic Modules mirrors the GCP REST API; there are [products](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/api/product.rb)
such as Compute or Container (GKE) that contains [resources](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/api/resource.rb),
[GCP resources](https://cloud.google.com/docs/overview/#gcp_resources) such as
Compute VM Instances or GKE Clusters.<commit_msg>Updated README for generating terraform-validator instead of terraform-google-conversion (#5359)<commit_after># Magic Modules

<img src="mmv1/images/magic-modules.svg" alt="Magic Modules Logo" width="300" align="right" />

## Overview

Magic Modules is a tool used to autogenerate support in a variety of open source DevOps
tools for Google Cloud Platform. [GCP "resource"](https://cloud.google.com/docs/overview/#gcp_resources)
definitions are encoded in a shared data file, and that data is used to fill in
tool-specific templates across each of the tools Magic Modules
generates.

Magic Modules generates GCP support for:

* Terraform
* Ansible
* InSpec

In addition, Magic Modules generates support for several companion
features/tools:

* Terraform Google Inventory Mapper
* Terraform in Cloud Shell

Importantly, Magic Modules *isn't* full code generation. Every change is made
manually; more than a code generator, Magic Modules is a force multiplier for
development. While many Magic Modules resources are defined exactly based on the
GCP API, we use Magic Modules to preemptively solve issues across each tool by
encoding our field-tested learnings from other tools in those definitions. In
effect, an issue solved in one tool will be solved for each other tool.

## Getting Started with Magic Modules

You can try out Magic Modules immediately with Open in Cloud Shell below; if
you're getting set up on a local workstation, this guide serves as a reference
to help you get it set up.

[![Open in Cloud Shell](http://gstatic.com/cloudssh/images/open-btn.svg)](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/magic-modules&tutorial=mmv1/TUTORIAL.md)

### Requirements

To get started, you'll need:

* Ruby 2.6.0
  * You can use `rbenv` to manage your Ruby version(s)
* [`Bundler`](https://github.com/bundler/bundler)
  * This can be installed with `gem install bundler`
* If you are getting "Too many open files" ulimit needs to be raised.
  * Mac OSX: `ulimit -n 1000`

### Preparing Magic Modules / One-time setup

**Important:**
Compiling Magic Modules can be done directly from the `mmv1` directory within this repository.
In the future we will add hybrid generation with multiple generators. All the information below
pertains only to the contents of the `mmv1` directory, and commands should be executed from
that directory.


To get started right away, use the bootstrap script with:

```bash
./tools/bootstrap
```

---

Otherwise, follow the manual steps below:

If you're generating the Terraform providers (`google` and `google-beta`),
you'll need to check out the repo(s) you're generating in your GOPATH. For
example:

```bash
git clone https://github.com/hashicorp/terraform-provider-google.git $GOPATH/src/github.com/hashicorp/terraform-provider-google
git clone https://github.com/hashicorp/terraform-provider-google-beta.git $GOPATH/src/github.com/hashicorp/terraform-provider-google-beta
```

Magic Modules won't work with old versions of the Terraform provider repos. If
you're encountering issues with vendoring and paths, make sure both MM and the
Terraform provider are running on up to date copies of `master`.

Once you've prepared the target folders for the tools, run the following to
finish getting Magic Modules set up by installing the Ruby gems it needs to run:

```bash
cd mmv1
bundle install
```

Now, you can verify you're ready with:

```bash
./tools/doctor
```

### Generating the Terraform Providers

Before making any changes, you can compile the Terraform provider you're working
on by running the following command. If Magic Modules has been installed
correctly, you'll get no errors.

The following commands should be run from the root directory of the repository.
OUTPUT_PATH should be set to the location of your provider repository, which is
recommended to be inside your GOPATH.

```bash
make terraform VERSION=ga OUTPUT_PATH="$GOPATH/src/github.com/hashicorp/terraform-provider-google"
make terraform VERSION=beta OUTPUT_PATH="$GOPATH/src/github.com/hashicorp/terraform-provider-google-beta"

# Only generate a specific product (plus all common files)
make terraform VERSION=ga OUTPUT_PATH="$GOPATH/src/github.com/hashicorp/terraform-provider-google" PRODUCT=dataproc
```

It's worth noting that Magic Modules will only generate new files when run
locally. The "Magician"- the Magic Modules CI system- handles deletion of old
files when creating PRs.

#### Generating terraform-validator

You can compile terraform-validator by running the following command.
If Magic Modules has been installed correctly, you'll get no errors.

```bash
make validator OUTPUT_PATH="/path/to/your/terraform-validator"
```

### Making changes to resources

Once again, see the Open in Cloud Shell example above for an interactive example
of making a Magic Modules change; this section will serve as a reference more
than a specific example.

Magic Modules mirrors the GCP REST API; there are [products](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/api/product.rb)
such as Compute or Container (GKE) that contains [resources](https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/api/resource.rb),
[GCP resources](https://cloud.google.com/docs/overview/#gcp_resources) such as
Compute VM Instances or GKE Clusters.
{'fraction_non_alphanumeric': 0.07941592086669808, 'fraction_numerical': 0.003579839849269901, 'mean_word_length': 4.927414852037968, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 18, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '277526', 'n_tokens_mistral': 3081, 'n_tokens_neox': 2948, 'n_words': 1285}