<commit_before>---
subcategory: "Dataproc"
layout: "google"
page_title: "Google: google_dataproc_cluster"
sidebar_current: "docs-google-dataproc-cluster"
description: |-
  Manages a Cloud Dataproc cluster resource.
---

# google\_dataproc\_cluster

Manages a Cloud Dataproc cluster resource within GCP.

* [API documentation](https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters)
* How-to Guides
    * [Official Documentation](https://cloud.google.com/dataproc/docs)


!> **Warning:** Due to limitations of the API, all arguments except
`labels`,`cluster_config.worker_config.num_instances` and `cluster_config.preemptible_worker_config.num_instances` are non-updatable. Changing others will cause recreation of the
whole cluster!

## Example Usage - Basic

```hcl
resource "google_dataproc_cluster" "simplecluster" {
  name   = "simplecluster"
  region = "us-central1"
}
```

## Example Usage - Advanced

```hcl
resource "google_service_account" "default" {
  account_id   = "service-account-id"
  display_name = "Service Account"
}

resource "google_dataproc_cluster" "mycluster" {
  name     = "mycluster"
  region   = "us-central1"
  graceful_decommission_timeout = "120s"
  labels = {
    foo = "bar"
  }

  cluster_config {
    staging_bucket = "dataproc-staging-bucket"

    master_config {
      num_instances = 1
      machine_type  = "e2-medium"
      disk_config {
        boot_disk_type    = "pd-ssd"
        boot_disk_size_gb = 30
      }
    }

    worker_config {
      num_instances    = 2
      machine_type     = "e2-medium"
      min_cpu_platform = "Intel Skylake"
      disk_config {
        boot_disk_size_gb = 30
        num_local_ssds    = 1
      }
    }

    preemptible_worker_config {
      num_instances = 0
    }

    # Override or set some custom properties
    software_config {
      image_version = "1.3.7-deb9"
      override_properties = {
        "dataproc:dataproc.allow.zero.workers" = "true"
      }
    }

    gce_cluster_config {
      tags = ["foo", "bar"]
      # Google recommends custom service accounts that have cloud-platform scope and permissions granted via IAM Roles.
      service_account = google_service_account.default.email
      service_account_scopes = [
        "cloud-platform"
      ]
    }

    # You can define multiple initialization_action blocks
    initialization_action {
      script      = "gs://dataproc-initialization-actions/stackdriver/stackdriver.sh"
      timeout_sec = 500
    }
  }
}
```

## Example Usage - Using a GPU accelerator

```hcl
resource "google_dataproc_cluster" "accelerated_cluster" {
  name   = "my-cluster-with-gpu"
  region = "us-central1"

  cluster_config {
    gce_cluster_config {
      zone = "us-central1-a"
    }

    master_config {
      accelerators {
        accelerator_type  = "nvidia-tesla-k80"
        accelerator_count = "1"
      }
    }
  }
}
```

## Argument Reference

* `name` - (Required) The name of the cluster, unique within the project and
	zone.

- - -

* `project` - (Optional) The ID of the project in which the `cluster` will exist. If it
	is not provided, the provider project is used.

* `region` - (Optional) The region in which the cluster and associated nodes will be created in.
   Defaults to `global`.

* `labels` - (Optional, Computed) The list of labels (key/value pairs) to be applied to
   instances in the cluster. GCP generates some itself including `goog-dataproc-cluster-name`
   which is the name of the cluster.

* `cluster_config` - (Optional) Allows you to configure various aspects of the cluster.
   Structure defined below.

* `graceful_decommission_timout` - (Optional) Allows graceful decomissioning when you change the number of worker nodes directly through a terraform apply.
      Does not affect auto scaling decomissioning from an autoscaling policy.
      Graceful decommissioning allows removing nodes from the cluster without interrupting jobs in progress.
      Timeout specifies how long to wait for jobs in progress to finish before forcefully removing nodes (and potentially interrupting jobs).
      Default timeout is 0 (for forceful decommission), and the maximum allowed timeout is 1 day. (see JSON representation of
      [Duration](https://developers.google.com/protocol-buffers/docs/proto3#json)).
      Only supported on Dataproc image versions 1.2 and higher.
      For more context see the [docs](https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters/patch#query-parameters)
- - -

The `cluster_config` block supports:

```hcl
    cluster_config {
        gce_cluster_config        { ... }
        master_config             { ... }
        worker_config             { ... }
        preemptible_worker_config { ... }
        software_config           { ... }

        # You can define multiple initialization_action blocks
        initialization_action     { ... }
        encryption_config         { ... }
        endpoint_config           { ... }
    }
```

* `staging_bucket` - (Optional) The Cloud Storage staging bucket used to stage files,
   such as Hadoop jars, between client machines and the cluster.
   Note: If you don't explicitly specify a `staging_bucket`
   then GCP will auto create / assign one for you. However, you are not guaranteed
   an auto generated bucket which is solely dedicated to your cluster; it may be shared
   with other clusters in the same region/zone also choosing to use the auto generation
   option.

* `temp_bucket` - (Optional) The Cloud Storage temp bucket used to store ephemeral cluster
   and jobs data, such as Spark and MapReduce history files.
   Note: If you don't explicitly specify a `temp_bucket` then GCP will auto create / assign one for you.

* `gce_cluster_config` (Optional) Common config settings for resources of Google Compute Engine cluster
   instances, applicable to all instances in the cluster. Structure defined below.

* `master_config` (Optional) The Google Compute Engine config settings for the master instances
   in a cluster.. Structure defined below.

* `worker_config` (Optional) The Google Compute Engine config settings for the worker instances
   in a cluster.. Structure defined below.

* `preemptible_worker_config` (Optional) The Google Compute Engine config settings for the additional (aka
   preemptible) instances in a cluster. Structure defined below.

* `software_config` (Optional) The config settings for software inside the cluster.
   Structure defined below.

* `security_config` (Optional) Security related configuration. Structure defined below.

* `autoscaling_config` (Optional)  The autoscaling policy config associated with the cluster.
   Note that once set, if `autoscaling_config` is the only field set in `cluster_config`, it can
   only be removed by setting `policy_uri = ""`, rather than removing the whole block.
   Structure defined below.

* `initialization_action` (Optional) Commands to execute on each node after config is completed.
   You can specify multiple versions of these. Structure defined below.

* `encryption_config` (Optional) The Customer managed encryption keys settings for the cluster.
   Structure defined below.

* `lifecycle_config` (Optional, Beta) The settings for auto deletion cluster schedule.
   Structure defined below.

* `endpoint_config` (Optional, Beta) The config settings for port access on the cluster.
   Structure defined below.
- - -

The `cluster_config.gce_cluster_config` block supports:

```hcl
  cluster_config {
    gce_cluster_config {<commit_msg>add note for preemptible_worker_config (#4695)<commit_after>---
subcategory: "Dataproc"
layout: "google"
page_title: "Google: google_dataproc_cluster"
sidebar_current: "docs-google-dataproc-cluster"
description: |-
  Manages a Cloud Dataproc cluster resource.
---

# google\_dataproc\_cluster

Manages a Cloud Dataproc cluster resource within GCP.

* [API documentation](https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters)
* How-to Guides
    * [Official Documentation](https://cloud.google.com/dataproc/docs)


!> **Warning:** Due to limitations of the API, all arguments except
`labels`,`cluster_config.worker_config.num_instances` and `cluster_config.preemptible_worker_config.num_instances` are non-updatable. Changing others will cause recreation of the
whole cluster!

## Example Usage - Basic

```hcl
resource "google_dataproc_cluster" "simplecluster" {
  name   = "simplecluster"
  region = "us-central1"
}
```

## Example Usage - Advanced

```hcl
resource "google_service_account" "default" {
  account_id   = "service-account-id"
  display_name = "Service Account"
}

resource "google_dataproc_cluster" "mycluster" {
  name     = "mycluster"
  region   = "us-central1"
  graceful_decommission_timeout = "120s"
  labels = {
    foo = "bar"
  }

  cluster_config {
    staging_bucket = "dataproc-staging-bucket"

    master_config {
      num_instances = 1
      machine_type  = "e2-medium"
      disk_config {
        boot_disk_type    = "pd-ssd"
        boot_disk_size_gb = 30
      }
    }

    worker_config {
      num_instances    = 2
      machine_type     = "e2-medium"
      min_cpu_platform = "Intel Skylake"
      disk_config {
        boot_disk_size_gb = 30
        num_local_ssds    = 1
      }
    }

    preemptible_worker_config {
      num_instances = 0
    }

    # Override or set some custom properties
    software_config {
      image_version = "1.3.7-deb9"
      override_properties = {
        "dataproc:dataproc.allow.zero.workers" = "true"
      }
    }

    gce_cluster_config {
      tags = ["foo", "bar"]
      # Google recommends custom service accounts that have cloud-platform scope and permissions granted via IAM Roles.
      service_account = google_service_account.default.email
      service_account_scopes = [
        "cloud-platform"
      ]
    }

    # You can define multiple initialization_action blocks
    initialization_action {
      script      = "gs://dataproc-initialization-actions/stackdriver/stackdriver.sh"
      timeout_sec = 500
    }
  }
}
```

## Example Usage - Using a GPU accelerator

```hcl
resource "google_dataproc_cluster" "accelerated_cluster" {
  name   = "my-cluster-with-gpu"
  region = "us-central1"

  cluster_config {
    gce_cluster_config {
      zone = "us-central1-a"
    }

    master_config {
      accelerators {
        accelerator_type  = "nvidia-tesla-k80"
        accelerator_count = "1"
      }
    }
  }
}
```

## Argument Reference

* `name` - (Required) The name of the cluster, unique within the project and
	zone.

- - -

* `project` - (Optional) The ID of the project in which the `cluster` will exist. If it
	is not provided, the provider project is used.

* `region` - (Optional) The region in which the cluster and associated nodes will be created in.
   Defaults to `global`.

* `labels` - (Optional, Computed) The list of labels (key/value pairs) to be applied to
   instances in the cluster. GCP generates some itself including `goog-dataproc-cluster-name`
   which is the name of the cluster.

* `cluster_config` - (Optional) Allows you to configure various aspects of the cluster.
   Structure defined below.

* `graceful_decommission_timout` - (Optional) Allows graceful decomissioning when you change the number of worker nodes directly through a terraform apply.
      Does not affect auto scaling decomissioning from an autoscaling policy.
      Graceful decommissioning allows removing nodes from the cluster without interrupting jobs in progress.
      Timeout specifies how long to wait for jobs in progress to finish before forcefully removing nodes (and potentially interrupting jobs).
      Default timeout is 0 (for forceful decommission), and the maximum allowed timeout is 1 day. (see JSON representation of
      [Duration](https://developers.google.com/protocol-buffers/docs/proto3#json)).
      Only supported on Dataproc image versions 1.2 and higher.
      For more context see the [docs](https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters/patch#query-parameters)
- - -

The `cluster_config` block supports:

```hcl
    cluster_config {
        gce_cluster_config        { ... }
        master_config             { ... }
        worker_config             { ... }
        preemptible_worker_config { ... }
        software_config           { ... }

        # You can define multiple initialization_action blocks
        initialization_action     { ... }
        encryption_config         { ... }
        endpoint_config           { ... }
    }
```

* `staging_bucket` - (Optional) The Cloud Storage staging bucket used to stage files,
   such as Hadoop jars, between client machines and the cluster.
   Note: If you don't explicitly specify a `staging_bucket`
   then GCP will auto create / assign one for you. However, you are not guaranteed
   an auto generated bucket which is solely dedicated to your cluster; it may be shared
   with other clusters in the same region/zone also choosing to use the auto generation
   option.

* `temp_bucket` - (Optional) The Cloud Storage temp bucket used to store ephemeral cluster
   and jobs data, such as Spark and MapReduce history files.
   Note: If you don't explicitly specify a `temp_bucket` then GCP will auto create / assign one for you.

* `gce_cluster_config` (Optional) Common config settings for resources of Google Compute Engine cluster
   instances, applicable to all instances in the cluster. Structure defined below.

* `master_config` (Optional) The Google Compute Engine config settings for the master instances
   in a cluster.. Structure defined below.

* `worker_config` (Optional) The Google Compute Engine config settings for the worker instances
   in a cluster.. Structure defined below.

* `preemptible_worker_config` (Optional) The Google Compute Engine config settings for the additional
   instances in a cluster. Structure defined below.
  * **NOTE** : `preemptible_worker_config` is
   an alias for the api's [secondaryWorkerConfig](https://cloud.google.com/dataproc/docs/reference/rest/v1/ClusterConfig#InstanceGroupConfig). The name doesn't neccasarily mean it is preemptible and is named as
   such for legacy/compatibility reasons.

* `software_config` (Optional) The config settings for software inside the cluster.
   Structure defined below.

* `security_config` (Optional) Security related configuration. Structure defined below.

* `autoscaling_config` (Optional)  The autoscaling policy config associated with the cluster.
   Note that once set, if `autoscaling_config` is the only field set in `cluster_config`, it can
   only be removed by setting `policy_uri = ""`, rather than removing the whole block.
   Structure defined below.

* `initialization_action` (Optional) Commands to execute on each node after config is completed.
   You can specify multiple versions of these. Structure defined below.

* `encryption_config` (Optional) The Customer managed encryption keys settings for the cluster.
   Structure defined below.

* `lifecycle_config` (Optional, Beta) The settings for auto deletion cluster schedule.
   Structure defined below.

* `endpoint_config` (Optional, Beta) The config settings for port access on the cluster.
   Structure defined below.
- - -

The `cluster_config.gce_cluster_config` block supports:

```hcl
  cluster_config {
    gce_cluster_config {
{'fraction_non_alphanumeric': 0.08099323972958919, 'fraction_numerical': 0.004745189807592303, 'mean_word_length': 3.3325823711630527, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5630450', 'n_tokens_mistral': 4450, 'n_tokens_neox': 4275, 'n_words': 1485}