<commit_before>---
layout: default
title: "Beam Contribution Guide"
permalink: /contribute/contribution-guide/
redirect_from: /contribution-guide/
---

# Apache Beam Contribution Guide

* TOC
{:toc}

The Apache Beam community welcomes contributions from anyone with a passion for data processing! Beam has many different opportunities for contributions -- write new examples, add new user-facing libraries (new statistical libraries, new IO connectors, etc), work on the core programming model, build specific runners (Apache Flink, Apache Spark, Google Cloud Dataflow, etc), or participate on the documentation effort.

We use a review-then-commit workflow in Beam for all contributions.

![Alt text]({{ "/images/contribution-guide-1.png" | prepend: site.baseurl }} "Workflow image")

**For larger contributions or those that affect multiple components:**

1. **Engage**: We encourage you to work with the Beam community on the [Apache JIRA issue tracker](https://issues.apache.org/jira/browse/BEAM) and [developer’s mailing list](http://beam.incubator.apache.org/use/mailing-lists/) to identify good areas for contribution.
1. **Design:** More complicated contributions will likely benefit from some early discussion in order to scope and design them well.

**For all contributions:**

1. **Code:** The best part ;-)
1. **Review:** Submit a pull request with your contribution to our [GitHub mirror](https://github.com/apache/incubator-beam/). Work with a committer to review and iterate on the code, if needed.
1. **Commit:** A Beam committer merges the pull request into our [Apache repository](https://git-wip-us.apache.org/repos/asf/incubator-beam.git).

We look forward to working with you!

## Engage

### Mailing list(s)
We discuss design and implementation issues on <EMAIL> mailing list, which is archived [here](https://lists.apache.org/list.html?<EMAIL>). Join by emailing [`<EMAIL>`](mailto:<EMAIL>).

If interested, you can also join the other [mailing lists](http://beam.incubator.apache.org/use/mailing-lists/) too.

### Apache JIRA
We use [Apache JIRA](https://issues.apache.org/jira/browse/BEAM) as an issue tracking and project management tool, as well as a way to communicate among a very diverse and distributed set of contributors. To be able to gather feedback, avoid frustration, and avoid duplicated efforts all Beam-related work should be tracked there.

If you do not already have an Apache JIRA account, sign up [here](https://issues.apache.org/jira/).

If a quick [search](https://issues.apache.org/jira/issues/?jql=project%3DBEAM%20AND%20text%20~%20%22the%20thing%20I%20want%20to%20contribute%22) doesn’t turn up an existing JIRA issue for the work you want to contribute, create it. Please discuss your idea with a committer or the [component lead](https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:components-panel) in JIRA or, alternatively, on the developer mailing list.

If there’s an existing JIRA issue for your intended contribution, please comment about your intended work. Once the work is understood, a committer will assign the issue to you. (If you don’t have a JIRA role yet, you’ll be added to the “contributor” role.) If an issue is currently assigned, please check with the current assignee before reassigning.

For moderate or large contributions, you should not start coding or writing a design document unless there is a corresponding JIRA issue assigned to you for that work. Simple changes, like fixing typos, do not require an associated issue.

## Design

To avoid potential frustration during the code review cycle, we encourage you to clearly scope and design non-trivial contributions with the Beam community before you start coding.

Generally, the JIRA issue is the best place to gather relevant design docs, comments, or references. It’s great to explicitly include relevant stakeholders early in the conversation. For designs that may be generally interesting, we also encourage conversations on the developer’s mailing list.

We suggest using [Google Docs](https://docs.google.com/) for sharing designs that may benefit from diagrams or comments. Please remember to make the document world-commentable and add a link to it from the relevant JIRA issue. You may want to start from this [template](https://docs.google.com/document/d/1qYQPGtabN5-E4MjHsecqqC7PXvJtXvZukPfLXQ8rHJs/edit?usp=sharing).

## Code
To contribute code to Apache Beam, you’ll have to do a few administrative steps once, and then follow a few guidelines for each contribution.

### One-time Setup

#### Intellij

#### Eclipse
 
Use a recent eclipse version that includes m2e. Currently we recommend Eclipse Neon.
Start eclipse with a fresh workspace in a separate directory from your checkout.

##### Install the m2e apt support

Beam uses apt annotation processing to provide auto generated code. One example is the usage of [google auto value](https://github.com/google/auto/tree/master/value). By default m2e does not support this and you will see compile errors.

Install m2e-apt

	Help 
	-> Eclipse Marketplace 
	-> Search for "m2 apt" 
	-> Install m2e-apt 1.2 or higher
	

Activate the apt processing

	Window 
	-> Preferences 
	-> Maven 
	-> Annotation processing 
	-> Switch to Experimental: Delegate annoation processing ... 
	-> Ok

##### Import the beam projects

	File 
	-> Import... 
	-> Existing Maven Projects 
	-> Browse to the directory you cloned into and select incubator-beam 
	-> make sure all beam projects are selected 
	-> Finalize

You now should have all the beam projects imported into eclipse and should see no compile errors.
 


#### [Potentially] Submit Contributor License Agreement
Apache Software Foundation (ASF) desires that all contributors of ideas, code, or documentation to the Apache projects complete, sign, and submit an [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt) (ICLA). The purpose of this agreement is to clearly define the terms under which intellectual property has been contributed to the ASF and thereby allow us to defend the project should there be a legal dispute regarding the software at some future time.

We require you to have an ICLA on file with the Apache Secretary for larger contributions only. For smaller ones, however, we rely on  [clause five](http://www.apache.org/licenses/LICENSE-2.0#contributions) of the Apache License, Version 2.0, describing licensing of intentionally submitted contributions and do not require an ICLA in that case.

#### Obtain a GitHub account
We use GitHub’s pull request functionality to review proposed code changes.

If you do not already have a personal GitHub account, sign up [here](https://github.com/join).

#### Fork the repository on GitHub
Go to the [Beam GitHub mirror](https://github.com/apache/incubator-beam/) and fork the repository to your own private account. This will be your private workspace for staging changes.

We recommend enabling Travis-CI continuous integration coverage on your private fork in order to easily test your changes before proposing a pull request. Go to [Travis-CI](https://travis-ci.org), log in with your GitHub account, and enable coverage for your repository.

#### Clone the repository locally
You are now ready to create the development environment on your local machine. Feel free to repeat these steps on all machines that you want to use for development.

We assume you are using SSH-based authentication with GitHub. If necessary, exchange SSH keys with GitHub by following [their instructions](https://help.github.com/articles/generating-an-ssh-key/).

Clone Beam’s read-only GitHub mirror.

    $ git clone https://github.com/apache/incubator-beam.git
    $ cd incubator-beam

Add your forked repository as an additional Git remote, where you’ll push your changes.

	$ git remote add <GitHub_user> <EMAIL>:<GitHub_user>/incubator-beam.git

You are now ready to start developing!

### Create a branch in your fork
You’ll work on your contribution in a branch in your own (forked) repository. Create a local branch, initialized with the state of the branch you expect your changes to be merged into. Keep in mind that we use several branches, including `master`, feature-specific, and release-specific branches. If you are unsure, initialize with the state of the `master` branch.

	$ git fetch --all
	$ git checkout -b <my-branch> origin/master

At this point, you can start making and committing changes to this branch in a standard way.

### Syncing and pushing your branch
Periodically while you work, and certainly before submitting a pull request, you should update your branch with the most recent changes to the target branch.

    $ git pull --rebase

Remember to always use `--rebase` parameter to avoid extraneous merge commits.

To push your local, committed changes to your (forked) repository on GitHub, run:

	$ git push <GitHub_user> <my-branch>	

### Testing<commit_msg>Reordered IDE section after code checkout and tweaked heading levels.<commit_after>---
layout: default
title: "Beam Contribution Guide"
permalink: /contribute/contribution-guide/
redirect_from: /contribution-guide/
---

# Apache Beam Contribution Guide

* TOC
{:toc}

The Apache Beam community welcomes contributions from anyone with a passion for data processing! Beam has many different opportunities for contributions -- write new examples, add new user-facing libraries (new statistical libraries, new IO connectors, etc), work on the core programming model, build specific runners (Apache Flink, Apache Spark, Google Cloud Dataflow, etc), or participate on the documentation effort.

We use a review-then-commit workflow in Beam for all contributions.

![Alt text]({{ "/images/contribution-guide-1.png" | prepend: site.baseurl }} "Workflow image")

**For larger contributions or those that affect multiple components:**

1. **Engage**: We encourage you to work with the Beam community on the [Apache JIRA issue tracker](https://issues.apache.org/jira/browse/BEAM) and [developer’s mailing list](http://beam.incubator.apache.org/use/mailing-lists/) to identify good areas for contribution.
1. **Design:** More complicated contributions will likely benefit from some early discussion in order to scope and design them well.

**For all contributions:**

1. **Code:** The best part ;-)
1. **Review:** Submit a pull request with your contribution to our [GitHub mirror](https://github.com/apache/incubator-beam/). Work with a committer to review and iterate on the code, if needed.
1. **Commit:** A Beam committer merges the pull request into our [Apache repository](https://git-wip-us.apache.org/repos/asf/incubator-beam.git).

We look forward to working with you!

## Engage

### Mailing list(s)
We discuss design and implementation issues on <EMAIL> mailing list, which is archived [here](https://lists.apache.org/list.html?<EMAIL>). Join by emailing [`<EMAIL>`](mailto:<EMAIL>).

If interested, you can also join the other [mailing lists](http://beam.incubator.apache.org/use/mailing-lists/) too.

### Apache JIRA
We use [Apache JIRA](https://issues.apache.org/jira/browse/BEAM) as an issue tracking and project management tool, as well as a way to communicate among a very diverse and distributed set of contributors. To be able to gather feedback, avoid frustration, and avoid duplicated efforts all Beam-related work should be tracked there.

If you do not already have an Apache JIRA account, sign up [here](https://issues.apache.org/jira/).

If a quick [search](https://issues.apache.org/jira/issues/?jql=project%3DBEAM%20AND%20text%20~%20%22the%20thing%20I%20want%20to%20contribute%22) doesn’t turn up an existing JIRA issue for the work you want to contribute, create it. Please discuss your idea with a committer or the [component lead](https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:components-panel) in JIRA or, alternatively, on the developer mailing list.

If there’s an existing JIRA issue for your intended contribution, please comment about your intended work. Once the work is understood, a committer will assign the issue to you. (If you don’t have a JIRA role yet, you’ll be added to the “contributor” role.) If an issue is currently assigned, please check with the current assignee before reassigning.

For moderate or large contributions, you should not start coding or writing a design document unless there is a corresponding JIRA issue assigned to you for that work. Simple changes, like fixing typos, do not require an associated issue.

## Design

To avoid potential frustration during the code review cycle, we encourage you to clearly scope and design non-trivial contributions with the Beam community before you start coding.

Generally, the JIRA issue is the best place to gather relevant design docs, comments, or references. It’s great to explicitly include relevant stakeholders early in the conversation. For designs that may be generally interesting, we also encourage conversations on the developer’s mailing list.

We suggest using [Google Docs](https://docs.google.com/) for sharing designs that may benefit from diagrams or comments. Please remember to make the document world-commentable and add a link to it from the relevant JIRA issue. You may want to start from this [template](https://docs.google.com/document/d/1qYQPGtabN5-E4MjHsecqqC7PXvJtXvZukPfLXQ8rHJs/edit?usp=sharing).

## Code
To contribute code to Apache Beam, you’ll have to do a few administrative steps once, and then follow a few guidelines for each contribution.

### One-time Setup

#### [Potentially] Submit Contributor License Agreement
Apache Software Foundation (ASF) desires that all contributors of ideas, code, or documentation to the Apache projects complete, sign, and submit an [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt) (ICLA). The purpose of this agreement is to clearly define the terms under which intellectual property has been contributed to the ASF and thereby allow us to defend the project should there be a legal dispute regarding the software at some future time.

We require you to have an ICLA on file with the Apache Secretary for larger contributions only. For smaller ones, however, we rely on  [clause five](http://www.apache.org/licenses/LICENSE-2.0#contributions) of the Apache License, Version 2.0, describing licensing of intentionally submitted contributions and do not require an ICLA in that case.

#### Obtain a GitHub account
We use GitHub’s pull request functionality to review proposed code changes.

If you do not already have a personal GitHub account, sign up [here](https://github.com/join).

#### Fork the repository on GitHub
Go to the [Beam GitHub mirror](https://github.com/apache/incubator-beam/) and fork the repository to your own private account. This will be your private workspace for staging changes.

We recommend enabling Travis-CI continuous integration coverage on your private fork in order to easily test your changes before proposing a pull request. Go to [Travis-CI](https://travis-ci.org), log in with your GitHub account, and enable coverage for your repository.

#### Clone the repository locally
You are now ready to create the development environment on your local machine. Feel free to repeat these steps on all machines that you want to use for development.

We assume you are using SSH-based authentication with GitHub. If necessary, exchange SSH keys with GitHub by following [their instructions](https://help.github.com/articles/generating-an-ssh-key/).

Clone Beam’s read-only GitHub mirror.

    $ git clone https://github.com/apache/incubator-beam.git
    $ cd incubator-beam

Add your forked repository as an additional Git remote, where you’ll push your changes.

	$ git remote add <GitHub_user> git@github.com:<GitHub_user>/incubator-beam.git

You are now ready to start developing!

#### [Optional] IDE Setup

Depending on your preferred development environment, you may need to prepare it to develop Beam code.

##### Intellij

_If you are an Intellij user, please contribute instructions! See [BEAM-835](https://issues.apache.org/jira/browse/BEAM-835)._

##### Eclipse
 
Use a recent eclipse version that includes m2e. Currently we recommend Eclipse Neon.
Start eclipse with a fresh workspace in a separate directory from your checkout.

1. Install m2e-apt: Beam uses apt annotation processing to provide auto generated code. One example is the usage of [google auto value](https://github.com/google/auto/tree/master/value). By default m2e does not support this and you will see compile errors.

	Help
	-> Eclipse Marketplace
	-> Search for "m2 apt"
	-> Install m2e-apt 1.2 or higher

1. Activate the apt processing

	Window
	-> Preferences
	-> Maven
	-> Annotation processing
	-> Switch to Experimental: Delegate annotation processing ...
	-> Ok

1. Import the beam projects

	File
	-> Import...
	-> Existing Maven Projects
	-> Browse to the directory you cloned into and select incubator-beam
	-> make sure all beam projects are selected
	-> Finalize

You now should have all the beam projects imported into eclipse and should see no compile errors.


### Create a branch in your fork
You’ll work on your contribution in a branch in your own (forked) repository. Create a local branch, initialized with the state of the branch you expect your changes to be merged into. Keep in mind that we use several branches, including `master`, feature-specific, and release-specific branches. If you are unsure, initialize with the state of the `master` branch.

	$ git fetch --all
	$ git checkout -b <my-branch> origin/master

At this point, you can start making and committing changes to this branch in a standard way.

### Syncing and pushing your branch
Periodically while you work, and certainly before submitting a pull request, you should update your branch with the most recent changes to the target branch.

    $ git pull --rebase

Remember to always use `--rebase` parameter to avoid extraneous merge commits.

To push your local, committed changes to your (forked) repository on GitHub, run:

	$ git push <GitHub_user> <my-branch>	

### Testing
{'fraction_non_alphanumeric': 0.07598834332215319, 'fraction_numerical': 0.005498432946610216, 'mean_word_length': 4.580853022399509, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 48, 'https://': 35, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3735253', 'n_tokens_mistral': 4913, 'n_tokens_neox': 4639, 'n_words': 2370}