<commit_before># **Apache Spot (Incubating) Release Process**

Apache Spot is open source software for leveraging insights from flow and packet analysis. It helps enterprises and service providers gain insight on their computing environments through transparency of service delivery and identification of potential security threats or attacks happening among resources operating at cloud scale.

While current threat intelligence tools help, identifying unknown threats and attacks remains a challenge. Apache Spot provides tools to accelerate companies’ ability to expose suspicious connections and previously unseen attacks using flow and packet analysis technologies. 
<br><br>

----

## **Overview**  

This document descrbes the Release Process to perform the official Apache SPOT (Incubating) release following the [Apache Software Fundation Release Policy](http://incubator.apache.org/guides/releasemanagement.html#best-practice). 

### **Requirements**

As a Release Manager (RM) you should create a code signing key to sign the release artifacts following this [guide](http://www.apache.org/dev/openpgp.html#generate-key).

Public key should be deatached and added into the KEYS file in SPOT Repo under: https://dist.apache.org/repos/dist/dev/incubator/spot/KEYS


### **Policy documents**

Frequently asked questions for making Apache releases are available on Releases FAQ page.

The Release Manager must go through the policy document to understand all the tasks and responsibilities of running a release.

### **Give a heads up**

The RM should first create an EPIC in Jira and then setup a timeline for release branch point. The time for the day the EPIC is created to the release branch point must be at least two weeks in order to give the community a chance to prioritize and commit any last minute features and issues they would like to see in the upcoming release.

The RM should then send the pointer to the EPIC along with the tentative timeline (Code Freeze) for branch point to the user and developer lists. Any work identified as release related that needs to be completed should be added as a subtask of the umbrella issue to allow users to see the overall release progress in one place.

    To: <EMAIL>

    Subject: Work on Spot <your release name>  (Incubating) Release has started + Code Freeze Date


    We are starting the process to prepare for SPOT <your release name> (Incubating) release. I have opened JIRA $jira to cover the features included in this release.

    If you have any JIRA in progress and would like to include in this release, please follow the process to do so. Code Freeze for final integration will be on $code_freeze_date.

    Feel free to comment on the JIRA if you have any comments/suggestions.

    Thanks,
    <Release Manager Name>

### **Sanitize Jira**

Before a release is done, make sure that any issues that are fixed have their fix version setup correctly. If the release number is not listed in the "fix version" field, as RM create a ticket for Infraestructure asking to create the value for the "fix version" field. 

Once the value is created by Infraestructure team, run the following JIRA query to see which resolved issues do not have their fix version set up correctly:

project = spot and resolution = fixed and fixVersion is empty

The result of the above query should be empty. If some issues do show up in this query that have been fixed since the last release, please bulk-edit them to set the fix version to '1.0'.

You can also run the following query to make sure that the issues fixed for the to-be-released version look accurate:

project = spot and resolution = fixed and fixVersion = '1.0'

### **Monitor active issues**

It is important that between the time that the umbrella issue is filed to the time when the release branch is created, no experimental or potentially destabilizing work is checked into the trunk. While it is acceptable to introduce major changes, they must be thoroughly reviewed and have good test coverage to ensure that the release branch does not start of being unstable.

If necessary the RM can discuss if certain issues should be fixed on the trunk in this time, and if so what is the gating criteria for accepting them.

### **Pull Request Validation**

All the features that will be included inthe release EPIC needs to have a proper Pull Request (PR) created following this guide. And needs three +1 votes wich at least one of them must be from the QA Team. 

Once the PR has votes, then the PPMC developer must merge the PR and the owner of that PR should close it properly in Github (in case it does not closes automatically).

### **Create the Release Candidate**



#### **Branch your release:**

git checkout -b <your release name> <commit sha1> 
push to origin:
git push origin <your release name>

#### **Tag your Branch release:**
Apply signed tag on release branch that will indicate the 

Example:
 git tag -u <GPG KEY ID> --sign <your release name>-rc# -m "Apache SPOT <your release name> (Incubating)" <SHA of HEAD of branch>

#### **Run RAT**
Apache Rat is a release audit tool, focused on licenses. Used to improve accuracy and efficiency when checking releases for licenses.

Download RAT: 
wget http://apache.claz.org//creadur/apache-rat-0.12/apache-rat-0.12-bin.tar.gz

Uncompress the code:
tar -zxvf apache-rat-0.12-bin.tar.gz
cd apache-rat-0.12

Now lets create the file to exclude the known extensions and files:
vi or nano .rat-excludes

Add the following exclude list:

.*md
.*txt
.gitignore
.gitmodules
.*png
.*json
.*csvss
.*less
.*ipynb
.babelrc
topojson.min.js

Save the File.

Run the rat tool as following.
java -jar apache-rat-0.12.jar -E /path/to/project/.rat-excludes -d /path/to/project/ > <to output file>.txt

If you have rat in the same directory as the SPOT Code you can verify as:
java -jar apache-rat-0.12.jar -E .rat-excludes -d ../apache-spot-1.0-incubating > apache-spot-1.0-incubating-rat-results.txt

If RAT find problems in the licences please fix the licence and run RAT again developers must fix their code and submit changes into the release branch, once there are no more findings. Upload RAT Results into subversion dev incubator repo for SPOT of the release


Make a tarball and gzip:

git archive -o ../apache-spot-<your release name>-incubating.tar --prefix=apache-spot-<your release name>-incubating/ <your tag/branch name>
gzip ../apache-spot-<your release name>-incubating.tar

Example:

$ git archive -o ../apache-spot-1.0-incubating.tar --prefix=apache-spot-1.0-incubating/ 1.0-incubating
$ gzip ../apache-spot-1.0-incubating.tar

Prepare MD5, SHA512 and ASC files from the source tarball:

md5 apache-spot-<your release name>-incubating.tar.gz > apache-spot-<your release name>-incubating.tar.gz.md5
shasum -a 256 apache-spot-<your release name>-incubating.tar.gz > apache-spot-<your release name>-incubating.tar.gz.sha256 
gpg2 --detach-sign -a apache-spot-<your release name>-incubating.tar.gz

Example:
$ md5 apache-hawq-src-2.1.0.0-incubating.tar.gz > apache-hawq-src-2.1.0.0-incubating.tar.gz.md5
$ shasum -a 256 apache-hawq-src-2.1.0.0-incubating.tar.gz > apache-hawq-src-2.1.0.0-incubating.tar.gz.sha256
$ gpg2 --detach-sign -a apache-hawq-src-2.1.0.0-incubating.tar.gz
 

Retrieve the subversion dev incubator repo for SPOT

Example: 
svn checkout https://dist.apache.org/repos/dist/dev/incubator/spot/ --username=<your apache user>
 
Create a local folder for the release (e.g. 1.0-incubating) in svn. 
svn mkdir -m "Creating SPOT <release number> dir" https://dist.apache.org/repos/dist/dev/incubator/spot/<release number> --username=<your apache user>

Example:
svn mkdir -m "Creating SPOT 1.0-incuabting dir" https://dist.apache.org/repos/dist/dev/incubator/spot/1.0-incuabting --username=rpanduro


Move the files into the release folder on local disk.
svn add <release folder>

Example:
svn add 1.0-incubating/

Commit artifacts:
svn commit -m '<custom message>' --username=<your apache user id>

Example: 
svn commit -m 'adding spot 1.0-incubating candidate release artifacts' --username=<your apache user id>

### **Validate the Build**

<To Be Added>



### **Running the Vote**

As per the Apache Incubator release [guidelines](http://incubator.apache.org/policy/incubation.html#Releases), all releases for incubating projects must go through a two-step voting process. First, release voting must successfully pass within the Apache SPOT (Incubating) community via the <EMAIL> mail list. Then, release voting must successfully pass within the Apache Incubator PMC via the <EMAIL> mail list.


Call for SPOT dev list votes Send an email to dev@ list. 

For example,

    To: <EMAIL><commit_msg>Updates Release Process README.md file<commit_after># **Apache Spot (Incubating) Release Process**

Apache Spot is open source software for leveraging insights from flow and packet analysis. It helps enterprises and service providers gain insight on their computing environments through transparency of service delivery and identification of potential security threats or attacks happening among resources operating at cloud scale.

While current threat intelligence tools help, identifying unknown threats and attacks remains a challenge. Apache Spot provides tools to accelerate companies’ ability to expose suspicious connections and previously unseen attacks using flow and packet analysis technologies. 
<br><br>

----

## **Overview**  

This document descrbes the Release Process to perform the official Apache SPOT (Incubating) release following the [Apache Software Fundation Release Policy](http://incubator.apache.org/guides/releasemanagement.html#best-practice). 

### **Requirements**

As a Release Manager (RM) you should create a code signing key to sign the release artifacts following this [guide](http://www.apache.org/dev/openpgp.html#generate-key).

Public key should be deatached and added into the KEYS file in SPOT Repo under: https://dist.apache.org/repos/dist/dev/incubator/spot/KEYS


### **Policy documents**

Frequently asked questions for making Apache releases are available on Releases FAQ page.

The Release Manager must go through the policy document to understand all the tasks and responsibilities of running a release.

### **Give a heads up**

The RM should first create an EPIC in Jira and then setup a timeline for release branch point. The time for the day the EPIC is created to the release branch point must be at least two weeks in order to give the community a chance to prioritize and commit any last minute features and issues they would like to see in the upcoming release.

The RM should then send the pointer to the EPIC along with the tentative timeline (Code Freeze) for branch point to the user and developer lists. Any work identified as release related that needs to be completed should be added as a subtask of the umbrella issue to allow users to see the overall release progress in one place.

    To: <EMAIL>

    Subject: Work on Spot <your release name>  (Incubating) Release has started + Code Freeze Date


    We are starting the process to prepare for SPOT <your release name> (Incubating) release. I have opened JIRA $jira to cover the features included in this release.

    If you have any JIRA in progress and would like to include in this release, please follow the process to do so. Code Freeze for final integration will be on $code_freeze_date.

    Feel free to comment on the JIRA if you have any comments/suggestions.

    Thanks,
    <Release Manager Name>

### **Sanitize Jira**

Before a release is done, make sure that any issues that are fixed have their fix version setup correctly. If the release number is not listed in the "fix version" field, as RM create a ticket for Infraestructure asking to create the value for the "fix version" field. 

Once the value is created by Infraestructure team, run the following JIRA query to see which resolved issues do not have their fix version set up correctly:

project = spot and resolution = fixed and fixVersion is empty

The result of the above query should be empty. If some issues do show up in this query that have been fixed since the last release, please bulk-edit them to set the fix version to '1.0'.

You can also run the following query to make sure that the issues fixed for the to-be-released version look accurate:

project = spot and resolution = fixed and fixVersion = '1.0'

### **Monitor active issues**

It is important that between the time that the umbrella issue is filed to the time when the release branch is created, no experimental or potentially destabilizing work is checked into the trunk. While it is acceptable to introduce major changes, they must be thoroughly reviewed and have good test coverage to ensure that the release branch does not start of being unstable.

If necessary the RM can discuss if certain issues should be fixed on the trunk in this time, and if so what is the gating criteria for accepting them.

### **Pull Request Validation**

All the features that will be included inthe release EPIC needs to have a proper Pull Request (PR) created following this guide. And needs three +1 votes wich at least one of them must be from the QA Team. 

Once the PR has votes, then the PPMC developer must merge the PR and the owner of that PR should close it properly in Github (in case it does not closes automatically).

### **Create the Release Candidate**

#### **Branch your release:**

* git checkout -b `<your release name>` `<commit sha1>` 

push to origin:

* git push origin `<your release name>`

#### **Tag your Branch release:**
Apply signed tag on release branch that will indicate the 

Example:
* git tag -u `<GPG KEY ID>` --sign `<your release name>`-incubating -m "Apache SPOT `<your release name>` (Incubating)" `<SHA of HEAD of branch>`

#### **Run RAT**
Apache Rat is a release audit tool, focused on licenses. Used to improve accuracy and efficiency when checking releases for licenses.

Download RAT: 
* wget http://apache.claz.org//creadur/apache-rat-0.12/apache-rat-0.12-bin.tar.gz

Uncompress the code:
* tar -zxvf apache-rat-0.12-bin.tar.gz
* cd apache-rat-0.12

Now lets create the file to exclude the known extensions and files:
* vi or nano .rat-excludes

Add the following exclude list:

    .*md
    .*txt
    .gitignore
    .gitmodules
    .*png
    .*json
    .*csvss
    .*less
    .*ipynb
    .babelrc
    topojson.min.js

Save the File.

Run the rat tool as following.
* java -jar apache-rat-0.12.jar -E /path/to/project/.rat-excludes -d /path/to/project/ > `<to output file>`.txt

If you have rat in the same directory as the SPOT Code you can verify as:
* java -jar apache-rat-0.12.jar -E .rat-excludes -d ../apache-spot-1.0-incubating > apache-spot-1.0-incubating-rat-results.txt

If RAT find problems in the licences please fix the licence and run RAT again developers must fix their code and submit changes into the release branch, once there are no more findings. Upload RAT Results into subversion dev incubator repo for SPOT of the release


Make a tarball and gzip:
* git archive -o ../apache-spot-`<your release name>`-incubating.tar --prefix=apache-spot-`<your release name>`-incubating/ `<your tag/branch name>`
* gzip ../apache-spot-`<your release name>`-incubating.tar

Example:
    $ git archive -o ../apache-spot-1.0-incubating.tar --prefix=apache-spot-1.0-incubating/ 1.0-incubating
    $ gzip ../apache-spot-1.0-incubating.tar

Prepare MD5, SHA512 and ASC files from the source tarball:

* md5 apache-spot-`<your release name>`-incubating.tar.gz > apache-spot-`<your release name>`-incubating.tar.gz.md5
* shasum -a 512 apache-spot-`<your release name>`-incubating.tar.gz > apache-spot-`<your release name>`-incubating.tar.gz.sha512 
* gpg2 --detach-sign -a apache-spot-`<your release name>`-incubating.tar.gz

Example:
    $ md5 apache-spot-1.0-incubating..tar.gz > apache-spot-1.0-incubating.tar.gz.md5
    $ shasum -a 512 apache-spot-1.0-incubating.tar.gz > apache-spot-1.0-incubating.tar.gz.sha512
    $ gpg2 --detach-sign -a apache-spot-1.0-incubating..tar.gz
 

Retrieve the subversion dev incubator repo for SPOT

Example:
* svn checkout https://dist.apache.org/repos/dist/dev/incubator/spot/ --username=`<your apache user>`
 
Create a local folder for the release (e.g. 1.0-incubating) in svn. 
* svn mkdir -m "Creating SPOT `<release number>` dir" https://dist.apache.org/repos/dist/dev/incubator/spot/`<release number>` --username=`<your apache user>`

Example:
    svn mkdir -m "Creating SPOT 1.0-incuabting dir" https://dist.apache.org/repos/dist/dev/incubator/spot/1.0-incuabting --username=`<your apache user>`


Move the files into the release folder on local disk.
* svn add `<release folder>`

Example:
    svn add 1.0-incubating/

Commit artifacts:
* svn commit -m '`<custom message>`' --username=`<your apache user id>`

Example: 
    svn commit -m 'adding spot 1.0-incubating candidate release artifacts' --username=`<your apache user id>`

### **Validate the Build**

`<To Be Added>`



### **Running the Vote**

As per the Apache Incubator release [guidelines](http://incubator.apache.org/policy/incubation.html#Releases), all releases for incubating projects must go through a two-step voting process. First, release voting must successfully pass within the Apache SPOT (Incubating) community via the <EMAIL> mail list. Then, release voting must successfully pass within the Apache Incubator PMC via the <EMAIL> mail list.


Call for SPOT dev list votes Send an email to dev@ list. 

For example,

    To: <EMAIL>
{'fraction_non_alphanumeric': 0.08006610439936175, 'fraction_numerical': 0.00945976749487121, 'mean_word_length': 4.0355810616929695, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 86, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 5, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7209343', 'n_tokens_mistral': 5165, 'n_tokens_neox': 4850, 'n_words': 2367}