# Streaming at Scale integration tests

## Integration test suite

The suite is designed to run in a fully automated manner in Azure DevOps. It
deploys each scenario from the streaming-at-scale project in a new resource
group, then runs a verification job in Azure Databricks that reads the entire
set of events generated by the scenario, and tests assertions on the throughput
and latency. For instance, after running the eventhubs-functions-cosmosdb
scenario at 5000 events/second, the verification job downloads the output data
stored in CosmosDB and computes the end-to-end latency and throughput
aggregated at one-minute interval. To account for some runtime variability, the
test asserts that a throughput of at least 90% of the expected throughput (in
that case, 270,000 events in a one-minute interval) was reached at least once.

Since the provisioning of an Azure Databricks workspace cannot be fully
automated at present, you must generate a PAT token of a preexisting workspace
and supply it to the pipeline.

## Installing the build agent

As the integration tests can run for more than 6 hours, they must be run on self-hosted Azure DevOps agents.

Create a project in Azure DevOps. Create an agent pool named "streaming-at-scale".

In the Azure portal, create an Azure VM with Ubuntu 18.04 LTS.

SSH to the VM and run the following commands interactively one a time.

```bash
# Install Azure DevOps agent. When prompted enter the Azure DevOps host and a PAT token with Agent Pool management permissions.
mkdir agent
cd agent
wget https://vstsagentpackage.azureedge.net/agent/2.155.1/vsts-agent-linux-x64-2.155.1.tar.gz
tar zxvf vsts-agent-linux-x64-2.155.1.tar.gz 
./config.sh 
sudo ./svc.sh install
sudo ./svc.sh start
# Install jq
sudo apt update
sudo apt install jq
# Install zip
sudo apt install zip
# Install az (Azure CLI)
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
# Install dotnet SDK
wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo add-apt-repository universe
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2
# Install maven
sudo apt install maven
```


## Creating the integration test pipeline in Azure DevOps

* Create a Databricks workspace in the Azure region of your choice:
  * tier: standard
  * make sure the workspace is deployed with a custom VNET (as the HDInsight
    Kafka setup will need to peer VNETs).
  * You can use the [Databricks VNET
  template](https://azure.microsoft.com/en-us/resources/templates/101-databricks-all-in-one-template-for-vnet-injection/),
  changing the tier to standard on the deployment screen.
* Install a build agent (instructions below).
* In Azure AD, create a service principal. Grant the service principal
  *Owner* permission on your subscription (we can't use the automatic
  service connection dialog in Azure DevOps, as that would only grant
  *Contributor* permissions. We need *Owner* in order to grant Azure Kubernetes
  Service permissions to pull containers from Azure Container Registry).
* In your Azure DevOps project settings, navigate to service connection and
  create an ARM service connection to your Azure subscription named
  'ARMConnection'. *Use the full version* of the dialog to use your service
  principal.
* Create a build pipeline:
  * As pipeline source, use https://github.com/Azure-Samples/streaming-at-scale.git.
  * As pipeline content, reference integration-tests/azure-pipelines.yaml.
  * Define the following  pipeline variables:

| Variable name          | Description                                    | Required? | Example    |
| --------------------   | ---------------------------------------------- | --------- | ---------- |
| LOCATION               | Azure region in which to deploy infrastructure | required  | eastus     |
| DATABRICKS_PAT_TOKEN   | (secret variable) Databricks PAT token for a Databricks workspace deployed in $LOCATION | required | dapi012345... |
| DATABRICKS_VNET | Databricks VNET name (default value: 'databricks-vnet') | optional | databricks-vnet |
| DATABRICKS_VNET_RESOURCE_GROUP | Resource Group containing the Databricks VNET | required | streamingitests |
| RESOURCE_GROUP_PREFIX  | Prefix used to name deployed resources. Must be globally unique, use a sufficiently unique string  | required | xyzzy0x4 |
| AGENT_VM_RESOURCE_GROUP | Resource group of the build agent VM  | required | streamingitests |
| AGENT_VM_NAME          | Name of the build agent VM  | required | streamingbuildagent |


## Running the integration tests

Running the pipeline will generate multiple runner jobs, and execute them in
sequence. Each scenario is deployed in a newly created resource group, that is
immediately deleted after the test is run.

At the end of the job output of the 'Run test' task is a link to the Databricks
job run URL for the verification job. You can open this URL to view the run
notebook output. At the end of the main notebook is a call to the
'verify-common' notebook job, open that notebook run to view detailed
statistics collected by the verification job.

