<commit_before># Building and Deploying the IoT Example System

This document describes how to build and deploy the IoT example system. 

You can download the code for the solution [here](https://github.com/mspnp/iot-journey/tree/master/src).

You should also download the [provisioning scripts](https://github.com/mspnp/iot-journey/tree/master/provision) that create the Azure assets required by the solution.

The solution comprises the following projects:

- ScenarioSimulator. This project *TBD* ...

- Device.Events. This project *TBD* ...

- Core. This project *TBD* ...

- ScenarioSimulator.ConsoleHost. This project *TBD* ...

- ScenarioSimulator.Tests and Core.Tests (in the UnitTests folder). These projects *TBD* ...

## Prerequisites

To use the sample solution, you must have an Azure account with a subscription. You should also have installed the following software on your computer:

- [Visual Studio](https://www.visualstudio.com/)

- [Azure SDK for .NET (Visual Studio tools)](http://azure.microsoft.com/downloads/)

- [Azure Powershell](https://azure.microsoft.com/documentation/articles/powershell-install-configure/)

## Overview

To build and deploy the system, perform the following steps:

1. Download and install the NuGet packages and assemblies required by the solution.

2. Create the required Azure assets and services.

3. Create the configuration file for the solution with your Azure account information.

4. Configure Stream Analytics.

5. Configure HDInsight.

6. Test and verify the solution.

The following sections describe these steps in more detail.

### Downloading and Installing the Required Assemblies

- Using Visual Studio, open the IoTJourney solution.

- Rebuild the solution. This action downloads the various NuGet packages referenced by the solution and installs the appropriate assemblies.

> **Note:** The rebuild process should report 1 error: *Before building this project, please copy mysettings-template.config to mysettings.config ...* You will do this in a later step. If rebuild process displays any other errors then clean the solution and rebuild it again.

### Creating the Required Azure Assets and Services

- Using the Azure portal, manually create the following items:

	- [A Service Bus namespace](https://msdn.microsoft.com/library/azure/hh690931.aspx). The namespace should be created in the Central US region. Set the namespace type to Notification Hub.

	- [A storage account](https://azure.microsoft.com/documentation/articles/storage-create-storage-account/). Set the location/affinity group to Central US.

- Start Azure PowerShell as administrator.

> **Note:** Windows Powershell must be configured to run scripts. You can do this by running the following PowerShell command before continuing:
> 
> `set-executionpolicy unrestricted`

- Move to the folder containing the provisioning scripts.

- Run the following command:

	`.\Provision-All.ps1`

- At the *ServiceBusNamespace* prompt, enter the name of the Service Bus namespace you created in step 1.

- At the *StorageAccountName* prompt, enter the name of the storage account you created in step 1.

- At the *StreamAnalyticsJobName* prompt, enter a name for the Stream Analytics job to be created (the script will create this item).

-  At the *SubscriptionName* prompt, enter the name of the subscription that you are using with your Azure account (this subscription should be the owner of the Service Bus namespace and the storage account).

-  In the Sign in to Windows Azure Powershell dialog box, provide the credentials for your Azure account.

### Creating the Configuration File

- Using File Explorer create of a copy of the file named *mysettings-template.config* in the RunFromConsole folder as *mysettings.config*

- In Visual Studio, open the *mysettings.config* file in the ScenarioSimulator.ConsoleHost project. 

- Using the Azure portal, find the connection string for the eventhub01 event hub in the Service Bus namespace that you created earlier.

- In Visual Studio, set the value of the Simulator.EventHubConnectionString key to the connection string for the eventhub01 event hub.

- Rebuild the solution. It should build successfully.

### Configuring Stream Analytics

### Configuring HDInsight

## Testing and Verifying the Solution

### Running the Simulator

### Using the Unit Tests

### Validating with Stream Analytics

### Validating by Using Hive Queries

### Validating by Inspecting Blob Data<commit_msg>Filled out empty sections - still needs more work to complete<commit_after># Building and Running the IoT Sample Solution

This document describes how to build and run the sample IoT solution. The solution simulates a large number of devices sending environmental readings for apartments in a smart building (for more information, see the [scenario](https://github.com/mspnp/iot-journey/blob/master/docs/journal/00-introducing-the-journey.md)). The simulator generates events that are posted to an Azure event hub. Azure Stream Analytics is used to process these events and save the results to blob storage. Optionally, the data can subsequently be processed by using a Hive query running on a Hadoop cluster. The query calculates the number of events generated by each device. 

The simulator enables you to test the following scenarios:

1. All devices sending readings that are within expected ranges.

2. Some devices send readings indicating that the temperature is excessive (over 30 degrees Celsius).

3. *Devices send readings that are not handled - What does this actually mean?*.

4. Devices send readings containing malformed data.

5. *LONG RUNNING EVENTS?*

You can download the code for the solution [here](https://github.com/mspnp/iot-journey/tree/master/src).

You should also download the [provisioning scripts](https://github.com/mspnp/iot-journey/tree/master/provision) that create the Azure assets required by the solution.

The solution comprises the following projects:

- ScenarioSimulator. This project *TBD* ...

- Device.Events. This project *TBD* ...

- Core. This project *TBD* ...

- ScenarioSimulator.ConsoleHost. This project provides the user interface to the simulator. It implements a menu that enables the user to run each of the scenarios described above.

- ScenarioSimulator.Tests and Core.Tests (in the UnitTests folder). These projects contain unit tests to verify that the simulator is running correctly *What do these tests actually show? TBD* ...

## Prerequisites

To use the sample solution, you must have an Azure account with a subscription. You should also have installed the following software on your computer:

- [Visual Studio](https://www.visualstudio.com/)

- [Azure SDK for .NET (Visual Studio tools)](http://azure.microsoft.com/downloads/)

- [Azure Powershell](https://azure.microsoft.com/documentation/articles/powershell-install-configure/)

## Building the Sample Solution

To build and deploy the system, perform the following steps:

1. Download and install the NuGet packages and assemblies required by the solution.

2. Create the required Azure assets and services.

3. Create the configuration file for the solution with your Azure account information.

4. Configure Stream Analytics.

5. Configure HDInsight (optional).

The following sections describe these steps in more detail.

### Downloading and Installing the Required Assemblies

- Using Visual Studio, open the IoTJourney solution.

- Rebuild the solution. This action downloads the various NuGet packages referenced by the solution and installs the appropriate assemblies.

> **Note:** The rebuild process should report one error: *Before building this project, please copy mysettings-template.config to mysettings.config ...* You will do this in a later step. If rebuild process displays any other errors then clean the solution and rebuild it again.

### Creating the Required Azure Assets and Services

- Using the Azure web portal, manually create the following items:

	- [A Service Bus namespace](https://msdn.microsoft.com/library/azure/hh690931.aspx). The namespace should be created in the Central US region. Set the namespace type to Notification Hub.

	- [A storage account](https://azure.microsoft.com/documentation/articles/storage-create-storage-account/). Set the location/affinity group to Central US.

- Start Azure PowerShell as administrator.

> **Note:** Windows Powershell must be configured to run scripts. You can do this by running the following PowerShell command before continuing:
> 
> `set-executionpolicy unrestricted`

- Move to the folder containing the provisioning scripts.

- Run the following command:

	`.\Provision-All.ps1`

- At the *ServiceBusNamespace* prompt, enter the name of the Service Bus namespace you created in step 1.

- At the *StorageAccountName* prompt, enter the name of the storage account you created in step 1.

- At the *StreamAnalyticsJobName* prompt, enter a name for the Stream Analytics job to be created (the script will create this item).

-  At the *SubscriptionName* prompt, enter the name of the subscription that you are using with your Azure account (this subscription should be the owner of the Service Bus namespace and the storage account).

-  In the Sign in to Windows Azure Powershell dialog box, provide the credentials for your Azure account.

### Creating the Configuration File

- Using File Explorer create of a copy of the file named *mysettings-template.config* in the RunFromConsole folder as *mysettings.config*

- In Visual Studio, open the *mysettings.config* file in the ScenarioSimulator.ConsoleHost project. 

- Using the Azure portal, find the connection string for the eventhub01 event hub in the Service Bus namespace that you created earlier.

- In Visual Studio, set the value of the Simulator.EventHubConnectionString key to the connection string for the eventhub01 event hub. Append the text `;TransportType=Amqp` to the end of the string

> **Note:** It is important to specify the transport type because the default protocol is not supported by Azure Event Hub and will trigger exceptions at runtime.

- Rebuild the solution. It should build successfully.

### Configuring Stream Analytics

The provisioning script should configure Stream Analytics automatically to receive events from the event hub and write a record of each event to blob storage. Event data is written using JSON format.

To verify that the configuration was successful, perform the following steps:

- Using the Azure web portal, open the Stream Analytics page.

- On the Stream Analytics page, verify that a Stream Analytics job has been created using the name that you specified when you ran the provisioning script (the name that you entered in response to the *StreamAnalyticsJobName* prompt).

- Click the job, and then in the menu bar click Configure

- On the Configuration page, verify that the storage account is set to the value that you specified when you ran the provisioning script (the name that you entered in response to the *StorageAccountName* prompt).

- In the menu bar click Inputs.

- On the Inputs page, verify that an input named input01 has been created and that the type of the input is Event Hub.

- Click input01.

- On the input01 page, in the General section, verify that:

	-  The Service Bus namespace is set to the value that you specified when you ran the provisioning script (the name that you entered in response to the *ServiceBusNamespace* prompt).

	-  The Event Hub name is set to eventhub01.

	-  The Event Hub policy name is set to ManagePolicy.

	-  The Event Hub consumer group is set to consumergroup01.

- In the Serialization section, verify that the event serialization format is set to JSON.

- Return to the job page, and then in the menu bar click Query

- On the Query page, verify that the following query is specified:
	`SELECT * FROM input01`

- Return to the job page, and then in the menu bar click Outputs.

- On the Outputs page, verify that an output named output01 has been created and that the output type is set to Blob Storage

- Click output01

- On the output01 page, in the General section, verify that:

	-  The storage account is set to the value that you specified when you ran the provisioning script (the name that you entered in response to the *StorageAccountName* prompt).

	-  The container is set to container01.

	-  The filename prefix is set to fabrikam.

	- In the Serialization section, verify that the event serialization format is set to JSON.

### Configuring HDInsight (Optional)

Perform the following steps to configure a Hadoop cluster by using HDInsight:

- Using the Azure web portal, open the HDInsight page.

- On the command bar, click New.

- In the New dialog box, select Hadoop. Enter a cluster name and admin password of your choice, specify the storage account that you used when you ran the provisioning script, and then click Create HDInsight Cluster.

> **Note:** It can take several minutes to create the cluster.

## Testing and Running the Solution

The following sections describe how to test and run the simulator.

### Running the Unit Tests

The UnitTests project ... (*What do these tests demonstrate? - TBD*)

To run the unit tests, on the Visual Studio menu bar click Test, click Run, and then click All Tests.

All tests should succeed without errors.

### Running the Simulator

Perform the following steps to run the simulator:

- Using the Azure web portal, open the Stream Analytics page.

- Select your Stream Analytics job, and on the command bar click Start.

- In the Start Output dialog box, select Job Start Time and then click the tick button.

- Wait for the Stream Analytics job to start before continuing.

- Using Visual Studio, run the ScenarioSimualtor.ConsoleHost project.

- On the menu, select option 1 (*Run NoErrorsExpected*). The simulator will generate events and echo their contents to the screen. The simulator should not report any exceptions.

- Allow the simulator to run for a couple of minutes and then press q.

- Press Enter to return to the menu.

- Repeat the previous three steps for options 2 to 5. As before, the simulator will generate and echo events and no exceptions should occur.

- On the menu, select option 6 to quit the simulator.

### Validating the Stream Analytics output

Perform the following steps to verify that events are being processed correctly by Stream Analytics:

- Using the Visual Studio Server Explorer window, connect to your Azure account.

- Expand the Storage node, expand the node that corresponds to your storage account, expand Blobs, and verify that a container named container01 has been created (you may need to refresh the display if the container does not appear).

- Double-click container01 to display the container01 contents pane.

- On the container01 contents, verify that the container has a folder named fabrikam.

- Double-click the fabrikam folder and verify that it contains one or more JSON files (files with random named but with the .json suffix).

- Double-click the most recent file to download and view the contents. Verify that the file contains a number of JSON formatted event records (there should be one record for each event that was generated when the simulator ran, although you probably didn't count them at the time!)

### Validating the Blob Data by Using a Hive Query (Optional)

Perform the following steps to process the data in blob storage by using a Hive query:

- Start Azure PowerShell as administrator.

> **Note:** Windows Powershell must be configured to run scripts. You can do this by running the following PowerShell command before continuing:
> 
> `set-executionpolicy unrestricted`

- Move to the folder containing the source code for the solution scripts, and then move to the Validation\HDInsight subfolder.

- Run the following command:

	`.\hivequery.ps1`

- At the *subscriptionName* prompt, enter the name of the subscription that  owns the storage account holding the blob data used by the simulator.

- At the *storageAccountName* prompt, enter the name of the storage account.

- At the *containerName* prompt, type container01 (this is the name of the container that the simulator uses to hold the blob data).

-  At the *clusterName* prompt, enter the name of the Hadoop cluster that you created in the section Configuring HDInsight.

-  In the Sign in to Windows Azure Powershell dialog box, provide the credentials for your Azure account.

- Verify that the message Successfully connected to cluster *cluster name* appears (where *cluster name* is the name of your Hadoop cluster)

- *NEED MORE INFORMATION HERE - WHERE DO THE QUERY RESULTS GO?*
{'fraction_non_alphanumeric': 0.056015850144092216, 'fraction_numerical': 0.004923150816522575, 'mean_word_length': 4.238050314465409, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 16, 'https://': 13, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5389615', 'n_tokens_mistral': 4161, 'n_tokens_neox': 3968, 'n_words': 2321}