This demo provides an end-to-end architecture to enable object detection using IoT Edge, IoT Central, and PowerApps.

For a quicker setup, you can use an ARM file to deploy all the required resources in the solution.

1. Setup your Azure subscription if you don't have one already. You can setup an account HERE.

2. Create a new Resource Group in the Azure Portal:

   • Login to the Azure portal.

   • Click the Create a resource button in the portal.

   • Enter the text Resource group in the search input field and select the displayed option.

   • Click the Create button in the new page.

   • Enter the required information:

     • Select the Subscription.
     • Enter the Resource group name.
     • Select the Region where to setup the Resource Group. Keep in mind that all resources will be deployed to this region so make sure it supports all of the required services. The template has been confirmed to work in West US 2.

   • Click the Review + Create button at the bottom of the page.

   • Click Create to finish the creation of the Resource Group.

Follow the steps to deploy the required Azure resources:

1. In the Azure portal select the Resource Group you created in the previous step.

2. Click the + Add button or Create resources.

3. Enter the text Template deployment in the search input field and select the display option.

4. Click the Create button.

5. Click the Build your own template in the editor link.

6. Click the Load file button, search and select the arm-template.json file inside the deployment folder and click Open.

7. Click the Save button at the left-bottom corner.

8. Select the Location where to run the deployment.

9. Review and update if required all the SETTINGS:

   • Prefix: This value will be added to the resource names.
   • Administrator Login: Username account for the SQL Server (default: theadmin).
   • Administrator Login Password: Password for the administrator account of the SQL Server (default: M1cro$oft2020).

10. Read and accept the TERMS AND CONDITIONS by checking the box.

11. Click the Purchase button and wait for the deployment to finish.

12. Review the output values:

    • Go to your Resource group and click Deployments from the left navigation.
    • Click Microsoft.Template
    • Click Outputs from the left navigation.
    • Save values for future use.

    IMPORTANT: You will need these values later in the setup.

    NOTE: Connection to the SQL server is allowed from all IP Addresses by default. To update this rule, follow the instructions in the Optional Steps section.

Some resources require some extra configuration.

Here we will run the script for the creation of the tables required by the solution.

1. In the Azure portal select the Resource Group you created earlier.

2. Select the SQL database resource.

3. Click the Query Editor option in the menu.

4. Enter the username and password that you used as parameters during deployment and click the Connect button.

5. Copy the following script to the Query area:

   CREATE TABLE alerts (
       IncidentId UNIQUEIDENTIFIER DEFAULT NEWID(),
       DeviceId NVARCHAR(50) NOT NULL,
       IncidentType NVARCHAR(50) NOT NULL,
       Status NVARCHAR(50) NOT NULL,
       BlobUrl NVARCHAR(255) NOT NULL,
       AlertNotified BIT DEFAULT 0,
       ReportedTime DATETIME NOT NULL,
       LastUpdated DATETIME
   )

6. Click the Run button.

7. You should be able to see the created table in the database.

8. Now replace the content of the Query area with the following code:

   CREATE PROCEDURE [dbo].[spFetchUpdateAlerts]
       @NewAlertsCount integer = NULL OUTPUT,
       @BlobUrl nvarchar(255) = NULL OUTPUT
   AS
   BEGIN
       SET NOCOUNT ON
   
       SET @NewAlertsCount =
       (
       SELECT COUNT(*)
       FROM dbo.alerts
       WHERE alerts.AlertNotified = 0 AND UPPER(alerts.Status) = 'UNRESOLVED'
       );
   
       IF (@NewAlertsCount = 0)
       BEGIN
       SET @BlobUrl = ''
       END
       ELSE
       BEGIN
       SET @BlobUrl = (
           SELECT BlobUrl
           FROM dbo.alerts
           WHERE alerts.AlertNotified = 0 AND UPPER(alerts.Status) = 'UNRESOLVED'
       );
       END
   
       DECLARE @MyTableVar table (
       [IncidentId] [uniqueidentifier] NULL,
       [DeviceId] [nvarchar](50) NOT NULL,
       [IncidentType] [nvarchar](50) NOT NULL,
       [Status] [nvarchar](50) NOT NULL,
       [BlobUrl] [nvarchar](255) NOT NULL,
       [AlertNotified] [bit] NULL,
       [ReportedTime] [datetime] NOT NULL,
       [LastUpdated] [datetime] NULL);
   
       UPDATE alerts
       SET alerts.AlertNotified = 1
       OUTPUT INSERTED.*
       INTO @MyTableVar
       WHERE alerts.AlertNotified = 0 AND UPPER(alerts.Status) = 'UNRESOLVED'
       SELECT *
       FROM @MyTableVar;
   END

9. Click the Run button.

Follow the next steps to setup the stream job.

1. In the Azure portal select the Resource Group you created earlier.
2. Select the Stream Analytics Job resource with the name ending with EdgeStreamJob.
3. Click Storage account settings from the left navigation.
4. Click Add storage account.
5. Select the storage account created in the ARM setup and add a new container named edgemodules.
6. Click Save and confirm.
7. Select Publish from the left navigation.
8. Click Publish and confirm;
9. Wait for the operation to complete.
10. Save the SAS URL as you will need this later in the device deployment.

1. Install Visual Studio Code (VS Code).
2. Install 64 bit Anaconda with Python version 3.7.
3. Install Docker Community Edition (CE). Don't sign in to Docker Desktop after Docker CE is installed.
4. Install the following extensions for VS Code:
   • Azure Machine Learning (Azure Account and the Microsoft Python will be automatically installed)
   • Azure IoT Hub Toolkit
   • Azure IoT Edge
   • Docker Extension
5. Restart VS Code.
6. Select [View > Command Palette…] to open the command palette box, then enter [Python: Select Interpreter] command in the command palette box to select your Python interpreter.
7. Enter [Azure: Sign In] command in the command palette box to sign in Azure account and select your subscription.

1. Launch Visual Studio Code, and open the $/device folder.

2. Update the .env file with the values for your container registry.

   • In the Azure portal select the Resource Group you created earlier.

   • Select the Container Registry resource.

   • Select Access Keys from the left navigation.

   • Update the following in device/.env with the following values from the Access Keys within the Container Registry:

     CONTAINER_REGISTRY_ADDRESS=<Login Server> (Ensure this is the login server and NOT the Registry Name)

     CONTAINER_REGISTRY_USERNAME=<Username>

     CONTAINER_REGISTRY_PASSWORD=<Password>

   • Save the file.

3. Sign in to your Azure Container Registry by entering the following command in the Visual Studio Code integrated terminal (replace <CONTAINER_REGISTRY_ADDRESS>, <CONTAINER_REGISTRY_USERNAME>, and <CONTAINER_REGISTRY_PASSWORD> with your container registry values set in the .env file).

   docker login -u <CONTAINER_REGISTRY_USERNAME> -p <CONTAINER_REGISTRY_PASSWORD> <CONTAINER_REGISTRY_ADDRESS>

   IMPORTANT: If you would llke to deploy your own model please refer to the Custom Vision Project Setup section in the Optional Steps section later in this document. You will need to increment the version number in the tag property of device/modules/ImageClassifierService/module.json if you wish to push another model after the initial one.

1. Open device/deployment.test-amd64.template.json.

2. Update the following desired properties for the EdgeStreamJob module with the value you got from Edge Stream Analytics Job earlier.

   "ASAJobInfo": "<ASA Blob URL>"

3. Next we need to update the Camera Capture module desired properties. In the Azure portal, select the Resource Group you created earlier.

4. Select the Storage Account resource with the name ending with storage.

5. Go to Access Keys in the left pane.

6. Copy the Storage account name and Key value from the output.

   • Alternatively, you can generate a SAS for the container.
   • Click on Storage Explorer (preview) from the left pane.
   • Expand BLOB CONTAINERS and right-click on forkliftimages and select Get Shared Access Signature.
   • Ensure Create and Write are selected from the permissions.
   • Obtain the key.

   NOTE: Either key will be fine for the next step. Creating a SAS key gives you more granular control over permissions to the container.

7. Update the following desired properties for the CameraCapture module.

   "VideoPath": "rtsp://admin:@<camera_IP>//h264Preview_01_main",
   "StorageAccountName" : "<Storage account name from previous step>",
   "StorageAccountKey" : "<Key from previous step>"

   NOTE: the video path with be specific to the camera in use. Check the manufacturer's documentation for more details.

8. Save the file.

1. Right click on device/deployment.test-amd64.template.json.
2. Select Build and Push IoT Edge Solution.
3. Wait for the process to complete.
4. Go to device/config and save deployment.test-amd64.json as you will need this for the IoT Central setup later.

Creating a new IoT Central environment is outside the scope of this document. Learn more about getting started here.

Once your environment is created, complete the following steps to configure it for the demo.

1. Sign in to your IoT Central environment.
2. Click + New.
3. Click Azure IoT Edge.
4. Click Next: Customize.
5. Click Browse.
6. Upload the device/config/deployment.test-amd64.json file.
7. Click Next: Review.
8. Click Create.

1. Click on Module ImageClassifierService.
2. Click Delete (and then Delete again).

   NOTE: this module doesn't have any properties or telemetry so is considered invalid during publishing.

3. Under Module EdgeStreamJob, click Manage.
4. Using + Add capability, add the following items. Each is of capability type Telemetry:
   • Display Name: message_type, schema: String
   • Display Name: event_type, schema: String
   • Display Name: timestamp, schema: DateTime
   • Display Name: count, schema: Integer
   • Display Name: blob_url, schema: String

   NOTE: all other values can be left as their default.

5. Click Save.
6. Click Views.
7. Click Visualizing the device.
8. Check count.
9. Click Add tile.
10. Click Save.
11. Click Publish (and then Publish again).

1. Click Rules.
2. Click + New.
3. Select your new Device Template from the drop down.
4. Add the following conditions:
   • message_type Equals alert
   • event_type Does not equal update
   • count Is greater than or equal to 0
   • blob_url Does not equal http://
5. Under Action, click + Webhook.
6. Enter Logic App for the Display name.
7. Insert the Device Alerts Logic App Endpoint from the outputs of the ARM template.
8. Click Save.

1. Click Devices.
2. Click + New.
3. Optionally customize the device ID or name.
4. Click Create.
5. Check the box next to the newly created device.
6. Click Migrate.
7. Select your new Device Template from the list.
8. Click Migrate.
9. Click on your new Device ID (in the device list).
10. Click Connect.
11. Copy the following values:
    • ID scope
    • Device ID
    • Primary key

    NOTE: these will be used to connect the device shortly using DPS.

The easiest option to create a device is to use the IoT Edge template available in Azure.

1. Provision a new VM using the template. Consider the following when provisioning:
   • You can use the existing resource group you have created or create a new one.
   • Ensure port 22 is open (this is the default).
   • Select a machine type with at least two vCPU cores.
   • If you haven't used SSH before/recently, password-based auth might be an easier option.
2. Once provisioning is complete, open your preferred shell.

   NOTE: ensure you have SSH installed. If you're not sure, you can always use the Cloud Shell.

3. Run ssh <your-username>@<your-machine-ip-address>.

   NOTE: your username will be the one you specified during provisioning and your machine IP address will be available from the Azure Portal (by viewing the VM you just created).

4. Once connected to the machine, run sudo nano /etc/iotedge/config.yaml.
5. Comment out the following section (i.e. add # before each line):

   provisioning:
       source: "manual"
       device_connection_string: "<ADD DEVICE CONNECTION STRING HERE>"

6. Uncomment out the following section (i.e. remove # before each line):

   provisioning:
       source: "dps"
       global_endpoint: "https://global.azure-devices-provisioning.net"
       scope_id: "{scope_id}"
       attestation:
         method: "symmetric_key"
         registration_id: "{registration_id}"
         symmetric_key: "{symmetric_key}"

7. Update {scope_id}, {registration_id}, and {symmetric_key} with the values you copied from the previous section.

   NOTE: scope_id refers to ID scope, registration_id refers to the device ID, and the symmetric_key refers to the primary key.

8. Press CTRL + X, Y, and the Enter to save and quit.
9. Run sudo systemctl restart iotedge.

Learn more here.

If you would like to onboard an actual device, you can follow instructions here:

• Linux
• Windows
• Linux containers on Windows

In this section we will setup the power app that will receive the alerts connection with the SQL Server and calling the created store procedure.

1. Login to the Power Apps site.
2. Click the Settings icon on the right corner and click the Admin center option.
3. In the new window click the Environments option in the left menu.
4. Click + New in the upper right hand corner, make sure to select SQL Database
5. Select the environment that you will be using and click the Settings option in the top menu.
6. Click the Product option to show more options.
7. Click the Features option from the list.
8. Find the Power Apps component framework for canvas apps section.
9. Make sure that the Allow publishing of canvas apps with code components option is set to On.
10. Click the Save button after make any change.
11. Close the window and go back to the main site.

1. Click the Data option in the left menu.
2. Click the Connections sub option from the list.
3. Click the + New connection button.
4. Click the SQL Server option.
5. Select the SQL Server Authentication option for the Authentication Type.
6. Enter the SQL Server Name and Database name from the outputs of the template deployment.
7. Click the Create button and wait for the connection to be created..

1. Click the Flows option in the left menu.
2. Click the + New option.
3. Select the Create from template option.
4. Select the PowerApps button option.
5. Select the PowerApps connector.
6. Click the New step button.
7. Search for the SQL Server option and find the Execute store procedure (V2) option.
8. Select the following options from the dropdown lists:
   • Server name
   • Database name
   • Procedure name
9. Add a new step and search for the Response action.
10. Select the OutputParameters from the stored procedure as the body.
11. Click the Show advanced options button.
12. Set the Response Body JSON Schema to:

    {
      "type": "object",
      "properties": {
         "NewAlertsCount": {
             "type": "number"
         },
         "BlobUrl": {
             "type": "string"
         }
      },
      "required": [
         "NewAlertsCount",
         "BlobUrl"
      ]
    }

13. Click the Save button.
14. You can test your flow to make sure that is working as expected.

1. Click the Apps option in the menu.
2. Click the Import canvas app button on the top.
3. Click the Upload button and select the .zip file under the deployment/powerapp/ folder.
4. Click the Configure button next to the SQL Server Connection resource.
5. Select the created connection and click the Save button.
6. Select the Configure button next to the Flow resource.
7. Select the created flow and click the Save button.
8. Click the Import button and wait for the app to be imported.
9. Open the App and allow the use of the SQL Server connection.

In this section we will describe some steps that are not required for the demo but allow for further customization if required.

Follow the next steps to update or remove the rule that allow the connection from all IP addresses.

1. In the Azure portal select the Resource Group you created earlier.
2. Find the SQL server resource and click it to see the detail.
3. Click the Firewalls and virtual networks option in the menu on the left under the Security section.
4. Check the AllowAllIps rule.
5. If you want to remove click the ... button next to the rule and click the Delete button.
6. If you want to update the rule, click and update the values of the Start IP and End IP columns of the rule with the values you want.

   NOTE: For a single IP set both values with the required IP.

7. Click the Save button.

In this section we will use the Custom Vision Project Generator App to create a Custom Vision project with all images and a trained iteration. This model is already provided in the solution, however, this will give you the flexibility to add more images if required.

NOTE: You will require Visual Studio 2017 or later for this setup.

1. Open Visual Studio.
2. From the top menu click the File > Open > Project/Solution.
3. Open src\CustomVisionLoaderApp\CustomVisionLoaderApp.sln.
4. Open the App.Config file and set the following settings values from ARM deployment output:
   • Key: Cognitive Services Account Key
   • Endpoint: Cognitive Services Account Endpoint
   • ResourceId: Cognitive Services Account Resource Id

1. In Visual Studio click the Play button and wait for the console app to start.
2. The app will go over all the required steps for the project generation using the information from the config file:
   • Load model from local file.
   • Create the project using the name in the config file.
   • Create the tags from the model.
   • Upload all the images one by one setting the tags.
   • Train the model.
   • Publish the iteration for the model to be ready for use.

   NOTE: this process may take several minutes depending on your connection.

3. Following output should be display if all the setup is correct:

   ***********************************************************
   *             Custom Vision Project Setup                 *
   *                                                         *
   ***********************************************************
   *** Loading local custom vision model from: C:\...\src\CustomVisionLoaderApp\CustomVisionLoaderApp\Resources\cv_data_model.json ***
   Tags to create: 2
   Image to tag: 119
   ** Starting Setup of project **
       Creating new object detection project: <project name>
       Creating tags
       Reading and Uploading images
       Uploading image 1 of 94
       ...
       Uploading image 93 of 94
       Uploading image 94 of 94
       Images upload is done.
       Training
       Done training!
   

After the project generation is finished we need to export the model from the portal.

1. Go to the Custom Vision portal and log in if you are not already.
2. Click the newly created project from the projects list.
3. Click the Performance option from the menu.
4. Click the Iteration 1 if it's not selected (should be by default).
5. Click the Export button for the iteration.
6. Click the TensorFlow icon button in the list.
7. Click the Export button.
8. Click the Download button.
9. Save the .zip file and extract the contents.
10. Copy and paste label.txt and model.pb to the device/modules/ImageClassifierService/app folder.

NOTE: If you have already deployed the provided model, you will need to increase the version value in module.json and create a new deployment to the device. These steps are outlined in the Container Registry Setup section.