Application Configuration

alwaysAI makes it easy to customize your application. You can change the engine, the accelerator, the models your app uses, add external Python packages and other dependencies. You can also package your application as a container, which can be run directly with Docker.

Change the Engine and Accelerator

An Engine and Accelerator must be specified for the core CV services. The engine is the software backend running the model, and the accelerator is the target hardware the engine is running on. Specify the engine and accelerator in the load() function. If only the engine is provided, the accelerator listed as default for that engine will be used. You can refer to the alwaysAI Model Catalog and the Supported Devices page to see if your model or device is supported by an engine and accelerator.

OpenCV’s DNN Engine

OpenCV’s DNN engine (DNN) will run on all supported devices and supports most models, so it’s a great starting point. The default accelerator is GPU, which attempts to run the model on the GPU and falls back to the CPU if not. If desired, you can manually select the CPU, which runs the model on the CPU.

Set the engine parameter of your core CV service as follows to use OpenCV’s DNN engine:

cv_service.load(engine=edgeiq.Engine.DNN)

or:

cv_service.load(
    engine=edgeiq.Engine.DNN, accelerator=edgeiq.Accelerator.CPU)

OpenCV’s CUDA Engine

OpenCV’s CUDA engine (DNN_CUDA) will perform inferencing on a CUDA-supported GPU. The default accelerator is NVIDIA, and for some models an additional performance boost comes with using the NVIDIA_FP16 accelerator.

Set the engine parameter of your core CV service as follows to use OpenCV’s CUDA engine:

cv_service.load(engine=edgeiq.Engine.DNN_CUDA)

NVIDIA’s TensorRT Engine

NVIDIA’s TensorRT engine (TENSOR_RT) is a high-performance model optimizer and inference engine for CUDA-supported GPU’s. The default accelerator is NVIDIA. Additionally, the DLA can be selected by using NVIDIA_DLA_0 and NVIDIA_DLA_1.

Set the engine parameter of your core CV service as follows to use NVIDIA’s TensorRT engine:

cv_service.load(engine=edgeiq.Engine.TENSOR_RT)

Hailo’s HailoRT Engine

Hailo’s RT engine (HAILO_RT) is a high-performance model optimizer and inference engine for Hailo’s accelerators.

Set the engine parameter of your core CV service as follows to use Hailo’s HailoRT engine:

cv_service.load(engine=edgeiq.Engine.HAILO_RT)

HailoRT libraries should be installed in the host machine to run the application on Hailo accelerator. Installing HailoRT libraries requires an additional installation procedure. Running the command below in a linux (amd64) machine shall install the Hailo RT firmware and Hailo PCIE Drivers.

In the host machine, run the command:

$ aai hailo install-pcie-driver

To uninstall the Hailo RT Firmware and Hailo PCIE Driver, run the command:

$ aai hailo uninstall-pcie-driver

Qualcomm’s QAIC RT Engine

Qualcomm’s QAIC RT engine (QAIC_RT) is a high-performance inference engine powered by Qualcomm’s AIC100 system-on-chip. It is specialized for Machine Learning and Deep Learning inference workloads.

Set the engine parameter of your core CV service as follows to use Qualcomm’s QAIC RT engine:

cv_service.load(engine=edgeiq.Engine.QAIC_RT)

OpenCV’s OpenVINO Engine for NCS2 and Myriad Processors

OpenCV’s OpenVINO engine (DNN_OPENVINO) enables inferencing on the Intel Neural Compute Stick 2 as well as other Myriad-based devices. It supports models built with the OpenVINO model optimizer as well as some other frameworks.

Set the engine parameter of your core CV service as follows to use OpenCV’s OpenVINO engine:

cv_service.load(engine=edgeiq.Engine.DNN_OPENVINO)

Change the Computer Vision Model

The alwaysAI model catalog provides pre-trained machine learning models that enable developers to quickly prototype a computer vision application without the need to create a custom model first. If you can’t find a model that suits your needs, you can upload a pre-trained model, or train a new model. To change the model in your application:

Update the app model dependency

Either navigate to the alwaysAI Model Catalog and find the model you want to add, or select from your personal models, and click on the model name to see details. Copy the model ID and run it with the aai app models add command in your terminal. For example, to use the MobileNet SSD model, the full command would be:

$ aai app models add alwaysai/ssd_mobilenet_v1_coco_2018_01_28

To remove your old model, run the alwaysai app models remove command:

$ aai app models remove alwaysai/mobilenet_ssd

To see the models that your app depends on, run the alwaysai app models show command:

$ aai app models show
Models:
  alwaysai/ssd_mobilenet_v1_coco_2018_01_28@2

For more details on adding models to your application you can visit this page.

Use the model in your application

The next step to using the new model in your application is to pass the model ID to the constructor of the object that will be using it. Classification, ObjectDetection and PoseEstimation all take a model ID as input. Paste the model ID into your app.py file as an input parameter:

obj_detect = edgeiq.ObjectDetection("alwaysai/ssd_mobilenet_v1_coco_2018_01_28")

Run the aai app install command to make sure your models are installed and the latest application code is available on your device.

If the model does not get installed or is not supported by the object you are using, you will get an error back telling you so.

Convert a Model to Tensor RT

For ONNX models, conversion to Tensor RT can be done at runtime by selecting the Tensor RT engine (TENSOR_RT) and appropriate accelerator.

For models using other frameworks, for example alwaysai/yolo_v3 and alwaysai/yolo_v4, you can use the alwaysAI CLI to convert the model for your particular Jetson device and JetPack version.

First, install the alwaysAI CLI directly on the device where the model will be used:

$ npm install -g alwaysai

Next, run the conversion command. Select the model from the alwaysAI model catalog or from your models:

$ aai model convert alwaysai/yolo_v3 --format tensor-rt

The full list of options is shown in the command help menu:

$ aai model convert --help
Usage: aai model convert model ID e.g. alwaysai/mobilenet_ssd <options>

  Convert an alwaysAI model to a different format

Options:

  --format <value>     : The output format of the model conversion
                          Allowed values {tensor-rt}
  [--output-id <str>]  : Model ID of the converted model
  [--version <num>]    : Version of the model to convert
  [--batch-size <num>] : Batch size if converting to tensor-rt

The model conversion will generate a new model package in the out/ directory. To use the model in your application, navigate to new model package directory and run aai model publish.

Update Application edgeIQ Version

The edgeIQ API will be frequently updated with new features, bug fixes, and enhancements. When a new version of edgeIQ is released, it is recommended that you update your applications to the latest version. The easiest way to update your edgeIQ version to the latest is to delete the Dockerfile in the app directory and run the app configure command to generate a new Dockerfile using the latest release.

$ rm Dockerfile
$ aai app configure

If you’ve customized your Dockerfile with additional commands, then the best option is to just edit the FROM line in the Dockerfile.

For example, change:

FROM alwaysai/edgeiq:${ALWAYSAI_HW}-2.2.0

to:

FROM alwaysai/edgeiq:${ALWAYSAI_HW}-2.2.1

Read the edgeIQ Release Notes to learn about the latest releases.

Once you’ve changed your Dockerfile, you must run aai app install for the changes to take effect.

Handle Application Dependencies

Once you start building more complex alwaysAI applications, you’ll likely use dependencies that are not included in the edgeIQ Docker image. There are two types of dependencies that are supported:

  1. Python Dependencies: These are packages that can be installed using pip.

  2. System Dependencies: These are packages that can be installed using apt-get.

Python Dependencies

To add a Python dependency to your app, add a requirements.txt file to your app directory and add the requirement, along with the version if necessary. For example, if your app requires the Requests Python module your requirements.txt would look like this:

Requests==2.22.0

During the app install command, the dependencies are installed in a Python virtual environment for your app.

System Dependencies

To add a system dependency to your app, add additional commands to your Dockerfile. For example, if your app depends on the VLC package, your Dockerfile would look like this:

ARG ALWAYSAI_HW="default"
FROM alwaysai/edgeiq:${ALWAYSAI_HW}-<version>
RUN apt-get update && apt-get install -y vlc

During the app install command the Docker image is rebuilt using the updated Dockerfile. Your app will run in a container based on the new Docker image.

edgeIQ Runtime Environment Base Images

The edgeIQ Runtime Environment is available as Docker images from alwaysAI Docker Hub. The format of the images is as follows:

alwaysai/edgeiq:<architecture/device>-<version>

Images are built for the following architectures:

  • aarch64

  • amd64

Additionally, an image is built for NVIDIA Jetson devices with the prefix “jetson-jp<jetpack-version>”, e.g.:

alwaysai/edgeiq:jetson-jp5.1-2.2.1

The latest release will be tagged with “latest” as the version. However, it is recommended to use a specific version tag and upgrade when new versions come out, since the API’s are constantly being updated and improved. Using the “latest” tag may lead to surprises when a new version is pulled down unexpectedly.

Select the Architecture or Device

The alwaysAI CLI takes advantage of Docker build arguments to automatically pick the right architecture or device. This is done by setting an argument before the FROM line of the Dockerfile which the CLI can overwrite:

ARG ALWAYSAI_HW="default"
FROM alwaysai/edgeiq:${ALWAYSAI_HW}-0.17.1

If you’d like to build the Dockerfile without using the CLI, just change ALWAYSAI_HW to match the architecture or name of your target device. The available target hardware types can be found using the CLI:

$ aai app configure --help
Usage: aai app configure [<options>]

  Configure this directory as an alwaysAI application

Options:

  ...
  [--hardware <value>] : Allowed values {jetson-jp4.6, jetson-jp4.6.1, jetson-jp4.6.2, jetson-jp4.6.3, jetson-jp4.6.4, jetson-jp5.1, armv7hf, aarch64, amd64, x86-trt-22.12, x86-trt-23.02, eyecloud-amd64, eyecloud-armv7hf, hailo-amd64}
  ...

Create a Docker Compose File for Your App

Docker compose can give you finer-grained control over your Docker configuration. The alwaysAI Device Agent will use a Docker Compose file you provide, or generate a new default one if not provided. To generate a Docker Compose file for your app, run:

$ aai app generate docker-compose

This will generate two files:

  • Dockerfile.standalone: This is a new Docker file based on the app’s Dockerfile and the start command configured in alwaysai.app.json

  • docker-compose.yaml: This is the default Docker Compose file used by alwaysAI apps.

The Docker Compose file includes the following configurations:

  • The network_mode: host flag tells docker to map the device’s network interfaces into the container. This enables access to the internet and the Streamer from outside the container.

  • The privileged: true flag is needed when working with USB devices.

  • The volume: /dev:/dev flag mounts the devices directory into the container so that cameras and USB devices can be accessed.

  • The volume: ~/.config/alwaysai:/root/.config/alwaysai flag mounts the credentials directory into the container so edgeIQ can authenticate.

For NVIDIA Jetson devices, the following options will be automatically added by the Device Agent:

  • runtime: nvidia ensures the NVIDIA drivers are loaded.

  • ipc: host is required when using JetsonVideoStream.

  • volume: /tmp/argus_socket:/tmp/argus_socket is required when using JetsonVideoStream.

To learn more about these options, visit the Docker Run reference page.

If you’d like to directly run your app, you can manage it with docker-compose commands. A couple useful commands are:

  • docker-compose build will build the Docker image(s) for the app

  • docker-compose up -d will bring up your app in detached mode (running in the background)

  • docker-compose logs will stream the incoming logs from your app

  • docker-compose down will bring down your app

Learn more about these commands at the Docker Compose reference page.

Related Tutorials