Application Configuration¶
alwaysAI makes it easy to customize your application. You can change the engine, the accelerator, the models your app uses, add external Python packages and other dependencies. You can also package your application as a container, which can be run directly with Docker.
Change the Engine and Accelerator¶
An Engine
and Accelerator
must be specified for the core
CV services. The engine is the software backend running the model, and the
accelerator is the target hardware the engine is running on. Specify
the engine and accelerator in the load()
function. If only the engine is
provided, the accelerator listed as default for that engine will be used.
You can refer to the alwaysAI Model Catalog and the
Supported Devices
page to see if your model or device is supported by an engine and accelerator.
OpenCV’s DNN Engine¶
OpenCV’s DNN engine (DNN
) will run on all
supported devices and supports most models, so it’s a great starting point.
The default accelerator is GPU
,
which attempts to run the model on the GPU and falls back to the CPU if not.
If desired, you can manually select the
CPU
, which runs the model on the CPU.
Set the engine
parameter of your core CV service as follows to use
OpenCV’s DNN engine:
cv_service.load(engine=edgeiq.Engine.DNN)
or:
cv_service.load(
engine=edgeiq.Engine.DNN, accelerator=edgeiq.Accelerator.CPU)
OpenCV’s CUDA Engine¶
OpenCV’s CUDA engine (DNN_CUDA
) will perform
inferencing on a CUDA-supported GPU. The default accelerator is
NVIDIA
, and for some models an
additional performance boost comes with using the
NVIDIA_FP16
accelerator.
Set the engine
parameter of your core CV service as follows to use
OpenCV’s CUDA engine:
cv_service.load(engine=edgeiq.Engine.DNN_CUDA)
NVIDIA’s TensorRT Engine¶
NVIDIA’s TensorRT engine (TENSOR_RT
) is a
high-performance model optimizer and inference engine for CUDA-supported
GPU’s. The default accelerator is NVIDIA
.
Additionally, the DLA can be selected by using NVIDIA_DLA_0
and NVIDIA_DLA_1
.
Set the engine
parameter of your core CV service as follows to use
NVIDIA’s TensorRT engine:
cv_service.load(engine=edgeiq.Engine.TENSOR_RT)
Hailo’s HailoRT Engine¶
Hailo’s RT engine (HAILO_RT
) is a
high-performance model optimizer and inference engine for Hailo’s accelerators.
Set the engine
parameter of your core CV service as follows to use
Hailo’s HailoRT engine:
cv_service.load(engine=edgeiq.Engine.HAILO_RT)
HailoRT libraries should be installed in the host machine to run the application on Hailo accelerator. Installing HailoRT libraries requires an additional installation procedure. Running the command below in a linux (amd64) machine shall install the Hailo RT firmware and Hailo PCIE Drivers.
In the host machine, run the command:
$ aai hailo install-pcie-driver
To uninstall the Hailo RT Firmware and Hailo PCIE Driver, run the command:
$ aai hailo uninstall-pcie-driver
Qualcomm’s QAIC RT Engine¶
Qualcomm’s QAIC RT engine (QAIC_RT
) is a
high-performance inference engine powered by Qualcomm’s AIC100 system-on-chip. It is
specialized for Machine Learning and Deep Learning inference workloads.
Set the engine
parameter of your core CV service as follows to use
Qualcomm’s QAIC RT engine:
cv_service.load(engine=edgeiq.Engine.QAIC_RT)
OpenCV’s OpenVINO Engine for NCS2 and Myriad Processors¶
OpenCV’s OpenVINO engine (DNN_OPENVINO
)
enables inferencing on the Intel Neural Compute Stick 2 as well as other
Myriad-based devices. It supports models built with the OpenVINO model
optimizer as well as some other frameworks.
Set the engine
parameter of your core CV service as follows to use
OpenCV’s OpenVINO engine:
cv_service.load(engine=edgeiq.Engine.DNN_OPENVINO)
Change the Computer Vision Model¶
The alwaysAI model catalog provides pre-trained machine learning models that enable developers to quickly prototype a computer vision application without the need to create a custom model first. If you can’t find a model that suits your needs, you can upload a pre-trained model, or train a new model. To change the model in your application:
Update the app model dependency¶
Either navigate to the alwaysAI Model Catalog
and find the model you want to add, or select from your personal models,
and click on the model name to see details.
Copy the model ID and run it with the aai app models add
command in your
terminal. For example, to use the MobileNet SSD model, the full command
would be:
$ aai app models add alwaysai/ssd_mobilenet_v1_coco_2018_01_28
To remove your old model, run the alwaysai app models remove
command:
$ aai app models remove alwaysai/mobilenet_ssd
To see the models that your app depends on, run the alwaysai app models show
command:
$ aai app models show
Models:
alwaysai/ssd_mobilenet_v1_coco_2018_01_28@2
For more details on adding models to your application you can visit this page.
Use the model in your application¶
The next step to using the new model in your application is to pass
the model ID to the constructor of the object that will be using it.
Classification
, ObjectDetection
and
PoseEstimation
all take a model ID as input. Paste the model ID
into your app.py file as an input parameter:
obj_detect = edgeiq.ObjectDetection("alwaysai/ssd_mobilenet_v1_coco_2018_01_28")
Run the aai app install
command to make sure your models are
installed and the latest application code is available on your device.
If the model does not get installed or is not supported by the object you are using, you will get an error back telling you so.
Convert a Model to Tensor RT¶
For ONNX models, conversion to Tensor RT can be done at runtime by selecting the
Tensor RT engine (TENSOR_RT
) and
appropriate accelerator.
For models using other frameworks, for example alwaysai/yolo_v3
and
alwaysai/yolo_v4
, you can use the alwaysAI CLI to convert the model for your
particular Jetson device and JetPack version.
First, install the alwaysAI CLI directly on the device where the model will be used:
$ npm install -g alwaysai
Next, run the conversion command. Select the model from the alwaysAI model catalog or from your models:
$ aai model convert alwaysai/yolo_v3 --format tensor-rt
The full list of options is shown in the command help menu:
$ aai model convert --help
Usage: aai model convert model ID e.g. alwaysai/mobilenet_ssd <options>
Convert an alwaysAI model to a different format
Options:
--format <value> : The output format of the model conversion
Allowed values {tensor-rt}
[--output-id <str>] : Model ID of the converted model
[--version <num>] : Version of the model to convert
[--batch-size <num>] : Batch size if converting to tensor-rt
The model conversion will generate a new model package in the out/
directory. To use the model in your application, navigate to new model package
directory and run aai model publish
.
Update Application edgeIQ Version¶
The edgeIQ API will be frequently updated with new features, bug fixes, and
enhancements. When a new version of edgeIQ is released, it is recommended that
you update your applications to the latest version. The easiest way to update
your edgeIQ version to the latest is to delete the Dockerfile in the app
directory and run the app configure
command to generate a new Dockerfile
using the latest release.
$ rm Dockerfile
$ aai app configure
If you’ve customized your Dockerfile with additional commands, then the best
option is to just edit the FROM
line in the Dockerfile.
For example, change:
FROM alwaysai/edgeiq:${ALWAYSAI_HW}-2.2.0
to:
FROM alwaysai/edgeiq:${ALWAYSAI_HW}-2.2.1
Read the edgeIQ Release Notes to learn about the latest releases.
Once you’ve changed your Dockerfile, you must run aai app install
for the changes to take effect.
Handle Application Dependencies¶
Once you start building more complex alwaysAI applications, you’ll likely use dependencies that are not included in the edgeIQ Docker image. There are two types of dependencies that are supported:
Python Dependencies: These are packages that can be installed using
pip
.System Dependencies: These are packages that can be installed using
apt-get
.
Python Dependencies¶
To add a Python dependency to your app, add a requirements.txt file to your app directory and add the requirement, along with the version if necessary. For example, if your app requires the Requests Python module your requirements.txt would look like this:
Requests==2.22.0
During the app install
command, the dependencies are installed in a Python
virtual environment for your app.
System Dependencies¶
To add a system dependency to your app, add additional commands to your Dockerfile. For example, if your app depends on the VLC package, your Dockerfile would look like this:
ARG ALWAYSAI_HW="default"
FROM alwaysai/edgeiq:${ALWAYSAI_HW}-<version>
RUN apt-get update && apt-get install -y vlc
During the app install
command the Docker image is rebuilt using the
updated Dockerfile. Your app will run in a container based on the new
Docker image.
edgeIQ Runtime Environment Base Images¶
The edgeIQ Runtime Environment is available as Docker images from alwaysAI Docker Hub. The format of the images is as follows:
alwaysai/edgeiq:<architecture/device>-<version>
Images are built for the following architectures:
aarch64
amd64
Additionally, an image is built for NVIDIA Jetson devices with the prefix “jetson-jp<jetpack-version>”, e.g.:
alwaysai/edgeiq:jetson-jp5.1-2.2.1
The latest release will be tagged with “latest” as the version. However, it is recommended to use a specific version tag and upgrade when new versions come out, since the API’s are constantly being updated and improved. Using the “latest” tag may lead to surprises when a new version is pulled down unexpectedly.
Select the Architecture or Device¶
The alwaysAI CLI takes advantage of Docker build arguments to automatically pick
the right architecture or device. This is done by setting an argument before the
FROM
line of the Dockerfile which the CLI can overwrite:
ARG ALWAYSAI_HW="default"
FROM alwaysai/edgeiq:${ALWAYSAI_HW}-0.17.1
If you’d like to build the Dockerfile without using the CLI, just change
ALWAYSAI_HW
to match the architecture or name of your target device. The
available target hardware types can be found using the CLI:
$ aai app configure --help
Usage: aai app configure [<options>]
Configure this directory as an alwaysAI application
Options:
...
[--hardware <value>] : Allowed values {jetson-jp4.6, jetson-jp4.6.1, jetson-jp4.6.2, jetson-jp4.6.3, jetson-jp4.6.4, jetson-jp5.1, armv7hf, aarch64, amd64, x86-trt-22.12, x86-trt-23.02, eyecloud-amd64, eyecloud-armv7hf, hailo-amd64}
...
Create a Docker Compose File for Your App¶
Docker compose can give you finer-grained control over your Docker configuration. The alwaysAI Device Agent will use a Docker Compose file you provide, or generate a new default one if not provided. To generate a Docker Compose file for your app, run:
$ aai app generate docker-compose
This will generate two files:
Dockerfile.standalone: This is a new Docker file based on the app’s
Dockerfile
and the start command configured inalwaysai.app.json
docker-compose.yaml: This is the default Docker Compose file used by alwaysAI apps.
The Docker Compose file includes the following configurations:
The
network_mode: host
flag tells docker to map the device’s network interfaces into the container. This enables access to the internet and the Streamer from outside the container.The
privileged: true
flag is needed when working with USB devices.The
volume: /dev:/dev
flag mounts the devices directory into the container so that cameras and USB devices can be accessed.The
volume: ~/.config/alwaysai:/root/.config/alwaysai
flag mounts the credentials directory into the container so edgeIQ can authenticate.
For NVIDIA Jetson devices, the following options will be automatically added by the Device Agent:
runtime: nvidia
ensures the NVIDIA drivers are loaded.ipc: host
is required when usingJetsonVideoStream
.volume: /tmp/argus_socket:/tmp/argus_socket
is required when usingJetsonVideoStream
.
To learn more about these options, visit the Docker Run reference page.
If you’d like to directly run your app, you can manage it with docker-compose
commands. A couple useful commands are:
docker-compose build
will build the Docker image(s) for the appdocker-compose up -d
will bring up your app in detached mode (running in the background)docker-compose logs
will stream the incoming logs from your appdocker-compose down
will bring down your app
Learn more about these commands at the Docker Compose reference page.
Related Tutorials