Hailo Support

Compile ONNX models for Hailo chips and upload the result to the AI Manager Cloud.

Support Level	AI Accelerator	CPU Architecture	Operating System	API/driver version
Hailo-8	Experimental	x86_64, aarch64	Ubuntu 20+	4.18.0, 4.19.0, 4.20.0
Hailo-8L	Experimental	x86_64, aarch64	Ubuntu 20+	4.18.0, 4.19.0, 4.20.0

Deploying to Hailo chips requires compiling ONNX models to Hailo-ONNX format. Because this compilation process is difficult to automate in the cloud, the model is compiled locally, then the compiled Hailo-ONNX file is uploaded to the AI Manager Cloud.

Compiling an ONNX Model: Requirements

• Python 3.8

• Python environment

• Hailo Dataflow compiler and HailoRT Python API installed in that environment

• A set of calibration images (similar to images used to train the model)

Compiling an ONNX Model: Example

This example covers the compilation steps for a Yolov4-tiny model trained on the COCO dataset. Most instructions apply to other ONNX models as well, though some steps require adjustments depending on the model.

object-detection-640x640.onnx

23MB

Open

A Yolov4-tiny model that's conforming to Nx's model requirements

To compile the model, run the Python script below after changing the ONNX path.

The code performs the following tasks:

1. it transpiles the ONNX model to another format optimized for Hailo,

2. it quantizes and optimizes the model using the supplied set of calibration images,

3. it compiles the model to a HEF and embeds it inside an ONNX file as an operator, while keeping the pre-processing and post-processing intact. This step returns an ONNX file that can have different input and output names and shapes.

4. Finally, a new metadata field, named chip is injected in the ONNX to save which Hailo chip the model was optimized for. This is needed by the AI Manager Cloud to determine the target chip of the model.

The value of chip can be either hailo for Hailo-8 chips or hailo-8l for Hailo-8L chips.

After completing the steps above, a new ONNX file is generated. The latter needs to have its IO metadata (names & shapes) adjusted. The Python script below is used for that purpose, it creates a new ONNX model with the adjusted inputs and outputs. The idea of the script is to make sure the generated ONNX is conforming to the ONNX requirements for Nx.

update_model_io.py

3KB

Open

Python script to adjust the ONNX input & output metadata.

To adapt these two scripts for any other ONNX model, make sure to check out the TODO comments and adjust them accordingly.

Deploying on a Machine with Hailo-8 or Hailo-8L Chips

1. First, verify that a compatible HailoRT driver is installed. Check this table to confirm the driver version is supported. For general Hailo driver installation, see the Hailo driver installation guide. Registration for the Hailo Dev Zone is required. For the Raspberry Pi AI HAT+ see here for install instructions. For the Raspberry Pi AI Kit see here.

2. Next, install the AI plugin.

3. Once complete, the Hailo runtime is selectable when enabling the Nx plugin, as shown below:

Hailo runtime available in the plugin - select it to enable Hailo hardware acceleration

1. After the installation is finished, the plugin interface will look something like this:

Plugin interface after installation - the Hailo runtime is active and ready to assign a model

1. To manually verify that the Hailo runtime is downloaded and set up, feel free to check out the content of the bin folder of the AI Manager and make sure it contains these files: - libhailort.so.4.xx.0 (xx is the minor version of the library) - libonnxruntime_providers_hailo.so - libonnxruntime_providers_shared.so - libRuntimeLibrary.so

user@system:~$ ls /opt/networkoptix-metavms/mediaserver/var/nx_ai_manager/nxai_manager/bin
installed_runtime.txt libhailort.so.4.17.0 libonnxruntime_providers_hailo.so libonnxruntime_providers_shared.so libRuntimeLibrary.so sclbld sclblmod

1. Finally, to deploy a model that can be accelerated on the Hailo chip, make sure that it has an application/x-hailo8 (Hailo-8) or application/x-hailo8l (Hailo-8L) artifact in the AI Manager Cloud:

The model must include a Hailo artifact: look for application/x-hailo8 or application/x-hailo8l in the model detail page.

If that is not the case, manually compile the ONNX model and upload it to the cloud as illustrated in the example above.

Limitations: Number of Parallel Models

The AI Manager can run multiple AI models simultaneously, but each Hailo chip (Hailo-8 or Hailo-8L) runs only one model at a time. The number of concurrent models on a machine is limited by the number of installed Hailo chips.

Monitoring

How to Enable Hailo Monitoring with hailortcli monitor

Monitoring Hailo usage with the hailortcli monitor command requires setting a specific environment variable. Follow these steps:

1. Edit the Mediaserver Service Configuration:

   Add the following line to the /etc/systemd/system/networkoptix-metavms-mediaserver.service file to set the necessary environment variable:

   Environment="HAILO_MONITOR=1"

   The updated configuration file should look like this:

   [Unit]
   Description=Network Optix Media Server
   After=network.target local-fs.target remote-fs.target
   Requires=networkoptix-metavms-root-tool.service
   
   [Service]
   PermissionsStartOnly=true
   ExecStartPre=/opt/networkoptix-metavms/mediaserver/lib/scripts/systemd_mediaserver_pre_start.sh
   ExecStart=/opt/networkoptix-metavms/mediaserver/lib/scripts/systemd_mediaserver_start.sh
   User=networkoptix-metavms
   Group=networkoptix-metavms
   Restart=always
   TimeoutStopSec=120
   KillMode=process
   TasksMax=8191
   LimitCORE=infinity
   Environment="HAILO_MONITOR=1"
   
   [Install]
   WantedBy=multi-user.target

2. Restart the NX Mediaserver:

   After updating the configuration file, restart the Network Optix Mediaserver on Linux or Windows for the changes to take effect, using one of the following commands:

   sudo systemctl restart networkoptix-metavms-mediaserver.service

   or

   sudo service networkoptix-metavms-mediaserver restart

3. Run hailortcli HAILO_MONITOR=1 hailortcli monitor

PCIe Descriptor Page Size Error

If you encounter the following error (actual page size may vary), it indicates that your host does not support the specified PCIe descriptor page size:

[HailoRT] [error] CHECK_AS_EXPECTED failed - max_desc_page_size given 16384 is bigger than hw max desc page size 4096

This issue is common on ARM64 devices like the Raspberry Pi AI Kit. To resolve it, add the following configuration to /etc/modprobe.d/hailo_pci.conf. If the file does not exist, create it and set the max_desc_page_size to the value mentioned in the error (e.g., 4096):

options hailo_pci force_desc_page_size=4096

Add this configuration by running the following command:

echo 'options hailo_pci force_desc_page_size=4096' | sudo tee -a /etc/modprobe.d/hailo_pci.conf

Reboot the machine for the changes to take effect, or reload the driver without rebooting by running these commands:

sudo modprobe -r hailo_pci
sudo modprobe hailo_pci

Experimental .ini Setting

Explicitly setting multiple AI runtime engines is controlled by an .ini file. This .ini file does not exist by default and must be created by the user. Create an empty .ini file as described in the Linux or Windows advanced settings page.

Then restart the Mediaserver on Linux or Windows:

sudo service networkoptix-metavms-mediaserver restart

Once the Mediaserver is restarted, the .ini file should be filled with defaults. Each setting should have a description in the .ini file.

Set multiple runtimes through the runtimesPerModel setting:

# The AI Manager will spawn this many runtimes per model. Default: 1
runtimesPerModel=2