> For the complete documentation index, see [llms.txt](https://nx.docs.scailable.net/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nx.docs.scailable.net/ai-manager-v6.1.4/ai-accelerators-support/advanced/hailo-support.md).

# Hailo Support

| Support Level | AI Accelerator | CPU Architecture | Operating System | API/driver version     |
| ------------- | -------------- | ---------------- | ---------------- | ---------------------- |
| Hailo-8       | Experimental   | x86\_64, aarch64 | Ubuntu 20+       | 4.18.0, 4.19.0, 4.20.0 |
| Hailo-8L      | Experimental   | x86\_64, aarch64 | Ubuntu 20+       | 4.18.0, 4.19.0, 4.20.0 |

Deploying to Hailo chips requires compiling ONNX models to Hailo-ONNX format. Because this compilation process is difficult to automate in the cloud, the model is compiled locally, then the compiled Hailo-ONNX file is uploaded to the AI Manager Cloud.

## Compiling an ONNX Model: Requirements

* Python 3.8
* Python environment
* **Hailo Dataflow compiler** and **HailoRT Python API** installed in that environment
* A set of calibration images (similar to images used to train the model)

## Compiling an ONNX Model: Example

This example covers the compilation steps for a Yolov4-tiny model trained on the COCO dataset. Most instructions apply to other ONNX models as well, though some steps require adjustments depending on the model.

{% file src="/files/YivPgrS8LGcNpNh2ErvJ" %}
A Yolov4-tiny model that's conforming to Nx's model requirements
{% endfile %}

To compile the model, run the Python script below after changing the ONNX path.

The code performs the following tasks:

1. it transpiles the ONNX model to another format optimized for Hailo,
2. it quantizes and optimizes the model using the supplied set of calibration images,
3. it compiles the model to a HEF and embeds it inside an ONNX file as an operator, while keeping the pre-processing and post-processing intact.\
   **This step returns an ONNX file that can have different input and output names and shapes.**
4. Finally, a new metadata field, named `chip` is injected in the ONNX to save which Hailo chip the model was optimized for. This is needed by the AI Manager Cloud to determine the target chip of the model.

{% hint style="info" %}
The value of chip can be either `hailo` for Hailo-8 chips or `hailo-8l` for Hailo-8L chips.
{% endhint %}

After completing the steps above, a new ONNX file is generated. The latter needs to have its IO metadata (names & shapes) adjusted. The Python script below is used for that purpose, it creates a new ONNX model with the adjusted inputs and outputs.\
The idea of the script is to make sure the generated ONNX is conforming to the ONNX [requirements](/ai-manager-v6.1.4/ai-models-support/custom-models/onnx-requirements.md) for Nx.

{% file src="/files/8tIKCIXyWQHtNxY0AJ3S" %}
Python script to adjust the ONNX input & output metadata.
{% endfile %}

{% hint style="info" %}
To adapt these two scripts for any other ONNX model, make sure to check out the TODO comments and adjust them accordingly.
{% endhint %}

## Deploying on a Machine with Hailo-8 or Hailo-8L Chips

1. First, verify that a compatible HailoRT driver is installed. Check this [table](/ai-manager-v6.1.4/ai-accelerators-support/supported-ai-accelerators.md) to confirm the driver version is supported.\
   For general Hailo driver installation, see the [Hailo driver installation guide](https://hailo.ai/developer-zone/software-downloads/). Registration for the Hailo Dev Zone is required.\
   For the Raspberry Pi AI HAT+ [see here](https://www.raspberrypi.com/documentation/accessories/ai-hat-plus.html#ai-hat-plus) for install instructions. For the Raspberry Pi AI Kit [see here](https://www.raspberrypi.com/documentation/accessories/ai-kit.html#ai-kit).
2. Next, install the AI plugin.
3. Once complete, the Hailo runtime is selectable when enabling the Nx plugin, as shown below:

<figure><img src="/files/0aUCSpJSaZdh2C98Acqu" alt="AI Manager plugin interface showing the Hailo runtime selected in the runtime dropdown"><figcaption><p>Hailo runtime available in the plugin - select it to enable Hailo hardware acceleration</p></figcaption></figure>

4. After the installation is finished, the plugin interface will look something like this:

<figure><img src="/files/sySOK1z7NsPK0wmjTD1T" alt="AI Manager plugin interface after Hailo runtime installation showing active pipeline configuration"><figcaption><p>Plugin interface after installation - the Hailo runtime is active and ready to assign a model</p></figcaption></figure>

5. To manually verify that the Hailo runtime is downloaded and set up, feel free to check out the content of the `bin` folder of the AI Manager and make sure it contains these files:\
   \- libhailort.so.4.xx.0 (xx is the minor version of the library)\
   \- libonnxruntime\_providers\_hailo.so\
   \- libonnxruntime\_providers\_shared.so\
   \- libRuntimeLibrary.so

```sh
user@system:~$ ls /opt/networkoptix-metavms/mediaserver/var/nx_ai_manager/nxai_manager/bin
installed_runtime.txt libhailort.so.4.17.0 libonnxruntime_providers_hailo.so libonnxruntime_providers_shared.so libRuntimeLibrary.so sclbld sclblmod
```

6. Finally, to deploy a model that can be accelerated on the Hailo chip, make sure that it has an `application/x-hailo8` (Hailo-8) or `application/x-hailo8l` (Hailo-8L) artifact in the AI Manager Cloud:

<figure><img src="/files/qZukOHfbFRQAdiuhEn6Z" alt="Model detail page showing available artifact download types including Hailo 8 ONNX and
     Hailo 8L ONNX entries"><figcaption><p>The model must include a Hailo artifact: look for <code>application/x-hailo8</code> or <code>application/x-hailo8l</code> in the model detail page.</p></figcaption></figure>

If that is not the case, manually compile the ONNX model and upload it to the cloud as illustrated in the example above.

## Limitations: Number of Parallel Models

The AI Manager can run multiple AI models simultaneously, but each Hailo chip (Hailo-8 or Hailo-8L) runs only one model at a time. The number of concurrent models on a machine is limited by the number of installed Hailo chips.

## Monitoring

**How to Enable Hailo Monitoring with `hailortcli monitor`**

Monitoring Hailo usage with the `hailortcli monitor` command requires setting a specific environment variable. Follow these steps:

1. **Edit the Mediaserver Service Configuration:**<br>

   Add the following line to the `/etc/systemd/system/networkoptix-metavms-mediaserver.service` file to set the necessary environment variable:<br>

   ```plaintext
   Environment="HAILO_MONITOR=1"
   ```

   \
   The updated configuration file should look like this:<br>

   ```plaintext
   [Unit]
   Description=Network Optix Media Server
   After=network.target local-fs.target remote-fs.target
   Requires=networkoptix-metavms-root-tool.service

   [Service]
   PermissionsStartOnly=true
   ExecStartPre=/opt/networkoptix-metavms/mediaserver/lib/scripts/systemd_mediaserver_pre_start.sh
   ExecStart=/opt/networkoptix-metavms/mediaserver/lib/scripts/systemd_mediaserver_start.sh
   User=networkoptix-metavms
   Group=networkoptix-metavms
   Restart=always
   TimeoutStopSec=120
   KillMode=process
   TasksMax=8191
   LimitCORE=infinity
   Environment="HAILO_MONITOR=1"

   [Install]
   WantedBy=multi-user.target
   ```
2. **Restart the NX Mediaserver:**<br>

   After updating the configuration file, restart the Network Optix Mediaserver on [Linux](/ai-manager-v6.1.4/support-and-troubleshooting/troubleshooting/controlling-the-server-and-the-plugin.md#os-linux) or [Windows](/ai-manager-v6.1.4/support-and-troubleshooting/troubleshooting/controlling-the-server-and-the-plugin.md#os-windows) for the changes to take effect, using one of the following commands:<br>

   ```bash
   sudo systemctl restart networkoptix-metavms-mediaserver.service
   ```

   \
   or<br>

   ```bash
   sudo service networkoptix-metavms-mediaserver restart
   ```
3. **Run hailortcli**\
   \
   `HAILO_MONITOR=1 hailortcli monitor`

<figure><img src="/files/gnlvlwLJBWAWHUbHrQRt" alt="Terminal output from hailortcli monitor showing runtime statistics"><figcaption></figcaption></figure>

## PCIe Descriptor Page Size Error

If you encounter the following error (actual page size may vary), it indicates that your host does not support the specified PCIe descriptor page size:

```
[HailoRT] [error] CHECK_AS_EXPECTED failed - max_desc_page_size given 16384 is bigger than hw max desc page size 4096
```

This issue is common on ARM64 devices like the Raspberry Pi AI Kit. To resolve it, add the following configuration to `/etc/modprobe.d/hailo_pci.conf`. If the file does not exist, create it and set the `max_desc_page_size` to the value mentioned in the error (e.g., 4096):

```bash
options hailo_pci force_desc_page_size=4096
```

Add this configuration by running the following command:

```bash
echo 'options hailo_pci force_desc_page_size=4096' | sudo tee -a /etc/modprobe.d/hailo_pci.conf
```

Reboot the machine for the changes to take effect, or reload the driver without rebooting by running these commands:

```bash
sudo modprobe -r hailo_pci
sudo modprobe hailo_pci
```

## Experimental `.ini` Setting <a href="#enable-.ini-settings" id="enable-.ini-settings"></a>

Explicitly setting multiple AI runtime engines is controlled by an `.ini` file. This `.ini` file does not exist by default and must be created by the user. Create an empty `.ini` file as described in the [Linux](/ai-manager-v6.1.4/ai-manager-plugin/advanced-configuration/enable-.ini-settings.md#os-linux) or [Windows](/ai-manager-v6.1.4/ai-manager-plugin/advanced-configuration/enable-.ini-settings.md#os-windows) advanced settings page.

Then restart the Mediaserver on [Linux](/ai-manager-v6.1.4/support-and-troubleshooting/troubleshooting/controlling-the-server-and-the-plugin.md#os-linux) or [Windows](/ai-manager-v6.1.4/support-and-troubleshooting/troubleshooting/controlling-the-server-and-the-plugin.md#os-windows):

```bash
sudo service networkoptix-metavms-mediaserver restart
```

Once the Mediaserver is restarted, the `.ini` file should be filled with defaults. Each setting should have a description in the `.ini` file.

Set multiple runtimes through the `runtimesPerModel` setting:

```ini
# The AI Manager will spawn this many runtimes per model. Default: 1
runtimesPerModel=2
```