> For the complete documentation index, see [llms.txt](https://nx.docs.scailable.net/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nx.docs.scailable.net/ai-manager-v6.1.4/ai-accelerators-support/advanced/hailo-support.md).

# Hailo Support

Supported Hailo-8 and Hailo-8L hardware, how to enable the Hailo runtime, get models for it and fix common problems.

| AI Accelerator | Support Level | CPU Architecture | Operating System          | API/driver version |
| -------------- | ------------- | ---------------- | ------------------------- | ------------------ |
| Hailo-8        | Experimental  | x86\_64, aarch64 | Ubuntu 20.04+, Debian 11+ | 4.17.0 to 4.20.0   |
| Hailo-8L       | Experimental  | x86\_64, aarch64 | Ubuntu 20.04+, Debian 11+ | 4.17.0 to 4.20.0   |

The Hailo runtime runs models on Hailo-8 and Hailo-8L PCIe or M.2 modules through HailoRT. Only the parts of a model that the chip cannot run (for example some pre- and post-processing steps) run on the host CPU.

To use it, install Hailo's driver, select the **Hailo** runtime in the AI Manager plugin, then use a model that has a Hailo artifact in the AI Manager Cloud.

{% hint style="info" %}
The AI Manager downloads the Hailo runtime for the HailoRT version it finds on the machine. Both chips are Experimental. Hailo-8L is detected and gets its own model type (`application/x-hailo8l`), but we have not verified it on hardware yet.
{% endhint %}

## Quick Start

1. Install Hailo's PCIe driver and HailoRT (4.17.0 to 4.20.0, same version for both), and the system libraries: [Requirements](#requirements).
2. Check that the plugin can see the chip: [Check the Installation](#check-the-installation).
3. Select the Hailo runtime in the AI Manager plugin: [Select the Hailo Runtime](#select-the-hailo-runtime).
4. Use a model with a Hailo artifact, for example by enabling the **Hailo** conversion when you upload it: [Convert a Model](#convert-a-model).
5. Optional: [configure the runtime](#configure-the-runtime).

## Requirements

| Item             | Requirement                                                                                                                                                                                         |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Card             | A **Hailo-8** or **Hailo-8L** module (PCIe or M.2)                                                                                                                                                  |
| Host             | x86\_64 or aarch64, running Ubuntu 20.04 or higher, or Debian 11 or higher                                                                                                                          |
| Hailo software   | Hailo's PCIe driver and HailoRT (`hailort-pcie-driver` and `hailort`, which provides `hailortcli`), version **4.17.0, 4.18.0, 4.19.0 or 4.20.0**. The driver and HailoRT must have the same version |
| System libraries | `libatomic1` and `libgomp1`                                                                                                                                                                         |

Install the system libraries with:

```bash
sudo apt install libatomic1 libgomp1
```

For the driver installation, see the [Hailo Developer Zone](https://hailo.ai/developer-zone/software-downloads/) (registration required). For the Raspberry Pi AI HAT+, [see here](https://www.raspberrypi.com/documentation/accessories/ai-hat-plus.html).

## Check the Installation

The plugin looks for devices in `/sys/class/hailo_chardev`, reads the HailoRT version with `hailortcli -v` and the chip type (Hailo-8 or Hailo-8L) with `hailortcli fw-control identify`. Run the same commands to see what the plugin sees:

```bash
hailortcli -v
hailortcli scan
hailortcli fw-control identify
ls /sys/class/hailo_chardev
```

<details>

<summary>Example output on a machine with four Hailo-8 modules</summary>

Shortened; `identify` prints one block per device, and `scan` does not sort the devices:

```
$ hailortcli -v
HailoRT-CLI version 4.20.0
$ hailortcli scan
Hailo Devices:
[-] Device: 0000:05:00.0
[-] Device: 0000:08:00.0
[-] Device: 0000:06:00.0
[-] Device: 0000:07:00.0
$ hailortcli fw-control identify
Executing on device: 0000:05:00.0
Identifying board
Control Protocol Version: 2
Firmware Version: 4.20.0 (release,app,extended context switch buffer)
Logger Version: 0
Board Name: Hailo-8
Device Architecture: HAILO8
...

Executing on device: 0000:08:00.0
...
$ ls /sys/class/hailo_chardev
hailo0  hailo1  hailo2  hailo3
```

</details>

* **Chip type:** `Device Architecture` is `HAILO8` or `HAILO8L`. If the machine mixes Hailo-8 and Hailo-8L modules, the first `Device Architecture` of the output decides the chip type. Other Hailo chips (for example Hailo-10H) are not detected.
* **No device, or an unsupported version:** the plugin does not offer the Hailo runtime and the UI shows no message about it. The reason is only in the plugin log, for example `Hailo-8 driver version <version> is not supported.` (log path: see [Troubleshooting](#troubleshooting)).

## Select the Hailo Runtime

1. Install the AI Manager plugin.
2. Once the plugin has detected the chip, the Hailo runtime is selectable when enabling the plugin, as shown below (see also [Runtime Selection](https://nx.docs.scailable.net/ai-manager-v6.1.4/ai-manager-plugin/configure-the-plugin/runtime-selection.md)). Open any camera's settings, go to the **Integrations** tab, click **AI Manager**, and select the Hailo runtime.

   <figure><img src="https://2982163321-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKGfC0TFSmuMDjbpLErO3%2Fuploads%2Fgit-blob-b48a3c1e34a4b8e963d43a3e5586000494b58ae8%2Fclient-ai-manager-runtime-hailo-tooltip.png?alt=media" alt="AI Manager panel with Generic CPU and Hailo runtime buttons, and a tooltip on the Hailo button reading Install this acceleration library on your server"><figcaption><p>Hailo runtime available in the plugin: select it to enable Hailo hardware acceleration</p></figcaption></figure>
3. After the installation is finished, the Integrations tab looks something like this, with a runtime active and a pipeline assigned:

   <figure><img src="https://2982163321-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKGfC0TFSmuMDjbpLErO3%2Fuploads%2Fgit-blob-782ddff28c543078e344332793470bfc943e4245%2Fclient-ai-manager-pipeline-selected.png?alt=media" alt="Camera Settings Integrations tab showing an active runtime in Device Overview and a pipeline assigned in the AI Pipeline dropdown"><figcaption><p>Integrations tab after installation - shown here with the Generic CPU runtime for illustration; the Device Overview panel reports Hailo once it is installed and selected</p></figcaption></figure>
4. To verify the runtime was downloaded, check the `inference_engine` folder of the AI Manager. It contains `libRuntimeLibrary.so`, the Hailo provider libraries and a `libhailort.so.<version>` that matches your HailoRT version. It also holds a header file and, while the runtime is loaded, a `libRuntimeLibrary.<N>.so` hard link (the number varies) that the plugin creates; ignore both:

   ```bash
   ls <server-install-dir>/mediaserver/var/nx_ai_manager/inference_engine
   ```

   Example output:

   ```
   installed_runtime.txt  libhailort.so.4.20.0  libonnxruntime_providers_hailo.so  libonnxruntime_providers_shared.so  libRuntimeLibrary.<N>.so  libRuntimeLibrary.so  ...
   ```

{% hint style="info" %}
The downloaded runtime bundles its own `libhailort`, so you do not install it separately. It must match the installed HailoRT version: after upgrading or downgrading HailoRT, select the Hailo runtime again.
{% endhint %}

## Convert a Model

The chip runs models compiled for it, not plain ONNX files. A model is usable on your machine when it has an artifact of the matching type in the AI Manager Cloud:

| Chip     | Artifact type           |
| -------- | ----------------------- |
| Hailo-8  | `application/x-hailo8`  |
| Hailo-8L | `application/x-hailo8l` |

<figure><img src="https://2982163321-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKGfC0TFSmuMDjbpLErO3%2Fuploads%2Fgit-blob-15345680b78e6d713262294280c408b340fd4f1a%2Fmodel-mimetypes.png?alt=media" alt="Model detail page showing available artifact download types including Hailo 8 ONNX and
     Hailo 8L ONNX entries"><figcaption><p>The model must include a Hailo artifact: look for <code>application/x-hailo8</code> or <code>application/x-hailo8l</code> in the model detail page.</p></figcaption></figure>

### Let the Cloud Convert Your Model

The simplest way: when you [upload your ONNX model](https://nx.docs.scailable.net/ai-manager-v6.1.4/ai-manager-cloud-ui/upload-your-model.md), enable the **Hailo** conversion in the hardware step. The Cloud compiles the model and adds the Hailo artifacts (Hailo-8 and Hailo-8L); no local Hailo tooling is needed.

Keep in mind:

* **Operators:** Hailo's compiler decides which operators it supports. For example, YOLOv8 and YOLOX models convert, while YOLO11 models (attention layers) failed with Hailo's compiler. The model also needs one image input with a fixed size.
* **Quantization:** the model is quantized to INT8, so detections can differ slightly from the CPU result (boxes by about 10 pixels in our tests).

### Compile and Upload Your Own Model

If the Cloud conversion fails or you want to control the compilation, you can compile the model yourself with Hailo's Dataflow Compiler (available from the Hailo Developer Zone) and upload the result. The compiler's HEF file is not what the Cloud takes: after `runner.compile()`, export the Hailo runtime ONNX model with `runner.get_hailo_runtime_model()`:

```python
import onnx

hef = runner.compile()                         # the HEF itself is not needed
onnx_model = runner.get_hailo_runtime_model()  # only possible on a compiled model

# The Hailo runtime supports ONNX opset 16 at most
from onnx import version_converter
if any(o.domain == '' and o.version > 16 for o in onnx_model.opset_import):
    onnx_model = version_converter.convert_version(onnx_model, 16)
onnx.save(onnx_model, 'model_hailo.onnx')
```

The Cloud recognizes a compiled Hailo model by its `HailoOp` node and reads the target chip from the ONNX metadata field `chip`:

| `chip` value                 | Artifact                |
| ---------------------------- | ----------------------- |
| `hailo8`, `hailo-8`, `hailo` | `application/x-hailo8`  |
| `hailo8l`, `hailo-8l`        | `application/x-hailo8l` |

The exported model has no `chip` field, and the Cloud then warns and assumes Hailo-8; an unknown value also counts as Hailo-8. The script below sets the field: use `hailo8l` for a model compiled for Hailo-8L.

The exported model keeps the input and output names generated by Hailo's tools (the input is named `input0`). The AI Manager matches tensors by name and expects the names defined in the [ONNX requirements](https://nx.docs.scailable.net/ai-manager-v6.1.4/ai-models-support/custom-models/onnx-requirements.md), so rename them before uploading with the script below.

Open the script and adjust every line marked `TODO`:

* the model path,
* the `chip` value,
* the input name,
* the shape of the `mask-` input,
* the shape of the output.

The script handles a model with a single output only and stops with a message otherwise.

{% file src="/files/8tIKCIXyWQHtNxY0AJ3S" %}
Python script to adjust the ONNX input & output metadata.
{% endfile %}

{% file src="/files/YivPgrS8LGcNpNh2ErvJ" %}
A Yolov4-tiny model that's conforming to the AI Manager's model requirements (a naming reference only: a plain ONNX file, not a compiled Hailo model)
{% endfile %}

## Configure the Runtime

The Hailo runtime works without any configuration. To change its defaults, create `runtime_args.json` in:

```
<server-install-dir>/mediaserver/var/nx_ai_manager/inference_engine/runtime_args.json
```

**Every value must be a JSON string** (`"2"`, not `2`); other types are skipped. A file that is not valid JSON is ignored entirely, with only a warning in the plugin log. For example:

```json
{
  "n_replicas": "2",
  "log_level": "0"
}
```

{% hint style="warning" %}
On Linux, reselecting a runtime in the AI Manager plugin, or clearing its cache, deletes `runtime_args.json`. Select the Hailo runtime first, then create the file, and only then restart the Mediaserver. A file written before the runtime was selected is removed again.
{% endhint %}

The deletion covers the whole content of the `inference_engine` folder, `runtime_args.json` included.

After making the changes, restart the Mediaserver, see [Restarting the Server](https://nx.docs.scailable.net/ai-manager-v6.1.4/support-and-troubleshooting/troubleshooting/controlling-the-server-and-the-plugin.md#restarting-the-server).

| Key          | Values (default in bold)                        | Meaning                                                                                                                                   |
| ------------ | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `log_level`  | `0` debug, **`1`** info, `2` warning, `3` error | Minimum level written to the runtime log file. Out of range counts as `1`                                                                 |
| `log_file`   | file path, **`runtime.log`**                    | Base name of the runtime log. Files are named `<log_file>.0`, `.1`, ... and a new one starts every 4 MiB. Use a path the server can write |
| `n_threads`  | `1` to `16`, **`4`**                            | Total CPU threads for the parts of the models that run on the host, split evenly over all models and replicas                             |
| `perf_mode`  | `eco`, `power`, **unset**                       | Only used when `n_threads` is not set: 40% (`eco`) or 90% (`power`) of the CPU cores                                                      |
| `n_replicas` | `1` or more, **`1`**                            | Sessions per model, each with its own worker thread. Raise it only if the host CPU, not the chip, limits the throughput                   |

There is no key to select a chip.

## Monitor

Monitoring Hailo usage with the `hailortcli monitor` command requires setting a specific environment variable. Follow these steps:

1. Add the following line to the `/etc/systemd/system/<service-name>.service` file (in the `[Service]` section, as in the example below) to set the necessary environment variable:

   ```ini
   Environment="HAILO_MONITOR=1"
   ```
2. Reload systemd and restart the Mediaserver (see [Restarting the Server](https://nx.docs.scailable.net/ai-manager-v6.1.4/support-and-troubleshooting/troubleshooting/controlling-the-server-and-the-plugin.md#restarting-the-server)) for the changes to take effect, using one of the following command sets:

   ```bash
   sudo systemctl daemon-reload
   sudo systemctl restart <service-name>.service
   ```

   or

   ```bash
   sudo systemctl daemon-reload
   sudo service <service-name> restart
   ```
3. Run `hailortcli`:

   ```bash
   HAILO_MONITOR=1 hailortcli monitor
   ```

<details>

<summary>Example of the updated service file (descriptions and paths depend on your installation)</summary>

```ini
[Unit]
Description=Media Server
After=network.target local-fs.target remote-fs.target
Requires=<server-user>-root-tool.service

[Service]
PermissionsStartOnly=true
ExecStartPre=<server-install-dir>/mediaserver/lib/scripts/systemd_mediaserver_pre_start.sh
ExecStart=<server-install-dir>/mediaserver/lib/scripts/systemd_mediaserver_start.sh
User=<server-user>
Group=<server-user>
Restart=always
TimeoutStopSec=120
KillMode=process
TasksMax=8191
LimitCORE=infinity
Environment="HAILO_MONITOR=1"

[Install]
WantedBy=multi-user.target
```

</details>

<figure><img src="https://2982163321-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKGfC0TFSmuMDjbpLErO3%2Fuploads%2Fgit-blob-a39295f59f88e63f9f9342cd41b9f6ccfc0c6139%2Fhailortcli-monitor-output.png?alt=media" alt="Terminal output from hailortcli monitor showing runtime statistics"><figcaption><p>Example output of <code>hailortcli monitor</code> (a yolov4 demo, not an AI Manager session)</p></figcaption></figure>

## Troubleshooting

| Log         | Location                                                             | Content                                                     |
| ----------- | -------------------------------------------------------------------- | ----------------------------------------------------------- |
| Runtime log | `<server-install-dir>/mediaserver/bin/runtime.log.0`                 | Runtime messages; the reason a model fails to load is there |
| HailoRT log | `hailort.log` in the same folder                                     | HailoRT messages                                            |
| Plugin log  | `<server-install-dir>/mediaserver/var/nx_ai_manager/nxai_plugin.log` | Plugin messages                                             |

| Symptom                                                                                                                                                   | Cause                                                                                                                                                                                                                               | Fix                                                                                                                        |
| --------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| The Hailo runtime is not offered                                                                                                                          | No device in `/sys/class/hailo_chardev`, `hailortcli` missing, or a HailoRT version other than 4.17.0 to 4.20.0. The UI shows no message; `nxai_plugin.log` has a line such as `Hailo-8 driver version <version> is not supported.` | Run the [checks above](#check-the-installation). Install the driver, or a supported version, then restart the Mediaserver. |
| `hailortcli scan` lists nothing and `/dev/hailo*` is missing                                                                                              | The PCIe driver is not installed or loaded                                                                                                                                                                                          | Install `hailort-pcie-driver`, reload it or reboot, then check `lsmod \| grep hailo`.                                      |
| The model fails to load with `Failed to create ORT session`; `hailort.log` says `Driver version (...) is different from library version (...)`            | The runtime was installed for another HailoRT version than the one now on the machine                                                                                                                                               | Select the Hailo runtime again so the matching one is downloaded.                                                          |
| A model with several inputs (for example `image-`, `mask-` and `nms_sensitivity-`) gives no detections; the runtime log says `Unexpected input data type` | An older Hailo runtime package bound inputs by position instead of by name                                                                                                                                                          | Select the Hailo runtime again to get the current package.                                                                 |
| `MLDataType for: tensor(float) is not currently registered` after switching from another runtime to Hailo                                                 | An older Hailo runtime package                                                                                                                                                                                                      | Select the Hailo runtime again to get the current package.                                                                 |
| `libatomic.so.1` or `libgomp.so.1: cannot open shared object file`                                                                                        | Missing system library                                                                                                                                                                                                              | `sudo apt install libatomic1 libgomp1`                                                                                     |
| Requests are accepted but no detections arrive; the runtime log has `Invalid Feed Input Name`                                                             | The model input name does not match the one the AI Manager sends                                                                                                                                                                    | Rename the model's inputs as described in [Compile and Upload Your Own Model](#compile-and-upload-your-own-model).         |
| `max_desc_page_size given ... is bigger than hw max desc page size`                                                                                       | The host does not support the PCIe descriptor page size                                                                                                                                                                             | See [PCIe Descriptor Page Size Error](#pcie-descriptor-page-size-error).                                                   |

### PCIe Descriptor Page Size Error

If you encounter the following error (actual page size may vary), it indicates that your host does not support the specified PCIe descriptor page size:

```
[HailoRT] [error] CHECK_AS_EXPECTED failed - max_desc_page_size given 16384 is bigger than hw max desc page size 4096
```

This issue is common on ARM64 devices like the Raspberry Pi AI Kit.

<details>

<summary>Fix the PCIe descriptor page size</summary>

To resolve it, add the following configuration to `/etc/modprobe.d/hailo_pci.conf`. If the file does not exist, create it and set the `max_desc_page_size` to the value mentioned in the error (e.g., 4096):

```bash
options hailo_pci force_desc_page_size=4096
```

Add this configuration by running the following command:

```bash
echo 'options hailo_pci force_desc_page_size=4096' | sudo tee -a /etc/modprobe.d/hailo_pci.conf
```

Reboot the machine for the changes to take effect, or reload the driver without rebooting. The driver cannot be unloaded while the Mediaserver uses the device, so stop the Mediaserver first (see [Controlling the Server and the Plugin](https://nx.docs.scailable.net/ai-manager-v6.1.4/support-and-troubleshooting/troubleshooting/controlling-the-server-and-the-plugin.md)), run these commands, then start it again:

```bash
sudo modprobe -r hailo_pci
sudo modprobe hailo_pci
```

</details>

## Limitations

| Limit              | Detail                                                                                                                                                                                                                            |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| One model per chip | A Hailo chip runs one model. Running several models on the same chip is not supported                                                                                                                                             |
| One chip is used   | The runtime opens a single Hailo chip, even when several are installed (checked on a machine with four Hailo-8 modules: HailoRT opens one device); the other chips stay unused. In practice, assign one Hailo model to the server |
