> For the complete documentation index, see [llms.txt](https://nx.docs.scailable.net/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nx.docs.scailable.net/ai-manager-v6.1.4/ai-models-support/custom-models/onnx-requirements.md).

# ONNX requirements

The previous sections explain how to create ONNX graphs from scratch, with examples to illustrate the concept. If a model is already trained on another platform or framework, export it directly and upload the resulting file to the Nx platform for deployment.

This page offers a comprehensive specification of an ONNX model needed to benefit from all the built-in functionalities in the AI Manager.

## ONNX Version **1.15.0** <a href="#onnx-version-1.15.0" id="onnx-version-1.15.0"></a>

To meet the compatibility requirements for AI Manager XPU runtimes, use or export only ONNX versions up to 1.15.0. This version is solely compatible with Python 3.11.

Runtimes are refreshed at least every six months; check back periodically for updates on the latest supported ONNX version.

To install ONNX 1.15.0 for Python 3.11, you can use the following pip command:

Copy

```bash
pip install onnx==1.15.0
```

For inference purposes, ONNX models are typically executed using the ONNX Runtime. The ONNX Runtime version 1.17.0 supports ONNX opset version 20 and is compatible with Python 3.11. To install the ONNX Runtime for CPU execution, use:

Copy

```bash
pip install onnxruntime==1.17.0
```

If you require GPU support, you can install the GPU version of ONNX Runtime:

Copy

```bash
pip install onnxruntime-gpu==1.17.0
```

Always ensure your development environment aligns with the above versions to maintain compatibility with your AI Manager XPU runtimes.

## Requirements for the ONNX Graph <a href="#requirements-for-the-onnx-graph" id="requirements-for-the-onnx-graph"></a>

An ONNX graph/model is an abstract representation of computational graphs that uses nodes to describe operators, such as matrix multiplication, convolutions, and addition.

Although custom operators and attributes can be defined in an ONNX model, the AI Manager only supports the predefined set of operators in the official [specification](https://onnx.ai/onnx/operators/) (also see the [ONNX operators list](https://github.com/onnx/onnx/blob/main/docs/Operators.md)). Each ONNX model includes an "opset\_version" property, representing the [Operator Set Version](https://github.com/onnx/onnx/blob/main/docs/Versioning.md#operator-sets). Testing has only evaluated versions up to 17.

The AI Manager supports specific types of ONNX models and variants. Currently, the AI Manager supports vision models that process RGB images, but it is not designed to handle textual or any other sensor data without additional coding.

ONNX supports graph inputs with dynamic shapes - for example, an image classification model can process images of any Height x Width without knowing the input shape at runtime. However, the AI Manager currently only supports static input dimensions at the graph boundary; dynamic shapes are supported within the graph itself.

To recap, the ONNX models that are supported must:

* be vision models,
* with static input shapes,
* expect to operate on RGB/Grayscale images,
* using only operators defined in this [list](https://onnx.ai/onnx/operators/),
* with versions up to 17.

## Named IO Specification <a href="#named-io-specification" id="named-io-specification"></a>

The AI Manager streamlines operations by leveraging names of ONNX inputs and outputs (IOs) through built-in functions. This facilitates efficient processing and management within the system.

Use [Netron](https://netron.app/) to visually inspect the ONNX graph and confirm that the inputs and outputs match the descriptions in this specification.

In the following sections, a comprehensive specification will be provided that details the input and output (IO) names of the model, along with their corresponding shapes and data types.

## Inputs Specification <a href="#inputs-specification" id="inputs-specification"></a>

A typical vision model can handle different input types, such as images and confidence scores, in various formats and shapes. The AI Manager only recognizes specific, commonly used input types for vision models.

The following list gives information about what inputs are supported and how they are expected to be: their names, shapes, and datatypes.

1. **Image input** (*Required*):
   * Description: This input is the primary means of providing images to the model for processing.
   * Name: "`image`"
   * Shape: The input expects images with two possible shapes: (1, channels, height, width) or (1, height, width, channels), where channels can be either 1 (for Grayscale images) or 3 (for RGB images).
   * Data Type: Floating-point numbers (Float32)
   * Expected value: A tensor containing image values after normalization.
2. **Probability input** (*Optional*):
   * Description: This optional input allows for configuring a threshold score to eliminate bounding boxes with lower detection scores (how likely the bounding box contains an object).
   * Name: "`nms_sensitivity`"
   * Shape: (1)
   * Data Type: Floating-point numbers (Float32)
   * Expected value: A floating-point number within the range of \[0, 1].
3. **Mask input** (*Optional*):
   * This input is designed for object detection models and regulates areas in images for model outcomes on a pixel-level granularity.
   * Name: "`mask`"
   * Shape: (Height, Width)
   * Data Type: Boolean (BOOL)
   * Expected value: A binary value (0 or 1) indicating whether the pixel is included (1) or excluded (0) from the model's processing.

While the Mask and Probability inputs are particularly beneficial for object detection models, it's noteworthy that they possess versatile applications within the ONNX graph, extending beyond their immediate utility.

## Outputs Specification: Supported ONNX Model Outputs <a href="#supported-onnx-model-outputs" id="supported-onnx-model-outputs"></a>

ONNX models can produce many kinds of outputs (scores, bounding boxes, masks, body skeleton, etc.) in various formats. The AI Manager currently parses two output types used by most vision applications: **Scores** (for image classification) and **Bounding Boxes** (for object detection). These outputs are standardized for compatibility within the AI Manager framework.

Below is an overview of each supported output type:

1. **Scores**:
   * Description: A list of confidence scores for the detected objects.
   * Naming Syntax: "`scores-class_id:class_name`". For example: "scores-0:Cat;1:Dog;2:Horse;3:Nothing".
   * Shape: (1, number of classes)
   * Data Type: Floating-point numbers (Float32)
   * Expected value: Each element of the list is interpreted as a probability/score value.
   * ***Usability***: These scores are used to visualize the model output by picking the category with the highest score.
2. **Alarms**:
   * Description: A list of boolean values for the detected objects.
   * Naming Syntax: "`alarm-class_id:class_name`". For example: "alarm-0:Cat;1:Dog;2:Horse".
   * Shape: (1, number of classes)
   * Data Type: Boolean values (0 or 1)
   * Expected value: Each element within the list denotes a condition that, when met, triggers an alarm.
   * ***Usability***: These boolean values serve the purpose of triggering an alarm whenever a True value is present within the list. For instance, if a cat, dog, or horse is detected, an alarm will be raised accordingly.
3. **Bounding Boxes (bboxes)**:
   * Description: Coordinates for the bounding boxes around detected objects represented as a matrix.
   * Naming Syntax: "`bboxes-format:xyxysc;class_id:class_name`" For example: "bboxes-format:xyxysc;0:Person;1:Bike;2:Car".
   * Shape: (Number of detected bboxes, 6)
   * Data Type: Floating-point numbers (Float32)
   * Expected value: Each entry in this list comprises 6 values representing the coordinates of the top-left corner and the bottom-right corner of the bounding box, followed by the model's confidence score, and the class ID (xmin, ymin, xmax, ymax, score, class), thus the format value in the output name being: xyxysc.
   * ***Usability***: These outputs can be seamlessly integrated with a built-in post-processing mechanism, configurable based on user settings. This post-processing capability extends to various functionalities, including visualization, line-crossing analysis, QR/Barcode scanning, or the application of bounding-box area blurring.

These IO names and formats determine which input values each model expects during inference and how to interpret the output values. Following these naming conventions enables the AI Manager to interact correctly with deployed models.

## Beyond Image Classification and Object Detection Models <a href="#beyond-image-classification-and-object-detection-models" id="beyond-image-classification-and-object-detection-models"></a>

Certain AI models generate raw outputs that require specialized processing, such as image segmentation or pose estimation models. These outputs cannot be directly incorporated into the ONNX model and necessitate hard-coded post-processing using external procedures. This involves extracting insightful information from the model's raw outputs, such as bounding boxes or scores, using specific processing code. The extracted scores or bounding boxes should adhere to the specifications outlined in the previous subsections to ensure compatibility with the AI Manager.