> For the complete documentation index, see [llms.txt](https://nx.docs.scailable.net/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nx.docs.scailable.net/ai-manager-v6.1.4/miscellaneous/nx-ai-certification-test.md).

# AI Certification Test

## Quickstart

The AI Certification test runs on any Ubuntu installation that is also compatible with Network Optix Server.

Ensure your device has at least a few gigabytes of free space, a working internet connection and a working Python 3 and Pip installation.

{% code title="Python requirements" %}

```bash
# on some systems you may need to install pip
# for example with the following command on ubuntu
sudo apt update
sudo apt install python3-pip
```

{% endcode %}

{% hint style="warning" %}
If you are testing a GPU or NPU, also make sure that the required drivers are installed, for example the Jetpack drivers for NVIDIA accelerators.
{% endhint %}

Execute the following commands:

{% code title="Download tests" %}

```bash
## Create and enter folder for test
mkdir nxai_test
cd nxai_test

## Download testing suite
wget https://artifactory.nxvms.dev/artifactory/nxai_open/NXAITest/nxai_test.tgz

## Unpack testing suite
tar -xvf nxai_test.tgz
```

{% endcode %}

{% code title="Ubuntu quirk" %}

```bash
# on newer ubuntu systems you need to activate a python venv
sudo apt install python3-venv
python3 -m venv ./ # create venv in current dir
source ./bin/activate # activate python venv in current dir
```

{% endcode %}

{% code title="Install acceleration library" %}

```bash
## Install all required Python packages
pip3 install -r requirements.txt

## Install AI Manager
./Utilities/install_nxai_manager.sh

## Install acceleration runtime
python3 Utilities/install_acceleration_library.py
#python3 Utilities/install_acceleration_library.py "Nx CPU"
```

{% endcode %}

{% hint style="info" %}
The `python3 Utilities/install_acceleration_library.py` command automatically detects the available hardware acceleration on the device. If the script finds more than one option, it pauses execution and prompts for a choice. To run the command without pausing, add the accelerator name, such as in the "Nx CPU" example.
{% endhint %}

{% code title="Download models" %}

```bash
## Download required models
python3 Utilities/download_models.py
```

{% endcode %}

{% hint style="warning" %}
The `download_models.py` command downloads approximately 3.5GB of model data; download time varies with connection speed.
{% endhint %}

{% code title="Run tests" %}

```bash
## Start test
python3 all_suites.py
```

{% endcode %}

{% code title="Gather info and print ID" %}

```bash
## Gather info
python3 Utilities/gather_hwinfo.py
```

{% endcode %}

```bash
## Upload results to cloud
python3 Utilities/upload_results.py
```

{% hint style="warning" %}
The test runs for a couple of hours and stresses the device. Do not power off the device.
{% endhint %}

## Introduction

The AI Certification Test verifies that a device is compatible and stable enough to run the AI Manager for extended periods.

The test runs multiple common model architectures, with different sizes and multiples of those models, to check compatibility and expected performance on the device. The test also runs long-duration operations to check for memory issues or performance degradation caused by overheating or other factors.

Finally, the test gathers all results in a folder that can be used to generate a report, either on the device or elsewhere. The report contains enough information to determine whether the device is compatible.

## Installing AI Manager

A script installs the AI Manager runtime locally within the test environment, so testing does not interfere with any existing installations on the device.

```bash
./Utilities/install_nxai_manager.sh
```

## Installing Nx Acceleration Library

Acceleration libraries act as layers between the AI Manager and the acceleration hardware. Any device can run the tests on CPU, without acceleration hardware. A device with AI acceleration hardware, such as an Nvidia CUDA device or a Hailo AI chip, uses it to accelerate the AI pipeline.

Running this script automatically detects available hardware. If more than one option is found, the script presents a list to choose from.

```bash
python3 Utilities/install_acceleration_library.py
```

## Download Models

The Certification Test checks whether a variety of models run on the device and libraries. After installing the acceleration library, download the correct models for the device.

{% hint style="warning" %}
This step could use a lot of data as many large files need to be downloaded. This might also take a long time depending on network conditions.
{% endhint %}

```bash
## Download required models
python3 Utilities/download_models.py
```

## Running Test

The AI Certification test includes a collection of tests that check different aspects of the device to confirm the AI Toolkit runs on it. To run all tests together, run:

```bash
python3 all_suites.py
```

This takes several hours to complete.

## Gathering Hardware Information

This command detects hardware information about the device and packages it with the test results, giving context for better analysis of the device's performance.

```bash
## Gather info
python3 Utilities/gather_hwinfo.py
```

This command prints the ID of the device. Use this ID to identify the device when the test results are uploaded. Run this command again at any time to retrieve the ID.

## Uploading Results

An endpoint accepts uploads and displays the test results for the device. After the test completes, run the following command to upload the results to the cloud:

```bash
python3 Utilities/upload_results.py
```

## Custom Model Benchmark

{% hint style="info" %}
The Custom Model Benchmark can only be used to benchmark models which have been uploaded to the AI Manager Cloud.
{% endhint %}

The test allows you to benchmark your own models on different devices. The Certification Test includes the functionality to add your own models to its benchmark test. If you already have that downloaded, you can skip downloading and extracting the benchmark test. If you are only interested in benchmarking a model, you can instead download the smaller benchmark package:

{% code title="Download benchmark" %}

```sh
## Create and enter folder for test
mkdir nxai_benchmark
cd nxai_benchmark

## Download testing suite
wget https://artifactory.nxvms.dev/artifactory/nxai_open/NXAITest/nxai_benchmark.tgz

## Unpack testing suite
tar -xvf nxai_benchmark.tgz
```

{% endcode %}

Install an AI Manager with runtime by following the steps at [#installing-ai-manager](#installing-ai-manager "mention")

Next, add as many models as you want to benchmark by running:

```bash
python3 Utilities/add_benchmark_model.py <Model ID>
```

And entering the ID of the model you want to test. The ID of your model can be found by navigating to your model in the [model cloud](https://admin.sclbl.nxvms.com/models). For example:

```bash
python3 Utilities/add_benchmark_model.py 4c5527f3-242a-4e17-b2e3-727dd6740f7c
```

After you have successfully added all the models you want to benchmark, you can download the model files from the cloud by running:

```bash
python3 Utilities/download_models.py
```

Once all your models have been downloaded, start the benchmark by running:

```bash
python3 Benchmark-Suite/run_suite.py
```

The test benchmarks all added models and gives an overview of how the model/device performs.

## Troubleshooting

If the device is working well but the test is not passing, contact support. Include log files so support staff can identify what is going wrong on the device.

Make sure you're in the root folder of the test suite and run the following command to gather all the log files:

```bash
find ./ -type f \( -name "*.log" -o -name "failed_output.json" \) -printf '%P\n' | xargs tar -czf test_logs.tgz Results/
```

This creates a file called `test_logs.tgz`. Send this file with the support request.