Nx AI Certification Test
Certification Test
Quickstart
The Nx AI Certification test should be able to run on any Ubuntu installation which is also compatible with Network Optix Server.
Ensure your device has at least a few gigabytes of free space, a working internet connection and a working Python 3 and Pip installation.
If you are testing a GPU or NPU, also make sure that the required drivers are installed, for example the Jetpack drivers for NVIDIA accelerators.
Execute the following commands:
Running the python3 Utilities/install_acceleration_library.py
command will automatically detect the available hardware acceleration that is available on your device. If the script finds more than one option, it will pause execution and ask you to choose. If you want to execute these commands without pausing, add an the accelerator name to the command, such as the "Nx CPU" example.
The download_models.py
command will download approximately 3.5GB of model data, depending on your connection this may take a while.
The test will run for a couple of hours and will stress the device. Do not power off your device.
Introduction
The Nx AI Certification Test is meant to test any device to ensure it is compatible and stable enough to run the Nx AI Manager for extended periods of time.
The test will attempt to run multiple common model architectures, with different sizes, and even multiples of those models to test compatibility and what can be expected to run on the device. The test will also run long operation tests to check if there are any memory issues or degradation in performance due to overheating or other reasons.
Finally, the test will gather all results in a folder which can be used to generate a report, either on the device or somewhere else. The report should contain enough information to determine if the device is indeed compatible.
Installing Nx AI Manager
A useful script is provided to install the Nx AI Manager runtime locally within the test environment. This will ensure that this testing does not interfere with any existing installations on the device.
Installing Nx Acceleration Library
Acceleration libraries act as layers between the Nx AI Manager and your acceleration hardware. Any device should be able to run the tests on CPU, without acceleration hardware. However if your device has AI acceleration hardware, such as an Nvidia Cuda device, or a Hailo AI chip, these can be used to accelerate the AI pipeline.
Running this script will automatically detect available hardware. If more than one is found, the script will present a list of options which you can choose from.
Download Models
The Certification Test will test a variety of models to see if they can run on your device and libraries. After installing the acceleration library the correct models for your device can be downloaded.
This step could use a lot of data as many large files need to be downloaded. This might also take a long time depending on network conditions.
Running test
The Nx AI Certification test includes a collection of tests to test different aspects of your device to ensure that the Nx AI Toolkit can run on your device. To run all of these tests in one large test, run:
This could take many hours to complete.
Gathering Hardware Information
This command will detect some hardware information about your device and package it with the test results. This will gives context to the test results and allows for better analysis about the device's performance.
Uploading Results
An endpoint was created where you can upload and view the test results of your device. After the test has completed, run the following to automatically upload the script to the cloud:
Custom Model Benchmark
The Custom Model Benchmark can only be used to benchmark models which have been uploaded to the Nx AI Cloud.
The test allows you to benchmark your own models on different devices. The Certification Test includes the functionality to add your own models to its benchmark test. If you already have that downloaded, you can skip downloading and extracting the benchmark test. If you are only interested in benchmarking a model, you can instead download the smaller benchmark package:
Install an Nx AI Manager with runtime by following the steps at Installing Nx AI Manager
Next, add as many models as you want to benchmark by running:
And entering the ID of the model you want to test. The ID of your model can be found by navigating to your model in the model cloud. For example:
After you have successfully added all the models you want to benchmark, you can download the model files from the cloud by running:
Once all your models have been downloaded, start the benchmark by running:
The test should now automatically benchmark all the models you've added and give you an overview of how this model/device performs.
Troubleshooting
If the device is working well, but the test is not passing, feel free to contact us for support. To make it easier for us to provide support, please include log files so that we can see what is going wrong on the device.
Make sure you're in the root folder of the test suite and run the following command to gather all the log files:
This will create a file called test_logs.tgz
, please send us this file with your request.
Last updated