MemryX Support
Summary
MX3
Experimental
x86_64
Ubuntu 22.04, Ubuntu 24.04
2.2.0
Introduction
About MemryX
MemryX, Inc. was established with the mission of accelerating Edge AI workloads by providing a highly efficient and scalable solution that is simple to deploy and support. MemryX’s AI accelerator chips, known as MXAs, offer server-like performance and accuracy at low power and low cost and are well-positioned for Edge AI applications, including autonomous driving, robotics, machine vision, and more.
The MemryX MX3 M.2 AI Accelerator module has an M-key 2280 form factor and includes 4 MXA chips. This production module is now supported by the AI Manager.
Workflow
The AI Manager simplifies the process of securely deploying your AI models on MemryX MX3 devices with the following steps:
Install all necessary software dependencies.
Install the Nx Meta Server on your machine and choose the MemryX runtime option.
Upload your model to the AI Cloud for compilation.
Compile and install the required post-processor.
Set up your inference pipeline.
Hardware and Software Requirements
Hardware Requirements:
Hardware requirements can be accessed in detail on the MemryX Developer Hub. Please see the following resource for hardware installation, requirements and recommendations.
In general, any modern x86_64 system with AVX2 and a suitable M.2 slot should work well with the MX3.
The MemryX driver and runtime libraries support multiple CPU and distribution combinations, provided via: apt package repository
Software Requirements:
Ubuntu 22.04 LTS
Ubuntu 24.04 LTS
Installation steps
Follow the runtime installation steps from the MemryX Developer Hub: Installation Instructions
The instructions will guide you through the installation of both the MemryX runtime and drivers, which are required to use the device with the AI Manager.
After installing the drivers and runtime, you will be prompted to reboot your system. After rebooting, please return to the installation steps on the Developer Hub and complete Step 7, which instructs you to install the MemryX utilities and plugins. These are strictly required for compatible use with AI.
Verify installation
To verify your MemryX installation, power on your system and run the following command:
The command should return the following to indicate that a single MemryX module is properly installed and ready to use:
If you wish to further verify the installation by running a simple workload on the accelerator module, please refer to the Hello, MXA! tutorial on the MemryX Developer Hub.
Getting Started
This guide walks you through deploying the Yolov8n model on an MemryX MX3 M.2 module using the AI Manager.
Before you begin, ensure you are familiar with the AI Manager and the process of running models on a CPU. If you need a refresher, please see the configuration section.
For this guide, we will use the following resources:
ONNX Model: The model is available at model.zip
Post-processor: The post-processor source is available at post_processor.zip
Upload model to AI Cloud
Upload your ONNX model to the AI Cloud following the steps outlined in the upload section.
Crucially, enable the MemryX conversion option within the model upload form and set the correct normalization values (as demonstrated in the image below).
The cloud platform will automatically convert the standard ONNX model into an MX3-compatible binary in the background. The model is ready for deployment once its status updates to ok, as shown below:
Select the MemryX runtime
The AI Manager provides native support for multiple AI accelerators. To route inference to the chip, select the MemryX runtime in the AI Manager settings page:
Install post-processor
Models running on the MX3 require an external post-processor to translate raw network output tensors (e.g., probability matrices) back into meaningful Nx metadata, such as object bounding boxes and classifications.
To deploy the provided YOLOv8m example, complete the following steps to compile and install the external C/C++ post-processor directly on the Nx Meta Server machine:
Extract the post-processor source code archive.
Execute the provided compilation script compile_install.sh. This builds the post-processor and automatically copies its config.json definition into the AI Manager's post-processors directory:
To apply the change, restart the Nx VMS server. Instructions are available in this section.
Configure pipeline
Within the Nx Meta Client, configure the pipeline as shown in the image below:
Enable inference on camera
The final step is to activate inference for your camera by toggling the Device Active switch on the AI Manager settings page (as shown below).
You should immediately begin seeing inference results in the Nx Meta Client after enabling the "Objects" tab.
To enable model inference on multiple cameras, simply repeat steps 4 and 5 for each camera.
Supported Models
An extensive list of supported models can be found in the MemryX Model Explorer.
Rather than providing a curated ‘model zoo’, the MemryX team emphasizes their solution’s ability to support a wide variety of models out-of-the-box by using BF16 activations, enabling users to run their own models without manual quantization or modifications.
The MemryX Model Explorer is a great resource to characterize the types of supported model architectures, but users are certainly not limited to this selection of models.
Monitoring
Monitor pipeline throughput
To see the number of frames the AI Manager is processing for a camera, enable the model FPS event. You can find detailed instructions on this page: Show AI inference framerate on video.
Monitor chip metrics
Chip Temperatures
There are two methods to view the chip temperatures. The first method is to use the C++ MxAccl API, which has two functions for temperature monitoring.
Use the following function to monitor the temperature of each chip on the specified device (docs):
To see the maximum chip temperature across all chips on the device, use (docs):
If you prefer to use the terminal for monitoring, you can run the following command to check the temperatures of all chips on your device:
For continuous monitoring from the terminal, try the following command to view updated temperatures in 1-second intervals:
Note that all temperature measurements are provided in degrees Celsius.
Chip Utilization
The C++ MxAccl API uses a metric referred to as ‘pressure’, which serves to approximate the throughput utilization of a given model on the device. You can read more about the pressure metric here.
You can use the following C++ function to retrieve the MXA’s current pressure status (docs):
Again, if you prefer to monitor utilization from the terminal, you can do so continuously in 1-second intervals using the following command:
Note that the percentage provided in this file is not measured in the same way as the pressure metric described above. It is more so an approximation of the current core utilization.
Troubleshooting
Missing bounding-boxes
If you are not seeing bounding boxes visualized in the Nx Client, check the following potential factors:
Model Accuracy: The model might not be accurate enough for the specific environment where it is deployed.
Corrupted Configuration: If the pipeline configuration has become corrupted, try re-installing the post-processor and reconfiguring the pipeline.
Throughput Saturation: If you assign the pipeline to too many high-framerate streams, the pipeline throughput may become significantly lower than the stream framerate, causing the AI Manager to drop most frames. To troubleshoot this, disable the AI Manager integration on all but one camera and gradually increase the number of cameras until you identify the machine's saturation limit.
Pipeline Incompatibility: If a camera on the server is assigned a model that is not compatible with the MemryX chip, the AI Manager integration will stop working for all cameras across the server. Verify that all cameras have the correct pipeline assigned. If any camera has an incompatible model, disable the AI Manager integration for that camera.
Technical Support
For technical questions or assistance, please reach out to us through the following channels:
MemryX Developer Support: MemryX Developer Community Forum
Nx Customer Support Portal: Network Optix