Accelerators Overview

How the AI Manager converts models into artifacts for each accelerator runtime.

The AI Manager accelerates AI model inference using built-in runtimes, each dedicated to a specific platform and AI accelerator (CPU, GPU, NPU, etc.). These runtimes integrate directly into the AI Manager, so new runtimes and acceleration types can be added.

When a user uploads a supported model to the AI Manager Cloud, the platform runs several conversion processes on the file, each generating a model artifact for one of the runtimes.

Example of a Teachable Machine Model

Suppose you've trained a teachable machine model, and exported it according to the guidelines mentioned here, the following model artifacts are stored to be used by the AI Manager when needed:

Model artifact types: each runtime uses a different artifact generated from the uploaded model.

• application/zip; source=original: is the original ZIP archive that's uploaded in the interface.

• application/zip; kind=teachable-machine: is the original ZIP archive that's uploaded in the interface with specific details for Teachable Machine.

• application/x-onnx: converts the TFLite generated earlier to ONNX, validated and optimized to run on CPU, Intel, and NVIDIA hardware. This step does not perform quantization on the model.

• application/zip; device=mxa: is the artifact generated by compiling the ONNX file into an optimized file dedicated only for MemryX hardware.

• application/x-onnx; device=hailo: is a custom ONNX generated specifically for Hailo-8 chips.

Any runtime or toolchain combination that follows the Open AI Accelerator eXchange (OAAX) standard is compatible with the AI Manager, so any installed runtime can be swapped for a new one. AI chip makers integrating with the AI Manager can refer to the documentation repository on GitHub.