SDKs
Edge SDK: ONNX and Core ML
Run Bio on the device itself: a robot, a phone, a site PC, a browser tab. No network call and no per-minute charge.
What runs on-device is the per-crop network. You give it a frame and the boxes of the animals in it (from any detector), and for each box it returns 17 body points, 8 behaviour probabilities and a 128-number identity embedding. Tracking, state rules and the forecast are not in this package yet. For the whole pipeline on your own machine, use the vitals-bio Python package.
Packages
| Package | Runtime | For | Checked how |
|---|---|---|---|
Python vitalsbio-edge | ONNX Runtime, or Core ML through coremltools | Linux boards, site PCs, Macs | Resize against OpenCV; ONNX against Core ML |
JavaScript @vitalsbio/edge | onnxruntime-web or onnxruntime-node | Browsers, Node, Electron | Against the Python package |
Swift VitalsBioEdge | Core ML | iPhone, iPad, Mac apps | It compiles. Not run on a phone, and not parity-checked |
Use it
Python
from vitalsbio_edge import BioEdge # pip install ./sdk/edge/python onnxruntime
bio = BioEdge("bionet.onnx") # or BioEdge("bionet-b1.mlpackage")
for a in bio.run(frame_rgb, [[x0, y0, x1, y1]]): # HxWx3 uint8 RGB; boxes in pixels
print(a["behaviour_top"], a["keypoints"][2]) # nose: [x, y, score]
JavaScript
import * as ort from "onnxruntime-web";
import { BioEdge } from "@vitalsbio/edge";
const bio = await BioEdge.load(ort, "/models/bionet.onnx");
const out = await bio.run({ data: rgba, width, height, channels: 4 }, [[x0, y0, x1, y1]]);
Swift
let bio = try VitalsBioEdge(modelURL: url) // bionet-b1.mlpackage let out = try bio.run(image: cgImage, boxes: [CGRect(x: 100, y: 80, width: 300, height: 200)])
Speed
Time for the network on one crop. REAL rows were measured on 2026-11-21 on one Mac, CPU only (the GPU and Neural Engine were not used), as the median of 10 to 30 runs on 48 crops from 3 real images. ESTIMATE rows were not run.
| Device | Runtime | Per crop | Crops a second | Label |
|---|---|---|---|---|
| Apple M5 Pro, CPU only | Core ML fp16, batch 1 | 9.0 ms | 111 | REAL |
| Apple M5 Pro, CPU only | Core ML fp16, batch 16 | 5.8 ms | 171 | REAL |
| Apple M5 Pro, CPU only | ONNX Runtime fp32, batch 1 | 14.6 ms | 68 | REAL |
| Apple M5 Pro, CPU only | ONNX Runtime fp32, batch 8 | 15.3 ms | 65 | REAL |
| Apple M5 Pro, CPU only | onnxruntime-web (wasm, 4 threads, in Node) | 91 ms | 11 | REAL |
| Apple M5 Pro, CPU only | Python end to end: crop, resize, network, decode | 17.7 ms | 57 | REAL |
| iPhone 15 or newer | Core ML, Neural Engine | 3 to 10 ms | 100 to 300 | ESTIMATE |
| Jetson Orin Nano | TensorRT fp16 | 5 to 15 ms | 65 to 200 | ESTIMATE |
| Raspberry Pi 5 | ONNX Runtime CPU | 150 to 300 ms | 3 to 7 | ESTIMATE |
How the estimates were made: iPhone and Jetson ranges come from the size of the network (a ViT-S/14, 22.6 million parameters, 224 px input) and typical public figures for that class on those chips. The Raspberry Pi range is the REAL ONNX row scaled by 10x to 20x, a rough guess. Use them for planning only, and run sdk/edge/tools/bench.py on your device for a number you can trust. This network is a vision transformer and is not a fit for small camera NPUs such as the RV1106.
Do the runtimes agree?
All REAL, measured here. These say the packages compute the same thing. They are not accuracy numbers; those are on what is live.
| Check | Result |
|---|---|
| Python resize against OpenCV, 240 random crops | At most 1 grey level apart on any pixel |
| JavaScript against Python: same frame, 8 boxes, 136 body points | Points at most 0.065 px apart; same top behaviour on 8 of 8 |
| ONNX fp32 against Core ML fp16, 48 crops: top behaviour | Same on 47 of 48 |
| ONNX fp32 against Core ML fp16: body points with a score of 0.3 or more | At most 3.4 px apart in the 224 px crop (median 0.26 px over all points) |
| ONNX fp32 against Core ML fp16: body points with a low score | Up to 218 px apart. Two near-equal peaks can swap, so filter points by score |