Docs
Docs pages

SDKs

Edge SDK: ONNX and Core ML

Run Bio on the device itself: a robot, a phone, a site PC, a browser tab. No network call and no per-minute charge.

What runs on-device is the per-crop network. You give it a frame and the boxes of the animals in it (from any detector), and for each box it returns 17 body points, 8 behaviour probabilities and a 128-number identity embedding. Tracking, state rules and the forecast are not in this package yet. For the whole pipeline on your own machine, use the vitals-bio Python package.

The model files (45 to 89 MB) are not a public download yet. Design partners get them with a release bundle. The current files are a dry-run release of the v0 heads: pose Live behaviour Live identity Pending

Packages

PackageRuntimeForChecked how
Python vitalsbio-edgeONNX Runtime, or Core ML through coremltoolsLinux boards, site PCs, MacsResize against OpenCV; ONNX against Core ML
JavaScript @vitalsbio/edgeonnxruntime-web or onnxruntime-nodeBrowsers, Node, ElectronAgainst the Python package
Swift VitalsBioEdgeCore MLiPhone, iPad, Mac appsIt compiles. Not run on a phone, and not parity-checked

Use it

Python

from vitalsbio_edge import BioEdge              # pip install ./sdk/edge/python onnxruntime

bio = BioEdge("bionet.onnx")                    # or BioEdge("bionet-b1.mlpackage")
for a in bio.run(frame_rgb, [[x0, y0, x1, y1]]):   # HxWx3 uint8 RGB; boxes in pixels
    print(a["behaviour_top"], a["keypoints"][2])   # nose: [x, y, score]

JavaScript

import * as ort from "onnxruntime-web";
import { BioEdge } from "@vitalsbio/edge";

const bio = await BioEdge.load(ort, "/models/bionet.onnx");
const out = await bio.run({ data: rgba, width, height, channels: 4 }, [[x0, y0, x1, y1]]);

Swift

let bio = try VitalsBioEdge(modelURL: url)      // bionet-b1.mlpackage
let out = try bio.run(image: cgImage, boxes: [CGRect(x: 100, y: 80, width: 300, height: 200)])

Speed

Time for the network on one crop. REAL rows were measured on 2026-11-21 on one Mac, CPU only (the GPU and Neural Engine were not used), as the median of 10 to 30 runs on 48 crops from 3 real images. ESTIMATE rows were not run.

DeviceRuntimePer cropCrops a secondLabel
Apple M5 Pro, CPU onlyCore ML fp16, batch 19.0 ms111REAL
Apple M5 Pro, CPU onlyCore ML fp16, batch 165.8 ms171REAL
Apple M5 Pro, CPU onlyONNX Runtime fp32, batch 114.6 ms68REAL
Apple M5 Pro, CPU onlyONNX Runtime fp32, batch 815.3 ms65REAL
Apple M5 Pro, CPU onlyonnxruntime-web (wasm, 4 threads, in Node)91 ms11REAL
Apple M5 Pro, CPU onlyPython end to end: crop, resize, network, decode17.7 ms57REAL
iPhone 15 or newerCore ML, Neural Engine3 to 10 ms100 to 300ESTIMATE
Jetson Orin NanoTensorRT fp165 to 15 ms65 to 200ESTIMATE
Raspberry Pi 5ONNX Runtime CPU150 to 300 ms3 to 7ESTIMATE

How the estimates were made: iPhone and Jetson ranges come from the size of the network (a ViT-S/14, 22.6 million parameters, 224 px input) and typical public figures for that class on those chips. The Raspberry Pi range is the REAL ONNX row scaled by 10x to 20x, a rough guess. Use them for planning only, and run sdk/edge/tools/bench.py on your device for a number you can trust. This network is a vision transformer and is not a fit for small camera NPUs such as the RV1106.

Do the runtimes agree?

All REAL, measured here. These say the packages compute the same thing. They are not accuracy numbers; those are on what is live.

CheckResult
Python resize against OpenCV, 240 random cropsAt most 1 grey level apart on any pixel
JavaScript against Python: same frame, 8 boxes, 136 body pointsPoints at most 0.065 px apart; same top behaviour on 8 of 8
ONNX fp32 against Core ML fp16, 48 crops: top behaviourSame on 47 of 48
ONNX fp32 against Core ML fp16: body points with a score of 0.3 or moreAt most 3.4 px apart in the 224 px crop (median 0.26 px over all points)
ONNX fp32 against Core ML fp16: body points with a low scoreUp to 218 px apart. Two near-equal peaks can swap, so filter points by score