Recipes
Robots: see the animal, plan around it
A feed pusher or a barn rover needs to know where the animals are, what they are doing, and where they will be in two seconds, without waiting on a network call. Leans on pose Live behaviour Live forecast Pending
On the robot: the Edge SDK
from vitalsbio_edge import BioEdge
bio = BioEdge("bionet.onnx") # Core ML on Apple hardware: BioEdge("bionet-b1.mlpackage")
while robot.running:
frame = camera.read_rgb() # HxWx3 uint8
boxes = detector(frame) # your animal detector: [[x0, y0, x1, y1], ...]
for box, a in zip(boxes, bio.run(frame, boxes)):
if a is None:
continue
if a["behaviour"]["lying"] > 0.8:
planner.keep_clear(box, margin_m=1.5) # don't nudge a resting animal
head = a["keypoints"][2] # nose: [x, y, score]
if head[2] >= 0.3:
planner.note_head(head[:2])
Measured on one Mac, CPU only: 9 ms a crop with Core ML and 15 ms with ONNX Runtime (REAL). A Jetson or a phone is an ESTIMATE until you run the benchmark on it. See the Edge SDK for the full table.
From the API: where they are heading
r = client.forecast(clip_bytes, species="cattle", max_seconds=5)
if r["capability"]["status"] != "live":
margin = 2.0 # not tested yet: plan with a wide margin
for a in r["animals"]:
if a["available"]:
f = a["frames"][-1]["forecast"] # the newest frame's forecast
for h, (x, y) in zip(f["horizons_s"], f["centre"]):
planner.avoid(x, y, at_s=h)
What this can and can't tell you
- Positions are in image pixels. Turning them into metres on the barn floor is your camera calibration.
- The forecast today is a constant-velocity baseline and its capability is Pending. An animal that is about to turn or stand up is exactly where it is wrong. Keep your own safety stop.
- A round trip to the hosted API is too slow for a control loop. Use it for planning and logging, and the Edge SDK for anything the robot reacts to.
- The Edge SDK does not include a detector. The one the hosted API uses is AGPL-3.0, which matters for a closed product.