AI Models for Frigate Enrichments

Frigate uses a handful of AI models — the object detector, plus the “enrichment” features that build on it (semantic search, face recognition, license plates, bird and audio detection, and more). This page lists every model, grouped by feature, and where each runs — the Apple Neural Engine (ANE), the GPU, or the CPU — on Fregata and, for comparison, in Docker.

The short version: object detection — the one thing that runs on every single frame — gets the Apple Neural Engine to itself. For everything else, Fregata uses the GPU for the models Frigate accelerates that Apple’s CoreML can run, and the CPU for the rest. After Fregata’s CoreML inference pipeline rewrite in v0.17.2.5 and v0.18.0-beta3, that’s nearly every enrichment model — there’s just one still waiting on the same treatment, and it isn’t a hot path (see below).

You don’t configure any of this — Fregata picks the right place automatically. The only related knob is model_size on the features that offer one (for example face recognition’s small vs large).

Where each model runs

The full picture — every model, the feature it powers, and where it runs across the three ways you might run Frigate:

Feature	Model (variant)	macOS Docker¹	Linux / Windows Docker²	Fregata (macOS)
Object detection	YOLOv9-t (the detector)	CPU	GPU / NPU	ANE
Semantic search	jinav1 (default model), large (default size)	CPU	GPU	GPU
	jinav1, small	CPU	CPU	text GPU · vision CPU
	jinav2 (opt-in, multilingual), large	CPU	GPU	GPU
	jinav2, small — not recommended, see below	CPU	CPU	CPU
Face recognition	FaceDet — finding faces	CPU	CPU	CPU
	FaceNet — small embedding	CPU	CPU	CPU
	ArcFace — large embedding (default)	CPU	GPU	GPU
License plate	YOLOv9-256 — plate detection	CPU	GPU	GPU
	PaddleOCR — text detection	CPU	GPU	GPU
	PaddleOCR — recognition_v4 (OCR)	CPU	GPU	GPU
Bird classification	MobileNetV2 (iNaturalist)	CPU	CPU	CPU
Audio detection	YAMNet	CPU	CPU	CPU
Audio transcription	sherpa-onnx — small (default)	CPU	CPU	CPU
	Whisper — large	CPU	GPU	CPU †
Custom classification	MobileNetV2 — training	CPU	GPU	CPU
	trained model (.tflite) — inference	CPU	CPU	CPU

¹ Docker Desktop on a Mac runs everything on the CPU. Docker’s Linux VM can’t reach the Mac’s Apple Neural Engine or GPU, so nothing is accelerated — the main reason Fregata exists.

² With a supported GPU or accelerator (NVIDIA/TensorRT, Intel/OpenVINO, a Coral, etc.). Without one, Linux/Windows Docker also runs everything on the CPU — same as the macOS Docker column.

† GPU on a GPU-equipped Frigate, but CPU on Fregata today. After Fregata’s CoreML inference pipeline rewrite in v0.17.2.5 and v0.18.0-beta3, every other model that used to carry this mark — semantic search, LPR, facial recognition — now runs 100% on the GPU. Whisper large is the last one still waiting on that same treatment. It is not a hot path (it runs per spoken phrase, not per frame), so the impact is small. See below.

The takeaway: Fregata is the only way to optimally hardware-accelerate Frigate on a Mac. macOS Docker is CPU-only across the board; Fregata puts object detection on the Apple Neural Engine and most enrichment models on the GPU — matching a GPU-equipped Linux/Windows box except for one model (Whisper large) that still falls back to the CPU for now.

Why object detection owns the Apple Neural Engine

The ANE is Apple’s dedicated, extremely power-efficient inference chip — but it works best running one model at a time. Object detection runs multiple times on every frame with motion from every camera, so it’s by far the most demanding and latency-sensitive job Fregata does. Pinning detection to the ANE — and routing everything else to the GPU or CPU — means a burst of face recognition or search indexing can never stall the detector. It’s also why Fregata needs Apple Silicon: Intel Macs have no Apple Neural Engine, and running detection on the CPU instead is dramatically slower than the ANE — too slow to be a real NVR.

Why some models use the GPU and others the CPU

Fregata is a faithful port of Frigate and keeps the same accuracy. Frigate already decides, per model, where it runs; Fregata honors those choices on Apple Silicon wherever the Mac’s GPU path can. Where a model offers a small and a large variant, these placements describe the variant named in the row — the two are different models and do not necessarily run in the same place:

• GPU (Apple’s CoreML) — the models Frigate accelerates and that CoreML can run: large semantic search, large facial recognition, and LPR in either large or small mode.

  large (ArcFace) is Fregata’s default for face recognition — a deliberate departure from upstream Frigate, which defaults to small. Face recognition runs only when a face is detected and large is far more accurate of the two embedding models. Choose small if you’d rather keep face recognition off the GPU entirely, or don’t need the extra accuracy.

• CPU, same as upstream — the models Frigate runs on the CPU everywhere: bird, audio detection, small face recognition, and the small speech-to-text model. These are small / TFLite models with no GPU path upstream either, so they run on the CPU like in all versions of Frigate.

One model runs on the CPU for now

The large (Whisper) audio transcription model hasn’t made the move to the GPU yet, so it still runs on the CPU. On a GPU-equipped Frigate it uses the GPU, so it’s slower on Fregata for now — but it isn’t a hot path (it runs only when requested, not automatically), so the real-world impact is small.

This list used to name four models. After Fregata’s CoreML inference pipeline rewrite in v0.17.2.5 and v0.18.0-beta3, the Semantic Search, Facial Recognition, and LPR AI Enrichments are now able to run 100% on the GPU too. Whisper large is the one still left.

When models download

Object detection and audio detection ship inside the app, so they work offline the moment you install Fregata. The other models download automatically the first time you turn on the feature that needs them, into Fregata’s model cache — a one-time download per model. Custom-model training additionally fetches TensorFlow on first use (a larger, one-time download); the model you train then runs on the CPU like any other.

First-run GPU compiles

Every model that lands on the GPU in the table above is also compiled for Apple’s CoreML the first time it’s loaded, then cached — later launches reuse the compiled result, and it’s only rebuilt if a future Fregata update changes the CoreML compiler underneath it. Most of these compiles finish in a few seconds and you won’t notice them.

Semantic search’s jinav2 model, at model_size: large, is the exception. Compiling it for the GPU the first time you enable jinav2 can use up to 10 GB of RAM and take up to about 10 minutes. During that window the process is pinned to a CPU core compiling the graph: there’s no GPU activity and no progress indicator, so it can look hung. It isn’t — give it time. This is a one-time cost tied to the Fregata version you’re running, not something that happens on every restart. If you don’t specifically need jinav2’s multilingual embeddings, the default jinav1 model gets you GPU-accelerated semantic search for a much smaller download and no first-run spike. See System requirements → Enrichment models for the full size breakdown.

jinav2 at model_size: small is not recommended

Switching to small avoids the compile spike above, but it isn’t a cheap alternative — it’s much slower than either jinav1 large or jinav2 large. int8-quantized models can’t reach CoreML at all (see the table above) — so every call pays a massive CPU cost with zero GPU acceleration. It’s also far less accurate. If you want jinav2 specifically, large is the size to use — the one-time compile cost is real, but what comes out the other side is fast. If you just want fast semantic search without the compile spike, stay on the default jinav1.

The first inference after each restart is slower too

Expected — no action needed

The very first semantic search, face match, or license-plate read after Fregata starts (or restarts) is often noticeably — sometimes tens of times — slower than normal. It’s a one-off delay, not a sign of a problem, and it clears itself after a call or two.

This is a different cost than the one-time GPU compile above, and unlike that one it isn’t cached — it happens again on every restart. When any of these models serves its first real request in a fresh Frigate process, it’s paying for work a busy model never has to redo: the CoreML/GPU session finishing whatever it deferred past session construction, plus one-time Python-side setup in the surrounding code (decoding the first image, initializing a tokenizer or feature extractor). None of that recurs once it’s done, so every call after the first one or two runs at full, benchmarked speed for the rest of that run.

Face recognition needs one extra restart the first time

The first time you enable face recognition, its models — including a small landmark-detection file — download in the background while Frigate keeps running. If a face is detected before that download finishes, face recognition can end up permanently unable to recognize anyone for the rest of that run: faces are still detected, but nothing is ever matched, even after the download completes. Restart Fregata once, after enabling face recognition and letting the initial download finish, and it works normally from then on. This only affects the very first time you turn the feature on — every later start already has the models on disk and is unaffected. See Troubleshooting → Face recognition never recognizes anyone after enabling it if you’re seeing the symptoms after the fact.