Apple Intelligence GenAI

Fregata can use the on-device Apple Intelligence model built into macOS as a GenAI provider for object descriptions, review item descriptions and chat. There is no server to run, no API key and no per-request cost, and the images from your cameras never leave the Mac. It is Fregata-exclusive: upstream Frigate has no apple_intelligence provider.

Not in Fregata 0.18.0 or earlier

Releases up to and including 0.18.0 do not recognize provider: apple_intelligence. It arrives in 0.18.0.1, and it needs macOS 27 or later (see What you need).

What it can do

GenAI role	Supported	What you get
descriptions	Yes	Text descriptions of tracked objects (shown in Explore) and descriptions and summaries of review items. Both send camera images to the model.
chat	Yes	The Chat page. The model runs Frigate’s chat tools itself, so it can search your tracked objects, check camera state and use the rest of the tools Frigate offers to any chat provider.
embeddings	No	Apple provides no embeddings model that Frigate’s semantic search can use. Keep semantic search on its own model.

Fregata always uses Apple’s on-device model. It never opts into Private Cloud Compute or any third-party model backend, so nothing about your cameras is sent off the Mac. Fregata also asks macOS for Apple’s permissive content-transformation guardrail setting instead of the default one, so that plain descriptions of people, vehicles and behavior are refused less often. That is not configurable, and Apple’s acceptable use requirements for the Foundation Models framework still apply.

What you need

• macOS 27 or later on an Apple Silicon Mac. Apple added image input to its on-device model in macOS 27, and descriptions need it.
• Apple Intelligence turned on in macOS System Settings, with its model finished downloading.
• A Fregata release version at least 0.18.0.1. Nothing else to install.

What Fregata does on a Mac that falls short:

• Below macOS 27: an apple_intelligence entry behaves exactly as if it were not configured. Nothing breaks, nothing is retried, and Fregata logs one warning when it loads the config, for example GenAI config 'apple' uses provider 'apple_intelligence', which requires macOS 27 or later. In the web UI the option is greyed out and labelled “requires macOS 27 or later”. A value already saved in config.yml stays displayed and selected, but you cannot pick it fresh.
• macOS 27, but Apple Intelligence is not ready: Fregata logs Apple Intelligence is not available on this Mac, followed by the reason macOS gave. It retries on its own, at most once a minute and only when a description or chat request needs it, so you do not have to restart Fregata after turning Apple Intelligence on or after the download finishes.

Apple chooses the model, and it depends on your Mac

There is no model to pick. Apple Intelligence is a single system-managed model, and Fregata does not choose, download or update it. macOS decides which on-device model your Mac runs, and Apple selects it based on the Mac’s hardware. That has practical consequences:

• No model setting. The Model field is greyed out in the web UI, and model: is ignored in config.yml, as are api_key and base_url.
• Results differ between Macs. Two Macs with identical configuration can produce different descriptions, at different speeds, from a model with different capabilities. Try it on the Mac that will actually run Fregata, not on another one.
• The context window is small and can vary. It holds the instructions, the prompt, every image and the model’s answer together, and it is far smaller than a cloud model’s. Fregata reads the real size from macOS instead of assuming one. When a review description would not fit, Fregata sends fewer frames, spread evenly across the clip and always including the first and last. A Mac whose model has a smaller window describes fewer frames. Chat drops the oldest messages first.
• Updates can change it. A macOS update can change the model Apple ships, so descriptions may shift after an update.
• Availability is Apple’s call. Whether Apple Intelligence can run at all on your Mac is decided by Apple and macOS. If macOS reports it unavailable, Fregata cannot work around it.

Not a like-for-like replacement for a large model

Descriptions from a small on-device model are usually shorter and less detailed than those from a large cloud model or a big local model, and the structured review output is less reliable (see Limitations). If description quality matters more than keeping everything on the Mac, compare a few real events against your current provider first.

Set up the provider

Adding the provider takes a few clicks in Settings, or one block in config.yml. A change saved in Settings applies at once. After editing config.yml by hand, restart Fregata.

Settings UI

1. In the web UI, go to Settings → Enrichments → Generative AI.
2. Click Add, then enter a Provider name of your choice, for example apple. Use letters, numbers, _ and -.
3. Set Provider to apple_intelligence. On a Mac below macOS 27 this option is greyed out and reads “apple_intelligence (requires macOS 27 or later)”. Once selected, API key and Model are greyed out too, because Apple Intelligence does not use them.
4. Under Roles, switch on Descriptions and Chat. Leave Embedding off (once Fregata has loaded the entry, Apple Intelligence does not offer it at all).
5. Click Save.

YAML

genai:
  apple:                          # the name is up to you
    provider: apple_intelligence
    roles:                        # always list roles; embeddings is not supported
      - descriptions
      - chat

model, api_key and base_url are not used by this provider, so leave them out.

Choose the roles explicitly

Each role can belong to only one provider. In the web UI a role that another provider already holds is greyed out. In config.yml, always list roles: an entry with no roles gets every role, including embeddings, and Fregata refuses to load the config with GenAI role 'embeddings' is assigned to both ... if another provider already has it. If you use semantic search with a GenAI provider, keep that on the other provider. See Frigate’s GenAI configuration docs for how roles work.

Chat needs nothing further. Descriptions are a separate opt-in for each kind, exactly as with any other GenAI provider. Turn on the ones you want next.

Turn on object descriptions

Object descriptions appear on tracked objects in Explore. Turn them on for one camera, or for every camera at once.

Settings UI

1. In the web UI, go to Settings → Camera configuration → Objects and pick the camera in the selector at the top. To turn descriptions on for every camera instead, go to Settings → Global configuration → Objects.
2. Under Advanced Settings, open GenAI object config.
3. Turn on Enable GenAI.
4. Optional: under GenAI objects, choose the object types to describe. Leave it empty to describe every tracked object.
5. Click Save.

If no provider has the Descriptions role, the page shows a warning.

YAML

cameras:
  front_door:
    objects:
      genai:
        enabled: true
        objects:                  # optional; empty or omitted means every tracked object
          - person

The same objects.genai keys work at the top level (objects:) to set the default for every camera.

Prompts, triggers and required zones work exactly as they do with any other provider. They are covered in Frigate’s object description docs.

Turn on review descriptions

Review descriptions add a description and summary to each review item. Alerts are described by default once this is on, and detections are opt-in.

Settings UI

1. In the web UI, go to Settings → Camera configuration → Review and pick the camera in the selector at the top. To turn descriptions on for every camera instead, go to Settings → Global configuration → Review.
2. Under GenAI config, turn on Enable GenAI descriptions.
3. Enable GenAI for alerts is on by default. Turn on Enable GenAI for detections to describe detections as well.
4. Optional: leave Review image source on preview. recordings sends sharper frames that use more of the small on-device context window.
5. Click Save.

If no provider has the Descriptions role, the page shows a warning.

YAML

cameras:
  front_door:
    review:
      genai:
        enabled: true
        alerts: true              # the default
        detections: false         # true to describe detections as well
        image_source: preview     # the default; recordings uses more of the context window

The same review.genai keys work at the top level (review:) to set the default for every camera.

Frigate’s review summary docs cover the remaining options.

Confirm it is working

• Object descriptions: when a tracked object of an enabled label ends, open it in Explore. The description appears with the object’s details. You can also regenerate it from there.
• Review descriptions: when an alert ends, its review item gets a description and summary.
• Chat: open the Chat page and ask about your cameras, for example “What happened at the front door today?”. The reply appears once the model has finished.
• The log: Apple Intelligence problems are logged as warnings in ~/Fregata/logs/frigate/current. See Troubleshooting Apple Intelligence.

Apple Intelligence provider options

These generation options tune the model. runtime_options accepts the same keys as provider_options and wins if a key appears in both. An invalid value is logged as a warning and ignored, and generation carries on with the defaults instead of failing.

Key	Type	Applies to	Description
temperature	number, 0 or higher	Descriptions and chat	How varied the output is. Lower is more repeatable.
maximum_response_tokens	integer, above 0	Descriptions and chat	Upper bound on the length of the answer. Fregata also reserves this much of the context window for the answer when it decides how many images fit. If unset, it reserves room for about 300 tokens.
sampling.mode	greedy or random	Descriptions and chat	greedy always takes the most likely next token, which is the most repeatable. random samples.
sampling.top	integer	Descriptions and chat	random mode only. Sample only from this many of the most likely tokens. Cannot be combined with probability_threshold.
sampling.probability_threshold	number	Descriptions and chat	random mode only. Sample from the most likely tokens whose combined probability reaches this value. Cannot be combined with top.
sampling.seed	integer	Descriptions and chat	random mode only. Fix the seed to make sampling repeatable.
instructions	string	Descriptions only	A persistent system-level prompt sent with every description request. It counts against the small context window, so keep it short. Chat takes its instructions from Frigate’s own chat prompt.

To set them, put the keys under Provider options (or provider_options) on your Apple Intelligence entry:

Settings UI

1. In the web UI, go to Settings → Enrichments → Generative AI and find your Apple Intelligence entry.

2. Expand Advanced. Provider options is a YAML box: type the options as a mapping, for example:

   temperature: 0.2
   maximum_response_tokens: 400
   instructions: "You describe security camera footage. Be brief and factual."

3. Click Save.

Runtime options, in the same Advanced group, works the same way and wins if a key is set in both.

YAML

genai:
  apple:
    provider: apple_intelligence
    roles:
      - descriptions
      - chat
    provider_options:
      temperature: 0.2
      maximum_response_tokens: 400
      instructions: "You describe security camera footage. Be brief and factual."

To make descriptions as repeatable as possible, use greedy sampling:

Settings UI

1. On the same Provider options box (Settings → Enrichments → Generative AI, under Advanced), enter:

   sampling:
     mode: greedy

2. Click Save.

YAML

genai:
  apple:
    provider: apple_intelligence
    roles:
      - descriptions
      - chat
    provider_options:
      sampling:
        mode: greedy

Limitations

• No embeddings, so semantic search cannot use this provider.
• Review output is best-effort structured. Review descriptions need structured output, and Fregata asks for it in the prompt rather than having the framework enforce it, so the model sometimes returns thinner or oddly shaped results than a large cloud model does. Fregata repairs one common slip (a list returned as a single string) before storing the result.
• Chat does not stream. The model finishes the whole answer first, then the Chat page shows it in pieces, so a long answer can sit on “Processing…” before any text appears.
• Chat tools run out of sight. The model calls Frigate’s tools itself, so the Chat page does not list which tools were used. Their results reach the model as text only, so a tool that would normally attach a live camera frame, such as the live-context tool, does not attach one.
• Chat only sees the current image. Images from earlier in a conversation are replaced by an “image omitted” note, and when a long conversation fills the context window the oldest messages are dropped first.
• Every request is a fresh session, with a 120 second limit.
• The Apple SDK is new. Fregata pins a specific apple-fm-sdk version and carries workarounds for known problems in it. Treat descriptions as best-effort and report anything odd on GitHub Discussions.

Troubleshooting Apple Intelligence

Look for these in ~/Fregata/logs/frigate/current. See Where the logs live for other ways to open it.

Log message	What it means	What to do
GenAI config '<name>' uses provider 'apple_intelligence', which requires macOS 27 or later	This Mac is below macOS 27, so the provider is not being used.	Upgrade macOS, or use another provider.
Apple Intelligence is not available on this Mac (<reason>)	macOS says Apple Intelligence is not ready. The reason is in the brackets.	Turn on Apple Intelligence in System Settings and let its model finish downloading. Fregata retries on its own within about a minute of the next request. If macOS says the Mac cannot run it at all, use another provider.
Apple Intelligence provider selected, but apple-fm-sdk is not installed	The Apple SDK is missing from this Fregata build.	Update Fregata to a release that includes the provider. If you already have one, please report it.
Apple Intelligence request timed out after 120 s	Generation took too long and was abandoned. The Mac may still be finishing it in the background.	Retry. If it keeps happening, reduce load on the Mac or send fewer images (for review descriptions, keep Review image source on preview, see Turn on review descriptions).
Apple Intelligence request exceeded the on-device context window even at minimum size	The text alone does not fit, even with no images.	Shorten instructions (see Provider options) and any custom GenAI prompts.
GenAI role 'embeddings' is assigned to both ...	The entry has the default roles, and another provider already has embeddings.	Set Roles (roles) on the Apple Intelligence entry to Descriptions and Chat only. See Set up the provider.
Apple Intelligence chat generation failed after N tool call(s) ... Retrying with tool results folded into the prompt as plain text	Informational. The model struggled after several tool calls, and Fregata retried once without tools.	Nothing. The answer may be less detailed. If the retry also fails, ask again with a simpler question.