# Apple Intelligence GenAI

Fregata can use the **on-device Apple Intelligence model** built into macOS as a GenAI
provider for object descriptions, review item descriptions and chat. There is no server to
run, no API key and no per-request cost, and the images from your cameras never leave the
Mac. It is **Fregata-exclusive**: upstream Frigate has no `apple_intelligence` provider.

:::note[Not in Fregata 0.18.0 or earlier]
Releases up to and including 0.18.0 do not recognize `provider: apple_intelligence`. It
arrives in 0.18.0.1, and it needs **macOS 27 or later** (see
[What you need](#what-you-need)).
:::

## What it can do

| GenAI role | Supported | What you get |
|---|---|---|
| `descriptions` | Yes | Text descriptions of tracked objects (shown in Explore) and descriptions and summaries of review items. Both send camera images to the model. |
| `chat` | Yes | The Chat page. The model runs Frigate's chat tools itself, so it can search your tracked objects, check camera state and use the rest of the tools Frigate offers to any chat provider. |
| `embeddings` | **No** | Apple provides no embeddings model that Frigate's semantic search can use. Keep semantic search on its own model. |

Fregata always uses Apple's **on-device** model. It never opts into Private Cloud Compute or
any third-party model backend, so nothing about your cameras is sent off the Mac. Fregata also
asks macOS for Apple's permissive content-transformation guardrail setting instead of the
default one, so that plain descriptions of people, vehicles and behavior are refused less
often. That is not configurable, and Apple's acceptable use requirements for the Foundation
Models framework still apply.

## What you need

- **macOS 27 or later** on an Apple Silicon Mac. Apple added image input to its on-device
  model in macOS 27, and descriptions need it.
- **Apple Intelligence turned on** in macOS System Settings, with its model finished downloading.
- **A Fregata release version at least 0.18.0.1.** Nothing else to install.

What Fregata does on a Mac that falls short:

- **Below macOS 27:** an `apple_intelligence` entry behaves exactly as if it were not
  configured. Nothing breaks, nothing is retried, and Fregata logs one warning when it loads
  the config, for example `GenAI config 'apple' uses provider 'apple_intelligence', which
  requires macOS 27 or later`. In the web UI the option is greyed out and labelled
  "requires macOS 27 or later". A value already saved in `config.yml` stays displayed and
  selected, but you cannot pick it fresh.
- **macOS 27, but Apple Intelligence is not ready:** Fregata logs `Apple Intelligence is not
  available on this Mac`, followed by the reason macOS gave. It retries on its own, at most
  once a minute and only when a description or chat request needs it, so you do not have to
  restart Fregata after turning Apple Intelligence on or after the download finishes.

## Apple chooses the model, and it depends on your Mac

There is no model to pick. Apple Intelligence is a single system-managed model, and Fregata
does not choose, download or update it. **macOS decides which on-device model your Mac runs,
and Apple selects it based on the Mac's hardware.** That has practical consequences:

- **No model setting.** The **Model** field is greyed out in the web UI, and `model:` is
  ignored in `config.yml`, as are `api_key` and `base_url`.
- **Results differ between Macs.** Two Macs with identical configuration can produce
  different descriptions, at different speeds, from a model with different capabilities. Try it
  on the Mac that will actually run Fregata, not on another one.
- **The context window is small and can vary.** It holds the instructions, the prompt, every
  image and the model's answer together, and it is far smaller than a cloud model's. Fregata
  reads the real size from macOS instead of assuming one. When a review description would not
  fit, Fregata sends fewer frames, spread evenly across the clip and always including the
  first and last. A Mac whose model has a smaller window describes fewer frames. Chat drops
  the oldest messages first.
- **Updates can change it.** A macOS update can change the model Apple ships, so descriptions
  may shift after an update.
- **Availability is Apple's call.** Whether Apple Intelligence can run at all on your Mac is
  decided by Apple and macOS. If macOS reports it unavailable, Fregata cannot work around it.

:::caution[Not a like-for-like replacement for a large model]
Descriptions from a small on-device model are usually shorter and less detailed than those
from a large cloud model or a big local model, and the structured review output is less
reliable (see [Limitations](#limitations)). If description quality matters more than keeping
everything on the Mac, compare a few real events against your current provider first.
:::

## Set up the provider

Adding the provider takes a few clicks in Settings, or one block in `config.yml`. A change
saved in Settings applies at once. After editing `config.yml` by hand, restart Fregata.

**In the Settings UI:**

1. In the web UI, go to **Settings → Enrichments → Generative AI**.
2. Click **Add**, then enter a **Provider name** of your choice, for example `apple`. Use
   letters, numbers, `_` and `-`.
3. Set **Provider** to `apple_intelligence`. On a Mac below macOS 27 this option is greyed
   out and reads "apple_intelligence (requires macOS 27 or later)". Once selected, **API
   key** and **Model** are greyed out too, because Apple Intelligence does not use them.
4. Under **Roles**, switch on **Descriptions** and **Chat**. Leave **Embedding** off (once
   Fregata has loaded the entry, Apple Intelligence does not offer it at all).
5. Click **Save**.

**In config.yml:**

```yaml
genai:
  apple:                          # the name is up to you
    provider: apple_intelligence
    roles:                        # always list roles; embeddings is not supported
      - descriptions
      - chat
```

`model`, `api_key` and `base_url` are not used by this provider, so leave them out.

:::caution[Choose the roles explicitly]
Each role can belong to only one provider. In the web UI a role that another provider already
holds is greyed out. In `config.yml`, always list `roles`: an entry with no `roles` gets every
role, including `embeddings`, and Fregata refuses to load the config with `GenAI role
'embeddings' is assigned to both ...` if another provider already has it. If you use semantic
search with a GenAI provider, keep that on the other provider. See Frigate's
[GenAI configuration docs](https://docs.frigate.video/configuration/genai/genai_config) for
how roles work.
:::

Chat needs nothing further. Descriptions are a separate opt-in for each kind, exactly as with
any other GenAI provider. Turn on the ones you want next.

## Turn on object descriptions

Object descriptions appear on tracked objects in Explore. Turn them on for one camera, or for
every camera at once.

**In the Settings UI:**

1. In the web UI, go to **Settings → Camera configuration → Objects** and pick the camera
   in the selector at the top. To turn descriptions on for every camera instead, go to
   **Settings → Global configuration → Objects**.
2. Under **Advanced Settings**, open **GenAI object config**.
3. Turn on **Enable GenAI**.
4. Optional: under **GenAI objects**, choose the object types to describe. Leave it empty to
   describe every tracked object.
5. Click **Save**.

If no provider has the **Descriptions** role, the page shows a warning.

**In config.yml:**

```yaml
cameras:
  front_door:
    objects:
      genai:
        enabled: true
        objects:                  # optional; empty or omitted means every tracked object
          - person
```

The same `objects.genai` keys work at the top level (`objects:`) to set the default for every
camera.

Prompts, triggers and required zones work exactly as they do with any other provider. They are
covered in Frigate's
[object description docs](https://docs.frigate.video/configuration/genai/genai_objects).

## Turn on review descriptions

Review descriptions add a description and summary to each review item. Alerts are described
by default once this is on, and detections are opt-in.

**In the Settings UI:**

1. In the web UI, go to **Settings → Camera configuration → Review** and pick the camera
   in the selector at the top. To turn descriptions on for every camera instead, go to
   **Settings → Global configuration → Review**.
2. Under **GenAI config**, turn on **Enable GenAI descriptions**.
3. **Enable GenAI for alerts** is on by default. Turn on **Enable GenAI for detections** to
   describe detections as well.
4. Optional: leave **Review image source** on `preview`. `recordings` sends sharper frames that
   use more of the small on-device context window.
5. Click **Save**.

If no provider has the **Descriptions** role, the page shows a warning.

**In config.yml:**

```yaml
cameras:
  front_door:
    review:
      genai:
        enabled: true
        alerts: true              # the default
        detections: false         # true to describe detections as well
        image_source: preview     # the default; recordings uses more of the context window
```

The same `review.genai` keys work at the top level (`review:`) to set the default for every
camera.

Frigate's [review summary docs](https://docs.frigate.video/configuration/genai/genai_review)
cover the remaining options.

## Confirm it is working

- **Object descriptions:** when a tracked object of an enabled label ends, open it in
  **Explore**. The description appears with the object's details. You can also regenerate it
  from there.
- **Review descriptions:** when an alert ends, its review item gets a description and summary.
- **Chat:** open the **Chat** page and ask about your cameras, for example "What happened at
  the front door today?". The reply appears once the model has finished.
- **The log:** Apple Intelligence problems are logged as warnings in
  `~/Fregata/logs/frigate/current`. See
  [Troubleshooting Apple Intelligence](#troubleshooting-apple-intelligence).

## Apple Intelligence provider options

These generation options tune the model. `runtime_options` accepts the same keys as
`provider_options` and wins if a key appears in both. An invalid value is logged as a warning
and ignored, and generation carries on with the defaults instead of failing.

| Key | Type | Applies to | Description |
|---|---|---|---|
| `temperature` | number, 0 or higher | Descriptions and chat | How varied the output is. Lower is more repeatable. |
| `maximum_response_tokens` | integer, above 0 | Descriptions and chat | Upper bound on the length of the answer. Fregata also reserves this much of the context window for the answer when it decides how many images fit. If unset, it reserves room for about 300 tokens. |
| `sampling.mode` | `greedy` or `random` | Descriptions and chat | `greedy` always takes the most likely next token, which is the most repeatable. `random` samples. |
| `sampling.top` | integer | Descriptions and chat | `random` mode only. Sample only from this many of the most likely tokens. Cannot be combined with `probability_threshold`. |
| `sampling.probability_threshold` | number | Descriptions and chat | `random` mode only. Sample from the most likely tokens whose combined probability reaches this value. Cannot be combined with `top`. |
| `sampling.seed` | integer | Descriptions and chat | `random` mode only. Fix the seed to make sampling repeatable. |
| `instructions` | string | Descriptions only | A persistent system-level prompt sent with every description request. It counts against the small context window, so keep it short. Chat takes its instructions from Frigate's own chat prompt. |

To set them, put the keys under **Provider options** (or `provider_options`) on your Apple
Intelligence entry:

**In the Settings UI:**

1. In the web UI, go to **Settings → Enrichments → Generative AI** and find your Apple
   Intelligence entry.
2. Expand **Advanced**. **Provider options** is a YAML box: type the options as a mapping, for
   example:

   ```text
   temperature: 0.2
   maximum_response_tokens: 400
   instructions: "You describe security camera footage. Be brief and factual."
   ```

3. Click **Save**.

**Runtime options**, in the same **Advanced** group, works the same way and wins if a key is
set in both.

**In config.yml:**

```yaml
genai:
  apple:
    provider: apple_intelligence
    roles:
      - descriptions
      - chat
    provider_options:
      temperature: 0.2
      maximum_response_tokens: 400
      instructions: "You describe security camera footage. Be brief and factual."
```

To make descriptions as repeatable as possible, use greedy sampling:

**In the Settings UI:**

1. On the same **Provider options** box (**Settings → Enrichments → Generative AI**, under
   **Advanced**), enter:

   ```text
   sampling:
     mode: greedy
   ```

2. Click **Save**.

**In config.yml:**

```yaml
genai:
  apple:
    provider: apple_intelligence
    roles:
      - descriptions
      - chat
    provider_options:
      sampling:
        mode: greedy
```

## Limitations

- **No embeddings,** so semantic search cannot use this provider.
- **Review output is best-effort structured.** Review descriptions need structured output,
  and Fregata asks for it in the prompt rather than having the framework enforce it, so the
  model sometimes returns thinner or oddly shaped results than a large cloud model does.
  Fregata repairs one common slip (a list returned as a single string) before storing the
  result.
- **Chat does not stream.** The model finishes the whole answer first, then the Chat page
  shows it in pieces, so a long answer can sit on "Processing..." before any text appears.
- **Chat tools run out of sight.** The model calls Frigate's tools itself, so the Chat page
  does not list which tools were used. Their results reach the model as text only, so a tool
  that would normally attach a live camera frame, such as the live-context tool, does not
  attach one.
- **Chat only sees the current image.** Images from earlier in a conversation are replaced by
  an "image omitted" note, and when a long conversation fills the context window the oldest
  messages are dropped first.
- **Every request is a fresh session,** with a 120 second limit.
- **The Apple SDK is new.** Fregata pins a specific `apple-fm-sdk` version and carries
  workarounds for known problems in it. Treat descriptions as best-effort and report anything
  odd on [GitHub Discussions](https://github.com/3rdBitLabs/Fregata/discussions).

## Troubleshooting Apple Intelligence

Look for these in `~/Fregata/logs/frigate/current`. See
[Where the logs live](/guides/troubleshooting/#where-the-logs-live) for other ways to open it.

| Log message | What it means | What to do |
|---|---|---|
| `GenAI config '<name>' uses provider 'apple_intelligence', which requires macOS 27 or later` | This Mac is below macOS 27, so the provider is not being used. | Upgrade macOS, or use another provider. |
| `Apple Intelligence is not available on this Mac (<reason>)` | macOS says Apple Intelligence is not ready. The reason is in the brackets. | Turn on Apple Intelligence in System Settings and let its model finish downloading. Fregata retries on its own within about a minute of the next request. If macOS says the Mac cannot run it at all, use another provider. |
| `Apple Intelligence provider selected, but apple-fm-sdk is not installed` | The Apple SDK is missing from this Fregata build. | Update Fregata to a release that includes the provider. If you already have one, please report it. |
| `Apple Intelligence request timed out after 120 s` | Generation took too long and was abandoned. The Mac may still be finishing it in the background. | Retry. If it keeps happening, reduce load on the Mac or send fewer images (for review descriptions, keep **Review image source** on `preview`, see [Turn on review descriptions](#turn-on-review-descriptions)). |
| `Apple Intelligence request exceeded the on-device context window even at minimum size` | The text alone does not fit, even with no images. | Shorten `instructions` (see [Provider options](#apple-intelligence-provider-options)) and any custom GenAI prompts. |
| `GenAI role 'embeddings' is assigned to both ...` | The entry has the default roles, and another provider already has `embeddings`. | Set **Roles** (`roles`) on the Apple Intelligence entry to Descriptions and Chat only. See [Set up the provider](#set-up-the-provider). |
| `Apple Intelligence chat generation failed after N tool call(s) ... Retrying with tool results folded into the prompt as plain text` | Informational. The model struggled after several tool calls, and Fregata retried once without tools. | Nothing. The answer may be less detailed. If the retry also fails, ask again with a simpler question. |