Skip to main content

Bring Your Own LLM: Endpoint Access and Fallback

Bring Your Own LLM lets you configure an OpenAI-compatible base URL, model name and API key for classification. Model choice does not change where the application itself runs.

Configure and test the endpoint

Use the provider’s documented base URL and exact model identifier. The application appends /chat/completions to that base URL. Test the connection in Settings with a synthetic input before introducing mailbox data.

Local Ollama exposes an API such as http://localhost:11434/v1. That address works only from a process that can reach the same loopback interface. The hosted service cannot reach your laptop’s localhost, and the current production URL guard rejects private and loopback addresses in custom endpoint settings. Do not expose an unauthenticated model server as a workaround.

Set it up step by step

  1. Open Settings in the main navigation. The section LLM Backend shows the Active model.
  2. Open Use a custom LLM backend.
  3. Enter the base URL in Custom LLM API URL. It must start with http:// or https://.
  4. Enter the API Key if your endpoint needs one. It is sent as a bearer token with each request. After saving, the key is no longer displayed in full.
  5. Enter the exact identifier in Model Name. If you leave it empty, the name default is sent.
  6. Click Test Connection. The test uses the values currently in the form, so you can test before saving. It sends one fixed test prompt with no mailbox content and waits up to 30 seconds. Connection Successful means the endpoint answered in the OpenAI chat format; otherwise you see Connection Failed with the error, and Show raw response displays what came back.
  7. Click Save Settings. The setting belongs to your user account and applies to all mailboxes you have connected. To return to the default model, empty the URL field and save.

The test checks only the endpoint in the form. It does not exercise the fallback described below.

A worked example

A fictional provider documents the base URL https://llm.example-provider.com/v1 and the model acme-small-instruct. Enter exactly these two values. The application then calls https://llm.example-provider.com/v1/chat/completions.

A common mistake is to paste the full path including /chat/completions. The suffix is then added a second time, and the test is likely to fail. Remove the suffix and test again.

What your endpoint receives

For classification the endpoint gets one chat request per decision. The prompt contains the sender, the subject, a preview of up to 500 characters of the message text, spam and authentication headers if your mail server added them, the description and domain of the mailbox, and your category names and descriptions. Attachments are not included.

The same endpoint is used when a sender is classified and when text is written for AI forwarding. For forwarding, up to 6,000 characters of the message text can be sent.

The answer has to name one of your categories exactly, as a short JSON object. If it cannot be read that way, the message is not assigned to a category and stays in the inbox. A model that ignores the format therefore leaves mail unsorted rather than filing it somewhere.

Check fallback as well as the selected model

The current code can attach the configured default model provider as a fallback for a custom endpoint. A failed custom request can therefore be attempted elsewhere. Selecting a model is not proof of exclusive use of that model.

For a requirement of local-only processing, verify the application deployment, chosen model, complete provider chain and observed outbound requests—including a synthetic outage test. A local endpoint can also serve Ollama cloud models, so check the actual model rather than relying on its URL.

Common questions

What happens when my endpoint is down or slow?

Timeouts, connection errors and temporary server errors are retried. When a fallback is attached, your endpoint gets fewer and shorter attempts before the request moves on to the fallback. If no endpoint answers, automatic processing stops at that message and tries it again on the next run. In a manual run the message is listed as failed and left untouched.

Can I use a model that runs on my own machine?

Only if the hosted service can reach it. The host name is checked when you save and when you test; names that resolve to private, loopback or other non-public addresses are rejected. A model server therefore needs a public address, and it should require authentication.

Is my model also used for forwarding summaries?

Yes. The text for AI forwarding is requested from the same endpoint, with the same fallback behavior.

Does a new model re-sort mail that was already handled?

No. A change applies to messages processed afterwards. Saved sender assignments also remain; see sender categories for when they are reused.

Review category quality after model changes

The per-message classifier uses sender, subject and a limited body preview. It chooses one category or none. Preview matching, excluded and ambiguous examples after changing models; this page does not provide measured accuracy rankings for providers.

See the Ollama guide, private sorting and sender categories.

Ready to put your inbox on autopilot?

Get Started Free