Agent Config

The agent section configures the LLM model, system prompt, tools, and behavior.

Basic Configuration

agent:
  model: anthropic/claude-sonnet-4-5
  system: system # References prompts/system.md
  tools: [get-user-account] # Available tools
  mcpServers: [figma, browser] # MCP server connections
  skills: [qr-code] # Available skills
  references: [api-guidelines] # On-demand context documents

Configuration Options

Models

Specify models in provider/model-id format, using a model from the model catalog. The same catalog is available programmatically from the Models API and the MCP list_models tool. It syncs with provider catalogs automatically, so new models appear shortly after release.

Supported Providers

Chat models outside the direct providers are reached through OpenRouter: openrouter/ plus OpenRouter's own model id. The catalog lists these routes, so copy one exactly as it's listed. A direct-provider chat model can also run through OpenRouter - for example on your own OpenRouter key - under the openrouter/ id the Models API returns as its openRouterId; copy that one exactly too, since OpenRouter's spelling can differ from the direct id (openrouter/anthropic/claude-opus-5.5 for anthropic/claude-opus-5-5). OpenRouter's variant ids with a : suffix (such as :free or :batch) and its ~ "latest" aliases aren't valid model ids. See OpenRouter routing for how these requests are routed.

The Vercel AI Gateway (vercel/...) is no longer a route: a protocol that names one fails validation with code RETIRED_MODEL_PROVIDER, and the message suggests the likely replacement - the direct id for the Anthropic, OpenAI, Google and SpaceXAI families, the OpenRouter id for everything else. The two gateways name some providers and models differently, so confirm the exact id in the model catalog.

Unsupported Models

Every model an agent uses must be one Octavus supports:

• Direct providers (Anthropic, Google, OpenAI, SpaceXAI, Octavus) - chat and image models must come from the model catalog, and video, speech, and transcription models from the tables in Generating video, Speech Generation, and Transcription.
• OpenRouter routes (openrouter/...) - chat models must come from the model catalog as well, which lists every model OpenRouter does, plus the routers Octavus has verified. The catalog syncs with OpenRouter's model list every few hours, so a model OpenRouter has just added becomes available after the next sync - or right away on your own OpenRouter key, which can run any OpenRouter model. Image, video, speech, and transcription models are always called on their provider directly, so they can't use an OpenRouter route.

A model outside the catalog or those tables, or an unknown provider, is rejected before it runs:

• At validation and deploy - a protocol that names one (including as an input's default) fails validation with code MODEL_NOT_SUPPORTED. An OpenRouter model the catalog doesn't list passes with a MODEL_NOT_IN_CATALOG warning instead when the project or organization has its own OpenRouter key.
• At runtime - a session whose model resolves to one (for example from a MODEL input) fails with a not_found_error stream error whose code is MODEL_NOT_SUPPORTED. The message links to the catalog. An unsupported video, speech, or transcription model fails its tool call or block with the same code.

Model Deprecations

Providers retire models on dates they announce in advance. When a provider announces a model's shutdown, the model catalog records it with the shutdown date and, when the provider names one, the replacement. A deprecation is a notice, not a restriction - it never changes what runs:

• Before the date, the model keeps working. Validation (on save, on deploy, in octavus validate, and in the MCP validate_platform_agent tool) passes a protocol that names it with a MODEL_DEPRECATED warning that gives the date and the replacement, and the agent editor and your project's agent list show the same warning without a re-save. The Models API and the MCP list_models tool return it as the model's deprecation, and the pricing page shows the date.
• From the date, the provider rejects requests to the model with a not_found_error. An agent with a backup model fails over to it on every step; an agent without one fails the turn. Once the model is gone from the catalog, a protocol that still names it fails validation with MODEL_NOT_SUPPORTED.

A deprecation covers every way of naming the model, including its OpenRouter route (openrouter/anthropic/... next to anthropic/...). A dated snapshot and its alias (gpt-4o-2024-08-06 and gpt-4o) are separate models with separate deprecations.

Move to another model before the date, and declare a backupModel on a different provider so that an early or unannounced shutdown never stops your agent.

Examples

# Anthropic Claude 4.5
agent:
  model: anthropic/claude-sonnet-4-5

# Google Gemini 3
agent:
  model: google/gemini-3-flash-preview

# OpenAI GPT-5
agent:
  model: openai/gpt-5

# OpenAI reasoning models
agent:
  model: openai/o3-mini

# SpaceXAI Grok
agent:
  model: x-ai/grok-4.6

OpenRouter routing

OpenRouter serves most models from several hosts and, by default, load-balances every request across them. Octavus applies its own routing policy to every OpenRouter request - on platform keys and on your own OpenRouter key alike - so a long agentic session behaves predictably:

• One host per session. A session's requests carry a sticky routing key, so the whole session stays on one host (see Prompt Caching - the key is sent unless cache: off).
• Reasoning is honored or reported. A thinking level is sent in the shape the model publishes for reasoning - an effort by name, a token budget, or an on/off switch - and a request that asks for reasoning is routed only to hosts that support it, so a declared level is never silently ignored (see Thinking). If none of a model's hosts support reasoning, nothing is sent: validation warns (THINKING_UNSUPPORTED_MODEL) and the session's execution log records it as a thinking-adjusted entry.
• One endpoint per provider. Models from the major providers, open-weight and proprietary alike, run only on the provider's own endpoint, which Octavus has verified for tool calling, reasoning and tool-result vision. A model its provider doesn't serve on OpenRouter itself - an older release, or an open-weight model only other hosts run - runs on OpenRouter's default routing, like a model from any other provider.

Billing. On platform keys, a request through OpenRouter is billed the exact cost OpenRouter reports for it: the serving host's rate at that moment (off-peak discounts and long-context tiers included) and any work a router did inside the request. For a model that runs on its provider's own endpoint, the rate the model catalog shows is the most a request costs; a model on OpenRouter's default routing is billed what the host that served it charged, which can differ from the rate shown.

The step-stats entry the execution log records after every model step carries upstream, the OpenRouter host that served it, servedModel, the model that answered, and reportedCost, what OpenRouter charged for the step (see Debugging); an OpenRouter error carries the host on provider.upstream. A host rejecting an image inside a tool result surfaces as a validation_error with code HOST_UNSUPPORTED_TOOL_CONTENT, which the backup model takes over from. A host rejecting structured output (responseType) outright surfaces the same way, with code HOST_UNSUPPORTED_RESPONSE_FORMAT.

On your own OpenRouter key, your account's provider restrictions remain the ceiling: if they exclude the host Octavus routes a model to, OpenRouter rejects the request with a 404 that names the cause and the backup model takes over.

Routers

An OpenRouter router - such as openrouter/openrouter/auto or openrouter/typesafe/jev-router - picks the model that answers each request. Name one anywhere a model can be named, the backup model and worker threads included. The catalog lists the routers Octavus has verified for agent runs alongside the models they pick from; on platform keys only those run, while your own OpenRouter key can run any router OpenRouter offers.

• Where a step can land. A router picks freely among the models it knows, but only on the endpoints Octavus routes the major providers to - never on another host. A request carrying structured output (responseType) is kept off the endpoints that refuse it.
• Billing. A router has no price of its own: each step is billed at the cost OpenRouter reports for the model the router picked. The Models API lists a router with pricingBasis: "served-model" and no rates.
• Thinking. A fixed thinking level is sent as the router's reasoning effort (max as its deepest, xhigh) and fixes the effort for every step. thinking: auto leaves the effort to the router; off (or no thinking) turns reasoning off for the picked model.
• One conversation per session. A session, and each named thread in it, is one conversation to the router whatever the cache setting, so a tool loop normally stays on one model. The model can change between user turns, and a router can pick again mid-turn when a request needs a capability the current model lacks, such as seeing an image in a tool result.
• Latency and cost. A router can escalate a hard step by consulting a stronger model before answering, which can take minutes and cost far more than the step's visible tokens. Your spend limits apply to the exact cost as with any model.
• Data. The router's provider reads the conversation text to decide which model answers.
• Context. Before a router's first step, the session budgets context against a conservative default window; after, against the window of the model that answered.

When the router reports its decision, step-stats carries it as router: why it chose (reason, such as initial, continuation or escalation), the effort the model ran at, and any advisorModel it consulted.

Your own OpenRouter key

On a project's or organization's own OpenRouter key, an agent can name any OpenRouter model, including one the catalog doesn't list: OpenRouter reports the cost of every request, and your account decides what it can run. Validation passes such a model with a MODEL_NOT_IN_CATALOG warning, and it bills a bandwidth fee of $0.30 per 1M tokens when the catalog has no rate for it. On platform keys, only catalog models run.

Dynamic Model Selection

The model field can also reference an input variable, allowing consumers to choose the model when creating a session:

input:
  MODEL:
    type: string
    description: The LLM model to use

agent:
  model: MODEL # Resolved from session input
  system: system

When creating a session, pass the model:

const sessionId = await client.agentSessions.create('my-agent', {
  MODEL: 'anthropic/claude-sonnet-4-5',
});

This enables:

• Multi-provider support - Same agent works with different providers
• A/B testing - Test different models without protocol changes
• User preferences - Let users choose their preferred model

The model value is validated at runtime to ensure it's in the correct provider/model-id format.

Backup Model

Configure a fallback model that activates automatically when the primary model encounters a transient provider error (rate limits, outages, timeouts), or when its provider rejects the model itself (a model it has shut down, or one your account has lost access to):

agent:
  model: anthropic/claude-sonnet-4-5
  backupModel: openai/gpt-4o
  system: system

When a provider error occurs, the system retries once with the backup model. If the backup also fails, the original error is returned, with the backup's failure attached as backup on the error (error.backup.model, error.backup.message, error.backup.code) so you can see why the failover did not rescue the request - for example a backup whose provider account is out of credits.

Key behaviors:

• Only transient provider errors and a rejection of the model itself trigger fallback - authentication and validation errors are not retried
• For streaming responses, fallback only occurs if no content has been sent to the client yet - except for a step the primary model voided: a safety refusal (content-filter), an empty response (NO_CONTENT_GENERATED), or a generation cut off at the output limit (OUTPUT_TRUNCATED). Those produced no committed answer, so the partial step is retracted (a step-discard event) and the backup model re-runs it
• A backup that fails hands the request back to the primary model: the primary's own transient retries and every later step of the turn run on the model you configured, not on the backup that just failed. A backup that succeeds stays active for the rest of the turn

Like model, backupModel supports variable references:

input:
  BACKUP_MODEL:
    type: string
    description: Fallback model for provider errors

agent:
  model: anthropic/claude-sonnet-4-5
  backupModel: BACKUP_MODEL
  system: system

System Prompt

The system prompt sets the agent's persona and instructions. The input field controls which variables are available to the prompt - only variables listed in input are interpolated.

agent:
  system: system # Uses prompts/system.md
  input:
    - COMPANY_NAME
    - PRODUCT_NAME

Variables in input can come from protocol.input, protocol.resources, or protocol.variables.

Input Mapping Formats

# Array format (same name)
input:
  - COMPANY_NAME
  - PRODUCT_NAME

# Array format (rename)
input:
  - CONTEXT: CONVERSATION_SUMMARY  # Prompt sees CONTEXT, value comes from CONVERSATION_SUMMARY

# Object format (rename)
input:
  CONTEXT: CONVERSATION_SUMMARY

The left side (label) is what the prompt sees. The right side (source) is where the value comes from.

Example

prompts/system.md:

You are a friendly support agent for {{COMPANY_NAME}}.

## Your Role

Help users with questions about {{PRODUCT_NAME}}.

## Guidelines

- Be helpful and professional
- If you can't help, offer to escalate
- Never share internal information

Agentic Mode

Enable multi-step tool calling:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  tools: [get-user-account, search-docs, create-ticket]
  agentic: true # LLM can call multiple tools
  maxSteps: 10 # Limit cycles to prevent runaway

How it works:

1. LLM receives user message
2. LLM decides to call a tool
3. Tool executes, result returned to LLM
4. LLM decides if more tools needed
5. Repeat until LLM responds or maxSteps reached

Output Limits and Loop Guard

Two optional safeguards bound the cost of a single LLM generation. Both apply per generation (one agentic step), not across the whole run - maxSteps is what bounds how many steps a run can take.

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  maxOutputTokens: 32000 # hard cap on output tokens per generation
  loopGuard: true # abort a generation that degenerates into a repeated string

maxOutputTokens

Caps the output tokens of a single generation, passed straight through to the provider's max_tokens. When omitted, the provider/SDK default applies - it varies by model and can be very large. Setting an explicit value gives you a predictable, model-independent ceiling on the cost of any one generation.

This is a per-step cap, not a budget for the whole run. A long agentic run makes many generations, each capped independently; use maxSteps to bound the number of steps.

It is a safety ceiling, not a way to shape answer length. A generation that hits the cap (finishReason: 'length') is incomplete - a cut-off answer, a partial tool call, or reasoning that consumed the whole budget - so the step fails as a provider_error with code OUTPUT_TRUNCATED rather than delivering the truncated output as a finished turn. A configured backup model takes the step first; the error is not retried on the same model. The same holds for a step that ends with no response at all (no text and no tool call, for example a reasoning-only generation): it fails with code NO_CONTENT_GENERATED, which is retried a bounded number of times on the model that produced it - the primary, or the backup once it has taken over - and falls back to the backup, instead of completing the turn with an empty message. An empty response that survives those retries is returned as non-retryable: the same context produced the same empty step on every attempt, so re-running the turn would only repeat it.

A prompt that tells the agent to end without replying ("stay silent and end", "don't send a summary") produces exactly this empty step - the model complies by emitting nothing, and the turn fails. Ask for a brief final message instead ("end with one line saying what you did, even if nothing needed attention"), and scope silence to the channels you mean (no Slack message, no email), not to the agent's reply itself.

loopGuard

Autoregressive models can occasionally degenerate into a repetition loop - emitting the same short string thousands of times until the output budget is exhausted. Set loopGuard: true to watch the streamed output and abort a generation as soon as it detects a short unit repeating past a threshold, so a degenerate turn stops after a few hundred tokens instead of burning the full output cap. The runaway tail is trimmed out of the stored message so it does not pollute later turns.

It is a plain on/off flag - the detection thresholds (how many consecutive repeats trip it, the longest unit considered) are fixed, conservative defaults, since they describe a degeneration detector rather than anything worth tuning per agent. Omit loopGuard (or set it to false) to disable it.

Image Delivery Limits

Model providers impose image constraints that change over time - most notably a maximum dimension per image, and stricter caps once a request carries many images. Set maxImageDimension to cap the longest side (in pixels) of every image in the model's view:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  maxImageDimension: 2000 # downscale any image above 2000px on its longest side

• On every request, an image whose longest side exceeds the cap is delivered downscaled to fit (aspect ratio preserved). The downscaled copy is made once and reused for the rest of the session, so a long session does not re-process its images on every request.
• The model is told. A downscaled image arrives with a short note giving the size it is shown at, its original size, and the limit - for example [chart.png is shown at 1265x2000, downscaled from 1850x2925 to fit the 2000px image limit] - so the agent knows its view of that image is reduced.
• An image the cap cannot fit is delivered by reference. When an over-cap image is too large to downscale, or its image data does not decode, it is never forwarded past the cap: in its place the model receives a short factual note with the image's dimensions, the limit, and its download URL - for example [capture.jpg (image/jpeg, 29600x15456) is not shown: it exceeds the 2000px image limit and it is too large to downscale. Download: https://...] - so the agent knows the image exists and can fetch it with its own tools. The turn continues either way.
• The cap only touches the images it actually changes. An image already within it is delivered exactly as it would be with no cap set - same bytes, same delivery path, no re-encoding and no quality change - so setting a cap costs nothing for the images it does not affect. Images uploaded with the client SDK or returned through the server SDK's tool-result normalizers carry their pixel dimensions (width / height on the file reference), so the runtime knows they fit without fetching them. Set the same fields on any file reference you build yourself, but only to the image's true pixel size: the runtime trusts them, so a smaller value lets an over-cap image skip the cap.
• This is a model-view transform only. Your stored conversation history, the files surface, and download URLs always keep the original full-resolution bytes, so nothing is lost.
• It is deterministic: the same image and cap always produce the same delivered image and note, so consecutive requests share the same prompt prefix and prompt caching keeps hitting.
• Omit the field to deliver images at full resolution and rely on whatever the provider does on its own.

Setting maxImageDimension is the recommended way to keep image-heavy sessions (screenshots, generated images, uploaded assets) from failing on a provider's per-image dimension limit. It bounds each image on its own; how many images of a long session ride each request is a question of the provider's delivery path, so it is a contextManagement concern: imageWindow keeps the model's view under a per-provider (or fixed) window, and a reactive safety net additionally recovers from image-count and byte limits (and from a provider tightening its limits below your cap) - see that page. Every adaptation is recorded in the session trace, so you can always see what the model actually received.

Inline image payload

Some delivery paths ship an image's bytes inside the request body rather than a URL the provider fetches: Gemini reads a tool result's images only as inline data, and every image sent through OpenRouter is shipped as bytes, since the hosts behind it fetch URLs with downloaders of their own that fail or stall on a long history. Those bytes are re-sent on every step of a turn, so the platform always bounds them: once the inline images in view exceed 32 MiB of image data - or less where the provider documents a smaller request-body ceiling (about 15 MB on OpenRouter) - the oldest are elided down to half that, each replaced where it was delivered by a short reference carrying its download URL, with the same block-wise pruning and the same pinning of the first user turn's attachments as the image window. It needs no configuration, never drops the most recent image, and does nothing for providers that fetch images by URL. Each prune is recorded in the session trace as an image-adapted entry, and every step's step-stats entry records the images its request carried and how (see Debugging).

maxImageDimension is also available per worker on the start-thread block; a worker inherits its parent agent's contextManagement.imageWindow.

Thinking

One vocabulary controls how much a model reasons, whatever the provider:

agent:
  model: anthropic/claude-opus-5-5
  thinking: high # off | low | medium | high | xhigh | max | auto

Omitting thinking is the same as off. Whenever a model reasons, its reasoning summary is requested, so thinking content streams to the UI and can be displayed to users.

How a level runs on a model

Every provider goes through the same rule, applied to what the model natively supports:

• A fixed level runs as the model's native level of the same name. When the model has no level of that name, it runs as the nearest one it has, and ties go up: xhigh on Claude Opus 4.6 (which has low to high and max) runs as max; medium on a model with low, high and max runs as high. A level above the model's top runs at its top, which is what max always means. On the few models that still take a token budget rather than named levels (Claude 4.5 and older, Gemini 2.5), each level is a budget inside the model's range.
• off sends the model's own off switch where it has one: thinking: disabled on Claude, effort none on the OpenAI models that accept it, a 0 budget on Gemini 2.5 Flash, enabled: false on OpenRouter. Where it has none - every Gemini 3 model, Grok, Claude Opus 5.5 and the Fable line, GPT-6 Astra and the o-series, and any OpenRouter model whose reasoning is mandatory - the lightest setting runs instead. off never sends nothing, which would run the model's own default (often its deepest level).
• auto keeps reasoning on and hands the depth to the model where the provider has such a mode (Anthropic adaptive thinking without an effort, Gemini dynamic thinking, OpenRouter's enabled, a router's own choice), and sends the provider's recommended default explicitly where it has none (medium on OpenAI, high on xAI), so auto never means no reasoning and never changes under a provider's default.
• A model that does not reason gets no reasoning setting at all, and a declared temperature stays. Otherwise a declared temperature is sent only when the model accepts one under the applied setting: Claude 4.7 and later and every Claude 5.x reject sampling parameters outright, OpenAI's reasoning models accept them only at effort none, Gemini 3 wants its defaults, Grok accepts them.

How a model maps is known per model: a curated table for the direct providers, the reasoning controls OpenRouter publishes for each of its models, and - for a model newer than either - the provider's current generation, with off running the lightest setting so the request is always accepted.

What ran is always visible

• Every model step's step-stats entry carries thinking: the declared level, the native setting it ran as (effort:xhigh, budget:16384, auto, disabled, or none), how the two relate (exact, substituted, floored, unsupported), and how certain the translation is (curated, gateway, derived, inferred, unknown). See Debugging.
• A level that does not run exactly as declared - a substitution, an off the model cannot honor, a model without reasoning, a declared temperature the applied setting rejects - is recorded once per run and model as a thinking-adjusted entry in the execution log.
• Saving a protocol that pairs a literal model with a literal level warns when the level will not run as declared: THINKING_LEVEL_SUBSTITUTED, THINKING_CANNOT_DISABLE, or THINKING_UNSUPPORTED_MODEL.

Hold a level constant within a thread where you can: Anthropic and OpenAI render it into the prompt, so changing it between requests starts the prompt cache over. A high level also needs room in maxOutputTokens, since reasoning counts toward the output cap on every provider.

Prompt Caching

Providers charge less for tokens served from their prompt cache (often 10% of the uncached rate). Octavus exposes a single cache field that picks the right retention policy per provider, so the stable prefix of your agent - tools, system prompt, and historical messages - gets billed at the cache-read rate on repeat requests.

agent:
  model: anthropic/claude-sonnet-4-5
  cache: auto # auto (default) | extended | off

Per-provider behavior

The cache field is provider-agnostic at the protocol level - each provider translates it into its own cache retention policy:

On OpenRouter, Claude models (openrouter/anthropic/*) cache only when a request asks, so on Anthropic's own endpoint (see OpenRouter routing) Octavus sends Anthropic's caching directive with the TTL your mode maps to - the same caching as the direct Anthropic provider. For every other model, cache controls the session's sticky routing key (session_id): with auto or extended a session's requests stay on the host whose prompt cache is warm, and the host's own caching does the rest. Two cases get no caching directive, so their Claude steps are not cached: an older Claude release that runs on OpenRouter's default routing, and a router, since the model it picks is only known once the request is served. On off, Octavus sends no caching directive and no routing key - except to a router, which always gets the key, since it is what keeps a conversation on the model the router picked. Providers that auto-cache (OpenAI on prefixes ≥ 1,024 tokens, Gemini 2.5+) may still cache transparently - off just disables Octavus's opt-in behavior.

Threads don't inherit

Named threads (created with start-thread) read their own cache field independently - they do not inherit the agent's cache value:

agent:
  cache: extended # 1-hour TTL on the main thread

handlers:
  summarize:
    Start summary:
      block: start-thread
      thread: summary
      # No cache field → defaults to 'auto' (5-minute TTL), NOT 'extended'
      system: summary-system

This is intentional: named threads are often used for short, one-shot work (summarization, classification) where the long TTL would be wasted. Set cache explicitly on start-thread when you do want it.

Cost trade-offs

• Cache reads are always much cheaper than uncached input on any provider - caching is effectively free if your prefix is stable.
• Cache writes on Claude models, direct or through OpenRouter, cost ~1.25× input for auto and 2× input for extended. OpenAI bills cache writes at ~1.25× input from GPT-5.6 on; earlier OpenAI models and Google don't charge separately for cache writes.
• Use extended only when the same prefix is genuinely reused across sessions that span hours; otherwise the higher write cost dominates the savings.

Skills

Enable Octavus skills for code execution and file generation:

skills:
  qr-code:
    display: description
    description: Generating QR codes

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  skills: [qr-code] # Enable skills
  agentic: true

Skills provide provider-agnostic code execution in isolated sandboxes. When enabled, the LLM can execute Python/Bash code, run skill scripts, and generate files.

See Skills for full documentation.

References

Enable on-demand context loading via reference documents:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  references: [api-guidelines, error-codes]
  agentic: true

References are markdown files stored in the agent's references/ directory. When enabled, the LLM can list available references and read their content using octavus_reference_list and octavus_reference_read tools.

See References for full documentation.

Image Generation

Enable the LLM to generate images autonomously:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  imageModel: openai/gpt-image-2.5-flare
  agentic: true

When imageModel is configured, the octavus_generate_image tool becomes available. The LLM can decide when to generate images based on user requests. The tool supports both text-to-image generation and image editing/transformation using reference images.

Supported Image Providers

Aspect Ratios and Resolution

The tool advertises aspectRatio - and, for models that support it, resolution - narrowed to what the configured model can actually produce. aspectRatio defaults to 1:1; a ratio outside a model's set is clamped to the nearest supported one.

Supported aspect ratios by model family:

Resolution is a Gemini 3 image feature. When a model has no resolution axis, the resolution field is not offered and any value is ignored (the model produces its default 1K output).

Image Editing with Reference Images

Both the agentic tool and the generate-image block support reference images for editing and transformation. When reference images are provided, the prompt describes how to modify or use those images.

Agentic vs Deterministic

Use imageModel in agent config when:

• The LLM should decide when to generate or edit images
• Users ask for images in natural language

Use generate-image block (see Handlers) when:

• You want explicit control over image generation or editing
• Building prompt engineering pipelines
• Images are generated at specific handler steps

Video

Video is two separate capabilities: understanding video (as model input) and generating video (as model output). Both are Google-centric today.

Understanding video

To let an agent watch video - summarize a recording, transcribe it, answer questions, or cite timestamps - run it on a Google Gemini model and pass the video as a file input. Gemini reads both the visual and audio track natively.

agent:
  model: google/gemini-3.5-flash
  system: system

Video understanding is model-dependent, and only Gemini reads video today. If you send a video file to a model that cannot see it (Anthropic, OpenAI), the platform does not silently hand over a useless URL - it tells the model the file is a video it cannot watch, and validation warns you at authoring time. The idiomatic pattern for an agent whose main model cannot see video is to define a worker on a Gemini model and delegate video files to it (see Workers); the video is read once inside the worker, and only the text result returns to the parent.

Uploaded videos up to 100MB (mp4, webm, quicktime, mpeg) are delivered to Gemini by URL - no size handling on your side.

Generating video

Set videoModel to enable the agentic octavus_generate_video tool, exactly like imageModel enables octavus_generate_image:

agent:
  model: anthropic/claude-sonnet-5
  system: system
  videoModel: google/veo-3.1-fast-generate-preview
  agentic: true

The tool generates a short clip (a few seconds) from a text prompt, and optionally from a starting image (image-to-video). It supports aspectRatio (landscape 16:9 or portrait 9:16), an optional durationSeconds, and an optional native-audio track. The clip is stored in Octavus storage and delivered into the conversation as a playable file, exactly like a generated image.

Video generation is Google-only today (Veo). Generation runs a slow provider job (tens of seconds to a couple of minutes) inside the turn, so the tool call stays pending until the clip is ready. Cancelling the run abandons it without delivering a partial clip.

Use the generate-video block (see Handlers) for deterministic, pipeline-style generation, the same way generate-image mirrors imageModel.

Speech Generation

Set speechModel to enable the agentic octavus_generate_speech tool, exactly like imageModel enables octavus_generate_image:

agent:
  model: anthropic/claude-sonnet-5
  system: system
  speechModel: openai/gpt-4o-mini-tts
  speechVoice: marin # optional default voice
  agentic: true

The tool turns a block of text into natural spoken audio and delivers it into the conversation as a playable audio file (with a download option) - the agent receives a reference (URL, format, size), never raw bytes. It supports an optional voice, an output format (default mp3), optional delivery instructions, and an optional language. The advertised voices and formats are narrowed to what the configured model supports, so the LLM only ever picks a valid option. Set speechVoice (a literal voice id or a variable reference) to fix the default voice used when the model does not specify one.

Use the generate-speech block (see Handlers) for deterministic, pipeline-style generation, the same way generate-image mirrors imageModel.

Transcription

Set transcriptionModel to enable the agentic octavus_transcribe_audio tool:

agent:
  model: anthropic/claude-sonnet-5
  system: system
  transcriptionModel: openai/gpt-4o-transcribe
  agentic: true

The tool takes the URL of an audio (or video) file and returns its transcript as text - so transcription works regardless of whether the chat model can natively "hear" the file. It auto-detects the spoken language by default, accepts an optional language hint, and can return timestamped segments (timestamps: true) where the model supports them (for OpenAI, whisper-1). Short transcripts return inline; long transcripts are delivered as a downloadable transcript file with only a bounded preview returned, so a multi-hour transcript never floods the model context. A file with no recognizable speech (silence, music, sound effects) is a successful result with an empty transcript and speechDetected: false, not an error - and not a statement about whether the file has an audio track.

The dedicated transcription API has per-request size/duration limits (OpenAI: ~25 MB / ~25 min). To transcribe long recordings in one pass, point a worker at a long-context multimodal model (e.g. Gemini) and delegate the file to it - the model transcribes or summarizes the whole recording directly. Use the transcribe-audio block (see Handlers) for deterministic, pipeline-style transcription.

Web Search

Enable the LLM to search the web for current information:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  webSearch: true
  agentic: true

When webSearch is enabled, the octavus_web_search tool becomes available. The LLM can decide when to search the web based on the conversation. Search results include source URLs that are emitted as citations in the UI.

This is a provider-agnostic built-in tool - it works with any LLM provider (Anthropic, Google, OpenAI, etc.).

Use cases:

• Current events and real-time data
• Fact verification and documentation lookups
• Any information that may have changed since the model's training

TODO List

Enable the LLM to maintain a structured task list while it works:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  todoList: true
  agentic: true

When todoList is enabled, the octavus_todo_write tool becomes available. The LLM creates and updates a list of items - each with id, content, and status (pending, in_progress, completed, cancelled) - and the platform emits a todo-update stream event with the resolved snapshot. The Client SDK accumulates updates into a single UITodoPart per assistant message, so consumers render an evolving "Plan" card without managing state themselves.

The list persists across messages: the LLM can use merge=true to update items by id (sending only the changed fields), or merge=false to replace the list entirely.

Use cases:

• Multi-step tasks where the user benefits from seeing progress
• Long-running agentic loops that should communicate intent
• Workflows where the agent plans before acting

View Images

Let the agent fetch an image from a URL into its own view so it can see and reason about the picture:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  viewImages: true
  agentic: true

When viewImages is enabled, the octavus_view_image tool becomes available. Give it the URL of an image - a message attachment (SMS/MMS, email, Slack), an image on a web page, or a file the agent produced - and the platform fetches it, verifies it is a real image, re-hosts it to storage, and attaches it to the conversation as vision. Only a compact confirmation is returned inline; the image arrives as a vision block on the next step, so context stays lean.

This is a provider-agnostic built-in tool - the model's own multimodal understanding does the seeing, so no extra model or API key is required. Common web image formats are supported (PNG, JPEG, GIF, WebP). Viewing is an explicit, agentic decision: images are never auto-inserted into context, so a tool that surfaces an image only as a URL hands that URL to octavus_view_image when the agent decides to look.

Use cases:

• Reading a photo texted or emailed to the agent (a receipt, a screenshot, an ID)
• Inspecting an image found on a web page
• Looking at an image file the agent generated or downloaded

Temperature

Control response randomness:

agent:
  model: openai/gpt-4o
  temperature: 0.7 # 0 = deterministic, 2 = creative

Guidelines:

• 0 - 0.3: Factual, consistent responses
• 0.4 - 0.7: Balanced (good default)
• 0.8 - 1.2: Creative, varied responses
• > 1.2: Very creative (may be inconsistent)

Streaming Cadence

Model providers frame their streamed output at different granularities - some emit small, word-sized deltas, others emit large multi-token chunks - so streamed text can arrive in lumps. The streaming option re-chunks visible text and reasoning on the wire so downstream renderers see a smooth, continuous stream, independent of the provider:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  streaming:
    chunking: word # off (default) | word | line
    delayMs: 15 # optional inter-chunk delay in ms

• chunking: off (the default) forwards provider deltas unchanged - no extra frames, no behavior change.
• chunking: word / line split text and reasoning into word- or line-sized deltas so a consumer rendering the raw stream sees continuous typing. The trade-off is more, smaller stream events.
• delayMs is an optional pause between emitted chunks, an integer between 0 and 100. Keep it small so a burst drains within the model's natural idle gap; omit it for the built-in default.

This shapes the wire stream, which is what you want when a backend consumes the SSE stream (via @octavus/server-sdk) and renders it elsewhere. If you render through the client SDK instead, prefer the zero-cost client-side textSmoothing option (see Client SDK -> Streaming); the two are independent and can be combined.

streaming resolves with the standard block > thread > agent precedence and is a no-op for structured output (responseType) blocks. It is a literal option (not variable-resolvable).

Dynamic Configuration

Like model, the temperature, thinking, speed, and maxSteps fields can also reference an input variable. Consumers choose values at session creation, so the same agent can be tuned per call without protocol changes:

input:
  TEMPERATURE:
    type: number
    description: Override temperature (0-2)
    optional: true
  THINKING:
    type: string
    description: Override thinking effort (low/medium/high/xhigh/max/auto, or "off")
    optional: true
  MAX_STEPS:
    type: integer
    description: Override max agentic steps
    optional: true

agent:
  model: anthropic/claude-sonnet-4-5
  temperature: TEMPERATURE
  thinking: THINKING
  maxSteps: MAX_STEPS
  system: system

When creating a session, pass the values in their natural type:

const sessionId = await client.agentSessions.create('my-agent', {
  TEMPERATURE: 0.7,
  THINKING: 'medium',
  MAX_STEPS: 5,
});

Accepted values

The resolver accepts the natural type for each field, plus a string fallback so consumers can pass values from form inputs without coercing first.

The protocol's input: declaration enforces what the consumer can pass. Pick type: number / type: integer if you want native numeric overrides; pick type: string (or type: unknown) if you also need to pass the "off" sentinel for temperature.

Explicit "off" vs not set

temperature and thinking accept an explicit "off" value to disable the field at session creation. This is different from omitting the variable:

• Variable not provided -> the field is unset: no temperature is emitted, and thinking is off (the same request as thinking: off)
• Variable provided as "off" -> the field is explicitly disabled

The distinction matters for temperature: an unset variable and "off" both leave it out, while any other value sends it. For thinking, "off" turns reasoning off on every model that allows it; see Thinking for what happens on a model that cannot. A declared temperature only reaches the model when the model accepts one under the applied thinking setting (most reject it while reasoning); use "off" on one to let the other take effect.

Validation

Variable references are caught at protocol validation time. If temperature: TEMPERATURE is declared but TEMPERATURE is missing from input: or variables:, the validator surfaces the error in the dashboard before the agent runs.

Thread-Specific Config

Override config for named threads:

handlers:
  request-human:
    Start summary thread:
      block: start-thread
      thread: summary
      model: anthropic/claude-opus-4-8 # Different model
      backupModel: openai/gpt-4o # Failover model
      thinking: low # Different thinking
      speed: fast # Fast mode for this thread (supported Opus models only)
      streaming: { chunking: word } # Smooth streamed text for this thread
      cache: off # Different cache mode (does not inherit from agent)
      maxSteps: 1 # Limit tool calls
      system: escalation-summary # Different prompt
      mcpServers: [figma, browser] # Thread-specific MCP servers
      skills: [data-analysis] # Thread-specific skills
      references: [escalation-policy] # Thread-specific references
      imageModel: openai/gpt-image-2.5-flare # Thread-specific image model
      webSearch: true # Thread-specific web search
      todoList: true # Thread-specific task list
      viewImages: true # Thread-specific image viewing

Each thread can have its own model, backup model, thinking level, speed, streaming cadence, cache mode, MCP servers, skills, references, image model, web search setting, task list setting, and image viewing setting. Skills must be defined in the protocol's skills: section. References must exist in the agent's references/ directory. Workers use this same pattern since they don't have a global agent: section - which is how a worker enables fast mode.

Full Example

input:
  COMPANY_NAME: { type: string }
  PRODUCT_NAME: { type: string }
  USER_ID: { type: string, optional: true }

resources:
  CONVERSATION_SUMMARY:
    type: string
    default: ''

tools:
  get-user-account:
    description: Look up user account
    parameters:
      userId: { type: string }

  search-docs:
    description: Search help documentation
    parameters:
      query: { type: string }

  create-support-ticket:
    description: Create a support ticket
    parameters:
      summary: { type: string }
      priority: { type: string } # low, medium, high

mcpServers:
  figma:
    description: Figma design tool integration
    source: remote
    display: description

skills:
  qr-code:
    display: description
    description: Generating QR codes

agent:
  model: anthropic/claude-sonnet-4-5
  backupModel: openai/gpt-4o
  system: system
  input:
    - COMPANY_NAME
    - PRODUCT_NAME
  tools:
    - get-user-account
    - search-docs
    - create-support-ticket
  mcpServers: [figma] # MCP server connections
  skills: [qr-code] # Octavus skills
  references: [support-policies] # On-demand context
  webSearch: true # Built-in web search
  todoList: true # Structured task tracking
  viewImages: true # View images from URLs
  agentic: true
  maxSteps: 10
  thinking: medium

triggers:
  user-message:
    input:
      USER_MESSAGE: { type: string }

handlers:
  user-message:
    Add message:
      block: add-message
      role: user
      prompt: user-message
      input: [USER_MESSAGE]
      display: hidden

    Respond:
      block: next-message