Agent Config

The agent section configures the LLM model, system prompt, tools, and behavior.

Basic Configuration

agent:
  model: anthropic/claude-sonnet-4-5
  system: system # References prompts/system.md
  tools: [get-user-account] # Available tools
  mcpServers: [figma, browser] # MCP server connections
  skills: [qr-code] # Available skills
  references: [api-guidelines] # On-demand context documents

Configuration Options

Models

Specify models in provider/model-id format. Any model supported by the provider's SDK will work.

Supported Providers

Examples

# Anthropic Claude 4.5
agent:
  model: anthropic/claude-sonnet-4-5

# Google Gemini 3
agent:
  model: google/gemini-3-flash-preview

# OpenAI GPT-5
agent:
  model: openai/gpt-5

# OpenAI reasoning models
agent:
  model: openai/o3-mini

# xAI Grok
agent:
  model: x-ai/grok-4.6

Dynamic Model Selection

The model field can also reference an input variable, allowing consumers to choose the model when creating a session:

input:
  MODEL:
    type: string
    description: The LLM model to use

agent:
  model: MODEL # Resolved from session input
  system: system

When creating a session, pass the model:

const sessionId = await client.agentSessions.create('my-agent', {
  MODEL: 'anthropic/claude-sonnet-4-5',
});

This enables:

• Multi-provider support - Same agent works with different providers
• A/B testing - Test different models without protocol changes
• User preferences - Let users choose their preferred model

The model value is validated at runtime to ensure it's in the correct provider/model-id format.

Backup Model

Configure a fallback model that activates automatically when the primary model encounters a transient provider error (rate limits, outages, timeouts):

agent:
  model: anthropic/claude-sonnet-4-5
  backupModel: openai/gpt-4o
  system: system

When a provider error occurs, the system retries once with the backup model. If the backup also fails, the original error is returned.

Key behaviors:

• Only transient provider errors trigger fallback - authentication and validation errors are not retried
• Provider-specific options (like anthropic:) are only forwarded to the backup model if it uses the same provider
• For streaming responses, fallback only occurs if no content has been sent to the client yet

Like model, backupModel supports variable references:

input:
  BACKUP_MODEL:
    type: string
    description: Fallback model for provider errors

agent:
  model: anthropic/claude-sonnet-4-5
  backupModel: BACKUP_MODEL
  system: system

System Prompt

The system prompt sets the agent's persona and instructions. The input field controls which variables are available to the prompt - only variables listed in input are interpolated.

agent:
  system: system # Uses prompts/system.md
  input:
    - COMPANY_NAME
    - PRODUCT_NAME

Variables in input can come from protocol.input, protocol.resources, or protocol.variables.

Input Mapping Formats

# Array format (same name)
input:
  - COMPANY_NAME
  - PRODUCT_NAME

# Array format (rename)
input:
  - CONTEXT: CONVERSATION_SUMMARY  # Prompt sees CONTEXT, value comes from CONVERSATION_SUMMARY

# Object format (rename)
input:
  CONTEXT: CONVERSATION_SUMMARY

The left side (label) is what the prompt sees. The right side (source) is where the value comes from.

Example

prompts/system.md:

You are a friendly support agent for {{COMPANY_NAME}}.

## Your Role

Help users with questions about {{PRODUCT_NAME}}.

## Guidelines

- Be helpful and professional
- If you can't help, offer to escalate
- Never share internal information

Agentic Mode

Enable multi-step tool calling:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  tools: [get-user-account, search-docs, create-ticket]
  agentic: true # LLM can call multiple tools
  maxSteps: 10 # Limit cycles to prevent runaway

How it works:

1. LLM receives user message
2. LLM decides to call a tool
3. Tool executes, result returned to LLM
4. LLM decides if more tools needed
5. Repeat until LLM responds or maxSteps reached

Output Limits and Loop Guard

Two optional safeguards bound the cost of a single LLM generation. Both apply per generation (one agentic step), not across the whole run - maxSteps is what bounds how many steps a run can take.

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  maxOutputTokens: 32000 # hard cap on output tokens per generation
  loopGuard: true # abort a generation that degenerates into a repeated string

maxOutputTokens

Caps the output tokens of a single generation, passed straight through to the provider's max_tokens. When omitted, the provider/SDK default applies - it varies by model and can be very large. Setting an explicit value gives you a predictable, model-independent ceiling on the cost of any one generation.

This is a per-step cap, not a budget for the whole run. A long agentic run makes many generations, each capped independently; use maxSteps to bound the number of steps.

loopGuard

Autoregressive models can occasionally degenerate into a repetition loop - emitting the same short string thousands of times until the output budget is exhausted. Set loopGuard: true to watch the streamed output and abort a generation as soon as it detects a short unit repeating past a threshold, so a degenerate turn stops after a few hundred tokens instead of burning the full output cap. The runaway tail is trimmed out of the stored message so it does not pollute later turns.

It is a plain on/off flag - the detection thresholds (how many consecutive repeats trip it, the longest unit considered) are fixed, conservative defaults, since they describe a degeneration detector rather than anything worth tuning per agent. Omit loopGuard (or set it to false) to disable it.

Image Delivery Limits

Model providers impose image constraints that change over time - most notably a maximum dimension per image, and stricter caps once a request carries many images. Set maxImageDimension to cap the longest side (in pixels) of every image in the model's view:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  maxImageDimension: 2000 # downscale any image above 2000px on its longest side

• On every request, an image whose longest side exceeds the cap is delivered downscaled to fit (aspect ratio preserved).
• The cap only touches the images it actually changes. An image already within it is delivered exactly as it would be with no cap set - same bytes, same delivery path, no re-encoding and no quality change - so setting a cap costs nothing for the images it does not affect.
• This is a model-view transform only. Your stored conversation history, the files surface, and download URLs always keep the original full-resolution bytes, so nothing is lost.
• It is deterministic (the same image and cap always produce the same delivered bytes), so prompt caching is unaffected.
• Omit the field to deliver images at full resolution and rely on whatever the provider does on its own.

Setting maxImageDimension is the recommended way to keep image-heavy sessions (screenshots, generated images, uploaded assets) from failing on a provider's per-image dimension limit. For agents that also declare contextManagement, a reactive safety net additionally recovers from image-count and byte limits (and from a provider tightening its limits below your cap) - see that page. Every adaptation is recorded in the session trace, so you can always see what the model actually received.

maxImageDimension is also available per worker on the start-thread block.

Extended Thinking

Enable extended reasoning for complex tasks:

agent:
  model: anthropic/claude-sonnet-4-5
  thinking: medium # low | medium | high | max

Thinking content streams to the UI and can be displayed to users.

How levels are applied

Each provider translates thinking into its own reasoning controls:

Prompt Caching

Providers charge less for tokens served from their prompt cache (often 10% of the uncached rate). Octavus exposes a single cache field that picks the right retention policy per provider, so the stable prefix of your agent - tools, system prompt, and historical messages - gets billed at the cache-read rate on repeat requests.

agent:
  model: anthropic/claude-sonnet-4-5
  cache: auto # auto (default) | extended | off

Per-provider behavior

The cache field is provider-agnostic at the protocol level - each provider translates it into its own cache retention policy:

On off, Octavus emits no explicit cache options. Providers that auto-cache (OpenAI on prefixes ≥ 1,024 tokens, Gemini 2.5+) may still cache transparently - off just disables Octavus's opt-in behavior.

Threads don't inherit

Named threads (created with start-thread) read their own cache field independently - they do not inherit the agent's cache value:

agent:
  cache: extended # 1-hour TTL on the main thread

handlers:
  summarize:
    Start summary:
      block: start-thread
      thread: summary
      # No cache field → defaults to 'auto' (5-minute TTL), NOT 'extended'
      system: summary-system

This is intentional: named threads are often used for short, one-shot work (summarization, classification) where the long TTL would be wasted. Set cache explicitly on start-thread when you do want it.

Cost trade-offs

• Cache reads are always much cheaper than uncached input on any provider - caching is effectively free if your prefix is stable.
• Cache writes on Anthropic cost ~1.25× input for auto and 2× input for extended. OpenAI and Google don't charge separately for cache writes.
• Use extended only when the same prefix is genuinely reused across sessions that span hours; otherwise the higher write cost dominates the savings.

Skills

Enable Octavus skills for code execution and file generation:

skills:
  qr-code:
    display: description
    description: Generating QR codes

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  skills: [qr-code] # Enable skills
  agentic: true

Skills provide provider-agnostic code execution in isolated sandboxes. When enabled, the LLM can execute Python/Bash code, run skill scripts, and generate files.

See Skills for full documentation.

References

Enable on-demand context loading via reference documents:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  references: [api-guidelines, error-codes]
  agentic: true

References are markdown files stored in the agent's references/ directory. When enabled, the LLM can list available references and read their content using octavus_reference_list and octavus_reference_read tools.

See References for full documentation.

Image Generation

Enable the LLM to generate images autonomously:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  imageModel: google/gemini-2.5-flash-image
  agentic: true

When imageModel is configured, the octavus_generate_image tool becomes available. The LLM can decide when to generate images based on user requests. The tool supports both text-to-image generation and image editing/transformation using reference images.

Supported Image Providers

Aspect Ratios and Resolution

The tool advertises aspectRatio - and, for models that support it, resolution - narrowed to what the configured model can actually produce. aspectRatio defaults to 1:1; a ratio outside a model's set is clamped to the nearest supported one.

Supported aspect ratios by model family:

Resolution is a Gemini 3 image feature. When a model has no resolution axis, the resolution field is not offered and any value is ignored (the model produces its default 1K output).

Image Editing with Reference Images

Both the agentic tool and the generate-image block support reference images for editing and transformation. When reference images are provided, the prompt describes how to modify or use those images.

Agentic vs Deterministic

Use imageModel in agent config when:

• The LLM should decide when to generate or edit images
• Users ask for images in natural language

Use generate-image block (see Handlers) when:

• You want explicit control over image generation or editing
• Building prompt engineering pipelines
• Images are generated at specific handler steps

Video

Video is two separate capabilities: understanding video (as model input) and generating video (as model output). Both are Google-centric today.

Understanding video

To let an agent watch video - summarize a recording, transcribe it, answer questions, or cite timestamps - run it on a Google Gemini model and pass the video as a file input. Gemini reads both the visual and audio track natively.

agent:
  model: google/gemini-3.5-flash
  system: system

Video understanding is model-dependent, and only Gemini reads video today. If you send a video file to a model that cannot see it (Anthropic, OpenAI), the platform does not silently hand over a useless URL - it tells the model the file is a video it cannot watch, and validation warns you at authoring time. The idiomatic pattern for an agent whose main model cannot see video is to define a worker on a Gemini model and delegate video files to it (see Workers); the video is read once inside the worker, and only the text result returns to the parent.

Uploaded videos up to 100MB (mp4, webm, quicktime, mpeg) are delivered to Gemini by URL - no size handling on your side.

Generating video

Set videoModel to enable the agentic octavus_generate_video tool, exactly like imageModel enables octavus_generate_image:

agent:
  model: anthropic/claude-sonnet-5
  system: system
  videoModel: google/veo-3.1-fast-generate-preview
  agentic: true

The tool generates a short clip (a few seconds) from a text prompt, and optionally from a starting image (image-to-video). It supports aspectRatio (landscape 16:9 or portrait 9:16), an optional durationSeconds, and an optional native-audio track. The clip is stored in Octavus storage and delivered into the conversation as a playable file, exactly like a generated image.

Video generation is Google-only today (Veo). Generation runs a slow provider job (tens of seconds to a couple of minutes) inside the turn, so the tool call stays pending until the clip is ready. Cancelling the run abandons it without delivering a partial clip.

Use the generate-video block (see Handlers) for deterministic, pipeline-style generation, the same way generate-image mirrors imageModel.

Speech Generation

Set speechModel to enable the agentic octavus_generate_speech tool, exactly like imageModel enables octavus_generate_image:

agent:
  model: anthropic/claude-sonnet-5
  system: system
  speechModel: openai/gpt-4o-mini-tts
  speechVoice: marin # optional default voice
  agentic: true

The tool turns a block of text into natural spoken audio and delivers it into the conversation as a playable audio file (with a download option) - the agent receives a reference (URL, format, size), never raw bytes. It supports an optional voice, an output format (default mp3), optional delivery instructions, and an optional language. The advertised voices and formats are narrowed to what the configured model supports, so the LLM only ever picks a valid option. Set speechVoice (a literal voice id or a variable reference) to fix the default voice used when the model does not specify one.

Use the generate-speech block (see Handlers) for deterministic, pipeline-style generation, the same way generate-image mirrors imageModel.

Transcription

Set transcriptionModel to enable the agentic octavus_transcribe_audio tool:

agent:
  model: anthropic/claude-sonnet-5
  system: system
  transcriptionModel: openai/gpt-4o-transcribe
  agentic: true

The tool takes the URL of an audio (or video) file and returns its transcript as text - so transcription works regardless of whether the chat model can natively "hear" the file. It auto-detects the spoken language by default, accepts an optional language hint, and can return timestamped segments (timestamps: true) where the model supports them (for OpenAI, whisper-1). Short transcripts return inline; long transcripts are delivered as a downloadable transcript file with only a bounded preview returned, so a multi-hour transcript never floods the model context.

The dedicated transcription API has per-request size/duration limits (OpenAI: ~25 MB / ~25 min). To transcribe long recordings in one pass, point a worker at a long-context multimodal model (e.g. Gemini) and delegate the file to it - the model transcribes or summarizes the whole recording directly. Use the transcribe-audio block (see Handlers) for deterministic, pipeline-style transcription.

Web Search

Enable the LLM to search the web for current information:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  webSearch: true
  agentic: true

When webSearch is enabled, the octavus_web_search tool becomes available. The LLM can decide when to search the web based on the conversation. Search results include source URLs that are emitted as citations in the UI.

This is a provider-agnostic built-in tool - it works with any LLM provider (Anthropic, Google, OpenAI, etc.). For Anthropic's own web search implementation, see Provider Options.

Use cases:

• Current events and real-time data
• Fact verification and documentation lookups
• Any information that may have changed since the model's training

TODO List

Enable the LLM to maintain a structured task list while it works:

agent:
  model: anthropic/claude-sonnet-4-5
  system: system
  todoList: true
  agentic: true

When todoList is enabled, the octavus_todo_write tool becomes available. The LLM creates and updates a list of items - each with id, content, and status (pending, in_progress, completed, cancelled) - and the platform emits a todo-update stream event with the resolved snapshot. The Client SDK accumulates updates into a single UITodoPart per assistant message, so consumers render an evolving "Plan" card without managing state themselves.

The list persists across messages: the LLM can use merge=true to update items by id (sending only the changed fields), or merge=false to replace the list entirely.

Use cases:

• Multi-step tasks where the user benefits from seeing progress
• Long-running agentic loops that should communicate intent
• Workflows where the agent plans before acting

Temperature

Control response randomness:

agent:
  model: openai/gpt-4o
  temperature: 0.7 # 0 = deterministic, 2 = creative

Guidelines:

• 0 - 0.3: Factual, consistent responses
• 0.4 - 0.7: Balanced (good default)
• 0.8 - 1.2: Creative, varied responses
• > 1.2: Very creative (may be inconsistent)

Dynamic Configuration

Like model, the temperature, thinking, speed, and maxSteps fields can also reference an input variable. Consumers choose values at session creation, so the same agent can be tuned per call without protocol changes:

input:
  TEMPERATURE:
    type: number
    description: Override temperature (0-2)
    optional: true
  THINKING:
    type: string
    description: Override thinking effort (low/medium/high/max, or "off")
    optional: true
  MAX_STEPS:
    type: integer
    description: Override max agentic steps
    optional: true

agent:
  model: anthropic/claude-sonnet-4-5
  temperature: TEMPERATURE
  thinking: THINKING
  maxSteps: MAX_STEPS
  system: system

When creating a session, pass the values in their natural type:

const sessionId = await client.agentSessions.create('my-agent', {
  TEMPERATURE: 0.7,
  THINKING: 'medium',
  MAX_STEPS: 5,
});

Accepted values

The resolver accepts the natural type for each field, plus a string fallback so consumers can pass values from form inputs without coercing first.

The protocol's input: declaration enforces what the consumer can pass. Pick type: number / type: integer if you want native numeric overrides; pick type: string (or type: unknown) if you also need to pass the "off" sentinel for temperature.

Explicit "off" vs not set

temperature and thinking accept an explicit "off" value to disable the field at session creation. This is different from omitting the variable:

• Variable not provided -> the field is unset; the provider uses its default behavior
• Variable provided as "off" -> the field is explicitly disabled (no temperature emitted, reasoning disabled)

The distinction matters because temperature and thinking are mutually exclusive at the provider level - several providers ignore temperature when reasoning is enabled. Use "off" to opt one out so the other takes effect.

Validation

Variable references are caught at protocol validation time. If temperature: TEMPERATURE is declared but TEMPERATURE is missing from input: or variables:, the validator surfaces the error in the dashboard before the agent runs.

Provider Options

Enable provider-specific features like Anthropic's built-in tools and skills:

agent:
  model: anthropic/claude-sonnet-4-5
  anthropic:
    tools:
      web-search:
        display: description
        description: Searching the web
    skills:
      pdf:
        type: anthropic
        description: Processing PDF

Provider options are validated against the model - using anthropic: with a non-Anthropic model will fail validation.

See Provider Options for full documentation.

Thread-Specific Config

Override config for named threads:

handlers:
  request-human:
    Start summary thread:
      block: start-thread
      thread: summary
      model: anthropic/claude-opus-4-8 # Different model
      backupModel: openai/gpt-4o # Failover model
      thinking: low # Different thinking
      speed: fast # Fast mode for this thread (supported Opus models only)
      cache: off # Different cache mode (does not inherit from agent)
      maxSteps: 1 # Limit tool calls
      system: escalation-summary # Different prompt
      mcpServers: [figma, browser] # Thread-specific MCP servers
      skills: [data-analysis] # Thread-specific skills
      references: [escalation-policy] # Thread-specific references
      imageModel: google/gemini-2.5-flash-image # Thread-specific image model
      webSearch: true # Thread-specific web search
      todoList: true # Thread-specific task list

Each thread can have its own model, backup model, thinking level, speed, cache mode, MCP servers, skills, references, image model, web search setting, and task list setting. Skills must be defined in the protocol's skills: section. References must exist in the agent's references/ directory. Workers use this same pattern since they don't have a global agent: section - which is how a worker enables fast mode.

Full Example

input:
  COMPANY_NAME: { type: string }
  PRODUCT_NAME: { type: string }
  USER_ID: { type: string, optional: true }

resources:
  CONVERSATION_SUMMARY:
    type: string
    default: ''

tools:
  get-user-account:
    description: Look up user account
    parameters:
      userId: { type: string }

  search-docs:
    description: Search help documentation
    parameters:
      query: { type: string }

  create-support-ticket:
    description: Create a support ticket
    parameters:
      summary: { type: string }
      priority: { type: string } # low, medium, high

mcpServers:
  figma:
    description: Figma design tool integration
    source: remote
    display: description

skills:
  qr-code:
    display: description
    description: Generating QR codes

agent:
  model: anthropic/claude-sonnet-4-5
  backupModel: openai/gpt-4o
  system: system
  input:
    - COMPANY_NAME
    - PRODUCT_NAME
  tools:
    - get-user-account
    - search-docs
    - create-support-ticket
  mcpServers: [figma] # MCP server connections
  skills: [qr-code] # Octavus skills
  references: [support-policies] # On-demand context
  webSearch: true # Built-in web search
  todoList: true # Structured task tracking
  agentic: true
  maxSteps: 10
  thinking: medium
  # Anthropic-specific options
  anthropic:
    tools:
      web-search:
        display: description
        description: Searching the web
    skills:
      pdf:
        type: anthropic
        description: Processing PDF

triggers:
  user-message:
    input:
      USER_MESSAGE: { type: string }

handlers:
  user-message:
    Add message:
      block: add-message
      role: user
      prompt: user-message
      input: [USER_MESSAGE]
      display: hidden

    Respond:
      block: next-message