Bringing Muse and SAM 3.1 to ChatbotX: An Omnichannel Chatbot That Handles Voice and Images

We integrated Muse Spark, Muse Image, Muse Voice Transcribe, and SAM 3.1 into an open-source chatbot platform. One API key enables them on every chat channel, with no new services to deploy.

ChatbotX is an open-source omnichannel chatbot platform. A single workspace can connect Messenger, Instagram, WhatsApp, Telegram, Zalo, TikTok, Threads, and Webchat. The platform ships with a flow builder, AI Agents, a shared inbox, and a public API, CLI, and MCP server that let other AI agents operate it.

During the hackathon, we focused on the AI layer: we added Muse as a new model provider for AI Agents and for AI flow steps. An AI Agent running on Muse can answer customers, read the images and documents they send, and then tag the contact, save data, or hand the conversation to a human. Inside flows, Muse transcribes WAV recordings, extracts structured data, generates images, and crops out exactly the object described, regardless of which channel the message came from.

Built during the hackathon (Oct 27 – Nov 4, 2026)

• #1586 feat(ai): add Muse (Meta Model API) provider: the Muse provider, Muse models for AI Agents and AI flow steps, image generation and editing, and speech-to-text.
• #1611 feat(flows): implement AI Segment Image step (SAM 3.1): the Segment Image step, powered by SAM 3.1.

Both pull requests were merged on November 1, 2026. Channel connectivity, the flow runtime, the inbox, and AI Agents existed before the hackathon.

What we built

Flow steps report their outcome through a Success or an Error branch. Segment Image adds a third branch, Skip, for when no matching object is found. A step writes its result to a custom field (a per-contact data field), where the next step can read it.

An example flow

A shoe store wants to handle complaints that include an audio recording and a product photo. Before the flow runs, two custom fields already hold values, written for example through the public API or by an earlier step: audio_url (the URL of a WAV file) and photo_url (the URL of the customer's product photo).

1. Speech to Text audio_url ─► transcript Success ─► 2 Error ─► "Please send a WAV file" 2. Extract Data transcript ─► order_id, issue Success ─► 3 Error ─► hand off to an agent 3. Segment Image photo_url, target "shoe" ─► crop_url Success ─► 4 Skip ─► "Please send another photo" Error ─► hand off to an agent 4. Analyze Image crop_url, "Describe any visible damage" ─► damage Success ─► 5 Error ─► hand off to an agent 5. Generate Text Agent prompt uses {{order_id}}, {{issue}}, {{damage}} ─► reply Success ─► 6 Error ─► hand off to an agent 6. Send message reply + image at crop_url 

Expected result: the customer receives a reply that cites the order number, the issue mentioned in the recording, and the damage visible in the photo, along with an image cropped down to the shoe. The order number, issue, and damage description are saved to custom fields, so a human agent can look them up immediately if the conversation is handed off. If the recording is not a valid WAV file, the flow stops on the Error branch of step 1 without calling Meta. The flow does not yet accept voice notes directly from Messenger or WhatsApp; see "What's next".

Architecture

Customer message │ webhook ▼ integration worker ───► POST /chat/completions muse-spark, muse-wonder AI Agent, text steps, Analyze Image │ queue "heavy" ▼ heavy worker ─────────┬► POST /images/* muse-image-1.0 Generate/Edit Image ├► POST /asr/transcribe muse-voice-transcribe-1.0 Speech to Text └► POST /responses (SSE) sam-3.1 Segment Image │ upload to S3, write custom field ▼ flow resumes Builder ──────────────► GET /models key check only 

Users connect Muse by pasting an API key under Settings → Integrations → Muse. The builder verifies the key with GET /v1/models and stores it per workspace; the key is never sent to the browser. Apart from that check, every model call happens in a worker, so webhooks return immediately and BullMQ handles retries. Lightweight steps call Meta directly from the integration worker. Heavy steps are dispatched to the heavy queue; when one finishes, the worker writes the result to a custom field and resumes the flow.

Chat and image generation go through the OpenAI-compatible provider in the Vercel AI SDK. Speech-to-text and SAM use their own request formats, so we call them directly over HTTP.

AI Agents running on Muse

The AI Agent is what customers interact with directly: it answers incoming messages on every channel without any flow. Muse can be the agent's primary model or one entry in a fallback chain. When Muse is selected, the agent's built-in tools run on Muse as well:

The image reader has no model of its own: the image is dispatched to the heavy queue and read with the agent's own provider and model. Because the Analyze Image step offers only Muse Spark models, agents that need to read images should use Muse Spark as well.

During the integration, we fixed a bug in AI Agent Actions: the model often left values empty when setting a custom field, because each action was identified only by an opaque random ID. The tool description now names the custom field behind each action:

1. ruleId: r1 When: The customer gives their order number Values to save when this rule matches (put each one the customer gave in values; omit any they did not give): - actionId a1: custom field "order_id" 

The image reader and document reader existed before the hackathon; the new work made them run on Muse and fixed the AI Agent Actions tool description.

Muse steps in flows

Every Muse flow step reads its input from a custom field and writes its result to a custom field, so steps can be chained as in the example flow above. The steps fall into three groups by data type.

Images: analyze, generate, edit, and segment

Analyze Image runs directly in the integration worker. During the integration, we found that this step reported success even when the provider returned an error: the AI SDK's textStream drops error parts, so the flow received an empty string. The step now keeps the error and rethrows it once the stream ends:

// apps/worker/src/integration/handlers/analyze-image/index.ts (simplified) // `textStream` drops error parts, so keep the provider error and rethrow // it once the stream ends — otherwise a failed call reports success. let streamError: unknown const result = streamText({ model: resolvedModel.model, // e.g. muse-spark-1.3 messages: [{ role: "user", content: [ { type: "text", text: step.prompt }, { type: "file", data: imageUrl, mediaType: "image/*" }, ]}], onError: ({ error }) => { streamError = error }, }) const { fullText } = await processStreamingText(result.textStream, noop, { sendParts: false }) if (streamError) throw streamError 

Generate Image, Edit Image, and Segment Image run on the heavy queue. The resulting image is re-encoded so it displays on every channel (challenge 2), uploaded to S3, and its URL is written to a custom field. Edit Image also inspects the file's actual bytes before sending it, instead of trusting content-type: CDNs often serve images as application/octet-stream, and a non-image file would otherwise fail inside the AI SDK as a retryable error. Image generation and editing fit scenarios other than the complaint flow above: for example, Edit Image can replace the background of a product photo a customer sends before it goes into a catalog, and Generate Image can create an illustration for a promotion to send along with a message.

Voice: Speech to Text

Muse Voice Transcribe accepts only 16-bit PCM WAV audio: mono, sampled at 16 or 24 kHz, at most 10 minutes long, in a request under 32 MB. Meta's documentation asks callers to convert other formats to WAV before uploading, but the worker has no ffmpeg to do that conversion. Instead, the worker inspects the file's RIFF header as soon as it is downloaded:

// apps/worker/src/heavy/handlers/speech-to-text.ts try { inspectMuseWav(audio) // reads only the RIFF header and chunk table, at most 64 chunks } catch (error) { if (error instanceof MuseWavFormatError) { throw new ExpectedHeavyStepError(`${MUSE_FORMAT_HINT}: ${error.message}`) } throw error } 

An unsupported file takes the Error branch with a specific reason (not mono, not 16-bit, and so on), and no request reaches Meta. The step also supports two optional Muse parameters, languageBias and keywords, which help Muse recognize product names and business-specific terms.

Text and data: Generate Text, Generate Text Agent, Extract Data

Generate Text produces text from a prompt. Generate Text Agent runs a configured AI Agent, including its tools, inside a flow and writes the reply to a custom field. Both use muse-spark-1.3, muse-spark-1.2, or muse-wonder-1.0, subject to the token and temperature handling described in challenge 1.

Extract Data turns text, an image, or a file into data fields such as order_id and issue. It builds a Zod schema from the list of fields to extract and asks the model for output that matches the schema (structured output):

// apps/worker/src/integration/handlers/extract-data/index.ts (simplified) const dynamicSchema = z.object(Object.fromEntries( step.extractFields.map(({ key }) => [key, z.string().nullable()]), )) const { output } = await generateText({ model: resolvedModel.model, // e.g. muse-spark-1.3 messages: [userMessage], output: Output.object({ schema: dynamicSchema }), }) // each non-null value is written to the custom field mapped to its key 

The Muse provider enables supportsStructuredOutputs, so the AI SDK includes the JSON Schema in the request. The result is validated against the schema before anything is written: a field the customer never mentioned comes back as null and is skipped, and output that does not match the schema sends the step to its Error branch. Combined with Speech to Text, a recording becomes data fields that flows, AI Agents, and human agents can all use.

Four engineering challenges

1. Muse speaks the OpenAI format, but not exactly

Problem. Muse accepts OpenAI-style requests, so we reused the existing OpenAI-compatible provider. Calling the real API revealed three differences:

Solution. Instead of patching every call site, we handle all three differences in transformRequestBody, a hook that rewrites the request body just before it is sent:

// packages/ai/src/core/muse.ts (simplified) export function transformMuseRequestBody(body) { const next = { ...body } // (1) Muse counts reasoning tokens against `max_tokens`. if (next.reasoning_effort == null) { next.reasoning_effort = maxTokens < MUSE_LOW_BUDGET_THRESHOLD ? "minimal" : "low" } next.max_tokens = maxTokens + MUSE_REASONING_HEADROOM_TOKENS // +1024 // (2) Temperature above 1 returns empty replies or 500s. if (next.temperature > MUSE_MAX_TEMPERATURE) next.temperature = MUSE_MAX_TEMPERATURE // (3) Muse rejects every `tool_choice` but "auto" with a 400. if (next.tool_choice === "none") { const { tools: _tools, tool_choice: _toolChoice, ...rest } = next return rest } if (next.tool_choice != null && next.tool_choice !== "auto") next.tool_choice = "auto" return next } 

Result. AI Agents and AI flow steps call Muse exactly as they call any other provider; every difference is contained in one function. All three fixes are covered by unit tests that run without an API key. The added headroom tokens are still billed, so for Muse, maxOutputTokens sets the budget for the reply text alone.

2. Natural replies on every channel

Problem. Two issues only surfaced once replies reached customers:

• When an agent calls a tool, the model often writes one sentence before the call and another after the result comes back. The AI SDK joins the two with no separator, so the customer receives text like "…the sleeve.I've noted…".
• The Muse image API accepts an output_format parameter (png, jpeg, or webp) that defaults to webp, and ChatbotX uses the default. WhatsApp, however, displays WebP files as stickers rather than images.

Solution. For text, we rebuild the stream and insert a paragraph break whenever the model starts a new step:

// packages/ai/src/core/stream.ts export async function* toStepSeparatedTextStream(stream) { let hasText = false let pendingStepBreak = false for await (const part of stream) { if (part.type === "start-step") { pendingStepBreak = hasText continue } if (part.type !== "text-delta" || !part.text) continue if (pendingStepBreak) { pendingStepBreak = false yield PARAGRAPH_SEPARATOR } hasText = true yield part.text } } 

Because most messaging channels do not let a bot edit a message after sending it, each paragraph is sent as its own message as soon as the model finishes writing it. For images, every WebP file is re-encoded before it is sent to a channel:

// apps/worker/src/heavy/handlers/channel-safe-image.ts (simplified) // WhatsApp accepts JPEG/PNG images but treats WebP as a sticker. export async function toChannelSafeImage(image) { const mimeType = image.contentType.split(";")[0]?.trim().toLowerCase() if (mimeType !== WEBP_MIME_TYPE) return image const transparent = await hasTransparency(sharp(image.buffer)) return await encodeChannelSafeImage(sharp(image.buffer), transparent) // PNG or JPEG } 

Result. Replies that involve tool calls arrive as separate, readable messages on every channel, and images generated by Muse display as images on WhatsApp. Images with real transparency stay PNG; everything else, including screenshots that carry an alpha channel but no transparent pixels, becomes JPEG at quality 90, far smaller than the equivalent PNG.

3. Reading SAM 3.1 output

Problem. SAM does not return JSON. Its result is text in a dedicated grammar, streamed over Server-Sent Events (SSE) from the /responses endpoint. Meta provides @meta-sam/parser, an optional package for reading that grammar, but the package is not MIT-licensed, so it cannot be included in ChatbotX, which is an MIT project. There are also two traps: SAM's end coordinates are inclusive, and phone photos usually store their orientation in EXIF metadata.

Solution. Meta allows clients to write their own parser against the documented grammar, which is what we did. The Segment Image step works as follows:

image URL ─► download ─► rotate by EXIF ─► reject > 25 MP ─► POST /responses (sam-3.1, target "shoe", SSE) ─► parse "box;x1=..;y1=..;x2=..;y2=..;w=..;h=.." lines ├─ no box ─► Skip └─ boxes ─► "largest" or "all" ─► crop (sharp) ─► S3 ─► Success any failure ─► Error 

We read the SSE stream with the eventsource-parser library, cap the buffer size, and ignore unknown fields. The parser reads only bounding boxes, skips masks, and validates every box before any cropping happens:

// packages/ai/src/server/sam.ts function parseBox(body, objectId, sourceSize) { // … read fields by key if (width !== sourceSize.width || height !== sourceSize.height) { // SAM saw different pixels than the caller will crop. throw invalidOutput(`frame ${width}x${height} does not match the source image …`) } if (x1 > x2 || y1 > y2 || x2 >= width || y2 >= height) { throw invalidOutput(`box ${x1},${y1},${x2},${y2} is outside its frame`) } // `x2`/`y2` are inclusive. return { objectId, left: x1, top: y1, right: x2 + 1, bottom: y2 + 1 } } 

Result. Crops are pixel-exact: the end coordinates are incremented by 1 because sharp crops with exclusive bounds. If an image were not rotated upright, SAM and sharp would see two different images; the frame-size check above catches exactly that mismatch. A photo without the requested object takes the Skip branch, so the flow can ask for another photo instead of failing. When SAM reports an error inside the stream after the HTTP response has already returned 200, only server_error and rate_limit_exceeded are retried.

4. Retrying without redundant model calls

Problem. Every model call costs money, and BullMQ retries failed jobs automatically. The worst case is a job that has already received a result from Meta but dies while writing it to custom fields: the retry calls Meta again and may re-read an input field that was just overwritten.

Solution. Two safeguards:

// packages/ai/src/server/muse-errors.ts /** Only Meta 408/429/5xx are worth a second paid attempt. */ export function isRetryableMuseStatus(status: number): boolean { if (PERMANENT_STATUSES.has(status)) return false // 400, 401, 403, 406, 413 return status === 408 || status === 429 || status >= 500 } // apps/worker/src/heavy/worker.ts (simplified) // A retry after a partial write replays the stored values: no provider // call, so no provider slot either. const replayed = attempt?.action === "replay" ? { status: "success", outputValues: attempt.outputValues } : undefined const result = replayed ?? (await runProviderStep(jobData)) 

The first safeguard applies to speech-to-text and SAM, the two clients that call Meta directly: only errors that can resolve on their own (timeouts, rate limits, and server errors) are retried. Their HTTP clients disable built-in retries (retry: 0), so BullMQ is the only place that decides to retry. Speech-to-text also sends the BullMQ job ID as Meta's sessionId, so the value stays the same across retries. The second safeguard applies to every heavy step: results are stored, and the step is marked as writing, before anything is written to custom fields.

Result. A heavy step that dies partway through a write replays its stored results instead of calling Meta again. For speech-to-text and SAM, permanent errors such as an invalid key or an oversized request are never retried. These safeguards reduce, but do not eliminate, repeated calls: if the failure happens before any result exists, BullMQ may still call the provider again.

Define Muse steps through the public API, CLI, and MCP

Each step is defined by a single Zod schema that drives the configuration dialog in the UI, validates the step in the worker, and backs the flow API. The .describe() text appears in the JSON Schema that agents read from GET /v1/schemas/flow-spec:

// packages/flow-config/src/steps/ai-segment-image.ts (excerpt) target: z.string().trim().min(1).max(160).describe( 'Short concrete noun phrase naming the object to find, e.g. "red car" or "product box". Not a command or question.', ), states: z.tuple([successStateSchema, skipStateSchema, errorStateSchema]).describe( "Exactly [success, skip, error]: success = object found, skip = no object found, error = processing failed.", ), 

The MCP server does not expose a separate tool per step, such as segment_image. Instead, an agent such as Claude or Muse Code uses the flow operations: flows.validate to check a flow, flows.create to create it, and flows.publish to publish it. This lets the agent build the example flow above without opening the UI.

What's next

• Open-weight models: run Muse Spark Open on Baseten or Fireworks and connect it to ChatbotX as an OpenAI-compatible provider, a provider type the platform already supports.
• SAM masks: the step currently uses bounding boxes only. Masks would enable background removal and product images with transparent backgrounds.
• Direct voice note support: convert audio from messaging channels to WAV before sending it to Muse, so a customer can simply send a voice note.
• Quality monitoring: a dashboard showing Success, Skip, and Error rates and cost for each Muse step.

Conclusion

Calling the Muse API was only the first step. Most of the work went into making Muse behave reliably inside a real messaging system: handling three differences from OpenAI, delivering replies and images that display correctly on each channel, reading SAM output without depending on a package that is not MIT-licensed, and retrying without redundant model calls.

The result is that a single API key enables Muse in both AI Agents and flow steps, on every channel ChatbotX supports. Other AI agents can build the same flows through the public API and MCP without opening the UI. The main limitation today is that Speech to Text accepts only WAV files and does not yet take voice notes directly from messaging channels.

Try it

You need PostgreSQL, Redis, S3-compatible storage, and a Meta Model API key. Muse needs no dedicated service; the builder and one worker process are enough.

git clone https://github.com/ChatbotXIO/ChatbotX && cd ChatbotX pnpm install docker compose up -d # Postgres, Redis, storage pnpm --filter @chatbotx.io/database db:migrate pnpm --filter builder dev # UI on http://localhost:3123 pnpm --filter worker dev # integration + heavy workers 

Open Settings → Integrations → Muse and paste your API key. Then create a flow with a Segment Image step and set its target to something like red shoe. Input images can be up to 25 megapixels; recordings must meet the WAV requirements described under Speech to Text.

To test the Muse provider and the SAM parser without an API key, run the unit tests:

pnpm --filter @chatbotx.io/ai exec vitest run \ __tests__/muse-provider.test.ts __tests__/sam.test.ts 

Source code

• Repository: github.com/ChatbotXIO/ChatbotX (MIT)
• Muse provider, image generation and editing, speech-to-text: PR #1586
• Segment Image with SAM 3.1: PR #1611