Writing a model provider for the Ortha copilot

A new backend is one factory returning a ModelProvider, plus one spec that runs the conformance kit.

Check first: the OpenAI-wire adapter already reaches Ollama, vLLM, llama.cpp, LM Studio, LiteLLM, OpenRouter, Azure and OpenAI. Write a package only when the wire format differs.

The port

From @orthacms/copilot-domain (no NestJS, no Drizzle, no vendor SDK):

interface ModelProvider {
    models(): readonly string[];                         // first = default
    capabilities(model?: string): Promise<ModelCapabilities>;
    stream(request: ModelRequest, signal?: AbortSignal): AsyncIterable<ModelStreamEvent>;
}

interface ModelRequest {
    model?: string;                        // omitted = your first model
    system?: string;
    messages: readonly ModelMessage[];     // { role: 'user' | 'assistant', content: blocks[] }
    tools?: readonly ModelTool[];          // { name, description, inputSchema }
    maxOutputTokens: number;
}

type ModelStreamEvent =
    | { type: 'text-delta'; text: string }
    | { type: 'tool-call'; id: string; name: string; input: unknown }   // input parsed
    | { type: 'done'; stopReason: ModelStopReason; usage: ModelUsage };

type ModelStopReason = 'end' | 'tool_use' | 'max_tokens' | 'refusal' | 'aborted';

interface ModelUsage {
    inputTokens: number;
    outputTokens: number;
    cachedInputTokens?: number;
    cacheWriteInputTokens?: number;
}

interface ModelCapabilities {
    model: string;
    toolCalling: boolean;
    streaming: boolean;
    vision: boolean;
    contextWindow: number;
    maxOutputTokens: number;
}

Helpers from the same package:

resolveModel(requested, available) — returns the model or throws UnknownModelError (NoModelsConfiguredError if the list is empty).
abortedEvent() — the done event for a cancelled call, zero usage.
isAbortError(error, signal) — whether a thrown error is the cancellation.

The contract

Exactly one done, last.
Text deltas concatenate to the full answer.
Tool calls are whole: an id, a name, and parsed input (not a JSON string).
Unknown model → UnknownModelError, and no request is sent.
Abort ends the stream, never throws out of it: emit abortedEvent().
An aborted call reports zero usage, even if text had streamed.
capabilities() with no argument describes the first model.

A complete provider

For a hypothetical backend “Acme” that streams newline-delimited JSON:

// packages/copilot-provider-acme/src/acme-provider.ts
import {
    abortedEvent,
    isAbortError,
    resolveModel,
    type ModelCapabilities,
    type ModelProvider,
    type ModelRequest,
    type ModelStopReason,
    type ModelStreamEvent
} from '@orthacms/copilot-domain';

export interface AcmeProviderConfig {
    baseUrl: string;
    apiKey: string;
    models: readonly string[];
    fetch?: typeof globalThis.fetch; // injection seam for tests
}

// The backend's wire format — one JSON object per line.
type AcmeLine =
    | { type: 'text'; text: string }
    | { type: 'tool'; id: string; name: string; arguments: string }
    | { type: 'end'; reason: 'stop' | 'tool' | 'length'; usage: { in: number; out: number } };

const STOP_REASONS: Record<'stop' | 'tool' | 'length', ModelStopReason> = {
    stop: 'end',
    tool: 'tool_use',
    length: 'max_tokens'
};

export function createAcmeProvider(config: AcmeProviderConfig): ModelProvider {
    const doFetch = config.fetch ?? globalThis.fetch;
    const url = `${config.baseUrl.replace(/\/+$/, '')}/v1/generate`;

    return {
        models: () => config.models,

        async capabilities(model?: string): Promise<ModelCapabilities> {
            return {
                model: model ?? config.models[0],
                toolCalling: true,
                streaming: true,
                vision: false,
                contextWindow: 128_000,
                maxOutputTokens: 8_192
            };
        },

        async *stream(
            request: ModelRequest,
            signal?: AbortSignal
        ): AsyncIterable<ModelStreamEvent> {
            // Throws UnknownModelError before any request is issued.
            const model = resolveModel(request.model, config.models);
            if (signal?.aborted) {
                yield abortedEvent();
                return;
            }

            try {
                const response = await doFetch(url, {
                    method: 'POST',
                    signal,
                    headers: {
                        'content-type': 'application/json',
                        authorization: `Bearer ${config.apiKey}`
                    },
                    // Translate messages and tools to your wire format here.
                    body: JSON.stringify({
                        model,
                        system: request.system,
                        messages: request.messages,
                        tools: request.tools,
                        max_tokens: request.maxOutputTokens
                    })
                });
                if (!response.ok || !response.body) {
                    throw new Error(`Acme answered HTTP ${response.status}`);
                }

                const decoder = new TextDecoder();
                let buffered = '';
                for await (const chunk of response.body) {
                    buffered += decoder.decode(chunk, { stream: true });
                    const lines = buffered.split('\n');
                    buffered = lines.pop() ?? '';

                    for (const line of lines) {
                        if (!line.trim()) continue;
                        if (signal?.aborted) {
                            yield abortedEvent();
                            return;
                        }
                        const event = JSON.parse(line) as AcmeLine;
                        if (event.type === 'text') {
                            yield { type: 'text-delta', text: event.text };
                        } else if (event.type === 'tool') {
                            yield {
                                type: 'tool-call',
                                id: event.id,
                                name: event.name,
                                input: JSON.parse(event.arguments)
                            };
                        } else {
                            yield {
                                type: 'done',
                                stopReason: STOP_REASONS[event.reason],
                                usage: {
                                    inputTokens: event.usage.in,
                                    outputTokens: event.usage.out
                                }
                            };
                            return;
                        }
                    }
                }
                if (signal?.aborted) {
                    yield abortedEvent();
                    return;
                }
                throw new Error('Acme closed the stream without an end event');
            } catch (error) {
                if (isAbortError(error, signal)) {
                    yield abortedEvent();
                    return;
                }
                throw error;
            }
        }
    };
}

If your client needs credentials, build it on first use, not in the factory, so a deployment that has not configured it still boots.

The conformance spec

runModelProviderConformance returns a report (null per passing check, otherwise a sentence). It needs no test framework; turn each of the twelve MODEL_PROVIDER_CONFORMANCE_CHECKS into a test:

// packages/copilot-provider-acme/src/acme-provider.spec.ts
import {
    MODEL_PROVIDER_CONFORMANCE_CHECKS,
    runModelProviderConformance,
    type ModelProviderConformanceReport,
    type ModelRequest
} from '@orthacms/copilot-domain';
import { createAcmeProvider } from './acme-provider';

let issued = 0;

// Each factory call gets a fresh provider over a stubbed transport.
const acme = (lines: object[]) =>
    createAcmeProvider({
        baseUrl: 'https://acme.test',
        apiKey: 'test-key',
        models: ['acme-large', 'acme-small'],
        fetch: async () => {
            issued += 1;
            return new Response(lines.map((l) => JSON.stringify(l)).join('\n') + '\n');
        }
    });

const request: ModelRequest = {
    messages: [{ role: 'user', content: [{ type: 'text', text: 'Hello?' }] }],
    maxOutputTokens: 256
};

describe('Acme ModelProvider conformance', () => {
    let report: ModelProviderConformanceReport;

    beforeAll(async () => {
        report = await runModelProviderConformance({
            text: () => ({
                provider: acme([
                    { type: 'text', text: 'Hello, ' },          // at least two deltas
                    { type: 'text', text: 'world.' },
                    { type: 'end', reason: 'stop', usage: { in: 12, out: 4 } }
                ]),
                request,
                answer: 'Hello, world.'
            }),
            toolCall: () => ({
                provider: acme([
                    { type: 'tool', id: 'call_1', name: 'admin_content_types', arguments: '{}' },
                    { type: 'end', reason: 'tool', usage: { in: 12, out: 9 } }
                ]),
                request
            }),
            unknownModel: 'acme-does-not-exist',
            requestsIssued: () => issued // proves "no request" on an unknown model
        });
    });

    it.each(MODEL_PROVIDER_CONFORMANCE_CHECKS)('%s', (check) => {
        expect(report[check]).toBeNull();
    });
});

Register it

// apps/server/src/plugins.ts
import { createAcmeProvider } from '@acme/copilot-provider-acme';

CopilotPlugin({
    providers: [
        ...copilotProviders(config),
        ...(process.env.ACME_API_KEY
            ? [{
                  name: 'acme',
                  provider: createAcmeProvider({
                      baseUrl: 'https://api.acme.example',
                      apiKey: process.env.ACME_API_KEY,
                      models: ['acme-large', 'acme-small']
                  })
              }]
            : [])
    ],
    config: config.plugins.copilot
});

Order is the default: the first registration serves a run that names no provider. Put yours first to make it the house default.
Register only when configured. A keyless backend that fails on the first message is worse than an absent one.
The name is permanent. Runs, the picker and saved conversations refer to it.

Check it is offered:

curl -s -b cookies.txt http://localhost:3000/api/copilot/models
# {"items":[..., {"provider":"acme","model":"acme-large"}, {"provider":"acme","model":"acme-small"}]}

Only an adapter may import a vendor SDK

The copilot’s domain and server packages never do. Keep the SDK inside your provider package.