LLM Client Layer

• Client
• ToolClient
• Message
• ToolDef
• ToolCall
• Usage
• OpenAIClient
• MockClient

The internal/llm package is astrolabe's thin abstraction over upstream inference providers. It exists so the rest of the pipeline — chiefly the agent loop in Agent Core — can talk to any OpenAI-compatible chat completions API (OpenAI, OpenRouter, Kimi, Qwen, Mistral, …) through two small interfaces, while tests run the exact same call sites against a scripted in-process fake. The package deliberately owns only the wire protocol: prompt construction, tool dispatch, and budget accounting live elsewhere package doc.

How it works

The two interfaces

Everything starts from Client, a one-method interface whose Chat sends a message list and returns the assistant's reply text Client. On top of it sits ToolClient, which adds ChatTools: the provider's native function-calling format, where tools go out as JSON-Schema function declarations and calls come back as structured entries on the assistant message ToolClient. This split exists because not every backend supports native tools — the agent pipeline requires ToolClient, and transports that only have plain chat (such as ACP agents) must adapt Chat to it rather than fake tool calling ToolClient doc.

ChatTools returns a full Message, not a string, and has a deliberate failure mode: a reply that carries neither text nor tool calls comes back together with ErrNoResponse, so the caller can inspect FinishReason and see why — for example "length", meaning the output budget was consumed by reasoning before any answer was produced ToolClient doc. ErrNoResponse is the package's single sentinel error, returned whenever a model response cannot be used ErrNoResponse.

The data model

The message and tool types mirror the OpenAI wire format closely but stay provider-neutral:

Usage values accumulate with Add, which is how a client's running total is kept across requests Usage::Add.

OpenAIClient: one request, end to end

OpenAIClient is the production implementation OpenAIClient. Its configuration fields cover the usual knobs — BaseURL, APIKey, Model, an injectable HTTPClient, optional MaxTokens/Temperature, a ReasoningEffort passthrough (which OpenRouter-style gateways map onto their own settings), and an Attempts retry bound defaulting to 3 OpenAIClient fields.

Both Chat and ChatTools funnel into one private method, complete, which does the real work OpenAIClient::complete. Chat simply calls it without tools and errors with ErrNoResponse if the reply is empty text OpenAIClient::Chat; ChatTools additionally serialises the ToolDef catalogue into the request and explicitly asks for parallel tool calls, so one turn can carry several lookups OpenAIClient::ChatTools.

Inside complete, the flow is:

1. Span setup. The request runs as a chat {model} client span following GenAI conventions, with attributes for model, tool choice, reasoning effort, max tokens, temperature, and — when content capture is enabled — truncated input messages span setup.
2. Retry loop. Up to Attempts tries; failures classified as transient (network errors, timeouts, HTTP 429/5xx) are retried with a linear backoff of attempt × 2s, and each retry is recorded as a span event retry loop.
3. HTTP. post sends one attempt to BaseURL + "/chat/completions" with a custom User-Agent (some providers behind bot protection reject Go's default header) and Bearer auth OpenAIClient::post. Network errors, timeouts, 429 and 5xx come back wrapped in an internal errTransient marker, which is what the retry loop checks errTransient.
4. Accounting and response shaping. On success the provider-reported usage is added to the client's running total (readable via Usage()) and stamped onto the span as GenAI token metrics usage accounting. The first choice is decoded into a Message, with reasoning text normalised across the two gateway spellings (reasoning_content vs reasoning) and tool calls unpacked into ToolCall values response shaping.

An empty Choices array also yields ErrNoResponse empty choices.

MockClient: scripted tests without API keys

MockClient stands in for the model in tests. Its Handler receives the exact message list a real provider would and returns reply text; an optional ToolHandler does the same for ChatTools and returns a full assistant Message, so tests can script tool-call rounds MockClient. When ToolHandler is nil, ChatTools falls back to wrapping Handler's text in a plain assistant message MockClient::ChatTools. Every call is recorded behind a mutex and inspectable via Calls, and the NewScriptedMock helper produces a client that hands back responses in order, then fails with a caller-chosen exhaustion error (defaulting to ErrNoResponse) NewScriptedMock.

Relationships

graph TD
    subgraph internal/llm
        Client[`Client` interface] --> ToolClient[`ToolClient` interface]
        OpenAIClient[`OpenAIClient`] --> Client
        OpenAIClient --> ToolClient
        MockClient[`MockClient`] --> Client
        MockClient --> ToolClient
        Message[`Message`] --> ToolClient
        ToolDef[`ToolDef`] --> ToolClient
        Usage[`Usage`] --> OpenAIClient
    end
    Agent["internal/agent loop"] -->|"ChatTools rounds"| ToolClient
    OpenAIClient --> Telemetry["internal/telemetry spans & metrics"]

The agent's ReAct loop is the primary consumer: each reasoning round is a ChatTools call carrying the tool catalogue from Agent Tools & Budget-style tool registrations, and the returned Message drives whether the agent answers or dispatches tools — see Agent Core. Telemetry-wise, every real request emits a GenAI-convention span and token metrics through Telemetry.

Reference

Generated by astrolabe from commit ecc180b3 (tag 0.4.0).