LLM Connections

New in Psyclone AIOS v2.2 — shipped 22 August 2026.

LLM Connections give CMSDK a first-class, declarative way to talk to language models. A named <llm> element in your spec — with an <llmvendor> descriptor inside it — becomes an LLMConnection object your code can use for a blocking request/response exchange (mode A) or token-by-token streaming (mode B), against OpenAI, Anthropic, AWS Bedrock, Google, or any custom HTTP endpoint. If you are building a real-time system that needs a language model in the loop — a voice agent, a planning layer, a supervised code assistant — this is the plumbing that keeps credentials out of your spec and vendor quirks out of your application code.

Why it matters

Most systems that call an LLM accumulate a private pile of glue: a vendor SDK per provider, an HTTP client, an SSE parser someone half-wrote, and an API key pasted into a config file. That glue is where outages and leaks live. LLM Connections replace it with one audited code path inside CMSDK itself — no third-party HTTP libraries, riding the same native transport the rest of the platform uses.

The failure mode it removes most directly is secret leakage. An inline API key in a spec is not merely discouraged — it is a typed configuration error (LLM_ERR_CONFIG_INLINE_SECRET) and the connection refuses to configure. Credentials are only ever references, resolved at connect time from an environment variable or a file, and the resolved value never appears in logs, error text or status output. A spec file can be committed, diffed and shared without a security review of its contents.

The second failure mode is silent vendor divergence. Streaming replies arrive in three genuinely different wire formats — SSE, raw HTTP chunked transfer, and AWS’s binary event-stream framing. Code that branches on vendor names rots as providers change. Here the transport is resolved once, at configure time, from the vendor descriptor; callers just call interactStream() and the connection drives the right decoder internally, including per-message CRC validation on the AWS format.

How it works

An <llm> element declares a named connection. Inside it, an <llmvendor> block carries the vendor type (openai, anthropic, bedrock, google, or custom), the endpoint URL, HTTP headers with %apikey%/%model% substitution, a <requesttemplate> that shapes the outgoing JSON, a <responsetemplate> JSON path (for example choices[0].message.content) that extracts the reply text, and a <models> list with a default. The LLMConnection class parses all of this into typed results — every misconfiguration returns a specific error code rather than crashing or guessing.

Two interaction modes are shipped. Mode A, interact(), is one blocking request/response exchange over the CMSDK HTTP client: the request is shaped from the template, sent, and the reply text extracted via the JSON path. Mode B, interactStream(), sends one request and delivers decoded incremental tokens to a callback as they arrive; the callback can cancel the stream early, and an assembled full reply is available at the end. Under mode B sits a real streaming transport stack: an SSE parser, an HTTP chunked-transfer decoder, a bounded byte-source pump, and an AWS event-stream frame parser that validates the prelude and per-message CRCs. The transport is chosen from <llmvendor transport="sse|eventstream|chunked">, or by vendor default (Bedrock streams event-stream; the others default to SSE).

Credential references take three forms: apikeyref="env:NAME" reads an environment variable; apikeyref="file:/path" reads a single-line key file (trailing whitespace trimmed); a bare name is the legacy form (environment variable, else an entry in an optional secrets file). An unresolvable reference fails typed at connect() with LLM_ERR_SECRET_UNRESOLVED. Each connection also keeps a stats record — uptime, bytes, request count, tokens in/out, streamed chunks, cost — retained after disconnect. Per-connection constraint fields (max tokens per request, requests per minute, concurrency, cost ceiling) are parsed and stored in 2.2; enforcement is a later step. Because the endpoints are HTTPS, SSL client verification matters: 2.2 verifies peer certificates against the OS trust store, and on Windows now imports the Windows ROOT system store so verification works on end-user machines, not just developer boxes.

How to use it

<llm name="main" apikeyref="env:OPENAI_API_KEY"
     maxtokensperrequest="4096" maxrequestspermin="60">
  <llmvendor type="openai"
             endpoint="https://api.openai.com/v1/chat/completions">
    <header name="Authorization" value="Bearer %apikey%"/>
    <requesttemplate>{"model":"%model%","messages":[...]}</requesttemplate>
    <responsetemplate>choices[0].message.content</responsetemplate>
    <models default="gpt-4o">
      <model name="gpt-4o"/>
      <model name="gpt-4o-mini"/>
    </models>
  </llmvendor>
</llm>

1. Declare the <llm> connection as above and export the referenced environment variable (or point apikeyref="file:..." at a key file) on the machine that will run it.
2. In C++, call configure() (or configureFromString()), then connect() — this validates config and resolves the secret; endpoint reachability is checked lazily on the first request.
3. Optionally selectModel("gpt-4o-mini") to switch within the declared model list.
4. Call interact(input, reply) for a blocking exchange, or interactStream(input, callback, userData) for incremental tokens.
5. Check stats() for tokens, bytes and request counts; reset() ends the connection but retains config and the final stats record.

When to use it / when not

• Use it when a CMSDK-based system needs LLM calls with clean credential handling, one code path across vendors, and streaming that survives vendor wire-format differences.
• Use it for supervised assistive loops — an LLM proposing content that your code validates before acting. Supervised, not autonomous: the roadmap’s LLM-tier builder (a model that writes and builds module code on request) is not shipped.
• Not yet: constraint enforcement (token/rate/cost ceilings are parsed but not enforced in 2.2), global="true" master-node proxying to nodes without internet access, preloaded per-connection contexts, and the PsyProbe LLM panel — all roadmap.
• Not for bulk offline batch inference — mode A is blocking and mode B is one stream per call; this is a real-time integration primitive, not a job queue.

Read the docs

• CMSDK API reference — the LLMConnection class, typed result codes, stream transports and stats.
• System Guide 5 — Security & SSL — certificate verification, allowselfsigned scopes, custom CAs via cafile/capath.
• User Guide 11 — Services & Interfaces — how services and network interfaces expose a Psyclone system.
• System Guide 11 — Roadmap — what is Shipped vs Roadmap around <llm>, including global proxying and the LLM-tier builder.

---

LLM Connections is one of the 2.2 architecture pillars. See also Builders & Supervisor — the other headline 2.2 capability — and Whiteboards & Dataflow for how model output flows through a system, or return to the Psyclone AIOS overview.