Heterogeneous compute nodes let one Psyclone AIOS system span many machines — mixing operating systems, CPU architectures and hardware capabilities — without changing a line of module code. A node is a Psyclone engine running on one computer; a space is a separate OS process inside a node; a component is a module, whiteboard or catalog placed onto one of them. Placement is a declaration in the PsySpec, not an architectural commitment. If you are building a real-time AI system where a GPU box does inference, a Windows machine drives a vendor SDK, and a small Linux board sits next to the sensors, this is the mechanism that makes those three machines behave as one system.
Why it matters
Real-time AI systems are rarely homogeneous. Cameras and microphones are attached to specific machines. Motor controllers and industrial buses often demand a particular OS. Model inference wants the box with the accelerator. Vendor SDKs are frequently Windows-only. The usual outcome is a set of separate programs glued together with bespoke sockets, message formats and reconnection logic — and that glue becomes the least reliable, least observable part of the product.
Psyclone removes that glue by making the machine boundary a placement attribute. Because components never call each other — they post typed messages and subscribe to types — a module does not know, and does not need to know, whether its correspondent is a thread in the same process, a process on the same host, or an engine on another continent. Moving a module from the GPU box to the sensor board is a PsySpec edit. The failure mode this eliminates is the one that costs most: rewriting working code because the deployment topology changed.
The second consequence is fault containment. Each space is its own OS process. If a component in a space crashes, the node restarts that space and recreates its components, and the components’ private data survives; the rest of the system keeps running. That is what allows commercial-grade code and experimental code to coexist in one deployment — an unstable vision prototype or a wayward Python interpreter can be quarantined in its own process rather than being allowed to take down the control loop beside it.
The third is bandwidth discipline. Distribution is only a win if you distribute in the right place. Psyclone’s design rule is explicit: bandwidth-heavy pipelines stay on one node, and only distilled results travel between nodes. Raw frames get processed on the machine the camera is plugged into; compact detections and classifications cross the network. Keeping that rule is what preserves the platform’s sub-100µs message bus behaviour for the local hot paths that need it.
How it works
Every Psyclone instance starts a local node, which manages communication and bookkeeping: subscriptions, services, interfaces, local components and local process spaces. A distributed system is one master node — the instance started with the PsySpec — plus any number of satellite nodes on other machines, which the master configures remotely. A satellite is simply Psyclone started with no spec, optionally with an explicit port=; it waits idle until it is claimed. The master’s own node is implicit and is referred to as Main.
There are three layers to manage, and they nest. A node is a machine boundary: one engine process per machine, with its own address, port, network links and — optionally — its own libraries. A space is an OS process within a node; every node automatically creates the internal space Root, in which all its components run as threads. A component is a module, whiteboard or catalog, placed with node= and space=. Anything with no node= attribute runs on the startup node.
Heterogeneity is handled at two points. Crank libraries are distributed globally by default, but a <node> element can carry a child <library> so a machine that needs its own build gets it; the .dll versus .so difference is resolved automatically from the library name. And spaces can be declared type="external", letting a third-party program — a GUI, a vendor runtime, a simulator — host Psyclone components inside its own process: the external program links CMSDK, creates a PsySpace, connects with the system ID, and fetches a PsyAPI per crank, which it then uses exactly like an internal module crank. External-space crashes are tolerated and reconnectable in the same way internal ones are.
Isolation is not free, and the guides are blunt about it: messages between spaces cross a process boundary via shared memory, and messages between nodes cross the network. Chatty component groups belong in the same space. Note also that DataMessage timestamps are synchronised across computers to microsecond resolution by the platform, so cross-machine ordering and latency analysis remain meaningful.
Multi-node is one system spread out. If you instead want several independent Psyclone systems to cooperate — one per robot, one per site — the mechanism is the remote query: an ordinary <query> plus host and port. The remote component answers as if it were local and the reply is routed back transparently; the receiver can tell the call came from elsewhere by the INTERSYSTEM_IDENTIFICATION, INTERSYSTEM_ADDRESS, INTERSYSTEM_PORT and INTERSYSTEM_SOURCENAME entries stamped onto the query.
Psyclone AIOS 2.2.0 hardened this path considerably. Bring-up is now multi-node aware: the supervisor groups components by node=, defers nodes it has not yet seen, routes Psyclone.Builder.* traffic to the target node, and forwards cross-node recipes and triggergroup joins, with ID-reservation adoption ordering, context ID-clash tie-break and SYNC_COMPONENT_INFO confirmation. A self-checking test suite, test=psymulti, brings up three real nodes in separate OS processes under supervisor orchestration on every supported platform. Psyclone.Ready now fires only once all components actually exist — across all nodes — and the node broadcasts Psyclone.SystemStatus on a five-second cadence.
How to use it
A two-machine deployment — a Linux sensor host and a Windows machine running a vendor SDK — is a handful of lines. Note that placement works identically for whiteboards and catalogs: storage components can live wherever the data is.
<psySpec>
<!-- satellite machines; the local master node is implicit ("Main") -->
<node name="SensorHost" address="192.168.20.20" port="11000" />
<node name="WinHost" address="otherhost" port="10000">
<library name="VendorLib" library="Vendor" />
</node>
<!-- crash-isolate the experimental tracker on the sensor host -->
<space node="SensorHost" name="TrackerSpace" />
<!-- raw frames never leave SensorHost; only detections cross the link -->
<module name="Camera" node="SensorHost">
<post name="Frames" type="input.video.raw" />
</module>
<module name="Tracker" node="SensorHost" space="TrackerSpace">
<trigger name="In" type="input.video.raw" />
<post name="Out" type="percept.human.detected" />
</module>
<module name="Actuation" node="WinHost">
<crank name="Drive" function="VendorLib::Drive" />
<trigger name="In" type="percept.human.detected" />
</module>
<whiteboard name="Percepts" node="Main" />
</psySpec>- Start an idle node on each worker machine:
./bin/linux64/Psyclone port=11000(no spec). It waits to be claimed. - Declare each machine as a
<node>in the master spec, with a fixed, reachable address and port. - Place components with
node=, keeping high-volume pipelines together on the machine that owns the hardware. - Add a
<space>per crash-prone or experimental component. A plainly-declared space is created on every node; restrict it withnode=. - Confirm every node can load the libraries its components need — global by default, per-node override available.
- Start the master with
spec=, then verify in PsyProbe: the Nodes section shows per-node performance, spaces and activity live, and the System Node Communication matrix shows which message types are crossing node links. - Rehearse failure: kill a space and a satellite node in staging and confirm the restart behaviour matches your expectations before production.
The bundled Examples/multinode.xml and multinodesimple.xml specs are runnable references.
When to use it / when not
| Use nodes and spaces when… | Think again when… |
|---|---|
| Hardware dictates placement: sensors, actuators, accelerators or vendor SDKs are tied to specific machines or operating systems. | Two components exchange high-volume data at high rates. Keep them in one space; a process or network boundary means copies. |
| You need crash isolation for experimental, third-party or Python-crank code alongside production control loops. | You are hoping for automatic load balancing. Placement is spec-driven; there is no hands-off rebalancer today. |
| A single machine cannot supply the CPU, memory or I/O the whole system needs. | The system fits comfortably on one machine. Distribution adds ports, links, clocks and failure modes to operate. |
| An external program — GUI, simulator, vendor runtime — must host components in its own process. | You want fully independent systems rather than one system. Use remote queries between separate Psyclone instances instead. |
Two caveats stated plainly. First, the migration="yes" attribute is parsed and sets a component’s migrate-allow flag, and much of the moving machinery exists in the engine, but automatic runtime migration is not a hands-off feature: the System Guide marks it Stub today, with move-to-node planned as a builder-owned action (SYS_BUILDER_MOVE) on the roadmap. Verify against your build before relying on it; for now, rebalancing means editing placement and restarting the affected spaces or nodes. The same applies to priority, which is stored but has no scheduling effect. Second, node-to-node and remote-query links carry your dataflow: apply the platform’s SSL configuration, where verification is on by default and self-signed certificates require an explicit allowselfsigned="yes". Design modules to be movable regardless — keep state in private data or catalogs, not in process-local globals.
Read the docs
- User Guide 10 — Distributed Systems: the developer’s view of nodes, spaces, external spaces and remote queries, with the PsySpec fragments and the
PsySpaceconnect/retry pattern in C++ and Python. - System Guide 8 — Distributed Admin: satellite start-up, the node/space/component layer table, the keep-dataflow-local topology rule, load balancing and the production checklist.
- User Guide 4 — Modules: cranks and libraries, one-shot versus continuous execution, private data, Python cranks (which automatically get their own space) and external modules.
- User Guide 13 — PsySpec XML Reference: every attribute of
<node>,<space>and<library>, with honest implemented / stub / roadmap status per attribute. - System Guide 5 — Security & SSL: securing node links and remote-query links, verification defaults and CA overrides.
- System Guide 11 — Roadmap Features: what is Shipped, Roadmap or Stub, including node-local builders and the current status of runtime migration.
- CMSDK API reference:
PsySpace,PsyAPIandDataMessagefor external-space hosts and crank authors.
Placement is one half of the story; what runs at any moment is the other. See Global Contexts for how a system switches whole sets of active components at runtime, Ephemeral Compute for components created and torn down on demand, and Builders & the Startup Supervisor for the 2.2 bring-up machinery that now orchestrates multiple nodes. Or start at the Psyclone AIOS overview.