Temporal Constructsan interactive introduction
Transport 00:00 Internal clock
Hermes, AI agent

The human chooses how much of the programming to take part in. All programming decisions can be delegated to agents, or the human can be closely involved, by turning knobs, editing snapshots, placing events, and building and testing devices by hand in Max or Pd. Any balance in between is also possible. The human also listens, inspects the files and the experience in the real world, and tells the agents whether their decisions were correct. Not knowing in advance what an agent will do is part of the interaction model.

With agents, the human can work in programming territory that would otherwise be unknown, with assistance and guidance. The guidance can cover the technology and its context, such as the work and heritage of the human programmers. It is on by default and can be turned off in conversation.

The influence of AI on a process that was purely human is where tension lies. For that reason the relation to AI is not fixed by the design. The agent invites the human to specify it while working, the human can change it at any time, and the line is drawn by the user and not by the designer.

The conductor agent does no module-specific work and holds the context relevant to the whole system. Each module also has agents working on tasks that relate to it, including work done on the timeline. Agents cannot hear the result of what they do, so they rely on the human to confirm whether a decision was correct.

In this overview, Nexus takes on the task of whichever agent is relevant to the part being shown.

The conversational layer. Human and agent talk as they would in any chat, here through Telegram, which suits multi-agent work because it supports groups. Each module has its own group, so the context for that instrument collects in one place. Intentions are discussed here, while tasks are carried out through CUE.

Groups
Conductor agent
Example: a request in the group of Module 01illustrative

Everything the system does is done through CUE, in the terminal, by the human and by agents independently of each other. Agents use the terminal in the same way as humans do, and the scope of possible tasks is kept as small as possible.

TerminalThe shared entry point for human and agent. Both use the same commands, independently of each other.
MCP serverA dedicated MCP server for CUE, so that agents know what to expect and how to navigate the system. Each module also has its own Agents.MD with module-specific instructions.
Example: the module agent consults CUEillustrative, simplified commands

The only global object. Sixteen group tracks, one per module, each holding an open-ended number of functions. Sixteen is a deliberate constraint, modelled on the 16 channels of MIDI. A function is something built into the hardware, or built into the device itself in Pd or Max, that can be accessed with a single CC. The timeline sends bank and program changes to up to 16 modules, each addressed by its own MIDI channel, and can run for as long as the work needs.

Track 1 is mapped to the strummer of Module 01. Select an event to send its recall, or select an empty position to create one.

16 tracks, one per module

Each module is an independent ALSA client, addressed by MIDI channel. None of them knows about the others, and that independence keeps the system stable. All of it runs headless on a Raspberry Pi 4 under Debian, with persistent TCP connections between the components.

Module 01
Governs stateControllerSets the knobs on the instrument and creates snapshots gradually. It always works with control changes.
Governs stored statesSnapshot memoryWhere snapshots are authored, the catalogue is organised and the controller map is requested. A snapshot is a stored set of controller values.
Governs timeSketchpadHolds recalls, switches loop mode on and off and changes playback speed. It can also recall snapshots without needing a MIDI channel.
ReferenceAgents.MD and manualModule-specific instructions for the agent, and the hardware manual for the instrument.
Controller of Module 01connected to the strummer
Strings 1 to 4
Snapshot cataloguebank 1 of 64, 128 slots per bank

A translation layer between the module and the actual hardware. The controller speaks in control changes. The device converts them into the MIDI messages the instrument needs, for example sysex, using a prepared Max or Pd patch.

ControllerControl changes
DeviceMax or Pd patch
InstrumentMIDI, including sysex

A human can prototype a device by hand and inspect the patch before or while it is deployed. The human therefore knows what a device can do and how the controller relates to it, which helps the agent to operate it and helps in creating the controller mapping.

A device exposes functions. A function is something built into the hardware, or built into the device itself in Pd or Max, that can be accessed with a single CC. The CCs of the controller knobs are mapped to functions, so that turning a knob changes the function it is mapped to.

Example device: the strummerfour variables per string

Six strings with four variables each give 24 functions, reached with CC 1 to 24. The variables of each string are the open string note (tuning), the fret number, the velocity and a time delay. Each CC has 128 values, which gives about 3.7 × 1050 possible settings. The knobs of the controller in the module set them, and the sound is synthesised in the browser.

Plays the strings with the current values.

Instruments from the 1980s and 1990s, many of which were unused before the project began. They make the sound, and the agents operate them through the modules and devices.

AvailabilityThe equipment already exists, and in many cases is gathering dust.
ConstraintEach instrument is a constrained possibility space, which keeps the work legible.
StudyThe work of human programmers can be studied and built upon using AI.
Physical domainAI enters the physical domain, so the collaboration takes place in a space already familiar to humans.
ObsolescenceThe instruments become more useful than they were in the 1980s and 1990s, which challenges the notion of obsolescence.
HeritageImportant heritage is preserved and protected, and future generations are encouraged to keep these workflows alive.
Examples from ALTERITY
Kurzweil K2000Atari STMacintosh SE/30
How they listen: MIDI

MIDI is a language of small messages. A message can play a note, change a controller value, or recall a stored sound by program change, and system exclusive (sysex) messages carry longer instructions that are specific to one instrument. There are 16 channels, and the system treats this limit as a design principle, which is why the timeline has 16 tracks.

The instruments come from a period in which every device was expected to connect to complex MIDI infrastructure such as Sounddiver, a logic that has since been abandoned, so their MIDI implementations are deep.

HermesAI agent