Synapse

A synapse is a ‘smart’ sync database that will automatically synchronise data between two or more connected synapses. It is a more powerful version of a sync database that can be used to synchronise an entire sync database across multiple Volt instances.

When two or more synapses are connected together, the Volt will monitor the underlying sync database for changes and automatically synchronise such that all connected synapses will have the same document content.

This is achieved by requiring that all documents in a given synapse have a schema. Most documents use Yjs shared types at the root (map, array, or text). Binary payloads use a blob document instead: the bytes live in a content-addressed store on the synapse, not in the CRDT.

As a reminder, every Yjs document can have multiple shared types defined at the root level, for example, YMap and YArray and YText. These are accessed using the relevant Yjs API method, for example:

const map = ydoc.getMap("my-map");
const array = ydoc.getArray("my-array");
const text = ydoc.getText("my-text");

By defining a schema for each of these top-level shared types, the Volt is able to deterministically detect when a document is modified and apply the changes to all other connected synapses.

Example

The following example demonstrates how to use a synapse to synchronise AIS vessel data across multiple Volt instances.

Setup

Note that the following example assumes that you have already created two Volts and have acquired connection credentials for them stored in ./volt-1.config.json and ./volt-2.config.json (see the Volt auth CLI command).

For the purposes of this example, we will assume that the Volt DIDs are did:volt:1 and did:volt:2 respectively.

To start, we need to create two synapses using the create-synapse command. Note that we pass the connection credentials for Volt 1 and Volt 2 to the command using the -c switch.

# Create a synapse with the alias @ais-synapse-1 on Volt 1
volt create-synapse "AIS vessel data" --alias ais-synapse-1 -c ./volt-1.config.json

# Create a synapse with the alias @ais-synapse-2 on Volt 2
volt create-synapse "AIS vessel data" --alias ais-synapse-2 -c ./volt-2.config.json

You can also use the fusebox to create synapses.

We now have two synapses, @ais-synapse-1 and @ais-synapse-2. We can connect them together using the create-synapse-connection command.

Below we connect synapse @ais-synapse-1 on Volt 1 to synapse @ais-synapse-2 on Volt 2. Note that we pass the connection credentials for Volt 1 to the command using the -c switch. In this case, the connection is created on Volt 1 and is owned by the identity in ./volt-1.config.json. It will attempt to authenticate on Volt 2 and establish a bi-directional synchronisation.

# Connect synapse @ais-synapse-1 on Volt 1 to synapse @ais-synapse-2 on Volt 2
volt create-synapse-connection @ais-synapse-1 did:volt:2 @ais-synapse-2 -c ./volt-1.config.json

You will need to ensure that Volt 2 has granted volt:database-write permission for @ais-synapse-2 to the Volt 1 identity.

In the event of any connection errors, the synapse will periodically retry to establish the connection until it succeeds.

Schema definition

Next, we need to define the schema for the shared types that will be used to store the AIS vessel data.

For example, the following JSON schema defines a property schema that can be used to store AIS vessel data. Copy this to a file called ais-trace-schema.json for use in the next step.

{
  "type": "object",
  "additionalProperties": {
    "type": "object",
    "x-yrs-storage": "shared",
    "properties": {
      "MMSI": { "type": "number" },
      "TSTAMP": { "type": "string" },
      "LATITUDE": { "type": "number" },
      "LONGITUDE": { "type": "number" },
      "COG": { "type": "number" },
      "SOG": { "type": "number" },
      "HEADING": { "type": "number" },
      "NAVSTAT": { "type": "number" },
      "IMO": { "type": "number" },
      "NAME": { "type": "string" },
      "CALLSIGN": { "type": "string" },
      "TYPE": { "type": "number" },
      "DRAUGHT": { "type": "number" },
      "DEST": { "type": "string" },
      "ETA": { "type": "string" }
    }
  }
}

x-yrs-storage

Note the x-yrs-storage custom JSON schema property in the above schema. This tells the Volt that the data is stored as a YJS shared type rather than a plain JSON object. The default storage type is json. This property can be used at any level in the schema, but only if the parent object is also a shared type, i.e. a plain JSON object cannot have a sub-property that is a shared type. This is useful for optimising the synchronisation deltas. For example, if the schema above omitted the x-yrs-storage property, then every time the LATITUDE property is updated, the entire Southern Cross object would be synchronised to the other synapse, even if only the LATITUDE property changed.

A document that conforms to this schema would look like this:

{
  "Southern Cross": {
    "MMSI": 203035273,
    "TSTAMP": "2024-11-19 20:10:00 GMT",
    "LATITUDE": 39.92121,
    "LONGITUDE": 7.60231,
    "COG": 144.9,
    "SOG": 4.7,
    "HEADING": 149,
    "NAVSTAT": 7,
    "IMO": 9666865,
    "NAME": "Southern Cross",
    "CALLSIGN": "KMUJ",
    "TYPE": 0,
    "DRAUGHT": 0,
    "DEST": "STARNMEER",
    "ETA": "11-26 20:10"
  },
  "Harbor Master": {
    "MMSI": 209083743,
    "TSTAMP": "2024-11-19 20:05:00 GMT",
    "LATITUDE": 44.6323,
    "LONGITUDE": 16.52741,
    "COG": 226,
    "SOG": 15.6,
    "HEADING": 231,
    "NAVSTAT": 0,
    "IMO": 9908828,
    "NAME": "Harbor Master",
    "CALLSIGN": "IMOP2",
    "TYPE": 70,
    "DRAUGHT": 0,
    "DEST": "ROTTERDAM",
    "ETA": "11-26 20:05"
  },
  ...
}

We now use the synapse-set-metadata CLI command to set the metadata for the synapse document that will contain the AIS vessel data.

# Set the metadata for the synapse document that will contain the AIS vessel data on Volt 1
volt synapse-set-metadata @ais-synapse-1 00000000-0000-0000-0000-000000000123 aisTrace map /path/to/ais-trace-schema.json -c ./volt-1.config.json

Note we only need to set the metadata on one of the synapses. The metadata will be synchronised to the other synapse automatically.

Writing data to a synapse

Let’s write some AIS vessel data to the synapse document, using the synapse-write-path CLI command.

# Write some AIS vessel data to the synapse document on Volt 1
volt synapse-write-path @ais-synapse-1 00000000-0000-0000-0000-000000000123 $.aisTrace.SouthernCross '{"MMSI": 203035273, "TSTAMP": "2024-11-19 20:10:00 GMT", "LATITUDE": 39.92121, "LONGITUDE": 7.60231}' -c ./volt-1.config.json

The data will be synchronised to the other synapse automatically.

We could also write data to the other synapse, note here we write to the LATITUDE property of the Southern Cross vessel, and we use the Volt 2 configuration file.

# Write some AIS vessel data to the synapse document on Volt 2
volt synapse-write-path @ais-synapse-2 00000000-0000-0000-0000-000000000123 $.aisTrace.SouthernCross.LATITUDE 39.9377 -c ./volt-2.config.json

Watching for changes to a synapse

We can also watch for changes to the synapse document using the synapse-watch-path CLI command. This will set up a long-lived connection to the synapse and print out the changes as they happen.

# Watch for changes to the synapse document on Volt 1
volt synapse-watch-path @ais-synapse-1 00000000-0000-0000-0000-000000000123 $.aisTrace.SouthernCross -c ./volt-1.config.json

We can watch for changes to specific vessels or any vessel using the wildcard *.

# Watch for changes to any vessel's NAVSTAT property
volt synapse-watch-path @ais-synapse-1 00000000-0000-0000-0000-000000000123 $.aisTrace.*.NAVSTAT -c ./volt-1.config.json

The output will indicate the path that was updated and the new value.

This will print out the changes as they happen.

Using the YJS API

So far we have been using the CLI to interact with the synapse. However, we can also use the standard Volt API to interact with the synapse via any of the client libraries.

When using the client libraries, we usually treat the synapse as a regular sync database and use the SyncProvider class to synchronise the data. The synapse running on the Volt will automatically synchronise the data to all connected synapses.

import grpc from "@grpc/grpc-js";
import { VoltClient } from "@tdxvolt/volt-client-grpc";
import { SyncProvider } from "@tdxvolt/volt-client-grpc";
import * as Y from "yjs";

const configPath = "./volt.config.json";

const voltClient = new VoltClient(grpc);
await voltClient.initialise(configPath);

const ydoc = new Y.Doc({ guid: "00000000-0000-0000-0000-000000000123" });

const syncProvider = new SyncProvider(Y, voltClient, "@ais-synapse-1", ydoc);

let aisTraceMap;

syncProvider.on("status", ({ status }) => {
  if (status === "connected") {
    console.log("connected");
    aisTraceMap = ydoc.getMap("aisTrace");

    aisTraceMap.observe((event) => {
      // Handle change event.
    });

    aisTraceMap.set("Southern Cross", {
      MMSI: 203035273,
      TSTAMP: "2024-11-19 20:10:00 GMT",
      LATITUDE: 39.92121,
      LONGITUDE: 7.60231,
    });
  } else {
    console.log("disconnected");
  }
});

syncProvider.startSync();

When writing data via the YJS API, as long as the data you write to the document conforms to the schema defined for the shared type, it will be synchronised to all connected synapses automatically. When using the YJS API, the synapse won't be able to prevent you from writing data that doesn't conform to the schema, but it might not be synchronised to other synapses.

Using the synapse API

The section above utilised the YJS API to read and write data to the synapse.You can also use any of the synapse API methods via the client libraries.

To set the metadata for a shared type use the SetSynapseDocumentMetadata API.

const request = {
  database_id: "@ais-synapse-1",
  document_id: "00000000-0000-0000-0000-000000000123",
  metadata: [
    {
      name: "aisTrace",
      type: "map",
      json_schema:
        '{"type":"object","additionalProperties":{"type":"object","x-yrs-storage":"shared","properties":{"MMSI":{"type":"number"},"TSTAMP":{"type":"string"},"LATITUDE":{"type":"number"},"LONGITUDE":{"type":"number"},"COG":{"type":"number"},"SOG":{"type":"number"},"HEADING":{"type":"number"},"NAVSTAT":{"type":"number"},"IMO":{"type":"number"},"NAME":{"type":"string"},"CALLSIGN":{"type":"string"},"TYPE":{"type":"number"},"DRAUGHT":{"type":"number"},"DEST":{"type":"string"},"ETA":{"type":"string"}}}}',
    },
  ],
};
const response = await voltClient.SetSynapseDocumentMetadata(request);
console.log(response);

To write data to a synapse use the WriteSynapsePath API.

const request = {
  database_id: "@ais-synapse-1",
  document_id: "00000000-0000-0000-0000-000000000123",
  path: "$.aisTrace.SouthernCross.LATITUDE",
  json: 39.9377,
};
const response = await voltClient.WriteSynapsePath(request);
console.log(response);

To watch for changes to a synapse use the WatchSynapsePath API.

const request = {
  start: {
    database_id: "@ais-synapse-1",
    document_id: "00000000-0000-0000-0000-000000000123",
    path: ["$.aisTrace.*.LATITUDE", "$.aisTrace.*.LONGITUDE"],
  },
};
const watchStream = await voltClient.WatchSynapsePath(request);

watchStream.on("data", (response) => {
  console.log(response);
});

watchStream.on("end", () => {
  console.log("watch ended");
});

watchStream.on("error", (err) => {
  console.log(err);
});

The synapse API methods will enforce the schema defined for the shared type and will fail if the data you write doesn't conform to the schema.

Blob documents

A blob document stores an opaque binary payload in the synapse blob store. There is no Yjs shared type for a blob: the document’s schema is a pointer to bytes that are addressed by their hash, and those bytes are transferred with dedicated blob RPCs rather than SyncDocument.

Use a blob document when the payload is a file (or other byte sequence) that should be replaced as a whole. Use map, array, or text roots when the data is structured JSON or collaboratively edited text.

VoltDrive uses blob documents for binary files and Y.Text documents for text files. The same blob APIs are available to any synapse client, not only VoltDrive.

Schema

A blob document’s __schema must be a single entry of type "blob". It cannot be mixed with map, array, or text roots, and a document cannot have more than one blob root.

const response = await voltClient.SetSynapseDocumentMetadata({
  synapse_id: "@ais-synapse-1",
  document_id: "photo-001",
  metadata: [{ name: "content", type: "blob" }],
  blob_hash: hash,
});

blob_hash is only valid on blob documents. For map, array, and text documents the server computes __hash from the Yjs content.

Content addressing

The blob hash is SHA-256 of the raw payload bytes, encoded as unpadded base64url. That value is both the blob store key and the document’s __hash once the document is bound.

Empty payloads are a special case: they are never written to the blob store. Binding a document to the SHA-256 of empty bytes is enough; every replica already holds that content.

Puts are idempotent. If the hash is already stored, PutSynapseBlob is a no-op. The server verifies that the received bytes match hash (and size) before committing a new blob.

APIs

Method	Role
HasSynapseBlob	Check whether a hash is already stored.
PutSynapseBlob	Upload bytes. First message is `start` (`synapse_id`, `hash`, `size`); later messages are payload chunks.
GetSynapseBlob	Download bytes by hash.
SetSynapseDocumentMetadata	Declare the blob schema and, when the bytes are already stored, bind them with `blob_hash`.

WriteSynapsePath, WatchSynapsePath, and SyncDocument do not apply to blob documents. Path writes and Yjs sync operate on map, array, and text roots only.

Create and bind

Typical content-first flow:

Hash the payload (sha256 of the raw bytes, unpadded base64url).
Call HasSynapseBlob. Skip the upload if exists is true, or if the payload is empty.
Otherwise stream the bytes with PutSynapseBlob.
Call SetSynapseDocumentMetadata with a single type: "blob" entry and blob_hash set to that digest. The server checks that the blob is stored (or that the hash is the empty digest) and writes __hash.

You can also create the metadata first and omit blob_hash. The document stays in draft until a later call binds it to a stored blob.

See Store binary data for a worked example.

Replication and document state

Synapse connections replicate blob payloads with the same Has / Get / Put RPCs. Metadata (__schema, __hash) still travels with the synapse meta document, so a replica can learn of a blob before it holds the bytes.

GetSynapseDocumentState and WatchSynapseDocumentState report whether this Volt holds the content advertised by __hash:

DRAFT — metadata exists, but __hash has not been published yet.
ANNOUNCED — __hash is published, but this Volt does not yet hold matching content.
READY — this Volt holds the payload (the blob is in the local store, or the hash is the empty digest).

Wait for READY before reading a blob document. For a blob, READY means the bytes are in the blob store; nothing is hashed from a Yjs document.

Permissions

HasSynapseBlob and GetSynapseBlob require volt:database-read on the synapse (volt:database-write also implies read). PutSynapseBlob and SetSynapseDocumentMetadata require volt:database-write.