Data Connector API

Declare the Entry

A data connector extension exposes an external data source, the API, service, or database it reads and writes, as collections in a Monospace workspace. A collection holds items, and each item carries named values called fields. Its entry in extension.config.json has "type": "dataConnector" and these fields:

A data source is a data connector configured in a workspace, with its own configuration and granted network permissions.

engine

Each permissions item is an object with exactly two fields:

The instance also rejects the whole extension at load time when a permission target passes the schema but can't be parsed (e.g., net:example.com:notaport), or when a reason contains only whitespace.

Entrypoint Rules

monospace extension build replaces engine.entrypoint with the built file, named <entry id>.js. The instance checks each entrypoint in the manifest against the extension directory. It rejects:

• absolute paths and Windows drive paths (e.g., C:/connector.js);
• backslash separators;
• any .. segment;
• a path that isn't a file;
• a symlink that resolves outside the extension directory.

A leading ./ and symlinks that stay inside the directory are accepted. One invalid entrypoint rejects the whole extension.

The entrypoint file must be one self-contained ES module whose default export has a setup function. monospace extension build checks for imports and a default export, but not for setup. A file without setup, or with a static import, fails when a data source starts. A dynamic import() inside a handler fails only when it runs. See One File, No Imports.

Import the Kit

@monospace/extension-kit has three import paths: /data-connector for defineDataConnector, its types and the error classes; /data-connector/protocol for the schema document and query types; and /data-connector/experimental for the schema helpers. The Extension Kit reference lists every export with its declaration.

The package has no runtime dependencies. The built entrypoint is a single file with no imports, so everything you use from the kit is bundled into it. See How Data Connectors Run.

Define a Data Connector

The entrypoint's default export is a descriptor. defineDataConnector builds one from a definition with a required setup and an optional preflight. The Stripe connector from Build a Stripe Data Connector parses its configuration in setup and returns its handlers; its other files hold the client, the schema and the operations:

import { defineDataConnector } from '@monospace/extension-kit/data-connector';
import { StripeClient } from './client';
import { schemaDocument } from './collections';
import { parseConfig } from './config';
import { readMany, readOne } from './read';
import { findResource } from './resources';
import { create, deleteMany, deleteOne, updateMany, updateOne } from './write';

export default defineDataConnector<unknown>({
    setup({ config: rawConfig }) {
        const config = parseConfig(rawConfig);
        const context = { client: new StripeClient(config.apiKey), config };

        return {
            introspect() {
                return schemaDocument(config.enableWrites);
            },

            async query({ query }) {
                // Engine sends only declared collections, and this connector declares no namespaces.
                // For anything else, throw a plain error.
                const resource = query.namespace === undefined ? findResource(query.collection) : undefined;
                if (resource === undefined) throw new Error(`Unknown collection \`${query.collection}\`.`);

                switch (query.op) {
                    case 'readMany': return await readMany(context, resource, query);
                    case 'readOne': return await readOne(context, resource, query);
                    case 'create': return await create(context, resource, query);
                    case 'updateMany': return await updateMany(context, resource, query);
                    case 'updateOne': return await updateOne(context, resource, query);
                    case 'deleteMany': return await deleteMany(context, resource, query);
                    case 'deleteOne': return await deleteOne(context, resource, query);
                    default: {
                        // `query` is `never` here, so an operation added to the kit fails to compile until it's handled.
                        const unhandled: never = query;
                        throw new Error('The Stripe connector received an operation it doesn\'t handle.', { cause: unhandled });
                    }
                }
            },

            async testConnection() {
                await context.client.ping();
                return { ok: true };
            },
        };
    },
});

defineDataConnector performs no I/O. It returns the definition plus a monospace marker, { kind: 'dataConnector', apiVersion: 1 }. Engine recognizes a descriptor by its setup function. A hand-written object with a setup function works without the kit.

Because the descriptor is a plain object, you can test a connector by calling setup and its handlers directly.

Receive the Configuration

preflight and setup each receive one argument, { config }. config is the data source's stored configuration object, passed through unchanged.

• Nothing validates it before your code runs. Engine doesn't check the configuration against any schema. Check every value you depend on in setup, and throw InvalidConfiguration when one is missing or wrong.
• Type it with the generic. defineDataConnector<Config> types config. The type is a compile-time assertion only. The default is Record<string, unknown>. The Stripe connector passes unknown and checks every value in parseConfig.
• It's a bag on purpose. New keys can be added beside config later without breaking your connector.

preflight

preflight({ config }) is optional. It returns { permissions }: the network permissions this data source needs beyond the manifest's, derived from the configuration (e.g., a host taken from a baseUrl setting).

Each permission is { permission, reason }, where permission is net: followed by a target (see Target Syntax), and reason is the text shown to whoever approves the grant.

Neither example connector uses preflight: their hosts are fixed, so they declare them in the manifest.

• Synchronous only. A preflight that returns a promise fails with "preflight must answer synchronously".
• No side effects. Don't make requests, open connections, or build caches in preflight. It runs before setup, every time the connector starts.
• Full validation belongs in setup. When a value preflight reads is invalid, throw InvalidConfiguration there. It deactivates the data source as it does from setup.

A permission is net: followed by a host, host:port, or *.host, never a URL. Each carries a reason, shown to whoever approves the grant. See Network Permissions.

setup

setup({ config }) runs once each time Engine starts the connector for a data source. It returns the handlers, or a promise of them.

• Keep client state in the closure. No handler receives the configuration, so build your client for the external data source in setup and close over it.
• It has a time limit. setup must settle within 30 seconds. See Data Connector Runtime Limits.
• Throw InvalidConfiguration for bad configuration, and AccessDenied for credentials the external data source refuses. Engine deactivates the data source until the extension is reloaded, because retrying can't fix either. Requests that reach it fail with 422 "Invalid extension configuration", with your message nested inside.
• A permission denial also deactivates it. A request in setup to a host outside the grant deactivates the data source when the denial reaches Engine, uncaught or kept as the cause of the error you throw.
• Missing handlers also deactivate it. A setup that returns no object, or an object without introspect or query, deactivates the data source the same way.
• Other throws don't deactivate the data source. Any other kit error class, or an error without one, fails the request that started the connector with 422 "Invalid extension configuration", and the next request starts it again.

See Data Connector Errors for the classes.

Handlers

setup returns an object of type DataConnectorHandlers, which lists each handler's signature:

• Missing required handlers deactivate the data source. If setup returns something that isn't an object, or lacks introspect or query, the connector doesn't start until the extension is reloaded.
• Unknown handlers are ignored. Engine logs "ignoring unrecognised handler(s)" with their names. Methods on a class prototype count.
• testConnection is optional. Leave it out when every call to the external data source costs quota. A connection test against a connector without it answers 422 with code 7002, "The connector does not offer a connection test". Creating a data source and previewing a schema skip the test. { ok: false } is reported as a failed connection, and stops a create or a preview like a thrown error.
• Handlers run concurrently. Engine doesn't wait for one request to finish before sending the next. Don't keep per-request state in the closure.
• Every handler must return a value. A handler that returns undefined or null fails with "<handler> returned no value".

Describe the Schema

introspect answers with a SchemaDocument. Engine calls it when a data source is created or reintrospected, and stores the result. It also runs, without storing anything, when a configuration is previewed before a data source exists. Every section defaults to empty.

A collection's identity is its namespace (or none) plus its name, so the same collection name can appear in different namespaces.

Collections

Fields

Field Types

PrimitiveFieldType is a closed set. An unrecognized name rejects the schema. Only decimal and string take params.

A field with isList: true takes an array of values of its type, or null.

Indexes

CanonicalIndex is { name, fields, kind }, where kind is 'primary', 'unique' or 'normal' and fields lists field names in order.

The primary index decides which keyed operations a collection can offer. See Keys and the Stable ID.

Relations

CanonicalRelation links a constrained collection, which holds the fields that point at the other side, to a referenced collection.

Relation fields expose a relation to callers. A to-one relation field returns a single related item. A to-many relation field returns several. The field on the constrained collection is always to-one. The field on the referenced collection is to-one only when every constrained field has its own single-field primary or unique index; a composite unique index doesn't count.

Each relation field is only offered when the collections' operations can serve it. See Relations.

Defaults

DefaultValue describes a field's default. The kit never evaluates it.

Engine evaluates engineManaged defaults before calling your connector. When a caller omits a field that the operation's input offers, Engine fills in a value. A create uses onCreate, and an update uses onUpdate. A field outside the member's data gets no engine-managed value, and neither does a foreign key whose relation input isn't offered; see Foreign Keys Are Set Through the Relation on Create. currentUser is null when the request has no current user.

Engine checks each default when the data source is created or reintrospected. A list field can't have one. An engineManaged default needs onCreate or onUpdate, and a field type other than unsupported. currentUser and randomUuid need a uuid field, currentTimestamp a date or time field, and a static value must fit the field's type.

A read-only, non-nullable field without a default blocks create on its collection. See Declaration Rules. With the helpers, a field is read-only unless a data list names it, so defineCollection rejects a create that leaves such a field out of every data list.

Example Schema Document

The full Art Institute connector, which Map an External Data Source walks through, returns this document from introspect. It's shown for the artists collection and the relation; the artworks collection is left out. The connector builds it with the helpers, and a hand-written SchemaDocument has the same shape:

{
  "collections": [
    {
      "name": "artists",
      "isReadonly": true,
      "fields": [
        {
          "name": "id",
          "type": {
            "name": "int32"
          },
          "isNullable": false,
          "isList": false,
          "isReadonly": true
        },
        {
          "name": "title",
          "type": {
            "name": "string",
            "params": {
              "length": "unlimited"
            }
          },
          "isNullable": false,
          "isList": false,
          "isReadonly": true
        },
        {
          "name": "birthDate",
          "type": {
            "name": "int32"
          },
          "isNullable": true,
          "isList": false,
          "isReadonly": true
        },
        {
          "name": "deathDate",
          "type": {
            "name": "int32"
          },
          "isNullable": true,
          "isList": false,
          "isReadonly": true
        }
      ],
      "indexes": [
        {
          "name": "artists_pkey",
          "kind": "primary",
          "fields": [
            "id"
          ]
        }
      ],
      "operations": {
        "readMany": {
          "filter": {
            "fields": {
              "id": {
                "operators": {
                  "equals": {},
                  "in": {}
                }
              },
              "birthDate": {
                "operators": {
                  "equals": {},
                  "in": {},
                  "lessThan": {},
                  "lessThanOrEquals": {},
                  "greaterThan": {},
                  "greaterThanOrEquals": {}
                }
              }
            },
            "logicalOperators": {
              "and": {},
              "or": {}
            }
          },
          "sort": {
            "id": {
              "directions": [
                "ascending",
                "descending"
              ]
            },
            "birthDate": {
              "directions": [
                "ascending",
                "descending"
              ]
            }
          },
          "limit": {},
          "offset": {},
          "output": [
            "id",
            "title",
            "birthDate",
            "deathDate"
          ],
          "meta": {
            "totalCount": true
          }
        },
        "readOne": {
          "filter": {
            "fields": {
              "id": {
                "operators": {
                  "equals": {},
                  "in": {}
                }
              },
              "birthDate": {
                "operators": {
                  "equals": {},
                  "in": {},
                  "lessThan": {},
                  "lessThanOrEquals": {},
                  "greaterThan": {},
                  "greaterThanOrEquals": {}
                }
              }
            },
            "logicalOperators": {
              "and": {},
              "or": {}
            }
          },
          "output": [
            "id",
            "title",
            "birthDate",
            "deathDate"
          ]
        }
      }
    }
  ],
  "relations": [
    {
      "name": "artworks_artist",
      "constrainedCollection": "artworks",
      "constrainedFields": [
        "artistId"
      ],
      "constrainedRelationFieldName": "artist",
      "referencedCollection": "artists",
      "referencedFields": [
        "id"
      ],
      "referencedRelationFieldName": "artworks",
      "onDelete": "noAction",
      "onUpdate": "noAction"
    }
  ]
}

operators and logicalOperators are maps from each operator to its options. No operator has options yet, so each value is {}. The helpers filter.field and filter.logical build these maps for you.

Build a Schema with the Helpers

@monospace/extension-kit/data-connector/experimental builds the same SchemaDocument from typed definitions, checks them, and derives the isReadonly flags from operations. The helpers are optional, and experimental within the experimental kit. Schema Helpers documents each helper, the checks it runs, and the Art Institute and Stripe schemas built with them.

Read a Query Request

Engine calls query with an envelope that holds one operation (QuerySerialized). The handler receives it as a QueryRequest, and a function typed with QueryRequest<'readOne'> receives only that operation:

{
  "query": {
    "op": "readMany",
    "collection": "articles",
    "select": ["id", "title", "wordCount"],
    "filter": { "cmp": "greaterThan", "left": { "field": "wordCount" }, "right": { "value": 1000 } },
    "sort": [{ "field": "title", "direction": "ascending" }],
    "limit": 100,
    "offset": 0
  }
}

Every operation carries op, collection, and namespace when the collection is in one. Collections and fields are named by their name, never their apiName.

Operations

• select lists the fields each returned item must carry.
• The query capabilities decide what a write selects. Without insertReturning, create arrives with select set to the stable-ID fields only, whatever the caller selected; answer each created item with them. Without updateReturning or deleteReturning, updates and deletes arrive as updateMany and deleteMany with select: [], filtered by the stable IDs Engine read first. See Declare Query Capabilities.
• Engine sends one create per item. data holds one object.
• data objects are keyed by field name and hold only fields the member's data declares.
• An absent filter adds no condition. A keyed operation still targets only its key.
• An absent limit means no limit. An absent offset means 0.

Keyed Operations

readOne, updateOne and deleteOne (KeyedOperation) address one item by key, an object keyed by the collection's primary-key field. Engine offers keyed operations only on a single-field primary key, so key has one entry.

{
  "query": {
    "op": "updateOne",
    "collection": "articles",
    "key": { "id": "art_42" },
    "filter": { "cmp": "equals", "left": { "field": "status" }, "right": { "value": "draft" } },
    "data": { "status": "published" },
    "select": ["id", "status"]
  }
}

• filter is a guard. The item must match both the key and the guard. Otherwise change nothing.
• Answer record. Answer { "record": <item> } with the item the operation read, updated or deleted, carrying every field in select. The answer takes no other key.
• Answer record: null on a miss. When the key matches no item, or the item fails the guard, answer { "record": null } and change nothing. A miss isn't an error, so don't throw. Callers see 404 "No record found".
• Never answer records. Engine fails a keyed operation that answers { "records": [...] }, whatever it holds, with 500 "Invalid extension response" and the reason "query answered with something this protocol does not define: unknown field records, expected record". A write your connector made stays applied.
• Keyed writes need their capability. Engine sends updateOne only to a connector that declares updateReturning, and deleteOne only with deleteReturning. Without it, the caller's keyed write reaches your connector as updateMany or deleteMany by stable ID.

{ "record": { "id": "art_42", "status": "published" } }

See Keys and the Stable ID.

Count Reads

When a caller asks for the total count, Engine first derives it from the page it read. A page shorter than limit, or a read without limit, establishes the total, unless it's empty at a non-zero offset.

Otherwise Engine sends a separate readMany with count: true, limit: 0 and no select. Recognize it by count: true: an ordinary read with an empty selection also arrives without select. Answer with totalCount, the number of items matching filter, ignoring limit and offset. Return no items.

{ "totalCount": 1284 }

If you omit totalCount, the caller's response carries no total.

Filters

FilterSerialized is an untagged union. Each node has exactly one of these shapes:

Each operand is { field: name } or { value: Value }. Read both sides of a comparison; don't assume the field is on the left.

{
  "and": [
    { "in": { "field": "status" }, "values": ["published", "featured"] },
    { "not": { "cmp": "equals", "left": { "field": "publishedAt" }, "right": { "value": null } } }
  ]
}

Negated caller operators arrive as not around the positive form (e.g., _neq as not around equals). _between arrives as and of greaterThanOrEquals and lessThanOrEquals. See Operations and Operators.

Sort

sort is an array of SortItemSerialized, applied in order:

Values

Value is untagged JSON: null, string, number, boolean, array, or object. Types without a JSON form of their own travel as strings: decimals, base64 bytes, UUIDs, dates and times. The table in Field Types lists each encoding.

int64 and uint64 values arrive as decimal strings, in data, in filter literals and in keys, so values beyond JavaScript's safe-integer range stay exact. In an answer, return a decimal string or a number. REST responses carry them as strings too. Narrower integers arrive as JSON numbers, and so do the elements of a list field: only single values become strings. An int64 or uint64 list element beyond JavaScript's safe-integer range loses precision when your connector receives it.

Answer a Query

A keyed operation answers { record } (see Keyed Operations). Every other operation answers with a QueryResultSerialized:

{
  "records": [
    { "id": "art_42", "title": "Designing Filters", "wordCount": 1850 },
    { "id": "art_43", "title": "Paging Large Collections", "wordCount": null }
  ]
}

Engine checks the fields of every item in records, and of a keyed answer's record, the same way. An answer that fails a check fails the request with 500 "Invalid extension response", with the reason nested inside:

• Carry every selected field. Use an explicit null for a missing value. A missing field fails the request with "Missing selected field".
• Extra fields on an item are ignored. Engine keeps only the fields in select. A non-keyed answer takes only records and totalCount, and any other key fails the request.
• Match the declared type. Engine converts each value by its field's type and fails the request when it can't, for example "Expected a value matching Int64, got a string" for "n/a".
• Some limits aren't checked. Engine rejects a negative value in an unsigned field and a value outside the 64-bit range. It doesn't check narrower integer widths such as int8, string length or nullability.
• Writes aren't rolled back. When your answer fails one of these checks, a write your connector already made stays applied. See No Transactions.
• Fill the page. Engine reads fewer items than limit as the end of the data: the caller sees no next page. When the external data source returns short pages, keep fetching until you reach limit or it runs out of items.
• Answers aren't streamed. A non-keyed answer arrives as one records array, and Engine converts every item in it before using any. Engine holds the whole answer in memory, so bound what a read without limit fetches. When a read hits your bound, throw QueryRejected instead of returning the items fetched so far: Engine treats every answer to a read without limit as complete.

Handle Requests You Didn't Declare

Engine offers callers only what your declaration lists, and rejects a caller's request outside it. It doesn't check the filters it sends to your connector against the called member's declaration. So some requests reach your connector in a shape the declaration doesn't list. Some come from caller shorthands and defaults that Engine expands, and others are calls Engine makes on its own. Serve them where you can. For a request you can't serve, throw QueryRejected, which fails it with 422 and passes your message to the caller.

A comparison on a field or with an operator the called member never declared can reach your connector too:

• A permission rule's condition. A field read rule's condition joins the filter of every call whose filter names the field it guards, including bulk updates and deletes. Saving the policy checks the condition against readMany, readOne, updateOne and deleteOne only, so it reaches updateMany and deleteMany unchecked. See Permission Rules.
• A stored declaration older than your code. After an extension update that narrowed the declaration without a schema refresh, Engine still offers what the stored declaration lists.

Never drop a comparison you can't translate: it can be a permission rule that restricts which items the caller may touch. Declare the fields and operators your rules' conditions use in every member they reach, and refuse any other undeclared comparison. No error class fits it, so throw a plain Error, which Engine reports as a failed query.

To catch them, check each request against your own declaration: the full Art Institute connector in Map an External Data Source reads it from schema.toSchemaDocument(). Engine's relation and read-back lookups on a key no declaration can carry are the exception: serve them.

Engine doesn't add a sort of its own. A request without a caller sort arrives without sort, so keep your connector's default order stable across pages.

Relation Reads

Engine resolves a relation field by reading the other collection, filtered by the relation's key field. It sends that read to your connector in one of two shapes:

• Batched: one readMany with in over the key values of every parent item, joined with and to the include's filter, with no limit and no offset.
• Per parent: one readMany per distinct, non-null parent key. Engine sends up to 16 of them at a time for one include, so they arrive concurrently. Each carries in with that one key value, joined with and to the include's filter, plus the include's sort and the resolved limit and offset.

For a key of several fields, or a key field whose filter declares equals and or but no in, equals on each field takes the place of in. See Resolve Relations.

A to-one include is always batched, because it takes no limit or offset. A to-many include is batched only when it resolves to no limit and no offset. A data connector can't apply a limit or offset to each parent's group of items within one read, so every other to-many include is read per parent.

The include's limit resolves like a top-level read's (see Page Size). The instance settings are MONOSPACE_QUERY_LIMIT_DEFAULT (the default, 100 unless configured) and MONOSPACE_QUERY_LIMIT_MAX (the maximum, unset unless configured):

• To-many includes are paged by default. With the default instance settings, a to-many include without limit carries limit: 100, so your connector receives one read per parent, and a caller sees at most 100 related items per parent.
• The default isn't checked against the maximum. With a default of 100 and a maximum of 50, an include without limit still arrives with limit: 100.
• A caller can set limit or offset on an include only when the related collection's readMany declares it. Otherwise the include always resolves as an omitted limit without offset.
• Only config.yaml can unset the default. MONOSPACE_QUERY_LIMIT_DEFAULT accepts only an integer; an empty value stops the instance from starting.
• A key of several fields, or a single key field whose filter declares equals and or but no in, arrives as an or of and-joined equals instead of in.
• Per-parent reads add up. A to-many include on a page of 100 parent items can send your connector up to 100 reads, up to 16 at a time. Concurrent caller requests each send their own.

Declare the filter each direction needs. See Resolve Relations.

See Also

• Extension Config and Manifest: the extension-level fields around the entry
• Operations and Operators: declare what each collection serves and which rules Engine enforces
• Data Connector Errors: error classes and when to throw each
• Extension Kit: every export of the kit, with its declaration
• Network Permissions: net: permissions in the manifest and from preflight
• How Data Connectors Run: the sandbox your connector runs in