Data Connector Module

Overview

@monospace/extension-kit/data-connector is the module a data connector extension builds on. A data connector reads and writes an external data source: an API, a service, or a database. It exposes that source as collections of items in a Monospace workspace. Each item carries named values called fields. The module exports defineDataConnector, the types around it, and the error classes a connector throws.

The kit has three import paths:

The package has no runtime dependencies. The built entrypoint is a single file with no imports, so everything you use from the kit is bundled into it. See How Data Connectors Run.

Each entry below shows the declaration from the kit's type definitions. The Data Connector API describes what Engine expects from the objects these types describe.

The examples are excerpts from three connectors. The Data Connector Quickstart shows the complete files of a trimmed Art Institute of Chicago connector, Map an External Data Source the full one (search.ts), and Build a Stripe Data Connector walks through the Stripe connector.

Define a Data Connector

defineDataConnector

Builds the descriptor that a data connector entrypoint exports as its default export.

function defineDataConnector<Config = Record<string, unknown>>(definition: DataConnectorDefinition<Config>): DataConnectorDescriptor<Config>;

Returns a DataConnectorDescriptor<Config>: a new object with the definition's properties plus monospace: { kind: 'dataConnector', apiVersion: 1 }. It performs no I/O and starts nothing, so a test can call setup and the handlers directly.

The Quickstart connector's entrypoint declares no configuration and returns three handlers:

import { defineDataConnector } from '@monospace/extension-kit/data-connector';
import { search } from './api';
import { readMany, readOne } from './read';
import { schema } from './schema';

const COLLECTIONS = new Set<string>(schema.collections.map((collection) => collection.name));

export default defineDataConnector({
    setup() {
        return {
            introspect() {
                return schema.toSchemaDocument();
            },

            async query({ query }) {
                // Monospace sends only declared collections, and this connector declares no namespaces.
                // For any other collection, throw a plain `Error`.
                if (query.namespace !== undefined || !COLLECTIONS.has(query.collection)) {
                    throw new Error(`Unknown collection \`${query.collection}\`.`);
                }

                switch (query.op) {
                    case 'readMany':
                        return await readMany(query);
                    case 'readOne':
                        return await readOne(query);
                    default:
                        // The schema declares no writes, so Monospace offers callers none.
                        throw new Error('The Art Institute of Chicago API is read-only.');
                }
            },

            async testConnection() {
                await search('artworks', { size: 0 });
                return { ok: true };
            },
        };
    },
});

See Define a Data Connector for what Engine does with the descriptor, and the Data Connector Quickstart for a walkthrough.

DataConnectorDefinition

The object you pass to defineDataConnector.

interface DataConnectorDefinition<Config> {
    preflight?: (options: DataConnectorSetupOptions<Config>) => DataConnectorPreflight;
    setup: (options: DataConnectorSetupOptions<Config>) => DataConnectorHandlers | Promise<DataConnectorHandlers>;
}

Validate the configuration in setup. preflight can also throw InvalidConfiguration when a value it reads is invalid, which deactivates the data source as it does from setup.

DataConnectorDescriptor

The value defineDataConnector returns: the definition plus the descriptor marker.

interface DataConnectorDescriptor<Config> extends DataConnectorDefinition<Config> {
    monospace: DescriptorMarker;
}

Export it as the entrypoint's default export. Engine recognizes a descriptor by its setup function, so a hand-written object with a setup function works without the kit.

DataConnectorSetupOptions

The one argument setup and preflight receive.

interface DataConnectorSetupOptions<Config> {
    config: Config;
}

The argument is an object so that new keys can be added beside config without breaking your connector. The Stripe connector types config as unknown and parses it in setup:

export default defineDataConnector<unknown>({
    setup({ config: rawConfig }) {
        const config = parseConfig(rawConfig);
        const context = { client: new StripeClient(config.apiKey), config };

See Receive the Configuration.

DataConnectorHandlers

The object setup returns: the functions Engine calls for a data source.

interface DataConnectorHandlers {
    introspect: () => SchemaDocument | Promise<SchemaDocument>;
    query: QueryHandler;
    testConnection?: () => TestConnectionResult | Promise<TestConnectionResult>;
    explainQuery?: (request: QuerySerialized) => unknown;
}

Handlers run concurrently: Engine sends the next request without waiting for the last one to finish. Don't keep per-request state in the setup closure. See Handlers for what Engine does with each, including what happens when a required handler is missing.

QueryHandler

The type of the query handler: a function from an operation's request to the answer that operation owes.

type QueryHandler<Op extends OperationName = OperationName> = (request: QueryRequest<Op>) => QueryResultFor<Op> | Promise<QueryResultFor<Op>>;

For each operation in Op, the function receives a QueryRequest<Op> and answers a QueryResultFor<Op>, or a promise of it:

The two answer shapes exclude each other, so each of these is a type error:

• { records: [] } or {} from a QueryHandler<'readOne'>
• { record: null } from a QueryHandler<'readMany'>
• { record: null, records: [] } from the query handler

DataConnectorHandlers.query is a QueryHandler for every operation, so it can hand each operation to a function typed for that one. With several operations in Op, the type doesn't pair a request with its answer: a query handler that answers { records: [] } to every operation type-checks. Type each operation's function for that operation to catch a wrong answer shape. The Quickstart connector's readOne declares its answer as QueryResultFor<'readOne'>, and answers record: null on a miss instead of throwing:

/**
 * Handles `readOne` with `readMany`: the key as a filter, any filter the caller adds, and a limit of
 * one item. Returns `record: null` when no item matches, and the caller gets `404`.
 */
export async function readOne(query: ReadOneOperationSerialized): Promise<QueryResultFor<'readOne'>> {
    // Monospace sends the key as `{ id }`. An ID that isn't a safe integer matches no item.
    const { id } = query.key;
    if (typeof id !== 'number' || !Number.isSafeInteger(id)) return { record: null };

    const key = { cmp: 'equals', left: { field: 'id' }, right: { value: id } } as const;
    const filter = query.filter === undefined ? key : { and: [key, query.filter] };
    const { records = [] } = await readMany({ op: 'readMany', collection: query.collection, select: query.select, filter, limit: 1 });
    return { record: records[0] ?? null };
}

The typeof id !== 'number' check fits this connector's int32 ids. An int64 or uint64 key arrives as a decimal string, so the same check misses every item in such a connector. See Values and Keyed Operations.

TestConnectionResult

What testConnection answers with.

interface TestConnectionResult {
    ok: boolean;
}

Answer { ok: true } when the connection works, and throw a kit error to report a failure. { ok: false } is also reported as a failed connection. The Quickstart connector sends one real request and answers { ok: true }:

            async testConnection() {
                await search('artworks', { size: 0 });
                return { ok: true };
            },

Declare Permissions

A data connector reaches only the hosts a data source's network permissions grant. Fixed hosts go in the manifest; preflight adds hosts derived from configuration. See Network Permissions.

DataConnectorPreflight

What preflight answers with.

interface DataConnectorPreflight {
    permissions: DataConnectorPermissions;
}

permissions lists the permissions this data source needs in addition to the manifest's. See Derive Permissions in preflight.

DataConnectorPermissions

A list of permissions, each with its reason.

type DataConnectorPermissions = DataConnectorPermission[];

DataConnectorPermission

One permission and the reason the connector needs it.

interface DataConnectorPermission {
    permission: DataConnectorNetTarget;
    reason: string;
}

DataConnectorNetTarget

A network permission target: net: followed by a host.

type DataConnectorNetTarget = `net:${string}`;

The type accepts any string after net:. The target is a host, host:port, or *.host, never a URL. See Target Syntax for the rules Engine applies.

Throw Errors

A data connector reports a failure by throwing one of these classes from setup or a handler. Each class carries its code, and Engine passes only the code and the message on. Data Connector Errors covers when to throw each class, the HTTP status callers see, and which failures deactivate a data source.

For anything no class describes, throw a plain Error. Engine reports it as a failed query.

ConnectorError

The abstract base class of every kit error class.

abstract class ConnectorError extends Error {
    readonly monospace: ConnectorErrorBrand;
    protected constructor(code: ThrownErrorCode, message: string, options?: ConnectorErrorOptions);
}

Its constructor is protected: throw one of the classes below, never ConnectorError itself. Every subclass sets name to the class's name and monospace to { code }, the brand Engine reads. cause from the options is set on the error as the standard Error cause.

Engine recognizes an error by its monospace property, not by instanceof ConnectorError. See Throw a Connector Error.

ConnectorErrorOptions

The optional second argument of every error class.

interface ConnectorErrorOptions {
    cause?: unknown;
}

cause stays on the JavaScript error and never reaches the caller. Keep a fetch rejection as cause: when the data source's permissions refused the request, Engine finds the denial there and reports a permission error instead. See Handle a Denied Request.

QueryRejected

Refuses a valid query or write the connector can't serve as asked, with a message for the caller.

class QueryRejected extends ConnectorError {
    constructor(message: string, options?: ConnectorErrorOptions);
}

Throw it for a request that fits your declaration but that the external data source can't serve. Examples are a missing filter it requires, a page beyond its window, or a value outside a field's rule. The caller can change the request, so write the message for them. The Quickstart connector refuses a read past the search API's window:

    // A read past the window still succeeds when fewer items match, so count the matches first and
    // end the read at the last match.
    if (end > SEARCH_WINDOW) {
        const { pagination } = await search(query.collection, { ...base, size: 0 });
        end = Math.min(end, pagination.total);

        if (end > SEARCH_WINDOW) {
            // The caller can narrow this read, so the message says how.
            throw new QueryRejected(`The Art Institute search API returns only the first ${SEARCH_WINDOW} results. Narrow the filter or read an earlier page.`);
        }
    }

Some shapes arrive although your declaration doesn't list them, because Engine composes them: not around an update's field rules, and or across permission rules. Throw QueryRejected for those you can't serve. For a request that names something you never declared, such as an unknown collection or operation, throw a plain Error. See Tell QueryRejected from a Plain Error.

UniqueConstraintViolation

Reports a write the external data source refuses because a value that must be unique already exists.

class UniqueConstraintViolation extends ConnectorError {
    constructor(message: string, options?: ConnectorErrorOptions);
}

NullConstraintViolation

Reports a write the external data source refuses because it requires a value the request left empty.

class NullConstraintViolation extends ConnectorError {
    constructor(message: string, options?: ConnectorErrorOptions);
}

ForeignKeyConstraintViolation

Reports a write the external data source refuses because of a reference: the item it points to doesn't exist, or other items still point to the one being deleted.

class ForeignKeyConstraintViolation extends ConnectorError {
    constructor(message: string, options?: ConnectorErrorOptions);
}

The Stripe connector maps Stripe's error codes to the three constraint classes. Not every missing object is an error. A retrieve answers a missing object as no item, and a keyed read then answers record: null. A list filtered by a missing referenced object answers no items.

A missing-object error that reaches the classifier becomes StripeObjectMissing, the connector's own subclass of Error. Its keyed writes answer it as record: null. Anywhere else, such as a bulk write or a later page of a list, it reaches Engine as a plain Error:

    switch (detail.code) {
        case 'resource_missing':
            // `param` names what is missing: `id` for the object in the path. On a write, any other
            // field is an object the written values refer to, such as a new price's `product`.
            if (method === 'POST' && detail.param !== undefined && detail.param !== 'id') return new ForeignKeyConstraintViolation(message);
            return new StripeObjectMissing(message);
        case 'resource_already_exists':
            return new UniqueConstraintViolation(message);
        case 'parameter_missing':
            return new NullConstraintViolation(message);
    }

InvalidConfiguration

Reports a data source configuration that is missing a value or has a wrong one, found without asking the external data source.

class InvalidConfiguration extends ConnectorError {
    constructor(message: string, options?: ConnectorErrorOptions);
}

Nothing validates a connector's configuration before setup receives it, so raising this is the connector's job. Thrown from setup, it deactivates the data source until the extension is reloaded. For credentials the external data source refuses, throw AccessDenied. The Stripe connector checks each value it reads:

/**
 * Nothing validates a connector's configuration before `setup` receives it, so this does. An
 * `InvalidConfiguration` from `setup` fails a connection test or a data source create with a 422.
 */
export function parseConfig(value: unknown): StripeConfig {
    if (!isObject(value)) throw new InvalidConfiguration('Stripe configuration must be an object.');

    const apiKey = value.apiKey;

    if (typeof apiKey !== 'string' || !/^(?:sk|rk)_(?:test|live)_\w+$/u.test(apiKey)) {
        throw new InvalidConfiguration('Stripe configuration `apiKey` must be a secret (`sk_`) or restricted (`rk_`) key.');
    }

See Use InvalidConfiguration and AccessDenied for Configuration.

AccessDenied

Reports that the external data source refuses this account. Its credentials are rejected, expired or revoked. Or they lack a scope, can't reach the resource, or exceed the plan.

class AccessDenied extends ConnectorError {
    constructor(message: string, options?: ConnectorErrorOptions);
}

Only someone with access to the external data source's account can fix it. Keep credentials out of the message: the Stripe connector uses a fixed one, because Stripe's own message echoes part of the key.

    // Only someone with access to the Stripe account can fix the key. Stripe's message for a rejected
    // key echoes part of the key, so it stays out.
    if (status === 401 || status === 403) return new AccessDenied(`Stripe rejected the API key (HTTP ${status}).`);

CollectionNotFound

Reports that a collection the connector declared no longer exists in the external data source, for example an object type deleted after the data source was introspected.

class CollectionNotFound extends ConnectorError {
    constructor(message: string, options?: ConnectorErrorOptions);
}

Engine sends only declared collections. For any other name, throw a plain Error, as the example connectors do.

ConnectionFailed

Reports that the external data source can't be reached or won't answer for now: a network failure, a timeout, maintenance, or a server error it reports as temporary.

class ConnectionFailed extends ConnectorError {
    constructor(message: string, options?: ConnectorErrorOptions);
}

The failure is temporary, so a caller can retry the request later. Wrapping a fetch rejection in it is safe when you keep the rejection as cause. The full Art Institute connector's search client:

    catch (error) {
        const isTimeout = error instanceof Error && error.name === 'TimeoutError';
        const reason = isTimeout ? `did not answer within ${TIMEOUT_MS / 1000} seconds` : 'could not be reached';
        // Keep the rejection as the `cause`: when the data source's permissions refused the request,
        // Engine finds that there and reports a permission error instead of a failed connection.
        throw new ConnectionFailed(`The Art Institute API ${reason}.`, { cause: error });
    }

    if (response.status === 429) {
        throw new RateLimited(`The Art Institute API rate limited the connector.${retryAfter(response)}`);
    }

    // Treat a server error as an outage: a later request may work.
    if (response.status >= 500) {
        throw new ConnectionFailed(`The Art Institute API answered HTTP ${response.status}.`);
    }

RateLimited

Reports that the external data source is throttling the connector: a rate limit, or a quota that resets.

class RateLimited extends ConnectorError {
    constructor(message: string, options?: ConnectorErrorOptions);
}

A caller can retry once the limit resets. For a limit that only a plan change lifts, throw AccessDenied. The ConnectionFailed example above throws it for an HTTP 429.

Read Error Codes

ErrorCode

Maps each error name to its protocol code, for code that recognizes an error without instanceof.

const ErrorCode: {
    readonly QueryRejected: 100;
    readonly UniqueConstraintViolation: 1002;
    readonly NullConstraintViolation: 1003;
    readonly ForeignKeyConstraintViolation: 1004;
    readonly InvalidConfiguration: 2001;
    readonly AccessDenied: 2002;
    readonly InvalidPermissions: 2003;
    readonly CollectionNotFound: 2004;
    readonly ConnectionFailed: 5001;
    readonly RateLimited: 5002;
    readonly MethodNotFound: -32601;
};

Compare with its entries, never with hard-coded numbers. The kit generates it from Engine's own list of codes. InvalidPermissions and MethodNotFound are raised only by Engine's side of the protocol, and no class throws them.

The full Art Institute connector's artifact smoke drives the built file and reads each refusal's code:

async function codeOf(promise) {
    try {
        await promise;
    }
    catch (error) {
        assert.ok(error instanceof Error, 'the connector throws an Error');
        return error.monospace?.code ?? PLAIN_ERROR;
    }

    assert.fail('expected the connector to throw');
}

One of its assertions checks that a not, which the connector can't translate, is refused with QueryRejected:

    assert.equal(await codeOf(read('artworks', { filter: { not: compare('equals', 'artistId', null) }, limit: 3 })), ErrorCode.QueryRejected);

See Read Codes from ErrorCode and Test a Data Connector.

ErrorCodeName

The name of any code in ErrorCode.

type ErrorCodeName = keyof typeof ErrorCode;

ShellErrorCodeName

The names of the codes only Engine's side of the protocol raises.

type ShellErrorCodeName = 'InvalidPermissions' | 'MethodNotFound';

InvalidPermissions (2003) reports a permission denial and MethodNotFound (-32601) a missing handler. The kit has no class for either.

ThrownErrorCodeName

The names of the codes connector code throws, each through its own error class.

type ThrownErrorCodeName = Exclude<ErrorCodeName, ShellErrorCodeName>;

ThrownErrorCode

The numeric codes connector code throws: the nine codes of the kit's error classes.

type ThrownErrorCode = (typeof ErrorCode)[ThrownErrorCodeName];

Identify Descriptors and Errors

Engine identifies kit values by plain properties, not by class or Symbol. Two bundled copies of the kit, or a connector written without it, work the same way.

DescriptorMarker

The monospace property defineDataConnector adds to a descriptor.

interface DescriptorMarker {
    kind: 'dataConnector';
    apiVersion: 1;
}

apiVersion describes the descriptor's shape, not the kit's package version. Engine recognizes a descriptor by its setup function, so the marker is optional in a hand-written descriptor.

ConnectorErrorBrand

The monospace property every kit error carries: the error's code.

interface ConnectorErrorBrand {
    code: ThrownErrorCode;
}

Only code crosses to Engine, together with the error's message. Engine passes the message on to API callers in the HTTP error response. Keep secrets and personal data out of the message. A thrown object whose monospace.code is a safe integer is read the same way as a kit class. See Keep Secrets out of Messages.

See Also

• Data Connector API: the entry, setup, the handlers, and the requests Engine sends
• Data Connector Errors: when to throw each class, and what callers see
• Protocol Types: the schema document and query types this module re-exports
• Schema Helpers: build the schema document from typed definitions
• Data Connector Quickstart: build a read-only connector step by step