Protocol Types

Overview

@monospace/extension-kit/data-connector/protocol holds the types of what a data connector extension exchanges with Engine: the schema document introspect answers with, the requests query receives, and the answers it returns. A data connector exposes an external data source as collections of items, and each item carries named values called fields.

The module exports types only. @monospace/extension-kit/data-connector re-exports every one of them, so import them from there:

import type { MonospaceObject, QueryResultFor, QueryResultSerialized, ReadManyOperationSerialized, ReadOneOperationSerialized, SortItemSerialized, Value } from '@monospace/extension-kit/data-connector';

Each entry below shows the declaration from the kit's type definitions. The Data Connector API describes what Engine sends and expects, and Operations and Operators the rules for declaring operations.

The examples are excerpts, with their imports and helpers left out. The Data Connector Quickstart shows the complete files of a trimmed Art Institute of Chicago connector, Map an External Data Source the full one (search.ts), and Build a Stripe Data Connector walks through the Stripe connector.

Read a Query Request

QuerySerialized

The envelope query receives: one operation under query.

interface QuerySerialized {
    query: OperationSerialized;
}

The envelope leaves room for further per-request context beside the operation. The query handler is typed with QueryRequest, the same envelope narrowed to the operations a function serves; QuerySerialized still types the reserved explainQuery handler. See Read a Query Request.

QueryRequest

The query request for the operations Op: the envelope, with query narrowed to those operations.

interface QueryRequest<Op extends OperationName> {
    query: Extract<OperationSerialized, {
        op: Op;
    }>;
}

QueryRequest<'readOne'> carries only a readOne, so a function typed with it reads query.key without narrowing op first. QueryHandler pairs it with the answer the operation owes.

OperationSerialized

One operation and its arguments, discriminated by op.

type OperationSerialized = ReadManyOperationSerialized | ReadOneOperationSerialized | CreateOperationSerialized | UpdateManyOperationSerialized | UpdateOneOperationSerialized | DeleteManyOperationSerialized | DeleteOneOperationSerialized;

Every operation carries collection, and namespace when the collection is in one. A readMany without select is either a count read (count: true, limit: 0) or a read with an empty selection. Each operation has a named type, from ReadManyOperationSerialized to DeleteOneOperationSerialized. A switch on op narrows the union to one of them, so a function that serves one operation takes its type, as the Quickstart connector's readMany does:

export async function readMany(query: ReadManyOperationSerialized): Promise<QueryResultSerialized> {

See Operations and Count Reads.

ReadManyOperationSerialized

A readMany request: the items that match filter, sorted and paged, or their count.

interface ReadManyOperationSerialized {
    op: 'readMany';
    collection: Identifier;
    namespace?: Identifier;
    select?: Identifier[];
    filter?: FilterSerialized;
    sort?: SortItemSerialized[];
    limit?: number;
    offset?: number;
    count?: boolean;
}

Answer with records, or with totalCount on a count read. The Quickstart connector's readMany takes it, and so does the Stripe connector's.

ReadOneOperationSerialized

A readOne request: one item by its primary key, with an optional guard.

interface ReadOneOperationSerialized extends KeyedOperation {
    op: 'readOne';
}

It extends KeyedOperation. Answer with record, or record: null on a miss. The Quickstart connector's readOne takes it (see QueryResultFor).

CreateOperationSerialized

A create request: one object in data per item to create.

interface CreateOperationSerialized {
    op: 'create';
    collection: Identifier;
    namespace?: Identifier;
    select: Identifier[];
    data: MonospaceObject[];
}

Answer with records: the created items, with the fields in select. Which fields select holds depends on the query capabilities. The Stripe connector's create takes it.

UpdateManyOperationSerialized

An updateMany request: one object in data, applied to every item that matches filter.

interface UpdateManyOperationSerialized {
    op: 'updateMany';
    collection: Identifier;
    namespace?: Identifier;
    select: Identifier[];
    filter?: FilterSerialized;
    data: MonospaceObject;
}

Answer with records: the updated items. The Stripe connector's updateMany takes it.

UpdateOneOperationSerialized

An updateOne request: data applied to one item by its primary key, with an optional guard.

interface UpdateOneOperationSerialized extends KeyedOperation {
    op: 'updateOne';
    data: MonospaceObject;
}

It extends KeyedOperation. Answer with record: the updated item, or null on a miss. Engine sends it only to a connector that declares updateReturning. The Stripe connector's updateOne takes it.

DeleteManyOperationSerialized

A deleteMany request: delete every item that matches filter.

interface DeleteManyOperationSerialized {
    op: 'deleteMany';
    collection: Identifier;
    namespace?: Identifier;
    select: Identifier[];
    filter?: FilterSerialized;
}

Answer with records: the deleted items. The Stripe connector's deleteMany takes it.

DeleteOneOperationSerialized

A deleteOne request: delete one item by its primary key, with an optional guard.

interface DeleteOneOperationSerialized extends KeyedOperation {
    op: 'deleteOne';
}

It extends KeyedOperation. Answer with record: the deleted item, or null on a miss. Engine sends it only to a connector that declares deleteReturning. The Stripe connector's deleteOne takes it.

OperationName

The name of an operation, as its op tag.

type OperationName = OperationSerialized['op'];

It's the union 'readMany' | 'readOne' | 'create' | 'updateMany' | 'updateOne' | 'deleteMany' | 'deleteOne', and the type parameter of QueryRequest, QueryResultFor and QueryHandler.

KeyedOperation

What readOne, updateOne and deleteOne share: a key for the collection's primary key, and an optional guard filter.

interface KeyedOperation {
    collection: Identifier;
    namespace?: Identifier;
    select: Identifier[];
    key: Record<Identifier, Value>;
    filter?: FilterSerialized;
}

key holds a value for the collection's primary-key field, by field name. Engine offers keyed operations only on a single-field primary key, so key has one entry. The item must match both the key and the guard. Answer with a KeyedQueryResultSerialized: { record: <item> }, or { record: null } when the key or the guard misses. A miss isn't an error: don't throw, and change nothing. Apply the key and the guard before you write, so a write never touches an item the guard excludes. See Keyed Operations.

FilterSerialized

A filter tree. Each node has exactly one of five shapes, told apart by which keys it has.

type FilterSerialized = StrictUnion<AndFilterSerialized | OrFilterSerialized | NotFilterSerialized | ComparisonFilterSerialized | InFilterSerialized>;

The type forbids the keys of the other shapes, so reading filter.and on any node type-checks. Checking a key against undefined, such as filter.cmp !== undefined, narrows the node to its named type, from AndFilterSerialized to InFilterSerialized. 'cmp' in filter doesn't narrow it. How a comparison treats null, and so what not around it matches, is up to your connector: the kit doesn't define it. The Quickstart connector translates the shapes it can serve and refuses the rest:

export function toQuery(filter: FilterSerialized): SearchBody {
    if (filter.and !== undefined) {
        return { bool: { filter: filter.and.map((child) => toQuery(child)) } };
    }

    if (filter.or !== undefined) {
        return { bool: { should: filter.or.map((child) => toQuery(child)), minimum_should_match: 1 } };
    }

    if (filter.in?.field !== undefined && Array.isArray(filter.values)) {
        // The search API rejects `null` in `terms`, and `terms` never matches a missing value. So `null`
        // members are left out, and `in` never matches an item without a value.
        return { terms: { [apiFieldName(filter.in.field)]: filter.values.filter((value) => value !== null) } };
    }

    if (filter.cmp === 'equals' && filter.left.field !== undefined && filter.right.value !== undefined) {
        const name = apiFieldName(filter.left.field);
        // `field = null` means "is null": the search API's `exists` is false for a field without a value.
        if (filter.right.value === null) return { bool: { must_not: { exists: { field: name } } } };
        return { term: { [name]: filter.right.value } };
    }

    throw new QueryRejected('The Art Institute connector answers only `equals` and `in`, joined with `and` or `or`.');
}

This connector leaves null members out of in, so a null field never matches it.

See Filters and Map Caller Operators to the Wire.

AndFilterSerialized

A node that matches when every child in and matches.

interface AndFilterSerialized {
    and: FilterSerialized[];
}

OrFilterSerialized

A node that matches when at least one child in or matches.

interface OrFilterSerialized {
    or: FilterSerialized[];
}

NotFilterSerialized

A node that matches when its child doesn't.

interface NotFilterSerialized {
    not: FilterSerialized;
}

ComparisonFilterSerialized

A node that compares two operands with the FilterOperator in cmp.

interface ComparisonFilterSerialized {
    cmp: FilterOperator;
    left: FilterOperandSerialized;
    right: FilterOperandSerialized;
}

The Stripe connector evaluates a comparison in a function that takes this type, after it has handled the other shapes:

function compare(
    resource: ResourceSpec,
    { cmp: operator, left: leftOperand, right: rightOperand }: ComparisonFilterSerialized,
    item: MonospaceObject,
): Truth {

InFilterSerialized

A node that matches when the operand in in is one of values.

interface InFilterSerialized {
    in: FilterOperandSerialized;
    values: InSetSerialized;
}

FilterOperandSerialized

One side of a comparison: a field reference or a literal value.

type FilterOperandSerialized = StrictUnion<{
    field: Identifier;
} | {
    value: Value;
}>;

Read both sides of a comparison; the field isn't always on the left.

InSetSerialized

The set side of an in: literal values, or a reference to a list field.

type InSetSerialized = {
    field: Identifier;
} | List;

FilterOperator

The comparison operators a cmp node carries.

type FilterOperator = 'equals' | 'greaterThan' | 'greaterThanOrEquals' | 'lessThan' | 'lessThanOrEquals' | 'contains' | 'icontains' | 'startsWith' | 'endsWith';

Negated caller operators arrive as not around the positive form, and _between as and of greaterThanOrEquals and lessThanOrEquals. See Map Caller Operators to the Wire.

SortItemSerialized

One sort key of a readMany. The sort array applies them in order.

interface SortItemSerialized {
    field: Identifier;
    direction: SortDirection;
    nulls?: SortNulls;
}

Engine sends nulls only when the caller sets it. The Quickstart connector offers a sort by id only, and sorts by id when the request has no sort:

/**
 * Translates the caller's sort. Only `id` declares a sort, so a sort on any other field gets a plain
 * `Error`. `id` is never null, so `nulls` has no effect. Without a sort, pages still need a stable
 * order, so the default is `id` ascending.
 */
function toSort(sort: readonly SortItemSerialized[]): { id: { order: 'asc' | 'desc' } }[] {
    if (sort.length === 0) return [{ id: { order: 'asc' } }];

    return sort.map(({ field, direction }) => {
        if (field !== 'id') throw new Error(`The Art Institute connector doesn't offer sorting by \`${field}\`.`);
        return { id: { order: direction === 'descending' ? 'desc' : 'asc' } };
    });
}

SortDirection

The direction of a sort key, and of a field's declared sort.

type SortDirection = 'ascending' | 'descending';

SortNulls

Where a sort key places null values.

type SortNulls = 'first' | 'last';

Answer a Query

QueryResultSerialized

What readMany, create, updateMany and deleteMany answer with.

interface QueryResultSerialized {
    records?: MonospaceObject[];
    totalCount?: number;
}

Every item carries every field in select, with an explicit null for a missing value. See Answer a Query for the checks Engine applies.

KeyedQueryResultSerialized

What readOne, updateOne and deleteOne answer with: one item, or null.

interface KeyedQueryResultSerialized {
    record: MonospaceObject | null;
}

record is required, and the answer takes no other key. Engine fails a keyed operation that answers { records: [...] }, {}, or totalCount beside record. A caller sees record: null as 404 "No record found". See Keyed Operations.

QueryResults

The answer each operation owes, by its op.

interface QueryResults {
    readMany: QueryResultSerialized;
    readOne: KeyedQueryResultSerialized;
    create: QueryResultSerialized;
    updateMany: QueryResultSerialized;
    updateOne: KeyedQueryResultSerialized;
    deleteMany: QueryResultSerialized;
    deleteOne: KeyedQueryResultSerialized;
}

The keyed operations owe a KeyedQueryResultSerialized, every other operation a QueryResultSerialized. QueryResultFor reads its answer types from this map.

QueryResultFor

The answer the operations Op owe, exclusive of the other shape's keys.

type QueryResultFor<Op extends OperationName> = StrictUnionMember<QueryResults[Op], QueryResults[OperationName]>;

QueryResultFor<'readOne'> is a KeyedQueryResultSerialized that also forbids records and totalCount, and QueryResultFor<'readMany'> a QueryResultSerialized that forbids record. For a union of operations, it's the union of their answers, and an object with keys of both shapes fits neither. Use it as the return type of a function that answers some operations, as the Quickstart connector's readOne does:

export async function readOne(query: ReadOneOperationSerialized): Promise<QueryResultFor<'readOne'>> {

MonospaceObject

One item: field values keyed by the field names the connector declared.

interface MonospaceObject {
    [field: string]: Value;
}

The Quickstart connector builds each item from exactly the selected fields:

/**
 * Returns exactly the selected fields, with `null` where the API has no value (the API returns `null`
 * or leaves the field out). Monospace checks each value against the field's declared type.
 */
function toItem(apiItem: Record<string, Value>, select: readonly string[]): MonospaceObject {
    const item: MonospaceObject = {};
    for (const name of select) item[name] = apiItem[apiFieldName(name)] ?? null;
    return item;
}

Value

Any value on the wire: untagged JSON.

type Value = null | string | number | boolean | Value[] | {
    [key: string]: Value;
};

Types without a JSON form of their own travel as strings: decimals, base64 bytes, UUIDs, dates and times. A single int64 or uint64 value travels as a decimal string in data, in filter literals and in keys. In an answer, Engine accepts a decimal string or a number. Narrower integers are JSON numbers, and so are the elements of an int64 or uint64 list field. See Values and Field Types.

List

An array of values.

type List = Value[];

Identifier

A non-empty field, collection or namespace name.

type Identifier = string;

Requests name collections and fields by their name, never their apiName.

Describe the Schema Document

SchemaDocument

What introspect answers with: the namespaces, collections and relations the data source exposes.

interface SchemaDocument {
    namespaces?: CanonicalNamespace[];
    collections?: CanonicalCollection[];
    relations?: CanonicalRelation[];
}

Every section defaults to empty. The schema helpers build one from typed definitions; the Stripe connector answers a read-only variant when writes are off:

/**
 * The declared schema. A data source with `enableWrites: false` declares reads only: the kit leaves
 * out every member that writes, and so answers every collection and field as read-only.
 */
export function schemaDocument(enableWrites: boolean): SchemaDocument {
    return schema.toSchemaDocument({ writes: enableWrites });
}

See Describe the Schema.

CanonicalNamespace

A named group of collections.

interface CanonicalNamespace {
    name: Identifier;
    collections?: CanonicalCollection[];
}

A collection's identity is its namespace (or none) plus its name, so the same collection name can appear in different namespaces.

CanonicalCollection

One collection: its fields, indexes, and the operations it serves.

interface CanonicalCollection {
    name: Identifier;
    apiName?: Identifier;
    fields?: CanonicalField[];
    indexes?: CanonicalIndex[];
    isReadonly: boolean;
    operations: SupportedOperations;
}

operations is required, and isReadonly: true blocks every create, update and delete. The schema helpers derive isReadonly: a collection is read-only when it declares no member that writes. See Collections for each property. The full Art Institute connector reads its own declaration back from the document, to check every filter and sort it receives:

// Engine offers callers what the declaration it stored for the data source declares, and doesn't
// check the filters it sends against it. That stored declaration is the one from when the data source
// was created or last refreshed: after an extension update that narrows `schema.ts`, it can still
// offer what this code no longer translates. So the translation below checks each filter and sort
// against `schema.ts` itself. A field, operator or sort it doesn't declare is not the caller's to
// fix: a plain error, which Engine reports as a failed query.
const DECLARATIONS = new Map<string, Declaration>((schema.toSchemaDocument().collections ?? []).map((collection) => [
    collection.name,
    {
        read: collection.operations.readMany ?? {},
        types: new Map(collection.fields?.map((field) => [field.name, field.type.name])),
    },
]));

CanonicalField

One field of a collection.

interface CanonicalField {
    name: Identifier;
    apiName?: Identifier;
    type: PrimitiveFieldType;
    isNullable: boolean;
    isList: boolean;
    defaultValue?: DefaultValue;
    isReadonly: boolean;
}

isReadonly: true keeps the field out of create and update data. The schema helpers derive it: a field is read-only when no data list names it. See Fields.

PrimitiveFieldType

A field's type: a name from a closed set, with params for decimal and string only.

type PrimitiveFieldType = {
    name: ParameterlessPrimitiveName;
    params?: never;
} | {
    name: 'decimal';
    params: DecimalPrecision;
} | {
    name: 'string';
    params: {
        length: 'unlimited';
        value?: never;
    } | {
        length: 'fixed';
        value: number;
    };
};

An unrecognized name rejects the schema. See Field Types for how each type travels on the wire.

ParameterlessPrimitiveName

The field type names that take no params.

type ParameterlessPrimitiveName = 'boolean' | 'bytes' | 'date' | 'dateTime' | 'dateTimeWithTimezone' | 'float32' | 'float64' | 'geometry' | 'int16' | 'int32' | 'int64' | 'int8' | 'json' | 'time' | 'uint16' | 'uint32' | 'uint64' | 'uint8' | 'unsupported' | 'uuid';

DecimalPrecision

The params of a decimal field: unconstrained, or a declared precision and scale.

type DecimalPrecision = {
    kind: 'unconstrained';
    precision?: never;
    scale?: never;
} | {
    kind: 'constrained';
    precision: number;
    scale: number;
};

DefaultValue

A field's default: supplied by the connector, or evaluated by Engine.

type DefaultValue = {
    source: 'connectorManaged';
    isAutoincrement?: boolean;
} | {
    source: 'engineManaged';
    onCreate?: EngineDefaultSource;
    onUpdate?: EngineDefaultSource;
};

connectorManaged means your connector or the external data source supplies the value. Engine evaluates an engineManaged default before it calls your connector, for a field the operation's input offers and the caller omits. See Defaults.

EngineDefaultSource

Where an engineManaged default's value comes from.

type EngineDefaultSource = {
    kind: 'currentTimestamp';
} | {
    kind: 'currentUser';
} | {
    kind: 'randomUuid';
    version: 'v4' | 'v7';
} | {
    kind: 'static';
    valueEncoded: Value;
};

currentUser and randomUuid need a uuid field, currentTimestamp a date or time field, and a static value must fit the field's type.

CanonicalIndex

One index of a collection.

interface CanonicalIndex {
    name: Identifier;
    fields: Identifier[];
    kind: IndexKind;
}

fields lists field names in order. The primary index decides which keyed operations a collection can offer. See Keys and the Stable ID.

IndexKind

The kind of an index.

type IndexKind = 'primary' | 'unique' | 'normal';

CanonicalRelation

A link from a constrained collection, which holds the fields that point at the other side, to a referenced collection.

interface CanonicalRelation {
    name: Identifier;
    constrainedNamespace?: Identifier;
    constrainedCollection: Identifier;
    constrainedFields: Identifier[];
    constrainedRelationFieldName?: Identifier;
    referencedNamespace?: Identifier;
    referencedCollection: Identifier;
    referencedFields: Identifier[];
    referencedRelationFieldName?: Identifier;
    onDelete: ReferentialAction;
    onUpdate: ReferentialAction;
}

constrainedFields and referencedFields pair up by position. See Relations for each property and Resolve Relations for what a relation needs to be offered.

ReferentialAction

The onDelete and onUpdate action of a relation.

type ReferentialAction = 'noAction' | 'restrict' | 'cascade' | 'setNull' | 'setDefault';

Declare Operations

A collection's operations declare what it serves: which members it offers, and for each member the filters, sorts, paging and fields it accepts. Engine offers callers nothing more. See Operations and Operators.

SupportedOperations

The operations a collection serves. An absent member isn't offered.

interface SupportedOperations {
    readMany?: ReadArguments;
    readOne?: ReadOneArguments;
    create?: CreateArguments;
    updateMany?: UpdateArguments;
    updateOne?: UpdateArguments;
    deleteMany?: DeleteArguments;
    deleteOne?: DeleteArguments;
}

Declaring a keyed member (readOne, updateOne, deleteOne) means accepting it with a key for the collection's single-field primary key, and answering it with one record or null; its filter governs only the guard. Without the connector's updateReturning or deleteReturning capability, Engine sends a declared updateOne or deleteOne as updateMany or deleteMany by stable ID, with select: [], and takes nothing from the answer. See Choose Members and What Engine Sends.

Engine checks every declaration when a data source is created or reintrospected. For example, a field list can't repeat a name, and output can't be empty. See Follow the Declaration Rules.

ReadArguments

What a readMany accepts. Callers can't pass an argument the declaration leaves out.

interface ReadArguments {
    filter?: FilterContract;
    sort?: Record<Identifier, SortField>;
    limit?: Record<string, never>;
    offset?: Record<string, never>;
    output?: Identifier[];
    meta?: {
        totalCount?: boolean;
    };
}

limit and offset are {} when present, and a connector can't set a ceiling on either. Without limit, a readMany still carries Engine's default limit when the caller sets none. meta.totalCount offers count reads. See Arguments.

ReadOneArguments

What a readOne accepts: the guard filter and the output fields.

interface ReadOneArguments {
    filter?: FilterContract;
    output?: Identifier[];
}

CreateArguments

What a create accepts: the fields callers may set, and the output fields.

interface CreateArguments {
    data?: Identifier[];
    output?: Identifier[];
}

An absent data and [] mean the same: callers set no field, and each created item arrives as {}.

UpdateArguments

What updateMany and updateOne accept.

interface UpdateArguments {
    filter?: FilterContract;
    data?: Identifier[];
    output?: Identifier[];
}

An absent data and [] mean the same: the update sets no field. When it also offers no nested relation write, it sets nothing, so Engine doesn't offer it.

DeleteArguments

What deleteMany and deleteOne accept.

interface DeleteArguments {
    filter?: FilterContract;
    output?: Identifier[];
}

FilterContract

The filters a member accepts: operators per field, and the logical operators that join them.

interface FilterContract {
    fields: Record<Identifier, FilterField>;
    logicalOperators?: Partial<Record<LogicalOperator, LogicalOperatorOptions>>;
}

No field is required in a filter, and every declared operator can occur any number of times anywhere in the tree. Filters name only the collection's own fields. The full Art Institute connector checks each comparison it receives against its contract:

/** Refuses `operator` on `field` unless `contract` declares it. A value in the field's place is a shape `translate` refuses. */
function requireOperator(contract: FilterContract | undefined, field: string | undefined, operator: DeclaredOperator): void {
    if (field === undefined) return;
    const operators = contract?.fields[field]?.operators;

    if (operators === undefined || !Object.hasOwn(operators, operator)) {
        throw new Error(`The Art Institute connector can't filter \`${field}\` with \`${operator}\`.`);
    }
}

Build these maps with filter.field and filter.logical.

FilterField

The operators a filter may apply to one field, each with its options.

interface FilterField {
    operators: Partial<Record<DeclaredOperator, DeclaredOperatorOptions>>;
}

DeclaredOperator

An operator a field can declare: every FilterOperator, plus in.

type DeclaredOperator = FilterOperator | 'in';

Which operators a field can declare depends on its type. See Match Operators to Field Types.

DeclaredOperatorOptions

The options of one declared operator. None exist yet: declare {}.

type DeclaredOperatorOptions = Record<string, never>;

LogicalOperator

The logical operators a filter can declare.

type LogicalOperator = 'and' | 'not' | 'or';

LogicalOperatorOptions

The options of one declared logical operator. None exist yet: declare {}.

type LogicalOperatorOptions = Record<string, never>;

SortField

The directions one field can be sorted in.

interface SortField {
    directions: SortDirection[];
}

Build it with sort.field.

See Also

• Data Connector API: what Engine sends in each request and expects in each answer
• Operations and Operators: declare what each collection serves, and the rules Engine enforces
• Data Connector Module: defineDataConnector, the handlers and the error classes
• Schema Helpers: build the schema document from typed definitions