Schema Helpers

Overview

@monospace/extension-kit/data-connector/experimental builds the SchemaDocument a data connector extension answers introspect with. A data connector exposes an external data source as collections of items, and each item carries named values called fields. The helpers describe those collections, fields and relations with typed definitions, check them, and serialize them.

The helpers are optional. introspect can return a hand-written SchemaDocument instead, with the same shape.

Each entry below shows the declaration from the kit's type definitions. Several signatures name the kit's internal helper types (e.g., DeepReadonly, Check, FieldReferences); those carry the compile-time checks and aren't exported.

Build a Schema

The full Art Institute connector declares its two read-only collections and the relation between them with the helpers. The Data Connector Quickstart builds a trimmed version that sorts only by id, without ranges or counts:

import { defineCollection, defineRelation, defineSchema, field, fields, filter, index, paging, sort } from '@monospace/extension-kit/data-connector/experimental';

// The operators the Art Institute's search API answers exactly. The relation's key fields, `id` and
// `artistId`, declare `in` as well as `equals`, because Engine resolves relations by filtering with `in`.
const exact = filter.field('equals', 'in');
const comparable = filter.field('equals', 'in', 'lessThan', 'lessThanOrEquals', 'greaterThan', 'greaterThanOrEquals');
/** Engine executes only `equals` on booleans. */
const booleanExact = filter.field('equals');
const eitherDirection = sort.field('ascending', 'descending');

const artworkFilter = {
    fields: { id: exact, artistId: exact, dateStart: comparable, isPublicDomain: booleanExact },
    logicalOperators: filter.logical('and', 'or'),
};

// The API's ids and years are small integers, so they're `int32`, which travels as a JSON number. A
// 64-bit integer would travel as a string. Neither collection declares a write, so both are read-only.
export const artworks = defineCollection({
    name: 'artworks',
    fields: {
        id: field.int32(),
        title: field.string({ nullable: true }),
        artistId: field.int32({ nullable: true }),
        artistTitle: field.string({ nullable: true }),
        dateStart: field.int32({ nullable: true }),
        dateEnd: field.int32({ nullable: true }),
        dateDisplay: field.string({ nullable: true }),
        isPublicDomain: field.boolean(),
        imageId: field.string({ nullable: true }),
    },
    indexes: { artworks_pkey: index.primary(['id']) },
    operations: {
        readMany: {
            filter: artworkFilter,
            sort: { id: eitherDirection, dateStart: eitherDirection },
            limit: paging.limit(),
            offset: paging.offset(),
            output: fields.all(),
            meta: { totalCount: true },
        },
        readOne: { filter: artworkFilter, output: fields.all() },
    },
});

const artistFilter = {
    fields: { id: exact, birthDate: comparable },
    logicalOperators: filter.logical('and', 'or'),
};

export const artists = defineCollection({
    name: 'artists',
    fields: {
        id: field.int32(),
        title: field.string(),
        birthDate: field.int32({ nullable: true }),
        deathDate: field.int32({ nullable: true }),
    },
    indexes: { artists_pkey: index.primary(['id']) },
    operations: {
        readMany: {
            filter: artistFilter,
            sort: { id: eitherDirection, birthDate: eitherDirection },
            limit: paging.limit(),
            offset: paging.offset(),
            output: fields.all(),
            meta: { totalCount: true },
        },
        readOne: { filter: artistFilter, output: fields.all() },
    },
});

export const schema = defineSchema({
    collections: [artworks, artists],
    relations: [
        // Engine resolves `artist` by filtering artists with `in` on `id`, and `artworks` by filtering
        // artworks with `in` on `artistId`. Both filters declare `in`, so both directions are offered.
        defineRelation({
            name: 'artworks_artist',
            from: artworks,
            to: artists,
            on: [['artistId', 'id']],
            names: { forward: 'artist', reverse: 'artworks' },
        }),
    ],
});

Return schema.toSchemaDocument() from introspect. The Data Connector API shows the document it produces for artists.

The define* functions and the field, index, default, filter, sort and paging builders return deep-frozen copies, so a definition can't change after the call.

Define Collections and Relations

defineCollection

Defines one collection: its fields, indexes, and the operations it serves.

function defineCollection<const F extends Readonly<Record<string, FieldDefinition>>, const C extends Omit<CollectionDefinition, 'fields' | 'indexes' | 'operations'> & {
    readonly fields: F;
} & FieldReferences<NoInfer<F>> & DerivedFlags>(definition: C & {
    readonly fields: F;
} & FieldReferences<NoInfer<F>> & Check<CollectionErrors<ExpandedCollection<NoInfer<C>>>>): DeepReadonly<ExpandedCollection<C>>;

Returns the definition, with every fields.all() replaced by the collection's field names, in the order Object.keys lists the fields map: integer-like names such as "10" first, in ascending order, then the rest in declaration order. Collections don't declare isReadonly: passing it is a type error. A collection is read-only exactly when operations declares no member that writes, and toSchemaDocument derives the flag.

defineCollection checks the field names in operations at compile time: the keys of filter.fields and sort, and the names in data and output. A name the collection doesn't declare is a TypeScript error such as Unknown operation field: <root>.articles.operations.readMany.titel. At runtime it throws SchemaDefinitionError; see Validation for the rules.

defineRelation

Defines a relation from the collection holding the pointing fields to the one it points at.

function defineRelation<const F extends CollectionDefinition, const T extends CollectionDefinition, const P extends readonly (readonly [
    FieldHint<NoInfer<F>['fields']>,
    FieldHint<NoInfer<T>['fields']>
])[], const N extends string, const Names extends RelationDefinition['names']>(definition: {
    name: N;
    from: F;
    to: T;
    on: P & readonly (readonly [
        FieldHint<NoInfer<F>['fields']>,
        FieldHint<NoInfer<T>['fields']>
    ])[] & PairChecks<NoInfer<F>, NoInfer<T>, NoInfer<P>>;
    names: Names;
    onDelete?: ReferentialAction;
    onUpdate?: ReferentialAction;
}): DeepReadonly<{
    name: N;
    from: F;
    to: T;
    on: P;
    names: Names;
    onDelete?: ReferentialAction;
    onUpdate?: ReferentialAction;
}>;

In the schema document, from becomes the constrained collection and to the referenced one. Each relation direction is offered only when the other collection's filter can serve it. See Resolve Relations.

defineSchema

Validates the whole schema and returns it with toSchemaDocument.

function defineSchema<const S extends SchemaDefinition>(definition: S & Check<SchemaErrors<NoInfer<S>>>): DefinedSchema<S>;

Returns a DefinedSchema. Call toSchemaDocument() on it in introspect.

defineNamespace

Defines a namespace, a named group of collections.

function defineNamespace<const N extends NamespaceDefinition>(definition: N): DeepReadonly<N>;

Register it in defineSchema({ namespaces }) and reference it from defineCollection({ namespace }). An empty name throws SchemaDefinitionError.

Declare Fields

field

Field constructors, one per field type. They describe fields; they never parse or generate values.

const field: {
    boolean: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "boolean";
    }, NoInfer<O>>;
    bytes: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "bytes";
    }, NoInfer<O>>;
    date: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "date";
    }, NoInfer<O>>;
    dateTime: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "dateTime";
    }, NoInfer<O>>;
    dateTimeWithTimezone: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "dateTimeWithTimezone";
    }, NoInfer<O>>;
    float32: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "float32";
    }, NoInfer<O>>;
    float64: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "float64";
    }, NoInfer<O>>;
    geometry: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "geometry";
    }, NoInfer<O>>;
    int8: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "int8";
    }, NoInfer<O>>;
    int16: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "int16";
    }, NoInfer<O>>;
    int32: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "int32";
    }, NoInfer<O>>;
    int64: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "int64";
    }, NoInfer<O>>;
    json: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "json";
    }, NoInfer<O>>;
    time: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "time";
    }, NoInfer<O>>;
    uint8: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "uint8";
    }, NoInfer<O>>;
    uint16: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "uint16";
    }, NoInfer<O>>;
    uint32: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "uint32";
    }, NoInfer<O>>;
    uint64: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "uint64";
    }, NoInfer<O>>;
    unsupported: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "unsupported";
    }, NoInfer<O>>;
    uuid: <const O extends FieldOptions = Record<never, never>>(options?: O) => DefinedField<{
        name: "uuid";
    }, NoInfer<O>>;
    string<const O extends StringFieldOptions = Record<never, never>>(options?: O): DefinedField<{
        name: "string";
        params: {
            length: "unlimited";
            value?: never;
        } | {
            length: "fixed";
            value: number;
        };
    }, NoInfer<Omit<O, "length">>>;
    decimal<const O extends DecimalFieldOptions = Record<never, never>>(options?: O): DefinedField<{
        name: "decimal";
        params: DecimalPrecision;
    }, NoInfer<Omit<O, "precision" | "scale">>>;
};

Each constructor takes an optional FieldOptions object and returns a field definition:

Fields default to scalar and non-nullable: field.int32() returns { type: { name: 'int32' }, isNullable: false, isList: false }. See Field Types for how each type travels on the wire.

field.string

A string field. field.string() is unlimited, and field.string({ length: 255 }) is fixed-length. Takes StringFieldOptions.

field.decimal

A decimal field. field.decimal() is unconstrained, and field.decimal({ precision: 12, scale: 2 }) declares a precision and scale. Pass both or neither; one alone is a type error. Takes DecimalFieldOptions.

FieldOptions

The options every field constructor takes.

interface FieldOptions extends Partial<Omit<FieldDefinition, 'type' | 'isNullable' | 'isList'>> {
    nullable?: boolean | undefined;
    list?: boolean | undefined;
    isReadonly?: never;
}

Fields don't declare isReadonly: passing it is a type error. A field is writable exactly when a data list in operations names it, and toSchemaDocument derives the flag.

StringFieldOptions

The options of field.string: every field option, plus length.

interface StringFieldOptions extends FieldOptions {
    length?: number | 'unlimited';
}

length is a number for a fixed length, or 'unlimited', the default.

DecimalFieldOptions

The options of field.decimal: every field option, plus precision and scale together or not at all.

type DecimalFieldOptions = FieldOptions & ({
    precision: number;
    scale: number;
} | {
    precision?: never;
    scale?: never;
});

defaultValue

Builds a field's defaultValue.

const defaultValue: {
    connectorManaged(options?: {
        isAutoincrement?: boolean;
    }): {
        readonly source: "connectorManaged";
        readonly isAutoincrement?: boolean;
    };
    engineManaged<const O extends DeepReadonly<Omit<Extract<DefaultValue, {
        source: "engineManaged";
    }>, "source">>>(options: O): DeepReadonly<O & {
        source: "engineManaged";
    }>;
};

A non-nullable field that no data list names needs a default, or defineCollection rejects a create on its collection. The Stripe connector marks the values Stripe sets itself:

// Values Stripe sets itself: callers never have to supply them on create.
const setByStripe = { defaultValue: defaultValue.connectorManaged() };
const id = field.string(setByStripe);
const created = field.dateTimeWithTimezone(setByStripe);
const nullableText = field.string({ nullable: true });

See Defaults for when Engine evaluates each kind.

defaultSource

Builds the onCreate and onUpdate sources of an engineManaged default.

const defaultSource: {
    static(valueEncoded: Value): DeepReadonly<Extract<EngineDefaultSource, {
        kind: "static";
    }>>;
    currentTimestamp(): Readonly<{
        readonly kind: "currentTimestamp";
    }>;
    currentUser(): Readonly<{
        readonly kind: "currentUser";
    }>;
    randomUuid(version: Extract<EngineDefaultSource, {
        kind: "randomUuid";
    }>["version"]): {
        readonly kind: "randomUuid";
        readonly version: "v4" | "v7";
    };
};

index

Builds a collection's indexes.

const index: {
    primary<const F extends readonly string[]>(fields: F): DeepReadonly<{
        kind: "primary";
        fields: F;
    }>;
    unique<const F extends readonly string[]>(fields: F): DeepReadonly<{
        kind: "unique";
        fields: F;
    }>;
    normal<const F extends readonly string[]>(fields: F): DeepReadonly<{
        kind: "normal";
        fields: F;
    }>;
};

index.primary(fields), index.unique(fields) and index.normal(fields) return { kind, fields }, with the field order preserved. A collection has at most one primary index. Keyed operations need a single-field primary index. See Keys and the Stable ID.

Declare Operations

These helpers build the arguments of a collection's operations. See Operations and Operators for what each member and argument means.

filter

Builds the parts of a member's filter.

const filter: {
    field<const O extends AtLeastOne<DeclaredOperator>>(...operators: O): {
        readonly operators: DeepReadonly<{
            readonly [P in O[number]]: NoOptions;
        }>;
    };
    logical<const O extends AtLeastOne<LogicalOperator>>(...operators: O): DeepReadonly<{
        readonly [P in O[number]]: NoOptions;
    }>;
};

filter.field

One field's entry in filter.fields. filter.field('equals', 'in') returns { operators: { equals: {}, in: {} } }. Takes at least one DeclaredOperator.

filter.logical

A filter's logicalOperators. filter.logical('and', 'or') returns { and: {}, or: {} }. Takes at least one of and, or and not.

The Stripe connector shares these declarations between its collections:

const exact = filter.field('equals', 'in');
/** Engine executes only `equals` on booleans. */
const booleanExact = filter.field('equals');
const createdRange = filter.field('equals', 'lessThan', 'lessThanOrEquals', 'greaterThan', 'greaterThanOrEquals');
const andOr = filter.logical('and', 'or');
/** Stripe lists newest first and can't sort any other way. */
const newestFirst = { created: sort.field('descending') };

Which operators a field can declare depends on its type. See Match Operators to Field Types.

sort

Builds the entries of a readMany's sort.

const sort: {
    field<const D extends AtLeastOne<SortDirection>>(...directions: D): DeepReadonly<{
        directions: D;
    }>;
};

sort.field

One field's entry in sort. sort.field('descending') returns { directions: ['descending'] }. Takes at least one SortDirection.

paging

Builds a readMany's limit and offset.

const paging: {
    limit(): {};
    offset(): {};
};

paging.limit

Offers limit on readMany. Returns {}.

paging.offset

Offers offset on readMany. Returns {}.

A readMany that declares neither still receives Engine's default limit when the caller sets none. See Handle Requests You Didn't Declare.

fields

Builds field lists inside defineCollection.

const fields: {
    all(): AllFields;
};

fields.all

Every field of the collection, as a data or output list. The names come in the order Object.keys lists the fields map: integer-like names such as "10" first, in ascending order, then the rest in declaration order. It returns a marker that defineCollection replaces with the field names. Write it inside the defineCollection call: assigned to a variable first, it widens to symbol, and defineCollection refuses it with a type error, unless you type the variable as AllFields.

fieldsOf

Builds field lists outside defineCollection, for a list several members share.

function fieldsOf<const F extends Readonly<Record<string, FieldDefinition>>>(map: F): {
    pick<const K extends AtLeastOne<keyof F & string>>(...names: K): DeepReadonly<K>;
};

fieldsOf(map).pick(...names) returns the named fields of map, in the order given. Editors complete the names, and the list keeps its literal type, so defineCollection still checks it. A name the map doesn't have is a type error, and pick throws SchemaDefinitionError for it at runtime. The Stripe connector shares one data list between the updates of products:

const productFields = {
    id: field.string(setByStripe),
    name: field.string(),
    description: nullableText,
    active: field.boolean(setByStripe),
    defaultPrice: field.string({ nullable: true }),
    created,
};
const productFilter = { fields: { id: exact, active: booleanExact, created: createdRange }, logicalOperators: andOr };
const productUpdate = fieldsOf(productFields).pick('name', 'description', 'active');

export const products = defineCollection({
    name: 'products',
    fields: productFields,
    indexes: { products_pkey: index.primary(['id']) },
    operations: {
        readMany: { filter: productFilter, sort: newestFirst, limit: paging.limit(), offset: paging.offset(), output: fields.all() },
        readOne: { filter: productFilter, output: fields.all() },
        // Stripe accepts a caller-chosen product id on create.
        create: { data: ['id', ...productUpdate], output: fields.all() },
        updateMany: { filter: productFilter, data: productUpdate, output: fields.all() },
        updateOne: { filter: productFilter, data: productUpdate, output: fields.all() },
        deleteMany: { filter: productFilter, output: fields.all() },
        deleteOne: { filter: productFilter, output: fields.all() },
    },
});

products declares writes, so the fields its data lists name are writable and every other field is read-only. Inline arrays of field names also work. A list typed string[], such as Object.keys(productFields), passes the compile-time check whatever it holds; defineCollection still rejects an unknown name when it runs.

AllFields

The type of the marker fields.all() returns.

type AllFields = typeof allFields;

Handle Validation Errors

SchemaDefinitionError

Thrown by the helpers when a definition breaks a structural rule.

class SchemaDefinitionError extends Error {
    readonly issues: readonly SchemaIssue[];
    constructor(issues: readonly SchemaIssue[]);
}

issues lists every problem found, as SchemaIssue objects. The message starts with Invalid schema definition: and lists each issue as path: message on its own line, for example collections.x.operations.readMany.output: Output must list at least one field. With literal definitions, TypeScript reports several of these mistakes before the code runs: unknown field names in operations, indexes and a relation's on.

SchemaIssue

One problem in a definition: where it is and what's wrong.

interface SchemaIssue {
    readonly path: string;
    readonly message: string;
}

Validation

• defineCollection checks the collection's names, fields and indexes. It rejects empty names, two fields with the same API name, an index without fields, a field repeated in one index, index fields that don't exist, and more than one primary index. A fixed string length, and a decimal's precision and scale, must be non-negative safe integers.
• defineCollection also checks operations. It rejects unknown field names and repeated names. It also rejects an empty filter.fields, operators, sort, directions or output. When the collection declares create, it rejects a non-nullable field without a default that no data list names, because nothing could fill it.
• defineRelation checks that on is non-empty, that no field repeats on either side, that both fields exist and are scalar, and that paired fields have the same type name. For example, uuid with string, int32 with int64, and date with dateTime don't pair.
• defineSchema checks the whole graph. It rejects duplicate collections, two collections with the same API name in one namespace, and unregistered namespaces and relation endpoints. It also rejects relation field names that collide with fields.
• defineNamespace rejects an empty name, and fieldsOf(…).pick a name its map doesn't have.

Rules that depend on field types, indexes or relations, such as which operators a type allows, are checked by Engine when the data source is created or reintrospected. See Declaration Rules.

Use the Definition Types

These types describe definitions as the helpers take and return them. Use them to type code that reads a collection's declaration.

SchemaDefinition

What defineSchema takes: collections, and optional namespaces and relations.

interface SchemaDefinition {
    readonly namespaces?: readonly NamespaceDefinition[];
    readonly collections: readonly CollectionDefinition[];
    readonly relations?: readonly RelationDefinition[];
}

DefinedSchema

What defineSchema returns: the frozen definition plus toSchemaDocument.

type DefinedSchema<S extends SchemaDefinition = SchemaDefinition> = DeepReadonly<S> & {
    toSchemaDocument: (options?: SchemaDocumentOptions) => SchemaDocument;
};

toSchemaDocument(options?) returns a new SchemaDocument each call. It lists namespaced collections under their namespace, and derives every isReadonly flag from operations. Relations get onDelete and onUpdate 'noAction' unless the definition sets them. Pass SchemaDocumentOptions to leave out the members that write.

SchemaDocumentOptions

The options of toSchemaDocument.

interface SchemaDocumentOptions {
    writes?: boolean;
}

writes: false leaves out create, updateMany, updateOne, deleteMany and deleteOne, so every collection and field is read-only. The default is true. The Stripe connector passes its enableWrites setting:

/**
 * The declared schema. A data source with `enableWrites: false` declares reads only: the kit leaves
 * out every member that writes, and so answers every collection and field as read-only.
 */
export function schemaDocument(enableWrites: boolean): SchemaDocument {
    return schema.toSchemaDocument({ writes: enableWrites });
}

CollectionDefinition

A collection as authored, without isReadonly.

interface CollectionDefinition extends DeepReadonly<Omit<CanonicalCollection, 'fields' | 'indexes' | 'isReadonly' | 'operations'>> {
    readonly namespace?: NamespaceDefinition;
    readonly fields: Readonly<Record<string, FieldDefinition>>;
    readonly indexes?: Readonly<Record<string, IndexDefinition>>;
    readonly operations: {
        readonly [M in keyof SupportedOperations]?: DeepReadonly<NonNullable<SupportedOperations[M]>> | undefined;
    };
}

The Stripe connector reads its declared filters and data lists through it:

/** A filter contract as `defineCollection` keeps it. */
export type DeclaredFilter = NonNullable<CollectionDefinition['operations']['readMany']>['filter'];

FieldDefinition

A field as authored, without its name and isReadonly.

type FieldDefinition = DeepReadonly<Omit<CanonicalField, 'name' | 'isReadonly'>>;

The Stripe connector maps each field's declared type to how its query code treats the values:

function kindOf(name: string, definition: FieldDefinition): FieldKind {
    // `id` is a kind of its own: the plan answers it by retrieving, not through a list parameter.
    if (name === 'id') return 'id';

    switch (definition.type.name) {
        case 'string': return 'string';
        case 'int64': return 'int64';
        case 'boolean': return 'boolean';
        case 'dateTimeWithTimezone': return 'timestamp';
        default: throw new Error(`Stripe field \`${name}\` has a type the connector doesn't handle: \`${definition.type.name}\`.`);
    }
}

IndexDefinition

An index as authored, without its name.

type IndexDefinition = DeepReadonly<Omit<CanonicalIndex, 'name'>>;

NamespaceDefinition

A namespace as authored.

interface NamespaceDefinition {
    readonly name: string;
}

RelationDefinition

A relation as authored.

interface RelationDefinition {
    readonly name: string;
    readonly from: CollectionDefinition;
    readonly to: CollectionDefinition;
    readonly on: readonly (readonly [
        string,
        string
    ])[];
    readonly names: {
        readonly forward: string;
        readonly reverse: string;
    };
    readonly onDelete?: CanonicalRelation['onDelete'];
    readonly onUpdate?: CanonicalRelation['onUpdate'];
}

See Also

• Data Connector API: what each part of the schema document means to Engine
• Operations and Operators: the declaration rules Engine checks
• Protocol Types: the schema document types the helpers produce
• Build a Stripe Data Connector: a schema with writes, defaults and shared field lists