Operations and Operators

Declare Operations

Every collection a data connector extension exposes declares operations: the members it serves and what each one accepts. A collection holds items, and each item carries named values called fields. A data source is the connector configured in a workspace, and the external data source is the system the connector reads and writes.

Engine derives what it offers callers from the declaration: the operations, the filter operators, the sort fields, and the fields callers can read and write. Engine rejects a caller request outside it before the request reaches your connector, except for the shapes listed below the example. An absent member or argument is not offered, and operations: {} offers nothing.

The artists collection of the full Art Institute connector, which Map an External Data Source walks through, serves a filterable, sortable, paged list with a total count, and single-item reads, with id as its single-field primary key. It declares its operations with the helpers from @monospace/extension-kit/data-connector/experimental:

import { defineCollection, defineRelation, defineSchema, field, fields, filter, index, paging, sort } from '@monospace/extension-kit/data-connector/experimental';

// The operators the Art Institute's search API answers exactly. The relation's key fields, `id` and
// `artistId`, declare `in` as well as `equals`, because Engine resolves relations by filtering with `in`.
const exact = filter.field('equals', 'in');
const comparable = filter.field('equals', 'in', 'lessThan', 'lessThanOrEquals', 'greaterThan', 'greaterThanOrEquals');
/** Engine executes only `equals` on booleans. */
const booleanExact = filter.field('equals');
const eitherDirection = sort.field('ascending', 'descending');
// …

const artistFilter = {
    fields: { id: exact, birthDate: comparable },
    logicalOperators: filter.logical('and', 'or'),
};

export const artists = defineCollection({
    name: 'artists',
    fields: {
        id: field.int32(),
        title: field.string(),
        birthDate: field.int32({ nullable: true }),
        deathDate: field.int32({ nullable: true }),
    },
    indexes: { artists_pkey: index.primary(['id']) },
    operations: {
        readMany: {
            filter: artistFilter,
            sort: { id: eitherDirection, birthDate: eitherDirection },
            limit: paging.limit(),
            offset: paging.offset(),
            output: fields.all(),
            meta: { totalCount: true },
        },
        readOne: { filter: artistFilter, output: fields.all() },
    },
});

filter.field(...), filter.logical(...), sort.field(...), paging.limit(), paging.offset() and fields.all() build the declaration's values. artists declares no member that writes, so the kit marks it read-only. In the schema document the connector returns, artistFilter becomes maps from each operator to its options:

{
  "fields": {
    "id": { "operators": { "equals": {}, "in": {} } },
    "birthDate": {
      "operators": {
        "equals": {},
        "in": {},
        "lessThan": {},
        "lessThanOrEquals": {},
        "greaterThan": {},
        "greaterThanOrEquals": {}
      }
    }
  },
  "logicalOperators": { "and": {}, "or": {} }
}

No operator has options yet, so each value is {}. See Schema Helpers for every helper, and Example Schema Document for the whole document the Art Institute connector returns for artists.

Engine checks a caller's request against the declaration, but not the filters it sends to your connector. Some requests reach your connector in a shape the declaration doesn't list. These include caller shorthands Engine expands: shorthand equality, _null, and sorting without a direction on a field that declares ascending. A default limit, Engine's own calls for relations and read-backs, and permission rule conditions are others. See Handle Requests You Didn't Declare.

Choose Members

A collection declares any of seven members:

A keyed member receives a key for the collection's primary key on every call. Its filter declares only the guard sent beside the key, never the key itself. See Use Keys and the Stable ID.

Arguments

• Field lists are sets. Don't repeat a name within one list.
• Build output and data with the helpers. fields.all(), fieldsOf(fields).pick(...) and inline lists let defineCollection check their names at compile time. See fields.all and fieldsOf.
• data on create. Without data, callers set none of this collection's fields: each item reaches your connector as {}, and your connector fills every field itself. A nested create into a related collection can still be offered, because it follows that collection's own create. A data field that isn't nullable and has no default is required in the caller's create input.
• data on an update. An update that can set no field and no nested relation write sets nothing, so Engine doesn't offer it.
• Narrow readMany.output for detail-only fields. A field listed in readOne.output but not in readMany.output is readable through readOne only. Each write member returns only the fields its own output lists.

Map Caller Operators to the Wire

Engine derives the filter operators callers see from what you declare. This table shows what each caller operator requires and how it reaches your connector.

Several conditions in one filter object combine with an implicit and. Without and in logicalOperators, a caller's filter object can hold only one condition. A filter object always holds at least one. See Filtering for the caller-side syntax.

Match Operators to Field Types

Engine accepts an operator on a field only if it can execute that operator on the field's type:

Every collection can declare and, or and not. Every field in the collection's fields can be declared in sort, in either direction. Relation fields can't be sorted.

Declaring an operator the type doesn't allow rejects the data source. The error names the collection, the path and the operator, for example "readMany.filter.fields.status declares greaterThan, which this engine cannot execute on Boolean".

Follow the Declaration Rules

Engine checks every declaration when a data source is created or reintrospected. A declaration that breaks one of the rules below rejects the whole request with a 422 whose message names the collection and what to declare. A schema document Engine can't read at all, such as one that repeats a field name in output or data, fails before these checks, with a 500.

If a newer Engine refuses a declaration it stored earlier, the collection loads with no operations and Engine logs an error. It stays that way until you reintrospect the data source.

Shape Rules

• Every collection declares operations. A collection without it is refused.
• Members name only the collection's own fields. A filter, sort, data or output that names an unknown field is refused.
• Lists aren't empty. filter.fields, each operators, sort, each directions and output must list at least one entry. data is the exception.
• Operators fit the field type. See Match Operators to Field Types.
• data lists writable fields. A read-only field or an unsupported field can't be in data. The helpers derive each field's isReadonly from the data lists, so they never conflict.

Member Rules

Some members can't be declared on some collections:

Rules for Engine's Own Calls

Engine re-reads and reads back items through readMany, addressing them by the collection's stable ID. The declaration must carry those calls:

• Write filters stay within readMany.filter. The filter of updateMany, updateOne, deleteMany and deleteOne can only use operators and logical operators that readMany.filter declares for the same fields.
• Keyed writes need equals on the stable ID. updateOne and deleteOne require equals on every stable-ID field, and and, in readMany.filter.
• Outputs carry the stable ID. readMany, updateMany and updateOne must list every stable-ID field in output. So must create when the connector doesn't declare insertReturning.
• Written items need a read-back path. Engine reads the items of a write by stable ID through readMany: after a create without insertReturning, before a delete without deleteReturning, and for every update. With updateReturning, Engine can fold an update's read into the update call, but the declaration must carry the read either way. A collection with a stable ID that serves such a write needs in on its stable-ID field and and in readMany.filter. Without in, equals on the field with and and or also works. A stable ID of several fields needs equals on each, with and and or. readMany.output must list the stable ID and every field the write's output lists.

A connector that doesn't declare every query capability adds more rules. See Declare Query Capabilities.

Some keys can't carry the read-back filter: a single boolean, bytes, json or geometry field, which has no in, or any key containing an unsupported field. Such a stable ID or relation key is exempt from the filter requirements of the read-back and relation rules only. Engine sends its read-backs and relation reads over it anyway, with an operator the declaration doesn't list. Serve them: see Handle Requests You Didn't Declare.

A relation key's unsupported fields are also exempt from its output requirement. A stable ID's aren't: readMany, updateMany and updateOne must list every stable-ID field in output, and no output can list an unsupported field, so a collection whose stable ID contains one can't declare those members.

The keyed-write rule also still applies: updateOne and deleteOne need equals on it and and in readMany.filter.

Permission Rules

Saving a policy, on create and on update, checks each rule's condition against the members that rule governs. A member that declares no filter can't carry one: saving the rule fails with "declares no <member>.filter, so it cannot be the subject of a permission rule".

A member the collection doesn't declare adds no constraint. A condition that names a field or operator one of these members doesn't offer fails with 422 "Invalid rule filter", and nothing is saved. The error's message names the member's filter input, such as …UpdateOneFilterInput. Its code is 4003 for an unknown field and 4012 for an unknown operator.

A saved condition joins the filter of the calls it applies to. Engine checks it against the members above when the policy is saved, not when it sends a call. A field read rule's condition also joins the filter of a bulk update or delete that filters by the guarded field, and reaches your connector even when that call's member doesn't declare the condition's fields or operators. Declare them there too, or refuse the call: see Handle Requests You Didn't Declare.

Declare Query Capabilities

A data connector entry declares its query capabilities in engine.capabilities.query in extension.config.json. Each one promises that a kind of write answers with the items it wrote, carrying every field in the request's select:

The capabilities apply to every collection of every data source that uses the entry. Absent, the entry declares none. A read-only connector declares none: Engine plans reads the same way with or without them.

What Engine Sends

Engine sends one create per item, with one object in data. The other writes are planned as follows, for a collection notes with the stable ID id:

• A filter by stable ID uses the shape the called member's filter declares. That's in on a single stable-ID field. Without in, and for a stable ID of several fields, it's equals on each field joined by and for each item, and or across the items.
• Other calls come on top, in both columns. An update field rule adds a readMany before the write, and relation fields in the caller's fields add reads of the related collection.
• Without updateReturning or deleteReturning, updateOne and deleteOne never reach your connector. Engine sends updateMany or deleteMany instead. When the first readMany finds no item for the key, Engine answers the caller 404 without sending a write.
• Engine takes nothing from an answer to select: []. Answer it with the items you wrote, or with no records.
• The extra calls aren't atomic. Another writer can change an item between them, and an external data source that is only eventually consistent can answer the read-back with stale data. Declare a capability whenever your connector can keep its promise.

Without updateReturning, a caller's update of the item n1 reaches your connector as this updateMany:

{
  "query": {
    "op": "updateMany",
    "collection": "notes",
    "select": [],
    "filter": { "in": { "field": "id" }, "values": ["n1"] },
    "data": { "title": "Renamed" }
  }
}

Rules Without a Capability

Each missing capability adds rules to the declaration. Engine checks them with the others when a data source is created or reintrospected, and refuses a declaration that breaks one with 422. These messages come from a collection notes whose connector declares no capability:

The read-back filter shape is in on a single stable-ID field with and, or equals on each stable-ID field with and and or. The last two messages name only what's missing. When updateMany is declared but its data misses a field, the message ends with "or `status` in `updateMany.data`". The first message names the capability in PascalCase; the others use the spelling of capabilities.query.

When the Capabilities Change

Engine reads an entry's capabilities when it builds a workspace, and keeps them until it rebuilds that workspace. A restart loads a new build with its new capabilities directly. With automatic reloading on, Engine loads a new build as soon as you install it. When that build's capabilities differ, every existing data source of the entry refuses requests that reach its connector with 503:

{
  "message": "Query failed",
  "source": {
    "message": "Data source is deactivated",
    "source": {
      "message": "the extension is deactivated",
      "source": {
        "message": "the connector now declares the query capabilities [insertReturning], but this data source was loaded with []. A restart applies the change."
      }
    }
  }
}

A restart of the instance applies the new capabilities, and so does anything else that rebuilds the workspace, such as creating another data source in it. Installing a build with the previous capabilities also restores the data source.

A stored declaration that relies on a capability the connector no longer declares breaks a rule above. After the rebuild, its collection loads with no operations: every request to it fails with 422 code 4010, such as "readMany operation not found for collection `notes`", and Engine logs "Loaded the collection without operations: its stored declaration is unusable. Reintrospect the source to restore them." Fix the declaration, then refresh the data source's schema.

Use Keys and the Stable ID

The stable ID is how Engine addresses one item. It's the collection's primary index or, without one, a unique index. With several unique indexes and no primary index, Engine picks one of them for the collection; declare a primary index to choose.

The keyed members (readOne, updateOne, deleteOne) need more: a primary index on exactly one field, of a type other than unsupported. A unique index isn't enough for them.

A readOne request for the article with ID art_42:

{
  "query": {
    "op": "readOne",
    "collection": "articles",
    "key": { "id": "art_42" },
    "select": ["id", "title"]
  }
}

• The key names the item. It has one entry: the primary-key field.
• The guard narrows it. A filter beside the key is a condition the item must also match. Otherwise change nothing.
• Answer one record. Answer { "record": <item> } with the item the operation read, updated or deleted.
• A miss answers record: null. When the key matches no item or the item fails the guard, answer { "record": null } and change nothing. A miss isn't an error, so don't throw. Callers see 404.
• Never answer records. Engine fails a keyed operation that answers { "records": [...] }, even with one item, and can't undo a write your connector already made.
• Compound external keys need a synthetic field for keyed members. If the external data source identifies an item by several values, expose a single string field that combines them and declare it as the primary index. A composite primary or unique index still works as the stable ID for the other members.

Engine can still read one item through readMany by stable ID, for example to return what an update wrote.

A collection without a primary or unique index has no stable ID. It can still declare readMany. It can declare create, updateMany and deleteMany only when the connector declares the matching query capability. Engine then returns what your connector answers for the write, without reading it back.

Resolve Relations

A relation links two collections. A to-one relation field returns a single related item; a to-many relation field returns several. You declare relations in the schema document's relations. See Relations for the shape.

A relation joins two collections through key fields: authorId on articles and id on authors in the example below.

Engine resolves a relation field by filtering the other collection by its key field. It sends a readMany with in on the key field, joined with and to any other conditions. Where the key field's filter declares equals and or but no in, and for a key of several fields, it uses equals on each field, with or, instead of in.

A to-one include is one batched call. A to-many include with a limit or an offset becomes one call per distinct, non-null parent key, up to 16 of them at a time.

To-many includes carry the default limit of the instance (your Monospace deployment) unless the caller sets one, so that's the usual case. See Relation Reads for every case.

For a relation from articles.authorId to authors.id, each direction needs its own call. Declare both rows to offer both relation fields:

List each relation's key field in the output of every member of its collection: authorId on articles, id on authors. A member whose output lacks the key field doesn't offer the relation.

• Unservable relations aren't offered. Engine accepts a relation direction your declaration can't carry this way. It doesn't offer that direction for reading or for nested _update. See Declare Nested Writes for the other nested writes.
• Engine logs what's missing. When the workspace loads, a warning names the collection, what the relation loses, and what to declare. Keys that no declaration can filter are exempt; see Rules for Engine's Own Calls.
• Relation keys pair by type name. Paired fields must be scalar and share a type name (e.g., int32 doesn't pair with int64). The schema helpers reject mismatches.
• No filters across relations. Callers can't filter through a relation into or out of a data connector collection, so your connector only ever receives filters on the collection's own fields.

Declare Nested Writes

Callers can write related items through a relation field with _create, _connect, _update, _disconnect and _delete. Engine splits a nested write into separate calls to each collection's own members.

The foreign key is the key field on the constrained collection (authorId in the example). Engine writes it itself, through one member:

• On the item being created: that collection's create.data must list it.
• On the item being updated: that collection's updateMany.data must list it. Engine addresses the item by its stable ID, so updateMany.filter also needs the read-back shape on it: in and and (or equals with and and or) for a single-field stable ID, or equals on each field with and and or for several.
• On the target: the target's create.data for _create, and its updateMany.data for _connect and _disconnect.

Engine offers each nested write only when the declarations below carry it, plus the extra conditions after the table for a to-one relation under an update. The member the write rides in must also list the relation's key field in its output.

• A create offers only _create and _connect. _update, _delete and _disconnect exist only in update input.
• A required to-one relation field offers no _delete or _disconnect. Either would leave the foreign key without a target.
• Under an update, a to-one write through a foreign key on the target clears the previous target first. Before _create or _connect attaches a new target, Engine clears the foreign key on the target that pointed at the updated item until then. The target's updateMany.data must list the foreign key, and its updateMany.filter must filter by it as in Resolve Relations. Otherwise neither write is offered.
• One-to-one relations need nullable sides under an update. When both relation fields are to-one, an update offers _create and _connect only when both relation fields are nullable. It offers _disconnect only when the other side is nullable as well.
• A _connect that writes the foreign key on the item being written doesn't check the target. Engine passes the key to your connector as is. Throw ForeignKeyConstraintViolation when the external data source rejects it.
• A _connect that writes the foreign key on the target checks it first. Engine reads the targets through their readMany, and fails unless every one exists.

Name Collections and Fields

Each collection, and each field in its fields, has a name and an optional apiName. Relation fields are named in the relation instead.

• name is what your connector sees. Every request names collections and fields by name. Items you return are keyed by name and carry every field in the request's select.
• apiName is what callers use. Without one, Engine derives it from name using the workspace's naming convention, camelCase by default. See API Names.
• Relation field names are used as declared. Engine doesn't apply the naming convention to them.
• Names collide across data sources. Collection API names are unique per workspace. When a derived name is taken, for example by another data source's articles, Engine prefixes the namespace or appends a suffix (articles_2).

Declare name in camelCase so that name and the API name match in a camelCase workspace.

API names are Monospace's own. A schema migration can rename a collection, field or relation field's API name on a data connector data source, and your connector keeps receiving name. See No Schema Changes from Monospace.

See Also

• Data Connector API: handlers, the schema document and the query wire format
• Data Connector Errors: what to throw when a request can't be served
• Data Connector Limitations: what a data connector can't declare
• Filtering: the caller-side filter syntax
• Map an External Data Source: turn an external data source's query grammar into a declaration