Data Connector Limitations

Overview

Each entry below names a limitation, what callers and connectors observe when an external data source (the API, service, or database a data connector extension reads and writes) runs into it, and the workaround. Run the Fit Check first to find the entries that apply to your external data source.

A data source is the connector configured in a workspace. It exposes the external data source as collections. A collection holds items, each a single data object with named values called fields.

Required Filters

No Required, Alternative, or Bounded Filters

A connector can't declare that a filter is required. It can't require one of several filters (e.g., "channel or id"), or bound a range (e.g., "at most 7 days"). Every declared operator is optional and can appear anywhere in the filter.

• What happens: callers can send an unfiltered or unbounded read, and Studio offers it. The connector has to refuse it at runtime.
• Workaround: refuse before making any request to the external data source. Throw QueryRejected with a message that names the missing filter, and document the requirement for your users. The caller gets 422 "The data source rejected the query", with your message nested inside.

No Filters Across Relations

Callers can't filter a connector collection by fields of a related collection (e.g., issues by their team's name), even when the external data source could. The same holds the other way: a collection in another data source can't be filtered through a relation into a connector collection.

• What happens: relation filters aren't offered on data connector collections. That includes permission rules: a rule can't filter through a relation that touches a connector collection.
• Workaround: expose the foreign key as a filterable field, and have callers filter by it directly.

Paging

Offset Paging Only

Engine and Studio page with limit and offset. The contract has no cursor paging.

• What happens: on an external data source that pages only by cursor, your connector has to walk its pages and discard items until it reaches offset. The cost grows with the page number.
• Workaround: cap how deep the connector walks, and refuse offsets beyond the cap with QueryRejected.

No Declarable Page Size or Depth Limit

A connector can't declare a maximum limit or a maximum offset. A caller's read without a limit arrives with the default of the instance (your Monospace deployment), 100 unless MONOSPACE_QUERY_LIMIT_DEFAULT changes it. When readMany declares limit, a caller can ask for a larger limit, or for no limit. When the instance sets MONOSPACE_QUERY_LIMIT_MAX, a larger limit is rejected and no limit means that maximum. The maximum doesn't cap the default.

• What happens: the connector receives page sizes and offsets the external data source can't serve in one request. Returning fewer items than limit tells Engine there are no more items.
• Workaround: page through the external data source until the requested limit is filled or it runs out. Refuse offsets past its maximum depth at runtime.

Query Shapes

Filter, Sort, Page, Select, Relate, Count

The query API offers filters, sorting, limit and offset, field selection, relations, and a total count, each where the collection declares it. It has no aggregation, grouping, full-text search, or vector search yet, for any data source. Keep up with what's coming on our Roadmap.

• What happens: an external data source's search or aggregate endpoints can't be exposed as query features.
• Workaround: map a search grammar onto the declared operators where it matches exactly (e.g., icontains for case-insensitive substring matching). Leave the rest undeclared.

Operators Depend on the Field Type

Engine executes only certain operators for each field type. For example, boolean, JSON, bytes, and geometry fields support only equals, and text operators exist only on strings.

• What happens: a declaration that uses an operator Engine can't execute on that field type fails data source creation, even when the external data source supports the operator.
• Workaround: declare the intersection of what the external data source supports and what Engine executes. See Operations and Operators for the per-type table.

A Count Without a Total Is Dropped

Declaring meta.totalCount doesn't guarantee callers a total. When the connector answers a count read without totalCount, Engine leaves the total out of the caller's response instead of failing the request.

• What happens: a caller that asked for a total gets none, and no error.
• Workaround: answer every count read with totalCount. When the external data source can't count a request, throw QueryRejected.

Engine Sends Shapes You Didn't Declare

Engine composes parts of the request itself:

• a default limit on a caller's read
• equals for a caller's shorthand { field: value }, and ascending order for a sort without a direction
• equals with null for a caller's _null: true, and not around it for _null: false
• permission rules joined with or, and not around field rules before an update
• one read per distinct parent key for a to-many relation (a relation field that returns several related items), up to 16 at a time, each with the include's limit, or the instance default (100) when the caller sets none
• one read without a limit for a to-one relation (a relation field that returns a single related item), and for a to-many include that resolves to no limit and no offset: limit=-1 or limit=0 without MONOSPACE_QUERY_LIMIT_MAX, or no limit on an instance that unsets its default (see Relation Reads)
• read-backs of updates, and, when the connector doesn't declare a write's query capability, readMany calls by stable ID around it and updateMany or deleteMany with select: [] in place of the caller's write (see What Engine Sends)
• count reads, which carry count: true and limit: 0. They arrive only when readMany declares meta.totalCount

The defaults and the permission wrappers, or and not, arrive whatever the collection declares. Engine doesn't check the comparisons inside the filters it sends against the called member's declaration either: a field read rule's condition can reach a bulk update or delete whose member doesn't declare its field. The declaration gates the rest:

• the shorthand arrives only where the field declares equals;
• _null arrives only on a nullable field that declares equals, in a filter that also declares not;
• relation reads arrive only for a direction the target collection's filter offers;
• read-backs arrive only on collections whose readMany can carry them (Engine checks that when it creates the data source);
• count reads arrive only when readMany declares meta.totalCount.

A relation read or read-back can also arrive undeclared, on a key that can't carry a declared filter. That's a single field whose type has no in (e.g., boolean), or a key with an unsupported field. Only the lookup's own operator arrives there undeclared: in on a single field, or equals on each of several. See Rules for Engine's Own Calls.

• What happens: the connector receives filters and arguments beyond what a caller typed.
• Workaround: handle these shapes, or refuse the ones you can't serve with QueryRejected. They're valid requests, so don't refuse them with a plain Error, which marks a request outside the declaration. Refuse any other undeclared comparison with a plain Error, and never drop it. See Handle Requests You Didn't Declare.

Keys

Single-Item Operations Need a Single-Field Primary Key

readOne, updateOne, deleteOne, and item URLs need a primary key made of one field, of a type Engine supports (not unsupported). A collection keyed by several fields in the external data source (e.g., channel and timestamp) has none of them.

• What happens: a declaration of a *One operation on that collection makes Engine reject any data source creation or reintrospection that includes the collection. Without those operations, the collection has no item routes. A unique index isn't enough.
• Workaround: declare a synthetic single-field primary key that encodes the parts reversibly (e.g., "<channel>:<timestamp>", escaping : inside a part). Keep the parts as separate filterable fields, and decode the key back into the lookup it stands for.

64-Bit Integers Arrive as Strings

Engine sends int64 and uint64 values as decimal strings, in item data, filter values and keys, so they stay exact beyond JavaScript's safe-integer range. It accepts a string or a number back, and returns them to callers as strings (e.g., "balance": "0"). Narrower integers stay numbers, and so do the elements of a list field: only single values become strings.

• What happens: a connector that checks typeof key === 'number' on an int64 key treats every key as a miss, and a comparison with a number literal never matches.
• Workaround: read int64 values as strings, and convert them with BigInt where you compute with them. Declare int32 when the external data source's values fit 32 bits and callers should get numbers. See Values.

Relations

Each Relation Direction Needs a Filter on Its Target

A relation has two sides (e.g., an article's author and an author's articles). Engine resolves each side by filtering the target collection by the relation's key fields. Each side is offered only when its target declares that filter, except for keys that can't carry one (see Rules for Engine's Own Calls).

• What happens: if the external data source can't filter by the referencing field (e.g., list articles by author), that side of the relation can't be read. The other side still works when its own target can filter.
• Workaround: declare the filter and output each side needs. See Resolve Relations. If neither side can be served, expose the reference as a plain field.

Unservable Relations Narrow Without an Error

A relation the declaration can't fully carry isn't rejected. Data source creation succeeds, and Engine offers less of it. Depending on what's missing, the relation can't be read or filtered, or is missing from one operation's results. It can also lose nested writes, including _create and _connect on a one-to-one relation under an update. See Declare Nested Writes.

• What happens: only a warning in the Engine log reports it, containing is not offered, offers no nested, or can be neither read.
• Workaround: after creating a data source, check the workspace's OpenAPI document for every relation you declared, and search the Engine log for those phrases.

Foreign Keys Are Set Through the Relation on Create

Create input doesn't accept a relation's foreign-key field. Callers set it on the relation instead, with _connect and the target's key or _create for a new target, where the target collection offers them.

• What happens: when the foreign key sits on the created item, Engine passes the _connect key to the connector without checking that the target exists. The external data source decides whether to accept it. When the key sits on the target instead, _connect fails unless every target item exists. A non-nullable relation whose key sits on the created item is required on create. The exception is a foreign key whose fields all have an engineManaged onCreate default that yields a value, on a relation that offers _connect: Engine then connects that value itself. A connectorManaged default doesn't lift the requirement.
• Workaround: when the external data source refuses a write that names a missing item, throw ForeignKeyConstraintViolation, so the caller gets 422. For references whose target often doesn't exist in the external data source (e.g., an external phone number), expose the reference as a plain field rather than a relation.

Relation Key Types Must Match

The schema helpers require scalar key fields with the same type name on both ends of a relation. For example, uuid doesn't relate to string, int32 doesn't relate to int64, and date doesn't relate to dateTime.

• What happens: the schema helpers reject the schema with Incompatible relation keys.
• Workaround: declare both key fields with the same type, converting values in the connector.

Writes

No Transactions

Engine runs no transaction across connector calls, and has nothing to roll back in the external data source. Whether a single request to it is atomic depends on the external data source.

• What happens: when a multi-item update, a nested write, or another multi-step write fails partway, the steps before the failure stay applied. So do your connector's writes when Engine rejects its answer afterwards (e.g., an item missing a selected field).
• Workaround: design writes to be safe to retry, and don't rely on rollbacks.

Bulk Writes Are Only as Atomic as the External Data Source

When the external data source writes one item at a time, a filter-based updateMany or deleteMany has to find the matching items first and then write each one.

• What happens: a concurrent writer can change an item between the read and the write. A write that fails partway leaves the items before it written, and the error has only its message to say how many.
• Workaround: use the external data source's conditional or idempotent write features where it has them. Start no new write after the first failure, and say in the message how far the write got. The Stripe example keeps the error's class and prefixes its message: "Deleted 3 of 4 products before this error: This product cannot be deleted because it has one or more user-created prices."

Writes Without Query Capabilities Take Several Calls

A connector that doesn't declare a write's query capability makes Engine add calls of its own. A create answers only the stable IDs, and Engine reads the created items back. An update or delete goes through updateMany or deleteMany by stable ID, between readMany calls. See What Engine Sends for each sequence.

• What happens: every such write costs extra connector calls, and each call your connector serves can cost requests to the external data source. Another writer can change an item between the calls, and an external data source that is only eventually consistent can answer the read-back with stale data. Each collection with such a write needs a stable ID and declarations that carry the extra calls. Without updateReturning, no update can change a stable-ID field.
• Workaround: declare the capabilities your connector can keep, so Engine uses the write's answer instead of reading the items itself. Permission rules and relation fields can still add reads. See Declare Query Capabilities.

No Field-Level Value Rules

A field can't declare the values it accepts (e.g., an enum, or "may only be cleared").

• What happens: the connector has to refuse invalid values at runtime.
• Workaround: validate values in query before calling the external data source, and throw QueryRejected with a message that names the field and the accepted values. The caller gets 422.

Answers Are Only Partly Checked

Engine converts each value your connector returns by its field's type, but doesn't enforce every declared constraint.

• What happens: a null passes for a non-nullable field, and a value that exceeds a narrower integer width (e.g., int8), a fixed string length, or a decimal's declared precision and scale passes too. A geometry field is the reverse: Engine can't convert geometry values yet, so any non-null value in it fails the request.
• Workaround: return only values that fit the declaration. See Answer a Query for what Engine does check.

Errors

No Automatic Retries

Engine doesn't retry a failed connector call, and RateLimited carries only a message. Engine answers it without a Retry-After header.

• What happens: a throttled request to the external data source fails the caller's request.
• Workaround: retry inside the connector where the request is safe to repeat, with a small, bounded number of attempts. Put the wait into the message, as the example connectors do: "Stripe rate limited the connector. Retry after 2 seconds."

Engine Doesn't Check Credentials

Engine runs the connector's testConnection on a connection test, and before it creates a data source or previews its schema. Engine itself never checks a credential. A connector whose setup, introspect, and testConnection make no authenticated request to the external data source accepts any credential.

• What happens: when none of setup, introspect and testConnection makes an authenticated request, a data source with a wrong credential is created. It fails once the connector first makes a request that needs the credential. A connector without testConnection still creates, and a connection test answers 422 code 7002, "The connector does not offer a connection test".
• Workaround: implement testConnection with an authenticated read from the external data source.

Configuration and Network

Configuration and Permissions Are Fixed at Creation

No API changes an existing data source's configuration or granted permissions. Reintrospecting keeps the original configuration and grant. We're working on letting you change a data source's configuration after it's created.

• What happens: rotating a credential or changing a host requires a new data source, with a new id.
• Workaround: recreate the data source.

Grants Must Match the Required Permissions Exactly

A data source starts only when its granted permissions exactly match the permissions the manifest and preflight require together. That means the same hosts, ports, and wildcards. Reasons aren't compared.

• What happens: a new connector version that changes the combined set of required permissions stops each existing data source that uses it, at the next request that reaches its connector. From then on the data source refuses every request that reaches its connector. Cached results are still served.
• Workaround: recreate those data sources and approve the new permissions, or reinstall a connector version whose required permissions match the stored grant.

Hosts Must Be Known Up Front

The sandbox allows only the hosts in the manifest plus those preflight derives from configuration, each as a host, host:port, or *.host.

• What happens: a request to any other host fails, including redirects to hosts you didn't predict (e.g., a regional host or a download CDN).
• Workaround: name those hosts, or a wildcard that covers them, in the manifest. See Network Permissions.

Configuration Isn't Validated by Engine

Engine passes the configuration JSON to the connector unchecked. Studio shows it as a raw JSON editor.

• What happens: typos and missing keys reach setup.
• Workaround: parse and validate the configuration in setup, and throw InvalidConfiguration with a message that names the problem.

Lifecycle

Data Sources Keep the Schema From Creation

A data source keeps the schema it was created or last reintrospected with. Installing a connector version with a changed schema doesn't update it.

• What happens: Engine keeps accepting requests based on the old schema. Requests the new code can't serve fail, and renamed collections keep their old names.
• Workaround: refresh the schema or recreate each data source after changing its declared schema.

No Schema Changes from Monospace

The connector declares the schema, so Monospace can't add, change, or remove collections, fields or namespaces on a data connector data source, or relations whose pointing fields are on one of its collections. A relation whose pointing fields live in another data source falls under that data source's rules. API names are Monospace's own, so those stay editable.

• What happens: a schema migration that changes the data source's schema fails. Operations that change only an API name succeed: renameCollection and renamePrimitiveField with dbName set to null, and renameRelationField. The connector keeps receiving its own names.
• Workaround: change the schema in the connector, then reintrospect the data source.

Engine Upgrades Can Disable Connector Collections

When a newer Engine can't use a stored declaration, it loads the collection with no operations and logs an error that says to reintrospect the source.

• What happens: the affected collections can't be queried until you reintrospect each data source.
• Workaround: after an Engine upgrade, check the log and reintrospect the data sources it names. If reintrospection is refused, update the connector so its declaration passes the newer rules first.

Collection Names Are Shared per Workspace

Collection names from all data sources in a workspace share one namespace.

• What happens: when a collection's derived API name is already taken, Engine picks a different one, with a prefix or a numeric suffix (e.g., customers_2). An explicit apiName is kept as declared.
• Workaround: give collections distinctive names in the connector.

Naming Style Can Be Mixed

Engine derives collection and field API names from name using the workspace's naming convention. The default is camelCase (e.g., unit_amount becomes unitAmount). Relation names are used exactly as declared.

• What happens: declarations in a different style produce an API that mixes both styles.
• Workaround: declare names in the workspace's convention, or set an explicit apiName. See Name Collections and Fields.

Runtime

Connectors run in a sandboxed Deno worker with web APIs, no Node APIs, no filesystem, and no persistence. Module loading and setup run under a timeout.

Need a Node.js-compatible API? Let us know.

See How Data Connectors Run for the sandbox and Data Connector Runtime Limits for the numbers.

See Also

• Data Connectors: live queries and the Fit Check
• Operations and Operators: the declaration rules Engine enforces
• Data Connector Errors: the error classes and when to throw them
• Map an External Data Source: patterns for paging, filters, and keys
• How Data Connectors Run: the sandbox and data source lifecycle