How Data Connectors Run

Overview

This page describes how the entries of a data connector extension run. Data connectors are the only extension entry type today, and the sandbox, lifecycle, and limits below apply to them specifically.

Engine runs each data connector extension inside a sandbox. Knowing its boundaries up front saves you from code that builds cleanly and then fails at runtime.

A data source is a connector configured in a workspace. It points at the external data source, the API, service, or database the connector reads and writes.

The model in one paragraph: every data source gets its own isolated Deno worker. The worker loads one self-contained JavaScript file and has web platform APIs but no Node APIs. It reaches only the hosts the data source was granted, and keeps nothing between restarts.

One Worker per Data Source

Single File, No Imports

The worker refuses every module import. The CLI bundles your TypeScript and its dependencies into one file, and fails the build if the output still imports anything.

You can use npm packages that run on web APIs alone, because the build inlines them. You can't use packages that need Node built-ins.

import { defineDataConnector } from '@monospace/extension-kit/data-connector';

import { readFileSync } from 'node:fs';

A build that still imports a module fails with extension.built_not_self_contained. A file that bypasses the CLI and imports a module fails to load with an extension cannot import …. A dynamic import() fails with the same message when it runs.

Available APIs

The worker exposes the web platform APIs of the Deno runtime, and nothing that reaches outside the sandbox.

Filesystem, environment, and subprocess calls throw Deno.errors.NotCapable.

Configuration, Not Environment Variables

A connector gets its settings and credentials from the data source's configuration, never from the environment. The Stripe connector from Build a Stripe Data Connector parses its configuration in setup. Its parseConfig throws InvalidConfiguration when the configuration is invalid:

import { defineDataConnector } from '@monospace/extension-kit/data-connector';
import { StripeClient } from './client';
import { schemaDocument } from './collections';
import { parseConfig } from './config';
import { readMany, readOne } from './read';
import { findResource } from './resources';
import { create, deleteMany, deleteOne, updateMany, updateOne } from './write';

export default defineDataConnector<unknown>({
    setup({ config: rawConfig }) {
        const config = parseConfig(rawConfig);
        const context = { client: new StripeClient(config.apiKey), config };

        return {
            introspect() {
                return schemaDocument(config.enableWrites);
            },

            async query({ query }) {
                // Engine sends only declared collections, and this connector declares no namespaces.
                // For anything else, throw a plain error.
                const resource = query.namespace === undefined ? findResource(query.collection) : undefined;
                if (resource === undefined) throw new Error(`Unknown collection \`${query.collection}\`.`);

                switch (query.op) {
                    case 'readMany': return await readMany(context, resource, query);
                    case 'readOne': return await readOne(context, resource, query);
                    case 'create': return await create(context, resource, query);
                    case 'updateMany': return await updateMany(context, resource, query);
                    case 'updateOne': return await updateOne(context, resource, query);
                    case 'deleteMany': return await deleteMany(context, resource, query);
                    case 'deleteOne': return await deleteOne(context, resource, query);
                    default: {
                        // `query` is `never` here, so an operation added to the kit fails to compile until it's handled.
                        const unhandled: never = query;
                        throw new Error('The Stripe connector received an operation it doesn\'t handle.', { cause: unhandled });
                    }
                }
            },

            async testConnection() {
                await context.client.ping();
                return { ok: true };
            },
        };
    },
});

const apiKey = Deno.env.get('STRIPE_API_KEY');

HTTP Through fetch

Use fetch for every call to the external data source. It sends a Monospace/<version> user agent unless you set User-Agent yourself. Engine sets no deadline on query, introspect or testConnection, so a request that never answers holds the caller's request open until the caller gives up. Give every outgoing request a timeout. It bounds that one request, not the whole operation: a read that pages sends several requests, each with its own deadline.

This excerpt from the full Art Institute of Chicago connector, from Map an External Data Source, sends one request with a 10-second deadline. It leaves out the imports, the types, and the helpers retryAfter and parseSearchResult:

const BASE_URL = 'https://api.artic.edu/api/v1';
/** Engine sets no deadline on `query` or `testConnection`, so every request carries its own. */
const TIMEOUT_MS = 10_000;
// …
/**
 * Sends one search request, and classifies every failure. Only an error's code and message leave the
 * connector, so the message names the API and the status, never the response body.
 */
export async function search(collection: string, body: SearchBody): Promise<SearchResult> {
    let response: Response;
    let text: string;

    try {
        response = await fetch(`${BASE_URL}/${collection}/search`, {
            method: 'POST',
            headers: { 'Content-Type': 'application/json', 'AIC-User-Agent': 'monospace-artic-connector' },
            body: JSON.stringify(body),
            // The deadline covers reading the body too.
            signal: AbortSignal.timeout(TIMEOUT_MS),
        });
        text = await response.text();
    }
    catch (error) {
        const isTimeout = error instanceof Error && error.name === 'TimeoutError';
        const reason = isTimeout ? `did not answer within ${TIMEOUT_MS / 1000} seconds` : 'could not be reached';
        // Keep the rejection as the `cause`: when the data source's permissions refused the request,
        // Engine finds that there and reports a permission error instead of a failed connection.
        throw new ConnectionFailed(`The Art Institute API ${reason}.`, { cause: error });
    }

    if (response.status === 429) {
        throw new RateLimited(`The Art Institute API rate limited the connector.${retryAfter(response)}`);
    }

    // Treat a server error as an outage: a later request may work.
    if (response.status >= 500) {
        throw new ConnectionFailed(`The Art Institute API answered HTTP ${response.status}.`);
    }

    // Any other failed status means the connector sent a request the API refuses.
    if (!response.ok) {
        throw new Error(`The Art Institute API answered HTTP ${response.status}.`);
    }

    return parseSearchResult(text);
}

• AbortSignal.timeout(TIMEOUT_MS) gives the request its deadline. When it passes, fetch rejects with an error named TimeoutError.
• Read the body inside the same try. The signal also aborts reading the body, so the deadline covers the whole response.
• Keep the rejection as the cause. A request to a host outside the data source's grant rejects with an error named NotCapable. Engine looks for it along the thrown error's cause chain, so ConnectionFailed with { cause: error } still reaches the caller as a permissions error, with your message as the innermost one. Without cause, the denial reads as a failed connection. See Handle a Denied Request.

Logging

Write logs with console. Engine discards stdout and stderr, so anything written there never appears. This logs the status of the fetch response above:

console.info('fetched artworks page', response.status);

await Deno.stdout.write(new TextEncoder().encode('fetched page\n'));

Every console call lands in the Engine log at info level, tagged with the data source and the connector id. Engine also exposes monospace.log(message, fields). It logs fields as JSON in a single params attribute instead of formatting them into the message.

No Persistence

The worker has nowhere to store data. Keep API clients, tokens, and caches in variables that setup creates, like the Stripe connector's context above. Expect to rebuild them after a restart.

Writing to disk throws instead:

await Deno.writeTextFile('./cursor.json', JSON.stringify({ offset: 100 }));

Network Access

A worker reaches only the hosts its data source was granted. An empty grant means no network at all.

The connector declares the hosts it needs in two places:

• Fixed hosts go in extension.config.json as net: permissions with a reason (e.g., net:api.stripe.com).
• Hosts that depend on configuration, such as a configurable base URL, come from the connector's optional preflight.

Whoever creates the data source approves the combined set. A fetch to any other host rejects with Deno's NotCapable error, and so does a redirect from a granted host to one outside the grant. Keep it as the cause when you wrap it, as HTTP Through fetch shows. See Network Permissions for the format and the approval flow.

Data Source Lifecycle

Engine starts a data source's worker when it first needs it. Every start walks the same sequence, from load to setup. Connection tests and schema previews start a temporary worker of their own, with the same sequence.

setup runs once per worker, not once per request. Anything it creates is shared by every request that worker serves.

Configuration

A data source's configuration is the JSON object you enter when you create it. In Studio, that's a raw JSON editor. Through the REST API, it's the request's config object, sent next to the approved permissions.

Restarts and Failures

The table describes a data source's own worker. A connection test or schema preview runs in a temporary worker, so a failure there, including one that deactivates it, doesn't affect the data source or the next test.

A thrown error inside query fails that request only. A readOne, updateOne or deleteOne that finds no matching item (a single data object in a collection) answers { record: null } instead of throwing. Unhandled promise rejections are logged and don't stop the worker. A rejection message longer than 4,096 UTF-16 code units is cut there and ends with …. console output isn't cut.

A handler that resolves can still fail the request. Engine checks the answer after the handler returns: an item missing a field the request selects, a value Engine can't convert to its field type, or a readOne, updateOne or deleteOne that answers records instead of record fails the request. For a write, the external data source has already applied the change, and Engine can't roll it back. See Answer a Query.

Install and Reload

Engine loads connector code from the extensions in its install location, at startup and, with MONOSPACE_EXTENSIONS__AUTO_RELOAD on, whenever an installed extension changes. After a reload, each data source whose entry id is still in the new build switches to the new code on the next request that reaches its connector. A data source whose entry was removed or renamed keeps running the old code until its worker restarts.

A reload swaps code only. Each data source keeps the schema it was created with, its configuration, its granted permissions, and the query capabilities its workspace was built with. A reloaded build that changes the entry's capabilities deactivates its existing data sources: requests that reach their connector fail with 503 until the instance restarts or the workspace is rebuilt. See When the Capabilities Change. See Update Data Sources After an Extension Update for what to do after each kind of change, and Install and Manage Extensions for the install location and reload settings.

See Also

• Data Connector Runtime Limits: timeouts and other hard numbers
• Network Permissions: declare and approve the hosts a connector reaches
• Data Connector API: preflight, setup, and the handlers
• Data Connector Errors: the error classes and when to throw them
• Data Connector Limitations: what a connector can't do today