Guides

Horizontal Scaling

Run several Monospace replicas against one shared system database.

Overview

Monospace runs in clustered mode by default. Clustered mode runs several replicas against one system database. Each replica can serve API requests; distribute traffic across ready replicas with a load balancer.

The replica that handles a change to your data model applies it immediately. The other replicas follow after a short delay.

What Clustered Mode Requires

Component	Requirement
System database	PostgreSQL. Replicas coordinate through it.
Event bus	RabbitMQ, reachable from every replica.
File storage	S3-compatible storage, or a local path marked as shared. See File Storage.

A replica that fails a requirement does not start. Its logs name the unmet requirement. For example, a replica with no RabbitMQ configuration logs this and exits:

logs

clustered mode requires RabbitMQ, but no `rabbitmq` config is set

Correct the configuration, then start the replica again.

Every replica needs the same configuration.

File Storage

Clustered mode accepts two file storage configurations: an S3-compatible bucket, or a local path marked as shared. It rejects any other local path, because a file uploaded to one replica would be missing on the others.

Point every replica at one S3-compatible bucket:

MONOSPACE_SYSTEM_FILE_STORAGE__PROVIDER=s3
MONOSPACE_SYSTEM_FILE_STORAGE__BUCKET=monospace-assets
MONOSPACE_SYSTEM_FILE_STORAGE__REGION=eu-central-1
MONOSPACE_SYSTEM_FILE_STORAGE__ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
MONOSPACE_SYSTEM_FILE_STORAGE__SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY

system_file_storage:
  provider: s3
  bucket: monospace-assets
  region: eu-central-1
  access_key_id: YOUR_ACCESS_KEY_ID
  secret_access_key: YOUR_SECRET_ACCESS_KEY

A network file system works too. Mount the same path on every replica, then mark it as shared:

MONOSPACE_SYSTEM_FILE_STORAGE__PROVIDER=local
MONOSPACE_SYSTEM_FILE_STORAGE__ROOT_PATH=/mnt/monospace-assets
MONOSPACE_SYSTEM_FILE_STORAGE__SHARED=true

system_file_storage:
  provider: local
  root_path: /mnt/monospace-assets
  shared: true

Mark a path as shared only when every replica mounts it. The flag turns off the startup check and nothing else; files on a private path remain unavailable to other replicas.

Configure Clustered Mode

Clustered mode is on by default. Every cluster setting is optional and falls back to its default:

Setting	Default	Effect
`cluster.enabled`	`true`	Turns clustered mode and its startup validation on or off.
`cluster.replica_id`	unset	Names the replica in logs and health output. Falls back to `HOSTNAME`, which container platforms typically set for you.
`cluster.cache_reconcile_interval`	`5m`	How often a replica checks the system database for changes it has not applied yet. Lower it to shorten the catch-up delay.
`cluster.cache_reload_timeout`	`15s`	How long a replica may fail to pick up changes before it reports itself unready.
`cluster.startup_lock_timeout`	`5m`	During startup, how long a replica waits for another replica to finish migrating. Raise it for long migrations.
`cluster.metadata_lock_timeout`	`15s`	How long a schema or settings write waits for a concurrent write to finish. Raise it when writes time out under load.
`shutdown_drain_delay`	`0s`	How long the replica keeps accepting new requests after it reports itself unready. Raise it when requests still arrive after that point.
`shutdown_grace_period`	`30s`	Total time from the shutdown signal to a forced exit. What remains after the drain delay is the time in-flight requests get to finish.

shutdown_drain_delay and shutdown_grace_period are top-level settings, not cluster settings. They apply to a single-node deployment too.

Configuration lists every cluster setting, and Server lists the shutdown settings.

Health Checks

Every replica exposes two routes. Neither route requires authentication, and both also work without clustered mode.

Route	Method	Purpose
`/api/system/live`	`GET`	Returns 200 while the process runs. Use it for a liveness probe.
`/api/system/ready`	`GET`	Returns 200 when the replica accepts traffic, and 503 when it does not. Use it for a readiness probe.

A replica reports itself unready until startup finishes, while it drains for shutdown, or when a required check fails. Redis becoming unavailable after startup reports degraded readiness while the replica continues accepting traffic. The response body lists each check and its status.

For more information about the system API, refer to System Endpoints.

Operations

Graceful Shutdown

On SIGTERM or SIGINT, a replica reports itself unready. It keeps accepting new requests for shutdown_drain_delay, then finishes its in-flight requests within shutdown_grace_period of the signal.

The drain delay must be shorter than the grace period. A replica with an equal or longer delay does not start.

Set the termination grace period of your platform higher than shutdown_grace_period.

Upgrades and Rollbacks

Back up the system database before every upgrade. Replace one replica at a time, and wait for its readiness before the next one. A rolling upgrade is safe when the new migrations do not break the replicas still on the old version.

A release with a destructive migration needs a full cluster restart. Stop every replica, then start the new version. In a rolling upgrade, the old replicas would keep querying a shape that no longer exists.

A rollback needs a database restore. A replica refuses to start when the system database records a migration newer than the latest one the replica knows. Restore the backup you took before the upgrade, then start the previous release.

Capacity Planning

Set max_connections on each database server high enough for the whole cluster. Each replica opens its own connection pool for the system database and for each data source. Leave headroom as you add replicas.