This is the full developer documentation for Karakoram
# Overview
> Karakoram is a multi-platform, self-hosted control plane for database and stateful-service orchestration inside your infrastructure, without Kubernetes or a…
Karakoram is a multi-platform, self-hosted control plane for database and stateful-service orchestration inside your infrastructure, without [Kubernetes](https://kubernetes.io/) or a hosted coordinator. A [Go](https://go.dev/) control plane provides a REST API, `karactl` CLI and web UI; providers supply infrastructure, while you operate the installation, credentials, data and recovery procedures. [Zerops](https://zerops.io/) is the reference provider. [Dokploy](https://dokploy.com/) and [Nomad](https://developer.hashicorp.com/nomad) are also available; see [Providers](/docs/providers/) for their status.
## Engines
[Section titled “Engines”](#engines)
Versions below are catalog defaults; each engine section lists supported versions and upgrade paths.
Release status and provider-specific availability are centralized in [engine availability](/docs/providers/#engine-availability).
| Engine | Distribution/version | Topologies | Backups | Section |
| ----------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ------------------------------------- |
| MongoDB | Percona Server for MongoDB 8.0.32-14; Percona Backup for MongoDB (PBM) 2.16.0-1 | Replica sets; sharded clusters | Logical/physical; point-in-time recovery (PITR) | [MongoDB](/docs/mongodb/) |
| RabbitMQ | RabbitMQ 4.3.6 | 1/3/5/7 brokers | Definitions only; no messages or PITR | [RabbitMQ](/docs/rabbitmq/) |
| Elasticsearch | Elasticsearch 9.5.5; optional same-version Kibana, Logstash and APM Server | Combined nodes; dedicated masters | Native S3 snapshots | [Elasticsearch](/docs/elasticsearch/) |
| Weaviate | Official Weaviate 1.40.1; mixed-license binary, community features only | Single; 3–9 data members with three fixed Raft voters | Native S3 full; no PITR | [Weaviate](/docs/weaviate/) |
| PostgreSQL | Percona PostgreSQL 17.11.1 default; majors 16/17/18; extension profiles | Single; Patroni/etcd HA (3/5); Citus distributed; ProxySQL client service | pgBackRest full/diff/incr and PITR | [PostgreSQL](/docs/postgres/) |
| Supabase | PostgreSQL 17 plus release-locked native services | Same database topologies; modular services | PostgreSQL data only; separate S3 object protection required | [Supabase](/docs/supabase/) |
| Keycloak | Official Keycloak 26.8.1; OpenJDK 21 | One native runtime with a PostgreSQL binding | Through the source PostgreSQL cluster | [Keycloak](/docs/keycloak/) |
| OpenBao | OpenBao 2.7.1 | 1/3/5 members; integrated Raft or bound PostgreSQL storage | Raft snapshots to S3; PostgreSQL storage uses source recovery; no PITR for Raft | [OpenBao](/docs/openbao/) |
| Temporal | Temporal 1.32.1 (also 1.31.4); ui-server 2.56.0 | Combined server; split stateless roles on bound PostgreSQL; Dokploy/Nomad only, not Zerops | Source PostgreSQL backup plus cluster-state manifest; requires binding service | [Temporal](/docs/temporal/) |
| MySQL | Percona Server for MySQL 8.4.11-11 default; majors 8.4/8.0 | Single; Group Replication single-primary (3/5); ProxySQL | XtraBackup full/incremental and GTID-based PITR | [MySQL](/docs/mysql/) |
| Percona XtraDB Cluster | PXC 8.4.11-11.1 default; majors 8.4/8.0; separate `pxc` engine driver | Galera cluster (3/5); one ProxySQL-routed writer | Separate PXC backup provenance; full/incremental and PITR | [PXC](/docs/mysql/) |
| Neo4j Community | Neo4j 2026.09.0 / 2026.10.0 (default); Temurin Java 21 | Single member only | Offline system + graph dumps; full outage; no PITR | [Neo4j](/docs/neo4j/) |
## Providers
[Section titled “Providers”](#providers)
Karakoram is multi-platform; a namespace selects the infrastructure provider that provisions its members.
| Provider | Role / runtime | Infrastructure and guide |
| -------- | ------------------ | ------------------------------------------------------------------------------------------------------- |
| Zerops | Reference provider | Managed Ubuntu runtimes, Local Storage and public routing in existing projects; [guide](/docs/zerops/). |
| Dokploy | Docker Swarm | Applications and node-local volumes on your Dokploy-managed Docker Swarm; [guide](/docs/dokploy/). |
| Nomad | Native jobs | Native jobs and dynamic host volumes on your operator-managed Nomad cluster; [guide](/docs/nomad/). |
See [Providers](/docs/providers/) for capability differences and [Operating costs](/docs/operating-costs/) for infrastructure costs.
## Across engines
[Section titled “Across engines”](#across-engines)
* Catalog-driven creation, resource configuration, disk growth and capability-gated [cluster operations](/docs/clusters/).
* [S3](https://aws.amazon.com/s3/)-compatible [storage, schedules and recovery](/docs/backups/).
* Multi-project [namespaces and authenticated relays](/docs/namespaces/).
* [Monitoring](/docs/monitoring/), logs and actor-attributed [audit activity](/docs/operations/#reviewing-audit-activity).
* Local accounts, [OIDC](https://openid.net/developers/how-connect-works/), namespace-aware RBAC and owner-scoped API tokens: [access control](/docs/access-control/).
* [Safe fleet operations](/docs/operations/), [control-plane backup/recovery](/docs/control-plane-backup/), [notifications](/docs/notifications/) and [backup drills](/docs/backup-drills/), verified offline.
**Warning:** Review [critical limits](/docs/limitations/) before production use: replica count does not guarantee independent hosts, Local Storage is single-host, and the control plane has one replica.
## Quick start
[Section titled “Quick start”](#quick-start)
Choose a provider in [Providers](/docs/providers/). For Zerops, use the dashboard-only [Quick start](/docs/getting-started/quickstart/), or the [installation reference](/docs/getting-started/install/) for local deployments, custom names and CLI setup. Dokploy and Nomad installation paths are linked there.
## Build from source
[Section titled “Build from source”](#build-from-source)
Requires Go **1.26+**, [Node](https://nodejs.org/) **22.6+**, [npm](https://docs.npmjs.com/) and [Make](https://cgit.git.savannah.gnu.org/cgit/make.git/):
```sh
make build-all # web UI and both Go executables
make test # Go unit/handler tests
# Integration-tagged tests (some require Docker; PostgreSQL/Supabase use native LXC):
go test -tags integration -count=1 -timeout 30m ./...
```
A Go-only `make build` without UI assets serves a “UI not built” page. See [UI development](https://github.com/KarakoramDev/Karakoram/blob/main/ui/README.md), [Contributing](/docs/contributing/) and the [end-to-end suite](https://github.com/KarakoramDev/Karakoram/blob/main/test/e2e/README.md); deployment tests create billable Zerops resources.
Coding agents: read [AGENTS.md](https://github.com/KarakoramDev/Karakoram/blob/main/AGENTS.md) for repository boundaries and contribution commands.
## Documentation index
[Section titled “Documentation index”](#documentation-index)
Keep `docs/` beside the repository’s `api/`, `ui/` and root README so relative source/reference links resolve.
* **Getting started:** [Overview](/docs/) · [Quick start](/docs/getting-started/quickstart/) · [Install](/docs/getting-started/install/) · [Providers](/docs/providers/) · [Operating costs](/docs/operating-costs/) · [Upgrade](/docs/getting-started/upgrade/)
* **Control plane:** [Architecture](/docs/architecture/) · [Clusters and operations](/docs/clusters/) · [Backups and storage](/docs/backups/) · [Namespaces and relays](/docs/namespaces/) · [Zerops provider](/docs/zerops/) · [Dokploy provider](/docs/dokploy/) · [Nomad provider](/docs/nomad/) · [Monitoring and events](/docs/monitoring/) · [Access control](/docs/access-control/) · [Security](/docs/security/) · [Operations and troubleshooting](/docs/operations/) · [Limitations](/docs/limitations/)
* **Interfaces:** [Web UI](/docs/web-ui/) · [CLI](/docs/cli/) · [REST API and SDK](/docs/api/) · [MCP quick start](/docs/mcp-quickstart/) · [MCP server](/docs/mcp/) · [Cluster data MCP](/docs/data-mcp/)
* **Operator agents:** [karakoram-operator skill](https://github.com/KarakoramDev/Karakoram/blob/main/skills/karakoram-operator/SKILL.md) · [Skill installation](https://github.com/KarakoramDev/Karakoram/blob/main/skills/README.md)
* **Control-plane recovery:** [State backups and provider rebuild runbooks](/docs/control-plane-backup/)
* **Backup verification:** [Scheduled and manual restore drills](/docs/backup-drills/)
* **Notifications:** [Webhook, Slack and email channels, signatures and delivery guarantees](/docs/notifications/)
* **MongoDB:** [Overview](/docs/mongodb/) · [Replica sets](/docs/mongodb/replica-sets/) · [Sharding](/docs/mongodb/sharding/) · [Users, roles and TLS](/docs/mongodb/access-and-tls/) · [Backups and PITR](/docs/mongodb/backups/) · [Operations](/docs/mongodb/operations/)
* **RabbitMQ:** [Overview](/docs/rabbitmq/) · [Access and TLS](/docs/rabbitmq/access-and-tls/) · [Definitions backups](/docs/rabbitmq/backups/) · [Operations](/docs/rabbitmq/operations/)
* **Elasticsearch:** [Overview](/docs/elasticsearch/) · [Kibana, Logstash and APM](/docs/elasticsearch/stack/) · [Access and security](/docs/elasticsearch/access-and-security/) · [Snapshots](/docs/elasticsearch/snapshots/) · [Operations](/docs/elasticsearch/operations/)
* **Weaviate:** [Overview](/docs/weaviate/) · [Access and TLS](/docs/weaviate/access-and-tls/) · [Backups](/docs/weaviate/backups/) · [Operations](/docs/weaviate/operations/)
* **PostgreSQL:** [Overview](/docs/postgres/) · [Extension profiles: PostGIS, pgvector, TimescaleDB](/docs/postgres/profiles/) · [Service bindings](/docs/postgres/bindings/) · [Distributed Citus](/docs/postgres/distributed/) · [Access and TLS](/docs/postgres/access-and-tls/) · [Backups and PITR](/docs/postgres/backups/) · [Operations](/docs/postgres/operations/)
* **Supabase:** [Overview](/docs/supabase/) · [Components and availability](/docs/supabase/components/) · [Operations](/docs/supabase/operations/)
* **Keycloak:** [PostgreSQL binding, routing and administrator recovery](/docs/keycloak/)
* **OpenBao:** [Overview](/docs/openbao/) · [API access and TLS](/docs/openbao/access-and-tls/) · [Raft backups and recovery](/docs/openbao/backups/) · [Operations and unseal](/docs/openbao/operations/)
* **Temporal:** [Overview and prerequisites](/docs/temporal/) · [Access and TLS](/docs/temporal/access-and-tls/) · [Operations and recovery](/docs/temporal/operations/)
* **MySQL and PXC:** [Overview](/docs/mysql/) · [Configuration](/docs/mysql/configuration/) · [Access and TLS](/docs/mysql/access-and-tls/) · [Backups and PITR](/docs/mysql/backups/) · [Operations and recovery](/docs/mysql/operations/) · [Native verification](/docs/mysql/verification/)
* **Neo4j Community:** [Overview](/docs/neo4j/) · [Access and TLS](/docs/neo4j/access-and-tls/) · [Offline backups](/docs/neo4j/backups/) · [Operations](/docs/neo4j/operations/)
* **Reference:** [Architecture decisions](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/README.md) · [Native artifacts](/docs/native-artifacts/) · [Verification](/docs/verification/) · [Contributing](/docs/contributing/) · [Code of Conduct](/docs/code-of-conduct/)
## Acknowledgements
[Section titled “Acknowledgements”](#acknowledgements)
Thank you to [Zerops](https://zerops.io) for providing free-of-charge infrastructure for developing and testing Karakoram.
Karakoram installs [Percona Server for MongoDB](https://www.percona.com/mongodb/software/percona-server-for-mongodb) and [Percona Backup for MongoDB](https://docs.percona.com/percona-backup-mongodb/latest/) from [Percona’s official repositories](https://repo.percona.com/). It independently implements orchestration; no [OpenEverest](https://openeverest.io/) code or UI is copied into this repository.
## License and trademarks
[Section titled “License and trademarks”](#license-and-trademarks)
Karakoram uses the [Apache License 2.0](https://github.com/KarakoramDev/Karakoram/blob/main/LICENSE); see [NOTICE](https://github.com/KarakoramDev/Karakoram/blob/main/NOTICE). Dependencies retain their licenses and notices. Percona Server for MongoDB and Percona Backup for MongoDB are Percona products and trademarks. OpenEverest, MongoDB and Zerops names and marks belong to their respective owners; Karakoram is independent and not affiliated with or endorsed by them.
# Access control
> Karakoram accounts, roles and tokens authorize control-plane actions, independently of provider infrastructure credentials and engine-native users.
Karakoram accounts, roles and tokens authorize control-plane actions, independently of provider infrastructure credentials and engine-native users.
The server enforces policy; hidden UI controls are not an authorization boundary. Engine identities belong in [MongoDB](/docs/mongodb/access-and-tls/), [RabbitMQ](/docs/rabbitmq/access-and-tls/) and [Elasticsearch](/docs/elasticsearch/access-and-security/). [Neo4j](https://neo4j.com/) native users are described in [Neo4j access](/docs/neo4j/access-and-tls/). Every Community user has full administrative privileges; control-plane RBAC does not add native database roles.
Idempotency replay is scoped to the authenticated identity, operation, resource path and namespace (including namespace query parameters). A retry in another namespace cannot receive a cached secret from the original namespace. Generated unconditional additional RBAC grants are checked on the same resource before execution or idempotency replay. [PostgreSQL](https://www.postgresql.org/) binding rotation therefore requires both `database-users:update` and `clusters:update`. Conditional grants (privileged database roles/tags and cross-cluster [OpenBao](https://openbao.org/) targets) are resolved against the request body or current engine state by their owning handlers. [Weaviate](https://weaviate.io/) user creation and API-key rotation recheck privileged-role access before returning cached keys; rotation resolves the user’s current native roles. Revoking `clusters:update` prevents privileged secret replay even while the principal retains its ordinary database-user permissions.
## Connected OAuth applications
[Section titled “Connected OAuth applications”](#connected-oauth-applications)
When [OAuth for MCP](/docs/mcp/#connect-with-oauth) is enabled, every connection requires interactive consent. Grants intersect the account’s current RBAC with approved scopes and optional namespace limits; they do not inherit browser [OIDC](https://openid.net/developers/how-connect-works/) group bindings. **Settings → Connected apps** lists and revokes your grants. `listOAuthGrants` and `deleteOAuthGrant` provide the same self-service API; `api-tokens:read` / `api-tokens:delete` allow administrators to manage others’ grants. Revocation invalidates every access and refresh token in the grant. OAuth connections cannot create or change API tokens; mint dedicated tokens with `karactl` or the web UI.
## Local accounts and first login
[Section titled “Local accounts and first login”](#local-accounts-and-first-login)
An empty account store creates `admin` from `KARAKORAM_ADMIN_PASSWORD`. Set it through a protected service secret; the [security model](/docs/security/) describes the generated-password logging fallback. The initial account must change its password: sign in at the HTTPS UI, change it, then sign in again before privileged work/token creation. Changing the bootstrap environment variable later does not reset an existing account.
| Rule | Value/behavior |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| Account name | 1–64 characters; starts with a letter; letters/digits/`_ . @ -` only |
| Password | At least 12 Unicode characters; at most 1024 UTF-8 bytes; Argon2id hash |
| Self-service change | Requires current password and a different new password |
| Admin reset | May reuse current value; revokes prior authentication and forces change |
| Login backoff | After five account/source failures: 1 second, doubling to 15 minutes; both correct/incorrect attempts receive 429 during delay |
Respect `Retry-After`/`details.retryAfterSeconds`. Password errors identify `currentPassword: invalid`, `newPassword: reused` or `newPassword: out_of_range`; reuse does not clear forced change. Duplicate accounts return 409 `already_exists`. Enabled/forced-change updates refresh `updatedAt` independently of password changes.
Use **Settings → Accounts** or a private request file:
```yaml
name: analyst
password:
enabled: true
mustChangePassword: true
```
```sh
karactl accounts create -f account.yaml
karactl accounts list
karactl accounts get analyst
karactl accounts set analyst -f account-update.yaml
karactl accounts reset-password analyst -f password-reset.yaml
karactl accounts delete analyst
```
Assign roles through policy, not account fields. Self-service change is **User menu → Change password**, `/change-password`, or `karactl accounts change-password -f FILE` with `currentPassword`/`newPassword`. Forced-change users cannot finish CLI login’s token creation first. See [local administrator recovery](/docs/operations/#recovering-local-admin-access).
## Default roles
[Section titled “Default roles”](#default-roles)
| Role | Grants |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `role:admin` | All resources/actions/objects; bootstrap `user:admin` is bound here |
| `role:operator` | All actions on `clusters`, `backups` (including export), `restores`, `database-users` (including roles), `operations`, `events`, `monitoring`; read `backup-storages`, `providers`, `namespaces` |
| `role:viewer` | Read operational resources, namespace metadata, storages and providers; no backup export, credentials, audit, accounts, tokens, RBAC or settings |
Operator has no namespace administration, credential reveal, audit or account/token/RBAC/settings administration. It can still disrupt workloads, restore data and change engine identities.
`cluster-data` is separate from lifecycle and credential-reveal access: neither viewer nor operator receives it by default. Its `read` and `write` actions apply to `/`. A restricted API token must explicitly include matching `cluster-data:read`/`cluster-data:write` permission, and the owner must retain the corresponding RBAC grants. Writes also require cluster and installation opt-in; see [cluster data MCP](/docs/data-mcp/#four-switches-for-writes). For example, add `p, user:analyst, cluster-data, read, prod/appdb` to your complete policy for narrowly scoped data reads.
Defaults apply to **fresh installations**; upgrades retain stored policy. Operator/viewer namespace visibility requires these additions if absent from the existing complete policy:
```csv
p, role:operator, namespaces, read, **
p, role:viewer, namespaces, read, **
```
Without namespace-read grants the UI shows only `default`, even with other cluster access. Narrower grants are valid. New accounts have no automatic role; permissions accumulate through direct rules/inheritance, subject to deny overrides.
## Policy format
[Section titled “Policy format”](#policy-format)
```text
p, SUBJECT, RESOURCE, ACTION, OBJECT[, allow|deny]
g, SUBJECT, role:ROLE
```
CSV `p` rules default to allow. Subjects are `user:NAME`, `group:OIDC_GROUP`, `role:NAME`; `g` binds users/groups/roles to roles. Comments start with `#`. Resource/action/object use glob matching: `*` does not cross `/`, while `dev/**` matches `dev` and its objects. No allow means deny; **any matching deny overrides allows across all subjects/roles**.
| Resource | Actions / object |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `namespaces` | create on global `*`; read/update/delete on bare namespace; validation/rotation use update |
| `clusters`, `database-users` | CRUD as exposed by route; `/`; database roles share `database-users` |
| `cluster-credentials` | read reveals credentials; `/` |
| `cluster-data` | read/write through the opt-in [data MCP gateway](/docs/data-mcp/); `/`; not included in viewer/operator defaults |
| `openbao-unseal-keys` | reveal; `/`; independent from ordinary credential read |
| `backups` | create/read/delete/export; `/` |
| `restores`, `operations` | Route-defined actions; `/` |
| `backup-storages` | CRUD; `/` |
| `events`, `monitoring` | read; `/` |
| `audit` | read; cluster object, bare namespace or global `*`, according to record scope |
| `accounts`, `api-tokens`, `rbac`, `settings`, `providers` | Route-defined actions; global `*`; token self-service exceptions below |
[OpenAPI](https://github.com/KarakoramDev/Karakoram/blob/main/api/openapi.yaml) defines route mappings. Lists, events/streams and `/metrics` filter/check actual objects; filters never widen access. Namespace metadata does not grant audit access. Cross-namespace source export and target permissions are documented in [backups](/docs/backups/); scoped examples are in [namespaces](/docs/namespaces/).
`events:read` or `audit:read` on `/` permits reading a cluster event, but only **`audit:read`** returns its optional `details` object in lists and streams; viewer/operator event access alone does not grant structured comparisons. Audit readers still receive masked sensitive values, never secrets; see [audit redaction and durability](/docs/security/#audit-attribution-and-durability).
OpenBao initialization-material reveal is administrator-only by default and uses `openbao-unseal-keys:reveal`, not `cluster-credentials:read`. Database-connection list/get requires `clusters:read` on OpenBao; create/delete requires `clusters:update` there **and** `database-users:create` / `database-users:delete` respectively on the target cluster. An OpenBao grant alone cannot authorize target identity changes. See [OpenBao access](/docs/openbao/access-and-tls/).
Normalization at startup/submission changes object `*` to `**`; legacy bare cluster/storage/audit patterns become `default/`. Bare namespace objects remain unchanged. Use `dev/**`, not bare `dev`, for namespace-wide audit; inspect the canonical policy after updates.
```csv
g, user:analyst, role:viewer
g, group:db-operators, role:production-operator
p, role:production-operator, clusters, *, prod/**
p, role:production-operator, clusters, delete, prod/critical, deny
p, role:production-operator, namespaces, read, prod
```
```sh
karactl rbac policy get
karactl rbac policy set -f complete-policy.csv
karactl rbac can --resource clusters --action delete --object prod/critical
karactl permissions
```
**Warning:** `policy set` consumes raw CSV and replaces the entire policy; examples above are additions, not replacements. Keep a protected copy and unrestricted administrator access.
The UI warns on missing admin assignments; confirmation does not bypass server validation. Replacement must leave a verifiable enabled account or currently verified OIDC-group identity with unrestricted administration and no denies. Invalid CSV identifies its line; rejected replacement preserves the previous policy. Keep a local break-glass administrator. A 403 identifies resource/action/object; check denies before retrying.
## OIDC setup and groups
[Section titled “OIDC setup and groups”](#oidc-setup-and-groups)
1. Register a confidential client with exact callback `https://example.com/api/v1/auth/oidc/callback`.
2. Set `KARAKORAM_PUBLIC_URL` to that HTTPS UI origin. Callback is derived; returned `redirectURL` is informative, not an override.
3. Emit valid `preferred_username` and a groups claim containing a JSON string array, such as `["db-operators"]`, in the **ID token**. Access-token-only groups do not suffice.
4. Configure **Settings → SSO**, or submit:
```yaml
oidc:
enabled: true
issuerURL: https://identity.example.com/realms/production
clientId: karakoram
clientSecret:
scopes: [openid, profile, email, groups]
groupsClaim: groups
```
```sh
karactl settings set-oidc -f oidc.yaml
karactl settings get
```
5. Add `g, group:db-operators, role:operator` or narrower bindings to the complete policy. Test SSO with low-privilege and operator identities while retaining local admin recovery.
Issuer URLs require HTTPS except loopback development. Client secrets are sealed/write-only; omitted/empty secrets retain the existing value only for unchanged issuer/client identity. Public settings expose `groupsClaim`/`scopes`, not `usernameClaim`, `allowedGroups` or an auto-creation toggle. Enabling OIDC enables account creation, **not automatic roles**.
Issuer/subject pinning prevents adopting existing local accounts through matching names. Session groups become `group:` subjects; sign in again after identity-provider group changes. Current Karakoram policy changes apply independently. See [protocol protections](/docs/security/).
Test-only issuer instructions live in the [end-to-end suite guide](https://github.com/KarakoramDev/Karakoram/blob/main/test/e2e/README.md), not production SSO setup.
## API tokens and browser sessions
[Section titled “API tokens and browser sessions”](#api-tokens-and-browser-sessions)
Tokens exercise their **owner’s current permissions**, optionally narrowed by immutable restrictions. Use a dedicated least-privilege automation account. Tokens begin `kk_`, are returned once and stored as SHA-256 hashes; disabled/deleted owners and forced password changes cannot retain ordinary token authority.
Password-current authenticated users may create/list/read/update/revoke their own tokens without `api-tokens` grants. Global read/update/delete grants permit the corresponding actions on other owners’ tokens. Creation always binds the caller; lists otherwise show only owned tokens.
`name` is required, trimmed, nonblank, 1–128 characters; duplicates are allowed, so revoke by unique ID. Optional `expiresAt` must be RFC3339 and strictly future (`invalid`/`out_of_range` errors). Creation without expiry lasts until revocation/owner invalidation; update omission preserves expiry, null removes it.
```yaml
name: nightly-automation
expiresAt: "2026-12-31T00:00:00Z"
```
```sh
karactl tokens create -f token.yaml
karactl tokens list
karactl tokens delete '' --yes
```
Securely capture creation’s one-time `secret`; later reads expose metadata only. Inject `KARAKORAM_TOKEN` from a secret manager, not command arguments/history. CLI login saves its own token; logout revokes it unless `--local` only removes local credentials. Bearers bypass CSRF, not RBAC.
`lastUsedAt` is best-effort, asynchronous usage metadata, not an authentication or audit guarantee. At most one usage update runs at a time with a one-second write budget; concurrent updates may be skipped when the writer is busy. Credential validity and owner permissions are still checked from persisted state on each request. Public version discovery reads its settings snapshot without waiting for the identity mutation mutex.
Browsers use rotating cookie sessions, not automation tokens; [cookie/CSRF guarantees](/docs/security/) are separate from token permissions. The UI refreshes expired access automatically; network/5xx errors preserve the session and offer retry. Repeated authentication rejection after refresh signs out instead of looping. See [CLI](/docs/cli/) and [API](/docs/api/).
## Restricted API tokens
[Section titled “Restricted API tokens”](#restricted-api-tokens)
Set `restrictions` when creating a token through the API, CLI flags or the UI’s **Restrictions** section. Omitted restrictions preserve the owner’s full authority; an explicit empty object is restricted and does not permit credential reveal. Restrictions only remove authority: owner RBAC and token restrictions must both allow an action, including after policy changes.
| Field | Effect |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `readOnly` | Only `read` actions are allowed. |
| `namespaces` | Allowlisted namespaces for namespace, cluster and storage objects. |
| `clusters` | Exact allowlist of `namespace/cluster` for cluster-scoped resources only. Namespace metadata and other namespaced objects, such as backup storages, are visible in the listed clusters’ namespaces regardless of their own names. Combined namespace and cluster namespace constraints intersect. |
| `permissions` | Allowlisted `resource:action` pairs; action may be `*`. Empty means all owner permissions. Names and actions are validated against the generated RBAC vocabulary. |
| `allowCredentialReveal` | Defaults to false; blocks `cluster-credentials` and operations marked as returning secrets, including token creation. |
Namespace/cluster-scoped tokens cannot access global objects. Collection routes remain available when their action is allowed, but filter each returned object. The same restrictions apply to events, SSE, metrics, operations, permission discovery and idempotent request replay. A read-only token may still be denied a read that returns credentials.
Restrictions cannot be changed with token update; mint a replacement instead. A restricted caller can mint only a subset of its own restrictions, never an unrestricted token. Token creation also requires credential-reveal permission and the route’s action/scope. Browser sessions and unrestricted tokens may choose any restrictions. A restricted caller with an expiring token must give every child token an expiry at or before its own; omitting/removing expiry is denied. Updating a token also requires its restrictions to be a subset of the caller’s and the resulting expiry to obey the same bound. A restricted caller therefore cannot rename or extend a broader sibling token. Browser sessions and unrestricted tokens retain the ordinary name/expiry update rules.
```sh
karactl tokens create --name inventory --read-only \
--scope-namespace prod --permission clusters:read
```
For MCP clients, `karactl mcp config` mints the right restricted token directly: read-only and optionally namespace-limited for the management endpoint, or scoped to one cluster with explicit `cluster-data` permissions for its data endpoint (`--cluster NS/NAME`). See the [MCP quick start](/docs/mcp-quickstart/).
OIDC group grants apply to browser sessions, not tokens: token authentication uses the owner’s user/role subjects. Give automation accounts explicit user or role bindings. See [ADR 0041](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0041-restricted-api-tokens.md).
# REST API and SDK
> The public contract is api/openapi.yaml, OpenAPI 3.0.3; use the specification embedded in your running server at GET /api/v1/openapi.yaml for…
The public contract is [api/openapi.yaml](https://github.com/KarakoramDev/Karakoram/blob/main/api/openapi.yaml), [OpenAPI](https://www.openapis.org/) 3.0.3; use the specification embedded in your running server at **`GET /api/v1/openapi.yaml`** for deployment-matched schemas and clients.
Most routes are under `/api/v1`. `/healthz` and authenticated `/metrics` are installation-root exceptions. The specification download is public, served as `application/yaml` and cacheable for five minutes; there is no bundled Swagger UI. Relay and agent protocols are not public management APIs.
Agents can use the embedded [MCP server](/docs/mcp/) at `/mcp`, with the same REST enforcement and API tokens.
## Authentication
[Section titled “Authentication”](#authentication)
Scripts use a Karakoram API token, **not a provider infrastructure credential** (such as a [Zerops](https://zerops.io/) integration token):
```http
Authorization: Bearer
```
Bearer tokens use their owner’s current permissions and need no CSRF header. Local `POST /auth/login` takes `{username, password}`, sets session cookies and returns `{account, csrfToken, expiresAt}`. Cookie-authenticated unsafe methods require **`X-Karakoram-CSRF`** matching the session token/readable CSRF cookie; login itself does not. `GET /auth/me` returns the current cookie session’s CSRF token, including after [OIDC](https://openid.net/developers/how-connect-works/). `POST /auth/refresh` rotates refresh credentials; reusing an older refresh token revokes that family. Complete mandatory password changes and reauthenticate before creating tokens or ordinary work.
OIDC uses `/auth/oidc/login` and `/auth/oidc/callback` with redirects/state/PKCE, not device-code login. Successful OIDC redirects to the installation root; `returnTo` currently does not change that. Account/password rules, token expiry/revocation and SSO setup live in [access control](/docs/access-control/); cookie protections live in [security](/docs/security/).
`POST /api-tokens` accepts optional immutable `restrictions`: `readOnly`, `namespaces`, `clusters` (`namespace/name`), `permissions` (`resource:action`, action may be `*`) and `allowCredentialReveal`. Token metadata returns the same fields. Omission is unrestricted; an explicit object defaults credential reveal to false. Updates accept only name/expiry. Restrictions intersect owner RBAC on every route and returned object, including replay and streams; permission discovery reports effective authority. See [restricted API tokens](/docs/access-control/#restricted-api-tokens) for the subset issuance rule and OIDC group limitations.
The UI bounds ordinary API requests to **30 seconds**, excluding SSE. Network/5xx failures preserve the session and offer retry. A 401 after a successful refresh, or a new 401 within **ten seconds** of that refresh, ends the session rather than repeatedly rotating credentials.
### Curl bootstrap
[Section titled “Curl bootstrap”](#curl-bootstrap)
Requires [Bash](https://cgit.git.savannah.gnu.org/cgit/bash.git/), [curl](https://curl.se/), [jq](https://jqlang.org/) and an enabled local account whose mandatory password change is complete. Replace the origin without a trailing slash. The protected temporary cookie jar and CSRF header are both needed for token creation:
```bash
set -euo pipefail
set +x
umask 077
SERVER='https://example.com'
COOKIE_DIR=$(mktemp -d)
trap 'rm -rf "$COOKIE_DIR"' EXIT
COOKIE_JAR="$COOKIE_DIR/cookies"
read -r -p 'Karakoram username: ' USERNAME
read -r -s -p 'Karakoram password: ' PASSWORD
printf '\n'
LOGIN=$(jq -n --arg username "$USERNAME" --arg password "$PASSWORD" \
'{username: $username, password: $password}' \
| curl --fail-with-body --silent --show-error \
-c "$COOKIE_JAR" -H 'Content-Type: application/json' \
--data-binary @- "$SERVER/api/v1/auth/login")
unset PASSWORD
if [ "$(printf '%s' "$LOGIN" | jq -r '.account.mustChangePassword')" = true ]; then
printf 'Complete the required password change and sign in again.\n' >&2
exit 1
fi
CSRF=$(printf '%s' "$LOGIN" | jq -er .csrfToken)
unset LOGIN
CREATED=$(curl --fail-with-body --silent --show-error \
-b "$COOKIE_JAR" -c "$COOKIE_JAR" \
-H 'Content-Type: application/json' -H "X-Karakoram-CSRF: $CSRF" \
--data '{"name":"automation"}' "$SERVER/api/v1/api-tokens")
KARAKORAM_TOKEN=$(printf '%s' "$CREATED" | jq -er .secret)
TOKEN_ID=$(printf '%s' "$CREATED" | jq -er .token.id)
unset CREATED
curl --fail-with-body --silent --show-error \
-H "Authorization: Bearer $KARAKORAM_TOKEN" "$SERVER/api/v1/auth/me"
curl --fail-with-body --silent --show-error \
-b "$COOKIE_JAR" -H "X-Karakoram-CSRF: $CSRF" \
-X POST "$SERVER/api/v1/auth/logout"
unset CSRF
```
Save the one-time `KARAKORAM_TOKEN` securely before closing the shell. `TOKEN_ID` identifies metadata/revocation, not the bearer secret. Logout closes the temporary session without revoking the API token. Revoke it with `DELETE /api-tokens/{id}` when no longer needed. Keep credentials out of shell tracing, logs and browser local storage.
## Authorization and credential reveal
[Section titled “Authorization and credential reveal”](#authorization-and-credential-reveal)
[OpenAPI](https://www.openapis.org/)’s `x-karakoram-rbac` declares each operation’s resource/action/object. Lists, streams and metrics filter individual authorized objects; collection access does not reveal all resources. See [access control](/docs/access-control/).
`GET /clusters/{cluster}/connection` is an explicit, audited reveal requiring `cluster-credentials:read`, with `Cache-Control: no-store`. It returns **engine driver-defined administrative credentials** and public CA, not an application identity. Ordinary cluster/status reads are password-free. Follow engine access procedures: [MongoDB](/docs/mongodb/access-and-tls/), [RabbitMQ](/docs/rabbitmq/access-and-tls/), [Elasticsearch](/docs/elasticsearch/access-and-security/). Storage keys/private PEM inputs are write-only; generated application passwords and API-token secrets are creation responses, not later GET results.
[PostgreSQL](https://www.postgresql.org/) and [Supabase](https://supabase.com/) use [native PostgreSQL access](/docs/postgres/access-and-tls/), not `/database-users` or `/database-roles`. Supabase’s separate GET `/clusters/{cluster}/supabase/credentials` requires `cluster-credentials:read` and returns API/Studio URLs and sensitive service/dashboard keys with no-store; do not confuse this with a normal cluster read.
## Operation safety metadata
[Section titled “Operation safety metadata”](#operation-safety-metadata)
Every [OpenAPI](https://www.openapis.org/) operation declares `x-karakoram-agent` immediately after its `x-karakoram-rbac` block. `go generate ./api/...` validates this inventory and emits `internal/api/agentmap.gen.go`; CI’s `make openapi-check` rejects missing, invalid or stale metadata.
| Field | Contract |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tool` | Unique resource/verb snake_case name, 1–64 characters, starting with a lowercase letter. PUT uses `replace`; PATCH uses `update`. |
| `exposed`, `reason` | Exposure defaults to true. Browser-session, streaming and API-description operations use `exposed: false`, a nonempty reason and no tool. |
| `risk`, `confirm` | `read` and `write` require `none`. `destructive` requires `confirm` or `name` (type the target’s name). |
| `secret` | Response can reveal credentials; defaults to false. Write-only inputs, namespace token status and password-free connection summaries do not count. |
| `async` | `none`, `operation` (poll an Operation), `generation` (observe cluster convergence), or `record` (poll a backup/restore record). |
| `idempotent` | Repeating identical arguments has no additional effect; defaults to false. This is not proof of side-effect freedom. |
| `untrusted` | Response carries engine or end-user content; defaults to false. Treat it as data, not instructions. |
A non-GET `read` requires `idempotent: true` and a reviewed operation-ID/method/path entry in `agentgen`’s side-effect-free allowlist. The allowlist covers permission checks and the four validation-only dry-run routes. Authentication observability is not a domain mutation.
Status, member, operation, event and log responses are untrusted, including nested engine messages in cluster and namespace responses. Restore and storage verification messages also receive this classification. Backup deletion uses `async: operation` because its asynchronous response is an Operation, while backup and restore creation use `record`. Namespace deletion returns no Operation or record, so it uses `none`; observe its lifecycle separately.
[OpenBao](https://openbao.org/)’s connection response is an exception to the administrative-credential set: it contains HTTPS endpoints and CA only. Initialization material has a separate confirmed reveal route and permission; see [OpenBao access and TLS](/docs/openbao/access-and-tls/).
The `openbao_unseal_keys_reveal` management tool is secret and requires explicit confirmation, as do creation and deletion of managed database connections. Restricted tokens need both `allowCredentialReveal` and the independent `openbao-unseal-keys:reveal` permission to reveal initialization material; ordinary `cluster-credentials:read` does not grant it. Connection list/get tools return password-free summaries and are idempotent reads.
## Resource families
[Section titled “Resource families”](#resource-families)
Paths below are relative to `/api/v1`; cluster suffixes follow `/clusters/{cluster}`. **All** remains subject to engine driver capabilities and RBAC. Use [OpenAPI](https://www.openapis.org/) for exact paths, payloads and permissions.
| Family | Routes or cluster suffixes | Engines |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Identity/policy/settings | `/auth/*`, `/accounts`, `/api-tokens`, `/permissions`, `/rbac/policy`, `/rbac/can`, `/settings` | All |
| Discovery/health | `/providers`, `/providers/{provider}`, `/backends`, `/version`; root `/healthz` | All |
| Namespaces | `/namespaces`, `/namespaces/{namespace}`, `/token`, `/validate` suffixes | All |
| Cluster state/connections | `/clusters`, `/clusters/{cluster}`; `/status`, `/members`, `/placement`, `/connection` | All |
| Lifecycle | `/restart`, `/resume`, `/retry-deployments`, `/rotate-tls`, `/members/{member}/replace`, `/system-users`, `/system-users/{user}/rotate` | Engine driver-supported capabilities |
| Compatibility/balancing | `/raise-fcv`, `/balancer` | MongoDB only |
| Database identities | `/database-users`, `/database-roles` and identity paths | MongoDB only |
| Native access | `/rabbitmq/vhosts`, `/rabbitmq/users`, `/rabbitmq/permissions`, `/rabbitmq/topic-permissions`, `/rabbitmq/policies`, `/rabbitmq/operator-policies` | RabbitMQ only |
| Native access | `/elasticsearch/users`, `/elasticsearch/roles`, `/elasticsearch/api-keys` | Elasticsearch only |
| Native PostgreSQL access | `/postgres/roles`, `/postgres/roles/{role}`, `/postgres/roles/{role}/password`, `/postgres/databases`, `/postgres/databases/{database}`, `/postgres/grants` | PostgreSQL and Supabase |
| PostgreSQL consumer bindings | `/postgres/bindings`, `/postgres/bindings/{binding}/rotate` | PostgreSQL source; metadata only, never credentials |
| Native MySQL access | `/mysql/users`, `/mysql/users/{user}`, `/mysql/users/{user}/password`, `/mysql/users/{user}/password/discard-old`, `/mysql/databases`, `/mysql/databases/{database}`, `/mysql/grants` | MySQL and PXC |
| PXC minority recovery | `/mysql/recover-quorum` | PXC only; explicit confirmation and provider fencing |
| Native Neo4j access | `/neo4j/users`, `/neo4j/users/{user}`, `/neo4j/users/{user}/password` | Neo4j Community only; all users have administrative privileges |
| Native Weaviate access | `/weaviate/users`, `/weaviate/users/{user}`, `/weaviate/users/{user}/rotate-key`, `/weaviate/roles`, `/weaviate/roles/{role}` | Weaviate |
| Supabase services | `/supabase/credentials`, `/supabase/smtp-password`, `/supabase/functions`, `/supabase/functions/{name}`, `/supabase/rotate-jwt-secret` | Supabase |
| OpenBao | `/openbao/unseal-keys/reveal`, `/openbao/database-connections`, `/openbao/database-connections/{connection}` | Confirmed initialization-material reveal; list/create/get/delete managed target connections |
| Temporal certificates | `POST /temporal/client-certificates` (issue once); `GET /temporal/client-certificates` (metadata) | Temporal |
| Temporal namespaces | `GET /temporal/namespaces`; `DELETE /temporal/namespaces/{temporalNamespace}` (asynchronous) | Temporal |
| Temporal UI secret | `PUT /temporal/ui/oidc-client-secret` (write-only) | Temporal |
| Recovery | `/backup-storages`; cluster `/backups`, `/restores` | All |
| Continuous recovery window | `/pitr` | MongoDB, PostgreSQL, Supabase, MySQL and PXC |
| Activity/cleanup | `/operations`, `/operations/{operation}`, `/operations/{operation}/cancel`, `/events`, `/events/stream`, `/orphaned-volumes`; cluster `/operations` | All |
| Observability | Cluster `/metrics`, `/members/{member}/logs`; root `/metrics` | All |
| Fleet operations | Cluster `/upgrade-agents`, `/members/{member}/move`; `/settings/agent-rollout` | Engine driver-supported rolling upgrades and relocation |
| Control-plane recovery | `/control-plane/backup-policy`, `/control-plane/backups` | Installation-wide |
| Notifications | `/notification-channels`, `/notification-channels/{channel}`, `/notification-channels/{channel}/test` | Installation-wide |
| Backup verification | Cluster `/backup-drills`, `/backup-drills/{drill}` | Engines with eligible successful backups |
[PostgreSQL](https://www.postgresql.org/) role PUT replaces attributes; PATCH changes only supplied fields. Database routes expose creation/read/deletion, not update; grants are bounded database/schema privileges. Generated login passwords are one-time no-store responses, with sealed idempotent replay. Native access uses `database-users` RBAC, with extra `clusters:update` authorization for privileged role/membership/password changes. See [payloads and limits](/docs/postgres/access-and-tls/). [Neo4j](https://neo4j.com/) native-user reads use `database-users:read`; create/delete/password changes additionally require `clusters:update` alongside the corresponding `database-users` permission. Password responses are no-store. The managed `neo4j` administrator is hidden from lists and cannot be mutated here. See [Neo4j access and TLS](/docs/neo4j/access-and-tls/).
See [consumer bindings](/docs/postgres/bindings/) for rotation and deletion/upgrade guards. In-place restores of a bound PostgreSQL source require `acknowledgeBoundConsumers: true`; otherwise HTTP 409 `bound_consumers` lists affected consumer identities.
Distributed PostgreSQL exposes `POST /clusters/{cluster}/actions/rebalance` with `{strategy?: string}` and returns 202 Operation (`type: rebalance`). It uses `clusters:update`, CSRF and idempotency conventions. Members expose a read-only `group` (0 coordinator); `CitusTopology` conditions report metadata drift. See [distributed PostgreSQL](/docs/postgres/distributed/). [MySQL](https://www.oracle.com/mysql/) native access uses `database-users` RBAC for application users at the fixed `%` host, databases and bounded database privileges. User PATCH changes only supplied connection-limit and lock fields. Create and password rotation return generated passwords only once, with `Cache-Control: no-store` and sealed replay when an idempotency key is supplied. Rotation accepts `retainCurrent` for dual passwords; discard-old removes the retained password. Database creation defaults to `utf8mb4` / `utf8mb4_0900_ai_ci`; deletion requires `force=true` for a nonempty database. Grant listing requires at least one `user` or `database` filter. Global privileges and `GRANT OPTION` are not exposed.
The topology contract accepts MySQL Group Replication `group` and [PXC](https://www.percona.com/mysql/software/percona-xtradb-cluster) `cluster` alongside `single`; engine driver catalogs identify which topologies each engine supports. Backup type enums include `incremental` for MySQL/PXC, separately from PostgreSQL’s `incr` spelling.
MySQL and PXC share the native access routes, not engine driver identity or interchangeable backups. See [access and TLS](/docs/mysql/access-and-tls/), [GTID-based recovery](/docs/mysql/backups/) and [explicit PXC minority recovery](/docs/mysql/operations/#explicit-pxc-minority-recovery). Quorum recovery requires cluster update permission, returns an accepted Operation and is not arbitrary SQL or a fencing bypass.
[Weaviate](https://weaviate.io/) uses [native users/roles](/docs/weaviate/access-and-tls/), not [MongoDB](https://www.mongodb.com/) identity routes. List/get requires `database-users/read`; user create/update/delete uses the corresponding action. Assigning any role other than `viewer` or `read-only` additionally requires `clusters/update`, including roles already present on user update. Key rotation requires `database-users/update` and, if the user’s current roles include any role beyond `viewer` or `read-only`, `clusters/update`. User deletion requires only `database-users/delete`, even for privileged users. Every role create/replace/delete requires both `database-users/update` and `clusters/update`. System/environment identities and built-in roles remain protected. Creation/key rotation returns no-store generated API keys; GET cannot retrieve them, while matching idempotency retries replay sealed results for 24 hours. Administrative reveal separately requires `cluster-credentials:read`.
[Supabase](https://supabase.com/) function GETs use `clusters:read`, mutations `clusters:update`; SMTP password is write-only. Function PUT replaces the relative-path/source map (100 functions maximum, 1 MiB each, nonempty `index.ts`). JWT rotation returns a durable operation. Source storage does not establish runnable Functions availability; enabled service artifacts must exist in the release lock. See [Supabase operations](/docs/supabase/operations/).
[Temporal](https://temporal.io/) certificate issuance requires `cluster-credentials:read`; certificate/namespace listing requires `clusters:read`; namespace deletion and UI [OIDC](https://openid.net/developers/how-connect-works/) secret writes require `clusters:update`, all scoped to the cluster. Issuance returns private material once with `no-store`; secret writes return no value. See [Temporal access/TLS](/docs/temporal/access-and-tls/) and [namespace/recovery operations](/docs/temporal/operations/).
Fleet disruption requests and cluster updates support `ignoreMaintenanceWindow=true` where declared in the specification. Without an override, accepted operations can remain Pending until the next configured window; acceptance is not permission to interrupt an in-progress member step. See [maintenance and relocation](/docs/operations/) and [agent rollout](/docs/getting-started/upgrade/).
Installation-wide agent rollout, control-plane backup and notification routes use `settings:read` or `settings:update`, not cluster permissions. Notification destinations and credentials are write-only; omitted secrets on replacement are retained only for an unchanged destination identity. Backup-drill creation uses `restores:create` on the source cluster; report reads use `restores:read`. Drill and control-plane backup responses are durable reports rather than cluster `Operation` resources: poll their respective inventories. See [notifications](/docs/notifications/), [state recovery](/docs/control-plane-backup/) and [restore drills](/docs/backup-drills/).
## Namespace parameters and lifecycle
[Section titled “Namespace parameters and lifecycle”](#namespace-parameters-and-lifecycle)
Object routes and creation default to **`default`**. Omitting `namespace` on cross-cluster collections (`/clusters`, `/operations`, `/events`, `/events/stream`, `/orphaned-volumes`, `/backup-storages`, root `/metrics`) includes all authorized namespaces; pass `namespace=NAME` to filter. Omit it rather than sending an empty value. Namespace is part of resource identity: retain it in caches and links.
For an explicit nonexistent namespace, authorized callers receive **404** `not_found` with `details.resource: namespace` and `details.name`. Without a permission there, collections remain empty and object requests retain their normal **403**, regardless of existence. Public/authentication routes do not resolve this parameter.
**`GET /namespaces` paginates normally**, after authorization filtering. Namespace creation accepts required `name`, `projectId` (provider target ID) and write-only `token`, plus optional `description`, `backend` (default `zerops`) and `config` (non-secret string settings). `GET /backends` requires `namespaces/read` on global `*` and lists registered kinds, capabilities, relay requirements and config keys; namespace responses include `backend` and enabled `capabilities`. Registered providers are `zerops`, `dokploy`, and `nomad`; unknown config keys return **400**. [Zerops](https://zerops.io/) accepts no config keys; its API URL is operator-only (`KARAKORAM_ZEROPS_API_URL`) and applies equally to legacy and new namespaces. [Dokploy](https://dokploy.com/)’s `url` and [Nomad](https://developer.hashicorp.com/nomad)’s `address` settings are restricted to operator-approved origins ([Dokploy target settings](/docs/dokploy/#target-settings), [Nomad target settings](/docs/nomad/#target-settings)).
Creation returns **201** after synchronous validation, then deploys a relay asynchronously only if the provider requires one (as Zerops does). **`DELETE /namespaces/{namespace}` returns 202 without a body or `Karakoram-Operation-Id`**: poll namespace lifecycle/health and events until removal, not `/operations`. Registration, token validation/rotation and guarded deletion belong in [namespaces](/docs/namespaces/). Recovery source namespace fields and source-export permissions belong in [backups](/docs/backups/).
## Requests and asynchronous work
[Section titled “Requests and asynchronous work”](#requests-and-asynchronous-work)
Create accepts `{name, spec}`; PUT replaces `{spec}`; PATCH accepts only `{spec: {...}}` as a JSON merge patch. Arrays replace entirely. Name/status/generation/timestamps are server-owned. Unknown fields, wrong types, invalid enums/bounds and trailing JSON produce **400** with `details.field` and `details.reason`. Explicit zero is not a request for defaults: resources and required member counts must satisfy their bounds, while optional component `replicas: 0` explicitly disables that component where the engine permits it. `spec.provider` is required; omitted version/topology/components/resources use that catalog’s defaults. [Go](https://go.dev/) input fields use pointers: nil requests defaults, a pointer to zero does not.
Resource defaults apply per field, not only when the entire `resources` object is absent, and are filled before engine validation. For example, Keycloak accepts `"resources":{"minDisk":1,"maxDisk":1}` with its catalog CPU/RAM defaults. Supplied bounds are retained and must meet engine-specific floors and ordering; an omitted maximum is at least its corresponding minimum.
Cluster create, PUT and PATCH apply engine defaults and validation, then check the namespace provider’s static capabilities before saving. A spec requiring an unavailable capability returns **400 `invalid_request`** naming the provider and missing capabilities (for example `publicTcp`); admission does not make a platform API request.
`spec.dataMcp` opts into per-cluster [data MCP](/docs/data-mcp/#enable-and-scope-access): `enabled`, `access`, engine-specific `scope`, `tools`, `auditQueryText` and companion `resources`; observed fields live in `status.dataMcp`. The MCP transport is **outside `/api/v1`**, at `/mcp/clusters/{namespace}/{cluster}`, with restricted bearer-token authentication and its own [request, result and rate limits](/docs/data-mcp/#confirmation-results-and-limits), including a 1 MiB request-body cap. REST session cookies and unrestricted API tokens do not authorize data MCP.
Cluster PUT/PATCH and queued actions commit against the cluster incarnation and desired generation inspected during admission. Concurrent changes return **409 conflict** rather than overwriting a newer spec or attaching work to a recreated cluster with the same name. Reload current state and retry the intended change. Suspended clusters still accept resume, full deletion, and backup deletion; ordinary runtime actions such as restart, backup, and restore require resume first.
`POST /clusters/{cluster}/retry-deployments?namespace=NS` requires `clusters/update` and returns `202` with the cluster after advancing its server-managed retry epoch and generation. It re-arms deployments in `DeploymentRetriesExhausted`; when none remain exhausted in the current epoch it returns `409 conflict` without a mutation. An `Idempotency-Key` replay returns the original result, not another retry budget. See [deployment recovery](/docs/clusters/#retry-exhausted-platform-deployments).
Send the declared Content-Type: usually `application/json`, but cluster PATCH uses `application/merge-patch+json` (`application/json` is also accepted there for older clients). The request-body limit is **2 MiB**, including chunked requests. Responses echo `X-Request-ID`; optionally supply 1–128 characters from `[A-Za-z0-9._:-]`. Invalid/repeated IDs are replaced. Include the ID, never credentials, in error reports.
Accepted cluster mutations have durable Operations. Cluster/backup/restore responses expose `operationId`; operation-bearing responses also set **`Karakoram-Operation-Id`**. Imperative work commonly returns **202**; cluster creation returns **201**, not completed provisioning. Cluster DELETE returns **202 with an Operation body**. Poll `/operations/{id}` and resource status/inventory. HTTP acceptance, terminal Operation phase and cluster health are distinct; see [operations and status](/docs/clusters/#operations-and-status). A wait timeout/disconnect does not cancel work.
Cancellation uses **`POST /operations/{operation}/cancel`**, requiring `clusters/update` on the operation’s persisted namespace/cluster; no namespace override is needed. Acceptance returns **202 Operation** plus the operation header. Pending work becomes Cancelled immediately. Supported Running checkpoints set `cancelRequested: true` and can remain Running during cleanup. Terminal/unsafe states return **409 conflict**, `details.reason: invalid_state`. Poll the same ID; cancellation is neither rollback nor reversal of accepted desired state. Engine guides define safe checkpoints.
## Server-side dry-run
[Section titled “Server-side dry-run”](#server-side-dry-run)
Preview cluster creation with `POST /dry-run/clusters`, replacement or merge patches with `PUT` / `PATCH /dry-run/clusters/{cluster}`, and restores with `POST /dry-run/clusters/{cluster}/restores` (all relative to `/api/v1`). Use the corresponding mutation’s body, content type and `namespace` query. The same authentication, cookie CSRF, RBAC (including recovery-source permissions), defaults, engine driver validation and read-only admission checks apply.
A successful preview returns **200 DryRunResult**: `operation` is `create`, `replace`, `patch` or `restore`; `cluster` contains the normalized candidate (the current target for restore), and `restore` is the prospective record when applicable. `changes` contains dot paths under `spec` with before/after values; create and restore return an empty changes list. Sensitive diff values are masked. `effects` describes generic version, member, resource, exposure, TLS, backup and recovery consequences; it does not predict engine-specific restarts. `warnings` contains applicable admission caveats. Placement warnings describe observed member/container placement, not a predicted node allocation. A create preview has no provisioned members and does not forecast co-location from the current Ready-node count; empty `warnings` and an empty `status.placement.warning` do not certify future host separation. After provisioning, inspect `/clusters/{cluster}/placement` and the cluster’s placement status. See [backend placement limits](/docs/limitations/).
Dry-run does not persist clusters, secrets, operations, restores or audit events, publish events, trigger reconciliation, or reserve/replay `Idempotency-Key`. Normal authentication may still update token last-used metadata. No operation ID is allocated, and an existing cluster’s generation is unchanged. Admission is a point-in-time observation, not a reservation or guarantee that a later mutation or restore will succeed. `karactl` exposes these endpoints with [`--dry-run`](/docs/cli/).
## Fleet operations and resilience contract
[Section titled “Fleet operations and resilience contract”](#fleet-operations-and-resilience-contract)
These API groups implement fleet operations and installation resilience:
| Group | Routes | Success contract and permission |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Agent rollout | `POST /clusters/{cluster}/upgrade-agents` | Optional `{includeCurrent}`; **202 Operation**, `clusters:update` on the cluster; **409** when agents are already current. |
| Agent rollout settings | `GET/PUT /settings/agent-rollout` | `AgentRolloutSettings` with `automatic`, `maxConcurrentClusters` (1–5) and read-only `currentBuild`; installation `settings:read/update`. |
| Member move | `POST /clusters/{cluster}/members/{member}/move` | **202 Operation**, `clusters:update`; placement guarantees depend on provider capabilities. |
| Control-plane backups | `GET/PUT /control-plane/backup-policy`, `GET/POST /control-plane/backups` | Installation `settings:read/update`; POST returns **202 ControlPlaneBackup**, not a cluster Operation. |
| Notifications | `GET/POST /notification-channels`, `GET/PUT/DELETE /notification-channels/{channel}`, `POST /notification-channels/{channel}/test` | Installation `settings:read/update`; test returns **200 NotificationTestResult**. |
| Backup drills | `GET/POST /clusters/{cluster}/backup-drills`, `GET /clusters/{cluster}/backup-drills/{drill}` | `restores:read/create` on the source cluster; POST accepts optional `{backup}` and returns **202 BackupDrill**. |
Lists use `items` and optional `nextCursor`. New mutations declare CSRF and optional `Idempotency-Key` headers. Notification destination URLs, signing secrets and SMTP passwords are **write-only**; responses expose only configured flags and secret-free status.
Cluster desired state adds `spec.maintenance{timeZone, windows[{schedule, duration}]}` and `spec.backup.drill{enabled, schedule, backupSchedule, timeout}`. Agent build identity is exposed alongside existing member agent connectivity as `agent.version`, `agent.commit` and `agent.build`; `status.agents` summarizes outdated/unknown members. `status.maintenance` reports the computed open window. `managedBy` is server-owned. Placement exposes provider-neutral `source` and `guaranteedAntiAffinity`.
The boolean query `ignoreMaintenanceWindow` is declared on cluster PUT/PATCH, restart, TLS rotation, member replacement/move, agent upgrade and FCV raise. It is an explicit per-request override, not a persisted desired-state policy.
## Safe retries with Idempotency-Key
[Section titled “Safe retries with Idempotency-Key”](#safe-retries-with-idempotency-key)
Supply the header only on operations declaring `IdempotencyKey` in [OpenAPI](https://www.openapis.org/): supported resource creation/update and lifecycle/recovery actions, not every mutation. Use a unique random key per intended mutation, **1–255 visible ASCII characters without spaces**, and reuse the same key/body for network retries.
* Scope includes account creation identity/[OIDC](https://openid.net/developers/how-connect-works/) issuer or the specific bearer token, method, route, namespace and path parameters. Omitted namespace and explicit `default` share a scope. Authentication, CSRF and authorization are checked again on replay.
* Successful **2xx** status/body/relevant headers are replayable for **24 hours**, including one-time secrets and operation IDs. Replay sets `Idempotent-Replayed: true`. It is the original response snapshot; query current state separately.
* JSON whitespace/key order do not affect matching; array order, values, omitted fields, explicit null and numeric spellings do. A changed body returns **409 conflict**, `details.field: Idempotency-Key`, `details.reason: conflict`. Concurrent in-flight duplicates return **409** with reason `invalid_state`; retry the same key/body after completion.
* **4xx/5xx are not cached**. Operation-backed mutations atomically save a recovery response with their accepted operation. If the server stops or the final response cannot be saved after acceptance, the same key replays the accepted resource/Operation snapshot and its original `Karakoram-Operation-Id`, rather than creating another action. The recovery response keeps the endpoint’s normal success status and response shape; once the ordinary successful response is saved, replay uses its exact status/body/headers. Non-operation-backed mutations can still have an ambiguous remote or response boundary: inspect their resources before retrying. Expiry permits new execution; a fresh key is not a recovery shortcut or an exactly-once guarantee.
`karactl` does not send this header and has no flag for it. Raw HTTP clients set `Idempotency-Key`; [Go](https://go.dev/) callers can pass a per-call `client.RequestEditorFn` that sets the header, or configure request editing with `client.WithRequestEditorFn`. Retain each mutation’s key across retries; never reuse one fixed key for unrelated writes or log replayed secrets.
A cluster PATCH or PUT whose normalized desired spec and supplied secret values are unchanged returns HTTP 200 without creating an operation or cancelling in-flight work. This no-op behavior does not require an `Idempotency-Key`. Changed desired state still creates a superseding operation; keyed retries replay their original response as described above. An explicit `ignoreMaintenanceWindow=true` still releases the accepted generation when the spec is unchanged: it persists that generation’s override and triggers reconciliation without replacing its in-flight operation. Repeating an already applied override does not trigger more work.
## Pagination and events
[Section titled “Pagination and events”](#pagination-and-events)
Lists return `{items: [...], nextCursor?: "..."}`. `limit` defaults to **100**, range **1–500**. Follow the opaque cursor with the same filters until absent; authorization precedes pagination. Multiple pages are not a transactional snapshot. Specialized status/log responses use their own schemas. Member logs are bounded tails: `process`/`lines` work, but `cursor`/`since` currently do not, and no next cursor is returned.
Member responses and cluster status members include `agent.connected`, optional `agent.configVersion` and `agent.lastSeenAt`, and an optional `agent.message` diagnostic. The message is sanitized using the same credential and base64 redaction rules as other diagnostic text and is omitted when empty.
`/events` and `/events/stream` share namespace/cluster/type/operationId/actor/resource filters. Actor/resource matches are exact, case-sensitive AND intersections. Types are **Normal, Warning, Audit**; lifecycle names such as `OperationFailed` are reasons. Unknown types/resources return 400; explicitly unreadable resource categories return 403, while unfiltered results omit unauthorized records. Cluster events require `events:read` or `audit:read`; namespace/global audit requires the corresponding namespace/global audit permission. Lists default oldest-first; `order=desc` pages newest-to-oldest.
Events may include optional `details`. For `reason: "ClusterSpecChangeRequested"`, it contains `changes?: EventChange[]` and `truncated?: boolean`. Each change has required `path: string`, optional JSON-valued `before`/`after`, and optional boolean `sensitive`/`truncated`:
```json
{
"details": {
"changes": [
{"path": "components.engine.resources.maxCpu", "before": 1, "after": 2}
]
}
}
```
Paths are dotted JSON field paths. Schedules and other named arrays use entry names or IDs rather than positions, for example `backup.schedules[nightly].retainCount`. An absent `before` or `after` normally means the field was absent; explicit JSON `null` remains `null`. Sensitive changes use the [redaction markers and rules](/docs/security/#audit-attribution-and-durability).
| Bound / flag | Meaning |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Value exceeds 1 KiB of encoded JSON | After redaction, replaced by `(N bytes)` and the change marked `truncated: true` |
| Details exceed 32 KiB | Values may be omitted with per-change and top-level `truncated: true`; if paths alone exceed the cap, only an ordered prefix remains; individual oversized paths may be shortened |
| Change `truncated: true` | A missing `before`/`after` means omitted content, **not unset** |
| `details.truncated: true` | The comparison may lack values or paths |
| Human-readable summary | At most 512 bytes; individual displayed values at most 64 bytes; [summary format and UI workflow](/docs/operations/#reviewing-audit-activity) |
Both list and stream omit the entire `details` object without [structured audit access](/docs/access-control/#policy-format). Clients should use `details.changes`, not parse the message, and check truncation flags before treating a comparison as complete. Older events may lack `details`. To distinguish a requested change from convergence, check [cluster status and Operations](/docs/clusters/#operations-and-status).
SSE uses `text/event-stream`, numeric IDs and stored types as `event:` names. Reconnect with **`Last-Event-ID`**; an explicit `cursor` query takes precedence. Preserve filters and the last dispatched ID. Parse complete blank-line-delimited frames, join `data:` lines, and ignore `:` comments; comments never advance the cursor. Authentication/permissions are revalidated during delivery and heartbeats.
The stream sends an immediately flushed connected comment and **15-second heartbeats**, each approximately **16 KiB** padded: budget roughly **1.1 KiB/s per connection** before compression, plus startup. Proxies must preserve `Cache-Control: no-cache, no-transform` and `X-Accel-Buffering: no`; configure nonbuffering delivery (`proxy_buffering off` where applicable). Small events can still wait until a heartbeat. Use durable lists/status when live delivery is stale, not repeated mutations.
## Errors
[Section titled “Errors”](#errors)
Errors are `{code, message, details?}`. Branch on code/field/reason, not English messages. An HTTP-successful poll can describe a failed Operation.
Cluster create, PUT and PATCH return `400 invalid_request` when fresh provider information positively establishes that public `spec.exposure.ipType` is unavailable. [Zerops](https://zerops.io/) checks current project IPv4/IPv6 addresses, not cached namespace identity. Select a provisioned family, provision the requested public address, or disable public exposure. A failed lookup or provider without authoritative family information does not reject the request: reconciliation reports `ExposureSupported=Unknown/PublicIPSupportUnknown` while attempting routing normally. [Dokploy](https://dokploy.com/) and [Nomad](https://developer.hashicorp.com/nomad) accept both family selections for native published ports; configured public addresses advertise endpoints, not an authoritative inventory of host ingress families.
Updates that preserve public/private mode, IP family and allowlist keep unchanged legacy exposure; an unrelated update is not rejected solely because that existing exposure is unavailable. New exposure choices or routing changes still undergo the availability check.
For already accepted requests, a definitive unsupported-family result sets `ExposureSupported=False/PublicIPTypeUnsupported`, `Ready=False`, and phase `Degraded`; the current generation’s convergence operation fails rather than remaining Running. Desired state is retained and reconciled, so correcting exposure (or provisioning the address) can recover it without recreating the database. Existing `400` response schemas are unchanged.
| HTTP | Meaning |
| --------- | --------------------------------------------------------------------------------------------------------------- |
| 400 | Invalid body/query/cursor/configuration/transition |
| 401 | Missing/expired/invalid authentication; invalid OIDC callback |
| 403 | RBAC/ownership/CSRF denial or `password_change_required` |
| 404 | Missing/not-visible resource or route; namespace rules above apply |
| 405 | Unsupported method on known route; inspect `Allow` |
| 409 | Conflicting state, unsafe cancellation, unsupported engine action or retry-key conflict |
| 413 / 415 | Body over 2 MiB / missing or unsupported Content-Type |
| 422 | Namespace token/project/relay-hostname validation failure |
| 429 | Login backoff; honor `Retry-After` and `details.retryAfterSeconds` |
| 500 | Generic unexpected failure, or sanitized dependency failure such as temporary namespace platform unavailability |
| 503 | Route-specific store/monitoring unavailability or disabled/unavailable OIDC issuer |
Actions and native credential/security mutations blocked by an unfinished initial restore return `409 conflict` with `details.resource: operation`, `details.name: ` and `details.reason: invalid_state`. This restore-owner check precedes capability, member-state and native-engine prechecks, so those checks cannot hide the actionable owner. Read operations, target deletion, own backup policy and the narrow pre-claim component amendment rules retain their separate admission behavior.
Outside that fence, an unsupported engine action returns `409 conflict` with `details.resource: operation`, `details.name: ` and `details.reason: unsupported`; the message identifies the engine. [PostgreSQL](https://www.postgresql.org/) and [Supabase](https://supabase.com/) do not support the generic system-user rotation action. Password rotation for **non-reserved PostgreSQL login roles** uses `POST /clusters/{cluster}/postgres/roles/{role}/password`; managed roles such as `postgres` remain engine-owned and are not made rotatable by that hint.
## Clients/SDK
[Section titled “Clients/SDK”](#clientssdk)
Install the public [Go](https://go.dev/) client at a release/commit matching the server:
```sh
go get github.com/KarakoramDev/Karakoram/pkg/client@
```
Use base URL **`https://example.com/api/v1`**, including the suffix, and the release’s required Go version. One-page example:
```go
package main
import (
"context"
"fmt"
"log"
"net/http"
"os"
"time"
"github.com/KarakoramDev/Karakoram/pkg/client"
)
func main() {
c, err := client.NewClientWithResponses("https://example.com/api/v1",
client.WithBearerToken(os.Getenv("KARAKORAM_TOKEN")),
client.WithHTTPClient(&http.Client{Timeout: 30 * time.Second}))
if err != nil { log.Fatal(err) }
r, err := c.ListClustersWithResponse(context.Background(), nil)
if err != nil { log.Fatal(err) }
if r.StatusCode() != http.StatusOK || r.JSON200 == nil {
log.Fatalf("list clusters: HTTP %d", r.StatusCode())
}
for _, cluster := range r.JSON200.Items { fmt.Println(cluster.Name) }
// Follow r.JSON200.NextCursor with the same filters for subsequent pages.
}
```
Transport/decoding errors and HTTP errors are separate: check status and the expected `JSON200`/`JSON201`/`JSON202` pointer; decode error bodies as `client.Error` when needed. Optional fields are pointers; check before dereferencing. Keep one-time secrets out of logs. Poll returned operation IDs, not resource names.
`StreamEvents` returns raw `*http.Response`, not decoded events. Close its body and use a streaming SSE parser, not `io.ReadAll` or a single JSON decoder. Inspect raw frames with:
```sh
curl --no-buffer --fail-with-body \
-H "Authorization: Bearer $KARAKORAM_TOKEN" \
-H 'Accept: text/event-stream' \
'https://example.com/api/v1/events/stream?namespace=default'
```
On reconnect send `Last-Event-ID`; follow the parsing/buffering rules above. Generate other-language clients from the deployment’s spec:
```sh
curl --fail-with-body https://example.com/api/v1/openapi.yaml -o openapi.yaml
openapi-generator-cli generate -i openapi.yaml -g python -o karakoram-python-client
python -m pip install ./karakoram-python-client
npx openapi-typescript openapi.yaml -o karakoram-api.d.ts
npm install openapi-fetch
```
Configure `/api/v1` base URL and bearer authentication. [TypeScript](https://www.typescriptlang.org/) generation produces types; pair with `openapi-fetch`, or generate a runtime client with `-g typescript-fetch`. Generated clients still need error handling, polling and SSE parsing. Repository regeneration belongs in [CONTRIBUTING](/docs/contributing/).
## OpenBao operations
[Section titled “OpenBao operations”](#openbao-operations)
POST `/clusters/{cluster}/openbao/unseal-keys/reveal` accepts `{"confirm":""}`, requires `openbao-unseal-keys:reveal`, returns no-store initialization material and audits disclosure without response secrets. Database-connection GET/list uses `clusters:read`; POST uses `clusters:update` plus target `database-users:create`; DELETE uses `clusters:update` plus target `database-users:delete`. Connections target Ready, TLS-enabled [PostgreSQL](https://www.postgresql.org/), PSMDB replica sets or [RabbitMQ](https://www.rabbitmq.com/) in the same namespace.
POST `/clusters/{cluster}/openbao/database-connections` example:
```json
{"name":"orders","targetCluster":"ordersdb","roles":[{"name":"reader","access":"readonly","defaultTtlSeconds":3600,"maxTtlSeconds":86400}]}
```
The returned secret-free record can be `pending`; use its final `credentialsPath` with a native [OpenBao](https://openbao.org/) token once `ready`. OpenBao issues credentials, not Karakoram. Shared restart/replace-member/rotate-tls routes and catalog-bound version patches drive maintenance. Shared backup/restore routes use `raft-snapshot`; cluster-create and restore requests accept top-level write-only `openbaoUnsealShares` for deleted-source recovery. See [operations](/docs/openbao/operations/) and [recovery request examples](/docs/openbao/backups/).
# Architecture
> Karakoram is a self-hosted, multi-platform control plane that reconciles engine clusters across infrastructure providers and namespace targets. Zerops is…
Karakoram is a self-hosted, multi-platform control plane that reconciles engine clusters across infrastructure providers and namespace targets. [Zerops](https://zerops.io/) is the reference provider; [Dokploy](https://dokploy.com/) and [Nomad](https://developer.hashicorp.com/nomad) are also available. See [Providers](/docs/providers/).
## Deployment and trust boundaries
[Section titled “Deployment and trust boundaries”](#deployment-and-trust-boundaries)
The diagram and concrete paths below show the Zerops provider deployment; other providers provide their own runtime layout, storage and public ingress.
Diagram source
```
flowchart LR
Client[Browser / karactl] -->|Public HTTPS| Routing[Zerops HTTP routing]
Routing -->|HTTP 8080| CP[Karakoram control plane]
CP --> State[(SQLite / karakoramdata)]
CP -->|Project-scoped credentials| Platform[Zerops REST API]
CP -->|Engine driver-authenticated administration| Engine[Engine runtime]
CP -->|Pinned TLS + member token / 9443| Agent[Node agent]
Agent -->|Protected configuration pull / TLS 9443| CP
Agent -->|Render files / supervise processes| Engine
```
* **Control plane:** one runtime service, normally `karakoram`, with one container and a `karakoramdata` Local Storage volume mounted at `/mnt/karakoramdata`. UI/API traffic enters through public HTTPS; the private agent listener uses TLS 9443. [SQLite](https://sqlite.org/) stores desired state, observations, operations, events and encrypted secret values. The master key is a separately managed service secret.
* **Data-bearing members:** singleton [Ubuntu](https://ubuntu.com/) 24.04 runtimes with independently identified persistent volumes and stable identities (Local Storage on Zerops). Increasing a runtime’s container count is not engine replication.
* **Stateless components:** engine driver-supported auxiliary runtimes do not have durable data volumes. Containers sharing a service identity are not independent data-bearing members.
* **Node agent:** `karakoram agent` supervises the selected engine and exposes a bounded private API, not a remote shell.
Private networking alone is not an authentication boundary. Agent traffic uses pinned trust roots and per-member bearer tokens. Public HTTPS, internal agent TLS and engine authentication/TLS protect different paths. See [Security](/docs/security/) for credentials, project-token authority and secret-storage limits. Backup transfer paths are documented in [Backups and storage](/docs/backups/).
## Namespaces and relay routing
[Section titled “Namespaces and relay routing”](#namespaces-and-relay-routing)
A namespace selects an immutable provider kind and target with its infrastructure credentials. Providers are the [Zerops provider](/docs/zerops/) (targets are Zerops projects), the [Dokploy provider](/docs/dokploy/) (Dokploy environments on a [Docker Swarm](https://docs.docker.com/engine/swarm/)), and the [Nomad provider](/docs/nomad/) (Nomad namespaces on an operator-managed cluster). `default` is the control plane’s own target and uses direct private networking. Additional namespaces use an outbound relay only when their provider’s `BackendInfo.Relay` requires one; all three current providers require it.
The relay opens an outbound WebSocket connection through public HTTPS and carries an inner mutually authenticated TLS tunnel. Engine driver-approved administration traffic and protected agent traffic use namespace-qualified streams; agent configuration pulls retain end-to-end TLS. A namespace token or relay outage interrupts management of that namespace, not every project. The relay is not application networking.
See [Namespaces and relays](/docs/namespaces/), the [relay threat model](/docs/security/#relay-threat-model) and [ADR 0018](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0018-multi-project-namespaces-relay.md).
## Infrastructure providers
[Section titled “Infrastructure providers”](#infrastructure-providers)
The server composition root registers compiled-in factories in `infra.Registry`. Each factory constructs and validates a namespace’s `Target`; engines use `infra.Backend`, not the platform client. Neutral `ServiceSpec` values distinguish runtimes from independently identified persistent volumes. `Deployment` pairs an executable artifact with a structured `RuntimeSpec`; each provider renders its own manifest and supplies its runtime `Layout`. Operations remain opaque handles polled through `OpStatus`, and errors use neutral classes.
Provider capabilities describe optional public routing, disk growth, quotas, placement, logs and other resource/runtime features. Engine capability gates reject unsupported requests. `GET /api/v1/backends` publishes registered kinds and capabilities; namespace responses identify their provider and enabled capabilities. All providers must pass the shared contract suite. See [ADR 0030](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0030-backend-neutral-infrastructure-contract.md); the registry does not imply support for an unregistered platform.
## Desired state and durable operations
[Section titled “Desired state and durable operations”](#desired-state-and-durable-operations)
Clusters separate desired `spec` from observed `status`, with generation tracking. Mutations persist intent before reconciliation. Per-cluster workers serialize synchronization and imperative operations; unrelated clusters can progress independently. Engine drivers persist checkpoints and platform identifiers, then observe completion before advancing disruptive work.
Engine driver persistence replaces each owned state value, not just its nonempty fields. [PostgreSQL](https://www.postgresql.org/), [Elasticsearch](https://www.elastic.co/elasticsearch) and [Supabase](https://supabase.com/) replace their complete `Cluster.Internal` namespace objects. Flat [MongoDB](https://www.mongodb.com/) and [RabbitMQ](https://www.rabbitmq.com/) state uses `internal/statejson.ReplaceOwned`: explicit JSON tags define owned root keys, omitted empty fields remove prior values, and unrelated workflow keys survive. Nested values are replaced whole; adding an optional owned field must not require a second, hand-maintained deletion list.
Neither HTTP acceptance nor a successful infrastructure configuration process proves engine health. There is no distributed exactly-once transaction across SQLite, infrastructure providers and engine/storage APIs: durable identifiers and observation constrain replay, while ambiguous outcomes require investigation. [Clusters and operations](/docs/clusters/#operations-and-status) defines generations, operation phases, health, cancellation and wait behavior.
## Node-local durability
[Section titled “Node-local durability”](#node-local-durability)
Agents cache assigned configuration on the data volume with owner-only permissions. Existing data-bearing members can start from that cache during control-plane downtime; first startup without a cache cannot. Stateless runtimes lose cached configuration when replaced.
Rendered credentials, certificates and configuration are restricted to the runtime owner. Explicit maintenance and process-stop holds are persisted separately from pulled configuration, so a configuration refresh or agent restart does not silently release them. Restore/restart recovery remains engine driver-specific: use the engine operations and recovery guides rather than manually overriding holds.
When the member executable is PID 1, a minimal init parent re-executes the real agent as its child. The parent forwards termination, interrupt and hangup signals, reaps adopted daemon descendants, and exits with the real agent’s status. This keeps orphan reaping separate from the agent’s `os/exec` command waits; daemonized database processes cannot remain zombies that block their supervisor’s lifecycle checks. Ordinary non-PID-1 members do not re-execute.
The shared process supervisor reaps each direct child independently of stdout and stderr EOF. After exit it allows up to 250 ms for final log draining, then closes its pipe readers so descendants retaining inherited descriptors cannot block exit status, restart scheduling or stop completion. Normal EOF finishes the drain immediately. Exit observation does not signal descendants; explicit stop/restart retains the configured stop signal to the child’s process group.
## Recovery, monitoring and events
[Section titled “Recovery, monitoring and events”](#recovery-monitoring-and-events)
The control plane owns scheduling and count retention; its downtime pauses new scheduling and retention. Already-started engine-side work can continue without the control plane. Engine backups and control-plane state backups protect different assets: preserve both the SQLite state and its master key as described in [state backup and recovery](/docs/operations/#control-plane-state-backup-and-recovery).
Agents expose member metrics for control-plane collection, history and authenticated [Prometheus](https://prometheus.io/) export. Persisted events feed live UI/API updates; the live event stream is an acceleration, not the persistence layer. See [Monitoring and events](/docs/monitoring/) and [API pagination and events](/docs/api/#pagination-and-events).
## Failure domains
[Section titled “Failure domains”](#failure-domains)
The control plane is single-replica. Persistent volumes are node-local on current providers. Zerops does not guarantee host separation; Dokploy and Nomad report authoritative node placement but cannot establish independent racks or physical hosts beneath the scheduler. Engine replication cannot compensate for every shared-host failure. See [Limitations](/docs/limitations/) before selecting availability and recovery objectives.
## Engine drivers
[Section titled “Engine drivers”](#engine-drivers)
The code calls an engine driver `engine.Provider`; the API and CLI retain `spec.provider`, `/api/v1/providers` and `--provider` for engine selection. Engine drivers are compiled into Karakoram, not dynamically installable plugins. Each owns its version catalog, validation/defaults, topology, health observation, lifecycle checkpoints, access APIs and backup semantics. Shared interfaces handle infrastructure, namespaces, secrets, operations and monitoring; capability metadata gates supported UI/API actions. A cluster’s engine driver is immutable.
Optional auxiliary services use ordered `ComponentDef`/`ComponentSet` declarations with per-component replicas and resources in `spec.components.extra`. Shared stateless reconciliation handles enable/disable stages through engine callbacks; credentials, application readiness, resizing and stateful safety remain engine-owned. Elasticsearch uses this for [Kibana](https://www.elastic.co/kibana), [Logstash](https://www.elastic.co/logstash) and [APM](https://www.elastic.co/observability/application-performance-monitoring); PostgreSQL for [ProxySQL](https://proxysql.com/); Supabase for its modular native services. See [ADR 0031](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0031-engine-component-model.md).
| Engine | Engine driver | Runtime and operational details |
| ----------------------- | --------------- | -------------------------------------------------------------------------------------------------------- |
| MongoDB | `psmdb` | [Overview](/docs/mongodb/) |
| RabbitMQ | `rabbitmq` | [Overview](/docs/rabbitmq/) |
| Elasticsearch | `elasticsearch` | [Overview](/docs/elasticsearch/) |
| PostgreSQL | `postgres` | [Percona, extension profiles, Patroni/etcd HA, Citus, ProxySQL and pgBackRest](/docs/postgres/) |
| Supabase | `supabase` | [PostgreSQL core plus modular native services](/docs/supabase/) |
| Keycloak | `keycloak` | [Native identity service with a managed PostgreSQL binding](/docs/keycloak/) |
| OpenBao | `openbao` | [Native secrets clusters, Raft recovery and leased database credentials](/docs/openbao/) |
| Temporal | `temporal` | [Stateless workflow roles on bound PostgreSQL](/docs/temporal/); requires the PostgreSQL binding service |
| MySQL | `mysql` | [Percona Server, Group Replication, ProxySQL and XtraBackup](/docs/mysql/) |
| Percona XtraDB Cluster | `pxc` | [Galera, quorum recovery and separate backup provenance](/docs/mysql/) |
| Neo4j Community | `neo4j` | [TLS-only singleton, native administrative users and offline dumps](/docs/neo4j/) |
| Weaviate | `weaviate` | [Native lifecycle, access management and recovery](/docs/weaviate/) |
Supabase wraps the PostgreSQL core rather than duplicating database lifecycle. Internal services use a protocol-transparent leader forwarder rather than ProxySQL; the client proxy retains explicit protocol limits. Native service availability comes from an embedded, checksum-pinned lock with operator mirror precedence, not floating downloads. See [ADRs 0034–0036](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/README.md) and [native artifacts](/docs/native-artifacts/).
See [ADR 0005](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0005-engine-provider-seam.md) for the engine driver seam and [Contributing](/docs/contributing/) for code layout and development workflows.
# Backup drills
> A successful backup is not proof that it can be restored. Backup drills exercise provisioning, native restore and readiness without modifying the source…
A successful backup is not proof that it can be restored. Backup drills exercise provisioning, native restore and readiness without modifying the source cluster. A successful drill proves the recorded checks, not application correctness or a guaranteed recovery time. Engine backups and [control-plane state backups](/docs/control-plane-backup/) protect different assets.
This Ops feature is **verified offline** (tests without a live provider). Qualify provisioning, native recovery and cleanup against your selected provider before relying on scheduled drills.
## Lifecycle and safety
[Section titled “Lifecycle and safety”](#lifecycle-and-safety)
A manual or scheduled drill selects a successful eligible backup and creates a disposable target in the **source namespace**, named `dr` followed by ten lowercase letters/digits. The target copies the source spec and resource sizes, but strips backup schedules, drill policy and public exposure, and disables monitoring. It uses the normal validated cluster-create path and provider deploy admission, including per-project caps. Server-owned `managedBy` identifies its source and exact drill ID; users cannot supply this marker. Managed targets are excluded from drill scheduling and automatic agent rollout.
Reports progress through `Pending`, `Provisioning`, `Restoring`, `Verifying`, `CleaningUp`, then `Succeeded` or `Failed`. Verification requires both cluster Ready and successful native restore inventory ([RabbitMQ](https://www.rabbitmq.com/) confirms definitions import). The report is persisted before cleanup. Cleanup uses ordinary cluster deletion with `keepVolumes=false` and `keepBackups=false`; it never deletes a target without matching drill ownership. Inspect cleanup phase and orphaned-volume inventory even after verification succeeds. The last 50 reports per source are retained after target deletion.
Deleting the source does not delete its drill reports or active cleanup bookkeeping. The controller records source disappearance as a failure for unfinished verification and continues ownership-fenced target cleanup; a drill already cleaning up preserves its recorded outcome. Reports remain retained after source deletion, subject to the same 50 completed reports per-source limit. Managed targets with missing reports remain protected from user mutation because completion cannot be established.
Cleanup is bounded to three delete-admission attempts and 30 minutes; exhausting either bound records Failed cleanup and orphaned-volume hostnames for operator review.
**Resource and cost:** every drill creates a real cluster with the source’s resources and uses backup bandwidth and storage. Allow spare quota/capacity and budget for its lifetime; admission may delay work. Installation-wide execution defaults to **one** drill; configure `KARAKORAM_BACKUP_DRILL_CONCURRENCY` on the control plane to an integer **1–4**. Excess drills remain queued.
Invalid concurrency values fail control-plane startup. Queued `Pending` time does not consume the drill’s server-side timeout.
## Scheduled policy
[Section titled “Scheduled policy”](#scheduled-policy)
Policy lives at `spec.backup.drill`:
```yaml
spec:
backup:
drill:
enabled: true
schedule: '0 4 * * 0'
backupSchedule: nightly
timeout: 6h
```
| Field | Rule |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `enabled` | Enables automatic runs; omit the policy or set false to disable scheduling. Manual runs remain separate. |
| `schedule` | Five-field cron in **UTC**, required for enabled scheduling. |
| `backupSchedule` | Optional source backup-schedule name; only successful backups from that schedule are eligible. Omit for any eligible successful backup. |
| `timeout` | Go duration, default `6h`, inclusive range `30m`–`48h`; bounds server-side execution from provisioning start, excluding the pending queue. |
Configure an existing cluster with a merge patch (other backup policy fields remain unchanged):
```sh
karactl -n production clusters patch orders -f drill-policy.yaml
# Or use convenience flags:
karactl -n production clusters patch orders --drill-enabled \
--drill-schedule '0 4 * * 0' --drill-backup-schedule nightly --drill-timeout 6h
```
The same four flags are available on cluster creation. Explicit flags update only their policy fields; `--drill-enabled=false` disables future scheduled drills. A request file remains authoritative over convenience flags. Disabling scheduling does not cancel accepted work.
## Manual runs and reports
[Section titled “Manual runs and reports”](#manual-runs-and-reports)
```sh
karactl -n production backup-drills run orders
karactl -n production backup-drills run orders --backup nightly-20261021 --wait --timeout 6h
karactl -n production backup-drills list orders --limit 20
karactl -n production backup-drills get orders
karactl -n production backup-drills get orders --wait --timeout 6h
karactl -n production backup-drills get orders -o json
```
Without `--backup`, the server chooses the latest successful eligible backup. Run returns the accepted report immediately unless `--wait` is supplied. List supports `--limit` and `--cursor`. Get includes checks, limitations, error, cleanup phase and orphaned volumes; JSON/YAML retain all structured fields. `--wait` polls this report through terminal status, exits nonzero for Failed, and defaults to a six-hour client timeout. Timeout/interruption does not cancel server work: use the printed `get ... --wait` command without creating another drill. The client timeout is independent of policy timeout.
API equivalents are `POST /clusters/{cluster}/backup-drills` with optional `{ "backup": "name" }`, `GET /clusters/{cluster}/backup-drills`, and `GET /clusters/{cluster}/backup-drills/{drill}` under `/api/v1`, using the namespace query parameter. **RBAC:** run requires `restores:create` on the source cluster; list/get/poll require `restores:read`. Waiting therefore needs both permissions. Policy changes use normal cluster create/update authorization.
### Timing fields
[Section titled “Timing fields”](#timing-fields)
* `createdAt`: report acceptance; `startedAt`: execution start after queuing.
* `readyAt`: restored target ready; `completedAt`: drill completion.
* `restoreDurationSeconds`: native restore `StartedAt` to `CompletedAt` when available, otherwise provisioning start to target readiness. It can include provisioning in the fallback case.
* `totalDurationSeconds`: overall recorded execution duration, distinct from restore duration.
* `dataAgeSeconds`: drill start minus backup `LastWriteTime`, falling back to backup completion. This is the age of the recovery data, not a promise about future recovery point objectives.
Missing timing fields mean the milestone or measurement is not yet available, not a zero-duration restore.
## Engine scope and limitations
[Section titled “Engine scope and limitations”](#engine-scope-and-limitations)
| Engine | Drill scope |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| MongoDB / PSMDB | Restore the selected supported PBM backup and confirm native inventory and cluster readiness; not an application query/assertion suite. |
| RabbitMQ | **Definitions only, no messages.** Confirms definitions import; cannot prove recovery of queued messages. The report explicitly records `definitions only, no messages`. |
| Elasticsearch | Restore the selected native snapshot and confirm inventory/readiness; not an application indexing or query benchmark. |
| PostgreSQL | Restore the selected pgBackRest backup and confirm inventory/readiness; this is backup restore, not an arbitrary PITR exercise. |
| Supabase | PostgreSQL backup restore scope; not proof of recovery of every external integration or non-database asset. |
Drills explicitly reject [Keycloak](https://www.keycloak.org/), [OpenBao](https://openbao.org/), [Temporal](https://temporal.io/), [MySQL](https://www.oracle.com/mysql/), [PXC](https://www.percona.com/mysql/software/percona-xtradb-cluster), [Neo4j](https://neo4j.com/), [Weaviate](https://weaviate.io/) and unknown engine drivers before provisioning. Their isolated clone/restore drill contracts are not implemented. In particular, a drill must not restore a binding-backed service’s shared [PostgreSQL](https://www.postgresql.org/) source in place. A persisted unsupported drill records a clear failure without creating a target; ordinary engine backup and restore support does not imply drill support.
See [engine backup capabilities](/docs/backups/#engine-recovery-capabilities) for backup types and recovery boundaries.
## Events and notifications
[Section titled “Events and notifications”](#events-and-notifications)
Success emits Normal `BackupDrillSucceeded`; failure or timeout emits Warning `BackupDrillFailed` on the source cluster. The failure notification category is **`backup-drill-failed`**. Configure [notification channels](/docs/notifications/) with that category (or all categories), and inspect the durable report for checks and cleanup details. Cleanup/orphaned-volume state remains visible independently of verification results.
# Backups and storage
> Register S3 destinations, manage backup inventory and schedules, and recover into existing or new clusters.
Register [S3](https://aws.amazon.com/s3/) destinations, manage backup inventory and schedules, and recover into existing or new clusters.
Use [backup drills](/docs/backup-drills/) to restore successful backups into disposable same-namespace clusters, record verification and timing, and clean up the target. Drills consume real infrastructure; they do not replace application-level recovery testing.
## Engine recovery capabilities
[Section titled “Engine recovery capabilities”](#engine-recovery-capabilities)
| Engine | Backup types and scope | PITR | Backup data path |
| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------- |
| [MongoDB](/docs/mongodb/backups/) | PBM `logical` (CLI default) / `physical`; incremental type not exposed; providerless legacy records belong here | Yes | Members write directly to S3 |
| [RabbitMQ](/docs/rabbitmq/backups/) | `definitions` only; **no queued messages**; restore merges definitions | No | Control plane exports through Management API, then compresses/uploads to S3 |
| [Elasticsearch](/docs/elasticsearch/snapshots/) | Native `snapshot` | No | Nodes write directly to S3 |
| [Weaviate](/docs/weaviate/backups/) | Native `full` via `backup-s3`; topology-preserving recovery | No | Members write directly to the cluster’s bound S3 storage |
| [PostgreSQL / Supabase](/docs/postgres/backups/) | pgBackRest `full`, `diff`, `incr`; PostgreSQL data only | When enabled | Current primary writes backups and WAL directly to S3 |
| [OpenBao](/docs/openbao/backups/) | Native `raft-snapshot`, up to 4 GiB; requires source seal material for recovery | No | Control plane streams native HTTPS snapshot to S3; unavailable with PostgreSQL storage |
| [Temporal](/docs/temporal/operations/#backups) | Source PostgreSQL `full`, `diff`, `incr` through the binding plus a manifest in Temporal cluster state | No Temporal timestamp restore | PostgreSQL source owns archives/retention; deleting a Temporal record does not delete source archives |
| [Neo4j Community](/docs/neo4j/backups/) | Offline `dump` of `system` and `neo4j`; **full outage**; schedules require `acceptBackupDowntime: true` | No | Member streams both dumps to S3, manifest last |
Backup payload transfer to [S3](https://aws.amazon.com/s3/) does not use the management relay; remote administration requests may. Control-plane storage verification does not prove member connectivity. Engine backups and [control-plane state backups](/docs/operations/#control-plane-state-backup-and-recovery) protect different assets. [Zerops](https://zerops.io/) live volume archives are not consistent engine backups.
## Register S3 storage
[Section titled “Register S3 storage”](#register-s3-storage)
Create the bucket separately. In **Settings → Backup storages**, register an existing [S3](https://aws.amazon.com/s3/)-compatible destination in the cluster’s namespace. Cluster backup settings and schedules resolve storage names within that namespace.
| Request field | Value / rule |
| -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | New registration: 2–63 lowercase letters/digits/hyphens, starting with a letter; trailing hyphen allowed |
| `type` | `s3` |
| `endpointURL` | HTTP(S) origin, e.g. `https://storage-.zerops.io`; optional port and root `/` allowed, no endpoint path prefix |
| `region` | Required S3 signing region |
| `bucket` | Existing bucket name |
| `prefix`, `description` | Optional |
| `forcePathStyle` | Defaults to `true`; `false` prefers virtual hosting for DNS endpoints, with automatic path-style fallback for IP endpoints and dotted HTTPS buckets |
| `verifyTLS` | Defaults to `true`; keep enabled with HTTPS |
| `credentials.accessKeyId`, `credentials.secretAccessKey` | Required credential pair; bucket listing plus object read/write/delete access scoped to the configured prefix |
Storage prefixes ignore leading and trailing slashes (`/` alone means the bucket root), while new registrations reject empty inner components, `.`/`..` components and backslashes; existing bindings retain their engine’s historical namespace mapping, shared by writers, inventory and cleanup.
Prepare an API-shaped YAML/JSON file using these fields; protect it with owner-only permissions and never commit or trace credentials.
```sh
karactl -n storages create -f /private/path/storage.yaml
karactl -n storages verify archive
karactl -n storages get archive
karactl -n storages set archive -f /private/path/storage-update.yaml
```
Set replaces the full configuration: include `type`, `endpointURL`, `region`, `bucket`, `forcePathStyle`, `verifyTLS` and any retained optional fields; omit `name`. Omit `credentials` entirely to retain the sealed pair, or supply both keys to rotate. Verify before revoking old credentials. Reads expose `credentialsConfigured`, never keys. Existing legacy names remain editable/readable/deletable.
Registration does not verify access. Verification performs temporary put/get/delete under the prefix; **HTTP 200 can contain `verified: false`**. The latest result/check time is persisted, not a history. Connection/credential edits clear it; description-only edits preserve it. UI **Options** describe transport, not successful **Verification**. Verification status records only the last explicit `storages verify` result and the control-plane clock at that time; backups and restarts do not refresh it, so after correcting host clocks or credentials, run `storages verify` again.
Inventories and restore-absence checks also require complete `s3:ListBucket` access for the configured prefix; a successful put/get/delete verification does not establish listing permission. Preserve `s3:GetObject`/HEAD access to existing metadata. Some S3 policies return HEAD403 rather than404 for a missing key when listing is restricted. A403 alone never proves absence: restore fencing requires an independently successful complete listing and all other native checks. Denied or partial listings retain the restore holds; repair the scoped policy rather than bypassing the fence.
Storage registration accepts DNS hosts, IPv4 literals and bracketed unscoped IPv6 literals, with an explicit `http://` or `https://` scheme and an optional port from 1–65535. Bare `host` / `host:port`, scoped IPv6, dotted IPv4-mapped IPv6 literals, URL credentials, query strings and fragments are rejected. For mapped IPv6, use hex-only notation or the equivalent IPv4 literal. Endpoint mount paths such as `/s3` are **not supported**: [pgBackRest](https://pgbackrest.org/) does not preserve the endpoint URL’s path. Supply an origin-level S3 endpoint instead. `prefix` scopes **object keys**, not an HTTP reverse-proxy mount, so copying `/s3` into `prefix` does not repair a mounted S3 service.
With `forcePathStyle: true`, requests use `https://endpoint/bucket/key`. With `false`, ordinary DNS endpoints use `https://bucket.endpoint/key`, requiring bucket-subdomain DNS and a matching certificate. Literal IPv4/IPv6 endpoints always use path style; dotted bucket names over HTTPS also use path style to avoid wildcard-certificate mismatch. HTTP dotted buckets may use virtual hosting. The [Go](https://go.dev/) object client, [PBM](https://docs.percona.com/percona-backup-mongodb/latest/), pgBackRest and [Supabase](https://supabase.com/) Storage share these rules; [Elasticsearch](https://www.elastic.co/elasticsearch) intentionally always uses path style. TLS validation still requires a certificate matching the effective hostname (or literal IP), and HTTP remains plaintext.
Old registrations with unsupported endpoint shapes remain readable and deletable. Their get/list response exposes `verification.verified: false`, `failureReason: endpoint_not_allowed` and shape-specific remediation even if a historical check succeeded. This is a current configuration diagnosis, not a new network probe. `storages verify` returns and persists the same actionable failure without making a request; the CLI exits nonzero. Repair the origin with `storages set`, then verify again. No stored endpoint, bucket, object key or retained-backup location is silently rewritten.
### Endpoint verification policy
[Section titled “Endpoint verification policy”](#endpoint-verification-policy)
Use an HTTP(S) origin: no credentials, non-root path, query, fragment or redirects. Public global-unicast destinations on 80/443 are permitted by default. For private S3 or another port, set `KARAKORAM_BACKUP_ENDPOINT_ALLOWLIST` on the **control-plane service**, then restart/redeploy. Entries are comma-separated exact `host` or `host:port`; host-only permits 80/443, host:port only that port. Do not broadly allowlist project services. The built-in exception is exactly `storage-prg1.zerops.io` on 80/443, including its private resolution; other regional private endpoints require configuration.
Loopback, link-local/metadata, unspecified/non-global destinations and `100.64.0.0/10` remain blocked even when allowlisted. Every connection validates all DNS answers and dials a checked numeric address. Proxies and redirects are disabled; the probe deadline is 30 seconds. Keep HTTPS/certificate verification enabled; plaintext HTTP is not made safe by a successful probe. This policy also guards control-plane retained-object cleanup, not all engine egress.
| `failureReason` | Check |
| ------------------------ | -------------------------------------------------------------------------- |
| `access_denied` | Credentials, read/write/delete permissions, signing region |
| `bucket_not_found` | Bucket, regional endpoint |
| `clock_skew` | Synchronize the control-plane host clock, then run `storages verify` again |
| `endpoint_not_allowed` | Origin, allowlist, DNS addresses, port, redirects |
| `tls_error` | Hostname/certificate chain; do not hide it by disabling verification |
| `unreachable`, `timeout` | Control-plane DNS/network/firewall or storage responsiveness |
| `unknown` | Configuration and storage-service diagnostics; do not publish credentials |
## On-demand backups and inventory
[Section titled “On-demand backups and inventory”](#on-demand-backups-and-inventory)
Register storage first, then use **Cluster → Backups → Backup now** or:
```sh
karactl -n backups create beforechange --type --storage archive
karactl -n backups list
karactl -n backups get beforechange
karactl operations wait
```
Select an explicit type from the engine table. Acceptance is not completion: follow **View operation**, then inspect backup phase (`Pending`, `Running`, `Succeeded`, `Failed`), timestamps, storage, type, native location and errors. Only `Succeeded` backups are recovery bases. Inventory distinguishes manual/scheduled/recovery-triggered work. A manual request needs a formed, active cluster with its current generation observed; unconverged/suspended/deleting clusters return `409`. See [operation and health semantics](/docs/clusters/#operations-and-status).
### Protect backups used by restores and clones
[Section titled “Protect backups used by restores and clones”](#protect-backups-used-by-restores-and-clones)
An accepted restore or create-from-backup reserves its source backup while the operation is in flight, including the clone’s initial member-provisioning phase. Deleting that backup (live or retained) returns `409 invalid_state` naming the dependent operation and target cluster. [PostgreSQL](https://www.postgresql.org/) expiration also protects referenced differential/incremental descendants of the requested set. Deleting the source cluster without retaining backups likewise returns `409`; wait for the dependent operation to complete or cancel it safely, or delete the source with `keepBackups=true` (`karactl clusters delete --keep-backups`). Retention sweeps cannot bypass this fence.
PITR restores/clones select their base set during execution. Until the dependent operation is terminal, Karakoram conservatively refuses deletion of any backup on that source and refuses source-cluster deletion even with `keepBackups=true`: retaining objects alone does not preserve the live source required by PITR. If source deletion or backup expiration is accepted first, PITR admission is refused until that destructive intent finishes and an available recovery point can be selected. Likewise, a named PostgreSQL child restore/clone is refused while any of its full/differential ancestors is already Deleting.
Admission and deletion intent are serialized: if deletion was accepted first, a new restore/create referencing that backup is refused instead. Retained, successful backups remain usable when the source was deleted with backups kept.
If inventory or the stored snapshot is authoritatively absent (for example, objects were removed outside Karakoram), the restore/create operation reaches `Failed` with source-backup guidance rather than staying `Restoring`. Inspect the operation and restore error, select another verified successful backup, and retry the restore or create a new target. Storage authorization, transport, timeout and service errors are not proof of absence and retain existing bounded retry/compensation behavior; do not delete restore holds or receipts manually.
## Schedules and count retention
[Section titled “Schedules and count retention”](#schedules-and-count-retention)
Configure **Backups** policy or patch `spec.backup`: `storage` selects the primary destination; `schedules` entries contain `name`, `cron`, engine driver-supported `type`, optional override `storage`, `retainCount` (at least 1), and `enabled`.
```sh
karactl -n clusters patch -f backup-settings.yaml --wait
```
Cron uses five fields in **UTC**. Retention keeps the newest successful backups **of that schedule only**, not manual/failed/pending backups. Excess backups are deleted through durable operations; this is count retention, not age/GFS or a guaranteed recovery duration. A patch replaces the schedules array: include every entry to retain. Set `enabled: false` to stop new runs.
Each retention sweep is fenced to the cluster incarnation and policy generation it inspected. A policy edit or source-cluster replacement abandons stale deletion candidates; a later sweep evaluates the current policy.
Unchanged enabled schedules retain a durable last-admitted occurrence. A failed enqueue does not consume that occurrence: the scheduler retries after storage recovers, including across control-plane restarts. Backup inventory, operation and cursor advancement commit atomically. Catch-up admits at most one overdue occurrence per schedule per reload (normally every 30 seconds) or cron callback, so recovery can queue delayed backups rather than running them at their original times. A new or edited schedule starts from its first successful observation; there is no retroactive guarantee for time before that cursor exists. Disabling and re-enabling a schedule starts a new cursor epoch once the disabled policy has been observed. Backward clock corrections do not repeat admitted occurrences. Engine-specific continuity requirements remain in the linked recovery guides.
## Restore and clone
[Section titled “Restore and clone”](#restore-and-clone)
**Warning:** restore mutates target state and can affect application access. Quiesce writers, plan downtime and preserve a recovery copy before in-place recovery; exact overwrite/merge semantics depend on the engine.
While an in-place restore runs, cluster `phase` is `Restoring` and the `Ready` condition is `False` with reason `Restoring`, even if a native health probe can still reach the old database processes. This applies to every engine, including definition-merge restores. Availability observations continue but cannot certify restore completion. A terminal restore operation leaves `Ready=False` with reason `RestoreFinished`; ordinary reconciliation must then observe database health before publishing `phase=Ready` and `Ready=True`. A successful operation alone is not an application-availability guarantee.
1. Select a successful compatible backup, or a engine driver-supported observed recovery time. Engine driver must match; topology/version/identity checks are engine-specific. Providerless legacy records are not engine-neutral artifacts; see the engine table.
2. Restore into an existing target, or clone into an independent new cluster without stopping the source:
```sh
karactl -n restores create recovery --backup beforechange --yes
karactl -n restores create clone --backup beforechange --into-new
karactl -n restores list
```
3. Follow the operation and destination restore/cluster records; verify actual recovered data and application authentication before reopening traffic. Acceptance is not a recovery drill.
Explicit cloning uses a complete cluster-create request with immutable `spec.dataSource` containing `cluster` and `backup` (or supported `pitrTime`), plus optional `namespace`. After initial recovery, dataSource is provenance, not a continuing dependency.
The initial data source remains a durable requirement across control-plane restarts and desired-generation changes. `Ready` requires positive native restore-completion evidence, not merely healthy database processes. An aborted initial restore leaves the target `Failed` with `RestoreIncomplete`; it never silently falls back to an empty database. Delete the target and create a new one from the preserved source to abandon that failed attempt. Failure withdraws readiness, not ownership of safe compensation: already-running native cleanup and any retained frozen [PostgreSQL](https://www.postgresql.org/) recovery journal continue to release their process/maintenance/router holds. Their failure observation cannot block that recovery, authorize a fresh restore, or turn the failed target Ready.
A create operation can also fail only its observation deadline/retry budget while an already-claimed native restore continues without abort or teardown. That is not a restore abort: positive native completion can make the target Ready, with an `InitialRestoreCompletedAfterTrackerFailure` audit event; the original create operation remains Failed. By contrast, explicit cancellation, native failure or partial/cancelled outcome, compensation, teardown, or failure before native claim durably aborts the initial restore. Such a target stays Failed/non-Ready even if a late native success appears. This evidence survives health refresh and suspension independently of the native success marker. The native claim is the first durable, incarnation-scoped bootstrap/import receipt, not the later creation of a bookkeeping Restore operation. Its original time is preserved across retries and restart. PostgreSQL/[Supabase](https://supabase.com/) can therefore finish an already-claimed bootstrap if the create tracker expires before DATA members join and the initial Restore record is enqueued. This continuation cannot change the source or reopen conflicting edit admission.
Retry diagnostics on a Running restore (for example a polling connection error) are not terminal failure evidence. They remain retryable; only actual terminal failure/cancellation or native compensation evidence records an abort, and positive completion clears the stale retry diagnostic.
Before native recovery is claimed, placement-only corrections (component replicas/resources) amend the **same create operation**, preserving the recovery source and its audit history. This permits correcting an unplaceable proxy without cancelling recovery. Source/target point, version, credentials, topology kind and other changes are refused with `409 invalid_state`. Once native recovery has been claimed, conflicting edits/actions are refused with `409` naming the owning operation until recovery completes. Target deletion remains allowed. Source backup deletion stays fenced until positive initial restore completion, including after a failed or cancelled create; deleting the target releases that dependency.
PostgreSQL and Supabase initial recovery joins and recovers DATA members without waiting for proxy deployment/placement. An unplaceable proxy therefore does not keep the recovery mutation fence closed indefinitely: after native recovery completes, an ordinary proxy placement/replica PATCH is admitted even while the create tracker is still waiting. Desired proxies remain an independent `Ready` requirement; successful data recovery alone does not make an unplaced proxy healthy. Supabase optional-service placement likewise does not gate core recovery.
Upgrade behavior: successful pre-marker PostgreSQL/Supabase and [MongoDB](https://www.mongodb.com/) clones are migrated only from matching completed native restore receipts; an old `Cancelled` create tracker caused by PATCH is not itself a failed restore. Matching native work already in flight is allowed to finish. [Elasticsearch](https://www.elastic.co/elasticsearch)’s existing completed marker and [RabbitMQ](https://www.rabbitmq.com/)’s successful definitions-import marker remain valid; an Elasticsearch accepted-only native submission is not promoted to success.
A legacy superseded create with no restore record or completion marker is different: the target may already contain user writes. It reports `Ready=False / LegacyRestoreUnproven`, naming the old create operation and the delete-and-recreate escape, without replaying initial restoration or authorizing empty initialization. Normal runtime self-healing, managed credentials, own backup scheduling and data-plane service/proxy exposure continue; withdrawing Ready is not a request to stop or unregister the database. Its initial-source obligation remains unproven and source deletion remains fenced. To recover the requested source data, delete that target and create a new target from the preserved source; do not discard its live data or force an implicit replay.
Own backup operations remain queueable because they do not supersede the restore owner and share the serialized native worker; this also permits the required post-restore fresh-base backup without reopening conflicting mutation admission. [OpenBao](https://openbao.org/) foreign/deleted-source recovery additionally needs the source’s unseal/recovery material. The top-level write-only `openbaoUnsealShares` field is accepted in restore and cluster-create requests, not persisted in desired spec. See [Raft restore, clone and supplied-share examples](/docs/openbao/backups/); a snapshot alone cannot recover a sealed server.
### Cross-namespace recovery
[Section titled “Cross-namespace recovery”](#cross-namespace-recovery)
Global `-n` selects the **target**; `--source-namespace` selects the source and defaults to the target when omitted:
```sh
karactl -n dev restores create orderscopy recovery \
--source-namespace prod --source-cluster orders --backup baseline --yes
```
New-cluster requests use `spec.dataSource.namespace`; in-place REST requests use `sourceNamespace`. Recovery reads the source backup’s recorded storage; ensure the applicable data path can reach it. Target backup settings must use target-local storage. The live-source `--into-new` convenience clone omits copied backup settings across namespaces; configure them afterward.
### PostgreSQL and Supabase recovery
[Section titled “PostgreSQL and Supabase recovery”](#postgresql-and-supabase-recovery)
See the focused [PostgreSQL recovery guide](/docs/postgres/backups/) and [Supabase recovery boundaries](/docs/supabase/operations/#recovery-boundaries). The shared rules below apply to both engine drivers.
Use an explicit `--type full`, `--type diff` or `--type incr`; the generic CLI default `logical` belongs to MongoDB. A differential requires an existing full; an incremental requires an existing consistent backup chain. The registered storage’s endpoint scheme, path-style addressing and TLS verification policy apply on every potential primary. Repository paths are isolated by namespace and cluster incarnation: `/postgres//-`, with stanza `kk`. Keep storage credentials and the control-plane master key.
[pgBackRest](https://pgbackrest.org/) automatic expiration is disabled. The Karakoram scheduler controls retention through `delete-backup` operations, but defers the newest full until a newer full exists and preserves ancestors of retained sets across schedules and manual requests. Skipped sets are reconsidered on later passes; actual counts can exceed the configured limit while dependencies require them. Explicitly deleting a full **also deletes its dependent differential and incremental sets**, including records from other schedules or manual requests; deleting a differential also deletes incrementals that reference it. The newest full cannot be deleted while the cluster is live. Rejected or failed expiration fails its operation and releases the queue for subsequent backups. WAL preceding the oldest remaining full chain is pruned by expiration. After source deletion, retained-backup deletion uses the recorded repository and native backup dependency metadata, rewrites checksummed pgBackRest inventory, and removes the whole isolated repository when its last backup is deleted.
Enable `spec.backup.pitr.enabled` with a default storage. `oplogSpanMinutes` controls PostgreSQL `archive_timeout` (one minute when omitted); this bounds WAL switching during low activity, not an exact commit-time recovery guarantee. The reported conservative interval starts at the oldest successful consistent backup’s stop time and ends at the current primary’s last successful archive completion time. No interval is advertised when PITR is disabled or archiving is failing. Archive completion is not a precise last-commit timestamp; missing objects or externally modified repositories can still make recovery fail.
In-place restore is destructive and offline. Single-member recovery stops PostgreSQL, performs a delta restore, starts the server and waits for promotion. HA recovery fences and stops every [Patroni](https://patroni.readthedocs.io/)/PostgreSQL member, resets only that cluster’s DCS scope, restores a seed through Patroni custom bootstrap, and rebuilds the remaining members before completing. Recovery continues archiving to the target’s stanza on a new timeline. When restoring from a different source repository, archiving remains disabled until the target repository configuration is published and the bootstrap archive override is removed on every member. For a different-source restore into an existing target repository, stanza-upgrade appends the restored system-ID history while retaining the target’s prior backup sets; it does not overwrite them with the source repository’s inventory. When PITR is enabled a new full backup is queued afterward; recovery is not protected by that new base until it succeeds. Recovery adopts the backup’s sealed system-role credentials after all existing members are stopped, before restarting the restored seed. This also applies to same-cluster backups taken before a system-password rotation; reveal connection credentials again after recovery rather than reusing an earlier password.
A clone’s `dataSource` may name a backup or a PITR time in the same or another namespace. Source and target engine drivers must match **exactly**: `postgres` cannot consume a `supabase` backup or vice versa. PostgreSQL major versions must match; source and destination topologies may differ. Bootstrap reads the source repository with archiving disabled, then switches to the target’s own repository and queues a full backup. Target-local backup storage must be configured in the clone request to establish that independent recovery history. Retained backup clones use the snapshot’s sealed system credentials, not a deleted source’s live credential scope. Supabase backups protect PostgreSQL data, not objects stored by its separate Storage service.
## Permissions
[Section titled “Permissions”](#permissions)
| Action | Required resource/action on namespace-qualified object |
| ------------------------------------------- | ------------------------------------------------------------------------ |
| Register/read/edit or verify/delete storage | `backup-storages/create`, `read`, `update`, `delete` respectively |
| Create/read/delete backups | `backups/create`, `read`, `delete` on source cluster |
| Save backup policy | `clusters/update` |
| Restore into existing / clone into new | Target `restores/create` / `clusters/create`, plus source `backups/read` |
| Cross-namespace recovery | Additionally source `backups/export`; export alone is insufficient |
## Delete or preserve recovery objects
[Section titled “Delete or preserve recovery objects”](#delete-or-preserve-recovery-objects)
Backup deletion is destructive; live-cluster cleanup is asynchronous and engine driver-specific. Active matching work can block deletion until cancelled/terminal. Use `backups delete --yes` and follow its outcome. See engine guides for shared-object and continuity rules, especially [MongoDB backup-data cleanup](/docs/mongodb/backups/).
If a backup-deletion operation fails, expires, or is cancelled before completion, its inventory leaves `Deleting` for `Failed` with the cleanup error and its original remote identity preserved. It is not a usable restore base. Repair the reported dependency and submit deletion again; the new operation safely retries cleanup of the same target instead of leaving an unretryable `Deleting` record. Startup also repairs historical inventory stranded by terminal deletion operations.
Suspended clusters still process backup-deletion operations without resuming ordinary runtime reconciliation. Object-store cleanup can therefore finish while runtimes remain absent. A engine driver whose native cleanup requires a running member can report that dependency failure; preserve the recorded target, resume/repair the dependency where needed, and retry deletion rather than removing inventory by hand.
Namespace/provider failures during suspended cleanup are reported in conditions and messages without clearing `Suspended`; only an explicit resume or deletion changes that lifecycle intent. After source-cluster deletion, failed backups with a recorded remote location or [PostgreSQL](https://www.postgresql.org/) repository metadata retry through direct retained-object cleanup, never through an operation stranded on the absent cluster. Inventory is removed only after remote cleanup succeeds.
[Cluster deletion](/docs/clusters/#delete-or-suspend) intentionally preserves backups with `--keep-backups`, independently of volumes. MongoDB, [Elasticsearch](https://www.elastic.co/elasticsearch), [RabbitMQ](https://www.rabbitmq.com/) and PostgreSQL/[Supabase](https://supabase.com/) retire runtime writers and delete (or intentionally retain) volumes before control-plane artifact cleanup. Elasticsearch, RabbitMQ and PostgreSQL/Supabase retry artifact access failures within a durable two-minute cleanup window; restarts do not reset that window. If exact owned cleanup cannot finish safely, cluster removal completes with affected inventory retained and a `BackupArtifactsRetained` warning (`SnapshotsRetained` for Elasticsearch). Successful destinations are checkpointed and are not deleted again after restart. Read-only source repositories and other incarnations remain untouched. Dependency admission still returns `409` before destructive intent when another operation needs the backups. Inspect retained-artifact warnings and exact-prefix removal guidance rather than treating terminal cluster success as proof of empty storage. Retained `backups list/get/delete` use the original name/namespace and source `backups/read`/`delete`. Recovery after source deletion is engine driver-specific: use an explicit create file where supported, or restore into an existing target following the engine guide. `--into-new` needs a live source spec. Retained listing returns `404` after the deleted source’s final backup record is removed.
Preserve the state database, master key, storage configuration and credentials: [S3](https://aws.amazon.com/s3/) objects alone do not reconstruct lost control-plane state. Retained inventory recreates neither the source resource nor its volumes. Storage removal returns `409` while referenced by clusters, schedules or backup records; deleting its registration does not empty the bucket.
### When cluster deletion stays pending
[Section titled “When cluster deletion stays pending”](#when-cluster-deletion-stays-pending)
A wait timeout does not undo deletion. Inspect the same namespace’s cluster, operation and events; repair the reported token/relay, S3 credential/bucket or platform-process dependency. Do not erase state, recreate the name or delete volumes to hide pending cleanup. Observe the existing operation; retry only after failure/nonacceptance, keeping intentional retention choices. Keep storage and namespace management access until cleanup finishes. Check cluster absence (or `Suspended`), retained inventory and [orphaned volumes](/docs/operations/#orphaned-platform-volumes) separately; orphan disks can still consume capacity and incur charges.
# karactl CLI
> karactl manages Karakoram through its REST API, not directly through infrastructure providers.
`karactl` manages Karakoram through its REST API, not directly through infrastructure providers.
## Install
[Section titled “Install”](#install)
Follow [install karactl](/docs/getting-started/install/#install-karactl) for release downloads and checksum verification. Match the CLI release to your server; connecting an older binary does not add commands. From a compatible release checkout with [Go](https://go.dev/) 1.26+:
```sh
go build -o /tmp/karactl ./cmd/karactl
install -Dm755 /tmp/karactl "$HOME/.local/bin/karactl"
```
Put `$HOME/.local/bin` on `PATH`. `make build` alternatively produces `bin/karactl`.
## Login and contexts
[Section titled “Login and contexts”](#login-and-contexts)
```sh
karactl --server https://example.com --context production login --username admin
karactl --context production health
karactl --context production whoami
karactl config set-context production --namespace production
karactl --context production logout
```
Login prompts without echo on a terminal; otherwise it reads the first stdin line. Supply noninteractive passwords from a secret manager or protected file, never arguments/history. Complete any mandatory password change in the UI first. Login opens a temporary cookie session, creates an owner-scoped API token, saves the context as current, then closes the session. Re-login saves the replacement before best-effort revocation of the previous token; follow any revocation warning.
`config set-context NAME` saves explicitly supplied `--server`, `--token` and `--namespace` flags, preserving other saved values. It selects the context only if none is current. Replacing a token this way clears its saved token ID. There is no `contexts` command; use `--context NAME` to select one. Outside login, an explicitly selected missing context is rejected.
Contexts are stored at `~/.config/karakoram/config.yaml`, independent of XDG overrides, with file mode 0600 and newly created directory mode 0700. Treat this as a credential file.
| Selection | Precedence/behavior |
| ------------------------- | --------------------------------------------------------------------------------------------- |
| Server/token | Explicit flag → `KARAKORAM_SERVER` / `KARAKORAM_TOKEN` → saved context |
| Context | `--context` → saved current context → `default` |
| Object namespace | `-n` → context namespace → `default` |
| Cross-cluster collections | Without explicit `-n`, include all authorized namespaces, even if the context has a namespace |
Server URLs accept the origin or `/api/v1` suffix. For automation inject `KARAKORAM_SERVER` and `KARAKORAM_TOKEN` securely. Tokens are Karakoram credentials, not provider infrastructure credentials. Logout normally revokes the matching saved token before clearing credentials; `logout --local` leaves server authority active. With an overriding token, revoke other saved tokens explicitly by ID if necessary.
## Requests, output and waits
[Section titled “Requests, output and waits”](#requests-output-and-waits)
* `-o table` is the default; detail output uses sections and list output summarizes nested fields. `-o json` and `-o yaml` preserve complete structured responses.
* `-f FILE` accepts API-shaped YAML/JSON; `-f -` reads stdin. A supplied file defines the body instead of convenience body flags. Use protected files for secrets. Exceptions: `rbac policy set` takes raw CSV; engine-native role/key commands use their documented definitions.
* Cluster create files contain `{name, spec}`. `edit` replaces the complete `{spec}` and can open the configured editor without a file. `patch`, `scale` and `upgrade` files contain `{spec: {...}}` merge patches; arrays replace the previous array, not append to it.
* Complete request files reject explicitly supplied convenience body flags (usage error, exit 2), on every command accepting such a file: creation/source flags, scale resource/replica/topology/component flags, upgrade/FCV version, balancer settings and RBAC permission fields must go in the file rather than silently overriding or being ignored. Put initial cluster provenance in `spec.dataSource` (`cluster`, optional `namespace`, and exactly one of `backup` or `pitrTime`); put restore provenance in `source`/`sourceNamespace`. Defaults do not override file fields. Execution controls such as namespace/output selection, API query flags, `--dry-run` and cluster `--wait`/`--timeout` remain supported; `restores create --file FILE --into-new NAME` still clones using the source recorded in the file. [Elasticsearch](https://www.elastic.co/elasticsearch)-native role/API-key definition files retain their documented merge/conflict behavior.
* `--dry-run` previews `clusters create`, `edit`, `patch`, `scale`, `upgrade` and `restores create` on the server using the same request body and namespace, without applying changes. `restores create --into-new --dry-run` previews creation of the cloned cluster. Table output has changes (paths and compact JSON before → after), effects and warnings; `-o json`/`-o yaml` preserves the complete plan. A valid plan exits 0; API validation failures keep the normal error exit status. Dry runs never prompt for destructive confirmation (including `-f -`) and cannot be combined with `--wait`.
* Create convenience flags default to `psmdb`. An omitted/empty `--version` leaves version selection to the selected engine driver’s catalog. For presets/components, inspect `providers get PROVIDER -o json`, customize a preset into a complete `{name, spec}` file, then use `clusters create -f FILE`. There is no `--preset` flag. See [MongoDB](/docs/mongodb/), [RabbitMQ](/docs/rabbitmq/) and [Elasticsearch creation](/docs/elasticsearch/#create).
* Maintenance convenience flags on cluster spec commands: repeat `--maintenance-window 'CRON;DURATION'` and select `--maintenance-time-zone IANA_NAME` (default UTC). `--maintenance-window ''` clears windows. With `-f`, put `spec: {maintenance: {timeZone: UTC, windows: [{schedule: "0 2 * * 0", duration: 2h}]}}` in the file instead; request files remain authoritative.
* `--ignore-maintenance-window` on restart, rotate-tls, replace-member, members move, raise-fcv, edit, patch, scale, upgrade and upgrade-agents sends the explicit per-operation/per-generation override as an API query parameter. It can accompany `-f` on commands accepting request files; do not put this control in the request body. It does not bypass quorum or other safety checks. See [maintenance semantics and exemptions](/docs/operations/#maintenance-windows); `clusters get` and `clusters status` include the computed maintenance status.
* [PostgreSQL](https://www.postgresql.org/)/[Supabase](https://supabase.com/) creation and current artifact restrictions: [PostgreSQL](/docs/postgres/), [Supabase](/docs/supabase/). Use `full`, `diff` or `incr` explicitly for their backups; generic `logical` is not a PostgreSQL type.
* [MySQL and PXC](/docs/mysql/) are separate engine drivers with a shared `mysql` access command group. Their backups use `full` or `incremental`, and PITR is GTID/binlog based; see [MySQL backups](/docs/mysql/backups/).
* [Neo4j](https://neo4j.com/) Community creation, native administrative users and offline outages: [Neo4j](/docs/neo4j/). Always pass `--type dump` for Neo4j backups; schedules require `engineOptions.acceptBackupDowntime: true`.
* [Weaviate creation](/docs/weaviate/#create) supports `single` and `cluster`; use `--type full` for its backups and its native `weaviate users|roles` groups for application access.
* `clusters connection NAME` prints a password-free summary. `--reveal` explicitly retrieves sensitive engine-admin credentials and CA; do not use that identity for applications or log its output. `clusters ca NAME` exports only PEM to stdout, regardless of output format, but still requires `cluster-credentials:read`. Follow [MongoDB access](/docs/mongodb/access-and-tls/), [RabbitMQ access](/docs/rabbitmq/access-and-tls/) or [Elasticsearch access](/docs/elasticsearch/access-and-security/).
* Cluster mutation `--wait` reports acceptance/operation ID on stderr, then phase changes and a fifteen-second heartbeat. The default timeout is **30m**. It waits for the operation and observed cluster generation; an **Error or Degraded cluster fails the wait**. `clusters wait NAME` observes the current desired generation or deletion/suspension without resubmitting.
* `operations wait ID` succeeds on **Succeeded**, even when the cluster is Degraded; **Failed/Cancelled** fail. Operation success is not a health guarantee. See [operations and status](/docs/clusters/#operations-and-status).
* Observation retries cover transient connection/timeouts and HTTP **408/429/502/503/504**, with bounded backoff and `Retry-After` handling for 429/503. Invalid URLs, untrusted certificates and nonexistent DNS names fail immediately. Timeout/interruption never cancels server work. Keep the operation ID and use the printed non-mutating resume command.
* Backup/restore creation has no `--wait`: use the returned operation ID with `operations wait`, then inspect `backups get` / `restores get`. Namespace deletion has no durable Operation or `--wait`; poll `namespaces get` and lifecycle events. See [namespaces](/docs/namespaces/).
* `operations cancel ID` follows the operation’s persisted namespace, not the selected namespace. It needs cluster-update permission; acceptance can leave Running work in cooperative cleanup and never rolls back desired state. Unsafe/terminal states return 409. See [REST API](/docs/api/).
* `clusters logs` returns a bounded log tail: `--lines` defaults to **100**, range **1–10000**; `--process` selects an engine driver-supported child process or `agent`. Child-process tails come from the authenticated member API. `agent` reads the member runtime’s provider application logs, including supervisor diagnostics and forwarded child output, not a separate supervised process; availability and retention depend on the provider. **`--cursor` currently has no effect**; the server also ignores the API’s `since` input and returns no next cursor.
* `events watch` streams until interrupted; see [pagination and SSE](/docs/api/#pagination-and-events). `openapi` downloads raw server YAML without authentication. `completion` generates scripts without configuration/server access. `version` always reports client metadata; absent/unreachable server metadata does not itself fail the command, and lookup is limited to five seconds.
`karactl` has **no `--idempotency-key` flag and does not send `Idempotency-Key`**. For keyed mutation retries use HTTP or the [Go](https://go.dev/) client; see [safe retries](/docs/api/#safe-retries-with-idempotency-key). Do not repeat a mutation merely because waiting or receiving its response failed.
## Command groups
[Section titled “Command groups”](#command-groups)
This index follows `karactl --help`; key subcommands are illustrative, not an exhaustive flag reference. **All** means a shared interface subject to engine driver capabilities and RBAC.
| Group/command | Purpose | Key subcommands | Engine applicability |
| ----------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| `accounts` | Local control-plane accounts | list, get, create, set, delete, reset-password, change-password | All |
| `auth` | Advanced cookie/OIDC flows | refresh, oidc-login, oidc-callback | All |
| `backends` | Registered infrastructure kinds and capabilities | list | All |
| `backups` | Backup inventory and creation/deletion | create, list, get, delete, pitr-window | All; pitr-window for engine drivers advertising PITR |
| `backup-drills` | [Disposable restore verification and reports](/docs/backup-drills/) | list, get, run | All; RabbitMQ definitions only |
| `clusters` | Cluster lifecycle/observation | create, list, get, status, members, placement, connection, ca, logs, metrics, edit, patch, scale, upgrade, upgrade-agents, restart, retry-deployments, replace-member, resume, rotate-tls, delete, wait, balancer, raise-fcv | All; balancer/raise-fcv MongoDB only |
| `completion` | Generate shell scripts | bash, zsh, fish, powershell | Local |
| `config` | Save connection contexts | set-context | Local |
| `control-plane` | [Coordinator state backups and recovery](/docs/control-plane-backup/) | backup-policy get/set, backups list/run | Installation-wide |
| `db-roles` | Custom database roles | list, get, create, set, delete | MongoDB only |
| `db-users` | Database users | list, get, create, set, delete | MongoDB only |
| `elasticsearch` | Native access | users, roles, api-keys | Elasticsearch only |
| `events` | Durable events/audit and SSE | list, watch | All |
| `health` | Control-plane health | — | All |
| `help` | Runtime help | any command path | Local |
| `login` | Password login and saved API token | — | All |
| `logout` | Revoke saved token/remove credentials | — | All |
| `mcp` | Stdio MCP bridge; management and per-cluster data MCP client configuration | config, config –cluster NS/NAME | Management: all; data: MongoDB, PostgreSQL, Supabase, Elasticsearch, RabbitMQ |
| `metrics` | Prometheus exposition | — | All |
| `mysql` | Native database access and PXC quorum recovery | users list/get/create/update/delete/rotate-password/discard-old-password, databases list/get/create/delete, grants list/grant/revoke; recover-quorum | MySQL and PXC; recover-quorum PXC only |
| `namespaces` | Project registration/token/relay health | list, get, create, update, validate, rotate-token, delete | All |
| `oauth-grants` | Connected OAuth application grants | list, delete | Control-plane accounts; administrators may manage all grants |
| `neo4j` | Native administrative users | users list/get/create/delete/password; create/password accept `--password-stdin` | Neo4j Community only; every user is an admin |
| `notification-channels` | [Durable operational notifications](/docs/notifications/) | list, get, create, replace, delete, test | Installation-wide; settings permissions |
| `openapi` | Download server specification | — | All |
| `operations` | Observe/cancel asynchronous work | list, cluster, get, wait, cancel | All; cancellation capability varies |
| `orphaned-volumes` | Inspect pending platform cleanup | list | All |
| `permissions` | Current effective permissions | — | All |
| `postgres` | Native database access | roles list/get/create/update/patch/delete/rotate-password, databases list/get/create/delete, grants list/grant/revoke | PostgreSQL and Supabase |
| `providers` | Catalogs/defaults/limits | list, get | All |
| `openbao` | Initialization material and dynamic-credential connections | unseal-keys reveal; database-connections list/get/create/delete | OpenBao only |
| `rabbitmq` | Native access | vhosts, users, permissions, topic-permissions, policies, operator-policies | RabbitMQ only |
| `rbac` | Permission checks and policy | can, policy get/set | All |
| `restores` | Recovery/clone inventory | create, list, get | All |
| `settings` | Control-plane configuration | get, set-oidc, agent-rollout get/set | All |
| `storages` | S3 destination registration/probe | list, get, create, set, delete, verify | All |
| `system-users` | Engine driver-managed identities | list, rotate | Engine driver-supported identities |
| `supabase` | API credentials, SMTP and edge functions | credentials, smtp-password set, functions list/get/deploy/delete, rotate-jwt-secret | Supabase only |
| `temporal` | Client certificates, workflow namespaces and write-only UI OIDC secret | client-certificate issue/list, namespace list/delete, ui-oidc-secret set | Temporal only |
| `tokens` | Control-plane API tokens | list, get, create, set, delete (alias revoke) | All |
| `version` | Client/server build metadata | — | All |
| `weaviate` | Native database access | users list/get/create/update/delete/rotate-key; roles list/get/create/update/delete | Weaviate only |
| `whoami` | Authenticated identity | — | All |
`karactl clusters retry-deployments NAME [--namespace NS] [-o json]` explicitly [re-arms exhausted platform deployments](/docs/clusters/#retry-exhausted-platform-deployments) after fixing the underlying platform failure; it requires `clusters/update` and returns a conflict if no deployment is exhausted.
`karactl clusters upgrade-agents NAME [--include-current] [--ignore-maintenance-window] [--wait] [--timeout 30m]` rolls out the current agent build without changing engine versions. `clusters get` shows `status.agents` target/outdated/unknown details; member output includes agent version/build. Current agents return HTTP 409 `AgentsCurrent` unless included explicitly. See [rolling out member agents](/docs/getting-started/upgrade/#rolling-out-member-agents).
`karactl clusters members move NAME MEMBER [--ignore-maintenance-window] [--wait] [--timeout 30m]` requests native member replacement with the current host excluded on placement-capable providers. It asks for confirmation (`--yes` for automation). [Nomad](https://developer.hashicorp.com/nomad)/[Dokploy](https://dokploy.com/) require another eligible node; [Zerops](https://zerops.io/) is best-effort and provides no host guarantee. Unsupported targets and unknown authoritative placement are refused. Inspect `clusters placement NAME` and the returned operation; see [moving a member](/docs/operations/#moving-a-member).
`karactl settings agent-rollout get` shows rollout settings and read-only `currentBuild`. `settings agent-rollout set [--automatic=true|false] [--max-concurrent-clusters N]` replaces settings (defaults: off and 1; range 1–5). Automatic rollout considers outdated, not unknown/legacy agents.
For engine-native payloads and permissions, use [RabbitMQ access](/docs/rabbitmq/access-and-tls/), [Elasticsearch access](/docs/elasticsearch/access-and-security/) and [MongoDB](/docs/mongodb/). For orphan inventory and manual cleanup, use [orphaned platform volumes](/docs/operations/#orphaned-platform-volumes). [PostgreSQL](https://www.postgresql.org/) payloads/TLS and proxy compatibility are in [PostgreSQL access](/docs/postgres/access-and-tls/); [Supabase](https://supabase.com/) credentials, SMTP and function constraints are in [Supabase operations](/docs/supabase/operations/). [Neo4j](https://neo4j.com/) user commands, CA trust and direct Bolt connection caveats are in [Neo4j access](/docs/neo4j/access-and-tls/); the managed `neo4j` account uses `system-users rotate`, not native user mutations. PostgreSQL `system-users rotate` supports enabled Data MCP identities only, not the revealed `postgres` administrator or other engine-owned accounts. Native role and binding rotations do not rotate that administrator. Read the [administrator credential containment limitation](/docs/postgres/access-and-tls/#administrator-credential-containment) before revealing or downloading a PostgreSQL connection credential.
## MCP clients
[Section titled “MCP clients”](#mcp-clients)
`karactl --context production mcp` bridges a local stdio [MCP](https://modelcontextprotocol.io/) client to the control plane’s `/mcp` using the same server/token resolution as other commands. Standard output contains MCP messages only; diagnostics go to stderr. Tool lists, calls, progress, cancellation, server instructions and human confirmation remain server-controlled. Narrow discovery with `--toolsets Clusters,Backups,Workflows`; this is not an authorization boundary.
`karactl mcp config` mints a dedicated restricted token and prints configuration for your client. Without `--cluster` it configures the management endpoint `/mcp`; with `--cluster NS/NAME` it configures that cluster’s [data MCP](/docs/data-mcp/) endpoint `/mcp/clusters/NS/NAME`. Both forms share `--client`, `--write`, `--token-name` and `--expires-in` (90 days by default; whole days such as `30d` or a [Go](https://go.dev/) duration such as `720h`), and both print the newly minted secret once: protect the output, heed the stderr warning and revoke the token by ID through `tokens delete` when no longer needed. The server must support restricted API tokens; older servers reject the request.
```sh
karactl --context production mcp config --client cursor --namespace production
karactl --context production mcp config --client vscode --expires-in 30d --output /secure/path/mcp.json
karactl --context production mcp config --client codex --stdio
karactl --context production mcp config --cluster production/appdb --client claude-code
```
**Management tokens** (no `--cluster`): clients are `claude-code` (shell command), `cursor`, `vscode`, `codex` (TOML) or `json` (generic JSON, default). Tokens are read-only with the name `mcp-CLIENT-DATE` unless overridden; `--write` permits mutations only where owner permissions and server mode allow them, and repeatable `-n/--namespace` restricts the token to namespaces. `--stdio` prints a `karactl mcp` command using the selected saved context, with no embedded token, and reveals the newly minted dedicated token once on stderr for separate provisioning; it does not replace the saved context’s token. No files are written unless `--output FILE` is supplied; this command-specific flag writes configuration with mode 0600, rather than selecting the global output format.
**Data tokens** (`--cluster NS/NAME`, always `namespace/name`, never the implicit namespace): clients are `claude-code`, `cursor`, `vscode` or `json`; `codex` is not accepted. The token is scoped to that namespace and cluster with `cluster-data:read`; `--write` adds `cluster-data:write` and removes the read-only restriction, but cluster, owner RBAC and installation write gates still apply. The default name is `mcp-data--`. `-n/--namespace`, `--stdio` and `--output` are usage errors (exit 2) together with `--cluster`. See [MCP security and confirmation](/docs/mcp/) and the [MCP quick start](/docs/mcp-quickstart/).
For [OAuth-connected applications](/docs/mcp/#connect-with-oauth), use `karactl oauth-grants list -o json` to inspect visible grants and `karactl oauth-grants delete GRANT_ID` to revoke a connection. Non-administrators see their own grants; administrators with the required API-token permissions may inspect and revoke all accounts’ grants. Deletion uses the normal destructive confirmation; pass `--yes` for noninteractive automation. These commands manage account-level grants, not namespace resources.
Use `karactl postgres bindings list CLUSTER` and `karactl postgres bindings rotate CLUSTER ID` for [managed consumer bindings](/docs/postgres/bindings/); neither returns credentials.
## Global flags
[Section titled “Global flags”](#global-flags)
| Flag | Use |
| ---------------------------- | ---------------------------------------------------------------------------------- |
| `--context NAME` | Select saved connection |
| `--server URL` | Override environment/context server |
| `--token TOKEN` | Override API token; prefer environment injection to avoid history/process listings |
| `-n, --namespace NAME` | Explicit namespace filter/object scope |
| `-o, --output FORMAT` | table (default), json, yaml |
| `--cookie NAME=VALUE` | Repeatable advanced-session cookie, not saved |
| `--csrf-token TOKEN` | Advanced-session CSRF value, not saved |
| `--insecure-skip-tls-verify` | Disable certificate verification; unsafe |
| `-h, --help` | Command help |
## Important command-specific flags
[Section titled “Important command-specific flags”](#important-command-specific-flags)
Run `karactl --help` for every flag. Query flags reach the API only when explicitly supplied; help defaults do not imply resource updates.
| Command | Important flags |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `notification-channels create/replace NAME` | create uses `--name`; `--type webhook\|slack\|email`, `--enabled`, repeatable `--category`/`--namespace`, or protected `-f` JSON/YAML; replace takes positional NAME |
| `notification-channels create/replace` destinations | Webhook: `--url`, `--signing-secret-stdin`; Slack: `--webhook-url-stdin` (secret, never an argument); email: `--smtp-host`, `--smtp-port`, `--security starttls\|tls`, `--username`, `--password-stdin`, `--from`, repeatable `--to` |
| `notification-channels test NAME` | Prints delivered/error; `-o json`/`-o yaml` retain structured result |
| `control-plane backup-policy set` | `--enabled`, `--storage [namespace/]name`, `--schedule` (UTC, default `0 3 * * *`), `--retain-count` (1–1000, default 14), `-f`; replaces policy |
| `control-plane backups list` | `--limit`, `--cursor`; `backups run` queues a manual run |
| `clusters create [name]` | `-f`, `--provider` (psmdb), `--version`, `--backup-storage`, repeatable `--schedule 'NAME=CRON,type=TYPE,retain=COUNT[,storage=NAME]'`, `--source-namespace`, `--wait`, `--timeout` |
| `clusters scale NAME` | `-f`, `--replicas`, CPU/RAM/disk `--min-*`/`--max-*`, `--start-cpu`, `--cpu-mode`, `--wait`, `--timeout`; topology/component flags are engine driver-specific |
| `clusters patch NAME` | `-f`, `--wait`, `--timeout` |
| `clusters delete NAME` | `--keep-volumes`, `--keep-backups` (both false unless supplied), `--yes`, `--wait`, `--timeout`; see [delete or suspend](/docs/clusters/#delete-or-suspend) |
| `backups create CLUSTER [NAME]` | `-f`, `--storage` (cluster default), `--type` (CLI default logical; choose the engine-supported type in [backup guidance](/docs/backups/)) |
| `backups list CLUSTER` | `--limit`, `--cursor` |
| `backup-drills list CLUSTER` | `--limit`, `--cursor`; table includes ID, phase, backup, trigger, restore duration, data age and creation time |
| `backup-drills get CLUSTER DRILL` | Checks, limitations, cleanup and error; `--wait`, `--timeout` (6h client default), `-o json/yaml` |
| `backup-drills run CLUSTER` | `--backup NAME` (otherwise latest successful eligible backup), `-f`, `--wait`, `--timeout` (6h client default); timeout does not cancel server work |
| `clusters create/patch` drill policy | `--drill-enabled`, `--drill-schedule` (UTC cron), `--drill-timeout` (server default 6h, range 30m–48h), `--drill-backup-schedule`; files remain authoritative; see [policy shape and scope](/docs/backup-drills/#scheduled-policy) |
| `restores create CLUSTER [NAME]` | `-f`, `--backup`, `--time` (RFC3339, where supported), `--source-cluster`, `--source-namespace`, `--into-new`, `--acknowledge-bound-consumers`, `--yes` |
| `namespaces create` | `--name`, `--project-id` (provider target ID), `--token-file`, `--description`, `--backend` (zerops), repeatable `--config key=value` (non-secret provider settings; Zerops accepts none, its API URL is operator-only); no `-f` |
| `storages create/set` | `-f`; `verify NAME` checks connectivity separately; list has `--limit`, `--cursor`; delete has `--yes` |
| `tokens create` | `-f` OR `--name`, `--expires` (RFC3339 or duration), `--read-only`, repeatable `--scope-namespace`, `--scope-cluster ns/name`, `--permission resource:action`, `--allow-credential-reveal` |
| `tokens set` | `-f` (name/expiry only); list has `--limit`, `--cursor`; delete/revoke has `--yes` |
| `mcp config` | `--client claude-code\|cursor\|vscode\|codex\|json` (json), `--write`, `--token-name`, `--expires-in` (90d; whole days or Go duration); management only: repeatable `-n/--namespace`, `--stdio`, `--output FILE`; data: `--cluster NS/NAME` (no `codex`; rejects `-n`, `--stdio`, `--output`) |
| `events list` | `--cluster`, `--type`, `--operation`, `--actor`, `--resource`, `--order asc\|desc`, `--limit`, `--cursor` |
| `events watch` | Same scope filters; `--cursor`, `--last-event-id`; no list limit/order |
| `operations list/cluster` | `--phase`, `--limit`, `--cursor`; list also `--cluster` |
| `operations wait/cancel ID` | wait: `--timeout` (30m); cancel: `--yes` |
### Weaviate native access
[Section titled “Weaviate native access”](#weaviate-native-access)
Use `karactl weaviate users create CLUSTER NAME --role viewer`, `users update CLUSTER NAME --role viewer --active` (or `--inactive`), and `users rotate-key CLUSTER NAME --yes`. `--role` is repeatable; update replaces the complete role list, and omission clears roles. User list/get/delete and role list/get/delete take the cluster and, for object actions, the name. Role create uses `roles create CLUSTER -f role.yaml`; replacement uses `roles update CLUSTER NAME -f role.yaml`, with `{name, permissions}`.
Generated user API keys are secret-bearing creation/rotation results, not later GET data; protect terminal output. Do not blindly retry after losing a mutation response: the CLI does not send idempotency keys. Protected identities and extra privilege grants follow the [Weaviate access guide](/docs/weaviate/access-and-tls/#management-authorization); [MongoDB](https://www.mongodb.com/) `db-users`/`db-roles` do not apply.
Key rotation requires `database-users/update` plus `clusters/update` when the target user’s current roles include anything beyond `viewer` or `read-only`. User deletion needs only `database-users/delete`, including for privileged users; protected identities remain excluded.
### Supabase native access
[Section titled “Supabase native access”](#supabase-native-access)
`karactl supabase credentials CLUSTER` masks API keys, the dashboard password and JWT secret in every output format. Pass `--show-secrets` only when sensitive terminal or redirected output is protected. The endpoint requires the separate `cluster-credentials:read` grant and returns `Cache-Control: no-store`.
`karactl supabase smtp-password set CLUSTER` reads the password through a hidden terminal prompt or the first stdin line; passwords are never accepted as command arguments. For example, pipe a secret manager’s output directly into this command.
Use `karactl supabase functions list CLUSTER`, `functions get CLUSTER NAME`, `functions deploy CLUSTER NAME DIR_OR_FILE`, and `functions delete CLUSTER NAME`. Deploy sends a map of relative file paths to source text; directories are recursive and require a nonempty `index.ts` entrypoint. A standalone source file is deployed as `index.ts`. Symlinks and special files are rejected, and source text is limited to 1 MiB per function. A cluster supports at most 100 functions. Names match `[a-z0-9][a-z0-9_-]{0,62}`; `main` is reserved for the runtime dispatcher, and parent traversal is forbidden. Sources are stored sealed but remain readable to callers with cluster-read permission, so do not embed application secrets in source files.
Function deletion and `karactl supabase rotate-jwt-secret CLUSTER` require confirmation (or `--yes`). Rotation queues a durable `rotate-jwt-secret` operation; track the returned ID with `karactl operations get` or `operations wait`. SMTP and function mutations require cluster-update permission; audit events contain identifiers, never passwords or source text.
### MySQL native access
[Section titled “MySQL native access”](#mysql-native-access)
The `mysql` command group administers application accounts and database-level grants:
```sh
karactl mysql users list CLUSTER
karactl mysql users get CLUSTER USER
karactl mysql users create CLUSTER -f user.json
karactl mysql users update CLUSTER USER -f user.json
karactl mysql users rotate-password CLUSTER USER --retain-current
karactl mysql users discard-old-password CLUSTER USER
karactl mysql databases list CLUSTER
karactl mysql databases get CLUSTER DATABASE
karactl mysql databases create CLUSTER -f database.json --character-set utf8mb4 --collation utf8mb4_0900_ai_ci
karactl mysql grants list CLUSTER --user USER
karactl mysql grants list CLUSTER --database DATABASE
karactl mysql grants grant CLUSTER -f grant.json
karactl mysql grants revoke CLUSTER -f grant.json --yes
karactl mysql databases delete CLUSTER DATABASE --force --yes
karactl mysql users delete CLUSTER USER --yes
```
Accounts use the fixed `'name'@'%'` identity and require TLS. Creation and password rotation return a server-generated `generatedPassword` once, using the same JSON/YAML/table output conventions as [PostgreSQL](https://www.postgresql.org/) role creation; save it securely and never log the response. `--retain-current` keeps the previous password valid during a credential rollout; `discard-old-password` removes that retained password.
User creation documents contain `name` and optional `locked` and `maxUserConnections`. `users update` sends a partial attribute update: omitted fields stay unchanged, and only `locked` and `maxUserConnections` may be changed. Passwords are managed separately through rotation, not supplied in user documents.
Database documents contain `name` and optional `characterSet` and `collation`. The creation flags override the corresponding document fields; these properties cannot be updated after creation. Database deletion refuses nonempty databases unless `--force` is supplied. Grant documents contain `user`, `database` and `privileges`; grants are database-level only, not global or administrative privileges. Listing grants requires a nonempty `--user` or `--database` filter (both may be supplied). User deletion, database deletion and privilege revocation require confirmation or `--yes`.
### PostgreSQL native access
[Section titled “PostgreSQL native access”](#postgresql-native-access)
The `postgres` command group also works with [Supabase](https://supabase.com/) clusters, which share the PostgreSQL core:
```sh
karactl postgres roles list CLUSTER
karactl postgres roles get CLUSTER ROLE
karactl postgres roles create CLUSTER -f role.json
karactl postgres roles update CLUSTER ROLE -f role.json
karactl postgres roles rotate-password CLUSTER ROLE --yes
karactl postgres databases create CLUSTER -f database.json
karactl postgres databases list CLUSTER
karactl postgres databases get CLUSTER DATABASE
karactl postgres grants list CLUSTER --database DATABASE
karactl postgres grants grant CLUSTER -f grant.json
karactl postgres grants revoke CLUSTER -f grant.json --yes
karactl postgres databases delete CLUSTER DATABASE --force --yes
karactl postgres roles delete CLUSTER ROLE --yes
```
A role document contains `name`, `login` (default true), `createdb`, `createrole`, `connectionLimit` (default -1), `memberOf`, optional RFC3339 `validUntil`, and optional `password`. Omitted passwords on login-role creation are generated and returned once as `generatedPassword`; rotation returns a new generated password. Responses are `no-store`; an identical request with the same idempotency key safely replays its sealed result. Never put passwords in command arguments or commit credential-bearing request documents. Privileged `superuser`, `replication`, and `bypassrls` flags, privileged memberships, and changes to existing privileged role passwords additionally require cluster-update permission. Engine system roles are hidden and immutable; Supabase service roles are protected too.
A database document contains `name`, `owner`, optional `encoding` (UTF8), `locale` (C.UTF-8), and `allowConnections` (true). Encoding and locale cannot be changed after creation. A grant document contains `role`, `database`, `privileges`, and optional `schema`: database privileges are CONNECT/CREATE/TEMP; schema privileges are USAGE/CREATE. Predefined memberships include pg_read_all_data, pg_write_all_data, pg_monitor, pg_read_all_stats, and pg_signal_backend.
Managed login passwords are sealed and synchronized to every [ProxySQL](https://proxysql.com/) member without restarting PostgreSQL. Roles created directly in SQL are not routable through ProxySQL. Use the proxy hostname, cluster CA, and `sslmode=verify-full` for TLS connections; public endpoints are advertised when enabled and configured by the infrastructure provider. Direct data-member endpoints remain private. Role/database deletion, grant revocation, and password rotation require confirmation (or `--yes`).
Access operations use authenticated internal POST actions on a freshly resolved primary. HA members also require the [Patroni](https://patroni.readthedocs.io/) leader lock; no writable primary produces HTTP 503. Only a precomputed SCRAM-SHA-256 verifier crosses the role-action protocol, never the plaintext password in SQL. Membership replacement is transactional; schema grants connect to the requested database. `roles patch CLUSTER ROLE -f patch.json` changes only supplied fields; `roles update` replaces role attributes.
ProxySQL does not implement every PostgreSQL protocol feature. In particular, clients requesting different result formats per column must select a uniform format (for pgx, use simple protocol or request text results). Supabase’s internal services bypass ProxySQL for features such as LISTEN/NOTIFY and logical replication.
### Restricted API tokens
[Section titled “Restricted API tokens”](#restricted-api-tokens)
```sh
karactl tokens create --name inventory --expires 720h --read-only \
--scope-namespace prod --permission clusters:read
```
Without `-f`, creation requires `--name`; combining a request file with any token creation flag is a usage error (exit 2). Repeat scope/permission flags to add entries. `--scope-namespace` restricts the new token; persistent `--namespace/-n` remains the request’s REST scope. `--expires` accepts an RFC3339 timestamp or a positive [Go](https://go.dev/) duration relative to creation.
Omitting all restriction flags creates an unrestricted token. Supplying any restriction flag creates an immutable deny overlay; credential reveal defaults to disabled, including on otherwise read-only tokens. List/get/create table output summarizes restrictions; JSON/YAML retain the full fields. `karactl login` creates an unrestricted token. See [restricted API tokens](/docs/access-control/#restricted-api-tokens) for scope, permission, issuance and [OIDC](https://openid.net/developers/how-connect-works/) rules. Distributed PostgreSQL adds `karactl postgres rebalance CLUSTER [--strategy NAME] [--wait]` (PostgreSQL only, not Supabase). `--wait` follows the returned operation; timeout does not cancel it. Worker groups scale through `clusters patch` using `spec.engineOptions.citus.workerGroups`. See [distributed PostgreSQL](/docs/postgres/distributed/) for artifact availability, coordinated recovery and limits.
### Temporal native access
[Section titled “Temporal native access”](#temporal-native-access)
```sh
karactl temporal client-certificate issue CLUSTER --common-name worker --ttl 720h --out-dir DIR
karactl temporal client-certificate list CLUSTER
karactl temporal namespace list CLUSTER
karactl temporal namespace delete CLUSTER NAME --yes
karactl temporal ui-oidc-secret set CLUSTER --from-file FILE
```
Certificate issuance requires exactly one of `--out-dir DIR` or `--stdout`; stdout contains the private key, so protect redirected output. Existing files are not overwritten. Secret input requires exactly one of `--from-file FILE` or `--from-stdin`; there is no argv secret flag. All commands accept `-n NS` for the Karakoram namespace, not the workflow namespace. See [Temporal access/TLS](/docs/temporal/access-and-tls/) for permissions, output modes, SDK configuration and [operations](/docs/temporal/operations/) for destructive namespace deletion.
## Safety and exit statuses
[Section titled “Safety and exit statuses”](#safety-and-exit-statuses)
Deletion, in-place restores, member replacement, compatibility-version raises, credential/TLS/namespace-token rotation, account password resets and operation cancellation prompt **Type the name to confirm** when stdin is a terminal. Compound identities use the final argument; in-place restore asks for the target cluster. `--yes`/`-y` skips confirmation and is required when stdin carries a request body. Without a terminal, one line containing `y` or `yes` (case-insensitive) confirms; another nonempty answer cancels (1), and empty input/EOF is a usage error (2), without mutation.
Repeated cluster deletion with unchanged retention options returns the same in-flight Operation; changing either option supersedes its tracking record. `storages verify` fails even after HTTP success when `verified: false`. Errors include available field/reason/resource context on stderr.
| Exit | Meaning |
| ---- | ----------------------------------------------------------------------------------------------- |
| 0 | Success |
| 1 | Generic failure, failed/cancelled operation, declined confirmation, failed storage verification |
| 2 | Usage/argument/flag error or missing noninteractive confirmation |
| 3 | HTTP 404 |
| 4 | HTTP 409 |
| 5 | HTTP 401/403 |
| 6 | HTTP 400/415/422 validation/media error |
| 7 | Wait deadline or HTTP 408/504; accepted work may continue |
## OpenBao access and recovery
[Section titled “OpenBao access and recovery”](#openbao-access-and-recovery)
```sh
karactl -n NS openbao unseal-keys reveal secrets --confirm secrets -o json
karactl -n NS openbao database-connections create secrets \
--name orders --target ordersdb --role reader:readonly \
--default-ttl 1h --max-ttl 24h
karactl -n NS openbao database-connections list secrets -o json
karactl -n NS openbao database-connections get secrets orders -o json
karactl -n NS openbao database-connections delete secrets orders --yes
karactl -n NS backups create secrets beforechange --type raft-snapshot --storage archive
```
Reveal prints real shares and the initial root token in every output format; protect stdout and offline custody. It requires the distinct `openbao-unseal-keys:reveal` grant and exact `--confirm`. Connection creation accepts `-f` or `--name`, `--target` and repeatable `--role name:readonly|readwrite[:database]`; TTL flags require positive whole seconds and fill roles without explicit TTLs. [RabbitMQ](https://www.rabbitmq.com/) roles in one connection must share the same TTL pair. The returned paths are read through [OpenBao](https://openbao.org/) with native tokens, not through `karactl`. Shared restart/replace-member/rotate-tls commands manage lifecycle; only catalog-listed forward versions are allowed. Use protected request files for write-only `openbaoUnsealShares` on deleted-source restores/clones. See [OpenBao access](/docs/openbao/access-and-tls/), [operations](/docs/openbao/operations/) and [snapshots](/docs/openbao/backups/).
# Clusters and operations
> A cluster is a namespace-scoped desired configuration, observed engine topology and durable operation history.
A cluster is a namespace-scoped desired configuration, observed engine topology and durable operation history.
## Cluster identity and capabilities
[Section titled “Cluster identity and capabilities”](#cluster-identity-and-capabilities)
Each cluster selects an immutable engine driver/engine. Data-bearing members are singleton runtimes with independently identified persistent volumes supplied by the namespace’s provider (Local Storage on [Zerops](https://zerops.io/)); auxiliary stateless components depend on the engine driver. Cluster names are 2–12 lowercase letters/digits, beginning with a letter, and unique within a [namespace](/docs/namespaces/). Use `-n ` to select one; without a selected namespace, single-object requests default to `default`.
The engine driver catalog advertises versions, upgrade paths, topology/component limits, backup types and operations. Query `karactl providers list` or `karactl providers get -o yaml`. Supported does not mean disruption-free: follow the linked engine’s prerequisites and limits.
| Capability | [MongoDB](/docs/mongodb/) (`psmdb`) | [RabbitMQ](/docs/rabbitmq/) | [Elasticsearch](/docs/elasticsearch/) | [PostgreSQL](/docs/postgres/) / [Supabase](/docs/supabase/) | [Temporal](/docs/temporal/) | [Neo4j Community](/docs/neo4j/) | [Weaviate](/docs/weaviate/) |
| --------------------------------------------- | --------------------------------------------------- | ----------------------------- | -------------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------- |
| Create | Yes | Yes | Yes | Yes; Supabase artifact availability gates components | Requires PostgreSQL source and binding service | Yes; single member | Yes |
| Scale CPU/RAM/disk | Yes; disk only grows | Yes; disk only grows | Yes; disk only grows | Yes; disk only grows and needs provider capability | Stateless role resources; durable disk belongs to source | Yes; disk only grows; memory changes interrupt service | Yes; disk only grows |
| Add/remove members | Yes | Yes | Yes | Single→HA; HA 3↔5; no HA→single | Per-role replicas within topology bounds | No | Cluster 3–9; drain/remove non-voters only; no automatic rebalance |
| Replace member | Yes | Yes | Yes | Yes | No separate replacement operation | No | Preserve identity; healthy/quorum or proven surviving replicas required |
| Rolling restart | Yes | Yes | Yes | HA; single interrupts service | Ordered by role; singleton roles interrupt service | Restart interrupts service | Non-voters before voters; single interrupts service |
| Upgrade | Catalog paths | Catalog paths | Catalog paths | Rolling minor; offline adjacent major for postgres; PG17 minor only for Supabase | Catalog paths; no skipped minors/downgrades | 2026.09.0→2026.10.0; offline, fresh backup required | Cataloged patch-forward/one minor forward; no downgrade |
| Feature compatibility version (FCV) promotion | Yes | No | No | No | No | No | No |
| TLS rotation | Yes | Yes | Yes | Yes; no CA replacement | Three-phase CA rotation | Yes; full outage | Rolling restart |
| System-user rotation | Yes | Yes | Yes | Yes | Binding-managed database credentials | Managed `neo4j` | Root key: two rolling passes; internal password: full restart |
| Public TCP exposure | Yes | Yes | Yes | Yes; selected client service | No; UI HTTPS only with OIDC | Bolt only | REST/gRPC with TLS; internal ports private |
| Backup types | Percona Backup for MongoDB (PBM): logical, physical | Definitions only; no messages | Native snapshots | pgBackRest full/diff/incr; database data only | Source full/diff/incr plus state manifest | Offline `dump`; system + graph; full outage | Native S3 full |
| Point-in-time recovery (PITR) | Yes | No | No | Yes, opt-in | No Temporal timestamp restore | No | No |
| In-place restore | Yes | Merge definitions | Native restore | Destructive/offline | Destructive source restore; shared-source acknowledgement | Destructive/offline | Destructive; same member count |
| Clone | Yes | Definitions only | Yes | Same engine driver and PG major only | No arbitrary destination; original source/name required | Neo4j dump; target version not older than source | Same member count; fresh target users/roles |
| Suspend/resume | Yes | Yes | Yes | Yes | Retain bound databases and disable login role | Yes | Yes |
| Cross-namespace clone | Yes | Definitions only | Yes | Yes; same engine driver and PG major | No arbitrary destination | Yes; Neo4j backup only | Yes; storage/recovery admission applies |
See [Backups and storage](/docs/backups/) for shared recovery mechanics and the engine guides for runtime, compatibility and checkpoint rules.
### Weaviate
[Section titled “Weaviate”](#weaviate)
[Weaviate](/docs/weaviate/) supports immutable `single` or `cluster` topology, fixed voter identities and typed options instead of raw engine configuration. Its internal transports require isolated private networking even when client TLS is enabled. Read [native access/TLS](/docs/weaviate/access-and-tls/), [draining, replacement and rotation](/docs/weaviate/operations/), and [full-backup recovery constraints](/docs/weaviate/backups/) before maintenance.
### PostgreSQL and Supabase
[Section titled “PostgreSQL and Supabase”](#postgresql-and-supabase)
[PostgreSQL](/docs/postgres/) adds `single` (one member) and `ha` (three/five [Patroni](https://patroni.readthedocs.io/) + [etcd](https://etcd.io/) members), a separate [ProxySQL](https://proxysql.com/) component, native roles/databases/grants and [pgBackRest](https://pgbackrest.org/) full/diff/incr/PITR recovery. It supports cataloged [Percona](https://www.percona.com/) majors 16/17/18, replica-first minor maintenance and offline backup-gated adjacent major upgrades. Single→HA and HA 3↔5 are supported; HA→single is rejected. See [access/TLS](/docs/postgres/access-and-tls/), [operations](/docs/postgres/operations/) and [backups](/docs/postgres/backups/).
[Supabase](/docs/supabase/) wraps that core, limits PostgreSQL to major 17 and adds independently sized stateless components. `spec.version` is the database version; application versions are release-locked. Internal database connections bypass ProxySQL through a leader-aware TCP forwarder. See [components/presets/dependencies and current artifact limits](/docs/supabase/components/) and [credentials, SMTP and function operations](/docs/supabase/operations/). Standard/full preset definitions do not imply their artifacts are currently available.
Both engine drivers support TLS/system-user rotation, public TCP subject to provider capability, restore/clone, replacement and retained-volume suspension. Backups match engine drivers exactly and protect database data, not separate Supabase object storage or all control-plane-held application state. [Shared recovery](/docs/backups/) describes namespace and retention rules.
### Temporal
[Section titled “Temporal”](#temporal)
[Temporal](/docs/temporal/) runs combined or split stateless workflow roles on an existing Karakoram PostgreSQL ≥12 source. It uses the control plane’s shared PostgreSQL binding service. Server versions and UI artifacts are locked separately. Topology, persistence options and history shard count are immutable; scaling is per role. See [access/TLS and OIDC](/docs/temporal/access-and-tls/) and [namespace, upgrade and recovery operations](/docs/temporal/operations/). Temporal backup manifests live in cluster state, while the PostgreSQL source owns archives and retention.
### MySQL
[Section titled “MySQL”](#mysql)
The `mysql` engine driver runs Percona Server in `single` or three/five-member, single-primary Group Replication topology. Group scaling clones new monotonic member identities from the ONLINE writer before joining them. Removal drains ProxySQL, hands the writer to an up-to-date ONLINE secondary when necessary, leaves Group Replication, and confirms the stopped runtime before deleting the ID-fenced runtime and volume. Rejected volume deletion is handed to orphan-volume inventory. Replacement joins a new identity before retiring its predecessor; cancellation before retirement rolls back the new identity, while cancellation after retirement begins completes the cutover under reconciliation ownership.
Single→group retains the existing seed volume and uses a new group identity and bootstrap epoch; group→single is rejected. Group topology requires ProxySQL. Resource changes, startup-only parameter changes and minor upgrades roll secondaries before the writer. The allowlisted `innodb_buffer_pool_instances` and `table_open_cache_instances` settings require a restart and remain pinned per member until its turn; the other allowlisted variables are applied live. The 8.0→8.4 major upgrade requires a successful current-version backup started since the durable upgrade request, publishes and acknowledges `innodb_fast_shutdown=0` before stopping each member, and upgrades the old writer last. Its departure lets Group Replication elect an upgraded writer; lower-version join exceptions are never enabled. A singleton upgrade is offline. Downgrades are rejected.
TLS rotation stages new member leaves under the existing CA, publishes them, and waits for configuration acknowledgement. [MySQL](https://www.oracle.com/mysql/) reloads client TLS in place; GR recovery uses the new leaves on its next connection. ProxySQL applies its updated TLS configuration. Formation, restore and active physical backups fence membership changes. Durable membership, retirement, rolling and TLS journals resume after control-plane restart.
### Percona XtraDB Cluster
[Section titled “Percona XtraDB Cluster”](#percona-xtradb-cluster)
The separate `pxc` engine driver supports only [Galera](https://galeracluster.com/) `cluster` topology with three or five data members and at least one ProxySQL replica. It shares MySQL-native access and physical backup lifecycle, but keeps its own artifact/version catalog, `pxc` repository prefix and restore provenance. MySQL↔[PXC](https://www.percona.com/mysql/software/percona-xtradb-cluster) conversion and cross-engine driver restore are rejected. Traffic requires Primary/Synced/readiness, with one proxy-routed writer; quorum loss rejects minority reads and writes. Full-outage recovery compares every stopped member’s recovered same-UUID Galera position; explicit minority recovery additionally fences missing owned members before bootstrap. See [MySQL/PXC configuration](/docs/mysql/configuration/), [operations and recovery](/docs/mysql/operations/) and [backup/UUID caveats](/docs/mysql/backups/).
## Create and change desired state
[Section titled “Create and change desired state”](#create-and-change-desired-state)
Use **Clusters → Create cluster** and the engine driver’s wizard, or prepare a complete `{name, spec}` YAML/JSON request using its engine guide:
```sh
karactl -n clusters create -f cluster.yaml --wait
karactl -n clusters list
karactl -n clusters get -o yaml
karactl -n clusters patch -f patch.yaml --wait
karactl -n clusters edit --wait
```
`spec` is desired state; `status` is observation. Accepted changes increment `generation`; `status.observedGeneration` records the generation reconciled. `Ready` alone can describe the previous generation: compare both numbers. `spec.version` is requested; `status.version` is observed, absent when unavailable and the oldest observed member version during a mixed-version rollout.
For OpenBao, `Initializing` describes first-time native formation, not the routine initialization/unseal checks on an already formed cluster. A maintenance-deferred update with no admitted member change retains `Ready` while healthy, with the previous `observedGeneration`. An admitted resource/configuration roll reports `Updating`, a version roll `Upgrading`, and native membership changes `Scaling`. Restore retains `Restoring`; unavailable or sealed members do not become healthy merely because an update is deferred.
PATCH merges only `{spec: {...}}`; arrays replace the whole previous array. Edit replaces the complete specification. The write is conditional on the cluster incarnation and generation that admission inspected: a concurrent edit returns `409` rather than silently erasing another accepted change. Reload the cluster, merge your intended changes, and retry. Queued actions are likewise rejected if that inspected cluster was deleted, replaced, suspended, or changed before admission committed. Omitted create fields select engine driver defaults; explicit zero is not a request for defaults. See [REST API](/docs/api/) for request validation and transport details.
Omitted PATCH collections retain their stored state, including absence; response-only empty arrays do not create a desired-state change. Supply `backup.schedules: []` or `exposure.allowlist: []` explicitly to clear the corresponding list.
### `spec.dataMcp`
[Section titled “spec.dataMcp”](#specdatamcp)
Optional [cluster data MCP](/docs/data-mcp/) exposes a separate token-authenticated data endpoint; it is disabled unless `enabled: true`. Fields are `enabled`, `access` (`read` by default or `readWrite`), engine-specific `scope` (`databases`, `schemas`, `indexPatterns`, `vhosts`), `tools.deny`, `tools.confirmWrites`, `auditQueryText` and companion `resources`. See the [field/default and scope reference](/docs/data-mcp/#enable-and-scope-access) before enabling. The observed `status.dataMcp` is separate from database membership and reports provider/access/readiness, not a token’s effective authority.
## Operations and status
[Section titled “Operations and status”](#operations-and-status)
| Layer | Meaning |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| HTTP acceptance | `202` accepts asynchronous work, not completion. Cluster creation returns `201`, also before convergence. Save the returned operation ID; see [API](/docs/api/). |
| Operation phase | `Pending`, `Running`, then `Succeeded`, `Failed` or `Cancelled`. Inspect `target`, `targetGeneration`, `step`, `message` and timestamps. |
| Cluster phase/health | `Creating`, `Initializing`, `Ready`, `Updating`, `Scaling`, `Upgrading`, `Restoring`, `Degraded`, `Error`, `Deleting` or `Suspended`. Inspect conditions and individual members/agents, not just the phase. |
**Operation `Succeeded` is not a lasting health guarantee.** Creation requires its target generation to be observed `Ready`; a diagnostic `Degraded` result keeps create tracking Running until recovery or its normal deadline. Other desired-state tracking can succeed after its target generation is observed in `Degraded` with an explanation. A newer generation cancels superseded tracking; failed/cancelled tracking does not undo desired state or stop convergence.
Reconciler-generated `PhaseChanged` events retain the new phase’s sanitized user-visible status message and a bounded summary of non-True conditions (`type / status / reason`), so the explanation remains available after recovery, including transitions without a generation change. They do not include cluster specs, internal state or condition message bodies. The admission-time transition to `Deleting` records only the phase arrow.
When an engine returns a failed or partial observation, fresh safety conditions (for example, a failed quorum condition) and safe error diagnostics remain visible, but that error cannot replace the inspected lifecycle phase or durable member identities with incomplete provisioning progress. Existing `Ready` becomes `Degraded`; pending lifecycle work stays pending. Every retained member’s health and agent-health claims become unknown/negative, and the failed sweep neither renews the observation lease nor advances `observedGeneration`. This applies to both ordinary reconciliation and independent availability renewal across all registered providers. A complete successful negative observation can still report fresh health for reachable siblings. An authoritative `Failed` or `Error` result (for example, an aborted initial restore) remains terminal even when returned with an error; it must not be replaced by the formerly serving phase. If no historical member inventory exists, identities from the partial observation remain visible with health withdrawn.
Formed clusters request health observation at most five seconds after the preceding reconciliation. Health is observed before recovery `Sync`; withdrawn readiness is persisted by the existing serialized cluster worker before platform repair can block. Read-only member probes allow four seconds for cold namespace/TLS connection setup and the native response (separate from restart/restore/backup action deadlines), and each complete health sweep has a five-second deadline. One immediate fresh sweep retry is the entire transient failure tolerance: a single failed probe can be absorbed by a successful retry without publishing a degraded intermediate result. Failed/stale samples are never reused as ready across cycles.
[MongoDB](https://www.mongodb.com/), [RabbitMQ](https://www.rabbitmq.com/) and [Elasticsearch](https://www.elastic.co/elasticsearch) collect project/service inventory once per sweep and overlap per-member platform reads, with at most eight provider method observations in flight per sweep. The same instance inventory is reused for placement and instance readiness checks; native replica-set and independent Elasticsearch reads run in parallel. Five-member regressions with 1.5-second platform calls complete within the existing five-second sweep. The deadline is unchanged: platform responses that cannot complete within it are unknown, not cached successes, and additional worker waves still consume the same bounded budget.
Provider inventory also bounds its own read traversal rather than hiding a serial per-runtime HTTP chain inside a single platform call. [Dokploy](https://dokploy.com/) overlaps read-only runtime observations while retaining its target lock; pending operation completion, legacy adoption and restoration decisions still execute serially under that lock. A real-provider regression with three persistent members and 600-millisecond platform HTTP responses completes a fresh sweep within five seconds. Concurrent instance diagnostics reduce in inventory order, and MongoDB FCV observations use bounded set probes with sorted successful fallback instead of depending on map iteration. [Nomad](https://developer.hashicorp.com/nomad) likewise bounds variable reads, service observations and missing-CNI allocation hydration to eight concurrent reads per traversal; pending-operation and missing-job mutation helpers retain their existing mutex/reload fences. [Zerops](https://zerops.io/) uses bulk paginated service/container inventories, not a per-runtime service-list traversal.
Dokploy and Nomad additionally enforce a shared ceiling of **eight HTTP observation GETs per provider/platform origin**, across nested traversals, provider instances and concurrently observed clusters in the same control-plane process. Admission honors cancellation. Slots are released after each response, before retry backoff; serialized lifecycle/mutation reads bypass the observation queue rather than being starved by health sweeps. Concurrent read failures are joined in stable input order, retaining cancellation and every observed `Retry-After` constraint; the latest deadline controls retry scheduling. A failed read-only prerequisite batch does not authorize mutation fallbacks.
A completed sweep that reports an unreachable or unhealthy member is a fresh negative observation, not a failure to observe the entire cluster. After the same-cycle retry, its `Degraded` status retains member-specific failure reasons and fresh health for reachable siblings. Those positive member observations still expire under the same sample-age lease. A failed/empty sweep or stale positive member sample cannot renew availability. During creation, expected not-yet-deployed agents remain unavailable without replacing deployment progress diagnostics or gating recovery/provisioning `Sync`.
While an ordinary reconciliation is busy, an independent five-second availability loop can refresh existing members from an isolated persisted snapshot, including during Creating, Initializing, Updating, Scaling, Upgrading or a failed lifecycle request. Member health does not imply lifecycle completion: these non-Ready phases, negative readiness/maintenance fences and the applied generation are preserved. A new cluster without members or prior observations has no availability to renew. Suspended, restoring and deleting intent excludes independent renewal; running imperative operations retain serialized observation/checkpoint ownership. Convergence-tracking records (including maintenance-deferred updates) do not own native operation receipts and do not block renewal. Publication checks that distinction atomically with the complete status/provider checkpoint, generation and incarnation; it writes status only, never provider state or applied generation. A slow `Sync` cannot by itself starve this publication. Ordinary and independent observation paths share a per-cluster publication gate from snapshot acquisition through the completed status write. In particular, finishing probes does not release the gate: an older delayed positive cannot overwrite a newer failure, or replace a newer lease with an older sample. Only successful pre-Sync publication releases the gate before Sync; failed writes retain it through the caller’s fallback state write. Final observation/publication reacquires the gate, so renewal is not serialized behind slow Sync. Unavailable agents/platform reads still fail closed under the unchanged lease. Reconciliation stages emit sanitized monotonic durations at DEBUG, or WARN when a stage exceeds the 15-second lease; fields contain cluster identity, stage and duration, not credentials, request bodies or private observation provenance.
Persisted health has a **15-second maximum sample age**: five seconds between cycles plus at most two five-second sweeps. Cluster/status/member API projections withdraw positive availability after that bound even if recovery, an operation, or a provider call blocks further publication. Sample age uses the start of the successful sweep, not condition transition times; missing/future timestamps are unknown. Legacy snapshots therefore require one successful health observation before they can report Ready. This is a bounded observation guarantee, not an instantaneous failure detector or a writable-transaction guarantee. After a CP restart, status reads Degraded/Unknown until the first post-start observation. Successful sweeps carry private provenance of the current control-plane process, independent of wall-clock timestamps. Old-process snapshots cannot replay Ready after a restart or clock correction, even if their wall time exactly matches a new sweep. Fresh current-process samples still require the normal timestamp lease; future and expired samples remain unknown. API projection never rewrites persisted observations or changes reconciliation’s readiness decisions. While one member or agent is hung, bounded reads can slow per-cluster observation publication enough that healthy members briefly report `ObservationStale`; this is fail-closed and clears once the sweep completes and a fresh observation is persisted. This lease also expires positive member health during Creating, Initializing, Scaling, Updating and Upgrading, even when overall readiness is already false. Current component, DCS/failover, replication, WAL-capacity and archive health conditions expire too; durable formation/certificate facts and definitive negative diagnostics are preserved.
MongoDB additionally requires a current primary `hello` and fresh reachability of its replica-set members. A surviving secondary’s heartbeat-delayed PRIMARY view alone cannot retain `WritablePrimary`. MongoDB, RabbitMQ and Elasticsearch probe member agents concurrently. RabbitMQ and [PostgreSQL](https://www.postgresql.org/) also overlap generic agent and native status requests because their HTTP transports establish separate TLS connections. All use the same shared four-second read-probe limit; the five-second sweep, 15-second readiness lease and same-cycle retry policy are unchanged.
MongoDB replication health uses the primary’s `replSetGetStatus` view, rather than a nearest secondary’s possibly partitioned view; unhealthy-member conditions name the affected member. [PBM](https://docs.percona.com/percona-backup-mongodb/latest/) inventory `list`/`status` reads race eligible members with independent four-second limits (config-server members only for sharded clusters). A blocked local PBM on one secondary cannot starve PITR configuration inventory or the reachable members’ health probes. Backup inventory runs alongside member observation with its own four-second budget: failure reports `Backup=False/InventoryUnavailable` and retains the last backup summary without discarding fresh member health or reporting every agent disconnected. Backup and restore initiation/cancellation receipts remain pinned; this parallelism applies only to read-only inventory.
A MongoDB member is healthy only when its native state is PRIMARY or SECONDARY, native health is 1, its agent and runtime are healthy, and its optime is no more than **10 seconds** behind the newest replica-set optime. Reachability alone does not establish replication health: a reachable secondary may temporarily fall behind while MongoDB changes away from a partitioned sync source. Such observed lag remains `health=false`; it is not an agent disconnection. `Replication/` enumerates affected members in member order with individual reasons (observation unavailable or measured replication lag). Any unavailable member observation takes precedence: status/reason is `Unknown/MemberObservationUnavailable`, even when another member has observed excessive lag. With only observed lag failures it is `False/ReplicationLag`. Lag classification requires successful, non-empty, all-running runtime inventory. The message is bounded to 2048 bytes, with an omitted-member count on overflow; individual member health and lag remain available in the member list.
`operations wait` succeeds on Operation `Succeeded` without checking cluster health. Mutation `--wait` first waits for the operation, then checks cluster generation/phase; `clusters wait` observes the cluster directly. Cluster waiting fails on `Degraded`/`Error`; deletion waits for absence or the requested `Suspended` state. A client timeout/disconnect does not cancel server work.
```sh
karactl -n clusters status
karactl -n clusters members
karactl -n operations list
karactl -n operations cluster
karactl operations wait
karactl operations cancel
```
Ordinary work defaults to 20 consecutive retryable failures or two hours of observed control-plane execution time; restores allow six hours. Successful steps reset consecutive failures; scheduled waits consume time, not failures. Elapsed budgets are persisted and measured with a monotonic clock: changing the host clock cannot expire active work, and control-plane downtime does not consume the budget. Engine driver-supported failure compensation has a separate six-hour observed-time window; expiration requires manual recovery. Tracking deadlines do not discard desired state or halt cleanup. If deletion later finishes, its failed tracking record is corrected to `Succeeded` atomically with cluster removal or suspension; startup also repairs historical failed deletion records whose original cluster is demonstrably absent.
Observed-time accounting includes waits between reconciliations and persists each observation before reloading tracking state. Deletion completion uses an explicit persisted cluster-incarnation equality token, not ordering of host-clock timestamps. For legacy tracking without that token, startup proves completion only when the same-named cluster is actually absent.
RabbitMQ rolling updates treat a refused agent reload connection and an explicitly booting broker observation as scheduled startup waits, including after the five-minute startup diagnostic threshold. These waits consume the operation’s observed-time budget, not its consecutive-failure budget. The member’s applied configuration and native cluster membership must still be verified before the rollout advances or succeeds.
Cancellation requires `clusters/update` on the operation’s recorded namespace/cluster. Pending work becomes `Cancelled` immediately, without reversing accepted spec changes. Running work is cancellable only at engine driver-supported safe checkpoints: acceptance sets `cancelRequested`, and cleanup may keep it Running. Terminal/unsafe work returns `409`. Cancellation is not rollback; wait for a terminal outcome before dependent actions. Checkpoint rules live in [MongoDB operations](/docs/mongodb/operations/) and [backups](/docs/mongodb/backups/), [RabbitMQ operations](/docs/rabbitmq/operations/), and [Elasticsearch operations](/docs/elasticsearch/operations/).
## Resources and autoscaling
[Section titled “Resources and autoscaling”](#resources-and-autoscaling)
| Setting/observation | Contract |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `cpuMode` | `SHARED` or `DEDICATED` |
| CPU bounds | Whole cores, at least 1; optional `startCpu` is whole and within bounds |
| RAM/disk bounds | GiB in the API; each maximum must be at least its minimum; component floors are engine driver-specific |
| Persistent disk | Existing volume grows; decreasing either desired disk bound is rejected |
| Member `autoscaling` | Effective runtime service bounds read from Zerops, not desired spec |
| Member `resources` | CPU cores, memory limit and root disk the container currently has, read live by the member agent; falls back to effective autoscaling (start or minimum CPU, minimum RAM/disk) when the agent is unreachable or predates the reading |
| `diskTotalBytes` | Mounted data-volume capacity; runtime autoscaling disk fields describe disposable root disk |
PostgreSQL (including the shared Supabase database core) reports effective autoscaling bounds for both data members and proxies when the backend supplies them (Zerops). These are read on each observation, not from the last applied member configuration. Nomad and Dokploy scheduler reservations/limits are not autoscaling observations and are not used as allocation fallbacks. Live agent allocation readings take precedence; an absent runtime has no effective bounds to report.
Disk admission is provider/component-specific, not a universal 5-GiB API floor. Explicit engine disk bounds accept values from 1 GiB at the HTTP schema layer; provider validation then enforces these floors for create, replacement and patch:
| Component | Minimum desired disk (GiB) |
| ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------- |
| [PostgreSQL](/docs/postgres/) / Supabase data; [MySQL/PXC](/docs/mysql/) data | 5 |
| [MongoDB](/docs/mongodb/) data, config servers and routers; [RabbitMQ](/docs/rabbitmq/); [OpenBao](/docs/openbao/) | 5 |
| [Elasticsearch](/docs/elasticsearch/) data/combined nodes and dedicated masters | 5 |
| [Neo4j](/docs/neo4j/); [Weaviate](/docs/weaviate/) | 1 persistent |
| [Keycloak](/docs/keycloak/); enabled proxies, Supabase services, [Temporal](/docs/temporal/) roles and Elastic optional components | 1 runtime |
Each maximum must remain at least its minimum. Omitted bounds use provider defaults, which may be larger. Zerops separately rounds imported persistent Local Storage volumes up to its 5-GB minimum; this does not raise the runtime disk floor. Nomad host volumes and Dokploy named volumes do not enforce requested capacity as a quota, and do not support disk growth. See [providers](/docs/providers/#capability-differences). These are validation floors, not workload sizing recommendations.
```sh
karactl -n clusters scale --min-cpu 1 --max-cpu 2 --wait
```
Use engine driver-supported `--component` and membership options from the engine guide, or a spec patch. In **Topology → Review & apply**, inspect **What will change** and **Merge patch to be sent**. Resource changes use per-member disruption precautions; even CPU-only changes can restart a runtime. Persistent capacity is configured on the volume, not the runtime root disk.
`Ready` plus matching generation means reconciliation completed, not that allocation changed instantaneously or provides a hard memory-allocation cap. CPU-mode rate limits surface `PlatformCooldown` with a persisted target and retry deadline; reconciliation resumes automatically, including after control-plane restart. The cluster remains `Updating` until the requested mode is applied. Scheduled cooldown waits do not emit repeated `ReconcileError` events; independent observation failures still log/emit them and use normal error backoff rather than waiting for the cooldown to expire.
Consecutive identical `ReconcileError` messages with the same actor and operation are recorded once per cluster worker. A changed message, actor or operation, or successful reconciliation resets suppression; a control-plane restart starts a new worker.
Rejected [Zerops](https://zerops.io/) resource updates and service imports remain authoritative. The `ReconcileError` event and cluster status message identify the service hostname, platform error code and safe validation details, followed by attempted CPU mode, CPU/start CPU, RAM, disk and container bounds. These values come from the actual platform request, including preserved settings, not only the cluster patch. Diagnostics never include credentials or environment values.
| Diagnostic | Limit / lifecycle |
| ------------------------ | -------------------------------------------------------------------------------------------- |
| Cluster status message | 512 bytes; clears the reconciliation failure after successful observation and reconciliation |
| Error/event details | 1024 bytes; long validation details or multi-service imports may be truncated |
| Waiting update operation | Retains its last error while waiting for convergence |
When a waiting PATCH does not converge, inspect `karactl clusters get -o yaml`, cluster events and the update operation’s `message` and last error (`error`). Do not independently change Zerops autoscaling or restart all members during reconciliation.
## Rolling restart
[Section titled “Rolling restart”](#rolling-restart)
```sh
karactl -n clusters restart --wait
```
Restart persists per-member progress and uses engine driver health/quorum gates before advancing. It is not a control-plane redeploy or agent executable upgrade; agents receive a newer executable on a engine driver-driven member deployment. A singleton is unavailable during its restart; multi-member rolling work is not a zero-disruption guarantee. For engine-specific order, drain and recovery rules, see [MongoDB](/docs/mongodb/operations/), [RabbitMQ](/docs/rabbitmq/operations/), [Elasticsearch](/docs/elasticsearch/operations/) and [Supabase](/docs/supabase/operations/#cluster-restart-and-member-software). Supabase includes every enabled application component after the [PostgreSQL](https://www.postgresql.org/) core, verifying each live instance’s process restart and readiness.
## Retry exhausted platform deployments
[Section titled “Retry exhausted platform deployments”](#retry-exhausted-platform-deployments)
`DeploymentRetriesExhausted` is terminal: normal reconciliation and control-plane restarts do not reset the exact-version deployment retry limit. Investigate and correct the platform failure before explicitly granting a fresh budget:
```sh
karactl clusters retry-deployments --namespace
karactl clusters retry-deployments --namespace -o json
```
The action requires `clusters/update`, atomically advances the server-managed `spec.retryEpoch` and desired generation, records the actor, old/new epochs and exhausted members/components in an event, and triggers reconciliation. It does not restart healthy members. If no deployment is exhausted in the current epoch, it returns `409 conflict` without changing the epoch. Reuse an `Idempotency-Key` when retrying the API request to replay its original result without granting another budget. Ordinary spec edits cannot reset or advance this epoch.
## Delete or suspend
[Section titled “Delete or suspend”](#delete-or-suspend)
**Warning:** default deletion removes runtimes, data volumes and owned recovery objects. Without retained volumes it cannot be resumed. Retention flags cannot recover already-deleted assets.
| `keepVolumes` | `keepBackups` | Result |
| --------------- | --------------- | --------------------------------------------------------------- |
| false (default) | false (default) | Full deletion, including owned backup cleanup |
| false | true | Cluster/volumes removed; recovery inventory retained |
| true | false | `Suspended`; volumes/identity/secrets retained, backups cleaned |
| true | true | `Suspended`; volumes and recovery objects retained |
```sh
# Suspend while retaining both assets; there is no separate suspend command.
karactl -n clusters delete --keep-volumes --keep-backups --wait --yes
karactl -n clusters resume --wait
# Destructive full deletion:
karactl -n clusters delete --wait --yes
```
Suspended member allocation is historical; agents are disconnected. Resume recreates runtimes on retained volumes. Normal scale-down/replacement deletes retired members’ volumes; cluster retention flags are not per-member retention.
Deletion needs `clusters/delete`; resume, patch/edit/scale/restart need `clusters/update`. Read/create need `clusters/read` and `clusters/create`, respectively, on namespace-qualified cluster objects. See [Access control](/docs/access-control/).
Running deletion cannot be cancelled. Repeating it with identical retention choices reuses the active operation. Before engine driver cleanup starts, changed retention choices supersede queued deletion tracking. A retain-volumes cleanup can be superseded by full deletion; stale suspension completion cannot clear that newer request. Once engine driver cleanup has started with removal of volumes or backups permitted, changing the corresponding flag to retain them returns `409`: already-issued deletions cannot safely promise retention. The cleanup-start fence is durable across control-plane restarts. Check cluster, backup and orphan-volume inventories separately. See [Backups and storage](/docs/backups/) for retained access and cleanup.
# Contributor Covenant Code of Conduct
> We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body…
## Our Pledge
[Section titled “Our Pledge”](#our-pledge)
We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
## Our Standards
[Section titled “Our Standards”](#our-standards)
Examples of behavior that contributes to a positive environment for our community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others’ private information, such as a physical or email address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a professional setting
## Enforcement Responsibilities
[Section titled “Enforcement Responsibilities”](#enforcement-responsibilities)
Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
## Scope
[Section titled “Scope”](#scope)
This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
## Enforcement
[Section titled “Enforcement”](#enforcement)
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at . All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the reporter of any incident.
## Enforcement Guidelines
[Section titled “Enforcement Guidelines”](#enforcement-guidelines)
Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
[Section titled “1. Correction”](#1-correction)
**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
### 2. Warning
[Section titled “2. Warning”](#2-warning)
**Community Impact**: A violation through a single incident or series of actions.
**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
### 3. Temporary Ban
[Section titled “3. Temporary Ban”](#3-temporary-ban)
**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
[Section titled “4. Permanent Ban”](#4-permanent-ban)
**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the community.
## Attribution
[Section titled “Attribution”](#attribution)
This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 2.1, available at .
Community Impact Guidelines were inspired by [Mozilla’s code of conduct enforcement ladder](https://github.com/mozilla/diversity).
For answers to common questions about this code of conduct, see the FAQ at . Translations are available at .
# Contributing to Karakoram
> Contribute to Apache-2.0 orchestration for MongoDB (Percona Server/PBM), RabbitMQ and Elasticsearch on Zerops.
Contribute to Apache-2.0 orchestration for [MongoDB](https://www.mongodb.com) ([Percona Server](https://www.percona.com/mongodb/software/percona-server-for-mongodb)/[PBM](https://docs.percona.com/percona-backup-mongodb/)), [RabbitMQ](https://www.rabbitmq.com) and [Elasticsearch](https://www.elastic.co/elasticsearch) on [Zerops](https://zerops.io).
Read [architecture](/docs/architecture/), [ADRs](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/README.md), [security](/docs/security/) and [limitations](/docs/limitations/). No [Kubernetes](https://kubernetes.io) dependencies or implied vendor/[OpenEverest](https://openeverest.io) affiliation or endorsement.
## Conduct and contribution process
[Section titled “Conduct and contribution process”](#conduct-and-contribution-process)
Follow the [Code of Conduct](/docs/code-of-conduct/): be respectful, critique ideas rather than people, keep discussion constructive and respect reviewers’ time. Harassment and discriminatory behavior are not accepted. Report conduct concerns to ; maintainers may remove content or block participants.
1. Search existing issues. For changes beyond a small fix, open an issue to agree on the problem/approach; architectural changes need an ADR update.
2. Create a topic branch and keep each pull request to one logical change.
3. Use clear commit messages: summary, blank line, then what changed and why.
4. Update affected code, callers, tests and documentation together; remove obsolete paths and run the applicable [checks](#checks).
5. Sign off every commit. Maintainers review pull requests and decide acceptance.
Every commit requires a `Signed-off-by: Your Name ` line under the [Developer Certificate of Origin 1.1](https://developercertificate.org/). `git commit -s` adds it. This certifies your right to contribute under Apache-2.0; use your real name, not anonymous/bot sign-offs.
## AI/LLM-assisted contributions
[Section titled “AI/LLM-assisted contributions”](#aillm-assisted-contributions)
AI assistance is allowed, but a human must submit, understand and take responsibility for every line, including correctness, security and licensing. Automated/bot-driven pull requests and unreviewed generated output are not accepted. Large/complex reconciliation, data-safety, backup/restore and security logic must be authored and fully understood by the contributor.
Disclose meaningful AI use in the pull request: tool and purpose. Check generated material for incompatible licenses, secrets and private data. Private AI self-review is allowed; do not post AI review bots into human review threads. Maintainers may request clarification or decline contributions. This policy is adapted from [Cloud Hypervisor’s guidelines](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/CONTRIBUTING.md#aillm-assistance--generated-code).
Coding agents must also read [AGENTS.md](https://github.com/KarakoramDev/Karakoram/blob/main/AGENTS.md) for repository layout, commands and invariants.
## Development setup
[Section titled “Development setup”](#development-setup)
Requirements: **[Go](https://go.dev) 1.26+**, [Make](https://cgit.git.savannah.gnu.org/cgit/make.git/), **[Node](https://nodejs.org) 22.6+** and [npm](https://docs.npmjs.com/). The UI test runner uses Node’s built-in [TypeScript](https://www.typescriptlang.org) support. Agent integration tests require [Docker](https://www.docker.com) and [Linux](https://kernel.org) host networking; unit tests need no Zerops account.
```sh
make ui
make build
make test
```
Binaries are `bin/karakoram` and `bin/karactl`. `make build-all` builds UI assets first. Plain Go builds without existing UI output do not embed the SPA. Build metadata uses Make’s `VERSION`, `COMMIT` and `BUILD_DATE` variables.
`make build` is for development. Publish artifacts with `make release`, not by copying `bin/`; see [Releasing](#releasing).
For frontend development, run `npm run dev` in `ui/`: [Vite](https://vite.dev) listens on 5173 and proxies API requests to localhost:8080. `KARAKORAM_API_URL` selects another provider; see [ui/README.md](https://github.com/KarakoramDev/Karakoram/blob/main/ui/README.md). Local server configuration is defined in [internal/server/config.go](https://github.com/KarakoramDev/Karakoram/blob/main/internal/server/config.go): use a persistent data directory, master key, public origin and real provider settings. Do not commit populated environment files or substitute fake production credentials.
## Releasing
[Section titled “Releasing”](#releasing)
Maintainers release from `main` for major changes, security fixes or accumulated fixes—not every commit. Pushes do not automatically create tags/releases.
Use `vMAJOR.MINOR.PATCH`: before 1.0, MINOR for features/breaking changes and PATCH for fixes/security; from 1.0, MAJOR for breaking changes, MINOR for compatible features, PATCH for fixes. Document breaking changes/upgrade steps. Prereleases such as `-rc.1` are allowed; deploy them only deliberately.
Default Zerops deploy/run configuration must remain compatible with the binary pinned in `deploy/zerops/RELEASE`, even on newer `main`. Ship binary-dependent configuration with its release or use `karakoram-source`. **Never edit `RELEASE` by hand:** the release tool writes its single SemVer tag.
### Prerequisites and release notes
[Section titled “Prerequisites and release notes”](#prerequisites-and-release-notes)
* Development tools above, [Bash](https://www.gnu.org/software/bash/), [Git](https://git-scm.com), [GNU tar](https://cgit.git.savannah.gnu.org/cgit/tar.git/)/[coreutils](https://www.gnu.org/software/coreutils/) (`date`, `sha256sum`), [gzip](https://cgit.git.savannah.gnu.org/cgit/gzip.git/), [OpenSSH](https://www.openssh.com) `ssh-keygen` supporting `-Y sign`/`-Y verify`.
* Publication: `gh` authenticated with public-repository release rights; Git push rights for `origin` tags and `main`.
* Signing key: `~/.config/karakoram/release-signing-ed25519`, overridden by `KARAKORAM_RELEASE_SIGNING_KEY`. Authorize its public key in [`deploy/allowed_signers`](https://github.com/KarakoramDev/Karakoram/blob/main/deploy/allowed_signers) for `release@karakoram.dev`; never commit the private key.
* Write `NOTES` outside the checkout (or commit before preparation): changes, security fixes, upgrade steps, state-schema/rollback compatibility. Notes are not generated. Review changes and run applicable [checks](#checks); release targets do not replace them.
### Prepare, build, and publish
[Section titled “Prepare, build, and publish”](#prepare-build-and-publish)
```sh
make release-prepare VERSION=v0.1.2
make release-build VERSION=v0.1.2
make release VERSION=v0.1.2 NOTES=../release-notes-v0.1.2.txt
```
| Target | Requirements and result |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `release-prepare` | Clean tree, `main` equal to `origin/main`, tag absent locally/remotely, version newer than pin. Detaches HEAD, writes pin, creates signed-off `Release v0.1.2` commit and annotated tag. Local `main` unchanged; no push. Review commit/tag. |
| `release-build` | Tag must exist and pin itself; signing key required. Dry run exports tag with `git archive`, builds UI once, server for Linux amd64/arm64 and CLI for Linux/macOS amd64/arm64 + Windows amd64. Writes server archives (binary/LICENSE/NOTICE), raw CLI binaries, `checksums.txt` and SSH signature to `dist/v0.1.2/`; verifies signature/all checksums. No publication/push. |
| `release` | Clean tree, HEAD at tag, `origin/main` ancestor; conflicting remote tag rejected. Rebuilds, pushes tag, creates draft from `NOTES`, downloads every asset, compares bytes and verifies signature/checksums. Then publishes (latest, or prerelease for suffixed tags) and fast-forwards `origin/main`, never pinning unpublished assets. |
Rerun the same publication command after failure: unfinished drafts are recreated; published releases are re-verified before fast-forwarding `main`. Afterwards:
```sh
git checkout main && git merge --ff-only origin/main
```
Reproducibility: `CGO_ENABLED=0`, `-trimpath`, `-buildvcs=false`, full tagged commit SHA and its UTC timestamp. Archive paths/order/owner/group/timestamps are normalized; gzip omits timestamps. Identical checksums require the **exact same Go toolchain**, tagged source and locked npm dependencies, not merely a compatible Go version. Inspect with `go version -m `.
### Signing-key rotation and verification
[Section titled “Signing-key rotation and verification”](#signing-key-rotation-and-verification)
Commit the new public key to `deploy/allowed_signers` before preparing/signing with it. Retain old keys for old releases; use OpenSSH `valid-after`/`valid-before` for validity windows. Distribute trust-file updates through a trusted channel.
Verify a complete local release:
```sh
RELEASE=v0.1.2
(cd "dist/$RELEASE" &&
ssh-keygen -Y verify -f ../../deploy/allowed_signers \
-I release@karakoram.dev -n karakoram-release \
-s checksums.txt.sig < checksums.txt &&
sha256sum --strict -c checksums.txt)
```
For downloads, [authenticate the signing key, verify the manifest signature, then the selected binary/archive checksum](/docs/getting-started/install/#install-karactl) before use.
## Code and change boundaries
[Section titled “Code and change boundaries”](#code-and-change-boundaries)
| Area | Responsibility |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `internal/server`, `internal/api` | Runtime wiring, strict HTTP contract, handlers and authorization |
| `internal/domain`, `internal/store` | Models, [SQLite](https://sqlite.org) migrations, durable intent/checkpoints/events/secrets |
| `internal/reconciler` | Serialized lifecycle/operations, observation and cleanup |
| `internal/engine` | In-process engine driver registry and reconciliation/data-plane interfaces |
| `internal/engine/psmdb`, `internal/engine/psmdb/pbm` | MongoDB topology/access/TLS and PBM backup/PITR/restore |
| `internal/engine/rabbitmq` | Broker lifecycle, Management API access and definitions recovery |
| `internal/engine/elastic` | Elasticsearch/Stack lifecycle, native security and snapshot recovery |
| `internal/infra/zerops`, `internal/namespaces`, `internal/relay` | Zerops wire formats, project lifecycle and relay transport |
| `internal/agent`, `internal/agentapi` | Engine process supervision/rendering and private protocol |
| `internal/auth`, `internal/rbac` | Identity, sessions/tokens and policy |
| `internal/scheduler`, `internal/events`, `internal/monitoring` | Schedules/retention, durable activity/SSE and metrics |
| `internal/cli`, `ui/`, `internal/ui` | CLI, typed [React](https://react.dev) application and embedded assets |
* Keep generic durability in reconciler/store; engine mechanisms stay within their engine driver; Zerops wire formats stay in the provider adapter.
* Engine driver metadata owns versions, topology constraints and UI defaults. Do not create a second CLI/UI catalog. Engine drivers are compiled in, not dynamically installable plugins.
* Perform at most one disruptive action per reconciliation step. Persist intent/checkpoints and remote identifiers before progressing, then observe completion. SQLite and remote services are not one transaction.
* Preserve agent config/control/receipt compatibility through rolling deployments. Maintenance/process holds survive restarts and pulls; ambiguous initiation is not permission to repeat destructive work.
* Engine tests must prove that an unchanged, Ready cluster submits no infrastructure mutations. Reuse `internal/engine/enginetest.AssertIdle` with the engine’s native/agent fixture: it uses the real Zerops layout and capabilities, round-trips cluster state through SQLite/JSON on every reconciliation, and requires twenty converged cycles with zero backend mutation submissions. Explicit `startCpu` values exercise decoded-pointer equality. Keep every supported engine/topology covered by `TestIdleReconcileZeroProviderMutations`; the recorder contract test guards the infrastructure mutation surface.
* Ordinary status/log/event reads must remain credential-free. Preserve separate `cluster-credentials:read`, per-object filtering and deny-overrides policies.
## Public API changes and regeneration
[Section titled “Public API changes and regeneration”](#public-api-changes-and-regeneration)
1. Edit [api/openapi.yaml](https://github.com/KarakoramDev/Karakoram/blob/main/api/openapi.yaml): schemas, `x-karakoram-rbac`, write-only secrets and errors.
2. Regenerate Go server/client/RBAC and TypeScript artifacts; do not hand-edit generated files.
3. Migrate handlers, CLI, UI and tests together. When CLI commands change, update the command-group table in [docs/cli.md](/docs/cli/) from recursive `karactl --help`.
4. Run the contract drift check with UI dependencies installed.
From the repository root:
```sh
make generate
(cd ui && npm ci && npm run gen:api)
make openapi-check
```
Go generation uses pinned [oapi-codegen](https://github.com/oapi-codegen/oapi-codegen) 2.4.1 for `internal/api/gen.go` and `pkg/client/client.gen.go`, plus the RBAC generator for `internal/api/rbacmap.gen.go`. TypeScript generation writes `ui/src/api/schema.d.ts` using lockfile-pinned [openapi-typescript](https://openapi-ts.dev). Keep handwritten public client helpers separate from generated files and independent of `internal/`; external consumers import `pkg/client`. See [Clients/SDK](/docs/api/#clientssdk).
`make openapi-check` rejects YAML aliases and regenerates/compares Go contract/client/RBAC and TypeScript files; it is a drift check, not live behavior verification. If editing introduces YAML anchors/aliases, normalize without changing meaning:
```sh
go run ./internal/tools/specnoalias
go run ./internal/tools/specnoalias -check
go generate ./api/...
(cd ui && npm run gen:api)
```
Normalization preserves existing component-response references; `-check` fails if anchors/aliases remain.
## Checks
[Section titled “Checks”](#checks)
```sh
make test
make test-integration
make lint
make openapi-check
```
Integration tests use real database containers; read their prerequisites. Skipped tests are not evidence of passing behavior. `make lint` runs Go vet, [staticcheck](https://staticcheck.dev) 2025.1.1 and UI typecheck/[ESLint](https://eslint.org); install UI dependencies first.
Live tests are opt-in and destructive to their resources:
```sh
# Securely load KARAKORAM_E2E_URL and KARAKORAM_E2E_TOKEN first.
make e2e
```
Follow [test/e2e/README.md](https://github.com/KarakoramDev/Karakoram/blob/main/test/e2e/README.md) for [S3](https://aws.amazon.com/s3/), IPv6 allowlist, CLI and [OIDC](https://openid.net/developers/how-connect-works/) settings. Use an isolated installation in a dedicated project: the suite mutates global policy/settings, so never run against production or concurrently with another live mutation suite. Do not modify the platform’s `core` service. Report scenarios exercised and results in the pull request.
## Security and licensing
[Section titled “Security and licensing”](#security-and-licensing)
Never commit credentials, master keys, passwords, private PEMs, signed upload/log URLs or populated secret fixtures. Protect CLI configuration/state archives as credentials; use synthetic fixtures, redacted reports and no shell tracing around secrets.
New source files need `SPDX-License-Identifier: Apache-2.0` and `Copyright 2026 The Karakoram Authors` comments. Verify incorporated code’s license compatibility and retain required headers/NOTICE attribution. Apache-2.0 grants no third-party trademark rights.
Report vulnerabilities privately through [security reporting](/docs/security/#reporting-vulnerabilities), : include impact, revision, prerequisites and a minimal redacted reproduction, not public exploit details or secrets.
# Control-plane backup and rebuild
> Control-plane backups preserve SQLite coordinator state, not database-engine data. Continue using database backups for application data. The control plane…
Control-plane backups preserve [SQLite](https://sqlite.org/) coordinator state, not database-engine data. Continue using [database backups](/docs/backups/) for application data. The control plane remains a single-replica service.
This Ops feature is **verified offline** (tests without a live provider). A successful state snapshot does not establish engine-data recovery or a provider rebuild recovery-time guarantee.
## Policy and runs
[Section titled “Policy and runs”](#policy-and-runs)
An administrator with `settings:read` can inspect policy and runs; changing policy and starting a run require `settings:update`.
```sh
karactl control-plane backup-policy set --enabled --storage production/state-archive \
--schedule '0 3 * * *' --retain-count 14
karactl control-plane backup-policy get -o yaml
karactl control-plane backups run -o json
karactl control-plane backups list --limit 100 -o yaml
```
`--storage` accepts `[namespace/]name`; without a namespace it uses the selected namespace, or `default`. The storage must already exist and be [S3](https://aws.amazon.com/s3/)-compatible; verification is recommended but not required. A policy-referenced storage cannot be deleted. `set` replaces policy, rather than patching it; defaults are disabled, `0 3 * * *`, and 14 retained successful backups. Supply all desired fields, or use an API-shaped `-f` file. Retention is 1–1000.
To reset to the initial unconfigured state, PUT a policy with `enabled: false` and `storage: {"namespace":"default","name":""}` (or use an API-shaped `-f` file). An enabled policy always requires an existing S3 storage; a nonempty storage reference is validated even when disabled. Resetting the policy does not remove existing backup records or artifacts. Manual runs require storage.
Schedules use five-field UTC cron with at most one catch-up. Manual runs use the same controller, storage policy and retention as scheduled runs. Only one backup executes at a time. Records progress through Pending, Running, Succeeded or Failed. Inspect the completed record: acceptance of `run` is not evidence of a usable backup. Failures emit `ControlPlaneBackupFailed` warning events.
A run interrupted while Running is marked Failed after control-plane restart, without automatic retry. Review the record and start a new manual run when needed.
## Object layout and custody
[Section titled “Object layout and custody”](#object-layout-and-custody)
Objects are stored under:
```text
/karakoram-control-plane//-.db.gz
/karakoram-control-plane//-.json
```
The timestamp is `YYYYMMDDTHHMMSSZ`. The JSON manifest contains `format: karakoram-state/v1`, `installationId`, `createdAt`, `karakoramVersion`, `commit`, `schemaVersion` (last migration filename), `keyFingerprint`, `sha256` and `sizeBytes`. SHA-256 and size cover the compressed archive. Retention removes archive/manifest pairs only for successful runs belonging to this installation, oldest first beyond the configured count.
A snapshot includes desired state, inventory, operation journals, accounts, policies, installation identity, sealed secrets and PKI. Secrets and private keys remain sealed using the original master key. **The master key is not in the backup.** Preserve `KARAKORAM_MASTER_KEY` independently, securely, and test recovery. The fingerprint detects a wrong key; it is not a substitute for a strong master key or trusted backup custody. Hashes detect corruption, not malicious replacement of both archive and manifest.
SQLite `VACUUM INTO` snapshots every table, including `postgres_bindings` and its source-incarnation/dependency metadata, not a hand-maintained table allowlist. Desired options, internal lifecycle journals and sealed cluster secrets for all registered engines are included, including [Keycloak](https://www.keycloak.org/), [OpenBao](https://openbao.org/), [Temporal](https://temporal.io/), [MySQL](https://www.oracle.com/mysql/), [PXC](https://www.percona.com/mysql/software/percona-xtradb-cluster), [Neo4j](https://neo4j.com/) and [Weaviate](https://weaviate.io/). These records do not include those engines’ external data or volumes.
The snapshot includes API-token hashes and immutable restriction metadata, [OAuth](https://oauth.net/2/) grant/token hashes, and sealed Data [MCP](https://modelcontextprotocol.io/) companion state/configuration. Raw API-token bearer secrets are not stored; Data MCP credentials stay in their existing sealed cluster-secret records. Snapshots are complete installation recovery artifacts, not restricted-token exports: protect backup storage and local files as installation-wide sensitive material. Restoring an older snapshot also restores its token revocation/expiry state; review and revoke stale tokens before reopening access. Export and restore are local administrator commands, not remotely exposed MCP tools; offline replacement requires explicit `--replace`.
The backup does **not** include engine volumes/data, binaries, deployment definitions, environment-supplied provider tokens, endpoint/public/advertise configuration or the master key. Independently preserve required environment and provider configuration. A sealed namespace credential stored in SQLite is restored, but that does not replace environment-supplied provider credentials.
## Local export and offline restore
[Section titled “Local export and offline restore”](#local-export-and-offline-restore)
Use the same data-directory and master-key environment as the server. Do not put keys in shell arguments or logs.
```sh
karakoram admin backup-state --out /secure/state.db.gz
# Produces /secure/state.db.gz and /secure/state.db.gz.json, both mode 0600.
```
Local export works while the server runs: it authenticates the supplied master key against persisted encrypted state before snapshotting or publishing a fingerprint, then uses a transactionally consistent WAL-safe SQLite snapshot and checks integrity. A fresh database with no encrypted state can be exported with its configured key. Preserve both files together.
Stop the control-plane service and disable automatic restarts before restoring. Set `KARAKORAM_DATA_DIR` to the destination and supply the original `KARAKORAM_MASTER_KEY` through your secret manager. Restore before first normal startup:
```sh
karakoram admin restore-state --file /secure/state.db.gz
```
For S3, use the archive object key, not the manifest key:
```sh
# Inject KARAKORAM_RESTORE_S3_ACCESS_KEY_ID and
# KARAKORAM_RESTORE_S3_SECRET_ACCESS_KEY securely into the environment.
karakoram admin restore-state --s3-endpoint https://s3.example.com \
--bucket state-backups --key prefix/karakoram-control-plane/INSTALLATION/TIMESTAMP-ID.db.gz \
--region us-east-1 --path-style
```
Only those two restore-specific environment variables supply S3 credentials. `--region` and `--path-style` are optional. Restore fetches the sibling `.json` object (replacing `.db.gz`), using the same guarded endpoint policy as backup storage: no redirects/proxy bypass, TLS verification, forbidden metadata/loopback destinations, and `KARAKORAM_BACKUP_ENDPOINT_ALLOWLIST` for permitted private endpoints. Do not weaken endpoint safety to recover an untrusted archive.
Restore verifies format, compressed size/hash, gzip, SQLite integrity, schema compatibility and matching key fingerprint before publishing. A mismatch reports both fingerprints and requires the original master key. A binary cannot restore a snapshot from a newer schema; use a compatible or newer release.
A running/open database is refused. An existing database is refused unless `--replace` is given. With replacement, SQLite first checkpoints the old database into a standalone main file and removes its WAL/SHM sidecars; failure to do so refuses replacement. A hard link preserves the old database as `karakoram.db.replaced-YYYYMMDDTHHMMSSZ`, and that backup directory entry is fsynced before publication. The authoritative `karakoram.db` name is never removed: a private, validated and fsynced temporary file atomically replaces it, followed by directory fsync. Interruption before publication leaves the old authoritative database, not an absent path that could initialize empty state. Keep replacement files until recovery is verified. Local archive and manifest are individually atomically published; if export is interrupted, do not use a mismatched pair.
## Provider rebuild runbooks
[Section titled “Provider rebuild runbooks”](#provider-rebuild-runbooks)
For every provider: isolate the old coordinator, retain engine services/volumes, redeploy only the control plane, restore state **before its first start**, then start exactly one coordinator with the same master key. Avoid having old and restored coordinators reconcile simultaneously. Preserve the original endpoint/advertise configuration and connectivity to members, object storage and provider APIs. Review readiness, events, operations and credentials after startup before enabling routine changes.
### Zerops
[Section titled “Zerops”](#zerops)
1. Stop or isolate the old control-plane service; do not delete database member services or their storage.
2. Redeploy the control-plane service with a persistent data mount and a compatible binary. Keep normal startup disabled while preparing recovery.
3. Restore the original master key, [Zerops](https://zerops.io/) provider token/project configuration and public/advertised endpoints from independent secret/configuration custody. These environment settings are not recovered by restoring SQLite.
4. Run offline restore against the mounted data directory, then enable normal startup. Check namespace connectivity and existing member trust before accepting mutations.
### Nomad
[Section titled “Nomad”](#nomad)
1. Stop the old control-plane allocation/job without purging member jobs or CSI volumes.
2. Redeploy the control-plane job with one allocation, the intended persistent state volume and compatible binary; use a one-shot recovery task or an allocation shell before enabling the server task.
3. Reinstall the same master key and the independently retained [Nomad](https://developer.hashicorp.com/nomad) endpoint, namespace/region, ACL token, TLS and advertised-address configuration; see [Nomad](/docs/nomad/).
4. Restore on the mounted volume, remove the recovery override and start one server allocation. Verify provider access and member connectivity before resuming automated work.
### Dokploy
[Section titled “Dokploy”](#dokploy)
1. Stop the old control-plane application/container and prevent automatic restart; retain database applications/services and volumes.
2. Redeploy only the control-plane application with its persistent data volume and compatible binary, temporarily overriding normal server startup for recovery.
3. Reapply the original master key and independently saved [Dokploy](https://dokploy.com/) API token, endpoint and deployment/public-address configuration; see [Dokploy](/docs/dokploy/).
4. Run offline restore on the mounted state directory, restore the normal command, then start one replica. Review application ownership and member reachability before permitting changes.
## Stale snapshots
[Section titled “Stale snapshots”](#stale-snapshots)
Restoring old state does not discover resources created after the snapshot: those resources become orphans and need operator inventory/reconciliation. Do not blindly recreate or delete them. In-flight operations can resume from their journals or fail because infrastructure changed since the snapshot. Review pending/running/failed work, provider resources and events before retrying mutations. Restoring state does not roll engine data back to snapshot time; perform engine-specific data recovery separately when required.
# Cluster data MCP
> Data MCP gives an MCP client tools for the data inside one cluster, not authority to create, restore or administer clusters. It is experimental in this…
Data [MCP](https://modelcontextprotocol.io/) gives an MCP client tools for the **data inside one cluster**, not authority to create, restore or administer clusters. It is **experimental** in this release: the interfaces and reviewed tool catalogue can change between minor versions. Each cluster has its own Streamable HTTP endpoint:
```text
https:///mcp/clusters/{namespace}/{cluster}
Authorization: Bearer
```
It uses the existing public listener; no public database port or companion port is needed. [MongoDB](https://www.mongodb.com/)/PSMDB uses `mongodb-mcp-server`, [PostgreSQL](https://www.postgresql.org/) and [Supabase](https://supabase.com/) use `pgedge-postgres-mcp`, and [Elasticsearch](https://www.elastic.co/elasticsearch) and [RabbitMQ](https://www.rabbitmq.com/) use native toolsets. See the [design decision](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0043-data-mcp-gateway.md), [agent access security model](/docs/security/#agent-access-mcp-security-model), [access control](/docs/access-control/) and the [MCP quick start](/docs/mcp-quickstart/#5-data-mcp-quick-start).
PostgreSQL extension profiles retain Data MCP support on single and HA topologies. Distributed PostgreSQL ([Citus](https://www.citusdata.com/)) rejects `spec.dataMcp.enabled: true` at admission because its group-aware access model is not supported by the PostgreSQL companion. [Keycloak](https://www.keycloak.org/) does not support Data MCP.
Data endpoints support stateless MCP `2026-07-28` and legacy initialize-based Streamable HTTP (`2025-11-25`, `2025-06-18`, `2025-03-26`), for both companion-backed and native toolsets. Companion connections negotiate an initialize-based revision independently of the client’s protocol; they do not forward the client’s protocol version.
Tool lists are caller-specific. On MCP `2026-07-28`, `tools/list` returns `cacheScope: "private"` and `ttlMs: 0`; clients must re-fetch to reflect current authorization and cluster tool filtering.
## Enable and scope access
[Section titled “Enable and scope access”](#enable-and-scope-access)
**Off by default:** an absent `spec.dataMcp` or `enabled: false` disables the endpoint and removes its managed data identities and companion when no longer needed. Disabled configurations can retain their settings for later use. Installation policy can also disable all [MCP](https://modelcontextprotocol.io/) access with `KARAKORAM_MCP=off`; the installation default is `readonly`, not permission to enable any cluster automatically.
For example, this cluster merge patch enables read-only access to one [MongoDB](https://www.mongodb.com/) database:
```yaml
spec:
dataMcp:
enabled: true
access: read
scope:
databases: [app]
tools:
deny: [mongodb-logs]
confirmWrites: true
auditQueryText: false
```
Apply it through the existing [cluster create/edit/patch workflow](/docs/clusters/#create-and-change-desired-state). Cluster changes require cluster update permission; that does not itself grant data access.
| `spec.dataMcp` field | Default and meaning |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled` | `false`; explicit opt-in |
| `access` | `read`; only `read` and `readWrite` are accepted |
| `scope.databases` | Empty; MongoDB or PostgreSQL/Supabase database names |
| `scope.schemas` | Empty; PostgreSQL/Supabase application schemas only |
| `scope.indexPatterns` | Empty; Elasticsearch index patterns only |
| `scope.vhosts` | Empty; RabbitMQ vhost names only |
| `tools.deny` | Empty; hides and refuses named tools, including write aliases |
| `tools.confirmWrites` | `false`; require confirmation for ordinary writes as well as destructive tools |
| `auditQueryText` | `false`; opt into argument text in audit details in addition to its hash |
| `resources` | Companion service resources; omitted selects shared CPU, `minCpu: 1`, `maxCpu: 1`, `minRam: 0.25`, `maxRam: 0.5` (RAM in GiB). Does not size the database or native toolsets. |
Each scope list accepts at most **64** nonblank entries of at most **256 bytes** before engine-specific validation. `tools.deny` accepts at most **64 unique**, nonblank names of at most **64 bytes**. Unknown upstream tools are denied regardless of this list. When enabling, the engine driver must support data MCP; fields for a different engine are rejected.
### Engine scope semantics
[Section titled “Engine scope semantics”](#engine-scope-semantics)
* **MongoDB/PSMDB:** only `databases`. Names must be valid UTF-8 without surrounding whitespace/control characters, fewer than 64 bytes, and exclude MongoDB’s prohibited name characters. `admin`, `local`, `config` and `$external` cannot be scoped. Empty scope grants the corresponding `readAnyDatabase`/`readWriteAnyDatabase` data role; explicit scope grants `read`/`readWrite` on the named application databases.
* **[PostgreSQL](https://www.postgresql.org/)/[Supabase](https://supabase.com/):** only `databases` and `schemas`. Identifiers must be nonempty, at most 63 bytes and contain no NUL. Schemas `information_schema` and names beginning `pg_` are rejected. Named databases and schemas must exist when grants reconcile. Empty database scope selects **`postgres`** for explicit object grants and the companion connection, **not every database**. With neither database nor schema scope, the reader additionally receives `pg_read_all_data`. Empty schema scope selects non-system schemas in the selected database(s). The companion connects only to the **first selected database**; database switching is unavailable. See the PUBLIC caveat below.
* **[Elasticsearch](https://www.elastic.co/elasticsearch):** only `indexPatterns`. Engine driver admission and native gateway routing share one rule: patterns must contain 1–255 bytes of lowercase ASCII letters, digits, `_`, `.`, `-`, `*` or `?`, and cannot begin with `.`, `-` or `_` (including `_all`). Other characters, including `+`, whitespace and non-ASCII letters, are rejected. Use simple patterns such as `app-*`. Empty scope resolves open indices, but hidden/system/dot indices remain inaccessible. Aliases and concrete backing targets are checked, not just the requested pattern.
* **[RabbitMQ](https://www.rabbitmq.com/):** only `vhosts`; names must contain 1–255 UTF-8 bytes without control characters. Empty scope grants existing vhosts, and reconciliation includes newly created vhosts. Every tool call must still name a permitted, nonempty `vhost`.
Prefer explicit, narrow scope even for read-only deployments. Scope controls managed grants and tool routing; it is not a substitute for reviewing pre-existing database privileges.
### Four switches for writes
[Section titled “Four switches for writes”](#four-switches-for-writes)
All four must allow writes; otherwise write and destructive tools are absent from `tools/list` and calls are refused:
1. Cluster `spec.dataMcp.enabled: true` and `access: readWrite`.
2. Restricted token explicitly includes `cluster-data:write` and is **not** `readOnly`.
3. Token owner currently holds RBAC `cluster-data:write` on `/` (and `cluster-data:read` to connect).
4. Installation sets `KARAKORAM_MCP=readwrite`.
A token never expands its owner’s current RBAC. Built-in viewer and operator roles do **not** include `cluster-data`; the administrator’s wildcard grant includes it. See [RBAC policy format](/docs/access-control/#policy-format).
## Connect an agent client
[Section titled “Connect an agent client”](#connect-an-agent-client)
Authenticate `karactl` to the intended installation and mint a dedicated restricted token:
```sh
karactl mcp config --cluster production/appdb
karactl mcp config --cluster production/appdb --client claude-code
karactl mcp config --cluster production/appdb --client cursor
karactl mcp config --cluster production/appdb --client vscode
# Requests write scope, but does not bypass the other three switches:
karactl mcp config --cluster production/appdb --write --client json
```
`--cluster` requires `namespace/name`; it does not use the ordinary implicit namespace. Formats are `json` (default: `url` and `headers`), `claude-code` (a `claude mcp add --transport http` command), `cursor` (`mcpServers`) and `vscode` (`servers`, with `type: http`). Each invocation creates a new token scoped to both that namespace and cluster, with `cluster-data:read`; `--write` adds `cluster-data:write` and removes the read-only restriction. The default lifetime is **90 days**; override `--expires-in` with a positive [Go](https://go.dev/) duration or whole days, such as `24h` or `90d`. `--token-name` overrides the default `mcp-data--` name.
**Output contains a credential shown only once.** Store it in protected client configuration; do not paste it into tickets, logs or source control. Revoke it through API token management when no longer needed. The bearer is a Karakoram API token, never a database password or infrastructure token.
The gateway rejects browser cookies, unrestricted/legacy tokens and missing explicit data permissions. Tokens must name at least one namespace or cluster; both restriction lists empty is denied, and every nonempty list must match the requested cluster. `Origin` must be absent or exactly match the configured public origin; [MCP](https://modelcontextprotocol.io/) sessions are bound to their token. Use the CLI’s cluster-scoped tokens rather than broad manually constructed restrictions.
## Reviewed tool catalogue
[Section titled “Reviewed tool catalogue”](#reviewed-tool-catalogue)
These are manifest-approved tools, not a guarantee that every upstream tool is available with every database grant. Tools also need upstream availability, effective access and absence from `tools.deny`. Denied and unrecognized upstream tools are never listed. Read tools always execute through the read identity, even when writes are enabled.
### MongoDB/PSMDB — `mongodb-mcp-server` 3.1.0
[Section titled “MongoDB/PSMDB — mongodb-mcp-server 3.1.0”](#mongodbpsmdb--mongodb-mcp-server-310)
| Class | Tools |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Read | `find`, `count`, `explain`, `aggregate`, `aggregate-db`, `collection-indexes`, `collection-schema`, `collection-storage-size`, `db-stats`, `list-collections`, `list-databases`, `mongodb-logs` |
| Write | `create-collection`, `create-index`, `insert-many`, `rename-collection`, `update-many` |
| Destructive | `delete-many`, `drop-collection`, `drop-database`, `drop-index` |
All approved tools force `connectionId: preconfigured`; connection overrides are refused. The companion configures a maximum of **100 documents per query** and **30,000 ms** query time, disables server-side JavaScript, telemetry, request overrides and connection/Atlas/export/knowledge features. Database grants remain authoritative: listing a tool does not confer administrative privileges.
### PostgreSQL and Supabase — `pgedge-postgres-mcp` v1.1.0 manifest
[Section titled “PostgreSQL and Supabase — pgedge-postgres-mcp v1.1.0 manifest”](#postgresql-and-supabase--pgedge-postgres-mcp-v110-manifest)
| Class | Tools |
| ----------- | ------------------------------------------------------------------------------------- |
| Read | `query_database`, `get_schema_info`, `count_rows`, `execute_explain`, `read_resource` |
| Write | None |
| Destructive | `query_database_write` |
`query_database_write` is a gateway alias for `query_database` using the write identity. **Every call to the alias requires confirmation**, because arbitrary SQL can be destructive; it is not an ordinary write-class tool. Database switching, LLM/embedding/knowledge tools are unavailable. [Supabase](https://supabase.com/) uses the dedicated [PostgreSQL](https://www.postgresql.org/) identity, not `service_role` or an RLS-bypassing application credential.
### Elasticsearch — `native-elasticsearch` manifest 1
[Section titled “Elasticsearch — native-elasticsearch manifest 1”](#elasticsearch--native-elasticsearch-manifest-1)
| Class | Tools |
| ----------- | ------------------------------------------------------------------------------------------- |
| Read | `indices_list`, `mapping_get`, `search`, `esql_query`, `count`, `shards_get`, `index_stats` |
| Write | `doc_index`, `doc_update` |
| Destructive | `delete_by_query` |
Search size defaults to **10**, capped at **100**, with a forced **5-second** server timeout and total-hit tracking capped at **10,000**. ES|QL accepts a conservative `FROM`-and-pipeline subset and appends **`LIMIT 500`**. Hidden, system and dot indices are never reachable, including through aliases; **data streams with hidden/dot backing indices are refused**, not transparently queried. Native request/response bodies are capped at **4 MiB**, before the smaller gateway result bound. `delete_by_query` uses a 5-second timeout, scroll size 100 and `max_docs: 500`; this is still a destructive operation.
### RabbitMQ — `native-rabbitmq` manifest 1
[Section titled “RabbitMQ — native-rabbitmq manifest 1”](#rabbitmq--native-rabbitmq-manifest-1)
| Class | Tools |
| ----------- | ------------------------------------------------------------------------------------------- |
| Read | `overview`, `queues_list`, `queue_get`, `exchanges_list`, `bindings_list`, `consumers_list` |
| Write | `messages_peek`, `message_publish` |
| Destructive | `queue_purge` |
Every tool requires an explicit permitted **`vhost`**, including `overview`. List tools default to page **1**, `page_size: 100`, maximum **100** per page. Some lists are paginated locally after a bounded Management API response, so page size is not a database-load guarantee.
`overview` returns only the exact requested vhost object, including message statistics, selected locally from the Management API vhost collection. It requests summary columns, but [RabbitMQ](https://www.rabbitmq.com/) 4.3.6 ignores that column selection and fails on paginated vhost collections. The single-vhost endpoint requires administrator privileges; no such privileges are granted to data identities. The tool fails closed if the collection exceeds the **4 MiB** response bound, even when the requested vhost alone would fit.
`messages_peek` **consumes and requeues** messages: it is a write, not a read-only inspection. Count defaults to **5**, capped at **20**. Only `ackmode: ack_requeue_true` and `encoding: auto` are accepted; payloads are truncated at **64 KiB**. `message_publish` accepts a UTF-8 payload of at most **64 KiB**. `queue_purge` permanently removes ready messages and always requires confirmation.
## Confirmation, results and limits
[Section titled “Confirmation, results and limits”](#confirmation-results-and-limits)
Destructive tools always require gateway-run confirmation; `tools.confirmWrites: true` extends this to ordinary writes. The client must support **form elicitation and multi-round-trip tool calls (MRTR)**: the initial result requests input and supplies signed `requestState`; the continuation supplies that state and an accepted `confirm: true` response. Approval is bound to the cluster, token, tool, class and canonical effective-argument SHA-256, expires within the call timeout and is single-use. Missing capability, changed arguments, expiry, replay, decline or unavailable confirmation **fails closed**, without executing the tool. Upstream elicitation is not exposed as an alternative confirmation path; upstream resources are not proxied.
Treat all returned content as **untrusted data, not instructions**. For upstreams without their own wrapping, the gateway adds a fresh per-call nonce boundary. [MongoDB](https://www.mongodb.com/)’s manifest declares its upstream untrusted-data wrapping. Structured/non-text results are normalized into text, with no parallel unwrapped structured result. Results are truncated with `[Result truncated]` to fit the byte budget; output is not a complete export mechanism.
Default gateway limits:
| Limit | Value |
| ---------------------------- | -------------------------------------------------------------------------------- |
| Tool-call rate per token | 60 calls/minute, token-bucket burst 10 |
| Concurrent calls per cluster | 4, shared across tokens |
| HTTP request body | 1 MiB |
| Serialized result budget | 256 KiB; usable text is smaller to reserve JSON escaping/envelope/boundary space |
| Tool call timeout | 60 seconds, including confirmation |
Rate/concurrency denial uses [MCP](https://modelcontextprotocol.io/) error **`-32029`**. Limits reduce exposure, not query cost: aggregations, scans, local list pagination and requeued-message reads can still load production.
## Managed identities and rotation
[Section titled “Managed identities and rotation”](#managed-identities-and-rotation)
Data credentials are generated and sealed, used privately by the gateway/companion and **never revealed through the API**. Logical system identities are `karakoram-mcp-ro` and, only for cluster `readWrite`, `karakoram-mcp-rw`. Narrowing scope replaces managed grants; disabling removes identities once reconciliation completes. Rotate them using the engine’s existing **rotate** operation, targeting these logical system-user names; do not create application users with these reserved names. Rotation updates companion configuration/fingerprints where applicable.
| Engine | Database enforcement |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| MongoDB/PSMDB | Users in `admin`; scoped `read`/`readWrite`, or `readAnyDatabase`/`readWriteAnyDatabase` when unscoped. No user/cluster administration roles. |
| PostgreSQL/Supabase | Actual LOGIN roles **`kk_mcp_ro` / `kk_mcp_rw`**, no superuser, createdb, createrole, replication or bypass-RLS; connection limit **10** each. `statement_timeout: 30s`, `idle_in_transaction_session_timeout: 60s`; reader additionally defaults `default_transaction_read_only: on`. Explicit grants are CONNECT, schema USAGE and table SELECT; writer adds INSERT/UPDATE/DELETE and scoped sequence USAGE/SELECT so serial/identity-default inserts work. Matching default table and sequence privileges cover application owner roles. API-driven role changes trigger an ACL refresh; objects and owners created directly in SQL are discovered by an additive refresh every ten minutes while readiness is observed. No ownership or DDL grants are added. Unscoped reader also receives `pg_read_all_data`. |
| Elasticsearch | Roles **`karakoram_mcp_ro` / `karakoram_mcp_rw`**: cluster `monitor`; scoped index `read`, `view_index_metadata`, `monitor`, plus `write` for RW. `allow_restricted_indices: false`; gateway separately excludes all hidden/system/dot indices. No index/user administration grants. |
| RabbitMQ | Both identities have **`monitoring`** tag. Per permitted vhost, Configure is `^$`, Read is `.*`; Write is `^$` for RO and `.*` for RW. The gateway, not the tag alone, prevents RO message consumption. |
**[PostgreSQL](https://www.postgresql.org/) PUBLIC caveat:** reconciliation does not revoke PUBLIC CONNECT/object grants, because that would change application privileges. Scope is not an isolation boundary for objects already accessible to PUBLIC. GUCs are role defaults, not a SQL sandbox or immutable resource quota. Review PUBLIC grants, callable functions and application RLS before enabling an agent. SQL that requires privileges not granted above will fail even after approval.
## Companion operations and status
[Section titled “Companion operations and status”](#companion-operations-and-status)
[MongoDB](https://www.mongodb.com/), [PostgreSQL](https://www.postgresql.org/) and [Supabase](https://supabase.com/) add one private stateless service named **`mcp`**, separate from database membership. It runs the Karakoram agent on the provider runtime ([Ubuntu](https://ubuntu.com/) 24.04 on [Zerops](https://zerops.io/)) with pinned [native artifacts](/docs/native-artifacts/#data-mcp-companion-artifacts): `mongodb-mcp-server` **3.1.0** or `pgedge-postgres-mcp` **1.1.0**. Development builds require **`KARAKORAM_ARTIFACT_MIRROR`** or a published **`KARAKORAM_ARTIFACT_RELEASE`** containing the exact locked MongoDB bundle because it has no upstream download URL. Admission fails if the artifact URL cannot be resolved. See [artifact resolution](/docs/native-artifacts/#download-and-verification).
PostgreSQL and Supabase companions connect directly to the active database members on private port **5432**, bypassing [ProxySQL](https://proxysql.com/) and [Supavisor](https://github.com/supabase/supavisor) even when configured. Multi-host connections use `target_session_attrs=prefer-standby` for reads (falling back to the primary when no standby is available) and `target_session_attrs=read-write` for writes. This preserves [pgEdge](https://www.pgedge.com/)/[pgx](https://github.com/jackc/pgx) extended-protocol compatibility; enabling Data [MCP](https://modelcontextprotocol.io/) does not enable a proxy reader listener. TLS-enabled clusters retain `sslmode=verify-full` with the cluster CA. All three providers provide companion-to-member private networking: Zerops project DNS, [Dokploy](https://dokploy.com/) target-overlay service aliases, and [Nomad](https://developer.hashicorp.com/nomad) native service discovery mapped to routable allocation addresses. Operator-managed network policies must permit these private connections; no public database exposure is required.
Companion resources default to shared CPU and 0.25–0.5 GiB RAM. `spec.dataMcp.resources` overrides those defaults. In an override, omitted/zero resource bounds, an omitted CPU mode and an omitted `startCpu` are unmanaged: providers with partial-update semantics may retain their existing values. Removing an override field does not request a reset or block later configuration and credential updates. Removing the entire resources override reapplies the controller’s explicit defaults while leaving fields outside those defaults unmanaged.
The companion supervises `mcp-ro` and, with cluster write opt-in, `mcp-rw`. Each upstream binds only to loopback and requires a process-local bearer. MongoDB receives connection/authentication through environment variables; pgEdge uses protected configuration and SHA-256 token files. Secrets are not command arguments. The authenticated agent routes `/v1/datamcp/ro` and `/v1/datamcp/rw` over private HTTPS port **9443**, using the existing dialer/relay; never publish that port. Credentials, desired spec and certificate changes advance config version and trigger reload. Disabling/deleting waits for ownership-checked companion cleanup, including volume-preserving cluster suspension.
On Nomad, an externally purged companion job does not block disabling Data MCP: reconciliation expires the missing job’s rollout receipt, deletes the retained logical service and clears its sealed companion identity and configuration. Database identity removal still follows the normal disable reconciliation.
Inspect `status.dataMcp` independently of database member readiness:
| Field | Meaning |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `phase` | Schema allows `Disabled`, `Provisioning`, `Ready`, `Degraded`; disabled clusters currently omit the entire status object |
| `kind` | `companion` or `native` |
| `upstream` | Manifest/provider name |
| `upstreamVersion` | Companion artifact version or reviewed native toolset manifest version |
| `access` | Provisioned **cluster-level** `read`/`readWrite`, not a particular token’s effective permission |
| `message` | Diagnostic such as artifact or companion readiness unavailable |
Companion readiness checks applied config version and the configured upstream processes. Native status reports the selected provider; it is not proof that a particular database query succeeds. Installation/token/RBAC write gates may still hide writes when status says `readWrite`.
## Risks and operating practice
[Section titled “Risks and operating practice”](#risks-and-operating-practice)
Stored text can contain prompt injection. Boundaries and confirmations help an agent recognize data but cannot prevent it from following malicious instructions. Use dedicated accounts/tokens, explicit scopes, read-only access by default and a client that actually shows meaningful confirmations. Never put production authority behind unattended blanket approvals.
Queries can consume CPU, memory, I/O and locks despite result bounds. Start with low-risk datasets, monitor database load and audit history, and review engine-specific privileges. Confirmations are not transactions or rollback. A transport/audit error after execution does not prove a mutation was undone; inspect state before repeating it.
Sessions, confirmation signing keys and consumed challenges, and rate/concurrency counters are local to each control-plane process. [MCP](https://modelcontextprotocol.io/) clients need instance affinity; restarting that process ends its sessions and pending confirmations.
Every `tools/call`, including denied, pending, declined, rate-limited and upstream-error calls, attempts an **`Audit`** event with resource **`cluster-data`**, reason `DataMcpToolCall`, cluster, actor, tool target, class, outcome, duration, confirmation state and **`argsHash`** (SHA-256 of canonical arguments; effective arguments after manifest defaults/clamps on admitted calls). `auditQueryText: true` additionally records `args`, which may contain sensitive query text or document values. Audit recording failure returns an error; it does not roll back a completed data change. Protect audit access and retention accordingly.
# Dokploy provider
> The dokploy provider runs Karakoram clusters on a Dokploy-managed Docker Swarm that you operate. It is experimental: it passes the shared provider…
The `dokploy` provider runs Karakoram clusters on a [Dokploy](https://dokploy.com)-managed [Docker Swarm](https://docs.docker.com/engine/swarm/) that you operate. It is **experimental**: it passes the shared provider conformance suite and was exercised with [MongoDB](https://www.mongodb.com/), [RabbitMQ](https://www.rabbitmq.com/) and [Elasticsearch](https://www.elastic.co/elasticsearch) on a three-node test Swarm, but it has no production track record. [Zerops](https://zerops.io/) remains the reference provider. Design rationale: [ADR 0032](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0032-dokploy-backend.md).
[PostgreSQL](/docs/postgres/) and [Supabase](/docs/supabase/) use the same provider-neutral runtime contract and have not been accepted on Dokploy. Members need access to locked native artifacts (including an operator mirror for source-built development entries) and registered [S3](https://aws.amazon.com/s3/) destinations. Public PostgreSQL TCP has no provider CIDR enforcement and needs explicit allow-all acknowledgement plus an external firewall. Supabase gateway public HTTPS uses the provider’s domain capability; unavailable component artifacts are still rejected.
## How it maps
[Section titled “How it maps”](#how-it-maps)
| Karakoram | Dokploy / Swarm |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Namespace target (`projectId`) | One Dokploy **environment** ID |
| Member runtime | One Dokploy application named after the member hostname (Dokploy suffixes the Swarm service name); one replica, stop-first updates and rollbacks, 120 s stop grace period |
| Member DNS name | A network alias equal to the member hostname on the target’s IPv4 overlay network; private services use DNSRR task addresses so peer certificate source-address verification sees matching DNS |
| Persistent volume | A Docker named volume on one node, owned by a “volume holder” application that is never started and gives the volume its own ID |
| Deployment | Karakoram uploads a build archive (Dockerfile + the release `karakoram` archive); Dokploy builds the image on the manager and pushes it to your registry; workers pull it |
| Anti-affinity | Volumes of one replica set (MongoDB) or cluster (RabbitMQ, Elasticsearch) are pinned to distinct Swarm nodes while enough nodes exist; runtimes follow their volume |
| Public TCP | Swarm ingress-published ports (routing mesh); exposed services switch to VIP endpoint mode, returning to DNSRR after port removal |
| Public HTTPS (Kibana, RabbitMQ management) | Dokploy domains served by Dokploy’s Traefik |
[Docker](https://www.docker.com/) is used on this provider only because Dokploy runs Swarm services. Each member image is a pinned [Ubuntu](https://ubuntu.com/) 24.04 base with the same package installation steps as on Zerops, and the engine processes run under the same `karakoram agent` binary as everywhere else. There is no Docker-in-Docker and no Docker socket in member containers.
Engine prepare commands use exec-form Dockerfile `RUN` instructions invoking [Bash](https://cgit.git.savannah.gnu.org/cgit/bash.git/) with `pipefail`; embedded newlines, shell quoting and heredocs are preserved rather than parsed as Dockerfile instructions.
### Platform deployment admission
[Section titled “Platform deployment admission”](#platform-deployment-admission)
Dokploy queues application deployments and builds uploaded Dockerfiles on its server. Karakoram therefore admits at most **two** in-flight deployments per Dokploy origin, shared across all environments/namespaces in the control-plane process. Set the control-plane environment variable `KARAKORAM_DOKPLOY_MAX_INFLIGHT_DEPLOYS` to an integer **>= 1** to change this limit; it is not a per-namespace setting.
Admission covers archive uploads, retained-source rebuilds and volume-wipe deployments. It reads owned applications across the server’s projects/environments and their authoritative deployment rows, adding local reservations while submission visibility is uncertain. A queued/running/unknown deployment blocks another submission for that application even if it was started outside Karakoram. Terminal `done`, `error` and `cancelled` rows release capacity. Waiting returns a typed deployment wait with a retry deadline at least 15 seconds away, for example `waiting for a platform deployment slot: 2/2 in flight (Dokploy server)`; it does not upload or enqueue another build.
Additional admission observations are bounded to **128 reads per attempt**, shared by project details, application deployment lists and exact-ID corroboration of reserved applications missing from inventory (plus one project-list request). This is an inventory-size ceiling, not a throughput limit: one project with 128 owned applications already exceeds it, even with no builds running. Waiting for deployments to finish or increasing the deployment limit will not resolve this diagnostic. Reduce the total project/owned-application inventory on that Dokploy installation (safely delete retired resources), or use a separate installation; splitting namespaces/environments on the same origin does not help. Unavailable or incomplete inventory fails closed. The API key must see all Karakoram applications on the installation. This is process-wide admission, not a distributed semaphore: independent control-plane processes and direct operator submissions must coordinate their own combined limit.
The cap also applies during initial cluster bootstrap; there is no quorum-group exception. Initial member deployment readiness means the agent supervisor is running and initialising, not that the database has formed a quorum. Stateless day-2 replacements retain native readiness when dependencies are healthy; the engine persists that choice for stable retries. Dokploy does not map either probe to a Docker `HEALTHCHECK`, because Swarm service discovery must be available to bootstrapping tasks. Application/native health, configuration acknowledgement and rolling-replacement catch-up remain enforced by Karakoram’s engine-level convergence gates. See [member startup and admission](/docs/operations/#deployment-admission-and-member-startup).
Synchronous Swarm `application.reload` operations reuse an existing image and do not enter Dokploy’s build queue, so they do not consume deployment slots. Existing pending build/lost-delivery fences still protect lifecycle operations. A terminal failed runtime build is returned as a service with `Failed` status, not as an identity lookup error. Engines can verify ownership and submit a fresh admitted deployment within their retry budget; observation alone never retries the build. A stalled or unknown-outcome build remains fenced and must not be superseded. A failed volume-wipe deployment/task remains terminal, retaining the holder, rather than resubmitting on every delete reconciliation. Ambiguous wipe submission responses retain their original fence.
Runtime archive uploads and retained-source builds reserve attempts durably in the application description before delivery, with at most **four attempts** for the same rendered desired deployment and retry epoch. Submission acceptance, waiting, control-plane restart and a failed create operation do not reset this budget. A successful completed deployment clears the consecutive attempt count; a changed rendered runtime/artifact or `clusters retry-deployments` grants a new budget. An authoritative enqueue refusal does not consume an attempt. Once exhausted, a subsequent deployment request reports `DeploymentRetriesExhausted`; the engine retains the failure status and emits a warning with the last safe platform error. Fix the platform/build cause before explicitly re-arming.
Dokploy reloads increment Swarm’s task `ForceUpdate`, even for a replica-only change. A pure count reduction back to an already-running count therefore uses temporary `stop-first`, `Parallelism=1` when at least two replicas remain running. This frees one replacement slot without deliberately stopping the entire service. The saved policy returns to `start-first` after the operation becomes terminal, without another reload; superseding specification changes and Start/Restart/Stop also restore it before replacing the pending operation, including Stop→Start. Singleton reductions retain `start-first` because `stop-first` would retire the only remaining serving task before its replacement starts.
This exception requires a matching application-specification receipt from a successful deployment or reload. Staged environment, resource, port or image changes cannot use the count-only exception; older records without that receipt conservatively keep their existing policy until an ordinary rollout establishes it. Unlike Nomad’s count-only reconciliation, Dokploy still replaces tasks because its API exposes no count-only Swarm update. Runtime Running is not native SQL/API readiness; the engine’s application-health and every-instance gates still apply. Specification receipts use the shared versioned HMAC-SHA256 helper, keyed by the control-plane-held Dokploy API credential with a dedicated value domain. This prevents a metadata reader from guessing low-entropy environment secrets offline. API-credential rotation invalidates older receipts conservatively until a successful ordinary apply establishes a receipt under the new credential.
### Resolving an unknown receipt or failed wipe
[Section titled “Resolving an unknown receipt or failed wipe”](#resolving-an-unknown-receipt-or-failed-wipe)
An unknown enqueue stays pending (a runtime may eventually report stalled/unknown-outcome); admission shows a previous-deployment or deployment-slot wait. A failed wipe reports `volume wipe deployment failed; the holder was retained` or `volume wipe task failed; the holder was retained`. It can block cluster deletion. Restarting alone does **not** clear durable receipts, and `retry-deployments` is not a Dokploy wipe-rearm command.
1. Identify the **exact application ID**, environment, generated `appName`, pinned node and volume from the Karakoram service/operation and the Dokploy application description. Preserve a copy of the description before any manual change. In Dokploy’s application Deployments UI, or `GET /api/deployment.all?applicationId=`, inspect the deployment IDs and terminal state, logs and queue entries. Compare against `pending.deployments` (runtime) or `wipePreviousDeployment` (holder) in the description. Inspect `docker service ps --no-trunc ` on a manager and the pinned node’s containers/mounts. Do not mistake an empty deployment list for absence: `GET /api/application.one?applicationId=` must return not-found to establish that the application was removed.
2. If the original deployment appears, let Karakoram track it; new authoritative rows replace the unknown capacity reservation automatically. If the exact application is absent, the reservation is retired automatically, including after restart. A wipe still waits for running tasks and final volume cleanup. Do not remove a data-bearing runtime application just to free capacity.
3. If no row appears, first obtain authoritative confirmation from the Dokploy queue/worker and request logs that the original request cannot still execute. **No row and elapsed time alone are not sufficient.** Stop every Karakoram control plane managing this installation before changing metadata. **For a runtime only**, when the request is confirmed never enqueued, edit that application’s description via Dokploy’s settings or `POST /api/application.update` with only `applicationId` and the updated `description`. Keep the existing prefix and all unrelated JSON fields intact; remove only `pending` and `mutationUntil`. Restart the control plane to discard its in-memory reservation. The next reconciliation can make a fresh admitted attempt. Never clear ownership, volume, node or writer-fence fields, and never edit Karakoram’s SQLite operation records.
4. For an **unknown or failed wipe**, use manual erasure and exact-holder removal, not a metadata reset. Clearing `wiping`, `wipePreviousTasks` or `wipePreviousDeployment` and restarting does **not** re-arm a wipe operation handle already persisted by an engine: that handle will keep waiting for its original task/deployment, even if Dokploy confirms nothing was enqueued. There is no product wipe-rearm action. If these fields were already cleared, the manual cleanup below still resolves the original tracked handle. With control planes stopped, conclusively exclude late enqueue delivery and confirm no queued/running deployment or task for that holder. Verify no runtime still mounts its exact named volume; a failed task is not proof of erased data. Correct the platform cause (node access, image pull, permissions), then manually complete the intended erasure on the original pinned node. This is destructive: back up anything that must survive. Remove the holder application by exact ID through Dokploy (`POST /api/application.delete`), remove its stopped task containers, and remove the **exact named Docker volume on the original node** once no container uses it. Never use a broad volume prune. Confirm application not-found, no running task/service, and volume absence on that node. Restart Karakoram: the existing wipe operation observes the absent holder, verifies cleanup and can finish cluster deletion. Keep the original node available until Karakoram acknowledges completion.
If queue delivery cannot be conclusively resolved, keep the receipt fenced and repair Dokploy observation/queue state; do not force a second potentially destructive deployment.
## Capabilities
[Section titled “Capabilities”](#capabilities)
| Capability | Dokploy | Notes |
| ------------------------------------------------------ | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `publicTcp` | yes | Ingress mode: every Swarm node forwards the port. Status and TLS names use the target’s `publicIPv4`/`publicIPv6` setting. |
| `tcpAllowlist` | no | No source-address filtering. Public TCP therefore requires an allowlist that covers the whole address family plus `allowAllAddresses: true`; restrict access with your own firewall in front of the nodes. |
| `publicHttps` | yes, with `domainSuffix` | Dokploy domains `--.`; needs wildcard DNS to the Traefik node(s). |
| `diskGrowth`, `volumeQuota` | no | Docker local volumes live on the node filesystem without a size limit; size requests are not enforced and disk growth is rejected. Monitor node disks. |
| `placement` | yes | Authoritative node per member (`placementGroup` = Swarm node hostname). |
| `logs` | yes | Platform logs through Dokploy. Swarm output can include current and retained tasks; Karakoram orders records before selecting the newest global tail, preserving source order for equal timestamps. Untimestamped/malformed records inherit their preceding transport neighbour’s timestamp (leading records use the first known timestamp); these timestamps are approximate. An entirely untimestamped response retains transport order. A quiet current task can still have older retained records in its tail. |
| `horizontalScaling` | yes | Stateless components (Kibana, Logstash, APM, mongos) scale by replicas. |
| `scopedTokens` | no | Dokploy API keys inherit the user’s rights; a member user restricted to one environment cannot read the registry or nodes this provider needs. Use an admin key of a Dokploy installation dedicated to Karakoram. |
| `exec`, `metrics`, `dedicatedCpu`, `envChangeRestarts` | no | Variable writes apply at the next restart. |
Swarm ownership checks resolve the service ID from `swarm.getNodeApps` before calling `docker.getConfig`; inspecting an application name can return its local build image instead. This also works for zero-replica services with no task containers and does not require cached metadata. An empty/unavailable inventory, or an ambiguous inspection of a listed service, blocks mutations until observation can be retried; it is not evidence of a foreign owner. Dokploy excludes service names starting with `dokploy-` from this inventory: these names (including generated application names for a `dokploy` hostname) cannot be inspected safely. Observation and lifecycle mutations refuse them rather than treating them as absent or falling back to ambiguous name inspection. Use hostnames outside this reserved prefix.
## Requirements
[Section titled “Requirements”](#requirements)
* Dokploy **v0.30.8** (the tested version) on an amd64 Docker Swarm; the API is unversioned, so test upgrades first.
* For more than one node: a container registry configured in Dokploy (Settings → Registry) that every node can pull from. Single-node Swarms may omit it.
* Outbound HTTPS from the manager during builds (Ubuntu, [Percona](https://www.percona.com/), RabbitMQ and Elastic package sources), from every node to [Docker Hub](https://hub.docker.com/) (volume deletion runs a one-shot task from the pinned `ubuntu:24.04` digest on the volume’s node), and from members to your S3 storage.
* Elasticsearch: `vm.max_map_count=262144` on every node (`/etc/sysctl.d`).
* Enough free disk on every node for volumes and images (each engine image is roughly 0.3–1.7 GB).
* An overlay network for Karakoram, normally a dedicated attachable one (the provider creates it through Dokploy when it is missing).
Optional node label: if any node carries `karakoram.data=true`, only labelled nodes receive data volumes (`docker node update --label-add karakoram.data=true `).
## Target settings
[Section titled “Target settings”](#target-settings)
| Config key | Environment variable (default namespace) | Meaning |
| ----------------------------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | `KARAKORAM_DOKPLOY_URL` | Dokploy origin, e.g. `https://dokploy.example.com` (no path) |
| — | `KARAKORAM_DOKPLOY_API_KEY` | API key (write-only token input for other namespaces) |
| `projectId` (namespace field) | `KARAKORAM_DOKPLOY_ENVIRONMENT_ID` | Dokploy environment ID |
| `network` | `KARAKORAM_DOKPLOY_NETWORK` | Attachable overlay network for member aliases; default `karakoram-<8 hex characters of the environment ID's SHA-256>`, created through Dokploy when missing |
| `registryId` | `KARAKORAM_DOKPLOY_REGISTRY_ID` | Dokploy registry used for built images |
| `publicIPv4`, `publicIPv6` | `KARAKORAM_DOKPLOY_PUBLIC_IPV4`, `_IPV6` | Address clients use for public TCP |
| `domainSuffix` | `KARAKORAM_DOKPLOY_DOMAIN_SUFFIX` | Wildcard DNS suffix for public HTTPS |
| `certificateType` | `KARAKORAM_DOKPLOY_CERTIFICATE_TYPE` | `letsencrypt` (default), `none` (Traefik default certificate) or `custom` |
**URL restriction.** A namespace’s Dokploy URL receives its API key, so the control plane only talks to approved origins: the operator’s `KARAKORAM_DOKPLOY_URL`, entries of `KARAKORAM_DOKPLOY_ALLOWED_URLS` (comma-separated origins), and — only while `KARAKORAM_DOKPLOY_ALLOWED_URLS` is unset — any `https://` origin whose host is a name or a public IP address (loopback, private, link-local and shared-address literals and `localhost` need an explicit allowlist entry). Production installations should set the allowlist. Redirects are never followed. Namespace settings cannot be changed after creation.
**API key rate limit.** Dokploy creates API keys with rate limiting enabled by default (in v0.30.8: 10 requests per 24 hours). Karakoram polls continuously, so create the key with rate limiting disabled (API: `user.createApiKey` with `rateLimitEnabled: false`). A namespace whose token suddenly turns invalid with HTTP 401 has usually hit this limit.
**Network isolation.** Member hostnames are DNS aliases on the configured network. Every application attached to that network can resolve and reach them, and two namespaces sharing a network could collide on names (for example the relay hostname), so the default network is per environment. Avoid `dokploy-network`, which every Dokploy application joins. Only services with public HTTPS enabled are additionally attached to `dokploy-network`, so Traefik can route to them; enabling or disabling public HTTPS therefore restarts that service. Engine authentication and the agent token still guard every port; the network is not an access-control boundary.
## Install the control plane on Dokploy
[Section titled “Install the control plane on Dokploy”](#install-the-control-plane-on-dokploy)
The control plane runs as one Dokploy application with a persistent volume; its default namespace manages one Dokploy environment directly over the shared overlay network.
1. In Dokploy create a project (for example `karakoram`) and, for clusters, a project or environment for the default namespace; note its environment ID (Dokploy UI URL or `GET /api/project.all`). Create an admin API key with rate limiting disabled (see [API key rate limit](#target-settings)) and, for multi-node Swarms, a registry. Before deploying the control plane, create its dedicated **attachable overlay** network (for example `karakoram`, via Dokploy’s network settings or `POST /api/network.create` with `{"name":"karakoram","driver":"overlay","attachable":true}`); its Swarm service needs the network before the provider can start.
2. Create an application `karakoram` with build type **Dockerfile**, Dockerfile path `deploy/dokploy/Dockerfile` and context `.`, from this repository (Git provider) — it downloads and verifies the signed release pinned in `deploy/zerops/RELEASE`. To deploy an unreleased build, upload a zip containing that Dockerfile as **`Dockerfile` at the zip root** and a linux `karakoram` binary alongside it (Drop source); set Dockerfile path **`Dockerfile`**, context `.`, and build argument `KARAKORAM_SOURCE=local`.
3. Advanced settings:
* Volume mount: named volume (for example `karakoram-data`) at `/var/lib/karakoram`.
* Swarm placement: pin to the Dokploy manager node (`node.hostname==`): the named volume is node-local, and the control plane then talks to Dokploy’s plain-HTTP service URL without leaving the host (Swarm overlay traffic between nodes is unencrypted). If it must run elsewhere, point `KARAKORAM_DOKPLOY_URL` at Dokploy’s HTTPS domain instead.
* Environment settings: turn off **Create .env file**, so Dokploy does not write the control plane’s secrets into the build directory.
* Swarm update and rollback config: `{"Parallelism": 1, "Order": "stop-first"}` — [SQLite](https://www.sqlite.org/) allows one writer.
* Swarm network: `[{"Target": "", "Aliases": ["karakoram"]}, {"Target": "dokploy-network"}]`. Members reach the control plane as `https://karakoram:9443`; `dokploy-network` lets [Traefik](https://traefik.io/traefik/) and the control plane reach Dokploy.
* Registry: your registry (multi-node).
4. Environment:
```sh
KARAKORAM_MASTER_KEY=<48 random characters; keep a separate backup>
KARAKORAM_ADMIN_PASSWORD=
KARAKORAM_PUBLIC_URL=https://karakoram.example.com
KARAKORAM_ADVERTISE_URL=https://karakoram:9443
KARAKORAM_DOKPLOY_URL=http://dokploy:3000
KARAKORAM_DOKPLOY_API_KEY=
KARAKORAM_DOKPLOY_ENVIRONMENT_ID=
KARAKORAM_DOKPLOY_REGISTRY_ID=
KARAKORAM_DOKPLOY_NETWORK=
KARAKORAM_DOKPLOY_PUBLIC_IPV4=
KARAKORAM_DOKPLOY_DOMAIN_SUFFIX=apps.example.com
```
`http://dokploy:3000` is Dokploy’s own service on `dokploy-network`; because it is the operator-set `KARAKORAM_DOKPLOY_URL`, it is accepted although it is plain HTTP. The image sets `KARAKORAM_BACKEND=dokploy` and `KARAKORAM_DATA_DIR=/var/lib/karakoram`. For a lab using Traefik’s default certificate, also set `KARAKORAM_DOKPLOY_CERTIFICATE_TYPE=none` and use certificate type `none` for the control-plane domain; clients must explicitly disable TLS verification. For private or nonstandard-port S3 storage, set `KARAKORAM_BACKUP_ENDPOINT_ALLOWLIST` to the endpoint’s `host:port` (for example `minio.internal:9000`), without a URL scheme; see [Backups and storage](/docs/backups/). 5. Domains: add `karakoram.example.com` → port **8080**, HTTPS on. Traefik forwards WebSocket upgrades, which the UI and relays need. Do not publish port 9443. 6. Deploy, then sign in at `KARAKORAM_PUBLIC_URL` as `admin`, complete the required initial password change, and sign in again before creating API tokens or using `karactl login` ([API bootstrap](/docs/api/#curl-bootstrap), [CLI](/docs/cli/#login-and-contexts)).
Back up `KARAKORAM_MASTER_KEY` and the `karakoram-data` volume ([Backups and storage](/docs/backups/)). Upgrades: redeploy with a newer release (`deploy/zerops/RELEASE`) or a newer local binary; stop-first update ordering keeps one writer.
## Managing Dokploy from another control plane
[Section titled “Managing Dokploy from another control plane”](#managing-dokploy-from-another-control-plane)
Additional namespaces of kind `dokploy` use the outbound [relay](/docs/namespaces/): Karakoram deploys one relay application into the target environment, which dials the control plane’s public URL. That URL must present a certificate trusted by the public CA bundle of the Ubuntu base image. Create the namespace with the target settings above, for example:
```sh
karactl namespaces create --name swarm --backend dokploy --project-id '' \
--config url=https://dokploy.example.com --config registryId= \
--config network=kk-swarm --config publicIPv4=203.0.113.10 --token-file dokploy-key.txt
```
## Operations and limitations
[Section titled “Operations and limitations”](#operations-and-limitations)
### Desired state and partial platform loss
[Section titled “Desired state and partial platform loss”](#desired-state-and-partial-platform-loss)
Karakoram owns its application records and their randomly suffixed Swarm service names. Use **Karakoram Stop**, not Dokploy Stop or `docker service scale ...=0`, for an intentional suspension. For an owned application with durable running intent, service observation recreates a missing Swarm service and restores an externally zeroed replica count, including a stop from Dokploy’s UI. It reuses the recorded built image when it is present on the manager; otherwise it requests one rebuild from the retained Drop Dockerfile source. Application identity, mounts, node placement, environment and configured replica count are retained.
The application description durably records desired state and the current lifecycle operation. A target-wide lock serializes provider instances within the control-plane process. Restoration submissions have a ten-second request deadline; an ambiguous response retains a one-minute delivery fence. Stop/Delete record their intent before waiting on that fence. A queued/running build must complete before another build, Stop or Delete is submitted: an accepted build can create the service much later, after a nominal stop would otherwise finish. A lost submission response with no new deployment record is deliberately not resubmitted: inspect Dokploy’s queue/deployment logs before resolving it. Archive bytes are hashed while rendering, and an identical retry recovers the saved attempt handle instead of uploading again. Queued and running builds have a separate six-hour progress bound: after it, the operation fails with a stalled/unknown-outcome diagnostic, but retains its exclusion because the original work may still commit.
Delete and stop waits older than **three minutes** report their elapsed age, exact owned-service desired replica count, and task desired/current state counts. The message distinguishes tasks still stopping from a Swarm inconsistency (`desired 0 but a task is desired-running`). Unknown/failed ownership or service inspection remains a wait, not evidence that deletion succeeded.
During PostgreSQL full-cluster deletion, that inconsistency can trigger a single **escalated removal** through Dokploy’s final `application.delete` step. This requires persisted full-delete intent, the exact CP-recorded application identity, matching immutable Swarm ownership labels/service identity, fresh zero-replica inspection, no active or queued deployment, and no competing pending mutation or delivery fence. Suspension/retained-volume cleanup and ordinary Stop never escalate. The attempt is durably recorded before submission; uncertain delivery is not replayed. The original receipt continues polling until application and running-task absence are observed, including after a control-plane restart. Missing queue permissions or incomplete observations disable escalation safely. The generic two-hour operation-tracking deadline still does not cancel desired cleanup; eventual authoritative deletion can complete the original operation. The preceding idempotent stop has a separate durable confirmation: if a delete journal survives before stop delivery (or after a refused stop), cleanup recovers the stop against the exact owned service without replacing the receipt. An explicitly observed native zero confirms older receipts without resending Stop. Missing or null replica counts are unknown, never proof of zero.
Resolve the original queue/build and obtain a terminal deployment receipt before retrying; never clear the receipt merely to force a second build. Failed terminal deployments require an explicit corrected deployment, not an automatic retry loop.
Cluster deletion does not require a failed runtime build to become healthy. Cleanup observes the recorded application identity without restoring or rebuilding it, and deletion supersedes the obsolete engine deployment wait. A queued/running mutation still retains its exact receipt: deletion waits for terminal delivery without submitting another build, then removes the owned application even when that deployment failed or was cancelled. An uncertain/stalled delivery remains fenced and requires the operator inspection described above. This observation mode is installed by shared cluster teardown for every engine, including suspension; it is not limited to PostgreSQL. Providers still verify resource ownership before removal. PostgreSQL’s stronger exact-ID intent remains required for its escalated-removal path. Public-route withdrawal records deletion intent before waiting on an active build, so that wait does not consume the ordinary consecutive-failure budget.
The ordinary operation failure budget is distinct from the exact-version `DeploymentRetriesExhausted` budget. A failed Dokploy build or an exhausted create/delete operation alone does not qualify for `retry-deployments`; its `409` does not mean deletion requires another deployment attempt. Use cluster deletion to remove a failed initial application.
An authoritative enqueue refusal (HTTP 4xx other than a request timeout) is different from lost/ambiguous acceptance: archive uploads, retained-source rebuilds and wipes retire that attempt’s local and durable receipt, permitting a later corrected retry. HTTP 429 still carries its `Retry-After` deadline. Transport failures, timeouts and 5xx responses retain the original exclusion; elapsed time is never permission to duplicate that submission. Waiters for the shared target guard honor request cancellation even while another application uploads an artifact. A canceled waiter does not detach or interrupt the already-running upload.
Automatic restoration refuses visible foreign service labels/identity, a missing or changed volume holder/mount/node, a wiping holder, and ambiguous Docker observations. Dokploy can return `null`/`[]` when Docker commands fail; those responses alone **do not prove absence**. Restoration requires a nonempty valid Swarm service inventory that omits the service name. An application record removed while its labelled Swarm service remains is reported as an orphan conflict, not adopted or replaced. Restore the original record from backup, or stop/remove the orphan after verifying its data and ownership before requesting replacement. No holder or named volume is automatically recreated by this repair.
Runtime metadata binds the original holder application ID and that holder’s own ownership marker, as well as its node and exact randomly named volume mount. Runtime and holder ownership markers are intentionally different because each marker authenticates its own service hostname; equality between them is not evidence of a valid attachment.
New volume-backed images hold an exclusive file lock on `.karakoram-writer.lock` for the member’s lifetime, so a recreated Swarm service cannot start a second writer while an old task is still draining after deletion. Existing images must be explicitly redeployed through Karakoram to acquire this fence before automatic service recreation is enabled. Legacy application metadata without durable desired state is migrated only while an owned running service is visible; if it is already missing/stopped, explicitly Start it after verifying retained data. A lost or physically damaged node-local volume still requires operator recovery or engine member replacement; a holder record is not a backup.
Application creation itself includes a non-serving ownership/import-spec receipt. Interrupted imports remain identifiable as CREATING and are resumed by replaying the original specifications; setup failures do not silently roll back or orphan that identity. Independent application search and known-ID reads corroborate environment listings. Same-named foreign or duplicate receipts are refused. Unknown creation outcomes are not blindly retried while their receipt is hidden. The in-process fence is not a durable distributed uniqueness primitive: after a control-plane restart, an API that simultaneously hides the committed receipt from all lookup paths can still defeat name-based discovery. Restore reliable platform visibility before retrying an unknown creation. The neutral service observation explicitly marks unfinished reservations as `importing`; ordinary reconciliation resumes setup rather than adopting a nonnull owned service as deployment-ready. Relay enrollment is not rotated mid-import, so replay retains the original sensitive environment.
Public-port changes record pending desired state before changing Dokploy rows. A matching row or an empty port list is not an apply receipt: retries complete the remaining Swarm reload after lost create/delete/reload responses. Stopped services retain their desired settings without being started. PostgreSQL whole-cluster cleanup withdraws saved port rows without requesting a replacement task immediately before deletion. Any already-issued restart keeps its exact journal and must converge before deletion submits Stop; ambiguous port-delivery and queued-build fences are not discarded.
Ingress-published TCP/UDP ports are Swarm-wide, not per application, environment, namespace or IP family: only one service can own a given ingress port/protocol. Before a new publication, Karakoram checks Dokploy’s Swarm service inventory, including unmanaged services and other environments. A confirmed conflict fails the operation promptly with `ExposureSupported=False` / `PublicPortInUse`; a concurrent conflict discovered after a refused reload restores the prior Dokploy port rows, endpoint settings, application status and update policy **without reloading the healthy private service**. The requested public exposure remains visible in the spec: free the port at its other owner or disable public exposure here. Dokploy omits `dokploy-*` services from that inventory and can hide Docker’s reason behind `Error reloading application`. When that explicit failure is followed by proof that the original owned private DNSRR service is still unchanged, Karakoram restores the staged settings and fails with `PublicPortPublishFailed` instead of claiming an unobserved port owner. Lost/unknown reload delivery retains its durable pending journal and blocks a new publication, private correction or other lifecycle mutation until the applied port specification and original replacement-task outcome are observed. A competing owner alone is not evidence that an outstanding request failed: it might free the port before that request reaches Docker. Only explicit completed refusal evidence permits rollback and a terminal clean-refusal claim. If the handler fails after applying the requested ports, Karakoram durably records that completed response; fresh matching service settings and the original replacement tasks’ completion repair Dokploy’s error status without a second reload. Stopped/undeployed settings-only changes do not acquire a reload fence and can finish after reconstruction without an applied-service receipt. If recording the completed response is interrupted before the rollback decision, later observation resumes either the applied-public outcome or the proven-unchanged-private refusal classification; it does not wait indefinitely for a refused publication to appear.
Resource and replica-count updates consume any completed prior deployment before merging new settings, so the platform settings and persisted intent stay aligned. Volume-less application updates use start-first rolling replacement; namespace relays remain serial stop-first singletons, including rollback, because two live relay sessions can evict one another.
Unavailable, drained/paused and missing original storage nodes are diagnosed separately. Reactivate/restore the original node and retained volume, or explicitly replace the member using healthy replicas or a verified backup; Karakoram does not relocate a local volume to empty storage. Volume-wipe completion is not acknowledged if the original cleanup node can no longer be identified.
HTTP 429 responses carry their `Retry-After` reset (seconds or HTTP date) to the control-plane scheduler without immediate transport retry or a short reset cap. The original unavailable classification and credential-silent detail are retained.
**Exclusivity requirement and API limitation.** Run one Karakoram control plane per target; all writes to its managed applications/service names must go through it. Dokploy v0.30.8 has neither metadata compare-and-swap nor an absent-only reload/deploy API. The shared lock is not a distributed lease across independent control planes. A privileged operator creating a foreign service under the exact randomly suffixed name *between* the ownership/absence check and Dokploy’s unconditional update-or-create can still race restoration; a queued build has the same limitation. Do not create or mutate foreign objects under those owned names. Visible foreign objects are refused, but the API cannot enforce the check atomically.
* **Node loss = member loss.** A member is pinned to the node that holds its volume. If the node fails, the member stays pending until the node returns; replace the member through Karakoram to rebuild it elsewhere from its replica set or cluster peers.
* **Member replacement needs headroom.** Karakoram replaces a member by adding the new one before removing the old one. When the cluster already uses every eligible node, the new member is placed on the node of the member it replaces, so that node must have free CPU and memory (Swarm reservations) for one extra member during the replacement; otherwise the new task stays pending until capacity is available. Add a node or free capacity first.
* **Changing public ports restarts the member** (Dokploy applies ports with a service update). Endpoint-mode/port cutovers use temporary `stop-first` with `Parallelism=0` (all replicas in one batch), rather than serial per-slot replacement, to avoid old/new endpoint overlap. This deliberately interrupts the whole service during cutover, including multi-replica proxies. The exact previous update policy is journaled and restored after completion or lifecycle supersession; ordinary updates retain their existing rolling policy. Public addresses still use the configured Swarm-node address and routing mesh.
* **Published PostgreSQL proxy endpoint recovery.** If every proxy task is natively healthy but the logical private endpoint fails, reconciliation can force a journaled stop-first update even when the public-port rows already match. Recovery is limited to three attempts per incident, at least five minutes apart; a successful logical endpoint probe acknowledges recovery and resets that budget. Task convergence alone never acknowledges recovery. A repair that was journaled but never delivered can consume the next bounded attempt after its startup deadline; queued or uncertain deployments are never superseded. Ownership and lifecycle delivery fences still apply. If the logical endpoint remains broken after the final completed attempt, the tracking operation fails with `PublishedEndpointRepairExhausted` and an actionable condition instead of remaining Running. Automatic repair stays stopped across control-plane restarts; inspect and repair the Swarm private DNS alias/VIP and overlay connectivity, or disable public exposure. A later successful logical probe resets the incident budget. Public addresses and ports are not changed by this repair.
* **Deleting a volume** wipes its contents with a one-shot task on its node and removes the holder application. Dokploy can remove Docker volume objects only on the manager; an empty volume object can remain on workers (`docker volume prune` on that node removes it).
* `keepVolumes` deletion keeps the holder application and its Docker volume; resuming the cluster reattaches it on the same node.
* Images are rebuilt by Dokploy for each deployment; the layer cache makes later members fast, but the first build of an engine version takes about one to two minutes. Use Dokploy’s cleanup settings for old images.
* Variable changes are stored immediately and applied at the next restart (`envChangeRestarts: false`).
* Elasticsearch’s S3 client trusts only public CAs; use a publicly trusted S3 endpoint for snapshots.
## Dokploy terms and licence
[Section titled “Dokploy terms and licence”](#dokploy-terms-and-licence)
Dokploy is Apache-2.0 except the `/proprietary` directory. Its [terms](https://github.com/Dokploy/dokploy/blob/v0.30.8/TERMS_AND_CONDITIONS.md) forbid “any commercial resale or redistribution of Dokploy as a service” without explicit consent. Running Karakoram on your own Dokploy for your own databases is internal use; offering Dokploy-backed databases to third parties as a service needs Dokploy’s written consent. Karakoram does not include or redistribute Dokploy.
# Elasticsearch
> Deploy Elasticsearch and its optional, version-coupled Elastic Stack components on Zerops.
Deploy [Elasticsearch](https://www.elastic.co/elasticsearch) and its optional, version-coupled Elastic Stack components on [Zerops](https://zerops.io/).
Opt-in [cluster data MCP](/docs/data-mcp/) provides native scoped Elasticsearch tools without a companion. Configure `scope.indexPatterns`; hidden/system/dot indices, including hidden data-stream backing indices, remain unreachable. Writes require four independent opt-ins.
## At a glance
[Section titled “At a glance”](#at-a-glance)
| Area | Supported configuration |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------- |
| Versions | `9.5.5` (default), `9.5.4`, `9.4.8`, `8.19.23`; use your installation’s catalog |
| Topologies | Combined master/data nodes or dedicated masters plus data nodes; counts below |
| Minimum RAM per instance | Data/combined: 1.5 GiB; dedicated master and Logstash: 1 GiB; enabled Kibana: 1.5 GiB on 9.x, 1 GiB on 8.x; APM: 0.25 GiB |
| Minimum disk | Elasticsearch nodes: 5 GiB persistent; optional components: 1 GiB runtime disk |
| Security | Elasticsearch HTTP/transport TLS required; `requireTLS` only |
| Backups | Native S3 snapshots; see [Snapshots](/docs/elasticsearch/snapshots/) |
| Components | Optional Kibana, Logstash and standalone APM Server at the exact Elasticsearch version |
[Access and security](/docs/elasticsearch/access-and-security/), [Stack components](/docs/elasticsearch/stack/), [Operations](/docs/elasticsearch/operations/), [platform limitations](/docs/limitations/) and [verification scope](/docs/verification/).
## Topologies and presets
[Section titled “Topologies and presets”](#topologies-and-presets)
| API topology | Node roles | Counts |
| ------------------ | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------ |
| `cluster` | Combined master/data/ingest/remote-cluster-client/transform | `components.engine.replicas`: 1–9 |
| `dedicatedMasters` | Master-only nodes plus data/ingest/remote-cluster-client/transform nodes | `components.extra.master.replicas`: 3 or 5; `components.engine.replicas`: 1–20 |
The UI labels these **Combined master and data nodes** and **Dedicated masters + data nodes**. Applications use data/combined nodes, not dedicated masters. Topology type is immutable.
| Preset | Initial composition |
| --------------- | ------------------------------------------------ |
| `dev` | 1 combined node + 1 Kibana |
| `prod` | 3 combined nodes + 1 Kibana |
| `observability` | 3 combined nodes + 1 Kibana + 1 Logstash + 1 APM |
| `large` | 3 dedicated masters + 3 data nodes + 1 Kibana |
Use one combined node for disposable development; usually choose 3/5 for production voting. Even counts are allowed with warnings. Two combined nodes need both voters: losing either stops writes. One data node has no redundant shard copy, regardless of master count. Presets do not establish independent failure domains.
Optional replicas are [Kibana](https://www.elastic.co/kibana) 0–3, [Logstash](https://www.elastic.co/logstash) 0–5 and APM 0–3. Zero disables a component. Each uses one service with multiple containers and ephemeral disks; components cannot attach to another cluster or upgrade independently. Without a preset, defaults are three combined nodes, or three dedicated masters plus two data nodes, with optional components disabled.
If a restart is waiting for an optional component to become ready, setting that component’s replicas to zero skips it in the restart pass. Normal reconciliation then deletes its service without requiring it to recover first. Skipped components are recorded separately from completed restarts, including across control-plane restarts; re-enabling one before the pass finishes requires its restart and readiness before completion. Enabled components and Elasticsearch nodes still retain their restart readiness gates.
## Sizing and heap
[Section titled “Sizing and heap”](#sizing-and-heap)
API RAM/disk values use GiB; CPU values use cores. Engine driver starting resources are:
| Component | RAM bounds | Disk bounds |
| ------------------ | ----------- | ------------------- |
| Data/combined node | 2–2 GiB | 5–20 GiB persistent |
| Dedicated master | 1.5–1.5 GiB | 5–20 GiB persistent |
| Kibana | 1.5–2 GiB | 1–5 GiB ephemeral |
| Logstash | 1.5–1.5 GiB | 1–5 GiB ephemeral |
| APM | 0.5–0.5 GiB | 1–5 GiB ephemeral |
The `dev` preset caps Kibana at 1.5 GiB. All components start with shared CPU bounds 1–2. CPU bounds are whole cores, at least one; maximum bounds cannot be below minima. Omitted resource bounds use engine driver defaults. Size for replicas, rebalance/recovery, disk watermarks and queues, not just current data size. More nodes cannot drain shards onto disks without headroom. See [resource changes](/docs/clusters/) for platform growth constraints.
Enabled Kibana 9.x requires at least **1.5 GiB minimum RAM** for creation, enablement, replica increases, intentional RAM changes and every version change (including patch upgrades). Kibana 8.x retains its 1-GiB admission floor; neither floor is a startup-time or workload-capacity guarantee. Disabled Kibana retains structural resource checks.
Existing smaller Kibana 9.x instances are not resized automatically: unchanged same-version updates and replica decreases/disable remain accepted. Status identifies the below-minimum allocation and directs an explicit increase of `spec.components.extra.kibana.resources.minRam` and `maxRam` together to at least 1.5 GiB. Before any version change, **resize Kibana first** at the current version and wait for readiness; a combined resize and upgrade is refused when the old enabled Kibana is below the target floor.
Heap derives from configured **minimum** RAM, not the current autoscaled allocation:
* Elasticsearch: equal initial/maximum heap; 40% through 4 GiB, otherwise 50%, rounded down to a 64-MiB multiple and clamped to 512–31744 MiB. At 2 GiB: 768 MiB.
* Kibana: Node.js old-space cap 60%, minimum 512 MiB; 614 MiB at 1 GiB and 921 MiB at 1.5 GiB. Legacy allocations retain this same formula.
* Logstash: equal initial/maximum heap 50%, minimum 384 MiB.
* APM: Go process, no JVM heap.
Heap excludes filesystem cache, native/direct memory, threads and queues. Elasticsearch’s cap does not guarantee compressed object pointers: 26 GiB is conservative on most systems; some permit about 30 GiB. Check startup logs and [Elastic JVM guidance](https://www.elastic.co/docs/reference/elasticsearch/jvm-settings). Small defaults are not production throughput sizing; review [Kibana](https://www.elastic.co/docs/deploy-manage/production-guidance/kibana-traffic-scaling-considerations) and [Logstash](https://www.elastic.co/docs/reference/logstash/jvm-settings) requirements.
Host kernel/process limits remain platform constraints. Check `vm.max_map_count` against Elastic’s recommendation of `1048576`; Karakoram cannot change the shared-host sysctl. A value of `262144` can meet bootstrap requirements without meeting that recommendation. Keep bootstrap checks enabled; memory locking is not supported. See [runtime rationale](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0025-elasticsearch-engine-runtime-model.md).
The inherited `nofile` **hard limit must be at least 65,535**. The runner raises the soft limit only to that inherited hard limit; it cannot raise the hard limit. Elasticsearch bootstrap checks still enforce the minimum.
## Create
[Section titled “Create”](#create)
1. Install/authenticate [karactl](/docs/cli/) or open **Clusters → Create cluster**.
2. Select the namespace, **Elasticsearch**, an advertised exact version, topology/preset and component resources. Names match `^[a-z][a-z0-9]{1,11}$`.
3. Keep private exposure initially. Review aggregate resources and placement constraints. The wizard clears dependent public flags, and disabled Logstash pipelines, when disabling components, including for clones.
4. Submit; inspect cluster health, members and observed generation, not just request acceptance.
Inspect the current catalog with `karactl providers get elasticsearch -o json`. There is no CLI `--preset`: copy a catalog `presets[].spec`, retain `provider: elasticsearch`, then customize it.
This canonical file creates an observability stack with private endpoints and nightly snapshots. [Register storage](/docs/backups/) named `archive` in the selected namespace first, or omit `backup`. Save as `elasticsearch.yaml`:
```yaml
name: search
spec:
provider: elasticsearch
version: 9.5.5
topology: {type: cluster}
components:
engine:
replicas: 3
resources:
cpuMode: SHARED
minCpu: 1
maxCpu: 2
minRam: 2
maxRam: 2
minDisk: 5
maxDisk: 20
extra:
kibana: {replicas: 1}
logstash: {replicas: 1}
apm: {replicas: 1}
tls: {mode: requireTLS, ca: selfSigned}
exposure: {public: false}
engineOptions:
elasticsearch: {settings: {}}
kibana: {public: false, settings: {}}
logstash: {pipelines: [], persistentQueue: false}
apm: {public: false, apiKeys: true}
backup:
storage: archive
schedules:
- name: nightly
cron: "0 2 * * *"
type: snapshot
retainCount: 7
enabled: true
```
```sh
karactl -n default clusters create -f elasticsearch.yaml --wait
karactl -n default clusters status search
karactl -n default clusters members search
```
The API accepts the same body at `POST /api/v1/clusters?namespace=default`; see [API mechanics](/docs/api/). For dedicated masters, change the topology and add `components.extra.master` with a permitted count; `engine` then means data-only nodes.
Settings belong in `engineOptions.elasticsearch.settings` or `engineOptions.kibana.settings`, not `engineConfig`. Undeclared fields and conflicting dotted/nested settings are rejected. Karakoram reserves roles, discovery, network/path/ports, security/TLS, [S3](https://aws.amazon.com/s3/) clients and Kibana connection/encryption/public-URL settings. See [component settings](/docs/elasticsearch/stack/). Patch files wrap changes in `{spec: {...}}`; arrays replace complete arrays.
## How it runs
[Section titled “How it runs”](#how-it-runs)
Each Elasticsearch node has a singleton [Ubuntu](https://ubuntu.com/) runtime and a separate Local Storage volume, including dedicated masters. This is not Zerops’s managed Elasticsearch service. Official exact-version archives are SHA-512-verified; unprivileged supervised processes use bundled runtimes. Formation bootstraps once, records cluster identity and removes bootstrap configuration; retained-volume resume does not bootstrap a different cluster. Security precedes optional component deployment. Local readiness is separate from formation and shard redundancy.
## Limits
[Section titled “Limits”](#limits)
No ML provisioning, arbitrary role pools, hot/warm/cold pools, version skew or downgrade is supported. No Elasticsearch shard-awareness policy maps replicas to physical hosts; see [shared placement/storage limitations](/docs/limitations/). Maintenance, ingress, queues and recovery have specific limits in [Operations](/docs/elasticsearch/operations/), [Access](/docs/elasticsearch/access-and-security/), [Stack components](/docs/elasticsearch/stack/) and [Snapshots](/docs/elasticsearch/snapshots/).
## Licensing
[Section titled “Licensing”](#licensing)
Karakoram is Apache-2.0, but contains no Elastic code and downloads rather than redistributes official default builds on user infrastructure. Elasticsearch/Kibana binaries use ELv2; AGPLv3/SSPL source options do not license those binaries. Logstash/APM offer OSS builds, but this engine driver uses default builds. Provisioning uses Basic-tier security, TLS, API keys, service accounts, snapshots and monitoring without bypassing paid-feature controls.
**Warning:** ELv2 restricts hosted/managed resale exposing substantial Elasticsearch/Kibana functionality. User-owned infrastructure and non-redistribution are not blanket exemptions; obtain appropriate Elastic rights and legal review before offering such services. OpenSearch is not this engine driver. See [ADR 0029](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0029-elastic-licensing.md), [Elastic’s FAQ](https://www.elastic.co/pricing/faq/licensing) and [ELv2 examples](https://www.elastic.co/licensing/elastic-license/faq); this is not legal advice.
# Access and security
> Connect clients and manage Elasticsearch, Kibana and intake identities separately from Karakoram accounts.
Connect clients and manage [Elasticsearch](https://www.elastic.co/elasticsearch), [Kibana](https://www.elastic.co/kibana) and intake identities separately from Karakoram accounts.
## Connect and reveal credentials
[Section titled “Connect and reveal credentials”](#connect-and-reveal-credentials)
Ordinary cluster status contains password-free endpoints. In Overview, expand **How to connect**; reveal credentials only for administration. `karactl clusters connection search --reveal` calls `GET /api/v1/clusters/{cluster}/connection?namespace=NAME`, requiring separate `cluster-credentials:read` permission. The response is sensitive and uses `Cache-Control: no-store`:
| Field | Meaning |
| ----------------------------- | -------------------------------------------------------------------- |
| Credential `elastic` | Human administrative superuser password, not an application identity |
| Credential `apm-secret-token` | APM intake token; not the provider writer password |
| `caCertificate` | Public cluster CA PEM |
Credential URIs describe the cluster; use returned [component endpoints](/docs/elasticsearch/stack/), not those URIs, for Kibana/APM. Do not reveal into CI/shared logs or terminal recordings.
**Admin-only health check:** this Bash recipe uses the revealed `elastic` superuser, authenticated [karactl](/docs/cli/), `jq` and `curl`. Run it from a trusted diagnostic workload in the namespace’s [Zerops](https://zerops.io/) project for private access; for public access, change `PUBLIC=false` to `PUBLIC=true` and use an allowed client. See [namespace networking](/docs/namespaces/) for private reachability.
```bash
(
set +x
set -euo pipefail
umask 077
work=$(mktemp -d)
trap 'rm -rf -- "$work"' EXIT
PUBLIC=false
karactl -n default clusters connection search --reveal -o json > "$work/connection.json"
url=$(jq -er --argjson public "$PUBLIC" \
'.connection.endpoints | map(select(.name == "elasticsearch" and .public == $public))[0].url // empty' \
"$work/connection.json")
jq -er '.caCertificate' "$work/connection.json" > "$work/ca.pem"
jq -er '.credentials[] | select(.username == "elastic") | "user = " + ("elastic:" + .password | @json)' \
"$work/connection.json" > "$work/curl.conf"
curl --config "$work/curl.conf" --fail-with-body --cacert "$work/ca.pem" \
"$url/_cluster/health?pretty"
)
```
Files are owner-only and removed on exit; protect temporary storage and the diagnostic host too. `jq -r` preserves actual PEM newlines; alternatively export with `karactl -n default clusters ca search`. Never use `-k`. Password clients need the CA, not transport certificates or private keys.
Provision scoped users/API keys from a trusted operator environment, then deliver only that app identity, returned endpoints and CA file through the application’s persistent runtime secrets/configuration. Do not install a Karakoram management token in the application. Supply multiple data-node endpoints for failover. Public clients must use provisioned public addresses and disable sniffing that discovers unreachable private hosts.
**Workload-side verification:** mount the application’s scoped key’s `encoded` value (base64 `id:key`) at `/run/secrets/elasticsearch-api-key` and the CA at `/run/secrets/elasticsearch-ca.pem`. Set `ELASTICSEARCH_URL` to a returned data-node endpoint. This checks the `orders-*` read grant shown below without a Karakoram token or admin identity:
```bash
(
set +x
set -euo pipefail
umask 077
work=$(mktemp -d)
trap 'rm -rf -- "$work"' EXIT
{ printf 'header = "Authorization: ApiKey '; tr -d '\r\n' < /run/secrets/elasticsearch-api-key; printf '"\n'; } > "$work/curl.conf"
curl --config "$work/curl.conf" --fail-with-body \
--cacert /run/secrets/elasticsearch-ca.pem \
"$ELASTICSEARCH_URL/orders-*/_search?size=0"
)
```
The secret header is read from an owner-only curl configuration, never a process argument. Use an existing authorized index pattern; a successful request verifies that workload’s CA, connectivity and read access, not cluster-wide health.
## Listeners and TLS
[Section titled “Listeners and TLS”](#listeners-and-tls)
| Listener | Port/protocol | Reachability |
| ----------------------- | --------------- | ----------------------------------------------------------- |
| Elasticsearch HTTP | 9200 HTTPS | Private nodes; optional public TCP to data/combined nodes |
| Elasticsearch transport | 9300 mutual TLS | Private nodes only |
| Kibana | 5601 HTTP | Private; optional public HTTPS subdomain |
| Logstash monitoring API | 9600 HTTP | Localhost only |
| Logstash Beats input | 5044 TCP | Private |
| Logstash HTTP input | 8080 HTTP | Private |
| APM intake | 8200 HTTP | Private, dual-stack `[::]`; optional public HTTPS subdomain |
| APM monitoring | 5066 HTTP | Localhost only |
Elasticsearch security is always enabled. HTTP and transport use per-node certificates under the cluster CA; transport requires client authentication with CA-based certificate verification. Components verify Elasticsearch HTTPS and CA trust automatically. Kibana/APM public routes terminate TLS at the balancer, forwarding HTTP on the private project network; [Logstash](https://www.elastic.co/logstash) inputs are not secured automatically.
CA sources are `selfSigned` or `userProvided`. The latter requires a certificate and matching signing private key, supplied through protected write-only TLS inputs; see [PKI input contract](https://github.com/KarakoramDev/Karakoram/blob/main/api/openapi.yaml). CA source/material cannot change in place. Leaf rotation uses `karactl clusters rotate-tls search --wait --yes` and watched PEM reload without Elasticsearch restart; it does not rotate the CA or Kibana encryption keys.
## Public access
[Section titled “Public access”](#public-access)
Elasticsearch `spec.exposure` explicitly enables public TCP, chooses IPv6 or dedicated IPv4, and applies the shared [public exposure allowlist and acknowledgement rules](/docs/security/#public-exposure-allowlists) with end-to-end Elasticsearch TLS. Dedicated masters and transport/management ports are not published. Routing status does not prove external reachability: test allowed and denied clients with CA validation.
**Warning:** Kibana/APM HTTPS subdomains are internet-reachable without source-IP restrictions. Elasticsearch’s database allowlist does not protect them. Kibana requires Elasticsearch/Kibana authentication; APM requires intake authentication. These switches do not publish Elasticsearch or Logstash.
Set `engineOptions.kibana.public` or `engineOptions.apm.public` and include the corresponding request-only `allowAllAddresses: true` when creating/enabling that route. An unchanged public route needs no new acknowledgement; disabling/re-enabling does. The acknowledgement is consumed, not persisted or inherited by future PATCH requests. The wizard/topology editor confirms this exposure and Review lists every public route even with private database access.
Setting either public switch to `false` revokes that component’s HTTPS route before configuration rolls or component-readiness waits, including an already-running roll. Revocation still waits for any accepted platform operation and uses the normal ownership checks and operation receipt; it does not mark the component ready or acknowledge the new generation. Enabling a route remains part of normal reconciliation. On an existing cluster, both route enable and disable still require an open [maintenance window](/docs/operations/#maintenance-windows) or generation override: a backend may replace the component task to change ingress. Use the override for urgent route withdrawal; readiness is not an additional prerequisite.
These are Zerops shared subdomains, not custom-domain HTTP locations; custom-domain source-IP policies cannot attach to them. Karakoram supplies Kibana’s public base URL and secure cookies. Zerops documents a 50-MB upload cap and does not recommend the shared subdomain balancer for production. Custom-domain management is outside this feature; see [Zerops public access](https://docs.zerops.io/references/networking/public-access).
## Users, roles and API keys
[Section titled “Users, roles and API keys”](#users-roles-and-api-keys)
Karakoram RBAC controls management; native identities control Elasticsearch/Kibana. **Elastic Access** manages users, roles, API keys and editable application grants. All routes below use `database-users` permissions (`read`, `create`, `update`, `delete`) on the selected cluster/namespace:
| Resource | Routes below `/api/v1/clusters/{cluster}/elasticsearch` |
| -------- | --------------------------------------------------------- |
| Users | `GET/POST /users`; `GET/PUT/DELETE /users/{username}` |
| Roles | `GET/POST /roles`; `GET/PUT/DELETE /roles/{role}` |
| API keys | `GET/POST /api-keys`; `DELETE /api-keys/{id}` invalidates |
Create a role before assigning it; unknown roles are rejected. User create bodies contain `{username,password,roles,fullName,email,enabled}`; passwords require eight characters and are write-only. Roles are string names. User reads cannot recover app passwords.
Save this role as `role.json` for read access to `orders-*` and Kibana’s Default space:
```json
{
"name": "orders-reader",
"cluster": [],
"indices": [{"names": ["orders-*"], "privileges": ["read", "view_index_metadata"], "allowRestrictedIndices": false}],
"applications": [{"application": "kibana-.kibana", "privileges": ["read"], "resources": ["space:default"]}]
}
```
Kibana needs both application and index grants. Use resources `["*"]` for all spaces or explicit `space:` entries. Kibana registers application privileges at startup; native acceptance of arbitrary names/privileges does not establish browser access. The editor warns for unknown forms, not against a live privilege inventory: known forms include `all`, `read`, `space_all`, `space_read`, and `feature_.` with `all`, `read`, `minimal_all`, `minimal_read` or subfeatures such as `url_create`. Custom application grants remain valid. Prefer [Kibana’s own role UI/API](https://www.elastic.co/docs/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-privileges) for general management.
Role bodies contain `cluster`, `indices` and `applications` arrays; empty arrays grant nothing. Writes omit response-only `reserved`; undeclared nested fields are rejected. Index entries support opaque JSON-string `query` and `fieldSecurity` with `grant`/`except` arrays. These restrictions require an appropriate paid licence, not Basic; unavailable features return an actionable 4xx rather than an unrestricted role. Reads/edits preserve restrictions, including empty grants. The UI accepts an index-privilege JSON array; API-key descriptors support identical restrictions. Errors identify the relevant cluster/index/application field, or `roleDescriptors..indices[i].query`/`.fieldSecurity` for keys; cross-cluster names in index diagnostics are not cluster-grant errors.
```sh
karactl elasticsearch roles create search orders-reader -f role.json
# APP_PASSWORD comes from protected secret input; keep tracing disabled.
printf '%s' "$APP_PASSWORD" | karactl elasticsearch users create search ordersreader \
--roles orders-reader --password-stdin
karactl elasticsearch users get search ordersreader
karactl elasticsearch roles update search orders-reader -f role.json
karactl elasticsearch users update search ordersreader --roles orders-reader
```
| Commands | Flags/semantics |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `users list`, `roles list`, `api-keys list` | `--limit` (100 default), `--cursor`; inherited namespace/output flags |
| `users create/update` | `--roles` replaces role names; `--full-name`, `--email`, `--disabled`/`--disabled=false`; omitted update flags retain values |
| Password changes | `--password-stdin`; required on create; never place secrets in arguments |
| `roles create/update` | Required `-f` JSON (`-` for stdin); command name authoritative and file name must match; omitted arrays empty; update replaces all privileges |
| `users/roles get` | `` |
| `users/roles delete` | ` --yes` |
Reserved/built-in and `karakoram*` users are hidden/protected; built-in roles are readable/assignable but immutable, and `karakoram*` roles are hidden/protected. Do not mutate managed identities through native CRUD.
API-key creation requires a name and a **nonempty** `roleDescriptors` map; inherited superuser privileges are never accepted. Save only that map in `descriptors.json`, for example:
```json
{"orders-read":{"cluster":[],"indices":[{"names":["orders-*"],"privileges":["read"]}],"applications":[]}}
```
```sh
karactl elasticsearch api-keys create search --name orders-query --expiration 30d -f descriptors.json
karactl elasticsearch api-keys list search
karactl elasticsearch api-keys invalidate search KEY_ID --yes
```
The REST body wraps the map in `{name,expiration,roleDescriptors}`; CLI `--expiration` accepts an Elasticsearch duration. Creation reveals `id`, `apiKey`, `encoded` once: store securely, never log it. Only Karakoram-created keys are listed/invalidated; lists omit secrets. Use the encoded value with `Authorization: ApiKey` for Elasticsearch, not as a Karakoram management token. Prefer explicit expiry/narrow grants; [APM intake keys](/docs/elasticsearch/stack/) need their own privileges.
Native user/role writes and API-key creation support [safe HTTP retries](/docs/api/#safe-retries-with-idempotency-key), including replay of one-time secrets. Without keyed retries, resolve uncertain creation by listing/invalidating unwanted keys first. Invalidation of an already invalidated key succeeds; retry and refresh inventory. A 404 remains an error. These access mutations are not durable cluster Operations. Deletion/invalidation needs confirmation: non-TTY callers pass `--yes` or pipe `y`; empty stdin never submits.
## Managed credentials and secret locations
[Section titled “Managed credentials and secret locations”](#managed-credentials-and-secret-locations)
Every node has file-realm `karakoram` (superuser control/recovery) and `karakoram-monitor` (cluster/index monitoring). The control identity remains usable when restoring native security state; it is not an app identity or a revealed credential. Kibana uses an `elastic/kibana` service-account token; Logstash/APM have separate scoped provider writers.
Regular reconciliation reasserts the engine driver-owned Logstash/APM writer roles, native users, enabled state and sealed passwords even when desired secrets have not changed. Native edits/deletion of these managed objects are repaired; customer-owned users and roles are not modified. Kibana’s service token is separately authenticated and replaced if revoked.
Node [S3](https://aws.amazon.com/s3/) keys reside in `elasticsearch.keystore`; rendered configuration, TLS keys, file-realm files and component secrets are owner-only (0600 files, 0700 directories). Kibana’s token/encryption keys and APM’s intake token/writer password are in component configuration; Logstash’s writer credential is injected into its process environment. These are accessible to privileged runtime administrators: see [trust boundaries and encryption at rest](/docs/security/).
`karactl system-users list search` lists rotation targets: `elastic`, `karakoram-monitor`, `kibana-token`, `logstash`, `apm`, `apm-secret-token`. Use `karactl system-users rotate search IDENTITY`; rotation republishes/restarts affected components. The internal file-realm control user is not a rotation target. Credential rotation does not change the CA or cluster-lifetime Kibana encryption keys.
# Operations
> Operate Elasticsearch nodes and Stack components with health-gated maintenance and shard-aware removal.
Operate [Elasticsearch](https://www.elastic.co/elasticsearch) nodes and Stack components with health-gated maintenance and shard-aware removal.
Native health, node-membership (`/_nodes`) and elected-master (`/_cat/master`) observations are required for Ready. A failed observation withdraws member health, preserves agent reachability separately, and reports actionable member/cluster messages. `ClusterHealth=Unknown/ObservationUnavailable` means native evidence is incomplete, not that a timeout proves unhealthy shards or a missing master. Fresh shard evidence establishing failure remains False if another probe fails. Previously acceptable green or yellow health becomes Unknown when required observations fail; acceptable yellow is not positive failure evidence. Inspect native API access, member processes and connectivity; restore fresh observations before maintenance. Successful fresh observations clear this diagnosis. The convergence membership fan-out also has a two-second read budget, so a paused JVM does not hold the serialized status worker until the native client’s longer administration timeout. Expired status snapshots still withdraw positive agent connectivity claims until a fresh sweep succeeds; inspect `ObservationStale` and agent last-seen times to distinguish this from a fresh failed agent RPC. Convergence-membership read failures identify the two-second budget and direct operators to inspect member responsiveness/connectivity.
## Scale, upgrade, restart and replace
[Section titled “Scale, upgrade, restart and replace”](#scale-upgrade-restart-and-replace)
Change counts/resources in **Topology & scaling** or a spec merge patch. The UI reviews the outgoing patch, not unrelated form defaults. For example, `scale.yaml`:
```yaml
spec:
components:
engine:
replicas: 5
```
```sh
karactl clusters patch search -f scale.yaml --wait
karactl clusters members search
karactl operations cluster search
```
New nodes use never-reused indices and hot-reloaded discovery seeds. Scale-down serially drains shards through allocation exclusions, then voting exclusions where needed. Overview distinguishes desired, observed and **draining** members until confirmed departure. Remaining data nodes must hold each index’s primary plus maximum replicas, including auto-expanded replicas evaluated after removal.
`ShardDrainBlocked` reports insufficient capacity or pinned shards, with a Warning on transition rather than every observation. Even green health can hide pinned primaries: three idle observations retaining the same nonzero shard count trigger a blocked drain. Progress/relocation resets that window; repeated flaps do not repeat the same warning. Allocation explanations identify indices, eligibility and filters when available; missing diagnostics do not suppress the condition.
To recover:
* Restore unavailable primaries first. Red health blocks voting exclusion/runtime/volume deletion, including vanished nodes and retried deletion; never force removal.
* Relax restrictive `index.routing.allocation.require.*`, `include.*` or `exclude.*` filters through Elasticsearch. More capacity/fewer replicas cannot free a pinned primary. Lower replica policies only if reduced redundancy is acceptable.
* Cancel an unfinished scale-down by restoring `components.engine.replicas` to cover **all observed data nodes**. This clears owned exclusions/blocked state and emits `ShardDrainCancelled`; further growth adds capacity. Once voting-excluded/deleting, removal must finish. Replacement removals cannot be cancelled this way.
Only Karakoram-owned exclusions are removed, including after interrupted writes. Dedicated masters skip shard-inventory waiting and proceed to voting exclusion, but retain primary-safety gates.
RAM/heap and restart-relevant settings trigger rolling maintenance; CPU/disk-only changes use nondisruptive updates within [platform constraints](/docs/clusters/). Optional replica changes adjust containers; zero deletes the service and ephemeral contents.
Upgrades allow newer advertised versions within a major, or `8.19.x` → `9.1+` (never `9.0`); targets come from the [catalog](/docs/elasticsearch/). Cross-major changes require a fresh successful snapshot. Resolve Upgrade Assistant/deprecation/index compatibility issues, then apply a version-only patch: `karactl clusters patch search -f upgrade.json --wait`. An in-place downgrade is not rollback: rebuild an earlier-version cluster using a compatible pre-upgrade snapshot.
```sh
karactl clusters restart search --wait
karactl clusters replace-member search MEMBER_NAME --wait --yes
```
Restart/replacement and other maintenance (backups/restores/rotations) reject requests before initial formation; restart/replacement also require an active cluster. Select observed member names, not bootstrap recovery targets.
Rolling disruption requires all planned masters present and green health, or narrowly acceptable yellow: only unplaceable replicas, no initialization/relocation. Red/unexplained yellow blocks progress. Maintenance restricts allocation to primaries, flushes, changes one node, waits for rejoin, restores allocation and waits for stable health. Data-only nodes precede non-elected masters; elected master last. After Elasticsearch: [Kibana](https://www.elastic.co/kibana) (all instances stopped together), [Logstash](https://www.elastic.co/logstash), APM. Status/Operation identify current member/component/step. Mixed Kibana versions are unsupported.
A failed deployment retry can recover the same absent target only with every other planned master and committed quorum; it cannot advance to another node while that target remains absent. Replacement joins a new node before draining the old and needs temporary capacity. Single-node replacement is rejected: use snapshots. One data node interrupts service during maintenance even with dedicated-master quorum; two combined nodes cannot preserve quorum with one stopped. Kibana has a separate interruption. [Suspend/delete](/docs/clusters/#delete-or-suspend) observations are historical: members show suspended/agents offline, not cached live health. [Interrupted restore and snapshot cleanup](/docs/elasticsearch/snapshots/) have additional retention rules.
On [Zerops](https://zerops.io/), the exact-version deployment retry budget is finite. `DeploymentRetriesExhausted` stops submissions for that node or stateless stack component, without hiding other members’ observations. The rolling journal and quorum checks remain in force. Correct the platform/runtime failure, then run `karactl --namespace NS clusters retry-deployments NAME` to re-arm deployment.
If a restarted Elasticsearch node cannot start because of a product-invalid setting, correct `engineOptions.elasticsearch.settings` and reapply it. The in-flight rolling pass publishes the correction to that same node while waiting for rejoin; it does not disrupt the next member. During upgrade, configuration repair keeps the journal’s target binary version and never attempts an in-place downgrade.
## Agent rollout and maintenance windows
[Section titled “Agent rollout and maintenance windows”](#agent-rollout-and-maintenance-windows)
`karactl clusters upgrade-agents search --wait` refreshes Karakoram agents without changing Elasticsearch or Stack versions, runtime identities or volumes. The durable rollout freezes the control-plane build and skips healthy members already reporting it; `--include-current` explicitly refreshes those members too. Data-only nodes roll first, then non-elected master-eligible nodes, the elected master, Kibana, Logstash and APM. Each node retains the master/quorum, shard-health, primaries-only allocation, flush and recovery gates above, and must report the exact target agent build before advancing. Stack components must also report that build and pass component readiness. Completed members are journaled across control-plane restarts. Cancellation is allowed between members, never while allocation or a stopped member needs recovery. If the control-plane build changes again, re-run the operation for the new build.
`spec.maintenance` windows gate the start of each new disruptive member step: restart, TLS rotation, agent rollout, engine upgrade, restart-required configuration/resource changes and scale-in (including optional component removal). Automatic database leaf renewal waits while more than seven days remain. A member already allocation-restricted or stopped always completes rejoin, allocation restoration and shard recovery even after the window closes.
Healing, create/scale-out, delete/suspend/resume, backup/restore, credential rotation, agent transport-certificate renewal and database certificates with at most seven days remaining are exempt. No windows means always open. Use `--ignore-maintenance-window` on a supported operation or spec update for an explicit override; it does not bypass native safety or deployment admission. See [maintenance policy](/docs/operations/) for schedules and override scope.
## Readiness and monitoring
[Section titled “Readiness and monitoring”](#readiness-and-monitoring)
| Process | Local readiness |
| --------------- | ---------------------------------------------------------------------- |
| `elasticsearch` | TCP listener reachable; does not establish formation/cluster health |
| `kibana` | `/api/status` returns HTTP 200; 503 and redirects are not ready |
| `logstash` | Every configured pipeline loaded and running via local monitoring APIs |
| `apm-server` | Intake TCP listener reachable; does not establish indexed delivery |
Inspect conditions, observed version/membership, elected master, component readiness, heap/disk and allocation. Green assigns primaries/replicas; yellow lacks replicas (sometimes expected on one data node); red lacks a primary. An `up` sample is not delivery/durability proof. Collection/authentication/history conventions: [Monitoring](/docs/monitoring/).
`IndexWriteBlocked=True` degrades an otherwise green cluster and names the affected indices and native settings (`index.blocks.read_only_allow_delete`, `index.blocks.read_only`, or `index.blocks.write`), including hidden/system indices. Free disk space or add capacity when flood-stage protection fires. Every supported Elasticsearch version [automatically releases flood-stage blocks](https://www.elastic.co/docs/troubleshoot/elasticsearch/fix-watermark-errors) after usage falls below the high watermark; Karakoram does not clear disk protections or operator-set blocks. Review intentional read-only/write restrictions before removing them. If index settings cannot be inspected, the condition is `Unknown` and the cluster is not reported Ready.
| Metric | Meaning |
| ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| `karakoram_elasticsearch_up` | Local availability |
| `karakoram_elasticsearch_cluster_status` | 0 green, 1 yellow, 2 red |
| `karakoram_elasticsearch_cluster_nodes` | Observed nodes |
| `karakoram_elasticsearch_active_shards`, `karakoram_elasticsearch_unassigned_shards`, `karakoram_elasticsearch_relocating_shards` | Allocation/recovery |
| `karakoram_elasticsearch_jvm_heap_used_bytes`, `karakoram_elasticsearch_jvm_heap_max_bytes` | Heap usage/capacity |
| `karakoram_elasticsearch_docs`, `karakoram_elasticsearch_store_bytes` | Documents/indexed storage |
| `karakoram_elasticsearch_indexing_total`, `karakoram_elasticsearch_search_query_total` | Workload counters; use rates |
| `karakoram_elasticsearch_gc_collection_seconds_total` | Accumulated GC time |
| `karakoram_elasticsearch_thread_pool_rejected_total` | Rejected-work counter |
| `karakoram_elasticsearch_fs_available_bytes` | Available data-filesystem space |
| `karakoram_kibana_up`, `karakoram_kibana_status_level` | Availability; status 0 available, 1 degraded, 2 unavailable |
| `karakoram_logstash_up`, `karakoram_logstash_events_in_total`, `karakoram_logstash_events_out_total`, `karakoram_logstash_jvm_heap_used_bytes` | Availability, event counters, heap |
| `karakoram_apm_up`, `karakoram_apm_requests_total`, `karakoram_apm_events_accepted_total` | Availability, request/event counters |
Generic process CPU/RSS/disk metrics remain available. Historical query steps can average categorical states: fractional cluster/Kibana status means mixed samples, not another native state. Use external alerting/history for independent availability monitoring.
## Troubleshooting
[Section titled “Troubleshooting”](#troubleshooting)
Check status, members, conditions and Operations first; retrieve protected component logs with `karactl clusters logs search MEMBER --process PROCESS`. For blocked drain, use the recovery actions above. For Logstash compile/start failures, [correct and reapply pipelines](/docs/elasticsearch/stack/). For Kibana initialization, distinguish allocated routes from ready applications; check proxy prefix handling. For HTTPS/network failures, check returned endpoint visibility, project reachability, CA trust and [public-route boundaries](/docs/elasticsearch/access-and-security/), never disable verification. For missing intake events, verify indexed data and producer replay, not just readiness. For repository/restore failures, preserve inventory and [snapshot recovery prerequisites](/docs/elasticsearch/snapshots/).
For Kibana startup under a tight memory limit, inspect native startup logs and the runtime’s memory limit/current usage and cgroup `memory.events` where available. Repeated `max` events indicate limit pressure, not proof of an OOM kill or permanent failure; listener refusal alone does not identify the cause. Enabled Kibana 9.x requires at least 1.5 GiB minimum RAM for new or changed admissions; 8.x retains 1 GiB. These floors are not startup-time guarantees. Apply an intentional resource change through `spec.components.extra.kibana.resources`, increasing both `minRam` and `maxRam` together. Increasing only the maximum does not increase the Node.js old-space cap. A deployed but not-yet-ready component can converge changed runtime resources before publishing its new heap and rechecking native readiness; an active platform operation must still finish first. Scheduler capacity and native initialization remain prerequisites for Ready.
Existing undersized 9.x Kibana remains admitted for unchanged same-version updates and replica decreases/disable, without automatic resizing. Status reports its below-minimum allocation and the explicit min/max RAM correction. Enabling, adding replicas or editing RAM must meet the active floor. For a version change targeting 9.x with Kibana enabled, including 9.x patch upgrades or 8→9, **resize Kibana first** to the target floor at the current version if needed; a combined resize and upgrade of an already enabled undersized instance is refused. Before changing version, the current generation must be fully applied (`status.observedGeneration == generation`) and the cluster must report `Ready`, including Kibana readiness. An accepted resize PATCH alone is not sufficient: wait for the pending change to finish before submitting the version PATCH. This convergence gate does not apply when Kibana is disabled or the target retains the 8.x floor. Existing cross-major backup requirements still apply.
When a child exits with a recognized Node/V8 fatal heap-limit marker, Kibana reports heap exhaustion with the configured old-space cap in MiB and directs an increase of `spec.components.extra.kibana.resources.minRam` and `maxRam` together (to at least 1.5 GiB on 9.x). This diagnostic is qualified by the applied configuration and failed child launch, remains available through retries, and clears after a later healthy start. It is not evidence of a cgroup OOM kill and does not depend on cgroup visibility. Status never copies raw fatal log lines into the diagnosis, and observation does not change sizing or restart policy.
With updated member agents on [Linux](https://www.kernel.org/) cgroup v2, an unavailable Kibana reports `MemoryPressure=True / CgroupMemoryPressure` when its own cgroup has a fresh increase in `memory.events:max` between uninterrupted samples less than ten seconds apart (shared only until ten seconds after the earlier sample), or nonzero `memory.pressure` `some avg10`. If an unavailable component has a readable counter but no fresh evidence, its readiness request takes a second sample one second later, bounded by the request context. This establishes a fresh comparison even on the first request or after a long gap between control-plane observations, without interpreting the accumulated delta across that gap as current pressure. The condition and instance-qualified readiness message explain how to increase RAM; the message includes numeric current/limit bytes when available. Old lifetime counters, historical PSI totals and high usage alone do not assert current pressure. Observation gaps, failed membership/mount resolution and counter resets discard the counter baseline. Parent-relative cgroup paths are rejected before normalization; oversized procfs/cgroup reads are rejected rather than truncated. Unsupported/unreadable cgroup files produce `MemoryPressure=Unknown` with reason `CgroupMemoryEvidenceUnavailable`, instance identity and instructions to inspect allocation-scoped host evidence or provide an operator-approved read-only scoped cgroup view. Unknown is not a pressure diagnosis. [Nomad](https://developer.hashicorp.com/nomad) `exec` tasks can have a read-only `/sys` sysfs mount without any cgroup2 submount: host cgroup files then exist but are invisible to the member agent. The product does not bypass that isolation, add mount privileges, or expose host-wide cgroups automatically. [Dokploy](https://dokploy.com/) and Zerops likewise require scoped cgroup visibility for a positive native diagnosis; no provider-specific host visibility is assumed. Older agents retain ordinary native readiness messages. Pressure is diagnostic, not a terminal configuration error or evidence of an OOM kill. Successful native readiness takes precedence; the condition disappears on the next fresh healthy observation or after pressure evidence expires. No restart or sizing mutation is performed by observation.
Karakoram retains the explicit **60% of minimum RAM** Node old-space cap (614 MiB at a legacy 1 GiB allocation, 921 MiB at 1.5 GiB). This caps V8 old space, **not total RSS**: young-generation heap, native/C++ allocations, buffers, code and other process/runtime overhead also consume memory. The allocation’s cgroup includes the member agent and file-backed memory as well. Startup/plugin initialization can consume additional memory; the observed limit hits do not identify which allocator or an optimizer as the cause. Raising the old-space fraction may worsen allocation pressure; lowering it may trade pressure for GC or V8 exhaustion. Neither is a proven fix for a particular workload.
Elastic’s [8.19 production guidance](https://www.elastic.co/guide/en/kibana/8.19/production.html#memory) and [current memory guidance](https://www.elastic.co/docs/deploy-manage/production-guidance/kibana-configure-memory) describe scalable heap sizing, not an unconditional startup RAM minimum. The current guidance documents 50% historically and 75% (capped at 4096 MB) for containers starting in 9.4; Karakoram deliberately supplies its own cap. Elastic recommends at least **2 GB** for reporting/detection/alerting/SLO and advanced licensed features, and [1 vCPU / 2 GB as a safe starting point for common analytics traffic](https://www.elastic.co/docs/deploy-manage/production-guidance/kibana-traffic-scaling-considerations). Use that workload baseline rather than treating the admission boundary as a production sizing recommendation. The 9.x admission floor is a supported operating boundary, not a claim that every smaller launch is impossible or every admitted workload fits. See Node’s [old-space flag](https://nodejs.org/docs/latest-v22.x/api/cli.html#--max-old-space-sizesize-in-mib) and [RSS/heap distinction](https://nodejs.org/api/process.html#processmemoryusage).
# Snapshots
> Back up Elasticsearch to S3 and recover data, global state and feature states through native snapshots.
Back up [Elasticsearch](https://www.elastic.co/elasticsearch) to [S3](https://aws.amazon.com/s3/) and recover data, global state and feature states through native snapshots.
## Create and schedule
[Section titled “Create and schedule”](#create-and-schedule)
Elasticsearch supports **`snapshot`**, not logical/physical backups or PITR. [Register storage](/docs/backups/) first. Nodes send snapshot data directly to S3. Always pass `--type snapshot`; the CLI defaults to `logical`.
```sh
karactl backups create search beforechange --type snapshot --storage archive
karactl backups get search beforechange
karactl backups list search
karactl operations cluster search
```
Wait for both Backup and Operation success. Partial/failed native snapshots are not recovery bases. Snapshots include indices/data streams, global state and feature states, including security/[Kibana](https://www.elastic.co/kibana); they exclude node files, private keys, keystores and repository registrations. Preserve protected control-plane state, its original encryption key and storage configuration separately.
Use Karakoram schedules/count retention, not Elastic SLM; the [create example](/docs/elasticsearch/#create) includes a schedule. [Scheduling rules](/docs/backups/) cover UTC cron, successful-backup counts, array replacement and control-plane downtime. ILM controls live-index retention, not backups.
Each referenced storage gets a named S3 client and verified repository with a cluster-UID-unique base path and one writer. Adding/removing clients or changing static endpoints requires rolling maintenance; keystore credential changes reload without restart. A removed storage retires only after no inventory, queued/running restore or unfinished initial-restore source references it. Unregistration does not delete snapshots; already deleted, unreferenced storage does not block retirement/configuration pulls. Never attach another writer to that prefix.
Elasticsearch 9.3+ requires S3 conditional writes. Verify actual repository semantics; do not bypass failures with unsafe compatibility settings. See [snapshot rationale](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0028-elasticsearch-security-and-snapshots.md).
## In-place restore
[Section titled “In-place restore”](#in-place-restore)
**Warning:** Quiesce external writers and plan downtime. Restore overwrites the destination; take a current recovery snapshot if needed.
```sh
karactl restores create search recovery --backup beforechange --yes
karactl restores get search recovery
karactl operations cluster search
```
Karakoram stops Kibana/[Logstash](https://www.elastic.co/logstash)/APM services at platform level, deletes all destination data streams and non-system indices (including hidden/closed objects), restores global state, aliases and feature states, waits for recovery, re-establishes managed credentials through the unaffected file realm and restarts components with current configuration. System indices restore through feature states, not blanket deletion. Application users, roles, API keys and persistent settings may revert. Verify data, authentication and delivery before reopening writers. Do not duplicate a restore after a client timeout.
Restore submission uses a durable per-operation receipt, not the presence of an index with a snapshot name. If the control plane loses the native response or cannot persist its successful response, it refuses to submit again and keeps managed components stopped. The Restore error requests reconciliation: inspect native `GET /_recovery?active_only=false&detailed=true`, the source snapshot, expected indices and acknowledged data. Do not delete the receipt or blindly retry. If a full restore cannot be established, keep external writers fenced, suspend with retained volumes to abandon the interrupted operation as described below, then resume and explicitly request a fresh restore. Legacy in-flight submission checkpoints without these receipts also require reconciliation rather than automatic replay.
Without durable native successful completion, the submission outcome remains **unknown**, even when the final HTTP response looks like a validation rejection (including HTTP 500 `snapshot_restore_exception` naming an open conflicting index, or a typed 4xx). Karakoram preserves the native error as evidence on the intent receipt, not as proof that restoration never started. Restore/Operation fails immediately with reconciliation guidance instead of spending a retry budget on a receipt that can never authorize another POST. Replaying an intent-only receipt after a crash or lost response is likewise terminal immediately; transient receipt-store read/write errors remain retryable.
In Elasticsearch 9.x, [RestoreService validation](https://github.com/elastic/elasticsearch/blob/v9.3.0/server/src/main/java/org/elasticsearch/snapshots/RestoreService.java#L1747-L1793) runs before restore routing is published **for that execution**. However, [TransportMasterNodeAction retries after master disconnect/NodeClosedException](https://github.com/elastic/elasticsearch/blob/v9.3.0/server/src/main/java/org/elasticsearch/action/support/master/TransportMasterNodeAction.java#L251-L274). A previous execution of the same HTTP request may already have started restoration before the retried execution rejects its now-open indices. One control-plane POST does not guarantee one native execution. Elastic’s [restore guidance](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore/restore-snapshot#delete-restore) also requires stopping indexing to prevent external index re-creation.
Terminal unknown-outcome does **not** undo the preceding destination deletion or release holds. Restore holds and stopped managed components remain in place; healthy client-created indices do not prove restoration and must not make the cluster Ready. Preserve the receipt, fence external writers, inspect the named conflict/reason, native `GET /_recovery?active_only=false&detailed=true`, source snapshot and acknowledged data, and preserve any needed destination data. To recover through Karakoram, suspend with `keepVolumes: true`, resume as described below, resolve the reported cause (including writers recreating indices), then explicitly request a fresh restore. For a conflicting index recreated by a client, first copy or snapshot any of its documents you need, then remove it with native `DELETE /` and confirm `HEAD /` returns 404 before the fresh restore. Do not delete receipts or restart managed components to bypass the hold. If error evidence cannot be persisted, the existing intent still remains outcome-unknown.
## New cluster from backup
[Section titled “New cluster from backup”](#new-cluster-from-backup)
Create a complete destination request using the [canonical shape](/docs/elasticsearch/#create) and this immutable `spec` fragment:
```yaml
dataSource:
cluster: search
backup: beforechange
```
```sh
karactl clusters create searchcopy -f clone.json --wait
karactl clusters status searchcopy
karactl operations cluster searchcopy
```
The source must still exist, use Elasticsearch and have a successful snapshot. Destination version cannot be older and may be at most one major ahead; [snapshot/index compatibility](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore) still applies. This is neither downgrade nor continuous replication. The destination forms, registers the source repository read-only, restores, establishes its own managed credentials and then uses its unique writable prefix.
Keep source backup/storage available through initial restore. After success, unrelated destination updates do not re-resolve source records; immutable `dataSource` remains historical provenance. For cross-namespace sources, set `dataSource.namespace`; see [permissions and recovery flow](/docs/backups/). Nodes require direct S3 reachability.
For a **deleted** source, create the destination without `dataSource`, wait for formation, then reference the retained backup’s original source:
```sh
karactl clusters create searchcopy -f destination.json --wait
karactl restores create searchcopy recovery --source-cluster search --backup beforechange --yes
karactl restores get searchcopy recovery
```
Add `--source-namespace NAME` when needed. Preserve its inventory, storage and protected control-plane state.
## Cleanup and interrupted restore
[Section titled “Cleanup and interrupted restore”](#cleanup-and-interrupted-restore)
`Backup.location` is `s3:///?repository=#`, a repository/snapshot reference shown in Backup details, **not a per-snapshot object prefix**. Snapshots share blobs: delete through Elasticsearch. Direct retained-backup deletion after source deletion is allowed only for the sole snapshot in a retired repository with no writer; otherwise register it elsewhere and use the native snapshot API.
Changing a storage registration’s prefix applies to future snapshots. Existing backups and interrupted backups retain their recorded base path for restore, deletion and continued polling; the recorded bucket and storage identity must still match. Keep the original bucket contents and access permissions available. A prefix edit does not migrate or copy objects, and changing a bucket/endpoint requires a separate storage migration plan.
With `keepBackups: false`, whole-cluster cleanup first retires runtime writers and deletes volumes (or records intentional `keepVolumes` retention). It then uses the control-plane S3 binding to remove only complete owned writable repository prefixes, including shared blobs, without contacting Elasticsearch. Read-only source repositories and prior incarnations are untouched. A durable two-minute artifact-cleanup window bounds storage/authentication/binding retries across controller restarts; successful repositories are checkpointed before inventory removal. On exhaustion, `SnapshotsRetained` names the recorded repository prefix and manual-removal guidance; cluster deletion completes while affected inventory remains. Restore the original binding, verify ownership, remove only that complete retired repository prefix, then remove retained inventory. Never delete individual shared snapshot blobs or guess ambiguous ownership. `keepBackups: true` preserves objects and inventory without a failure warning, independently of `keepVolumes`; accepted restore/clone dependencies still fence destructive deletion with `409`. See [deletion choices](/docs/clusters/#delete-or-suspend).
Suspending with `keepVolumes: true` during restore abandons it. Resume records `RestoreAbandoned`, clears restore-owned destructive settings after rejoin, restores component intent and skips interrupted initial-restore checkpoints. Retained data may be partially restored: inspect before workloads; request a fresh explicit restore if necessary. Initial restore is not automatically retried.
# Kibana, Logstash and APM
> Configure the optional components of an Elasticsearch cluster.
Configure the optional components of an [Elasticsearch cluster](/docs/elasticsearch/).
## Endpoints
[Section titled “Endpoints”](#endpoints)
Use returned endpoints, not guessed platform addresses. The endpoint names are `elasticsearch`, `kibana`, `apm`, `logstash-beats` and `logstash-http`. Public entries retain the same name and carry `public: true`; there are no `kibana-public` or `apm-public` names. Select the public entry explicitly. See [listeners, network access and credentials](/docs/elasticsearch/access-and-security/#connect-and-reveal-credentials).
## Kibana
[Section titled “Kibana”](#kibana)
Open the returned [Kibana](https://www.elastic.co/kibana) URL. Initial administration can use `elastic`; use a regular native user with appropriate roles for routine work. Karakoram’s UI login is not a Kibana identity. The automatically installed service-account token authenticates Kibana’s provider, not browser users.
Set custom settings under `spec.engineOptions.kibana.settings`. If using `server.basePath`, the reverse proxy must strip the prefix. `server.rewriteBasePath` is managed as `false`: omit it from new requests; previously stored values are ignored. Shared, cluster-lifetime session/saved-object/reporting encryption keys support multiple replicas. [Public routing](/docs/elasticsearch/access-and-security/) is a separate opt-in.
A route can exist before application initialization finishes. Wait for [component readiness](/docs/elasticsearch/operations/); a suspended component must be resumed rather than treated as starting.
Kibana, [Logstash](https://www.elastic.co/logstash) and APM readiness is checked on every live runtime instance using provider-provided individual addresses, not one service-DNS sample. Each instance must acknowledge the current configuration and pass its native application probe, and the live count must meet the configured replica count. Extra live instances during a rollout must pass as well; stopped/disappeared instances are excluded from application probes. Missing inventory, unusable addresses and a failing sibling keep the component and cluster non-Ready, with the instance identity and available native reason in the component member’s status. The same check gates initial join and rolling completion.
## Logstash pipelines and queues
[Section titled “Logstash pipelines and queues”](#logstash-pipelines-and-queues)
Enable `spec.components.extra.logstash.replicas` and size RAM/runtime disk for throughput and buffering. With no user pipelines, Karakoram supplies dual-stack [Beats](https://www.elastic.co/beats) and HTTP inputs and a CA-verified Elasticsearch data-stream output. Inputs bind `::`; input exposure and ports are in [Access and security](/docs/elasticsearch/access-and-security/). Custom listeners do not automatically provision additional routing or secure inputs.
`engineOptions.logstash.pipelines` contains `{id, config}` entries, not raw `pipelines.yml` documents.
| Limit | Value |
| --------- | --------------------------------------------------- |
| Pipelines | At most 10 |
| ID | Unique; `^[a-z][a-z0-9_-]{0,31}$` (1–32 characters) |
| Config | Non-whitespace content; at most 64 KiB per pipeline |
Pipeline substitutions are supplied through the process environment:
| Variable | Value |
| -------------------------- | ------------------------------------------------------------------- |
| `${KARAKORAM_ES_HOSTS}` | Space-separated HTTPS data/combined-node URLs for URI-list settings |
| `${KARAKORAM_ES_HOST}` | First Elasticsearch HTTPS URL |
| `${KARAKORAM_ES_USER}` | Scoped writer username |
| `${KARAKORAM_ES_PASSWORD}` | Writer password |
| `${KARAKORAM_ES_CA}` | Rendered Elasticsearch CA path |
For a custom JSON HTTP intake, save this merge patch as `logstash.json`:
```json
{
"spec": {
"components": {"extra": {"logstash": {"replicas": 1}}},
"engineOptions": {
"logstash": {
"persistentQueue": true,
"pipelines": [{
"id": "jsonlogs",
"config": "input { http { host => \"::\" port => 8080 codec => json } }\noutput { elasticsearch { hosts => [\"${KARAKORAM_ES_HOSTS}\"] user => \"${KARAKORAM_ES_USER}\" password => \"${KARAKORAM_ES_PASSWORD}\" ssl_enabled => true ssl_certificate_authorities => [\"${KARAKORAM_ES_CA}\"] ecs_compatibility => v8 data_stream => true data_stream_type => \"logs\" data_stream_dataset => \"application\" data_stream_namespace => \"default\" } }"
}]
}
}
}
}
```
```sh
karactl clusters patch search -f logstash.json --wait
```
The default writer covers `logstash-*`, `ecs-logstash-*`, `logs-*` and `metrics-*`. Other destinations need a separately authorized writer/role. Never embed `elastic` or real secrets in pipeline source, settings or committed files; see [Elastic output options](https://www.elastic.co/docs/reference/logstash/plugins/plugins-outputs-elasticsearch).
Confirmed `ConfigurationError`/`PluginLoadingError` failures appear in member status and `ComponentsReady=False`, with parser positions when available but without raw source/credentials. They fail the current generation’s convergence operation on observation instead of its normal two-hour deadline; transient startup/monitoring failures retain that deadline. The warning is deduplicated per applied configuration/error. Failed convergence retains the cluster and desired state and does not block independent snapshots, restores or queued operations. Correct the pipeline and submit a new patch: changed configuration is republished even after failed initial deployment or rolling rejoin. Use protected component logs for full diagnostics. Readiness does not establish event delivery.
**Warning:** `persistentQueue: true` persists only to each container’s runtime disk. It survives process restart, but not redeployment, version upgrade, container replacement or service deletion. Queues are neither replicated nor snapshot-protected. Budget disk for every pipeline and retain upstream replay when event loss is unacceptable; see [queue semantics](https://www.elastic.co/docs/reference/logstash/persistent-queues).
## APM intake
[Section titled “APM intake”](#apm-intake)
Enable `spec.components.extra.apm.replicas`. This is standalone APM Server, not Fleet orchestration. `engineOptions.apm.apiKeys` defaults to true; false disables intake API-key authentication while retaining secret-token authentication. Enable public intake only through the [explicit public-route policy](/docs/elasticsearch/access-and-security/).
Configure APM agents with the returned server URL and the revealed intake token. Intake authentication is separate from the APM-to-Elasticsearch writer. Send bearer tokens only over trusted HTTPS: private HTTP intake is unencrypted.
Elasticsearch’s built-in `apm-data` plugin supplies APM templates, ILM policies and ingest pipelines on supported versions (8.15+). Karakoram does not run a separate asset setup job; asset presence is not proof of event delivery.
For manual intake, obtain `APM_URL` from an HTTPS endpoint and `APM_SECRET_TOKEN` from protected credentials without recording either secret in logs. This Bash recipe needs `jq`, `curl` and GNU `date`; it sends NDJSON metadata before a transaction with a current Unix-microsecond timestamp:
```bash
(
set +x
set -euo pipefail
umask 077
work=$(mktemp -d)
trap 'rm -rf -- "$work"' EXIT
printf '%s' "$APM_SECRET_TOKEN" \
| jq -Rs '"header = " + ("Authorization: Bearer " + . | @json)' \
> "$work/curl.conf"
TIMESTAMP_US=$(date +%s%6N)
printf '%s\n' \
'{"metadata":{"service":{"name":"intake-example","agent":{"name":"manual","version":"1.0.0"}}}}' \
"{\"transaction\":{\"id\":\"0123456789abcdef\",\"trace_id\":\"0123456789abcdef0123456789abcdef\",\"name\":\"GET /example\",\"type\":\"request\",\"timestamp\":$TIMESTAMP_US,\"duration\":1.2,\"span_count\":{\"started\":0},\"sampled\":true}}" \
| curl --config "$work/curl.conf" --fail-with-body -i "$APM_URL/intake/v2/events" \
-H 'Content-Type: application/x-ndjson' --data-binary @-
)
```
Expected success is `202 Accepted`; verify indexed data afterward ([intake API](https://www.elastic.co/docs/solutions/observability/apm/elastic-apm-events-intake-api)). For API-key intake, use an APM-authorized key with `Authorization: ApiKey` and its encoded value, keeping it in protected curl configuration as above. A general index key is not automatically valid for intake: use [APM authorization](https://www.elastic.co/docs/solutions/observability/apm/apm-server/apm-agent-authorization). Karakoram supports application descriptors such as `{"application":"apm","privileges":["event:write","config_agent:read"],"resources":["*"]}` through its [native access APIs](/docs/elasticsearch/access-and-security/).
# Install Karakoram
> Choose an infrastructure provider before installing. Karakoram is multi-platform; provider capabilities and operator responsibilities differ. See Providers…
Choose an infrastructure provider before installing. Karakoram is multi-platform; provider capabilities and operator responsibilities differ. See [Providers](/docs/providers/) and [Operating costs](/docs/operating-costs/).
| Provider | Installation path |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Zerops — reference provider | [Zerops provider guide](/docs/zerops/); follow this page or the dashboard-only [Quick start](/docs/getting-started/quickstart/) to install. |
| Dokploy | [Install the control plane on Dokploy](/docs/dokploy/#install-the-control-plane-on-dokploy). |
| Nomad | [Install the control plane on Nomad](/docs/nomad/#install-the-control-plane-on-nomad). |
The deployment steps below are for the **[Zerops](https://zerops.io/) provider**, in an existing Zerops project. CLI installation is shared across providers.
## Prerequisites
[Section titled “Prerequisites”](#prerequisites)
* Permission to import services and deploy code in an existing project. Organization Owner/Admin access and project-creation rights are not required.
* **Control-plane capacity:** one [Ubuntu](https://ubuntu.com/) 24.04 container, shared CPU, 0.5–1 GiB RAM, 1–2 GiB runtime disk and a separate 5 GiB Local Storage volume. A LIGHT project can host this footprint; cluster services require additional capacity. Review engine minimums ([MongoDB](/docs/mongodb/), [RabbitMQ](/docs/rabbitmq/), [Elasticsearch](/docs/elasticsearch/)) and the wizard’s aggregate footprint before creating clusters.
* A Zerops token: **Settings → Access Tokens Management → Custom access per project → Full access**, for this project only, **sudo mode off**. Read-only access cannot provision, update secrets, grow storage or configure routing. Save the one-time value in a password manager; see [Security](/docs/security/) and [Zerops permissions](https://docs.zerops.io/features/rbac#integration-tokens).
* Free service names `karakoram` and `karakoramdata`, or customize them below.
* For local deployment: authenticated [zcli](https://docs.zerops.io/references/cli) with deployment access. The default prebuilt setup needs no [Go](https://go.dev/) or [Node](https://nodejs.org/) toolchain locally or on Zerops.
## Setups: prebuilt release or source build
[Section titled “Setups: prebuilt release or source build”](#setups-prebuilt-release-or-source-build)
Both setups in root `zerops.yaml` use the same runtime and state volume:
| Setup | Build and use |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `karakoram` (default) | Downloads the upstream pinned binary; verifies the checksum manifest’s SSH signature, archive SHA-256 and binary version. Seconds; no Go/Node. Download or verification failure fails the build. |
| `karakoram-source` | Builds the exact deployed commit and UI with Node 22 and Go; allow minutes. For forks, development and unreleased fixes. Requires the complete checkout, including `ui/package-lock.json`, and outbound Go toolchain/module registry and npm access. Zerops `go@1` starts with Go 1.22; `go 1.26.0` selects the required toolchain automatically. |
[`deploy/zerops/RELEASE`](https://github.com/KarakoramDev/Karakoram/blob/main/deploy/zerops/RELEASE) contains the pinned SemVer tag. The v0.2.0 release checkout pins `v0.2.0`; a development checkout can still pin the previous published release. **Never edit it by hand:** only the [release tool](/docs/contributing/#releasing) updates it. A published tag pins itself; `main` between releases pins the latest published release, not newer source. Prebuilt binaries already embed release version, tagged commit and reproducible build date.
For source builds select `zeropsSetup: karakoram-source` in imports, `--source` with the helper, or `--setup karakoram-source` with zcli. Forks must select source: the default always downloads KarakoramDev’s release, never the fork’s source.
## Required secrets and generated settings
[Section titled “Required secrets and generated settings”](#required-secrets-and-generated-settings)
| Setting | Source and purpose |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `KARAKORAM_MASTER_KEY` | Import generates 48 random characters; encrypts stored credentials. Preserve for the state database’s lifetime; back up separately. |
| `KARAKORAM_ADMIN_PASSWORD` | Import generates 24 random characters; creates the first `admin` only, not a password-reset mechanism. |
| `KARAKORAM_ZEROPS_TOKEN` | Supply the project token; templates leave it empty. |
| `KARAKORAM_ZEROPS_PROJECT_ID` | Pipeline uses injected `projectId`; no manual secret needed. |
| `KARAKORAM_DATA_DIR` | `/mnt/karakoramdata`; contains persistent `karakoram.db` state. |
| `KARAKORAM_PUBLIC_URL` | Pipeline derives HTTPS origin from injected `zeropsSubdomain`. |
| `KARAKORAM_BACKUP_ENDPOINT_ALLOWLIST` | Optional trusted private/nonstandard-port S3-verifier destinations; see [Backups and storage](/docs/backups/). Normally unset for Zerops storage. |
| `KARAKORAM_ZEROPS_API_URL` | Optional non-default Zerops API endpoint. |
| `KARAKORAM_OAUTH` | `on` or `off` (default `off`): embedded OAuth authorization for MCP. |
| `KARAKORAM_OAUTH_DCR` | `on` or `off` (default `off`): public dynamic client registration, only effective with OAuth on. |
| `KARAKORAM_OAUTH_CLIENT_HOSTS` | Optional comma-separated host allowlist for HTTPS client ID metadata documents; never permits private destinations. |
| `KARAKORAM_MCP` | Embedded [MCP server](/docs/mcp/): `off`, `readonly` (default), or `readwrite`; invalid values fail startup. |
Zerops reserves `ZEROPS_`: user secrets named `ZEROPS_TOKEN` or `ZEROPS_PROJECT_ID` fail with `userDataZeropsPrefixForbidden`. Use `KARAKORAM_ZEROPS_*`.
**Warning:** Put secrets only on the Karakoram service, not in root `zerops.yaml`, source control, deployment logs or shared project variables. Preserve default service environment isolation. Set an explicit bootstrap password through service secrets on custom deployment paths: if absent, the server generates and logs an initial admin password once.
For a custom domain, replace `run.envVariables.KARAKORAM_PUBLIC_URL` in the pipeline with its HTTPS origin and redeploy. A same-named secret does not override a pipeline runtime variable. Secret changes require restart/redeployment to reach running processes.
## Installation sources
[Section titled “Installation sources”](#installation-sources)
Use [`deploy/zerops/import.yaml`](https://github.com/KarakoramDev/Karakoram/blob/main/deploy/zerops/import.yaml) for public-repository imports, or [`import-local.yaml`](https://github.com/KarakoramDev/Karakoram/blob/main/deploy/zerops/import-local.yaml) for local deployment:
```sh
git clone https://github.com/KarakoramDev/Karakoram
cd Karakoram
```
**Warning:** These are new-install templates. Never re-import over an existing installation; use [Upgrade](/docs/getting-started/upgrade/).
## Local checkout installation
[Section titled “Local checkout installation”](#local-checkout-installation)
Import from the checkout root:
```sh
PROJECT_ID=''
zcli project service-import deploy/zerops/import-local.yaml -P "$PROJECT_ID"
```
Open **karakoram → Environment variables → Secret variables**, set `KARAKORAM_ZEROPS_TOKEN`, and preserve the generated master key and bootstrap password. Deploy the pinned prebuilt release:
```sh
zcli service push karakoram -P "$PROJECT_ID" --setup karakoram \
--zerops-yaml-path zerops.yaml --workspace-state clean --disable-logs
zcli service enable-subdomain karakoram -P "$PROJECT_ID"
```
`--workspace-state clean` uploads only committed `HEAD`; zcli’s default (`all`) also uploads untracked, non-ignored files. Do not add `--no-git`: it uploads the whole checkout, including private imports and `node_modules`. The [helper](#one-command-helper) instead stages only four required files. Commit source/pipeline changes before a clean push.
The local template has neither `buildFromGit` nor an inline pipeline: it creates a `READY_TO_DEPLOY` runtime without code. The push uploads deployment files and downloads the release. Enable the subdomain after deployment: the initial runtime had no HTTP port.
### Build provenance for local pushes
[Section titled “Build provenance for local pushes”](#build-provenance-for-local-pushes)
Only `karakoram-source` needs these inputs. `--no-git` source uploads lack [Git](https://git-scm.com/) metadata; obtain:
```sh
git describe --tags --always
git rev-parse HEAD
```
Merge their **literal output** into the existing `zerops.yaml`, preserving all other pipeline settings:
```yaml
zerops:
- setup: karakoram-source
build:
envVariables:
KARAKORAM_VERSION:
KARAKORAM_COMMIT:
```
Refresh both values for each push. These are remote **build** inputs; workstation exports, `run.envVariables` and post-deploy runtime edits cannot relink the executable. The explicit commit overrides Git discovery; the pipeline records a UTC build date. Without provenance the API/UI show `unknown (built without git metadata)`.
Local builds accept the same inputs (or explicit [Make](https://cgit.git.savannah.gnu.org/cgit/make.git/) `VERSION`/`COMMIT` overrides):
```sh
KARAKORAM_VERSION="$(git describe --tags --always)" \
KARAKORAM_COMMIT="$(git rev-parse HEAD)" make build
```
Check public `GET /api/v1/version` for `version`, `commit`, `buildDate` and `apiVersion`.
### Use different service names
[Section titled “Use different service names”](#use-different-service-names)
Use private copies of the local import and pipeline, not the helper. For runtime `dbcontrol` and volume `dbcontroldata`:
1. Change both import `hostname` values and import that copy; set the token on the renamed runtime.
2. Rename setup `karakoram` to `dbcontrol`, `karakoram-source` to `dbcontrol-source`, and its `extends` to `dbcontrol`. In the base setup, change `run.volume.hostname` to `dbcontroldata` and `run.envVariables.KARAKORAM_DATA_DIR` to `/mnt/dbcontroldata`. Mount hostname/path must agree. No-Git source builds need provenance under the renamed source setup.
3. Update explicit hostname references, including `KARAKORAM_ADVERTISE_URL` if configured. Default internal/public URLs use injected runtime hostname/subdomain; no hard-coded replacement is needed. Keep `./karakoram server` unchanged.
4. Stage the private prebuilt pipeline and every file it reads; `--no-git` is safe here because this is not the checkout:
```sh
UPLOAD_DIR="$(mktemp -d)"
mkdir -p "$UPLOAD_DIR/deploy/zerops"
cp /private/path/zerops.yaml "$UPLOAD_DIR/zerops.yaml"
cp deploy/allowed_signers "$UPLOAD_DIR/deploy/"
cp deploy/zerops/RELEASE deploy/zerops/fetch-release.sh "$UPLOAD_DIR/deploy/zerops/"
zcli service push dbcontrol -P "$PROJECT_ID" --setup dbcontrol \
--working-dir "$UPLOAD_DIR" --zerops-yaml-path zerops.yaml --no-git --disable-logs
rm -r "$UPLOAD_DIR"
zcli service enable-subdomain dbcontrol -P "$PROJECT_ID"
```
For source, push from the checkout with `--setup dbcontrol-source`, `--zerops-yaml-path /private/path/zerops.yaml` and `--workspace-state clean`; a whole-checkout `--no-git` upload requires excluding private files first.
Use the resulting origin for browser, CLI and external URL references.
### One-command helper
[Section titled “One-command helper”](#one-command-helper)
To supply the token before import, create a private untracked copy of `import-local.yaml`, fill its token and restrict access:
```sh
./deploy/zerops/install.sh "$PROJECT_ID" /private/path/import-local.yaml
```
Without that optional argument, the helper uses the token-free template:
```sh
./deploy/zerops/install.sh "$PROJECT_ID"
```
Then add the token in the GUI and restart/redeploy for readiness. Using existing zcli authentication, the helper imports, pushes setup `karakoram`, and enables the subdomain. It stages only `zerops.yaml`, `deploy/allowed_signers`, `deploy/zerops/RELEASE` and `deploy/zerops/fetch-release.sh` in a temporary upload directory. It never loads/stores tokens, creates projects, starts a VPN, renames services or overwrites installations. Delete private import copies afterward.
To build the exact checkout:
```sh
./deploy/zerops/install.sh --source "$PROJECT_ID" /private/path/import-local.yaml
```
`--source` selects `karakoram-source` and uploads the whole checkout with `--no-git`. Keep private imports outside it, exclude sensitive files and configure [provenance](#build-provenance-for-local-pushes) first.
## Import YAML installation from a public repository
[Section titled “Import YAML installation from a public repository”](#import-yaml-installation-from-a-public-repository)
1. Obtain [`deploy/zerops/import.yaml`](https://github.com/KarakoramDev/Karakoram/blob/main/deploy/zerops/import.yaml). Its `buildFromGit` points to `https://github.com/KarakoramDev/Karakoram`; no fork required. Forks need a plain **public** [GitHub](https://github.com/)/[GitLab](https://about.gitlab.com/) URL without trailing `.git` and `zeropsSetup: karakoram-source`; private repositories are unsupported.
2. Set the empty token in a private copy, or add it in the GUI immediately after import.
3. Choose **Import services** in the existing project’s dashboard, or run:
```sh
PROJECT_ID=''
zcli project service-import /private/path/import.yaml -P "$PROJECT_ID"
```
4. Wait for build and deployment, not just import acceptance. If the token was empty, set it and restart/redeploy after the build.
Both templates require `#yamlPreprocessor=on` to generate secrets. Multi-argument preprocessor functions require a space after each comma. `buildFromGit` is a one-time source import, not an automatically configured webhook integration.
## Install karactl
[Section titled “Install karactl”](#install-karactl)
Match the CLI release to the **deployed control-plane version**, not the latest release independently. Read `https://example.com/api/v1/version`; with an existing CLI, `karactl --server https://example.com version` shows client/server versions. If the server reports unknown provenance, identify its deployed release/revision before choosing a binary.
[Release assets](https://github.com/KarakoramDev/Karakoram/releases) cover [Linux](https://www.kernel.org/)/[macOS](https://www.apple.com/macos/) (`linux`/`darwin`, `amd64`/`arm64`) and [Windows](https://learn.microsoft.com/en-us/windows/) (`windows-amd64.exe`) from v0.1.0; signed manifests start at **v0.1.2**. Linux amd64:
```sh
RELEASE=v0.2.0
ASSET=karactl-linux-amd64
BASE="https://github.com/KarakoramDev/Karakoram/releases/download/$RELEASE"
curl --fail --location --proto '=https' --proto-redir '=https' \
--remote-name "$BASE/$ASSET" \
--remote-name "$BASE/checksums.txt" \
--remote-name "$BASE/checksums.txt.sig" &&
curl --fail --location --proto '=https' --proto-redir '=https' --output allowed_signers \
"https://raw.githubusercontent.com/KarakoramDev/Karakoram/$RELEASE/deploy/allowed_signers" &&
ssh-keygen -Y verify -f allowed_signers -I release@karakoram.dev \
-n karakoram-release -s checksums.txt.sig < checksums.txt &&
sha256sum --ignore-missing -c checksums.txt &&
install -Dm755 "$ASSET" "$HOME/.local/bin/karactl"
```
Authenticate the signing key against a trusted checkout or independently trusted fingerprint: downloading it beside the binary does not establish authenticity. Verify in the download directory; proceed only after signature success and the selected asset reports `OK`. On macOS use `shasum -a 256 --check --ignore-missing checksums.txt` for checksums; Windows needs [OpenSSH](https://www.openssh.com/) signature verification and SHA-256 verification. Add `$HOME/.local/bin` to `PATH`.
Alternatively build the matching checkout with Go 1.26+:
```sh
go build -o /tmp/karactl ./cmd/karactl
install -Dm755 /tmp/karactl "$HOME/.local/bin/karactl"
```
## First sign-in and CLI connection
[Section titled “First sign-in and CLI connection”](#first-sign-in-and-cli-connection)
1. Wait for `karakoram` to become `ACTIVE`, open its **Public access** HTTPS subdomain, and check:
```sh
curl --fail --show-error https://example.com/healthz
```
2. Read the service’s `KARAKORAM_ADMIN_PASSWORD`; sign in as `admin`, choose a unique password of at least 12 characters, and sign in again before creating API tokens. Changing the bootstrap variable does not reset existing accounts; see [admin recovery](/docs/operations/#recovering-local-admin-access).
3. Connect using the changed password:
```sh
karactl --server https://example.com --context production login --username admin
karactl --context production health
karactl --context production version
```
Login prompts without echo, creates an API token and stores the context. Protect CLI configuration as a credential.
4. For automation, create a token for a restricted account through the API or `karactl tokens create -f token.yaml` (`name`, optional `expiresAt`). Save its one-time `kk_` secret in a secret manager:
```sh
export KARAKORAM_SERVER='https://example.com'
# Supply KARAKORAM_TOKEN securely without logging it.
karactl health
```
Karakoram API tokens are not Zerops tokens; they inherit their owner’s current [Karakoram permissions](/docs/access-control/#api-tokens-and-browser-sessions).
Add other projects through [namespace registration and rotation](/docs/namespaces/#add-validate-and-rotate).
## Deployment boundaries
[Section titled “Deployment boundaries”](#deployment-boundaries)
* Use HTTPS for the public UI/API; review exposure and origin requirements in [Security](/docs/security/). **Do not publish port 9443**, the private authenticated agent endpoint.
* Retain one control-plane container and `deploy.temporaryShutdown: true`; review [availability and placement limits](/docs/limitations/).
* Configure proxies for [events and pagination](/docs/api/#pagination-and-events); a custom domain alone does not disable buffering.
* Back up [control-plane state and master key](/docs/operations/#control-plane-state-backup-and-recovery) separately from [engine backups](/docs/backups/).
For startup, deployment or credential failures, see [Operations and troubleshooting](/docs/operations/).
# Quick start
> Choose a provider, install the control plane, then create a cluster and back it up. See Providers for capability differences and Operating costs for costs.
Choose a provider, install the control plane, then create a cluster and back it up. See [Providers](/docs/providers/) for capability differences and [Operating costs](/docs/operating-costs/) for costs.
| Provider | Quick-start path |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Zerops — reference provider | Follow the dashboard-only steps below; no checkout or CLI required. |
| Dokploy | Follow [Install the control plane on Dokploy](/docs/dokploy/#install-the-control-plane-on-dokploy), then use the engine guides linked below. |
| Nomad | Follow [Install the control plane on Nomad](/docs/nomad/#install-the-control-plane-on-nomad), then use the engine guides linked below. |
The steps below use the **[Zerops](https://zerops.io/) provider**. You need a Zerops account and existing project. LIGHT hosts the control plane; clusters need **additional capacity**. Check engine sizing below and the wizard’s total against project capacity. Other Zerops deployment paths: [Install](/docs/getting-started/install/).
## 1. Create a Zerops token
[Section titled “1. Create a Zerops token”](#1-create-a-zerops-token)
Open **Settings → Access Tokens Management**. Choose **Custom access per project → Full access** for this project only, with **sudo mode off**. Save the one-time token securely; see [Security](/docs/security/) for the privilege boundary.
## 2. Import Karakoram
[Section titled “2. Import Karakoram”](#2-import-karakoram)
Choose **Import services** in the project. Copy this template, then replace the empty `KARAKORAM_ZEROPS_TOKEN` with your token before importing:
```yaml
#yamlPreprocessor=on
# SPDX-License-Identifier: Apache-2.0
# Copyright 2026 The Karakoram Authors
# Set buildFromGit to your fork's public URL when deploying a fork, and set
# zeropsSetup to karakoram-source: the default setup deploys the upstream release.
# To deploy a local checkout, use import-local.yaml and install.sh.
services:
- hostname: karakoramdata
type: local-storage:single@1
verticalAutoscaling:
minDisk: 5
maxDisk: 5
- hostname: karakoram
type: ubuntu@24.04
minContainers: 1
maxContainers: 1
enableSubdomainAccess: true
buildFromGit: https://github.com/KarakoramDev/Karakoram
zeropsSetup: karakoram
envSecrets:
KARAKORAM_MASTER_KEY: <@generateRandomString(<48>)>
KARAKORAM_ADMIN_PASSWORD: <@generateRandomString(<24>)>
# Replace before import, or set in the service GUI and restart after import.
# Zerops reserves ZEROPS_* for system variables.
KARAKORAM_ZEROPS_TOKEN: ""
verticalAutoscaling:
cpuMode: SHARED
minCpu: 1
maxCpu: 1
minRam: 0.5
maxRam: 1
minDisk: 1
maxDisk: 2
```
This is [`deploy/zerops/import.yaml`](https://github.com/KarakoramDev/Karakoram/blob/main/deploy/zerops/import.yaml). Default `karakoram` downloads/verifies the signed release pinned on `main` (seconds); `zeropsSetup: karakoram-source` builds the exact commit (minutes). Wait for **Active**. Delete token-bearing YAML; preserve the generated master key separately.
## 3. Sign in
[Section titled “3. Sign in”](#3-sign-in)
1. Open **karakoram → Public access** and its HTTPS subdomain.
2. Read `KARAKORAM_ADMIN_PASSWORD` under **Environment variables → Secret variables**.
3. Sign in as `admin`, set a different password of at least 12 characters, then sign in again.
## 4. Add storage
[Section titled “4. Add storage”](#4-add-storage)
**Storage is required before any backup.** Create an [S3](https://aws.amazon.com/s3/) bucket, for example using Zerops Object Storage. Open **Settings → Backup storages**, then **Create**:
| Field | Value |
| --------------------------------- | --------------------------------------------------------- |
| Name | e.g. `backups`; 2–63 lowercase letters, digits or hyphens |
| Endpoint URL | Object Storage endpoint shown in Zerops |
| Region | `us-east-1` for Zerops storage |
| Bucket | Your bucket name |
| Access key ID / Secret access key | Object Storage access details |
Keep other defaults. Choose **Verify** and confirm read/write/delete access. See [Backups and storage](/docs/backups/) for other endpoints and retention rules.
## 5. Create a cluster
[Section titled “5. Create a cluster”](#5-create-a-cluster)
Choose **New cluster**, select the engine and name, then:
| Engine | Wizard choices for a first cluster | Next page |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| MongoDB | **Percona Server for MongoDB → replica set**; 1 member only for disposable tests, or 3 voters for quorum-based failover when placement permits. Host separation is [not guaranteed](/docs/limitations/#zerops-failure-domains-and-storage). | [Create and size](/docs/mongodb/#create) |
| RabbitMQ | **RabbitMQ**; 1 broker for disposable tests or 3 brokers. | [Create and configure](/docs/rabbitmq/#create-and-configure) |
| Elasticsearch | **Elasticsearch → Combined master and data nodes**; `dev` preset for disposable tests or `prod` for three combined nodes. | [Create and size](/docs/elasticsearch/#create) |
Review member sizing and total project capacity. Select backup storage; keep the advertised version, TLS required and public access off. Review and create; wait for **Ready**.
## 6. Take a backup
[Section titled “6. Take a backup”](#6-take-a-backup)
Open **Cluster → Backups → Backup now**. This is asynchronous: follow the operation and wait until backup inventory reports **Succeeded**. If storage was skipped, add it before retrying.
* [MongoDB](/docs/mongodb/backups/): [Percona Backup for MongoDB](https://docs.percona.com/percona-backup-mongodb/latest/) (PBM) logical or physical backup.
* [RabbitMQ](/docs/rabbitmq/backups/): definitions only—no messages or point-in-time recovery (PITR).
* [Elasticsearch](/docs/elasticsearch/snapshots/): native snapshot.
## 7. Connect an application
[Section titled “7. Connect an application”](#7-connect-an-application)
Use the private network from an application service in the same project. **Revealed credentials are the engine’s administrative identity, not application credentials.** Create a scoped application identity and deliver its credentials, endpoints and CA through runtime secrets/configuration:
* [MongoDB users, roles and TLS](/docs/mongodb/access-and-tls/)
* [RabbitMQ access and TLS](/docs/rabbitmq/access-and-tls/)
* [Elasticsearch access and security](/docs/elasticsearch/access-and-security/)
## 8. Connect an AI agent
[Section titled “8. Connect an AI agent”](#8-connect-an-ai-agent)
The control plane serves a read-only [MCP server](/docs/mcp/) at `/mcp` out of the box. Mint a restricted token with `karactl mcp config --client claude-code|cursor|vscode|codex` and paste the printed configuration into your client; the [MCP quick start](/docs/mcp-quickstart/) covers each client, write access and per-cluster [data MCP](/docs/data-mcp/).
## What next
[Section titled “What next”](#what-next)
* [CLI](/docs/cli/), [cluster operations](/docs/clusters/) and [Backups and storage](/docs/backups/).
* [Upgrade](/docs/getting-started/upgrade/) existing installations; never re-import the installer.
* [Limitations](/docs/limitations/) before production; [Operations](/docs/operations/) for troubleshooting.
# Upgrade and recover Karakoram
> Upgrade the control plane and recover its state; engine-version changes are separate operations documented under each engine.
Upgrade the control plane and recover its state; engine-version changes are separate operations documented under each engine.
## Upgrading to v0.4.2
[Section titled “Upgrading to v0.4.2”](#upgrading-to-v042)
Back up control-plane state using the [consistent pre-upgrade backup](#consistent-pre-upgrade-state-backup), preserve the original `KARAKORAM_MASTER_KEY` separately, and retain the previous executable with its pre-upgrade state before deploying v0.4.2.
```sh
git fetch origin --tags && git checkout v0.4.2
zcli service push karakoram -P "$PROJECT_ID" --setup karakoram \
--zerops-yaml-path zerops.yaml --workspace-state clean --disable-logs
```
There are **no new SQLite migrations** from v0.4.1 to v0.4.2; the inventory in `internal/store/migrations` still ends at `017_backup_drills.sql`. There are no new engine resource minimums, Go dependency changes or native artifact lock changes in this release. The v0.4.1 memory floors still apply. Control-plane rollback does not undo engine or infrastructure changes.
### Changed behavior
[Section titled “Changed behavior”](#changed-behavior)
* Dokploy retains a maximum of four build attempts per rendered deployment content/retry epoch across control-plane restarts. Exhausting this budget reports `DeploymentRetriesExhausted`; inspect deployment logs and correct the failed deployment rather than expecting indefinite retries.
* Temporal rejects an initial data source at admission because it does not support create-from-backup. This does not remove its separate restore support.
* Keycloak fills each omitted resource field independently from catalog defaults, including disk-only requests; it does not resize existing clusters automatically.
* Resume and recovery restore only previously admitted exposure during a closed maintenance window. Deferred exposure changes remain deferred and unacknowledged.
### Fixed in v0.4.2
[Section titled “Fixed in v0.4.2”](#fixed-in-v042)
Unless noted otherwise, the following issues affected both v0.4.0 and v0.4.1:
* **MySQL Group Replication formation:** generated recovery credentials now fit MySQL’s limit, with sanitized startup diagnostics. Delete and re-create never-formed legacy groups; do not force bootstrap or edit coordinator state. Existing single-member clusters receive compatible recovery credentials before their first adoption into a group.
* **PXC formation on Zerops:** startup persists the required files and directory chain without a filesystem-wide sync that the platform rejects.
* **OpenBao rolling status:** Updating or Upgrading remains visible between member replacements until convergence completes. This was a v0.4.1 regression; v0.4.0 was not affected by the intermediate Ready status.
* **Clone readiness:** OpenBao, MySQL, PXC, Neo4j and Weaviate now acknowledge completed initial restores so clone creation can finish and report Ready. Temporal now rejects unsupported initial data sources rather than accepting a clone request it cannot complete.
* **Keycloak resource defaults:** omitted CPU, memory and disk fields receive their catalog defaults independently, including requests supplying only disk.
* **Dokploy deployment retries:** failed initial builds remain observable and retryable, with a bounded build-attempt budget retained across control-plane restarts, rather than leaving creation blocked behind a failed build.
* **MySQL Group Replication primary handoff:** restart, replacement and rolling maintenance can select a new primary using the native MySQL function.
* **Resume and exposure recovery:** already-admitted TCP and HTTPS exposure can recover during a closed maintenance window without admitting deferred exposure changes. This covers the affected database and service engines, including OpenBao resume and Supabase gateway recovery. This was a v0.4.1 regression; v0.4.0 was not affected.
* **MySQL and PXC full backups:** the control plane recognizes XtraBackup’s native full-backup checkpoint type, allowing full backups and their dependent restore, clone and point-in-time recovery baselines. This fix needs only a control-plane upgrade, not a member-agent update. Take and verify a new full backup; a previously failed backup is not a recovery baseline.
* **MySQL and PXC in-place restore:** the restore operation now releases the server stop holds during re-formation so the restored seed and followers can start and restore can complete. This affected v0.4.0 and v0.4.1 but was previously masked by the full-backup metadata defect described above.
* **MySQL Group Replication restore follower re-clone:** in-place recovery can re-clone followers after restoring the seed. Clone recipients fence clients while preparing old data for replacement, and rejected native starts can be retried only when copying never began. PXC uses its own Galera recovery path.
* **MySQL and PXC point-in-time recovery:** binlog archiving now preserves its deadline across configuration refreshes instead of being postponed indefinitely. Archive lag and recovery-window status now reflect the completed archive rather than configuration refresh timing.
* **ProxySQL configuration reloads:** the ProxySQL agent no longer reloads unchanged configuration every minute, which briefly routed writes to replicas and caused occasional failed writes.
* **MySQL clone failure diagnostics:** operation events now report the clone stage and, when available, the MySQL error number without exposing SQL, credentials or raw server error messages.
### Existing MySQL and PXC agents
[Section titled “Existing MySQL and PXC agents”](#existing-mysql-and-pxc-agents)
The MySQL Group Replication clustered-restart/primary-handoff, in-place restore follower re-clone, MySQL/PXC PITR binlog archiving and ProxySQL reload fixes are **agent-side**, as are the new clone diagnostics. Existing MySQL and PXC clusters receive these fixes **only by being re-created with v0.4.2 agents**. For example, take and verify a backup, clone or restore it into a new cluster on v0.4.2, verify the new member builds and recovered data, then switch clients. Back up and verify an independent recovery path before deleting any cluster; preserve application data when planning re-creation.
`upgrade-agents` covers **PostgreSQL, Supabase, MongoDB, RabbitMQ and Elasticsearch only**; it does not cover MySQL or PXC. Upgrading only the control plane or restarting the native database process does not replace agent code. The separate in-place restore **stop-hold release** fix is control-plane-side and does not itself need an agent upgrade; successful follower re-clone still requires the updated agents described above.
### Other engine agents
[Section titled “Other engine agents”](#other-engine-agents)
For an upgrade **from v0.4.1**, the v0.4.2 fixes for **OpenBao** (rolling status, clone readiness and exposure recovery), **Neo4j** and **Weaviate** (clone readiness and exposure recovery), **Keycloak** (resource defaults and exposure recovery), and **Temporal** (initial-data-source rejection) are **control-plane-side**. Upgrading the control plane is enough for these fixes; these engines do not need re-creation for a v0.4.1 → v0.4.2 upgrade. Dokploy initial-build retry and shared exposure-recovery fixes also run in the control plane. The v0.4.2 exposure-recovery fixes for **PostgreSQL (including Citus), Supabase, MongoDB, RabbitMQ and Elasticsearch** are also control-plane-side; these fixes do not themselves require an agent rollout.
OpenBao, Neo4j, Weaviate, Keycloak and Temporal still do **not** support `upgrade-agents`. This statement does not install agent-side fixes from older releases: when upgrading from v0.4.0, also follow the separate v0.4.1 agent guidance below. MySQL/PXC formation durability and sanitized startup/clone diagnostics include agent-side changes; use new v0.4.2 members, not merely a database-process restart.
### Known issues in v0.4.2
[Section titled “Known issues in v0.4.2”](#known-issues-in-v042)
* **PostgreSQL administrator password rotation remains unavailable.** The revealed `postgres` password cannot be rotated in place. Follow [administrator credential containment](#postgresql-administrator-password-cannot-be-rotated-yet).
* **PXC after an in-place restore on Zerops:** after either a full or PITR restore, one member can be flagged `GTIDHistoryDiverged` and the cluster reports Degraded even though the checked data and writes are correct. Investigation is ongoing; verify recovered data and replication with native SQL rather than treating the operation’s success as complete acceptance. The flag can block incremental backups and PITR continuity, and taking a new full backup has not reliably cleared it. This was not reproduced on Dokploy.
* **Nomad live coverage of the new engines is incomplete.** Implemented capabilities do not establish full lifecycle/recovery acceptance. OpenBao, Temporal, MySQL/PXC, Neo4j, Weaviate, Keycloak and Citus remain EXPERIMENTAL, as does Data MCP; see [engine availability](/docs/providers/#engine-availability).
* **Temporal is not supported on Zerops:** membership requires private IPv4 networking, while Zerops private networking is IPv6. Use Dokploy or Nomad within their documented capability limits.
* **Weaviate cannot start on Zerops** (also present in v0.4.0 and v0.4.1): members never become Ready because Zerops private networking is IPv6-only and Weaviate’s cluster address is not advertised for IPv6. Nomad and Dokploy are not affected. Use Weaviate on Dokploy or Nomad until the next release.
Other continuing limitations include [empty Docker volume objects after Dokploy deletion](/docs/dokploy/) and PostgreSQL’s configuration acknowledgement wait after binding/access changes; backups still require readiness.
## Upgrading from v0.4.0 to v0.4.1
[Section titled “Upgrading from v0.4.0 to v0.4.1”](#upgrading-from-v040-to-v041)
Back up control-plane state using the [consistent pre-upgrade backup](#consistent-pre-upgrade-state-backup) and preserve the original `KARAKORAM_MASTER_KEY` separately before deploying. Retain the v0.4.0 executable with its pre-upgrade state. Then deploy v0.4.1:
```sh
git fetch origin --tags && git checkout v0.4.1
zcli service push karakoram -P "$PROJECT_ID" --setup karakoram \
--zerops-yaml-path zerops.yaml --workspace-state clean --disable-logs
```
There are **no new SQLite migrations** in v0.4.1; the migration inventory still ends at 017. This does not make rollback of subsequent engine or infrastructure changes automatic. Engine packages and member agents are separate from the control-plane executable.
### Member-agent rollout and changed minimums
[Section titled “Member-agent rollout and changed minimums”](#member-agent-rollout-and-changed-minimums)
Before any PostgreSQL member redeployment, follow the [version-drift inventory](#postgresql-exact-versions-and-existing-version-drift) below. Recover an access-held restore before requesting an agent rollout. On a healthy cluster, `karactl -n NS clusters upgrade-agents NAME --wait --timeout 30m` supports **PostgreSQL (including Citus), Supabase, MongoDB, RabbitMQ and Elasticsearch**. It retains engine versions and honors maintenance windows and health/quorum gates. Follow [member-agent rollout](#rolling-out-member-agents) and verify every member’s reported build afterward.
**OpenBao, Temporal, MySQL, PXC, Neo4j, Weaviate and Keycloak do not support `upgrade-agents`.** Existing executables are not replaced by restarting the control plane or native process. Agent-side fixes reach these engines only through control-plane-managed member re-creation; use deletion/recreation for disposable clusters, or the engine’s documented backup/new-target recovery where supported. Preserve and verify valuable data first. There is no generic data-preserving agent-only upgrade for these engines; consumer deletion is especially destructive, as explained below. Do not use volume retention or copying binaries as a substitute.
Agent-side changes include Keycloak JDBC and Temporal SQL driver settings; MySQL Group Replication pre-bootstrap observation and message-cache budgeting; MySQL/PXC address-family-aware SQL listeners; PXC numeric Galera binding, managed access serialization and archive eligibility; and Neo4j idempotent restore credential recovery and its reload/error handling. PostgreSQL agents carry the listener/proxy, TimescaleDB and coordinated Citus recovery fixes. Control-plane-only fixes, including idle resource-update suppression, status publication and maintenance admission, apply without replacing agents.
MySQL Group Replication and PXC now require **`minRam >= 2` GiB per data member**, even with custom buffer settings; raise `maxRam` with it when necessary. MySQL single-member deployments retain their 1 GiB floor. Both providers retain the 5 GB data-disk floor; proxy resources are separate. Create/update and dry-run admission reject undersized specifications, including unrelated updates to an existing undersized cluster. Existing clusters are **not automatically resized** by deployment of the control plane. Correct their requested RAM through supported resource maintenance, or delete/re-create an unsuccessful fresh deployment. A RAM-only resize is not proof that a new agent was installed. The Group Replication message-cache fix specifically needs new data-member agents; the PXC admission floor itself is control-plane-only. See [MySQL resource guidance](/docs/mysql/configuration/#mysql-group-replication-only). The higher floor is not a bootstrap workaround: MySQL Group Replication cannot form on any provider in this release, and PXC cannot form on Zerops. See [known issues in v0.4.1](#known-issues-in-v041).
Explicit 1 GB disks are now admitted for Keycloak, Neo4j and Weaviate, matching their engine minimums instead of the old shared 5 GB request floor. This does not shrink existing volumes or add disk growth to Dokploy/Nomad.
There are **no new native release-asset bundles**. The four Percona Server lock entries add Ubuntu 24.04’s `libtirpc3t64` runtime dependency; preparation installs it through APT. Exact-byte native artifact mirrors need no additional bundle, but restricted/offline installations must provide this package and its dependencies through their Ubuntu package source. Source-built control planes must use a published artifact release explicitly when needed; see [native artifacts](/docs/native-artifacts/).
Disruptive changes for the new engines, including runtime-replacing HTTPS, hostname and public TCP route changes, now wait for maintenance admission. Use a generation-scoped explicit override only when intended; safety checks still apply. Unchanged updates no longer cancel in-flight work. Status now retains observed members on failed reads, distinguishes version drift from configuration acknowledgement waits, and reports build inventory independently of rollout support. These corrections do not turn missing evidence into health.
### Keycloak and Temporal PostgreSQL proxy compatibility
[Section titled “Keycloak and Temporal PostgreSQL proxy compatibility”](#keycloak-and-temporal-postgresql-proxy-compatibility)
v0.4.0 Keycloak can fail before readiness when its PostgreSQL binding uses ProxySQL: the proxy rejects mixed text/binary result-column formats and named cursor portals. Disabling binary transfer alone is insufficient: Keycloak’s Hibernate fetch size of 64 still creates named portals in transactions. New consumer deployments use Keycloak JDBC `preferQueryMode=simple`, covering both restrictions even with explicit fetch sizes. JDBC parameter encoding and TLS hostname verification remain enabled, but server-side preparation and incremental cursor fetching are disabled; large results are buffered in memory. Temporal uses the `postgres12` (lib/pq) plugin with `disable_prepared_binary_result=yes`, including for explicitly prepared visibility queries; lib/pq binds the unnamed portal. The writer proxy remains in place. OpenBao 2.7.1 PG storage uses pgx but only single-column reads, so the mixed-format rejection does not affect it.
**Upgrading the control plane alone does not install these agent-rendered settings. Neither provider supports `upgrade-agents`; process restart/reload also retains the old executable.** Do not copy binaries into running members.
For a failed, disposable consumer with **no valuable application state**, save its API create request (not response-only fields), record the source/binding and database names, then after deploying the fixed control plane and configured member artifact, recreate **only the consumer**:
```sh
karactl -n NS clusters delete CONSUMER --wait --yes
karactl -n NS clusters create -f consumer.json --wait --timeout 30m
karactl -n NS clusters status CONSUMER -o json
karactl -n NS clusters members CONSUMER -o json
```
This is full deletion: it **drops the consumer-owned bound databases and login**. Never target the source PostgreSQL cluster. Verify the new member’s build identity and native readiness, not merely a Ready binding. A Keycloak deployment that never passed initial Liquibase startup may have only incomplete bootstrap state, but confirm there is no imported/pre-existing data or external writes before deleting it. Do not generalize this to an already-used Keycloak cluster.
Temporal can accept writes before a later visibility lookup fails. Readiness failure or the proxy error does **not** prove it is empty: deletion can destroy workflow histories, running executions/timers/tasks, namespace metadata and SQL visibility. For any valuable Temporal or Keycloak data, stop application writes, take and verify a source PostgreSQL backup, retain the current consumer and binding, and arrange an explicitly reviewed data-preserving migration or deployment procedure. The disposable recreation sequence is not that procedure; this release does not add a generic in-place agent rollout for these providers.
**Do not use `--keep-volumes` suspend/resume as an agent-upgrade shortcut.** Binding release with retention disables the login and removes its binding record; a subsequent new binding incarnation can reject the retained database owner rather than safely adopt it. Retention is not a transparent redeployment or a backup. Do not edit internal state, role ownership or platform logical records to bypass this fence.
### PostgreSQL exact versions and existing version drift
[Section titled “PostgreSQL exact versions and existing version drift”](#postgresql-exact-versions-and-existing-version-drift)
In v0.4.0, installing PostgreSQL profiles could upgrade a previously exact-pinned server as an APT dependency. For example, a cluster assigned Percona PostgreSQL 17.10.1 with PostGIS could actually run 17.11.1 while member status still reported 17.10.1. This affects the shared PostgreSQL/Supabase core on all providers. Native-artifact mirrors do not freeze the separate APT repositories.
**Before refreshing existing member agents, changing profiles/resources, or replacing a data runtime, inventory every member with `SELECT version()` over a verified connection.** Old agents echo assigned versions, so v0.4.0 status is not proof of the installed release. Record SQL results and take a backup. If a member runs a newer catalog release than requested, explicitly update the cluster’s desired release to that observed release (or a supported newer one) before any runtime preparation. Do not install older packages over the existing data directory to make the binary match the old specification.
For PostgreSQL 16, SQL and older agents may report only the upstream release `16.15`, with no Percona package revision. Select catalog version `16.15.1` for that release (or a newer supported same-major catalog version). Missing product revisions are not treated as downgrades or unresolved drift; an explicitly reported newer product revision still blocks an older pin. The corrective engine upgrade rolls members normally, including when they still run v0.4.0 agents. Wait for its generation to converge before `upgrade-agents`. An agent-only rollout before correcting the stale database pin is rejected rather than reinstalling older database packages. Public member versions retain the actual SQL release, which can remain `16.15` after alignment to catalog `16.15.1`.
v0.4.1 keeps the requested server package as the APT candidate throughout profile installation. Updated member agents report the running Percona product release, not configuration intent. An unavailable SQL version observation is empty. An unexpected observed data-member release raises `VersionDrift=True` with the member, observed/desired releases and recovery guidance, and the cluster is not Ready. Expected source releases during a forward rollout remain Updating instead of producing drift incidents. A confirmed mismatch survives missing observations as `VersionDrift=Unknown` until fresh SQL confirms the target; missing observations alone do not create drift on unaffected members. The specification is never rewritten automatically. Preparing a runtime below its observed server release is rejected, including agent-only redeployment.
For an affected 17.10.1-assigned/17.11.1-running cluster, explicitly select 17.11.1, complete the same-major reconciliation and agent refresh, verify SQL and member status agree on every member, then take a new backup. Only then request an adjacent major upgrade. Major-upgrade admission reports unresolved assignment/ observation drift explicitly rather than rejecting an actual-release backup as though no current backup existed. Review [PostgreSQL version-drift recovery](/docs/postgres/#existing-version-drift) before proceeding. This upgrade neither downgrades engines nor recreates clusters.
### PostgreSQL restores stuck at access
[Section titled “PostgreSQL restores stuck at access”](#postgresql-restores-stuck-at-access)
v0.4.0 accidentally sent every managed-role recovery password request through the binding mutation scope. The agent rejected it before SQL execution because normal binding mutations require an OID-fenced journal. This affects every provider, including single-member and HA PostgreSQL/Supabase; it requires a managed login role present in the recovered catalog. It is not an IPv6, TLS, extension-profile or PostgreSQL minor-version error. v0.2.0 and v0.3.0 do not contain this regression.
An affected restore is Failed and the cluster Degraded at `access`, with client HBA rules and proxy users intentionally held closed. **Do not submit another restore or edit HBA files.** Preserve the control-plane database/master key, backup objects and sealed recovery credentials.
For a stuck cluster with only ordinary managed roles (no `kb_` binding roles), deploy the fixed control plane first, preserving its state and master key. The fixed control plane uses the original password-only protocol accepted by v0.4.0 member agents: the existing journal automatically resumes, rekeys roles, publishes access and releases client holds. No new API action is required. Wait for `RestoreRecovery` to clear, restore-resource success, working verified SQL and fresh cluster readiness before refreshing agents. The original Failed operation remains historical evidence. If version drift also exists, resolve the actual/desired release mismatch as described above **after recovery and before any member redeployment**; this recovery does not downgrade PostgreSQL.
New control-plane/member-agent pairs also support password-only recovery of binding identities, gated by the member’s client-held state. **An already stuck v0.4.0 source containing recovered `kb_` roles cannot use `upgrade-agents` as an escape:** its restore journal owns the operation queue. Skipping binding rekey is not safe: normal binding ownership checks do not reapply a password rotated since the backup, and could leave a nominally repaired consumer unable to log in. There is no in-place legacy-agent bypass. Preserve the backup and use the supported delete/new-target recovery in [failed restore and client holds](/docs/postgres/backups/#failed-restore-and-client-holds). Remove dependent consumers/bindings through their supported lifecycle first if they prevent source deletion. Do not copy executables into running containers.
Before another restore, complete a supported member-agent rollout on a healthy cluster. A failed destructive restore never opens incomplete data merely because a retry budget expires: repair and automatic continuation, or explicit target abandonment and new-target recovery, are the supported outcomes.
### PostgreSQL administrator password cannot be rotated yet
[Section titled “PostgreSQL administrator password cannot be rotated yet”](#postgresql-administrator-password-cannot-be-rotated-yet)
**v0.4.1 does not support in-place rotation of the `postgres` administrator credential returned by the connection endpoint.** `system-users rotate` returns HTTP 409 with an explanation; the other internal PostgreSQL system roles are also not rotatable through this command. Native application-role password rotation and managed binding rotation do not rotate the administrator. In-place administrator rotation is planned for the next release.
If this credential is exposed, restrict access at both private and public network boundaries and terminate existing privileged sessions; removing public exposure alone does not disconnect them. Treat the exposure as a superuser compromise and assess data, schema and role changes. If retaining data, quiesce writers and restore or clone trusted data into a **fresh cluster**, which gets new system credentials. Verify the recovered data, new authentication and rejection of the exposed password on every new client route. Physical recovery also carries application identities from the source catalog; inspect them for compromise.
Migrate applications and bindings deliberately: binding sources are immutable, so bound consumers require recreation and an explicit data migration. Resolve deletion guards, retire the compromised source through the normal lifecycle, verify its endpoints/runtimes are gone, and securely retire exposed connection files. In-place restore, suspension/resume and TLS rotation are not password revocation; manual SQL password changes or private rekey calls are not supported workarounds. Follow the detailed [administrator credential containment procedure](/docs/postgres/access-and-tls/#administrator-credential-containment) and [CLI rotation limitations](/docs/cli/#command-groups).
### Citus restores require updated member agents
[Section titled “Citus restores require updated member agents”](#citus-restores-require-updated-member-agents)
v0.4.0 Citus restores could stall at credential recovery because system-role changes were implicitly propagated to workers whose client access was intentionally fenced. v0.4.1 performs node-local, coordinated credential recovery on every group primary before releasing access; it does not relax client fences to make recovery succeed.
**Both the updated control plane and updated member agents are required.** Older agents reject the new local password-recovery request before SQL execution; the restore fails closed with recovery holds retained rather than downgrading to an unsafe request. Updating the control plane alone does not replace member executables. Complete the supported agent rollout on a healthy formation before starting a restore.
An already failed v0.4.0 Citus restore keeps its recovery journal and holds: neither another in-place restore nor `upgrade-agents` can take ownership. Do not cancel recovery, edit HBA rules, clear journals or copy executables into running members. Preserve control-plane state, the master key, sealed recovery credentials, the retained backup inventory and all group repositories/WAL. Recover by abandoning the failed target with backups retained:
```sh
karactl -n NS clusters delete FAILED_TARGET --keep-backups --yes --wait
```
Do **not** add `--keep-volumes`. Wait for deletion and verify the original runtimes and volumes are gone. Confirm the retained coordinated backup and archives still exist, then prepare a complete cluster-create request named `citus-recovery.json` for a **new** target with fixed member artifacts, source-compatible PostgreSQL version, distributed topology, matching per-group replica count, worker-group count, Citus database and profile requirements. Configure the target’s registered backup storage and resources. Within `spec`, select the retained backup by name:
```json
"dataSource": {
"namespace": "NS",
"cluster": "FAILED_TARGET",
"backup": "RETAINED_BACKUP"
}
```
Use backup-name selection, not a timestamp or live-source PITR request: the retained aggregate resolves its own coordinated named point after source deletion.
The retained PostgreSQL source is identified by its recorded pgBackRest repository and backup label, with the aggregate’s group repositories, labels, named restore point and replay bounds for Citus. It does not require a `location` field or a snapshot-time native-binary URL. v0.4.0 incorrectly required `location` after source deletion even though coordinated backups do not populate it; v0.4.1 accepts their native provenance instead. Existing complete retained records need no metadata rewrite or new backup. Incomplete records still fail closed; a display location cannot replace missing native provenance or sealed recovery credentials.
This admission fix requires only a control-plane update. The **new target** must still use the fixed member agents described above; the control-plane fix does not make old member executables safe for Citus credential recovery.
Submit the new target once:
```sh
karactl -n NS clusters create -f citus-recovery.json --wait --timeout 30m
karactl -n NS clusters status NEW_TARGET -o json
karactl -n NS clusters members NEW_TARGET -o json
```
Verify every member’s fixed build, successful recovery, current Ready status, generation convergence, Citus metadata and archive health. Before reopening writes, verify recovered application data and authentication through the coordinator endpoint. This is **new-target recovery**, not success of the original failed restore. See [coordinated Citus recovery](/docs/postgres/distributed/#coordinated-backup-and-pitr) and [failed restore and client holds](/docs/postgres/backups/#failed-restore-and-client-holds).
### Known issues in v0.4.1
[Section titled “Known issues in v0.4.1”](#known-issues-in-v041)
The following issues published for v0.4.1 are **fixed in v0.4.2**: MySQL Group Replication formation on every provider; PXC formation on Zerops; MySQL/PXC full-backup failures; OpenBao briefly reporting Ready during rolling updates; OpenBao/MySQL/PXC/Neo4j/Weaviate clones never reporting Ready; and delayed resume/public-exposure recovery under a closed maintenance window. See the [v0.4.2 upgrade guidance](#upgrading-to-v042) for affected versions and required re-creation of existing MySQL/PXC clusters for agent-side fixes.
The PostgreSQL `postgres` administrator password also cannot be rotated yet; see [its containment procedure](#postgresql-administrator-password-cannot-be-rotated-yet). Temporal remains unsupported on Zerops.
## Upgrading from v0.3.0 to v0.4.0
[Section titled “Upgrading from v0.3.0 to v0.4.0”](#upgrading-from-v030-to-v040)
**Back up control-plane state and preserve the original `KARAKORAM_MASTER_KEY` before starting v0.4.0.** Use the [consistent pre-upgrade backup](#consistent-pre-upgrade-state-backup); retain the old binary and state together for recovery. Existing namespaces, clusters, credentials and policies are retained; upgrading does not migrate their provider or opt them into new engine topologies.
### Automatic state changes
[Section titled “Automatic state changes”](#automatic-state-changes)
Opening the store at startup applies these migrations in filename order, skipping recorded migrations. Each SQL migration and its completion record commit in one transaction; no separate migration command is needed.
| Migration | Purpose |
| ------------------------------- | ------------------------------------------------------------------------- |
| `013_postgres_bindings.sql` | PostgreSQL consumer bindings, source identity and dependency metadata |
| `014_ops_cluster_fields.sql` | Adds managed-cluster ownership and maintenance-override generation fields |
| `015_notifications.sql` | Durable notification delivery outbox and retry indexes |
| `016_control_plane_backups.sql` | Control-plane backup run records and single-active-run index |
| `017_backup_drills.sql` | Backup drill reports and cluster/active-run indexes |
Migrations are **forward-only**. Downgrading an upgraded store to v0.3.0 is not supported; an older executable is not a schema rollback. To roll back, stop the new control plane and restore the consistent pre-upgrade state with its original master key and matching old binary. This does not roll back engine data or infrastructure changes made after the snapshot.
### PostgreSQL binding database access
[Section titled “PostgreSQL binding database access”](#postgresql-binding-database-access)
[PostgreSQL](https://www.postgresql.org/) bindings are new in v0.4.0. Databases Karakoram creates for [PostgreSQL bindings](/docs/postgres/bindings/) do not keep PostgreSQL’s default PUBLIC `CONNECT` and `TEMPORARY` privileges: Karakoram revokes both from PUBLIC and grants them only to the binding’s owning role; explicit grants Karakoram manages, such as Data [MCP](https://modelcontextprotocol.io/) readers scoped to that database, are kept. Another binding’s role cannot connect to a consumer’s database.
Karakoram re-checks these privileges on reconcile, not through the [SQLite](https://sqlite.org/) migration: if PUBLIC access reappears (for example, granted by hand or restored from older data), it is revoked again and sessions of roles without `CONNECT` are disconnected. Applications must use the owning binding identity or deliberately managed grants. This concerns binding databases only, not a blanket revocation on unrelated PostgreSQL databases.
### Source-build prerequisites
[Section titled “Source-build prerequisites”](#source-build-prerequisites)
[`go.mod`](https://github.com/KarakoramDev/Karakoram/blob/main/go.mod) now declares **`go 1.26.0`**, with no separate `toolchain` directive. Source builders need [Go](https://go.dev/) 1.26 or newer; the [README build prerequisites](/docs/#build-from-source) also require [Node](https://nodejs.org/) 22.6+, [npm](https://docs.npmjs.com/) and [Make](https://cgit.git.savannah.gnu.org/cgit/make.git/). `make build-all` builds the UI before both executables; `make build` alone does not build UI assets. Prebuilt installations need no Go or Node toolchain.
### New runtime environment variables
[Section titled “New runtime environment variables”](#new-runtime-environment-variables)
`internal/server/config.go` adds no settings in this release. These new variables are read by the operational controllers/transports or offline recovery command:
| Variable | Default | Effect |
| ---------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `KARAKORAM_BACKUP_DRILL_CONCURRENCY` | `1` | Installation-wide concurrent drill limit, integer 1–4; invalid values fail startup |
| `KARAKORAM_NOTIFY_ENDPOINT_ALLOWLIST` | unset | Comma-separated exact `host` or `host:port` exceptions for notification destinations; see [outbound safety](/docs/notifications/#delivery-and-outbound-safety) |
| `KARAKORAM_RESTORE_S3_ACCESS_KEY_ID` | unset | S3 access key for offline `karakoram admin restore-state` |
| `KARAKORAM_RESTORE_S3_SECRET_ACCESS_KEY` | unset | S3 secret key for offline `karakoram admin restore-state` |
Restore credentials are needed only for [S3](https://aws.amazon.com/s3/) recovery, not ordinary startup. `KARAKORAM_BACKUP_ENDPOINT_ALLOWLIST` and `KARAKORAM_ARTIFACT_MIRROR` already existed; they are not new settings. Native-test binary paths and development hooks are not installation requirements. See [notifications](/docs/notifications/), [backup drills](/docs/backup-drills/) and [offline state restore](/docs/control-plane-backup/#local-export-and-offline-restore).
### New native artifacts
[Section titled “New native artifacts”](#new-native-artifacts)
Installations mirroring native artifacts must add the following exact filenames from [`lock.json`](https://github.com/KarakoramDev/Karakoram/blob/main/internal/engine/nativeartifacts/lock.json), preserving their locked SHA-256 and size. Artifacts are selected by engine/version/profile, not all downloaded at control-plane startup.
| Feature | New files |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| OpenBao | `openbao_2.7.1_linux_amd64.tar.gz`, `openbao-plugin-database-mongodb_linux_amd64_v1.tar.gz`, `openbao-plugin-kms-aws_linux_amd64_v1.tar.gz` |
| Keycloak | `keycloak-26.8.1.tar.gz` |
| TimescaleDB | `timescaledb-pg16-2.31.0-ubuntu24.04-amd64.tar.gz`, `timescaledb-pg17-2.31.0-ubuntu24.04-amd64.tar.gz`, `timescaledb-pg18-2.31.0-ubuntu24.04-amd64.tar.gz` |
| Citus | `citus-pg16-14.2.0-ubuntu24.04-amd64.tar.gz`, `citus-pg17-14.2.0-ubuntu24.04-amd64.tar.gz`, `citus-pg18-14.2.0-ubuntu24.04-amd64.tar.gz` |
| Temporal | `temporal_1.31.4_linux_amd64.tar.gz`, `temporal_1.32.1_linux_amd64.tar.gz`, `ui-server_2.56.0_linux_amd64.tar.gz` |
| Percona Server | `Percona-Server-8.4.11-11-Linux.x86_64.glibc2.35-minimal.tar.gz`, `Percona-Server-8.4.10-10-Linux.x86_64.glibc2.35-minimal.tar.gz`, `Percona-Server-8.0.46-37-Linux.x86_64.glibc2.35-minimal.tar.gz`, `Percona-Server-8.0.45-36-Linux.x86_64.glibc2.35-minimal.tar.gz` |
| PXC | `Percona-XtraDB-Cluster_8.4.11-11.1_Linux.x86_64.glibc2.35-minimal.tar.gz`, `Percona-XtraDB-Cluster_8.0.46-38.1_Linux.x86_64.glibc2.35-minimal.tar.gz` |
| XtraBackup | `percona-xtrabackup-8.4.0-7-Linux-x86_64.glibc2.35-minimal.tar.gz`, `percona-xtrabackup-8.0.35-36-Linux-x86_64.glibc2.35-minimal.tar.gz` |
| Neo4j / Java | `neo4j-community-2026.09.0-unix.tar.gz`, `neo4j-community-2026.10.0-unix.tar.gz`, `OpenJDK21U-jre_x64_linux_hotspot_21.0.12.1_1.tar.gz` |
| Weaviate | `weaviate-v1.40.1-linux-amd64.tar.gz`, `weaviate-v1.39.12-linux-amd64.tar.gz` |
[TimescaleDB](https://www.timescale.com/) and [Citus](https://www.citusdata.com/) bundles are Karakoram-built; development builds need a mirror or explicit `KARAKORAM_ARTIFACT_RELEASE` containing their exact locked bytes. Other additions have official upstream URLs. [Weaviate](https://weaviate.io/) is upstream-only, mixed-license and **not redistributed as a Karakoram release payload**. A private mirror is not permission to redistribute upstream software. See [native artifacts](/docs/native-artifacts/) for resolution order, provenance and licensing.
### Deploy on Zerops
[Section titled “Deploy on Zerops”](#deploy-on-zerops)
After the state/key backup, deploy the published tag from the checkout root:
```sh
git fetch origin --tags
git checkout v0.4.0
zcli service push karakoram -P "$PROJECT_ID" --setup karakoram \
--zerops-yaml-path zerops.yaml --workspace-state clean --disable-logs
```
Set `PROJECT_ID` as below, then follow [deployment and readiness](#deployment-and-readiness) and [member-agent rollout](#rolling-out-member-agents). Review [engine/provider status](/docs/providers/#engine-availability) and the [verified-offline Ops features](/docs/operations/) before enabling new functionality.
[Temporal](https://temporal.io/) is available on [Dokploy](https://dokploy.com/) and [Nomad](https://developer.hashicorp.com/nomad) only in v0.4.0. [Zerops](https://zerops.io/) support is pending IPv6 private networking: the current Temporal membership runtime requires a private IPv4 route to the bound PostgreSQL writer.
## Upgrading from v0.2.0 to v0.3.0
[Section titled “Upgrading from v0.2.0 to v0.3.0”](#upgrading-from-v020-to-v030)
**Back up control-plane state and preserve the original `KARAKORAM_MASTER_KEY` before starting v0.3.0**, as for every upgrade ([consistent pre-upgrade backup](#consistent-pre-upgrade-state-backup)). Namespaces, clusters, secrets, policies and API tokens are retained; no cluster recreation is required.
### Automatic state changes
[Section titled “Automatic state changes”](#automatic-state-changes-1)
No [SQLite](https://sqlite.org/) migrations are added after `012_sanitize_diagnostics.sql`. [OAuth](https://oauth.net/2/) grants and API-token restrictions use existing tables, and dry-run previews run in read-only transactions. Migrations remain forward-only; keep the pre-upgrade backup for rollback.
### Agent access is on, read-only
[Section titled “Agent access is on, read-only”](#agent-access-is-on-read-only)
After the upgrade the control plane serves the [MCP server](/docs/mcp/) at `/mcp` because `KARAKORAM_MCP` defaults to `readonly`: holders of valid API tokens, including existing unrestricted ones, can call read-only tools through the same RBAC and audit path as the REST API. Nothing becomes readable that the token could not already read. To disable the endpoint, set `KARAKORAM_MCP=off` in the control plane’s service environment ([Zerops](https://zerops.io/): the service’s environment variables, or `envVariables` in the import YAML) and restart; set `readwrite` only deliberately. New variables, all documented in [Install](/docs/getting-started/install/#required-secrets-and-generated-settings):
| Variable | Default | Effect |
| ------------------------------ | ---------- | ------------------------------------------------------------- |
| `KARAKORAM_MCP` | `readonly` | `off`, `readonly` or `readwrite`; invalid values fail startup |
| `KARAKORAM_OAUTH` | `off` | Embedded OAuth authorization server for MCP connectors |
| `KARAKORAM_OAUTH_DCR` | `off` | Dynamic client registration, only with OAuth on |
| `KARAKORAM_OAUTH_CLIENT_HOSTS` | unset | Optional allowlist of client-metadata hosts |
Existing API tokens stay unrestricted; mint [restricted tokens](/docs/access-control/#restricted-api-tokens) for agents with `karactl mcp config`. Stored RBAC policies are kept. The new `cluster-data` resource is covered by the administrator wildcard but not by the built-in viewer or operator roles; grant it explicitly per `/`. [Data MCP](/docs/data-mcp/) stays off for every cluster until `spec.dataMcp.enabled: true` is set on that cluster.
### New native artifacts
[Section titled “New native artifacts”](#new-native-artifacts-1)
Data MCP companions for [MongoDB](https://www.mongodb.com/), [PostgreSQL](https://www.postgresql.org/) and [Supabase](https://supabase.com/) use two new entries in `lock.json`, downloaded only when a cluster enables data MCP: `mongodb-mcp-server-3.1.0-ubuntu24.04-amd64.tar.gz` is a Karakoram-built release asset with no upstream URL, and `pgedge-postgres-mcp-server_1.1.0_linux_x86_64.tar.gz` comes from the official [pgEdge](https://www.pgedge.com/) release. Operators running a `KARAKORAM_ARTIFACT_MIRROR` must add both files with their exact locked bytes; development builds need a mirror or explicit `KARAKORAM_ARTIFACT_RELEASE` containing the locked MongoDB bundle. See [data MCP companion artifacts](/docs/native-artifacts/#data-mcp-companion-artifacts).
### Deploy on Zerops
[Section titled “Deploy on Zerops”](#deploy-on-zerops-1)
After the state/key backup, deploy the published tag from the checkout root:
```sh
git fetch origin --tags
git checkout v0.3.0
zcli service push karakoram -P "$PROJECT_ID" --setup karakoram \
--zerops-yaml-path zerops.yaml --workspace-state clean --disable-logs
```
Set `PROJECT_ID` as below, then follow [deployment and readiness](#deployment-and-readiness) and [member-agent updates](#rolling-out-member-agents). Data MCP companions are new private services created only when a cluster opts in.
## Upgrading from v0.1.x to v0.2.0
[Section titled “Upgrading from v0.1.x to v0.2.0”](#upgrading-from-v01x-to-v020)
**Back up control-plane state and preserve the original `KARAKORAM_MASTER_KEY` before starting v0.2.0.** Use the [consistent pre-upgrade backup](#consistent-pre-upgrade-state-backup) below, not a live copy of the [SQLite](https://sqlite.org/) main file. Let engine operations settle and retain the old revision and backup for recovery.
Existing [Zerops](https://zerops.io/) namespaces and clusters retain their provider, resource identities, secrets and desired specifications; no cluster recreation or provider migration is required. Stored namespaces without a provider kind normalize to `zerops`, and omitted namespace-create kinds still default to Zerops. [PostgreSQL](https://www.postgresql.org/) and [Supabase](https://supabase.com/) are new engines; upgrading the control plane does not change an existing cluster’s engine driver or engine version.
### Automatic state changes
[Section titled “Automatic state changes”](#automatic-state-changes-2)
Startup applies the following migrations added after v0.1.2, in order, without a separate migration command:
| Migration | Purpose |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `010_lifecycle_recovery.sql` | Marks failed delete tracking successful when the matching cluster incarnation is already absent; releases backup inventory stuck in `Deleting` after terminal failed/cancelled deletion so deletion can be retried. |
| `011_metrics_retention.sql` | Adds the `metrics(timestamp,id)` index for bounded retention scans. |
| `012_sanitize_diagnostics.sql` | Rewrites legacy diagnostic JSON and event details using the current sanitizer in bounded, resumable batches; leaves secret values and row metadata intact. |
Migration 011 builds an index over existing metrics; migration 012 rewrites legacy diagnostics. These can extend first startup on large stores. Measure on an isolated copy and allow enough readiness time; see [Monitoring](/docs/monitoring/#upgrading-an-existing-metrics-database). Migrations are forward-only: an older binary is not a schema rollback.
### API and specification compatibility
[Section titled “API and specification compatibility”](#api-and-specification-compatibility)
The `/api/v1` API and existing Zerops cluster spec shapes remain. The [OpenAPI](https://www.openapis.org/) changes are principally additive, but clients must account for these response and admission changes:
* Namespace responses now require `backend` and `capabilities` and may include non-secret `config`; namespace creation accepts optional `backend` and `config`. `projectId` remains the field name for the provider target ID. Strict clients that reject new response properties need updated models.
* `GET /backends` lists compiled-in provider metadata and requires `namespaces/read` on global `*`.
* Cluster create/PUT/PATCH reject missing provider capabilities with `400 invalid_request`. A newly selected public IP family that fresh provider information proves unavailable is rejected; unchanged legacy exposure is preserved and unknown availability does not reject admission.
* `POST /clusters/{cluster}/retry-deployments` explicitly re-arms exhausted deployment budgets. `spec.retryEpoch` is server-managed/read-only, not an ordinary spec-edit control.
* Backup or source-cluster deletion can return `409 invalid_state` while an in-flight restore/create depends on it; PITR dependencies also protect the source/history. Wait for completion or safe cancellation rather than bypassing the guard.
* PostgreSQL role/grant and Supabase credentials, SMTP, functions and JWT-rotation endpoints are added. Topology, backup-type, member-role and credential enums gain engine-specific values; member responses may include `componentVersion`, and agent status may include `message`. Some engine-administration reads now document `503` for unavailable dependencies.
* Restore readiness is explicitly false during recovery and remains false with `RestoreFinished` until ordinary reconciliation observes database health. Do not infer database readiness from operation completion alone.
Use the matching v0.2.0 CLI/SDK models; see the [API reference](/docs/api/) and [OpenAPI contract](https://github.com/KarakoramDev/Karakoram/blob/main/api/openapi.yaml). No existing path or request property was removed in the v0.1.2-to-v0.2.0 contract diff; expanded enums, stricter admission and deletion guards are notable behavior changes, not a blanket promise that every old client handles new responses.
### Deploy on Zerops
[Section titled “Deploy on Zerops”](#deploy-on-zerops-2)
After the state/key backup, deploy the published tag from the checkout root:
```sh
git fetch origin --tags
git checkout v0.2.0
zcli service push karakoram -P "$PROJECT_ID" --setup karakoram \
--zerops-yaml-path zerops.yaml --workspace-state clean --disable-logs
```
Set `PROJECT_ID` as below. Preserve the existing volume, master key, secrets, public origin and single-writer deployment settings. Then follow [deployment and readiness](#deployment-and-readiness) and [member-agent updates](#rolling-out-member-agents); upgrading the control-plane binary alone does not replace running member binaries.
### New optional providers
[Section titled “New optional providers”](#new-optional-providers)
Zerops remains the reference provider. [Dokploy](https://dokploy.com/) and [Nomad](https://developer.hashicorp.com/nomad) are additional options, not an automatic migration for existing installations. Review [Providers](/docs/providers/), then the [Dokploy installation](/docs/dokploy/#install-the-control-plane-on-dokploy) or [Nomad installation](/docs/nomad/#install-the-control-plane-on-nomad). Keep the existing default namespace on its original target; additional targets use explicit provider selection and credentials through [Namespaces](/docs/namespaces/).
## Prerequisites
[Section titled “Prerequisites”](#prerequisites)
For new installations use [Install](/docs/getting-started/install/). The concrete deployment, maintenance-container and ownership commands below are for the **[Zerops](https://zerops.io/) provider**. [Dokploy](/docs/dokploy/#install-the-control-plane-on-dokploy) and [Nomad](/docs/nomad/#install-the-control-plane-on-nomad) use their guides’ singleton/stop-first deployment paths with the same consistent-state and master-key backup requirements. Zerops upgrades need the target checkout, authenticated [zcli](https://docs.zerops.io/references/cli) with deployment access, runtime secrets and Local Storage maintenance-container access. Use `zcli login` with a Zerops token; avoid shared terminals/shell history.
From the checkout root, set:
```sh
PROJECT_ID=''
```
Commands assume default service names; substitute customized names consistently.
## Before deploying
[Section titled “Before deploying”](#before-deploying)
1. Read target-release compatibility notes. Retain the previous source revision/application version.
2. Let provisioning, backup, restore, scaling and upgrade operations settle. Avoid desired-state changes during maintenance. Engine processes continue independently, but reconciliation, new operations, schedules and monitoring pause while the control plane is down; choose a window that will not miss required backups.
3. Take the consistent state backup below; preserve `KARAKORAM_MASTER_KEY` separately. State without its original key cannot decrypt credentials.
4. Keep the same volume, secrets, project ID and public origin. Never rerun the installer or regenerate the master key.
## Consistent pre-upgrade state backup
[Section titled “Consistent pre-upgrade state backup”](#consistent-pre-upgrade-state-backup)
**Warning:** A live copy of [SQLite](https://sqlite.org/)’s main file or a live volume tarball can omit committed WAL changes. There is no `karakoram admin backup-db` command.
1. Stop the runtime:
```sh
zcli service stop karakoram -P "$PROJECT_ID"
```
Wait until the stop finishes and the runtime is `STOPPED`; no control-plane writer may remain.
2. Open **karakoramdata → Web shell**. The maintenance container mounts state at `/data` (runtime: `/mnt/karakoramdata`). Archive outside it:
```sh
umask 077
tar -czf /tmp/karakoram-state-before-upgrade.tar.gz -C /data .
```
Include the whole directory: database, remaining WAL/SHM sidecars and persisted identities. Never attach another running writer.
3. Download through the maintenance container’s **File browser**. Store encrypted off-platform, record version/time and restrict access: the archive contains authentication state and encrypted infrastructure secrets. `/tmp` and same-volume copies are not durable protection against host/volume loss.
4. Leave stopped for immediate deployment; otherwise restart the old version:
```sh
zcli service start karakoram -P "$PROJECT_ID"
```
See [Zerops Local Storage maintenance](https://docs.zerops.io/local-storage/how-to/manage). Automated consistent backups can use SQLite’s native backup API/tooling; do not assume the runtime includes `sqlite3`.
## Deploy
[Section titled “Deploy”](#deploy)
### Prebuilt release (default)
[Section titled “Prebuilt release (default)”](#prebuilt-release-default)
After state/key backup, select the published release:
```sh
git fetch origin --tags
git checkout v0.4.1
zcli service push karakoram -P "$PROJECT_ID" --setup karakoram \
--zerops-yaml-path zerops.yaml --workspace-state clean --disable-logs
```
Choose your target tag, or update `main`: [`deploy/zerops/RELEASE`](https://github.com/KarakoramDev/Karakoram/blob/main/deploy/zerops/RELEASE) pins the latest published release, not newer source. Tags pin themselves. **Never edit the pin by hand**; only the [release tool](/docs/contributing/#releasing) changes it.
The default setup verifies the manifest’s SSH signature, archive SHA-256 and binary version; allow seconds, with no [Go](https://go.dev/)/[Node](https://nodejs.org/) toolchain. `--workspace-state clean` uploads only committed `HEAD`; do not add `--no-git`, which uploads the whole checkout.
For [Git](https://git-scm.com/)-connected services, redeploy the chosen tag or `main` through [Zerops](https://zerops.io/) controls; its ref determines the pin. Installer `buildFromGit` is a one-time import, not webhook configuration. Never rerun the helper or reimport to upgrade.
Download, signature/checksum or build failure never reaches deployment and leaves the running version untouched. A service left stopped after backup remains stopped until started or successfully deployed.
### Source builds and provenance
[Section titled “Source builds and provenance”](#source-builds-and-provenance)
Use `--setup karakoram-source` for the exact checked-out commit, including UI and static Go binary (minutes): forks, development or unreleased fixes. Git imports select `zeropsSetup: karakoram-source`.
Before source pushes without Git metadata, merge literal `git describe --tags --always` / `git rev-parse HEAD` output into the source setup’s `build.envVariables`; see [source provenance](/docs/getting-started/install/#build-provenance-for-local-pushes). Explicit commit overrides discovery; the pipeline records UTC build time. These are build inputs, not runtime settings. Prebuilt binaries already carry provenance; missing source metadata appears as `unknown (built without git metadata)` in the API/UI.
Prefer `--workspace-state clean` and commit intended source/pipeline changes before pushing. `--no-git` uploads the whole checkout: keep private imports outside it and exclude other sensitive files first. The [source helper](/docs/getting-started/install/#one-command-helper) uses this no-Git path.
### Deployment and readiness
[Section titled “Deployment and readiness”](#deployment-and-readiness)
Both setups replace the executable in the existing runtime. Keep `deploy.temporaryShutdown: true` and one-container maximum: stop the old writer before starting the new one. Brief UI/API downtime is intentional; do not use overlapping deployments. Automatic state migrations need no separate command. Deployment probes `/healthz` on port 8080 (120-second failure timeout, 10-second retries). Wait for deployment completion and `ACTIVE`, then check:
```sh
curl --fail --show-error https://example.com/healthz
karactl --context production health
karactl --context production version
```
Check clusters, operations, backup inventory, member agents and logs without exposing secrets. Healthy control-plane readiness does not prove engine recovery works.
## Existing single-project installations
[Section titled “Existing single-project installations”](#existing-single-project-installations)
Existing clusters, events and metrics enter namespace `default`; existing secrets and master key remain valid. Stored policy is normalized: `*` becomes `**`, while bare cluster patterns such as `orders*` become `default/orders*`, retaining their old project scope. Namespace rules keep bare names. Review [access control](/docs/access-control/).
Upgrades retain stored policy rather than adding fresh-install namespace-read defaults. If operators/viewers should enumerate namespaces, merge these rules into the **complete existing policy**, preserving administrator access; use narrower grants where appropriate:
```csv
p, role:operator, namespaces, read, **
p, role:viewer, namespaces, read, **
```
Without namespace-read grants the UI shows only `default`, even with remote-cluster access. Remote relays update automatically; expect brief management interruptions and check desired/reported versions and connected state in [Namespaces](/docs/namespaces/#relay-health).
## Rolling out member agents
[Section titled “Rolling out member agents”](#rolling-out-member-agents)
Upgrading or restarting the control plane does **not** replace existing member agents. A member receives the current executable on its next control-plane-managed deployment. Do not copy binaries into running containers or add another supervisor.
Inspect `karactl clusters get NAME` or `clusters status NAME`: `status.agents` reports `targetBuild`, `targetVersion`, and the `outdated` and `unknown` member names. Member output includes `agent.version`, `agent.commit` and `agent.build`. The `AgentsCurrent` condition is informational: `True/Current`, `False/Outdated`, or `Unknown/Unreported`; it never makes a cluster unready or Degraded. An `AgentsOutdated` event signals outdated member agents.
Build inventory is independent of support for `upgrade-agents`: all registered providers publish identities returned by their existing generic agent status probes, including MySQL/PXC, Keycloak, OpenBao, Temporal, Neo4j and Weaviate whose agent-only rollout is unsupported. A healthy native engine is not evidence of a current agent executable. Legacy agents without build metadata remain unknown. PostgreSQL proxy, Supabase, Temporal and Keycloak instance observations conservatively report a non-current or unknown identity when replicas disagree, rather than letting a current replica hide an incomplete rollout.
PostgreSQL proxy members report the identities observed on their live instances, independently of native proxy health. If instances disagree, member inventory selects a non-current build (or an unknown identity for a legacy instance), so one current replica cannot hide an incomplete rollout. Older control planes can omit proxy build metadata even when the agent reports it; upgrading the control plane corrects this inventory without redeploying the proxy. Missing inventory metadata alone does not explain a Running rollout: completion reads the agent’s reported executable build directly, after platform deployment and health gates.
```sh
karactl clusters upgrade-agents NAME --wait --timeout 30m
# Include agents already reporting the current build:
karactl clusters upgrade-agents NAME --include-current --wait
```
The durable `upgrade-agents` operation redeploys the current executable at the **same engine version**, using each engine’s safe, one-member-at-a-time rolling sequence and health/quorum gates. Unknown/legacy agents are eligible for a manual run. If every agent is current, the API returns HTTP 409 with code `AgentsCurrent` unless `--include-current` is set. Cancel with `karactl operations cancel ID`; cancellation is cooperative between members, not a rollback of completed deployments.
Rollouts respect [maintenance windows](/docs/operations/#maintenance-windows). Use `--ignore-maintenance-window` for an explicit override; this does not bypass engine safety gates. Timing out or interrupting `--wait` does not cancel the rollout; retain the operation ID and use `operations wait ID`.
An agent rollout can be claimed even if ordinary desired-state synchronization fails (for example, because the old agent contains the bug the rollout repairs). This exception applies only to `upgrade-agents`: it does not start unrelated queued actions or displace an already Running operation. Provider claim policy, maintenance windows, native health checks and bounded operation retries still apply. Delete and keep-volumes suspend already use a separate cleanup path and do not require successful ordinary synchronization.
The enqueue-frozen rollout target is not silently retargeted after another control-plane upgrade. A Pending PostgreSQL rollout targeting a previous control-plane build is claimed and fails explicitly before deployment, with a message to re-run `upgrade-agents`. Keep that receipt for evidence and submit a new rollout targeting the current control plane. An already Running rollout retains its provider-specific recovery/checkpoint safeguards.
PostgreSQL resumes an in-flight member after a control-plane restart. If a failover changed the WAL timeline while a replica was being redeployed, it persists a fresh barrier from the current primary of the same replication group and system identity. Completion requires the replica to report streaming, the current leader and timeline, no lag, and replay at or beyond that barrier; WAL positions from different timelines are never compared. A different system identity fails explicitly rather than treating a foreign replica as caught up. Barrier refresh requires successful observations of every group member and a unique self-reported primary with healthy DCS. Local replica status alone is not proof of its WAL source: the control plane also reads the selected primary’s replication view to confirm exactly one streaming sender for that member and acknowledged replay past the barrier, on the expected primary system/timeline. This bounded read-only query uses the existing `kk_monitor` identity, namespace-aware native connection and cluster-CA hostname verification. It requires no member-agent upgrade. Missing SQL access or replication proof never acknowledges the member; do not weaken TLS or grant arbitrary SQL access to bypass a failed proof. Unproven catch-up remains subject to the existing retry/deadline budgets; restore replication and follow the failure’s `re-run upgrade-agents` guidance.
A changed control-plane build can finish acknowledging the final already deployed member against the original frozen target and retire that operation. It cannot deploy remaining members for the old target: that requires a new rollout. Success of the old operation does not imply agents match the new control plane; inspect inventory before requesting the new rollout.
Agent rollout requires an advertised engine driver operation backed by a one-member disruption journal. [Keycloak](https://www.keycloak.org/), [OpenBao](https://openbao.org/), [Temporal](https://temporal.io/), [MySQL](https://www.oracle.com/mysql/), [PXC](https://www.percona.com/mysql/software/percona-xtradb-cluster), [Neo4j](https://neo4j.com/) and [Weaviate](https://weaviate.io/) do not currently implement that contract: automatic rollout reports unsupported agents without scheduling any disruption, and manual rollout is rejected. Enabling automatic rollout never opts these engines into a generic restart or redeployment fallback.
Automatic rollout is **off by default**:
```sh
karactl settings agent-rollout get
karactl settings agent-rollout set --automatic=true --max-concurrent-clusters 2
karactl settings agent-rollout set --automatic=false --max-concurrent-clusters 1
```
These installation-wide settings include a read-only `currentBuild` (`version`, `commit`, `build`). `set` replaces the settings; omitted flags default to `automatic=false` and `maxConcurrentClusters=1` (allowed range **1–5**).
When enabled, the controller checks every **60 seconds**, enqueuing rollouts only for clusters with **outdated** agents, bounded by `maxConcurrentClusters` across the installation and respecting maintenance windows. Unknown/legacy agents alone do not trigger automatic rollout: run manually. A failed rollout for a target build is **not retried automatically** for that build; fix the cause and re-run manually.
## Rollback
[Section titled “Rollback”](#rollback)
* A pre-deployment build failure leaves the current application unchanged. After temporary shutdown, inspect service/process state; start/redeploy the previous version if necessary.
* Redeploy an old binary only if it supports the current schema. Migrations are forward-only; a pre-namespace binary requires pre-upgrade state.
* Otherwise stop **every writer**, preserve failed-version state, and restore the complete pre-upgrade archive into an **emptied** state directory through the maintenance container. Never merge stale WAL/SHM or identity files. Restore `zerops` ownership, restrictive permissions and the matching master key; deploy the old binary before starting it.
* State rollback rewinds specifications, operations, tokens, sessions and inventory—not external services or engine data. Review changes and in-flight operations before resuming reconciliation. It does not undo engine-level upgrades or compatibility changes; see [MongoDB](/docs/mongodb/operations/), [RabbitMQ](/docs/rabbitmq/operations/) and [Elasticsearch operations](/docs/elasticsearch/operations/), and [Backups and storage](/docs/backups/) for data recovery.
For [Weaviate](https://weaviate.io/), follow [native upgrade operations](/docs/weaviate/operations/#restart-and-upgrade): only cataloged patch-forward or one-minor-forward changes, never downgrades or skipped minors. Members roll with quorum/health gates; restoring the control-plane archive does not reverse native data changes.
Retain the archive, master key and previous revision until the upgrade and agent transition are accepted.
## Version notes
[Section titled “Version notes”](#version-notes)
### Unreleased: fleet operations and recovery
[Section titled “Unreleased: fleet operations and recovery”](#unreleased-fleet-operations-and-recovery)
* Member build identity and explicit rolling agent upgrades keep executable rollout separate from database-version upgrades. Automatic rollout is opt-in; existing and legacy agents are not redeployed merely because the control plane restarts.
* Optional cluster maintenance windows defer new disruptive work. Emergency healing and near-expiry certificate recovery remain exempt; explicit overrides never bypass engine safety gates.
* Placement reports distinguish authoritative anti-affinity from best-effort evidence. Member relocation uses native replacement and synchronization, and cannot promise a new host on providers without placement support.
* Control-plane backup policy, durable notification deliveries and backup-drill reports add forward-only [SQLite](https://sqlite.org/) migrations. Preserve a consistent pre-upgrade snapshot and the original master key; an older binary must not open newer state.
* State snapshots do not contain the operator master key or external provider credentials. Rebuild with the original key, endpoint identity and valid credentials; review external changes since the snapshot before resuming reconciliation.
* Import the [Grafana dashboards and Prometheus rules](/docs/monitoring/) and configure [notification channels](/docs/notifications/) deliberately. Existing installations retain their RBAC policy; installation-wide settings still require explicit settings access.
### v0.1.2
[Section titled “v0.1.2”](#v012)
* Default [Zerops](https://zerops.io/) deployments use signed prebuilt releases pinned by the checkout, not unreleased source; forks/unreleased fixes require `karakoram-source`. Preserve the state volume, master key and single-writer deployment settings.
* Signed CLI checksum manifests start with v0.1.2; [verify the signature and checksum](/docs/getting-started/install/#install-karactl) and match the deployed server version.
* Cluster specification changes record `ClusterSpecChangeRequested` audit events with field-level details for `audit:read`; see [Reviewing audit activity](/docs/operations/#reviewing-audit-activity).
* Platform resource-update rejections expose the hostname, error code and attempted bounds; see [Resources and autoscaling](/docs/clusters/#resources-and-autoscaling).
### v0.1.1
[Section titled “v0.1.1”](#v011)
Healthy clusters need no action. Project-scoped Full access tokens work without sudo mode; existing resources gain current ownership evidence during reconciliation.
### Legacy installations (before v0.1.1)
[Section titled “Legacy installations (before v0.1.1)”](#legacy-installations-before-v011)
For unverified ownership or repeated `zerops HTTP 400 (processNotFound)`, upgrade to v0.1.1 or newer first. Expired imports have unknown outcomes: reconciliation observes existing services and imports missing ones; other expired actions use bounded retries.
If initial provisioning still reports unverified ownership, choose one recovery path:
* Delete only never-deployed, data-free runtime/volume services from the failed import so reconciliation recreates them. A member volume appends `v` to its hostname (e.g. `m0v`). **Never delete healthy members or retained data.**
* Supply a **temporary**, project-scoped Full access sudo-mode token for **one ownership-verification reconcile only**. For `default`, change `KARAKORAM_ZEROPS_TOKEN` and restart. Then restore the non-sudo token, restart again and **revoke the temporary sudo token**. Other namespaces use [token rotation](/docs/namespaces/#add-validate-and-rotate).
Hostnames alone never prove [service ownership](/docs/security/#service-ownership).
# Keycloak
> Karakoram runs the official Keycloak 26.8.1 native distribution on Ubuntu 24.04 with openjdk-21-jre-headless. The archive URL, SHA-256, byte size,…
Karakoram runs the official **[Keycloak](https://www.keycloak.org/) 26.8.1** native distribution on [Ubuntu](https://ubuntu.com/) 24.04 with `openjdk-21-jre-headless`. The archive URL, SHA-256, byte size, Apache-2.0 license and source revision are pinned in the native artifact lock. [Keycloak’s supported configurations](https://www.keycloak.org/server/supported-configurations) include [OpenJDK](https://openjdk.org/) 21 and [PostgreSQL](https://www.postgresql.org/) 14–18.
Keycloak is an independent engine driver, not a PostgreSQL cluster variant. Create and make ready a `postgres` source first, then select `keycloak`, the `single` topology and one engine replica. The UI’s Basics step accepts the source cluster and namespace. Engine driver-specific options are:
```json
{
"database": {
"cluster": "identitydb",
"namespace": "default",
"name": "keycloak"
},
"hostname": "https://login.example.com",
"publicHttps": true,
"proxyHeaders": "xforwarded"
}
```
`database.cluster` is required; omitted namespace uses the consumer namespace and omitted database name uses `keycloak`. The binding source and database are immutable after creation. Unknown fields, password fields, raw engine configuration, extra components, direct public TCP exposure and consumer backup schedules are rejected. Choose at least one CPU, 1.5 GiB RAM and 1 GiB runtime disk. The runtime’s filesystem is disposable: authoritative realms, identities, sessions and configuration are in the bound PostgreSQL database.
## Database and credential boundary
[Section titled “Database and credential boundary”](#database-and-credential-boundary)
Before deploying any runtime, the control plane ensures a binding with a dedicated non-superuser owning login and database. Only that login’s password, resolved in-process through `pgbinding.Service.Password`, reaches the consumer. Source administrator credentials never reach Keycloak. JDBC uses the binding writer endpoint with `sslmode=verify-full` and an owner-only CA file. A source that cannot supply verify-full trust is rejected.
The managed JDBC URL sets `preferQueryMode=simple`. ProxySQL 3.0.11 rejects named cursor portals and mixed text/binary result-column formats. Keycloak 26.8.1 configures Hibernate’s statement fetch size to 64, which makes pgjdbc use named portals inside transactions even if binary transfer or server-side preparation is disabled. Simple mode avoids both protocol limitations, including explicit JDBC fetch sizes; `defaultRowFetchSize=0` alone cannot override those. pgjdbc still safely encodes JDBC `PreparedStatement` parameters, but does not use server-side prepared statements or incremental cursor fetching. Results are buffered in memory, so large queries can consume more member memory. TLS hostname verification and the binding’s writer endpoint remain unchanged; the same configuration also works with direct singleton sources.
The same datasource supplies Liquibase, Hibernate and the default Infinispan JDBC-ping discovery SQL; simple mode also covers discovery’s prepared statements. The managed scheduler uses Keycloak’s basic timer, not a Quartz JDBC job store. This is not support for custom extensions that require `LISTEN` or other unsupported proxy features. During PostgreSQL primary failover, the writer endpoint remains stable, but in-flight requests can fail until the JDBC pool replaces failed connections. Verify new authentication and persisted realm/user operations after recovery, not merely an already-cached session. See the [binding failover boundary](/docs/postgres/bindings/#protocol-compatibility-and-primary-failover).
The setting is rendered by the member agent, not the control plane. Existing consumers need a newly deployed agent; ordinary restart and `upgrade-agents` are not a supported rollout for this provider. Follow the [v0.4.1 consumer rollout procedure](/docs/getting-started/upgrade/#keycloak-and-temporal-postgresql-proxy-compatibility), including its data-retention warnings.
The member configuration is sealed at rest, including the generated temporary `admin` password under `keycloak/admin`. It is not returned through ordinary cluster reads. Member pulls are restricted to the allocated member identity and use the existing authenticated agent channel. On the runtime, database and bootstrap credentials live only in mode-0600 files under `/karakoram/keycloak/`; they are not placed in process arguments or environment variables. The optimized distribution is built with `kc.sh build --db=postgres --health-enabled=true` before `start --optimized`.
Each reconciliation resolves the binding again. Credential generation, endpoint, CA, hostname or proxy-header changes republish the sealed configuration and trigger an agent reload. The agent starts the configured runtime on first apply and stops then starts it when configuration changes, unless maintenance, an operator stop hold or the desired process list prevents startup. There is only one runtime, so rotation interrupts authentication. Applied configuration versions and management health must both match before the cluster is ready.
The desired generation is acknowledged after the sole live agent accepts its configuration and runtime resources/routing have converged. Periodic health polls continue without keeping that generation pending. Missing agents, stale applied configuration, agent-reported configuration errors and application changes deferred by a maintenance window do not acknowledge it. Credential and transport refreshes remain exempt from that window, but accepting those refreshes does not acknowledge a deferred application change. Native management health independently controls Ready.
## Routing and health
[Section titled “Routing and health”](#routing-and-health)
Application HTTP listens on private port **8080** for a trusted TLS-terminating ingress. Management HTTP on **127.0.0.1:9000** is not exposed by the runtime’s port declarations. Native readiness calls `/health/ready` and requires HTTP 200 with status `UP`. Agent health is independently protected by cluster-scoped TLS on port 9443.
Successful agent observations carry a control-plane sample timestamp. Native-ready results renew the shared availability lease only while that sample is fresh; failed or expired observations withdraw health and connectivity claims. A control-plane upgrade repairs missing observation timestamps without replacing the serving agent or changing its credentials.
`hostname` accepts an explicit HTTP(S) origin and sets Keycloak’s issuer/front-end URL only; it does not create a route for that origin. Omitting it disables strict hostname checks for private development only. `publicHttps` requires an explicit HTTPS origin and `forwarded` or `xforwarded` proxy headers; the infrastructure provider must support public HTTPS subdomain routing.
On Dokploy and Zerops, Karakoram publishes HTTPS only on the platform-generated domain, not the configured `hostname`. To serve the configured origin, provide your own custom-domain route or reverse proxy forwarding that origin to the generated route; DNS alone is not enough. Without that mapping, clients can reach discovery through the generated route while the returned issuer remains the configured `hostname`; this does not establish reachability at that origin. Nomad has no `publicHttps` capability and no generated public HTTPS route; operators must supply their own ingress to the private runtime instead.
**Only trust an ingress that overwrites the selected headers**; otherwise clients can forge their external address/scheme. A private HTTP endpoint is not an Internet-facing production endpoint.
Create, replace and patch admission, including dry runs, reject `publicHttps: true` when the namespace provider does not advertise `publicHttps`. This includes [Nomad](https://developer.hashicorp.com/nomad) and [Dokploy](https://dokploy.com/) namespaces without configured HTTPS routing.
On an existing runtime, changes to `hostname`, `proxyHeaders` and `publicHttps` respect the maintenance window. Public routing can replace the backend service’s task even though application settings are delivered separately in sealed agent configuration. A closed window defers both changes, not just the agent reload. The last admitted HTTPS policy is retained separately from runtime route receipts. Resume and runtime recovery restore that policy even while the window is closed; a deferred `publicHttps` change remains unpublished and its desired generation is not acknowledged until the window opens or the operation override admits it. The admitted `hostname` and `proxyHeaders` settings are also retained independently of disposable member configuration, so resume cannot publish a previously deferred application change when recreating that configuration. Retained legacy member configuration is migrated before suspension deletes it. If both that configuration and the admitted-settings receipt are already absent on a legacy resume, the control plane cannot infer old settings from a newer desired spec: an open window or desired-state maintenance override is required to admit them. A stale configuration hash without its secret is also unknown admission: it does not block finishing interrupted suspension, and the window/override path can admit the desired settings on resume. Actual secret-store failures and malformed saved configuration remain errors rather than admission fallbacks. Older state migrates retained applied routing before cleanup. If no applied policy survives and the desired generation is unacknowledged, recovery defaults to private access rather than publishing an unadmitted route.
Exactly **one replica** is supported. Multi-instance JDBC-ping discovery has not been qualified for this engine driver; advertising three replicas without that proof would imply unverified availability. Runtime maintenance, resource changes and credential rotation cause downtime even when the source PostgreSQL cluster is highly available.
Resource reconciliation compares values, including an explicit `startCpu`, not the identity of the decoded optional value. An unchanged resource specification does not submit another deployment. After upgrading the control plane, a failed redundant resize from an older build is retired only when its target equals the previously accepted resources and the owned runtime still reports those resources. The control plane then resumes agent/configuration and native readiness observation; retiring the receipt alone does not mark the cluster Ready. Real failed resource changes remain failures. This repair requires only a control-plane upgrade, not replacement of an already serving Keycloak agent. Historical failed create operations remain historical failures even if background convergence later recovers.
## Establishing an administrator and recovery
[Section titled “Establishing an administrator and recovery”](#establishing-an-administrator-and-recovery)
There is deliberately no new credential-reveal API: existing reveals are engine driver-specific, and Keycloak does not implement the database-user/role data-plane contract merely to expose a password. Use Keycloak’s supported offline recovery command with the rendered database configuration:
1. Stop the Karakoram member agent/runtime using the infrastructure’s maintenance controls. Prevent automatic restart while recovering. All Keycloak nodes must remain stopped.
2. As the runtime owner, set `JAVA_HOME` to `/usr/lib/jvm/java-21-openjdk-amd64`, then run:
```sh
/opt/karakoram/keycloak-26.8.1/keycloak-26.8.1/bin/kc.sh \
--config-file=/karakoram/keycloak/keycloak.conf \
bootstrap-admin user --optimized
```
Enter a distinct temporary administrator name and password at the interactive prompt; do not put passwords in command arguments, logs or shell history. Keep the existing protected file and CA in place so recovery uses exactly the same verified database connection.
3. Restart the runtime and sign in at the configured origin’s `/admin/`. Create a permanent administrator with strong authentication. Remove both the recovery user and the original temporary `admin` account.
Startup bootstrap settings only create an administrator when the master realm is first initialized; changing them does not reset an existing account. See [upstream administrator recovery](https://www.keycloak.org/server/bootstrap-admin-recovery), including service-account recovery when realm policy prevents password login.
## Backups and deletion
[Section titled “Backups and deletion”](#backups-and-deletion)
Back up, restore and upgrade the **source PostgreSQL cluster**, not the disposable consumer runtime. A source restore affects every consumer sharing it; coordinate an outage and recovery across those consumers. A source cannot be deleted while the binding exists.
Consumer deletion first disables managed public routing and deletes the owned runtime, observing completion before releasing the binding. The binding identity is derived from the consumer and source specification, so deletion also releases a partially provisioned binding when the first binding attempt failed. Normal deletion drops the bound database and role. `keepVolumes` maps to binding `keepData`: preserve the database and disable the login (`NOLOGIN`). It does not preserve a Keycloak runtime volume. Do not treat deletion retention as a backup.
## Native qualification
[Section titled “Native qualification”](#native-qualification)
The opt-in `TestNativeKeycloak` test in `internal/agent/keycloak` starts and stops its own native PostgreSQL and Keycloak processes. Set `KARAKORAM_KEYCLOAK_NATIVE=1`, `KEYCLOAK_HOME`, `JAVA_HOME` and `POSTGRES_HOME` to verified, unpacked distributions. The test uses the production `Render` function without changing the generated settings, verifies management readiness, obtains a bootstrap administrator token without logging it, and checks least-privilege role attributes and TLS sessions. It uses application port 8080 and loopback management port 9000; run it in an isolated environment where those ports are free. Multi-instance discovery is intentionally outside this qualification.
# Limitations
> These cross-engine constraints define the operating model; engine-specific limits are linked below.
These cross-engine constraints define the operating model; engine-specific limits are linked below.
## Zerops failure domains and storage
[Section titled “Zerops failure domains and storage”](#zerops-failure-domains-and-storage)
* **No anti-affinity guarantee on [Zerops](https://zerops.io/).** Separate services/volumes and dedicated CPU do not prove independent hosts, racks or zones. One host failure can remove several voters. Placement groups are best-effort observations: missing/changing signals can produce unknown, false or stale groups; distinct groups do not certify future separation. The [Dokploy](/docs/dokploy/) and [Nomad](/docs/nomad/) providers pin each member’s volume to a node and keep one replica set or cluster on distinct nodes while enough nodes exist; they report that node as authoritative placement but cannot protect against shared racks or physical hosts below the scheduler.
* **Local Storage is single-host, non-replicated storage.** Each runtime is co-located with its volume. Runtime root disks are wiped on deployment; keep engine data only on assigned per-member volumes. Shared filesystems are unsupported. Replication and backups protect different failure modes.
* **Local Storage minimum is 5 GB.** Mounted capacity follows the volume service’s `maxDisk`, not runtime disk or volume `minDisk`. Growth is supported; shrinking is rejected. Resource/autoscaling behavior belongs in [clusters](/docs/clusters/).
* **Live volume archives are not consistent database backups.** Use [engine-native backups](/docs/backups/) and a separately [consistent control-plane snapshot](/docs/operations/#control-plane-state-backup-and-recovery).
## Provider-specific limitations
[Section titled “Provider-specific limitations”](#provider-specific-limitations)
[Zerops](/docs/zerops/) is the reference provider. [Dokploy](/docs/dokploy/) and [Nomad](/docs/nomad/) are additional providers; their status is documented on the [Providers page](/docs/providers/). See the [provider comparison](/docs/providers/#capability-differences) before selecting one.
* **Dokploy:** node-local [Docker](https://www.docker.com/) volumes have no enforced quota or supported growth. The operator owns Swarm, node disks, registry, DNS and firewall. Admin API keys are not target-scoped; use a dedicated installation. Public TCP has no provider CIDR filtering; public HTTPS needs a domain suffix and wildcard DNS.
* **Nomad:** dynamic `mkdir` host volumes have no enforced quota or supported growth. The operator owns dedicated clients, private-copy rootfs, routed networking, ACLs, disk monitoring and load balancing. Trust every submitter who can place jobs on the clients; namespace metadata is not hostile-tenant isolated. Managed public HTTPS is unavailable.
* **Public TCP on both providers:** requires explicit `allowAllAddresses: true` and an allowlist covering the whole address family. Enforce narrower access externally. Nomad static ports need load-balancer forwarding to the correct member node; they are not a routing mesh.
* **[PostgreSQL](https://www.postgresql.org/) and [Supabase](https://supabase.com/):** both use the shared runtime contract on these providers, but their guides do not claim engine acceptance. Locked native artifacts and registered [S3](https://aws.amazon.com/s3/) destinations must be reachable. Supabase public gateway HTTPS is rejected on Nomad.
## Control-plane and platform operations
[Section titled “Control-plane and platform operations”](#control-plane-and-platform-operations)
* **One control-plane replica; no control-plane HA.** [SQLite](https://sqlite.org/) WAL requires a single writer. Keep `maxContainers: 1` and `temporaryShutdown`; upgrades interrupt management. Lost state cannot be reconstructed by automatically adopting existing services. Preserve consistent state and the original master key; see [recovery](/docs/operations/#control-plane-state-backup-and-recovery).
* **REST polling, not platform watches.** Zerops Processes are asynchronous. Request acceptance is not completion or engine health. Desired resource settings are not current allocation; CPU-mode changes can enter `PlatformCooldown`. See [operations and status](/docs/clusters/#operations-and-status).
* **Repeated reconciliation errors are suppressed.** Consecutive identical `ReconcileError` records are suppressed per worker; see [resources and autoscaling](/docs/clusters/#resources-and-autoscaling) for attribution and reset rules.
* **Platform rejections remain authoritative.** Diagnostic messages are bounded and never include credentials or environment values; see [resource diagnostics](/docs/clusters/#resources-and-autoscaling).
* **Schedules run in the control plane.** Downtime pauses new scheduled requests and retention. Unchanged enabled schedules with an existing durable cursor retry missed admissions after recovery, at most one occurrence per reload/callback; new or edited schedules have no retroactive guarantee. Already-running engine work has separate semantics; see [backups](/docs/backups/).
* **Deletion can leave charged storage.** Platform refusals can move verified owned volumes into [orphan inventory](/docs/operations/#orphaned-platform-volumes); completing cluster deletion does not prove every volume is gone.
* **No container-exec API.** Node work uses a bounded authenticated agent, not arbitrary remote SSH/exec. [Admin recovery](/docs/operations/#recovering-local-admin-access) uses service secrets/restart or a trusted offline maintenance environment.
* **Deployment environments are snapshots.** Secret changes require restart/redeploy; runtime YAML environment values override service secrets. Member provisioning imports infrastructure before artifact deployment. Resource updates send complete vertical/horizontal settings.
* **Existing targets and cataloged engines only.** Karakoram does not create infrastructure projects/clusters; Zerops is the reference provider, with [Dokploy](/docs/dokploy/) and [Nomad](/docs/nomad/) providers. The system `core` service is outside its ownership. Public [Git](https://git-scm.com/) import requires a public repository; private source needs local deployment. Members use prebuilt artifacts.
## Namespaces and relays
[Section titled “Namespaces and relays”](#namespaces-and-relays)
* **One namespace per provider target.** Names are namespace-local for clusters/storage. Provider kind, namespace name and target identity are immutable; only descriptions change. The `default` infrastructure credential is environment-managed; API rotation returns 409. See [namespace lifecycle](/docs/namespaces/).
* **One relay per additional target; no relay HA.** All current providers require it. Its loss interrupts management/config pulls/monitoring there, not engine replication. It is not application connectivity. Remote management requires outbound public HTTPS and WebSocket forwarding; the default namespace remains direct.
* **One ordered tunnel shares congestion and failure fate.** EOF closes both directions; TCP half-close is not propagated. The relay is not a general VPN or bulk-backup transport. Keepalive/reconnect behavior does not establish a longevity or throughput SLA.
* **Removal guards retained resources.** Clusters, storages, retained backup records and orphan inventory must be cleared first. Removal deletes the owned relay, not its project or RBAC policies; review grants before name reuse.
* **Bounds do not eliminate DoS.** Bounded handshakes/messages/streams still share listener/network resources. Project DNS/network remain trusted; inner relay TLS does not replace engine TLS. See [relay threat model](/docs/security/#relay-threat-model).
## Access and networking
[Section titled “Access and networking”](#access-and-networking)
* On the **Zerops provider**, public TCP requires project IPv6 or dedicated IPv4; shared HTTP/IPv4 routing is not a generic engine TCP gateway. Verify client address-family connectivity and account for dedicated IPv4 cost.
* Zerops public route synchronization is project-wide, including other clusters’ changes. `Ready`/`isSynced` do not establish external reachability. Check allowed and denied clients independently; retain TLS/authentication as well as allowlists. Other providers have the [exposure limits above](#provider-specific-limitations).
* Infrastructure administrators remain trusted. On Zerops, **Full access without sudo mode cannot read sensitive environment values**; this does not remove mutation authority. See [Security](/docs/security/) for token authority, public exposure and TLS boundaries; Karakoram is not a sandbox against infrastructure administrators.
* Credential/leaf rotation is not general CA-source or master-key rotation. Use engine access guides for supported TLS transitions and certificate requirements.
### Certificate and member-credential lifecycle
[Section titled “Certificate and member-credential lifecycle”](#certificate-and-member-credential-lifecycle)
* **Leaf recovery requires valid trust roots and a reachable control plane.** Agent HTTPS leaves are renewed through the member’s authenticated configuration pull, independently of database health or rolling restart gates. An expired or future-dated agent leaf therefore does not have to authenticate a control-plane-to-member status request to receive its replacement. CA verification, hostname verification and bearer authentication remain enabled. Database certificates still follow engine-specific reload/rolling safety rules.
* **Renewal needs writable control-plane state.** If persisting a replacement internal HTTPS leaf fails, new handshakes can continue using the active identity only while it is currently valid. Repair the storage fault before that identity expires; expired or future-dated identities are never used as a continuity fallback.
* **Keep clocks synchronized.** New certificates are backdated by five minutes, not indefinitely. A verifier more than five minutes behind the issuing clock can reject a new certificate as not yet valid; a sufficiently advanced clock can see certificates as expired. Correct the affected clock first. With a currently valid CA, reconciliation replaces future-dated leaves after clock correction. It never replaces an established CA merely because the clock makes it invalid.
* **CA rollover is not generally supported.** Internal and relay CAs are long-lived but finite; issuing a new leaf cannot extend CA validity. Most engine drivers reject cluster CA replacement; [Temporal](https://temporal.io/) explicitly supports a [three-phase cluster CA rotation](/docs/temporal/access-and-tls/#revocation-and-rotate-tls) that invalidates all prior client certificates. Monitor CA expiration externally and arrange supported, coordinated migration; do not replace pinned roots, delete PKI state or recreate data automatically as a renewal workaround. Leaf rotation alone is not a CA migration.
* **Member bearer tokens are non-expiring and not hot-rotatable.** They are separate from browser sessions and API access tokens. Changing only the stored member token does not update a running member’s startup environment and breaks both management directions with authentication failures. Recovery/rotation requires coordinated delivery of the same token to control-plane state and the affected member environment followed by member redeployment/restart. Existing database processes may continue under cached configuration while management is disconnected.
## Observability, auth and tooling
[Section titled “Observability, auth and tooling”](#observability-auth-and-tooling)
* Built-in monitoring is short-term and control-plane-dependent, not independent HA alerting; see [collection, retention and sample freshness](/docs/monitoring/).
* Agent log tails are bounded, not a durable archive; see [log limits and retention](/docs/operations/#health-and-logs).
* Proxies may buffer event delivery. Preserve replay cursors and read durable events/status rather than resubmitting work; transport details live in [API events](/docs/api/#pagination-and-events).
* [OIDC](https://openid.net/developers/how-connect-works/) publicly configures scopes/groups claim, not arbitrary internal authentication options. Username uses `preferred_username`; no public `allowedGroups`/`usernameClaim`/`autoCreate` toggles exist. Login captures group subjects rather than continuously querying the issuer; automatic identity creation grants no roles. See [access control](/docs/access-control/).
* CLI authentication uses saved bearer tokens, not browser/device SSO or a standalone contexts manager. See [CLI](/docs/cli/). Generated-contract consistency checks do not establish behavior; see [verification](/docs/verification/).
## Cluster data MCP
[Section titled “Cluster data MCP”](#cluster-data-mcp)
* [Data MCP](/docs/data-mcp/) is opt-in, limited to [MongoDB](https://www.mongodb.com/)/PSMDB, PostgreSQL, Supabase, [Elasticsearch](https://www.elastic.co/elasticsearch) and [RabbitMQ](https://www.rabbitmq.com/). MongoDB/PostgreSQL/Supabase add a private companion service; native Elasticsearch/RabbitMQ do not.
* Reviewed manifests deny unknown tools. No upstream resource proxying, connection switching or credential reveal is available. Clients without form elicitation/MRTR cannot execute tools requiring confirmation.
* Result/rate/concurrency bounds are not query-cost guarantees. Stored data can inject prompts; confirmations do not provide rollback. RabbitMQ peeking consumes/requeues and therefore needs write access. Elasticsearch hidden/system/dot indices and streams backed by them are refused.
* PostgreSQL’s companion connects to the first scoped database (default `postgres`); existing PUBLIC grants are not revoked and can weaken scope isolation. See [scope and identity caveats](/docs/data-mcp/#managed-identities-and-rotation).
## Known issues in v0.4.2
[Section titled “Known issues in v0.4.2”](#known-issues-in-v042)
* PostgreSQL’s `postgres` administrator password cannot be rotated in place; follow [credential containment](/docs/getting-started/upgrade/#postgresql-administrator-password-cannot-be-rotated-yet).
* After a PXC in-place full or PITR restore on Zerops, a member can be flagged GTID-history-diverged and the cluster Degraded despite correct checked data and writes; incremental backups/PITR continuity can be blocked. Investigation continues; verify data natively. This was not reproduced on Dokploy.
* Nomad live coverage of the new engines remains incomplete; EXPERIMENTAL status remains unchanged.
* Temporal remains unsupported on Zerops because it requires private IPv4.
* Weaviate cannot start on Zerops: members never become Ready because Zerops private networking is IPv6-only and Weaviate’s cluster address is not advertised for IPv6 (also in v0.4.0 and v0.4.1). Nomad and Dokploy are not affected; use Weaviate there until the next release.
See [known issues in v0.4.2](/docs/getting-started/upgrade/#known-issues-in-v042) for details. The formation, full-backup, rolling-status, clone-readiness and resume/exposure issues previously listed for v0.4.1 are [fixed in v0.4.2](/docs/getting-started/upgrade/#fixed-in-v042).
## Engine limits
[Section titled “Engine limits”](#engine-limits)
* [MongoDB limits](/docs/mongodb/#limits)
* [RabbitMQ limits](/docs/rabbitmq/#limits)
* [Elasticsearch limits](/docs/elasticsearch/#limits)
* [Weaviate limits](/docs/weaviate/#limits): fixed three-voter clusters; only non-voters can drain and scale down; scale-out does not rebalance shards. Internal transports are plaintext and require isolated private networking. Full S3 recovery preserves member count; no incremental backups or PITR. Official binaries are mixed-license; only community features are enabled.
* [PostgreSQL limits](/docs/postgres/#limits): [ProxySQL](https://proxysql.com/) has protocol/driver restrictions; SQL-created users are not automatically proxy-routable. Strict synchronous HA can block writes, and major upgrades are offline and backup-gated.
* [Supabase limits](/docs/supabase/#options-and-limits): PostgreSQL 17 only; missing locked component/extension artifacts are rejected. Standard/full preset definitions do not imply available binaries. Database backups omit Storage objects and control-plane-held function/service secrets.
* [OpenBao limits](/docs/openbao/#options-and-limits): 1/3/5 members, TLS-only, no seal migration, PITR, performance/DR replication or automated lost-quorum recovery. Shamir re-unseal depends on the control plane and its original master key; auto-unseal depends on the external transit/KMS key. PostgreSQL storage uses the control plane’s shared PostgreSQL binding service; a server built without that service rejects it and does not advertise the choice. RabbitMQ connection roles must share a default/maximum TTL pair per connection.
* [Temporal limits](/docs/temporal/#limits): Zerops is not supported yet because membership requires private IPv4; use Dokploy or Nomad. Requires a PostgreSQL ≥12 source through the control plane’s shared PostgreSQL binding service; no multi-cluster replication or namespace authorization. Namespace descriptions cannot be cleared. Restores require recorded manifests in Temporal cluster state, the original source/database names and shared-source acknowledgement when applicable; deleting Temporal backup records does not remove PostgreSQL archives.
* [Neo4j Community limits](/docs/neo4j/#access-and-limits): one member and one user database; every user is an administrator. No clustering, online backup, RBAC, native metrics endpoint, PITR or incremental backups. Dumps, restores, restarts and upgrades interrupt all service.
* [MySQL and PXC limits](/docs/mysql/): engine drivers cannot be switched in place; MySQL single-primary Group Replication and PXC [Galera](https://galeracluster.com/) have distinct recovery/provenance rules. TLS is required; private replication ports need operator-enforced isolation.
* [Keycloak limits](/docs/keycloak/): exactly one runtime, maintenance downtime, PostgreSQL binding required; recovery protects source database data separately from the coordinator’s sealed administrator state.
* [Native artifacts](/docs/native-artifacts/) currently target [Linux](https://www.kernel.org/) amd64/[Ubuntu](https://ubuntu.com/) 24.04. Development use of source-built locked artifacts needs a configured mirror; mirrors cannot add absent capabilities.
New-engine and [Citus](https://www.citusdata.com/) release status is centralized in [engine availability](/docs/providers/#engine-availability). [Fleet operations](/docs/operations/), [notifications](/docs/notifications/), [control-plane recovery](/docs/control-plane-backup/) and [backup drills](/docs/backup-drills/) are verified offline, not live-provider acceptance or recovery-time guarantees.
# MCP server
> Karakoram embeds an MCP server at https://YOUR-CONTROL-PLANE/mcp. It executes management requests in-process through the same REST authentication,…
Karakoram embeds an [MCP](https://modelcontextprotocol.io/) server at `https://YOUR-CONTROL-PLANE/mcp`. It executes management requests in-process through the same REST authentication, authorization, validation, idempotency and audit path. It never sends telemetry. For a step-by-step client setup, start with the [MCP quick start](/docs/mcp-quickstart/).
## Enable and authenticate
[Section titled “Enable and authenticate”](#enable-and-authenticate)
`KARAKORAM_MCP=readonly` is the default: only read-risk tools are available. Set `readwrite` to expose mutations and destructive tools, or `off` to make `/mcp` return 404. Invalid values prevent startup. Environment changes require a restart. For an import YAML, add the setting to the Karakoram service’s environment variables (retain existing variables):
```yaml
services:
- hostname: karakoram
envVariables:
KARAKORAM_MCP: readonly
```
Use a dedicated Karakoram API token in the HTTP `Authorization: Bearer …` header, not a [Zerops](https://zerops.io/) project token. Keep tokens in your client’s secret store, never in a checked-in project configuration. Cookies are ignored. Token owner permissions are evaluated again for each REST operation; revocation stops subsequent calls. Prefer restricted, read-only tokens scoped to the namespaces and clusters the agent needs. Credential reveal requires its own permission and token restriction allowance.
The endpoint supports stateless MCP `2026-07-28` and legacy initialize-based Streamable HTTP (`2025-11-25`, `2025-06-18`, `2025-03-26`). It does not serve deprecated HTTP+SSE. Legacy sessions are memory-only, principal-bound and expire after 30 idle minutes; at most 256 are retained.
Tool lists are filtered by the caller’s current permissions and selected toolsets. On MCP `2026-07-28`, `tools/list` returns `cacheScope: "private"` and `ttlMs: 0`, so lists are immediately stale and must not be shared between callers.
## Client configuration
[Section titled “Client configuration”](#client-configuration)
[Claude Code](https://code.claude.com/docs/en/overview) supports a remote HTTP server:
```sh
claude mcp add --transport http karakoram https://YOUR-CONTROL-PLANE/mcp --header 'Authorization: Bearer YOUR_API_TOKEN'
```
[Cursor](https://cursor.com/) (`.cursor/mcp.json`) and clients accepting `mcpServers` use this generic shape (substitute the token through the client’s supported secret mechanism):
```json
{
"mcpServers": {
"karakoram": {
"url": "https://YOUR-CONTROL-PLANE/mcp",
"headers": {"Authorization": "Bearer YOUR_API_TOKEN"}
}
}
}
```
[VS Code](https://code.visualstudio.com/) (`.vscode/mcp.json`) uses `servers` and supports prompted secret inputs:
```json
{
"inputs": [{"id": "karakoram-token", "type": "promptString", "description": "Karakoram API token", "password": true}],
"servers": {
"karakoram": {
"type": "http",
"url": "https://YOUR-CONTROL-PLANE/mcp",
"headers": {"Authorization": "Bearer ${input:karakoram-token}"}
}
}
}
```
Client configuration formats change; use the client’s remote Streamable HTTP option, endpoint URL and bearer header if its format differs.
## Tools, completion and confirmation
[Section titled “Tools, completion and confirmation”](#tools-completion-and-confirmation)
Tool names are stable snake_case names from the [OpenAPI](https://www.openapis.org/) agent metadata. Path and query parameters are top-level arguments; JSON request bodies are under `body`. Nullable fields accept JSON `null`, including merge-patch removals and removing an API token’s expiry. Lists are sorted and filtered by server mode, current RBAC and token restrictions. Narrow large lists using OpenAPI tags, for example `/mcp?toolsets=Clusters,Backups`; names are case-insensitive. Include `Workflows` to retain workflow tools when narrowing.
`wait` observes an operation ID, a cluster generation, a backup or restore, with a default 120-second and maximum 600-second wait. Cancelling the client wait never cancels the underlying operation. `cluster_diagnose` gathers status, members, recent operations, warning events and backups; unavailable or unauthorized sections are omitted with notes.
Mutations are asynchronous even when REST returns a cluster instead of HTTP 202. Results include `operationId` when the REST response supplies it. Use `wait`; after any timeout or broken stream, check state before resubmitting. Identical arguments derive the same idempotency key within a ten-minute time bucket; crossing a bucket is not a replay guarantee.
Destructive tools require a human elicitation response. The server writes the approval summary, including dry-run changes/effects where available, and binds approval to the principal, tool and exact arguments for five minutes. Summaries identify the namespace and full nested resource path where applicable. Some operations additionally require typing the actual resource’s name: for example, deleting a backup requires the backup name, not its parent cluster name. Decline, cancel, missing or expired approval, altered arguments, or a mismatched name execute nothing. There is no model-settable confirmation argument. Clients without elicitation must use `karactl` or the web UI instead. Acceptance and rejection are audited; successful management actions carry an `(mcp)` actor suffix.
## Security
[Section titled “Security”](#security)
Readonly mode classifies side effects, not confidentiality: secret-bearing operations return actual material when authorized, returned logs, events and database-controlled strings are untrusted data, and tool annotations are guidance while REST authorization and server-side confirmation are enforcement. The [agent access security model](/docs/security/#agent-access-mcp-security-model) describes credentials, enforcement and what is not protected. Operationally: the public URL must match the control plane origin (other browser `Origin` values are rejected), run behind HTTPS, and store the master key securely because confirmation state and retry identities use keys derived from it.
## Connect with OAuth
[Section titled “Connect with OAuth”](#connect-with-oauth)
Set `KARAKORAM_OAUTH=on` and restart the control plane. [OAuth](https://oauth.net/2/) is off by default. Configure the connector with `https:///mcp`. Its 401 challenge links to protected-resource metadata, which advertises the embedded authorization server. Use the same public HTTPS origin as `KARAKORAM_PUBLIC_URL`.
Clients may use an HTTPS client ID metadata document. Its `client_id` must equal the document URL; `client_name` and `redirect_uris` are required. Optionally set `KARAKORAM_OAUTH_CLIENT_HOSTS=client.example,other.example` to restrict metadata hosts. Private/loopback/link-local destinations and HTTP redirects are never accepted, even for allowlisted hosts. Metadata is cached briefly; failed fetches are cached for five seconds. Slow metadata retrieval does not block existing connections’ token authentication, refresh or revocation.
For clients requiring dynamic registration, additionally enable `KARAKORAM_OAUTH_DCR=on`; registration is otherwise unavailable. Only public clients (`token_endpoint_auth_method: none`) are supported. Client redirects must match exactly; HTTP loopback IP redirects may vary only in port.
Sign in with your ordinary Karakoram browser account (local or SSO); SSO returns to the consent flow after login. Every authorization shows a new consent page with the client, verified metadata host, redirect and scopes. Review these before approving; optionally enter a comma-separated namespace limit. Declining sends `access_denied` to the validated client redirect. Authorization uses single-use codes and mandatory S256 PKCE.
| Scope | Authority (always intersected with your current RBAC) |
| ----------------------- | ----------------------------------------------------- |
| `karakoram:read` | Default; read-only, without credential reveal. |
| `karakoram:write` | Allows writes permitted by your account and MCP mode. |
| `karakoram:credentials` | Allows credential reveal permitted by your account. |
Access tokens last 10 minutes and work only at `/mcp`, not the REST API. Refresh tokens rotate on use; replay disconnects the entire grant. Grants last at most 30 days. Account lifecycle changes are enforced immediately. OAuth connections cannot create or change API tokens; mint dedicated tokens with `karactl` or the web UI.
Open **Settings → Connected apps** to inspect clients, scopes, namespace limits, creation and last-use times, and revoke a connection. API equivalents are `listOAuthGrants` and `deleteOAuthGrant`. Administrators with `api-tokens` permissions may inspect/revoke all accounts’ grants. Grant issuance, revocation and refresh-token reuse are audited without recording credentials.
Authorization responses identify this installation with the `iss` parameter; clients should compare it with discovered issuer metadata before exchanging a code. Discovery and authorization follow [RFC 9728](https://www.rfc-editor.org/rfc/rfc9728), [RFC 8414](https://www.rfc-editor.org/rfc/rfc8414) and the [MCP authorization profile](https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization).
## karactl bridge and configuration
[Section titled “karactl bridge and configuration”](#karactl-bridge-and-configuration)
For clients requiring stdio, run `karactl --context production mcp`. The bridge connects to the selected control plane’s `/mcp` with its API bearer token, without an `Origin` header. Server URL and token use the normal CLI flag/environment/context precedence. Standard output is exclusively MCP JSON-RPC; diagnostics use stderr. Instructions, tool discovery, calls, progress, cancellation, ping and confirmation round trips are forwarded. Both native multi-round-trip input requests and legacy elicitation are supported; the bridge never approves an action itself. Server mode, permissions, token restrictions and confirmation remain authoritative.
Use `--toolsets Clusters,Backups,Workflows` to narrow discovery on the server. This does not grant permission or change server mode.
`karactl mcp config --client claude-code|cursor|vscode|codex|json` creates a dedicated restricted API token and prints remote HTTP client configuration. The default client is `json`, token name is `mcp-CLIENT-DATE`, and expiry is 90 days. Repeat `--namespace NAME` to restrict its scope; `--write` removes the token’s read-only restriction but cannot override owner RBAC or server readonly mode. Override the name with `--token-name` or lifetime with `--expires-in 30d` ([Go](https://go.dev/) durations such as `720h` are also accepted).
The same command with `--cluster NAMESPACE/NAME` mints a cluster-scoped data token for the per-cluster [data MCP](/docs/data-mcp/#connect-an-agent-client) endpoint instead; `--namespace`, `--stdio` and `--output` do not apply there.
```sh
karactl --context production mcp config --client cursor --namespace production
karactl --context production mcp config --client vscode --expires-in 30d --output /secure/path/mcp.json
karactl --context production mcp config --client codex --stdio
```
Claude Code output is a shell command, [Codex](https://developers.openai.com/codex/) output is TOML, and other clients receive JSON. Remote configurations include the endpoint and `Authorization: Bearer` header. Protect this output: the newly created secret is printed once with a warning. `--stdio` prints a command using the selected saved context, never embeds a token in the snippet, and reveals the new dedicated token separately on stderr for provisioning. It does not modify the context; save that token into a dedicated context if the bridge should use its restrictions.
No files are written unless `--output FILE` is set, which writes the snippet with mode 0600. Do not commit generated configurations containing credentials. See the [CLI guide](/docs/cli/#mcp-clients) for context and output details.
# MCP quick start
> Connect an AI agent to a Karakoram installation in a few minutes. The management endpoint https://YOUR-CONTROL-PLANE/mcp exposes the REST API as MCP tools;…
Connect an AI agent to a Karakoram installation in a few minutes. The management endpoint `https://YOUR-CONTROL-PLANE/mcp` exposes the REST API as [MCP](https://modelcontextprotocol.io/) tools; each cluster can additionally expose the data inside it at `/mcp/clusters/{namespace}/{cluster}`. This page covers the common path; the [MCP server](/docs/mcp/), [cluster data MCP](/docs/data-mcp/) and [agent access security model](/docs/security/#agent-access-mcp-security-model) pages are the reference.
## 1. Check the server mode
[Section titled “1. Check the server mode”](#1-check-the-server-mode)
`/mcp` is enabled after installation in **read-only** mode: `KARAKORAM_MCP=readonly` is the default, so agents can inspect clusters, backups, operations and events but cannot change anything. Set `readwrite` to also expose mutations and destructive tools (destructive tools still require a human confirmation), or `off` to make `/mcp` return 404. On [Zerops](https://zerops.io/), add the variable to the Karakoram service’s environment and restart; in an import YAML the setting is:
```yaml
services:
- hostname: karakoram
envVariables:
KARAKORAM_MCP: readwrite
```
Check the effective mode without credentials: an enabled endpoint answers `401` with a `WWW-Authenticate: Bearer` challenge; a disabled one answers `404`.
```sh
curl -s -o /dev/null -w '%{http_code}\n' https://YOUR-CONTROL-PLANE/mcp
```
The mode applies to the whole installation. Whether a particular agent may write is additionally decided by its token and its owner’s RBAC, so the server can stay in `readwrite` while most agents hold read-only tokens.
## 2. Mint a restricted token
[Section titled “2. Mint a restricted token”](#2-mint-a-restricted-token)
Sign in with `karactl` once ([login and contexts](/docs/cli/#login-and-contexts)), then let `karactl mcp config` create a dedicated API token and print the configuration for your client:
```sh
karactl --context production mcp config --client claude-code --namespace production
```
| Flag | Effect |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--client claude-code\|cursor\|vscode\|codex\|json` | Output format; `json` (default) is a generic `mcpServers` snippet |
| `-n/--namespace NAME` | Restrict the token to a namespace; repeatable. Without it the token sees every namespace its owner can see |
| `--write` | Remove the token’s read-only restriction. It never exceeds the owner’s RBAC or the server mode |
| `--expires-in 90d` | Lifetime as whole days or a Go duration; 90 days by default |
| `--token-name NAME` | Override the default `mcp-CLIENT-DATE` |
| `--output FILE` | Write the snippet to a file with mode 0600 instead of stdout |
| `--stdio` | Print a `karactl mcp` stdio command for the selected saved context instead of embedding a token; the new token is printed separately and is not saved anywhere |
The token secret is shown **once**. Keep it in the client’s configuration or secret store, never in a repository or ticket. Revoke it with `karactl tokens delete ID --yes` or in **Settings → API tokens** when the agent no longer needs it. The same page creates equivalent tokens by hand under **Restrictions** (read-only, namespaces, clusters, permissions); `karactl tokens create --name mcp-agent --read-only --scope-namespace NAME` does the same from the shell ([restricted API tokens](/docs/access-control/#restricted-api-tokens)).
## 3. Connect your client
[Section titled “3. Connect your client”](#3-connect-your-client)
All remote configurations use the endpoint URL and an `Authorization: Bearer` header over Streamable HTTP. Clients that only speak stdio run the `karactl mcp` bridge instead; it forwards everything, including confirmations, to the server.
### Claude Code
[Section titled “Claude Code”](#claude-code)
Run the printed command, substituting your values:
```sh
claude mcp add --transport http karakoram https://YOUR-CONTROL-PLANE/mcp --header 'Authorization: Bearer YOUR_API_TOKEN'
```
`--client claude-code --stdio` prints `claude mcp add --transport stdio karakoram -- karactl --context production mcp` instead, which keeps the token in `karactl`’s saved context rather than in [Claude Code](https://code.claude.com/docs/en/overview)’s settings. `mcp config` never updates a saved context, so that command uses the context’s existing credential: first save the newly printed restricted token into a dedicated context (`karactl config set-context mcp-agent --server https://YOUR-CONTROL-PLANE --token NEW_TOKEN`) and use `--context mcp-agent` in the command. Use `/mcp` inside Claude Code to check the connection.
### Claude Desktop
[Section titled “Claude Desktop”](#claude-desktop)
[Claude Desktop](https://claude.com/download) (and claude.ai) connect to remote servers through **custom connectors**, which are reached from [Anthropic](https://www.anthropic.com/)’s cloud, so the control plane’s HTTPS origin must be reachable from the internet. Custom connectors authenticate with [OAuth](https://oauth.net/2/): set `KARAKORAM_OAUTH=on` on the control plane ([connect with OAuth](/docs/mcp/#connect-with-oauth)), then open **Customize → Connectors → Add custom connector**, enter `https://YOUR-CONTROL-PLANE/mcp` and choose **Use [Claude](https://claude.com/)’s published identity** as the OAuth client. That option uses a client ID metadata document, which Karakoram accepts without dynamic registration; **Register automatically** additionally requires `KARAKORAM_OAUTH_DCR=on`. Each user signs in with their own Karakoram account and approves the scopes and namespace limit on the consent page; the connection appears under **Settings → Connected apps**. Fixed bearer headers for custom connectors are currently a beta feature in Claude that is not available to every organization.
Without OAuth, run the stdio bridge locally. Choose **Settings → Developer → Edit Config** in Claude Desktop and add the server to `claude_desktop_config.json` (this is the output of `--client json --stdio`):
```json
{
"mcpServers": {
"karakoram": {
"command": "karactl",
"args": ["--context", "production", "mcp"]
}
}
}
```
If the desktop app cannot find `karactl`, use its absolute path. As with Claude Code, point `--context` at a dedicated context holding the restricted token, not your administrator context. Restart Claude Desktop after editing the file.
### Cursor
[Section titled “Cursor”](#cursor)
`--client cursor` prints an `mcpServers` entry with `url` and `headers`. Put it in `.cursor/mcp.json` in the project or `~/.cursor/mcp.json` for all projects, for example with `--output ~/.cursor/mcp.json` when the file does not exist yet. [Cursor](https://cursor.com/) stores the header value in that file, so keep project files out of version control or use `--client cursor --stdio`, which references the saved context instead of a token.
### VS Code
[Section titled “VS Code”](#vs-code)
`--client vscode` prints a `servers` entry with `type: http`. Put it in `.vscode/mcp.json` (workspace) or your user profile’s `mcp.json`. To keep the token out of the file, replace the header value with a prompted input as shown in the [MCP server guide](/docs/mcp/#client-configuration); [VS Code](https://code.visualstudio.com/) asks for the secret once and stores it securely.
### Codex
[Section titled “Codex”](#codex)
`--client codex` prints a TOML table for `~/.codex/config.toml` (or a trusted project’s `.codex/config.toml`):
```toml
[mcp_servers.karakoram]
url = "https://YOUR-CONTROL-PLANE/mcp"
http_headers = { Authorization = "Bearer YOUR_API_TOKEN" }
```
To avoid writing the token into the file, replace `http_headers` with `bearer_token_env_var = "KARAKORAM_MCP_TOKEN"` and export that variable where [Codex](https://developers.openai.com/codex/) runs. `--client codex --stdio` prints a `command`/`args` table for the bridge instead. With `KARAKORAM_OAUTH=on`, `codex mcp login karakoram` starts the OAuth flow.
### Other clients
[Section titled “Other clients”](#other-clients)
Any spec-compliant client works. Use its remote Streamable HTTP option with the endpoint URL and the bearer header (`--client json` prints `url` and `headers`), or OAuth when enabled. The server speaks MCP `2026-07-28` and the legacy initialize-based revisions `2025-11-25`, `2025-06-18` and `2025-03-26`; it does not serve the deprecated HTTP+SSE transport. Stdio-only clients run `karactl --context production mcp` as the server command.
To inspect the endpoint before wiring it into an agent, the MCP Inspector lists the tools your token can see:
```sh
npx @modelcontextprotocol/inspector --cli --transport http --server-url https://YOUR-CONTROL-PLANE/mcp \
--header 'Authorization: Bearer YOUR_API_TOKEN' --method tools/list
```
## 4. Try it
[Section titled “4. Try it”](#4-try-it)
Start with questions, not changes; everything below works in read-only mode:
* “Which clusters can you see, and what phase is each in?” (`clusters_list`, `cluster_status_get`)
* “Diagnose cluster `appdb` in namespace `production`.” (`cluster_diagnose` gathers status, members, recent operations, warning events and backups)
* “When was the last successful backup of `appdb`?” (`backups_list`)
* “What would change if `appdb` had three members? Preview only.” (`cluster_update_dry_run` returns the changed paths, effects and warnings without applying anything)
In `readwrite` mode with a `--write` token, “Take a backup of `appdb` and wait for it” calls `backup_create` and then `wait`; mutations are asynchronous and `wait` observes the operation. A destructive request such as deleting a backup pauses in your client for a confirmation written by the server; some confirmations require typing the resource’s actual name. If your client does not support elicitation, the tool returns an error naming the equivalent `karactl` command and nothing executes.
Clients with tool limits can narrow the list on the server: append `?toolsets=Clusters,Backups,Workflows` to the endpoint URL ([OpenAPI](https://www.openapis.org/) tags; `Workflows` keeps `wait` and `cluster_diagnose`).
## 5. Data MCP quick start
[Section titled “5. Data MCP quick start”](#5-data-mcp-quick-start)
[Cluster data MCP](/docs/data-mcp/) gives an agent tools for the data inside one cluster through a separate per-cluster endpoint. It is off per cluster until you enable it, and it needs the `cluster-data` permission, which the built-in viewer and operator roles do not have.
1. Enable it on the cluster with a merge patch (read-only, one [MongoDB](https://www.mongodb.com/) database in this example):
```yaml
spec:
dataMcp:
enabled: true
access: read
scope:
databases: [app]
```
```sh
karactl --context production -n production clusters patch appdb -f datamcp.yaml
```
MongoDB, [PostgreSQL](https://www.postgresql.org/) and [Supabase](https://supabase.com/) clusters start a small companion service; [Elasticsearch](https://www.elastic.co/elasticsearch) and [RabbitMQ](https://www.rabbitmq.com/) use built-in tools. Wait until `status.dataMcp.phase` is `Ready` (`karactl -n production clusters get appdb -o json`); the web UI shows the same on the cluster’s **Data MCP** card.
2. Make sure your account holds `cluster-data:read` on `production/appdb`. Administrators already do; otherwise add `p, user:NAME, cluster-data, read, production/appdb` to the [policy](/docs/access-control/#policy-format).
3. Mint a cluster-scoped token and connect:
```sh
karactl --context production mcp config --cluster production/appdb --client claude-code
```
`--cluster` accepts `claude-code`, `cursor`, `vscode` and `json`; `--namespace`, `--stdio` and `--output` do not apply. The printed configuration targets `https://YOUR-CONTROL-PLANE/mcp/clusters/production/appdb`.
4. Ask: “Which collections are in database `app`, and what does a typical document look like?” Read tools run through a dedicated read-only database identity. Writes require [all four switches](/docs/data-mcp/#four-switches-for-writes): cluster `access: readWrite`, a `--write` token, owner RBAC `cluster-data:write` and `KARAKORAM_MCP=readwrite`; destructive tools always ask for confirmation.
Treat everything an agent reads from a database as untrusted data; see [risks and operating practice](/docs/data-mcp/#risks-and-operating-practice).
# MongoDB
> Manage Percona Server for MongoDB (PSMDB) and Percona Backup for MongoDB (PBM) on Zerops.
Manage [Percona Server for MongoDB](https://docs.percona.com/percona-server-for-mongodb/) (PSMDB) and [Percona Backup for MongoDB](https://docs.percona.com/percona-backup-mongodb/) (PBM) on [Zerops](https://zerops.io/).
Opt-in [cluster data MCP](/docs/data-mcp/) uses a private `mongodb-mcp-server` companion and separate scoped read/write users. Configure `scope.databases`; system databases cannot be scoped. It is disabled by default and independent of ordinary credential reveal.
## At a glance
[Section titled “At a glance”](#at-a-glance)
| Setting | Supported value / default |
| ------------------------------- | ------------------------------------------------------------------------------------------ |
| Engine driver | `psmdb` |
| PSMDB catalog | `6.0.29-23` (deprecated); `7.0.40-22`, `7.0.43-23`, `8.0.29-13`; **`8.0.32-14` (default)** |
| PBM | `2.16.0-1`; Debian package pins append `.noble` |
| Topologies | `replicaSet`: 1/3/5/7 data-bearing members; `sharded`: config set, shard sets and mongos |
| Resource floors | Whole CPU cores ≥1; RAM ≥0.5 GiB; disk ≥5 GiB; maxima ≥minima; start CPU within bounds |
| Engine driver resource defaults | SHARED CPU, min/max 1 core, 1 GiB RAM, 5 GiB disk; omitted maxima follow effective minima |
| Database TLS | `requireTLS`, generated per-cluster CA |
| Recovery | PBM `logical` / `physical` snapshots and PITR |
| Public TCP | Opt-in, CIDR allowlist; project IPv6 or dedicated IPv4 |
Query your installation with `karactl providers get psmdb -o yaml`. Catalog defaults are resolved at creation, not automatic upgrades. Package availability is not an upstream lifecycle guarantee; PSMDB 8.3 Technical Preview is excluded. [Operations](/docs/mongodb/operations/) describes supported upgrade paths.
## How it runs
[Section titled “How it runs”](#how-it-runs)
Each mongod has a singleton Zerops runtime and persistent Local Storage volume. Karakoram deploys its agent and pinned Percona packages, bootstraps the first member through MongoDB’s localhost exception, and establishes reserved management accounts. Sharded creation initializes config servers, shard sets and routers, then registers shards.
Runtime names are `{cluster}m{index}`, `{cluster}s{shard}m{index}`, `{cluster}c{index}` and `{cluster}r`; volumes append `v`. Replacement indexes are monotonic. Routers are stateless. Agents supervise engine processes and cache protected configuration; see [runtime operations](/docs/mongodb/operations/).
## Create
[Section titled “Create”](#create)
Select the target [namespace](/docs/namespaces/), then **Clusters → Create cluster → MongoDB**. The wizard exposes:
| Step | MongoDB choices |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Basics | Name, exact version, replica set or sharded topology, optional preset |
| Topology / Resources | Member/shard/config-server/router counts; component CPU/RAM/disk bounds; validated YAML overrides; routers have no durable data-disk editor |
| Backups | Registered storage or None, logical/physical schedules, PITR interval, optional backup/time seed and source namespace |
| Access & TLS | TLS mode/CA and optional certificate pairs, exposure/IP family/CIDRs, monitoring |
| Review | All settings, aggregate runtime CPU/RAM, data volumes/capacity, separate runtime disks and placement warning |
**Custom** uses no preset; **dev** selects one member without redundancy; **prod** selects three voters; **sharded** selects two three-member shards, three config servers and one router. Presets use the resource defaults above and remain editable; none certifies capacity or host separation. Check aggregate footprint against project capacity; object storage is additional, and totals are not a price quote or initial allocation.
CLI example (after [CLI authentication](/docs/cli/)):
```sh
karactl -n default clusters create demo --provider psmdb --topology replicaSet --members 3 --wait
```
This uses CLI convenience resource bounds; use the explicit bounds in the replica-set guide when sizing a fixed footprint.
Use the [replica-set checklist and CLI alternative](/docs/mongodb/replica-sets/) or [sharded creation example](/docs/mongodb/sharding/). Connect using [scoped application users and TLS](/docs/mongodb/access-and-tls/), not revealed administrative credentials.
## Limits
[Section titled “Limits”](#limits)
* Names and topology type are immutable; names are 2–12 lowercase alphanumeric characters starting with a letter.
* No arbiters/custom voting layout or per-shard independent component specs; [topology bounds and shard administration](/docs/mongodb/sharding/) apply.
* No shard removal or replica-set-to-sharded conversion; migrate into a compatible new deployment.
* Singletons incur maintenance downtime; multiple voters do not establish [independent failure domains](/docs/limitations/).
* The RAM floor is for disposable tiny workloads, not a load/recovery guarantee; reserve WiredTiger/PBM headroom.
* No arbitrary package versions, downgrades or skipped-major upgrades; [binary upgrades and explicit FCV](/docs/mongodb/operations/) are separate.
* Overrides cannot replace managed networking/authentication/TLS/replication/sharding/process/logging/dbpath/cache settings.
* Public seeds are not external replica-set HA; read [public exposure and discovery](/docs/mongodb/access-and-tls/#public-exposure-and-discovery).
* Shared [SeaweedFS](https://seaweedfs.com/) dbpaths and live volume archives are not supported MongoDB recovery substitutes; use [PBM](/docs/mongodb/backups/).
* Restore is disruptive, compatibility-bound and not necessarily a complete logical rollback; [restore and clone](/docs/mongodb/backups/) define coverage and retained-snapshot requirements.
* Count retention does not guarantee PITR duration; inspect observed continuous windows after backup/storage changes.
* [Monitoring and process tails](/docs/mongodb/operations/#monitoring-vocabulary) have bounded coverage; see [log retention](/docs/operations/#health-and-logs).
* Existing agents update on member deployment, not control-plane restart; lost control-plane state cannot safely adopt members. Preserve [state and original encryption key](/docs/operations/#control-plane-state-backup-and-recovery).
# Users, roles and TLS
> Manage MongoDB identities and database transport; Karakoram accounts authorize the control plane separately.
Manage [MongoDB](https://www.mongodb.com/) identities and database transport; [Karakoram accounts](/docs/access-control/) authorize the control plane separately.
## Application users and custom roles
[Section titled “Application users and custom roles”](#application-users-and-custom-roles)
Use **Users → Add user** or an API-shaped `app-user.yaml`:
```yaml
username: appwriter
database: appdb
roles:
- role: readWrite
db: appdb
```
```sh
karactl -n default db-users create demo -f app-user.yaml
karactl -n default db-users list demo
karactl -n default db-users get demo appdb appwriter
```
Omit `password` to receive `generatedPassword` **once**; save it before closing the UI success dialog. Subsequent reads cannot recover it. `db-users set CLUSTER DATABASE USER -f FILE` changes roles/password; omission retains the password. `db-users delete CLUSTER DATABASE USER` removes the identity. UI empty-password fields mean generation on create, retention on edit; supplied passwords are write-only.
Supplied passwords need ≥8 Unicode characters and MongoDB SCRAM’s SASLprep (Unicode 3.2) profile. Punctuation, accented Latin and supported CJK are accepted; emoji, prohibited/unassigned characters, controls and invalid bidirectional text are rejected. Explicit empty passwords return 400 with `details.field: password`, reason `out_of_range` for length or `invalid` for profile violations, without echoing secrets.
Custom roles accept inherited role references and MongoDB resource/action documents, for example `app-role.yaml`:
```yaml
name: appreader
database: appdb
roles: []
privileges:
- resource: {db: appdb, collection: orders}
actions: [find]
```
Use `db-roles create demo -f app-role.yaml`, `list demo`, `get demo appdb appreader`, `set demo appdb appreader -f FILE` or `delete demo appdb appreader`. Built-in roles are referenced, not editable custom roles; reserved identities are protected. `cluster`/`anyResource` resources are not database/collection substitutes. These operations require Karakoram `database-users` permissions; grant workloads only required database privileges.
Deleting an assigned/inherited role returns **409**, reason `invalid_state`, naming dependents. Remove references with user/role updates, then retry DELETE; no force-delete exists. The UI lists known dependents but does not revoke them, and the server rechecks stale UI data.
## System users and rotation
[Section titled “System users and rotation”](#system-users-and-rotation)
Reserved `admin` accounts have independent generated passwords:
| Account | Roles on `admin` unless noted |
| ---------------- | ------------------------------------------------------------------------------------------------------ |
| `clusterAdmin` | `clusterAdmin`: topology administration |
| `userAdmin` | `userAdminAnyDatabase`: users/roles and restored-user reconciliation |
| `clusterMonitor` | `clusterMonitor`, `read@local`: monitoring |
| `backup` | `backup`, `restore`, `clusterMonitor`, `readWrite`, custom `pbmAnyAction` |
| `databaseAdmin` | `readWriteAnyDatabase`, `readAnyDatabase`, `dbAdminAnyDatabase`, `backup`, `restore`, `clusterMonitor` |
**Warning:** `pbmAnyAction` grants `anyAction` on `anyResource`; [PBM](https://docs.percona.com/percona-backup-mongodb/) is not read-only. Database administration and user administration also carry broad/escalation authority. Never use these system accounts in applications.
```sh
karactl -n default system-users list demo
karactl -n default system-users rotate demo backup --yes
karactl -n default operations cluster demo
```
Rotation changes credentials per set and reloads local backup/monitor credentials member by member. Follow the durable operation; do not manually reset managed passwords. Listings reveal no passwords. Successful rotation advances durable UTC `lastRotatedAt` (**Last rotated**); absence means no recorded success, and failed/running rotation does not advance it.
The separate generated intra-cluster keyfile is owner-only **0400**, not an application credential or TLS key. Public reads do not return it; no public keyfile-rotation command exists. Preserve encrypted cluster secrets during [state recovery](/docs/operations/#control-plane-state-backup-and-recovery).
## TLS modes and generated CA
[Section titled “TLS modes and generated CA”](#tls-modes-and-generated-ca)
| Mode | Database traffic |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `requireTLS` (default) | TLS required; password clients need the trusted CA, not client certificates (`allowConnectionsWithoutCertificates: true`) |
| `preferTLS` | Members send TLS; plaintext clients remain allowed |
| `disabled` | No database TLS; authentication/keyfile and separately protected agent transport remain |
The generated cluster CA uses ECDSA P-256, distinct from installation internal PKI. Member leaves cover private hostnames, localhost IPs and known project public IPs. Trust this CA; never disable hostname/certificate verification.
Patch `spec.tls: {mode: requireTLS, ca: selfSigned}` with `clusters patch demo -f FILE --wait`. Changes use rolling passes, waiting for configuration acknowledgement and replication catch-up between members/passes. Enabling traverses `disabled → allowTLS → preferTLS → requireTLS`, stopping at the requested mode; disabling reverses it. Internal `allowTLS` accepts both but sends plaintext; it is not a public option. Directly mixing disabled/preferTLS can break replication/stepdown.
`karactl clusters rotate-tls demo --wait --yes` rotates leaves under the existing CA, not the master key or CA source. CA source **and stored signing certificate/key are immutable**: changed material returns **409**; identical resubmission is accepted. Controller-issued database leaves renew within 30 days of expiry, staged one member at a time behind disruption/replication checks. Agent HTTPS leaves are published independently and hot-swapped through authenticated configuration pulls, so expired or future-dated agent transport cannot block its own renewal. See [certificate lifecycle limits](/docs/limitations/#certificate-and-member-credential-lifecycle).
## User-provided PKI
[Section titled “User-provided PKI”](#user-provided-pki)
For `tls.ca: userProvided`, provide paired write-only `caCertificate`/`caPrivateKey` PEM fields: matching signing CA/key, RSA ≥2048 or ECDSA P-256/P-384/P-521, with CA/signing constraints. A public CA bundle alone is insufficient. Use a dedicated cluster CA, not an organizational root.
Optional paired `certificate`/`privateKey` must match, be currently valid, verify under that CA for **serverAuth and clientAuth**, and cover every current member hostname plus **IP SAN `127.0.0.1` for PBM** and the selected public IP when exposed. One supplied leaf serves relevant members; new monotonic hostnames may need expanded SANs before replacement/scaling. Private keys are encrypted and omitted from ordinary reads.
Supplied member leaves do not auto-renew: within 30 days status reports `TLSCertificate=False`, reason `ExpiringSoon`, then `Expired`. Replace the pair before expiry. Agent/controller-issued leaves still renew under a supplied CA. Explicit leaf rotation issues under the current CA; do not assume it preserves a supplied leaf unchanged.
Use protected create/patch files and YAML block scalars, not terminal history/screenshots. Existing userProvided clusters may omit CA inputs to retain stored material. The editor’s ability to enter another CA source does not override immutable-source validation.
## Connection details and application recipe
[Section titled “Connection details and application recipe”](#connection-details-and-application-recipe)
Ordinary status provides private hosts, replica-set name, password-free URI and provisioned public endpoints. **Reveal credentials** / `clusters connection demo --reveal` requires `cluster-credentials/read` on `/`, returns `Cache-Control: no-store`, and reveals **databaseAdmin only**, never application passwords. Do not use that URI in an app or retain full reveal JSON in logs.
Provision from a trusted **operator workstation**, with [authenticated karactl](/docs/cli/) and `jq`; the application needs no Karakoram token or CLI. Wait for [healthy observed generation](/docs/mongodb/replica-sets/).
1. Create the scoped user above and capture its one-time password with tracing disabled:
```sh
set +x
NAMESPACE=default
CLUSTER=demo
APP_PASSWORD=$(karactl -n "$NAMESPACE" db-users create "$CLUSTER" \
-f app-user.yaml -o json | jq -er .generatedPassword)
karactl -n "$NAMESPACE" clusters ca "$CLUSTER" > ca.pem
```
CA export requires credential-read permission and uses the reveal endpoint; do not retain its other fields. For an existing user, load the saved password instead of recreating it.
2. Read hosts/replica-set from **non-reveal status**, then URI-encode the application password:
```sh
SUMMARY=$(karactl -n "$NAMESPACE" clusters status "$CLUSTER" -o json \
| jq -c .connection)
HOSTS=$(printf '%s' "$SUMMARY" | jq -er '.hosts | join(",")')
RS=$(printf '%s' "$SUMMARY" | jq -r '.replicaSet // empty')
PASSWORD=$(printf '%s' "$APP_PASSWORD" | jq -sRr @uri)
OPTIONS='authSource=appdb&tls=true'
if [ -n "$RS" ]; then OPTIONS="$OPTIONS&replicaSet=$RS"; fi
MONGODB_URI="mongodb://appwriter:$PASSWORD@$HOSTS/appdb?$OPTIONS"
```
Replica sets supply `rs0`; mongos summaries omit it. Encode usernames too if changed to characters requiring escaping.
3. Persist `MONGODB_URI` and, if stored separately, the application password as **[Zerops](https://zerops.io/) application-service secret environment variables**, not source/pipeline variables. Ship the public multiline `ca.pem` in the app’s deployment/configuration to a readable fixed path, such as `/var/www/ca.pem`. Restart/redeploy the application to load updated secrets. A shell `export` affects only a test shell; it is not persistent service configuration.
4. Run clients **inside the database namespace’s project network**, or use a deliberately validated public access plan:
```sh
mongosh "$MONGODB_URI" --tls --tlsCAFile /var/www/ca.pem \
--eval 'db.orders.insertOne({example: "tls"}); db.orders.findOne()'
```
```python
import os
from pymongo import MongoClient
with MongoClient(os.environ["MONGODB_URI"], tlsCAFile="/var/www/ca.pem") as client:
client.admin.command("ping")
print(client.appdb.orders.find_one())
```
**UI alternative:** **Users → Add user**, save the generated password, then use ordinary **Overview** hosts/replica-set. Reveal only to **Download ca.pem** / **Copy CA**; if clipboard fails, download or select the visible PEM preserving line breaks. **How to connect (MongoDB)** snippets start from an administrative URI: replace identity, password and `authSource` with the scoped user’s values, then perform the same secret/CA handoff above.
## Public exposure and discovery
[Section titled “Public exposure and discovery”](#public-exposure-and-discovery)
Private-only is default. Public MongoDB TCP needs project IPv6 (default `IPV6`) or dedicated IPv4, not shared IPv4/HTTP subdomains; clients need the corresponding connectivity. Patch exposure, for example:
```yaml
spec:
exposure:
public: true
ipType: IPV6
allowlist: ["2001:db8:1234::/64"]
```
Replace this documentation CIDR with authorized client ranges; follow the shared [public exposure allowlist and acknowledgement rules](/docs/security/#public-exposure-allowlists). The UI clears confirmation when settings change.
Managed listeners explicitly enable `bindIpAll` and IPv6; neither is overridable. Members share the project public IP with distinct mapped ports. Disabling exposure removes owned routes even when no writable primary exists or replicas are lagging, after pending platform operations and ownership checks; it does not mark the cluster Ready or acknowledge the new generation. Publishing routes still requires native health. Routing synchronization is project-wide; Ready/`isSynced` is not proof of external reachability. Keep `requireTLS`, narrow CIDRs and verify allowed/denied access; never publish agent port 9443.
**Warning:** Public replica-set seeds still advertise private member names/ports. No automatic external DNS/SNI split-horizon solution is configured; DNS-to-shared-IP alone does not fix per-member port mappings. External clients must resolve/reach discovered endpoints with valid TLS. `directConnection=true` is diagnostic, not HA/failover. Treat `publicHosts` as routes, not validated external discovery. [Sharded](/docs/mongodb/sharding/) mongos endpoints do not require external discovery of shard members.
## Reading placement
[Section titled “Reading placement”](#reading-placement)
`clusters placement` fingerprints are best-effort. Co-location warnings group matching **known** fingerprints; `unknown` means placement is unobserved, not co-location. Distinct fingerprints do not certify host/zone separation; see [platform limitations](/docs/limitations/).
# Backups and PITR
> Use PBM for consistent MongoDB snapshots, point-in-time recovery and clone recovery.
Use [PBM](https://docs.percona.com/percona-backup-mongodb/) for consistent [MongoDB](https://www.mongodb.com/) snapshots, point-in-time recovery and clone recovery.
## At a glance
[Section titled “At a glance”](#at-a-glance)
| Capability | Requirement / behavior |
| ---------- | ---------------------------------------------------------------------------------------------------------- |
| `logical` | PBM logical snapshot; CLI backup-type default |
| `physical` | PSMDB physical snapshot; recovery replaces the data directory |
| PITR | Successful base snapshot plus continuous PBM-observed oplog slices |
| Transfer | Members ↔ S3 directly, never through the management relay |
| Isolation | Effective storage prefix includes a stable cluster UID; preserve each snapshot’s recorded namespace/prefix |
Register and verify [S3](https://aws.amazon.com/s3/) storage using [Backups and storage](/docs/backups/), which owns common inventory, schedule/count-retention and retained-backup flows. Verification from the control plane does not prove every member can reach S3. Preserve state, original encryption key and storage credentials; object storage alone cannot reconstruct recovery ownership.
## On-demand backups and schedules
[Section titled “On-demand backups and schedules”](#on-demand-backups-and-schedules)
```sh
karactl -n default backups create demo beforechange --type logical --storage archive
karactl -n default backups create demo physicalbase --type physical --storage archive
karactl -n default backups list demo
karactl -n default backups get demo beforechange
karactl -n default operations cluster demo
```
UI: **Cluster → Backups → Backup now**; storage is required. Follow the operation and inventory to **Succeeded**, not just request acceptance/[PBM](https://docs.percona.com/percona-backup-mongodb/) initiation. Inventory includes PBM name, type/storage, times, size/error where available, and `location`: the snapshot’s object prefix `//` within its recorded storage bucket; `trigger` is `manual`, `schedule` or `restore` (automatic post-restore base). The prefix identifies snapshot data, not sibling `/.pbm.json` metadata or shared PITR slices. It is populated when the native snapshot name is known, including inventory-discovered logical, physical and incremental snapshots and PITR base snapshots. Legacy empty locations are filled when the snapshot is next observed by inventory using its recorded prefix (or the prefix adopted by inventory if none was recorded); unobserved snapshots remain unchanged. Existing nonempty locations are preserved. Deletion continues to use the recorded PBM prefix/name, not `location`.
Manual requests require a formed active cluster with current generation observed. Creating/suspended/deleting/unconverged clusters return **409** with phase/generation context, even if an older Ready phase remains visible. Failed/expired/cancelled/retired operations mark unfinished backup inventory Failed. Never-started or Failed records without a started PBM snapshot can be removed immediately when no matching active operation exists; started snapshots need provider cleanup before name reuse. Missing storage returns **404** (`backupStorage`); duplicate name **409** `already_exists`; unsupported type **400** listing valid types.
Both PBM types can be scheduled. Create-time shorthand:
```sh
# Add to clusters create:
--backup-storage archive --schedule 'nightly=0 2 * * *,type=logical,retain=7,storage=archive'
```
For edits use `spec.backup.schedules` in an API-shaped patch; see [schedule rules](/docs/backups/). PBM deletes excess successful scheduled snapshots durably. Base/oplog deletion can shorten PITR coverage; inspect the observed window after retention/storage changes.
## PITR
[Section titled “PITR”](#pitr)
1. Configure primary `spec.backup.storage` and complete a base backup.
2. Set `spec.backup.pitr.enabled: true`; `oplogSpanMinutes` is integer minutes, minimum 1, default 10.
3. Run `karactl -n default backups pitr-window demo`; choose a UTC RFC3339 timestamp inside a reported **continuous** window.
Future/outside/gap timestamps return **400**, `details.field: source.pitrTime`, reason `out_of_range`, with UTC `earliest`/`latest`; bounds do not imply intervening gaps are covered. Initialization/no coverage returns **409** `invalid_state`, including on the window endpoint. Coverage is rechecked during execution; permanently invalid targets fail terminally. Once a set is ready with [PBM](https://docs.percona.com/percona-backup-mongodb/) storage configured, convergence during scaling/upgrades/reloads does not itself hide established coverage; agent/storage failures still can.
Quota, slicing failure, history gaps, base deletion and storage changes affect recovery. After restore, PITR stays disabled until a fresh snapshot on **primary storage** succeeds. The required automatic base cannot be cancelled. If it fails/disappears, at most three replacement attempts are queued with warnings without blocking cluster convergence. After exhaustion, take a successful manual primary-storage backup; a snapshot queued after restore also satisfies the gate. Check the new timeline before trusting recovery.
## In-place restore
[Section titled “In-place restore”](#in-place-restore)
**Warning:** Stop application writers and plan downtime. Restore overwrites data and may restore application users/roles; preserve a current recovery copy first.
```sh
karactl -n default restores create demo recoverbase --backup beforechange --yes
# Replace the illustrative time with an observed covered point:
karactl -n default restores create demo recovertime --time 2026-01-15T12:00:00Z --yes
karactl -n default restores get demo recoverbase
```
Use exactly one `source.backup` / `source.pitrTime`; both/neither return **400**, field `source`, reason `conflict`. Missing backup returns **404** with source context; duplicate restore name **409** `already_exists`, before enqueueing. Source defaults to target; require matching engine driver/topology/engine major and a successful compatible snapshot, particularly for physical recovery. Missing engine driver metadata denotes legacy [PSMDB](https://docs.percona.com/percona-server-for-mongodb/), not a engine driver-neutral backup.
Logical restore is **not full rollback**: replica-set restore replaces backed-up namespaces but leaves extra collections/databases untouched. Sharded logical restore drops every database present in the backup, removing later collections within those databases; databases absent from the backup remain. For an exact copy, quiesce/drop extras deliberately, use physical recovery, or restore into a new cluster.
Karakoram serializes disruptive work, disables PITR and holds sharded routers/balancer. Physical restore suppresses automatic mongod/[PBM](https://docs.percona.com/percona-backup-mongodb/) restart; mongod may be offline and PBM may exit successfully. Progress uses PBM storage configuration while [MongoDB](https://www.mongodb.com/) is unavailable. Recovery restarts processes, clears maintenance, resets system users, resyncs target storage and restores router/balancer state. Logical restore does not use this offline lifecycle. Phase is **Restoring** with checkpoint message, then health observation on success or Degraded/error on terminal failure.
Sharded restore captures the original balancer state durably before disabling it. A balancer that was intentionally disabled remains disabled; cleanup only re-enables one that this operation observed enabled. Older interrupted checkpoints without a captured state leave the balancer disabled rather than guessing operator intent; inspect and enable it explicitly if appropriate.
Validate all members/agents, application authentication, collection data, balancer, inventory and fresh base before reopening writers. PBM `done` alone is insufficient.
## Clone and cross-namespace recovery
[Section titled “Clone and cross-namespace recovery”](#clone-and-cross-namespace-recovery)
```sh
karactl -n default restores create demo cloneproof --backup beforechange --into-new democopy
# PITR clone: substitute a real covered timestamp
karactl -n default restores create demo clonepoint --time 2026-01-15T12:00:00Z --into-new pointcopy
karactl -n default restores list democopy
```
Clone leaves the source running. Alternatively create a compatible cluster with immutable `spec.dataSource` containing `cluster`, exactly one `backup`/`pitrTime`, and optional source `namespace`; see [generic clone flow](/docs/backups/). Destination initialization queues a deterministic restore. After initial recovery, dataSource is provenance only; later changes do not need the source. It is not continuous replication.
[PBM](https://docs.percona.com/percona-backup-mongodb/) restores source application users. Logical recovery preserves its performing backup account, then uses it to reset destination system users. Physical snapshots retain an independently encrypted snapshot-time `userAdmin` recovery identity. Legacy physical recovery can protect a still-existing source’s unchanged snapshot-era userAdmin; unavailable/changed credentials block recovery before mutation. Modern logical/physical snapshots can survive source deletion; missing provenance/topology/version or unrecoverable physical identity returns **409**. Never manually back up a clone while PBM still points at the source prefix.
PITR clone still requires a live source and observed window. A deleted source needs an explicit complete create file rather than `--into-new` (which copies a live spec), or restore into an existing compatible target with `--source-cluster`.
For cross-namespace recovery, global `-n` selects target, `--source-namespace` source; source needs `backups/read` **and** `backups/export`, plus target mutation permission. PBM temporarily uses the recorded source-namespace storage; target members must reach it directly, then return to their isolated configuration. Convenience clone omits copied cross-namespace backup settings: configure target-local storage afterward. See [cross-namespace flow](/docs/backups/).
## PBM recovery and cancellation
[Section titled “PBM recovery and cancellation”](#pbm-recovery-and-cancellation)
[Operation limits](/docs/clusters/#operations-and-status) apply. Restore compensation has a separate six-hour generic failure-cleanup window (with the shorter fail-closed observation rules below); it is not rollback. Backup polling exhaustion and legacy failed-start cleanup enter **read-only recovery with a two-minute observed-execution budget**, persisted across CP restarts. Legacy cleanup checkpoints receive one fresh observation window on upgrade. Each engine driver attempt is capped at two minutes; with the default one-minute worker backoff, failure is published within five minutes of entering this recovery window (excluding CP downtime, queue scheduling and unavailable state storage).
Before initiating a backup, Karakoram waits for [PBM](https://docs.percona.com/percona-backup-mongodb/) status to report at least one healthy primary/secondary from each participating replica set, including config servers. A Ready [MongoDB](https://www.mongodb.com/) primary alone does not prove PBM storage readiness: a first configuration/resync can finish before PBM refreshes its health record, causing an empty native nomination. The operation reports `ready` progress while waiting and fails without initiation after a persisted two-minute observed-execution readiness budget. Status outages or malformed/missing readiness data do not authorize initiation.
A failed, unknown, or malformed initiation response without a native name now fails the requested record and operation immediately with **initiation outcome unconfirmed** and operator guidance, rather than repeatedly returning the immutable failed receipt for the ordinary retry budget. This does not assert that PBM rejected or stopped the command. No automatic re-initiation, cancellation, deletion, worker restart, or reconfiguration is issued; inspect native work before retrying. The durable `done` checkpoint prevents replay after CP restart.
Backup recovery never replays a failed/unknown start receipt and never cancels, deletes, restarts workers or reconfigures a possibly-running backup. Recovery may claim a snapshot only when the native name came from **this operation’s actual successful structured initiation result**, not merely from a matching timestamp, type, storage or serialized queue. For example, an accepted/named backup whose later polling exhausted retries can still complete truthfully after native observation. Pinned PBM 2.16 JSON mode emits its initiation result only after its acceptance wait; there is no supported partial progress-name capture. A CLI interrupted before that JSON result leaves no attributable name. Full-cluster scope is independently checked: selective `namespaces` metadata cannot satisfy a full backup request.
For an operation-attributed full backup, positive native success completes both record and operation Succeeded; native failure completes both Failed. Without an operation-specific identity, the original record and operation fail closed with candidate artifact names and operator guidance. Ordinary inventory reconciliation then adopts an unclaimed native snapshot under its **own separate native-name record**; it must not claim that a manual backup fulfilled the failed request. Native polling also expires Failed with operator guidance if the outcome remains unconfirmed within the observation budget. Repair/verify native work before submitting another backup. An unnamed initiation receipt cannot gain ownership proof from a CP or member restart. Cluster deletion’s verified-prefix sweep (below) covers unlisted artifacts or reports `BackupArtifactsRetained`.
After ordinary restore timeout, cleanup observes an already-started PBM restore to terminal state before restarting databases. PBM `done` can leave Restore Succeeded while Operation records the limit failure; fresh-base/PITR gating still applies. Cleanup expiry does not prove PBM stopped or authorize starting database processes.
For a logical restore, an orphaned native `running` status is not progress. Karakoram checkpoints the native command/lock owner’s supervisor session and successful child-launch counter while observing the restore. A later, higher launch counter **in the same supervisor session** proves the previous child was reaped before replacement. If that restore’s lock heartbeat and native progress transition are also at least **two minutes** old, Restore/Operation fail with the native name, launch evidence, possible OOM/restart diagnosis and recovery action. Create-from-backup also fails rather than presenting an empty healthy target as Ready. This is a genuine lost-worker abort, not just an expired create tracker. No restore is re-initiated, and holds remain because other native workers may still need inspection.
Wall-clock process birth ordering is **never** replacement proof, even on a single host: clocks can skew or step relative to MongoDB cluster time. A paused live process keeps its launch counter and cannot be classified dead. First observation, legacy agents without launch identity, a different supervisor session, missing lock ownership and slow progress are ambiguous. They retain ordinary bounded operation deadlines, not inferred native failure. Deploying CP alone cannot retroactively prove an old unobserved worker died; do not restart a member merely to manufacture evidence. Physical restores are excluded because their process restarts are intentional. With an established comparable identity and reachable observers, classification occurs on the first poll after the two-minute stale window. Observer-only authentication/transport failures are diagnostics, not restore retry-budget failures; normal native polling remains authoritative. Observer authentication uses the active restore-scoped PITR identity when present.
Ambiguity is visible, not silent: if both the named restore’s native progress and lock heartbeat are stale for the two-minute window, Karakoram publishes `RestoreWorker / Unknown / RestoreWorkerUnconfirmed` on the cluster and a `RestoreWorkerUnconfirmed` event associated with the Restore operation and its active create operation. The Restore operation message explains that the worker cannot be confirmed alive, native restoration is not re-initiated, and the operator may inspect native logs/memory or delete the target. This warning also works after a CP-only upgrade on legacy agents with no launch identity. It is **not terminal failure or abort evidence**. Fresh heartbeat/native progress or a classified terminal outcome clears the warning; repeated stale polls do not emit duplicate transition events.
PBM’s supervised Go runtime also receives an explicit **soft** memory target. For each participating mongod, the control plane derives `GOMEMLIMIT` from its **applied** minimum allocation (`minRam`), minus its **active** WiredTiger cache, 256 MiB for mongod non-cache memory, 64 MiB for agents/CLI and 64 MiB of allocation headroom. A missing active-cache field uses the agent’s historical 0.25 GiB cache default, never a recomputation from pending desired `maxRam`. With normally applied engine driver cache defaults, fixed 1 GiB therefore gets a 384 MiB PBM target; fixed 2/4/8 GiB members get 896/1920/3968 MiB. An unapplied desired RAM/cache increase cannot inflate the current target. Go’s limit controls heap/GC pacing, **not a hard process-RSS ceiling**: live buffers, non-Go memory, native engine growth and GC CPU pressure still matter. `GOGC` is not overridden.
Applied sizing travels with every replica/shard/config-server mongod in the PBM environment. Legacy members without persisted applied RAM are checked against the provider’s authoritative current service bounds, not desired spec. If those bounds are unavailable or unknown, new logical recovery is refused with `PBMRestoreMemoryBudgetUnknown`; no 1 GiB allocation is invented. Existing native work keeps its nonterminal legacy diagnostic instead of failing because an optional sizing observation is unavailable.
New logical restore/clone recovery validates the **minimum applied PBM budget across every participating mongod** against a **320 MiB floor** before source configuration or native restore initiation. Below it, the operation fails with `PBMRestoreMemoryBudgetTooSmall`, naming the limiting member/component and budget. Correct applied `minRam`/`maxRam` before recovery. This floor reserves room for live archive/BSON/bulk/driver state; it is not a claim that every possible workload fits at the boundary. Physical restores are not subject to this logical admission rule. Already-named or uncertain legacy native work is not retroactively rejected or re-initiated by a new sizing policy.
Logical concurrency uses that same minimum applied member budget, allowing 256 MiB of BSON reader queue per collection and 128 MiB per insertion worker. Two collections are used only if two one-worker budgets fit, then workers are capped at ten per collection. Fixed 1 GiB uses one/one, 2 GiB two/one, 4 GiB two/five and 8 GiB two/ten. Batch size remains 500: pinned mongo-tools flushes below its approximately 48 MB wire byte limit. These concurrency settings alone do not account for Go GC slack. The 64 MiB download-buffer setting applies to PBM’s configurable downloader; pinned 2.16 logical namespace reads instead use the ordinary one-worker 8 MiB [S3](https://aws.amazon.com/s3/) arena, plus archive composition/decompression/BSON buffers.
New agent binaries apply the target **only on the next actual PBM process launch**, reporting it as `processes.pbm-agent.goMemoryLimitBytes` in agent status. A config pull or budget change never restarts a possibly-mutating PBM restore. Existing agents and already-running children may lack an applied target; Restore operation diagnostics then report `PBMMemoryBudgetUnknown` at the first restore-time observation instead of claiming enforcement. Deploy new agents on fresh recovery targets to exercise the bounded-runtime path; do not restart a held restore to retrofit a limit.
Small members trade parallelism/GC CPU for headroom, not data correctness. Inspect allocation OOM events, observed PBM launch limit and native progress; choose larger RAM before recovery when the workload requires it. Increasing RAM or restarting CP cannot resume an already-killed native restore. Keep writers quiesced and use explicit fresh-target recovery below, never replay the failed receipt.
A published native restore failure remains **Restore Failed / Operation Failed** after compensating cleanup. The error retains the native restore name and original PBM error. Partial data or authentication changes cannot be excluded: PBM 2.16 does not provide affirmative no-mutation proof for a logical restore that fails during primary loss, even if the last sampled status was `starting` or mongorestore reports zero document counters. Karakoram does not automatically re-initiate it, change its native identity or issue another request identity. `Ready` describes database health, not successful restore or unchanged data. A native failure with uncertain consistency leaves the cluster **Degraded / operator action required**, even after processes and members become healthy; runtime liveness alone does not clear that durable safety state.
The operator must keep application writers quiesced, verify that native restore workers have stopped, and assess collection data and application/system authentication before choosing recovery. Explicitly authorize a **new restore with a new restore name** (duplicate names are rejected), or recover a verified backup into a fresh compatible cluster. A new request is a new destructive execution, not a resume or automatic retry. Only a later successful, verified restore under a fresh restore name completes recovery and clears the consistency-uncertain safety state. Preserve the failed Restore, Operation and receipts as evidence; compensation is not rollback.
`karactl operations cancel ID` supports ordinary backups and restores **before data mutation**, including source-inventory waits; initiating/applying restore cannot be cancelled. Required post-restore base cancellation is rejected; legacy missing/unreadable trigger inventory returns **409** until safety can be established. Cancelled backup/restore inventory is Failed with the cancellation summary. Wait for cooperative cleanup; cancellation is not undo.
PBM initiation pins a command member and durable request identity. Observation and idempotent configuration/resync commands can use surviving eligible members (config-server members for sharded clusters), including offline physical-restore progress. Once an attributable native operation name is durably recorded, backup/restore resumes polling without replaying initiation, even if the receipt-owning member is unavailable.
An interrupted receipt **without** a recorded native name reports `PBMReceiptUnknown`, naming the operation request and member. Karakoram cannot safely attribute an unrelated timestamp-matching native operation to that request or prove that an absent inventory entry means initiation never happened. Inspect `pbm list --out=json`, `pbm list --restore --out=json`, matching `describe-backup` / `describe-restore`, PBM logs and the persisted Operation using a surviving eligible member and the recorded storage prefix. Preserve receipts, storage and physical-restore holds; never erase receipts, change request identity, switch initiation nodes or launch a duplicate restore. If provenance cannot be established, recover a verified backup into a new cluster and keep the ambiguous target fenced. There is no automatic destructive retry or supported receipt-editing shortcut.
Missing restore metadata is tolerated as **restore starting** only for one minute after successful acceptance, with a persisted deadline; later disappearance and unrelated command failures are errors.
Logical named restores also consult `.pbm.restore/` in their recorded source storage. An empty `pbm list --restore`, historical non-primary skip, listener log, or reconstructed process birth timestamp never proves that an old command cannot execute later.
PBM’s storage-backed `describe-restore --config` can return exit success with only a name/type and an empty status even for an empty restore prefix. That synthetic shell is not native progress or completion. For a never-observed logical restore it must still pass the complete absence and active-fence checks below; a stored nonempty status continues through native-outcome polling.
For a logical replica-set restore that has **never** published metadata, a bounded absence path requires exact not-found from every eligible member, complete storage absence, the unique native command at least one minute old, and no conflicting operation in either `pbmLock` or `pbmLockOp`. All operation types (backup, PITR, replay, resync, restore and unknown types) block admission; only this restore’s own stale restore lock is exempt. These observations must remain complete across a persisted further one-minute window. Admission is refreshed immediately before **each** member restart, so work that appears after the initial window or between restarts prevents the next restart. The product then **actively restarts PBM agents on every member**, using stable identified supervisor requests—not another destructive restore command. Each agent must confirm its old child exited through the supervisor’s wait and a replacement started. Immediately before launch and on acknowledgment, the member clock must be at least the original command timestamp plus **30 seconds**; a clock behind that guard keeps the old process stopped. Identified durable receipts prevent a lost response from repeating the restart. Every complete final observation revalidates all receipts for the current supervisor session, process generation, capability and clock before accepting metadata/storage absence. No per-member verification cursor survives a failed or interrupted observation.
After all members confirm replacement, Karakoram rechecks every member’s metadata, all PBM operation locks and complete recorded storage listing. Only continued absence permits compensating recovery to restore target configuration and release holds. Restore and Operation then become **Failed** with `RestoreNeverStarted` and “data unchanged; re-run the restore”; ordinary fresh health observation can return the cluster to Ready. Karakoram never resubmits the destructive command or reports Succeeded from absence.
If native metadata appears during restart admission or final absence checking, Karakoram consumes that observation through normal native-outcome polling instead of continuing absence fencing. Native `done` or explicit failure enters the corresponding existing recovery path; running, unclassified or subsequently missing metadata keeps the restore holds in place. No further agent restarts or destructive restore initiation are allowed from that observation, and it cannot be converted into `RestoreNeverStarted`.
Any unconfirmed exit/start, changed supervisor/process generation, clock guard failure, transport error or unknown restart outcome keeps the restore fenced. Unknown restart state is durable; later healthy observations cannot revive it as a successful fence. Local stopped intent prevents config pulls or automatic backoff from bypassing a rejected launch. Physical/sharded restores, previously observed native restores, active locks and partial/unavailable metadata/storage observations do not use this terminal-absence path.
The `pbmRestoreFence` capability and explicit restart acknowledgment require updated member agents as well as the control plane. Legacy agents remain fenced under the default 20-failure or six-hour restore-execution limit, followed by a separate six-hour failure-cleanup bound. Expiry does not prove PBM stopped, establish unchanged data, or authorize destructive retry. Every unsuccessful attempt invalidates its absence window, including dependency resolution/restore validation before engine driver entry and inventory/storage/runtime-recovery failures inside the engine driver.
## Delete backup data and recover pending deletion
[Section titled “Delete backup data and recover pending deletion”](#delete-backup-data-and-recover-pending-deletion)
**Warning:** [PBM](https://docs.percona.com/percona-backup-mongodb/) deletion is irreversible and can shorten PITR coverage. Keep registered storage/location/credentials valid until cleanup completes.
`karactl backups delete demo beforechange --yes` on a live cluster waits asynchronously for PBM inventory absence. Active matching work returns **409** naming backup/state; cancel it first. After source deletion, snapshot DELETE is synchronous **204**, agent-free, and removes only recorded `//...` and `/.pbm.json`, then identity/record. Other snapshots, PITR and namespace markers remain. Partial failure leaves Deleting; repeat DELETE safely, including after restart. Already absent returns 204. Never-started retained Failed records need no storage; unsafe provenance returns 409.
Cluster deletion without backup retention first removes runtime services and (unless retained) volumes. It never authenticates to [MongoDB](https://www.mongodb.com/)/PBM: broken managed users or unreachable members do not block deletion. Once every database runtime is authoritatively absent, the control plane uses its registered backup-store binding to sweep only the verified cluster-incarnation prefix, including unlisted snapshots, PITR and markers. Agent unreachability alone never authorizes a sweep. Recorded backup prefixes must match the binding; earlier-incarnation retained snapshots remain independent. `--keep-backups` skips all artifact and inventory deletion. Active restore/clone dependency fences still return **409** before deletion is accepted.
Object cleanup has a durable **two-minute total window**, starting after runtime/volume retirement. A storage failure, unsafe/missing ownership or exhausted window emits **`BackupArtifactsRetained`**, identifying the storage/bucket/exact prefix when known and manual removal guidance; deletion still completes and affected backup inventory remains available under the original name/namespace. Inspect events as well as terminal success: success means cluster removal, not a promise that every remote artifact was removed. Restore the original storage binding and remove only the recorded incarnation prefix with object-store tooling; never remove a source/other incarnation prefix. Then remove retained backup records through the API. Suspended clusters need not resume for verified-prefix cleanup. Resume clears cleanup checkpoints and recreates PBM initialization markers even with unchanged backup settings.
Version-aware cleanup uses `s3:ListBucketVersions` / `s3:DeleteObjectVersion` to remove versions/markers. Unavailable/denied version listing falls back to ordinary visible-key deletion: noncurrent versions require separate credentials/lifecycle cleanup. Successful listing followed by denied version deletion remains an error. Empty-namespace confirmation uses that listing mode; large namespaces progress over bounded passes. If storage prefix changed, restore the original prefix before retrying.
For pending cluster deletion, inspect same-namespace cluster status, operations and events. Repair the named [S3](https://aws.amazon.com/s3/) bucket/prefix/write-delete access, [namespace relay](/docs/namespaces/#relay-health), or platform process; unformed clusters may never have had usable PBM. Observe the existing operation; do not repeatedly DELETE, erase state/volumes or recreate the name. Retry only failed/unaccepted deletion with the **same intentional retention choices**, following [Delete or suspend](/docs/clusters/#delete-or-suspend). Check retained inventory and [orphan volumes](/docs/operations/#orphaned-platform-volumes) separately; keep namespace management available until cleanup finishes.
# MongoDB operations
> Scale, upgrade and diagnose managed PSMDB members and PBM processes.
Scale, upgrade and diagnose managed [PSMDB](https://docs.percona.com/percona-server-for-mongodb/) members and [PBM](https://docs.percona.com/percona-backup-mongodb/) processes.
## Resource scaling
[Section titled “Resource scaling”](#resource-scaling)
Use [Clusters and operations](/docs/clusters/) for desired bounds versus observed allocation, disk growth, platform cooldowns and generic restart semantics; use [replica sets](/docs/mongodb/replica-sets/) for membership and [sharding](/docs/mongodb/sharding/) for component counts.
```sh
karactl -n default clusters scale demo --min-cpu 1 --max-cpu 2 \
--min-ram 2 --max-ram 2 --cpu-mode SHARED --wait
karactl -n default clusters scale demo --min-disk 5 --max-disk 20 --wait
```
RAM changes alter managed WiredTiger cache (`max(0.25, 0.5 × (maxRam − 0.5))` GiB) and require controlled reload. Minimum-sized members still need PBM/container headroom. Sharded engine bounds apply to every shard; configServer/router resources are separately selectable. Do not independently edit platform autoscaling or restart database runtimes in parallel during reconciliation.
In **Topology → Review & apply**, inspect **What will change** and **Merge patch to be sent**: member sync/promotion/removal, router capacity, added shards and resource reloads differ. CPU-only changes do not move data but may restart a member; this is not a no-downtime promise.
## Version catalog and paths
[Section titled “Version catalog and paths”](#version-catalog-and-paths)
[Overview](/docs/mongodb/) owns exact versions/defaults. Query `karactl providers get psmdb -o yaml` for deployed `upgradePaths`. Only forward same-major or adjacent-major changes are allowed; no downgrade/skipped major. Adjacent-major upgrades require observed FCV equal to the old major. Retaining FCV does not make binary downgrade supported.
### Required pre-upgrade backups
[Section titled “Required pre-upgrade backups”](#required-pre-upgrade-backups)
For any advertised path with `requiresBackup: true`, PUT/PATCH version changes require a **Succeeded**, timestamped backup from the same cluster/namespace, completed within **24 hours** and not before cluster creation. When observed/current desired version and successful convergence history are known, completion must also follow that version’s latest successful create/upgrade convergence; unrelated resource edits do not reset the cutoff. Pending/Running/Failed/Cancelled backups do not qualify.
Missing prerequisites return **409 conflict**, `details.resource: cluster`, cluster name, reason `invalid_state`, naming path/remedy. Complete a [backup](/docs/mongodb/backups/), then retry within 24 hours; no bypass exists. UI backup age/button state is not proof of eligibility. The current catalog does not mark any path `requiresBackup`; nevertheless take and test a recovery point before every upgrade.
## Rolling binary upgrade
[Section titled “Rolling binary upgrade”](#rolling-binary-upgrade)
1. Complete a backup, verify storage/recovery and target application/driver compatibility.
2. Check healthy replica sets, caught-up secondaries, connected agents and source-major FCV; finish competing restore/replacement work.
3. Submit an offered path, for example from healthy 7.0 binaries/FCV:
```sh
karactl -n default clusters upgrade demo --version 8.0.32-14 --wait
karactl -n default clusters members demo
karactl -n default clusters status demo
```
Secondaries upgrade first, then primary stepdown/former-primary deployment, waiting for exact pinned binary and replication convergence per member. Sharded order is config servers, shards, routers; prior balancer state is saved/stopped/restored. Lost quorum/unhealthy members pause progression.
**Warning:** Singletons are unavailable during restart/upgrade. Multi-member rolling order still permits brief write failures; configure appropriate driver retries/read/write concerns and plan maintenance.
**Upgrades** shows observed **Running version**, possibly mixed during rollout, separately from **Desired version**. Missing observations mean **not yet observed**, not deployment success.
[Zerops](https://zerops.io/) exact-version deployment retries have a hard budget. `DeploymentRetriesExhausted` records the blocked member and stops submissions for that target while other members remain observable and can finish provisioning. An unstarted member is not treated as bootstrapped or upgraded, and existing replication/quorum fences remain in effect. Correct the platform/runtime failure, then run `karactl --namespace NS clusters retry-deployments NAME` to re-arm.
## Explicit FCV promotion
[Section titled “Explicit FCV promotion”](#explicit-fcv-promotion)
Binary upgrades do not raise featureCompatibilityVersion automatically. Observe application compatibility with old FCV, then use **Upgrades → FCV** or:
```sh
karactl -n default clusters raise-fcv demo --version 8.0 --wait --yes
```
Use major form, not `8.0.32-14`. Wrong/non-deployed major returns **400** with field guidance before enqueueing; unfinished binary convergence returns **409** `invalid_state`. Every relevant member must report exact desired package version. Pre-acceptance agent/relay observation failure returns **503 unavailable** with retry hint and no operation. After acceptance, transient observation errors retry within bounded limits; genuine runtime validation fails with its reason. FCV uses confirmation for 7.0+ and enables on-disk features Karakoram cannot demote.
For stuck/wrong requests, inspect `operations cluster demo`, `operations get ID`, members/status/FCV and events in the database namespace. `karactl operations cancel ID` (API `POST /api/v1/operations/{id}/cancel`) is safe only **before `apply-fcv`**. Acceptance is 202, not completed cancellation; wait to terminal. A 409 `invalid_state` means inspect the new checkpoint, not submit competing work. Cancellation cannot undo applied FCV.
Once terminal and healthy, re-read observed FCV; promote only if still needed and all binaries match. Never edit operation records or change desired binaries to justify a wrong FCV target. Preserve diagnostics and use [MongoDB](https://www.mongodb.com/)’s supported recovery if native FCV partially failed.
## Rolling restart and overrides
[Section titled “Rolling restart and overrides”](#rolling-restart-and-overrides)
`karactl -n default clusters restart demo --wait` checkpoints members, checks quorum/health and processes secondaries before primary. It restarts database processes, not the agent executable.
Use `karactl -n default clusters upgrade-agents demo --wait` to deploy the current control-plane executable to the **same runtimes and volumes**, without changing MongoDB versions. Config servers precede shards and routers; each replica set rolls secondaries before its primary, with safe-restart, step-down and replication catch-up gates. Members already reporting the target build are skipped unless `--include-current` is set. Completed members are journaled and never repeated after a control-plane restart. Cancellation is safe between members, not during an active member change. If the control plane changes build during a rollout, the operation fails before another deployment; re-run it to freeze the new target.
`spec.maintenance` windows gate new engine upgrades, rolling resources/configuration changes and scale-in, plus automatic database leaf renewal while more than seven days remain. A member already restarting finishes rejoining even if the window closes. Creation/scale-out, healing, agent transport certificates and database leaves within seven days of expiry are exempt. Imperative restart, TLS rotation, FCV promotion, replacement and agent upgrades wait before claim; `--ignore-maintenance-window` explicitly overrides supported actions. See [maintenance policy](/docs/operations/).
Use validated `engineConfig` YAML only for non-managed settings; see the [overview limits](/docs/mongodb/). Restart/replacement/rotation and Running desired-state/deletion work have no safe general abort; agent rollout cancellation is available only between members. [Operation semantics](/docs/clusters/#operations-and-status) distinguish cancellation from rollback.
## Processes, files and readiness
[Section titled “Processes, files and readiness”](#processes-files-and-readiness)
`mongod` serves data/config members; `mongos` serves routers; PBM placement is described in [sharding](/docs/mongodb/sharding/). Platform `karakoram agent readiness` probes **TCP 127.0.0.1:27017 only**, without credentials/environment; it does not validate users, replication or PBM. Authenticated engine driver observations determine [cluster health](/docs/mongodb/replica-sets/).
Ready requires an observed, writable primary in every replica set (including config servers and each shard), with data members in PRIMARY/SECONDARY state and within the managed replication-lag gate. Reachable secondary processes alone are not healthy write service. `Replication/` reports `NoWritablePrimary` when election/quorum cannot provide a verified primary. Restore retained voters, disk capacity, DNS and connectivity; Karakoram does not force replica-set reconfiguration or reinitialize a previously formed set. A missing last data copy requires recovery from a verified backup, not an empty replacement. A lost secondary may be safely reseeded only from a verified retained writable primary.
Every live mongos instance must acknowledge its configuration and perform an authenticated majority read of config-server metadata. One healthy logical DNS endpoint or a running mongos PID cannot hide a failed sibling or unavailable config-server quorum. `Router/` identifies failed instance observations.
Normal steady-state reconciliation reapplies the five reserved system-user credentials/roles and `pbmAnyAction`; it does not rewrite customer-created users or roles. If the retained `userAdmin` credential or privileges no longer authorize repair, `ManagedSystemUsers` reports `CredentialRepairRequired`. Restore that account’s managed credential and `userAdminAnyDatabase` access using a surviving authorized administrator; restarting MongoDB does not repair authentication drift.
Reconciliation bounds each replication observation (connect, writable-primary check and replication status together) to four seconds. Managed-user connection checks, native `rolesInfo`/`usersInfo` reads, and each router instance status read also have independent four-second observation budgets. An expired read fails closed and identifies the read and connectivity/responsiveness checks; user/role mutations retain their original administration context and are not replayed by the observation timeout.
Agents pin internal trust and pull assigned protected configuration. With usable cache they can run through control-plane outages; without it, first configuration retries every five seconds. Normal pull is every 60 seconds, with immediate reload available to the controller. Activated member snapshots prevent simultaneous disruptive adoption; stateless router replacement loses cache.
| Under `DATA_DIR` | Recovery significance |
| ---------------------------------------------------------------- | ------------------------------------------------------ |
| `karakoram/config.json` | Atomic 0600 config cache with locally required secrets |
| `karakoram/keyfile` | Internal authentication key |
| `karakoram/mongodb.pem`, `ca.pem`, `mongod.conf` / `mongos.conf` | Protected rendered files; private material owner-only |
| `karakoram/controls.json` | Atomic 0600 durable maintenance/stopped-process holds |
| `karakoram/pbm-receipts/` | Durable 0600 PBM initiation outcomes |
| `db/` | Persistent mongod database files |
Supervisor owns start/stop/backoff and clean SIGTERM; do not launch duplicate mongod/PBM supervisors. Explicit stops survive pulls/restarts; only explicit maintenance-off or process start/restart releases corresponding holds. Never edit controls to undo [physical restore](/docs/mongodb/backups/).
## Logs
[Section titled “Logs”](#logs)
```sh
karactl clusters logs demo demom0 --process mongod --lines 100
karactl clusters logs demo demom0 --process pbm-agent --lines 100
# Router: actual member name, --process mongos
```
API: `GET /api/v1/clusters/{cluster}/members/{member}/logs?process=mongod&lines=100`. Default is mongod. Tails hold at most **1000 in-memory lines/process**, lost on agent restart; cursor/since are not a durable archive. Zerops runtime/deployment/application logs cover early startup, forwarded child output and supervisor diagnostics. Signed log URLs/tokens are credentials; never paste them into tickets. No arbitrary-file/exec endpoint exists.
## Monitoring vocabulary
[Section titled “Monitoring vocabulary”](#monitoring-vocabulary)
Enable `spec.monitoring.enabled` (CLI create default true), then **Monitoring** or `karactl clusters metrics demo --range 1h --step 30s -o json`. Agents authenticate with `clusterMonitor` and collect process/filesystem data. [Shared monitoring](/docs/monitoring/) owns scrape/history/API range semantics and authorization; MongoDB ships exporter/dashboard, not [PMM](https://www.percona.com/software/database-tools/percona-monitoring-and-management)/query analytics.
Every public metric has `namespace`, `cluster`, `member`; `set`/`role` are status properties, not guaranteed labels. Availability-dependent metrics may be absent, notably mongos oplog/WiredTiger values. Counter JSON series are reset-aware rates, not raw counters.
| Prometheus metric | Type/unit | JSON series / meaning |
| --------------------------------------- | -------------------------- | -------------------------------------------------------------------- |
| `karakoram_mongodb_up` | Gauge 0/1 | `up`: authenticated serverStatus success |
| `karakoram_mongodb_uptime_seconds` | Gauge seconds | `uptime` |
| `karakoram_opcounters_total` | Counter operations | `opcounters.total`: operations/s |
| `karakoram_mongodb_opcounters_total` | Counter; `operation` | `opcounters.byOperation`: insert/query/update/delete/getmore/command |
| `karakoram_connections_current` | Gauge count | `connections.current` |
| `karakoram_connections_available` | Gauge count | `connections.available` |
| `karakoram_repl_lag_seconds` | Gauge seconds | `repl.lagSeconds`: latest visible optime difference |
| `karakoram_repl_member_state` | Gauge state code | `repl.memberState`: MongoDB myState |
| `karakoram_oplog_window_seconds` | Gauge seconds | `oplog.windowSeconds`: local first/last span, not PITR coverage |
| `karakoram_cache_used_bytes` | Gauge bytes | `cache.usedBytes` |
| `karakoram_cache_max_bytes` | Gauge bytes | `cache.maxBytes` |
| `karakoram_process_cpu_seconds_total` | Counter CPU seconds | `cpu.percent`: rate ×100, JSON clamped 0–100 |
| `karakoram_ram_used_bytes` | Gauge bytes | `ram.usedBytes`: mongod/mongos RSS, excludes PBM/container overhead |
| `karakoram_ram_total_bytes` | Gauge bytes | `ram.totalBytes`: proc/cgroup available bound |
| `karakoram_disk_used_bytes` | Gauge bytes | `disk.usedBytes`: DATA_DIR filesystem |
| `karakoram_disk_total_bytes` | Gauge bytes | `disk.totalBytes`: DATA_DIR capacity |
| `karakoram_pbm_agent_up` | Gauge 0/1 | `pbm.up`: process running, not registration/backup health |
| `karakoram_mongodb_documents_total` | Counter; `operation` | `documents.total`: inserted/updated/deleted/returned |
| `karakoram_mongodb_network_bytes_total` | Counter bytes; `direction` | `network.bytes`: bytesIn/bytesOut |
[Prometheus](https://prometheus.io/) throughput: `rate(karakoram_opcounters_total[5m])`; never rate gauges. Raw CPU `rate(karakoram_process_cpu_seconds_total[5m]) * 100` can exceed clamped UI values on multicore work.
## Troubleshooting
[Section titled “Troubleshooting”](#troubleshooting)
| Symptom | Action |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Deployment/config stuck | Read operation checkpoint/events, generation and agent config/connectivity; inspect early Zerops logs |
| Empty charts | Check collection, agents, credentials/CA, range and enough counter samples; missing data is not healthy zero |
| Scrape 403 | Check `monitoring/read` on namespace/cluster and token/account status |
| Relay/config pull unavailable | Check [relay health](/docs/namespaces/#relay-health), public origin and namespace routing; retain CA validation |
| PBM up, no backups | Inspect PBM registration and backup records, not process-up alone. The local PBM URI must not set `directConnection=true`: agents on secondaries must discover the primary. Run one `pbm-agent` per `mongod`, none on `mongos`. |
| Offline during restore | Follow [PBM recovery](/docs/mongodb/backups/); do not release holds/start competing processes |
| Memory near bound | Include PBM/other processes; inspect Zerops allocation and runtime logs |
| Application connection failure | Check project DNS/network, saved password/authSource/roles, then cluster CA; do not bypass verification |
# Replica sets
> Create and maintain a managed PSMDB replica set named rs0.
Create and maintain a managed [PSMDB](https://docs.percona.com/percona-server-for-mongodb/) replica set named `rs0`.
## Create
[Section titled “Create”](#create)
### Web UI
[Section titled “Web UI”](#web-ui)
1. Select the database project’s [namespace](/docs/namespaces/), then **Clusters → Create cluster → [MongoDB](https://www.mongodb.com/)**.
2. Choose **replicaSet** and an exact [catalog version](/docs/mongodb/). Select **dev** only for disposable single-member use, or **prod** for three voting members.
3. Review member resources and total project capacity. Keep private exposure and `requireTLS`; configure [backup storage](/docs/backups/) if recovery is required.
4. Review and create. Follow the resulting operation and **Topology** until the desired generation is observed and active members are healthy.
### CLI alternative
[Section titled “CLI alternative”](#cli-alternative)
[Install and authenticate karactl](/docs/cli/), then select the namespace with `-n NAME` (omitted single-object namespace defaults to `default`):
```sh
karactl providers get psmdb
karactl -n default clusters create demo --provider psmdb \
--topology replicaSet --members 3 --version 8.0.32-14 \
--min-ram 1 --max-ram 1 --min-disk 5 --max-disk 5 \
--tls-mode requireTLS --wait
```
Use a version advertised by your installation. Explicit bounds avoid the CLI’s larger resource maxima. REST requires `spec.provider`; omission selects defaults only for optional fields, while explicit zero/unknown fields are rejected. See [API validation](/docs/api/) and [OpenAPI](https://github.com/KarakoramDev/Karakoram/blob/main/api/openapi.yaml).
## Observe health
[Section titled “Observe health”](#observe-health)
```sh
karactl -n default clusters get demo -o yaml
karactl -n default clusters members demo
karactl -n default clusters placement demo
karactl -n default operations cluster demo
karactl -n default events list --cluster demo
```
Ready means the desired `generation` is observed by `observedGeneration` **and** authenticated member observations show healthy replication. Check phase/message/conditions, role/state/lag, agent connectivity/configVersion and observed binary versions. A caught-up secondary is not merely a [Zerops](https://zerops.io/) ACTIVE service or an open TCP port. [Operation success alone is not a health guarantee](/docs/clusters/#operations-and-status).
`spec.version` is desired; `status.version` reflects observed binaries, with missing observations unknown. Leaving members remain visible until removed; the UI counts active members and leaving members separately. Condition transition timestamps change on status/reason transitions, not every poll.
Writable-primary readiness requires a freshly selected primary’s `hello` and fresh reachability of the set’s members, not just a secondary’s potentially stale replication view. A failed observation is retried once within the same bounded cycle; it never extends cached readiness. See the shared [health freshness and transient tolerance contract](/docs/clusters/#operations-and-status) for the 2-second read-probe, 5-second sweep and 15-second maximum sample-age bounds.
Use [application connection instructions](/docs/mongodb/access-and-tls/); private hosts belong to the namespace’s project, not necessarily the control-plane project.
## Add, remove and replace members
[Section titled “Add, remove and replace members”](#add-remove-and-replace-members)
```sh
karactl -n default clusters scale demo --members 5 --wait
karactl -n default clusters scale demo --members 3 --wait
karactl -n default clusters replace-member demo demom1 --wait --yes
```
Choose an actual inventory name for replacement. Added members join non-voting, priority zero; after initial sync, healthy replication and lag ≤10 seconds, they become voters. Scale-down removes high-index members, checks voting majorities and steps down a primary when required. Replacement uses a new hostname, so indexes may have gaps; it never creates an eighth voter at the seven-voter limit. Retired member volumes are removed: cluster-level retention is not per-member retention.
Unhealthy replication/quorum blocks disruptive work; repair it instead of repeatedly resubmitting. If replacement fails while removing the old member, ordinary convergence finishes removal before promoting the caught-up replacement; failure is not rollback. Inspect voting state before raising FCV. A singleton replacement necessarily interrupts availability.
For resource changes/restarts, see [MongoDB operations](/docs/mongodb/operations/). For deletion, suspension and resumption, use [Delete or suspend](/docs/clusters/#delete-or-suspend); for [PBM](https://docs.percona.com/percona-backup-mongodb/) cleanup, see [backups](/docs/mongodb/backups/).
# Sharding
> Provision config servers, shard replica sets and mongos routing for PSMDB.
Provision config servers, shard replica sets and mongos routing for [PSMDB](https://docs.percona.com/percona-server-for-mongodb/).
## At a glance
[Section titled “At a glance”](#at-a-glance)
| Component | Shape / omitted-field default |
| -------------- | --------------------------------------------------------------------------------------------------------- |
| Config servers | Replica set `cfg`; 1/3/5/7 members, default 3 |
| Shards | `rs0`, `rs1`, …; 1–128 shards, default 1; each set uses the engine component’s 1/3/5/7 members, default 3 |
| Routers | Stateless mongos service; 1–32 containers, default 1 |
These are validation bounds, not load-tested maxima or a project-capacity guarantee. Each mongod has its own runtime/volume; mongos has no durable database volume. All services remain in the selected namespace’s project. The [management relay](/docs/namespaces/) is not cross-project replication or application networking.
## Create
[Section titled “Create”](#create)
In **Create cluster → [MongoDB](https://www.mongodb.com/)**, choose **Sharded**, configure each component, and review the complete runtime/storage footprint. The **sharded** preset is described in the [overview](/docs/mongodb/).
CLI alternative:
```sh
karactl -n default clusters create analytics --provider psmdb \
--topology sharded --shards 2 --members 3 --config-servers 3 --routers 2 \
--version 8.0.32-14 --min-ram 1 --max-ram 1 \
--min-disk 5 --max-disk 5 --tls-mode requireTLS --wait
karactl -n default clusters members analytics
```
Use `-f` with a complete API-shaped create request for distinct `engine`, `configServer` and `router` resource blocks; see [OpenAPI Components](https://github.com/KarakoramDev/Karakoram/blob/main/api/openapi.yaml). Explicit zero, including `--shards 0`, is invalid. Engine driver defaults above differ from CLI convenience defaults, notably its two routers.
## Connect and shard collections
[Section titled “Connect and shard collections”](#connect-and-shard-collections)
Applications connect to **mongos only**, not config/shard members. The password-free connection summary contains router hosts and omits `replicaSet`; follow the [application-user TLS recipe](/docs/mongodb/access-and-tls/). Private connections need access to the database project.
Karakoram registers shard infrastructure; it does not select shard keys, shard collections or distribute application data automatically. Use an authorized MongoDB client through mongos to administer database/collection sharding. Adding a shard does not prove existing collections are sharded or balanced.
## Add shards and scale components
[Section titled “Add shards and scale components”](#add-shards-and-scale-components)
```sh
karactl -n default clusters scale analytics --shards 3 --wait
karactl -n default clusters scale analytics --routers 3 --wait
karactl -n default clusters scale analytics --component configServer \
--min-ram 2 --max-ram 2 --wait
karactl -n default clusters scale analytics --members 5 --wait
```
`--component` selects `engine` (default), `configServer` or `router`; `--config-servers` and `--routers` select their counts. Engine replicas/resources apply uniformly to every shard. New shard sets bootstrap and register before balancing. Existing membership changes use [initial-sync/quorum safeguards](/docs/mongodb/replica-sets/). Shard count cannot decrease; reducing members within a shard is different from removing it.
## Balancer
[Section titled “Balancer”](#balancer)
```sh
karactl -n default clusters balancer get analytics
karactl -n default clusters balancer set analytics --enabled=false
karactl -n default clusters balancer set analytics --enabled=true
```
Inspect balancer status separately from member health; manual disablement is not automatically an error. Binary upgrades save, stop and restore its prior state. Restores coordinate balancer/router holds; do not override those holds during an operation.
While restore holds routers stopped or they are restarting, balancer GET returns HTTP 503 `unavailable`; retry after router connectivity recovers. Database-user and database-role reads use the same availability contract. Authentication and non-transient command failures are not reported as temporary unavailability.
## Router readiness
[Section titled “Router readiness”](#router-readiness)
Router readiness uses an authenticated `clusterMonitor` read of `config.shards` through each mongos, with primary read preference and majority read concern. The metadata command has a two-second deadline within the three-second probe budget; the MongoDB driver derives the server-side time limit from that deadline. One config-server member is supported but has no config-server redundancy; majority is one in that topology. A running mongos or an open TCP port alone does not satisfy readiness. `Router/` / `ConfigMetadataUnavailable` reports a failed probe. Native command failures include the bounded MongoDB numeric code and code name in the condition and agent log. Client-side timeout, cancellation, network, TLS certificate/handshake and authentication failures use fixed diagnostic categories. Server-selection failures identify typed handshake causes from the observed servers when available, rather than hiding them behind the selection deadline. Raw native messages, connection URIs and credentials are not exposed by this diagnostic.
## Backups, access and failure domains
[Section titled “Backups, access and failure domains”](#backups-access-and-failure-domains)
[PBM](/docs/mongodb/backups/) runs on every config/shard mongod, never mongos; sharded PBM commands use `cfg`. Recovery is whole-cluster and disruptive, not a per-shard restore or topology conversion.
[Public exposure](/docs/mongodb/access-and-tls/#public-exposure-and-discovery) publishes mongos only. Multiple router containers share one service hostname, not independent per-container public discovery. Extra routers add process-level redundancy, not [host/rack separation](/docs/limitations/); their caches disappear on replacement. See [operations](/docs/mongodb/operations/) for component upgrades and resource changes.
# Monitoring and events
> Use built-in charts for short-term inspection, authenticated Prometheus scraping for external retention/alerting, and durable events/status for operational…
Use built-in charts for short-term inspection, authenticated [Prometheus](https://prometheus.io/) scraping for external retention/alerting, and durable events/status for operational progress.
## Built-in dashboard
[Section titled “Built-in dashboard”](#built-in-dashboard)
Enable `spec.monitoring.enabled` and open **Cluster → Monitoring**. The CLI create convenience path enables monitoring by default. The control plane scrapes agents every **30 seconds** and retains **24 hours** of persisted samples by default. Control-plane downtime interrupts collection/aggregation; this is not independent availability monitoring.
Collection uses at most eight concurrent member requests, each bounded by the scrape interval (at most 30 seconds). A slow member does not serialize every other target behind its timeout. Retention pruning runs independently of new ingestion, including when discovery or metric insertion fails. Storage failures are logged and retried; fresh in-memory usage/federation observations remain available even when their historical samples cannot be persisted. Pruning can still fail on a completely full or read-only filesystem: restore writable headroom rather than deleting the database or WAL.
Global retention lookups use a timestamp index. Each sweep deletes at most 4,096 expired rows in transactions of at most 256 rows, yielding the writer between batches and using a two-second admission/execution context. Saturated or deadline-limited sweeps wake one coalescing maintenance worker, which starts another bounded sweep after yielding for 25 ms; it does not wait another 30-second scrape interval. The worker stops and is joined on collector shutdown. This removes a fixed 4,096-rows-per-interval ceiling, but retention remains best-effort under writer contention, slow disks, or sustained ingestion beyond available delete throughput; 24 hours is not a hard storage-size guarantee. Slow writer admission in retention and token-usage maintenance is logged after two seconds with the operation name; it does not expose SQL parameters or credentials. A one-second token-usage admission timeout normally completes before that warning threshold. HTTP requests exceeding two seconds also emit a warning with request ID, method, matched route pattern and elapsed duration; raw URLs and query values are not logged. These completion diagnostics cannot replace a failure-time goroutine dump when a request never returns.
### Upgrading an existing metrics database
[Section titled “Upgrading an existing metrics database”](#upgrading-an-existing-metrics-database)
Migration `011_metrics_retention.sql` creates `(timestamp,id)` in one [SQLite](https://sqlite.org/) transaction with its migration marker. Building it scans and sorts existing metrics: startup time and temporary/WAL disk headroom depend on retained rows and storage speed, not just the current cluster count. The control plane opens its listeners only after all migrations finish; `/healthz` and the API are unavailable during the build. Do not bypass this by deleting state or reporting healthy before migration completes.
Before a large-table upgrade, preserve a consistent database/WAL backup and master key, measure the upgrade against a restored copy on comparable storage in an isolated environment with no access to member/platform endpoints, and allow sufficient free disk and startup time. [Zerops](https://zerops.io/)’s supplied readiness `failureTimeout` is **120 seconds**: increase it before deploying if the measured migration plus normal startup can exceed that window. The supplied [Nomad](https://developer.hashicorp.com/nomad) job uses `health_check = "task_states"`; a running allocation is **not** API readiness. Wait explicitly for `/healthz` and `/api/v1/version` after deployment rather than treating allocation/deployment status as proof that migration completed.
A failed or interrupted index transaction does not leave a committed migration marker; reopening retries it. Preserve authoritative state and repair disk/write access rather than repeatedly killing an index build. An older binary whose schema includes migrations through `010_lifecycle_recovery.sql` can reopen this additive index and ignores the unknown `011` marker; its existing metric SQL remains compatible. This statement is specific to `011`, not a guarantee that every other schema or durable-state change supports downgrade. Keep a single active control-plane writer; rolling back the binary also restores that binary’s older, unbounded retention implementation.
```sh
karactl -n default clusters metrics demo --range 1h --step 30s -o json
karactl -n default clusters members demo
```
The UI offers 15m/1h/6h/24h ranges with 15s/30s/2m/10m steps. The `range` URL parameter preserves selection; default is 1h. Buckets average sampled values: fractional state codes can represent mixed states, not a new state. Counter series become reset-aware rates; missing data is not healthy zero.
Namespaces have independent histories even for same-named clusters. Select the namespace explicitly; omitted namespace selects `default`. Agent connectivity, member status and engine health remain separate signals: exporter reachability does not prove quorum or successful application work.
Disable collection with a merge patch setting `spec.monitoring.enabled: false`; history remains until retention prunes it.
## Metrics API and SDK
[Section titled “Metrics API and SDK”](#metrics-api-and-sdk)
`GET /api/v1/clusters/{cluster}/metrics?namespace=&range=1h&step=30s` returns JSON `series` with member/name/unit/labels and timestamped points. Range/step use [Go](https://go.dev/) durations: positive range ≤24h; positive step ≤range. Defaults are 1h and 30s. The [Go client walkthrough](/docs/api/#clientssdk) supplies authentication/client setup.
Historical JSON is distinct from installation-root `/metrics` Prometheus text. A generated SDK client for root endpoints needs the origin, not `/api/v1`, as its base URL. Empty results are not evidence of health.
## Prometheus scraping
[Section titled “Prometheus scraping”](#prometheus-scraping)
Scrape **`GET /metrics` at the installation root**, not `/api/v1/metrics`. Metrics carry authoritative `namespace`, `cluster` and `member` labels; include namespace in queries/alert grouping. `set`/`role` are not guaranteed metric labels.
Use a **Karakoram API token**, not a provider infrastructure credential. Its owner needs `monitoring, read, /`; individual objects are filtered by policy. Without `namespace`, all authorized namespaces are included; `?namespace=` restricts the scrape. A signed-in browser’s access cookie also works, but unattended scrapers should use bearer tokens, not expiring browser cookies.
```yaml
scrape_configs:
- job_name: karakoram
scheme: https
metrics_path: /metrics
scrape_interval: 30s
authorization:
type: Bearer
credentials_file: /etc/prometheus/karakoram-token
static_configs:
- targets: ["karakoram.example.com:443"]
```
Mount an owner-readable token-only credentials file from a secret manager; do not embed tokens in shared YAML. Keep server-certificate verification enabled; configure CA trust for a private issuer rather than `insecure_skip_verify`. `karactl metrics` reads this endpoint too.
The endpoint serves current scrape snapshots, not an indefinitely renewable cache. A failed member scrape omits that exporter’s series and exposes `karakoram_member_scrape_success=0`. Failed target discovery immediately invalidates cached observations for all previously known targets. Federation also checks sample age on every request: after **two collector intervals** (**60 seconds** at the default 30-second cadence), or for missing/future sample timestamps, exporter series are absent and scrape success is zero. Repeated Prometheus requests cannot renew that lease while collection is stalled. Previously persisted historical series are unchanged. This metric lease is separate from the cluster API’s 15-second readiness lease; exporter reachability does not prove application availability or quorum. Use external retention and independent application checks for alerting.
## Control-plane state metrics
[Section titled “Control-plane state metrics”](#control-plane-state-metrics)
The same `/metrics` response includes durable cluster state even when `spec.monitoring.enabled` is false or the member collector is unavailable. Cluster series require `monitoring:read` on each namespace-qualified cluster. Installation-wide backup, notification and certificate series additionally require `settings:read` on `*`; a cluster monitoring grant does not grant that visibility. An explicit `?namespace=` selection excludes installation-wide series. State inventory read errors fail the scrape rather than reporting healthy zeroes. These observations reflect stored state, not independent live database probes.
Every cluster series has `namespace` and `cluster`; extra labels are listed below. All are gauges except the notification failure counter.
| Metric | Extra labels | Meaning |
| ------------------------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| `karakoram_cluster_info` | `provider`, `version`, `phase` | Value 1 with observed engine version and stored phase. |
| `karakoram_cluster_ready` | — | 1 only for Ready; suspended, creating and deleting clusters also report 0. |
| `karakoram_cluster_degraded` | — | 1 for Degraded or Error. |
| `karakoram_cluster_condition` | `type`, `status`, `reason` | Value 1 identifies a current condition, not its health polarity. |
| `karakoram_cluster_members_co_located` | — | 1 when multiple members share a known observed placement group; 0 does not guarantee anti-affinity. |
| `karakoram_member_agent_outdated` | `member` | 1 for a member reported outdated; unknown agent builds are not asserted outdated. |
| `karakoram_backup_last_success_timestamp_seconds` | `schedule` | Latest successful completion; one series per enabled schedule plus `schedule=""` for all backups, including manual ones. |
| `karakoram_backup_last_failure_timestamp_seconds` | — | Latest failed backup completion. |
| `karakoram_certificate_expiry_timestamp_seconds` | `certificate` | Certificate expiry; installation certificates use empty cluster identity. |
| `karakoram_backup_drill_enabled` | — | Whether scheduled drills are enabled. |
| `karakoram_backup_drill_last_success_timestamp_seconds` | — | Latest successful drill completion. |
| `karakoram_backup_drill_last_failure_timestamp_seconds` | — | Latest failed drill completion. |
| `karakoram_backup_drill_last_restore_duration_seconds` | — | Restore duration from the latest successful report, not the longest or latest failed report. |
| `karakoram_deployment_retries_exhausted` | — | 1 while a deployment is terminally fenced pending a desired-state change or explicit retry. |
Installation metrics have no cluster labels: `karakoram_control_plane_backup_enabled`, `karakoram_control_plane_backup_last_success_timestamp_seconds`, `karakoram_control_plane_backup_last_failure_timestamp_seconds`, and `karakoram_notification_deliveries_failed_total{channel}`. Notification counts include deliveries that exhausted retries, not transient failed attempts; use `increase()` or `rate()` to handle counter resets. Channel credentials and delivery destinations are never metric labels.
Timestamp gauges are Unix seconds; zero means no recorded terminal result, not the current time or healthy state. Pending/in-progress records cannot renew a success timestamp. Backup completion freshness differs from the backup’s data recovery point. Results summarize retained durable inventory: deleting backup records or rotating drill-report retention can remove old evidence. Conditions are ternary and condition-specific: `True` can mean a fault, so inspect `type` and `reason` rather than treating every true condition as healthy.
## Grafana dashboards and Prometheus alerts
[Section titled “Grafana dashboards and Prometheus alerts”](#grafana-dashboards-and-prometheus-alerts)
Import the supplied JSON dashboards from [`deploy/grafana/dashboards`](https://github.com/KarakoramDev/Karakoram/tree/main/deploy/grafana/dashboards): `overview.json`, `mongodb.json`, `rabbitmq.json`, `elasticsearch.json`, `postgres.json` and `supabase.json`. Select a Prometheus datasource through the `datasource` variable; no fixed datasource UID or credentials are embedded. For file provisioning, configure a [Grafana](https://grafana.com/) dashboard provider for the mounted dashboard directory and provision the datasource separately. Namespace, cluster and member selectors are populated from member scrape series: select **All** to include state-only clusters that do not have member monitoring.
The overview covers readiness, conditions, backup/drill freshness, certificate lifetime, placement, agent drift, deployment failures and installation signals. Engine dashboards use native metric semantics: reset-aware `rate()`/`increase()` for counters, `max` rather than sums for [Elasticsearch](https://www.elastic.co/elasticsearch)’s repeated cluster-wide values, and separate [PostgreSQL](https://www.postgresql.org/) primary-observed replication lag from replica-local replay backlog. Missing series are unknown, not healthy zeroes.
Load [`deploy/prometheus/karakoram-rules.yaml`](https://github.com/KarakoramDev/Karakoram/blob/main/deploy/prometheus/karakoram-rules.yaml) through Prometheus `rule_files`, reload Prometheus, and route its `severity` labels with [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/). Give the scrape token both cluster monitoring and installation settings read permissions if installation alerts are wanted. Alert annotations link to the relevant operational runbooks.
Review thresholds before enabling paging: backup freshness assumes daily schedules (26 hours), drill freshness assumes weekly schedules (eight days), certificate warnings start at 14 days and become critical at three days, and agent drift warns after 24 hours. Enabled schedules with no successful result are stale. Disable policies intentionally rather than silencing failing runs. Cluster-not-ready alerts can include planned suspension or provisioning. Monitor Prometheus’s own scrape `up` signal too: missing state metrics after a failed scrape must not be interpreted as recovery.
For a degraded cluster, inspect conditions and recent operations before restarting members. For backup/drill alerts, inspect completed reports, verification checks, cleanup inventory and storage access; backup process availability alone does not establish recoverability. For deployment retry exhaustion, correct the provider error before explicitly retrying. Certificate alerts require reviewing certificate identity and renewal conditions, not disabling TLS verification.
## Engine metric vocabularies
[Section titled “Engine metric vocabularies”](#engine-metric-vocabularies)
Metric names, units, availability and engine-specific health interpretation live in:
* [MongoDB operations](/docs/mongodb/operations/)
* [RabbitMQ operations](/docs/rabbitmq/operations/)
* [Elasticsearch operations](/docs/elasticsearch/operations/)
* [Neo4j Community operations and metrics](/docs/neo4j/operations/#monitoring): authenticated Query API/JVM observations and filesystem store size, not an Enterprise Prometheus endpoint; all ten `neo4j_*` series are gauges.
* [Weaviate operations and exact metric vocabulary](/docs/weaviate/operations/#metrics): availability, object/shard/collection counts, query/request/batch counters, vector-index capacity/tombstones, LSM bytes and async queue length; native 2112 is private and wildcard-bound.
Use `rate()` for counters, not gauges; distinguish process availability from recovery coverage and application correctness.
### PostgreSQL
[Section titled “PostgreSQL”](#postgresql)
PostgreSQL members expose the following `karakoram_postgres_` metrics. Database series have a `database` label; [ProxySQL](https://proxysql.com/) pool series have `host` and `hostgroup` labels. Counters reset when their underlying PostgreSQL statistics or ProxySQL process resets. An unavailable SQL or [Patroni](https://patroni.readthedocs.io/) endpoint omits its series rather than reporting a healthy zero.
| Suffix | Type | Meaning |
| ------------------------------------------------------------------------------------------------------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `transactions_committed_total`, `transactions_rolled_back_total` | Counter | Database transactions committed or rolled back. |
| `blocks_read_total`, `blocks_hit_total` | Counter | Database disk blocks read or found in the buffer cache. |
| `tuples_returned_total`, `tuples_fetched_total`, `tuples_inserted_total`, `tuples_updated_total`, `tuples_deleted_total` | Counter | Database tuple activity from `pg_stat_database`. |
| `connections`, `max_connections` | Gauge | Current database providers and the server connection limit. |
| `database_size_bytes` | Gauge | Total size of each database. |
| `replication_lag_bytes`, `replay_backlog_bytes` | Gauge | Primary-observed transport-plus-replay lag and replica-local received-but-not-replayed WAL respectively. Replica backlog excludes WAL that has not arrived. |
| `archiver_failed_total` | Counter | Failed WAL archive attempts. |
| `last_archived_age_seconds` | Gauge | Age of the last successful WAL archive; zero before the first archive. Correlate with archive status and backup configuration. |
| `patroni_primary`, `patroni_timeline` | Gauge | Patroni’s local primary role (1/0) and PostgreSQL timeline on HA members. |
| `proxysql_connections_used`, `proxysql_connections_free` | Gauge | Used and idle provider pool connections. |
| `proxysql_queries_total` | Counter | Queries sent through each provider pool. |
PostgreSQL readiness means that local SQL accepts connections; proxy readiness means that its frontend listener is open and its runtime writer hostgroup has an ONLINE provider at observation time. Neither is a quorum guarantee, and a ready proxy does not make the next connection infallible: clients keep reconnect and retry logic after member failures. ProxySQL defaults to writer-only routing. Optional read/write split rules are deliberately simple and do not guarantee causal reads or classify every SQL construct. Native replication, LISTEN/NOTIFY and other session-sensitive protocols should connect directly to PostgreSQL.
### Supabase application components
[Section titled “Supabase application components”](#supabase-application-components)
Supabase application agents expose these series with a `component` label (in addition to the shared target identity labels):
| Metric | Type | Meaning |
| ----------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `karakoram_supabase_up` | Gauge | 1 when the component’s current readiness check succeeds, otherwise 0. This is component readiness, not PostgreSQL quorum, whole-stack health or an end-to-end application probe. |
| `karakoram_supabase_restarts_total` | Counter | Supervised restarts of the component process. Use reset-aware `rate()` or `increase()`; agent/runtime replacement can reset the counter. |
An agent without a configured component omits both series. A failed scrape is missing data, not an emitted zero; correlate with agent connectivity. The Supabase dashboard uses these series alongside PostgreSQL core metrics. See [Supabase operations](/docs/supabase/operations/#monitoring).
## Events and event streams
[Section titled “Events and event streams”](#events-and-event-streams)
```sh
karactl -n default events list --cluster demo
karactl -n default events watch --cluster demo
karactl -n default operations cluster demo
```
Persisted events and operations support incident review; streams provide timely notification, not authoritative completion. Preserve replay position and filters when reconnecting. Event categories such as `Warning` differ from lifecycle reasons such as `OperationFailed`; use the [API pagination/event contract](/docs/api/#pagination-and-events) for filtering, SSE and proxy behavior. Audit scope/coverage is defined in [Security](/docs/security/#audit-attribution-and-durability).
To query [Data MCP](/docs/data-mcp/) audit events, use `karactl -n default events list --cluster demo --resource cluster-data` or `GET /api/v1/events?namespace=default&cluster=demo&resource=cluster-data`. The same resource filter is supported by event streams. Reading these events requires `events:read` or `audit:read` on the cluster; `cluster-data:read` alone does not grant audit visibility. Explicit unauthorized resource filters return 403, while unfiltered listings omit unauthorized events.
## Troubleshooting
[Section titled “Troubleshooting”](#troubleshooting)
| Symptom | Check/action |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| 403 scrape | Account/token status and per-cluster `monitoring:read` permission. |
| Empty charts | Enabled collection, time range, enough counter samples, agent connectivity and engine monitor credentials/CA. |
| Stale samples | Latest successful observation, failed scrapes and independent availability checks. |
| One namespace unavailable | Project token and [relay health](/docs/namespaces/#relay-health); unrelated namespaces continue. |
| Remote scrape/config pull fails | Public-origin reachability and namespace-qualified routing; never disable CA/hostname checks. |
| Memory near limit | Inspect complete runtime allocation/process usage; a process RSS series is not total container memory. |
# MySQL and Percona XtraDB Cluster
> Karakoram offers two native MySQL-compatible engine drivers. Choose the replication model before creating a cluster; switching engine drivers is not a…
Karakoram offers two native [MySQL](https://www.mysql.com/)-compatible engine drivers. Choose the replication model before creating a cluster; switching engine drivers is not a topology change.
| | Percona Server for MySQL | Percona XtraDB Cluster (PXC) |
| ------------------ | -------------------------------------------------------------- | -------------------------------------------------------------------- |
| Engine driver | `mysql` | `pxc` |
| Server majors | 8.4 LTS default; 8.0 for existing workloads | 8.4 LTS default; 8.0 for existing workloads |
| Topology | `single`: 1 member; `group`: 3 or 5 | `cluster`: 3 or 5 members only |
| Replication | Group Replication, single-primary, `MYSQL` communication stack | Galera certification, encrypted IST/SST |
| Application writer | Group-elected primary | One ProxySQL-routed writer, despite Galera’s multi-writer capability |
| ProxySQL | 1–5 replicas; zero permitted only for `single` | 1–5 replicas required |
| Backup types | XtraBackup `full`, `incremental`; optional binary-log PITR | Same lifecycle, separate engine driver provenance and repository |
| Native access | MySQL users, databases and database-level grants | Same MySQL access interface |
Use `karactl providers get mysql -o json` or `karactl providers get pxc -o json` for exact available versions, presets, resources and upgrade paths. Version strings are engine driver-specific; do not substitute a [Percona Server](https://docs.percona.com/percona-server/) version for a [PXC](https://docs.percona.com/percona-xtradb-cluster/) version. Catalog defaults currently select Percona Server **8.4.11-11** and PXC **8.4.11-11.1**. A compatible, checksum-locked [XtraBackup](https://docs.percona.com/percona-xtrabackup/) release is selected with the server; its own compatibility check is not bypassed. Native artifacts target [Linux](https://www.kernel.org/) amd64 with the declared [Ubuntu](https://ubuntu.com/) 24.04 runtime dependencies; see [native artifacts](/docs/native-artifacts/).
## Create
[Section titled “Create”](#create)
The UI’s **New cluster** wizard lists both engine drivers. Choose a catalog preset, inspect topology and resources, then review TLS, exposure, backups and engine options. **MySQL access** is available on both engine drivers when authorized. A successful create response accepts asynchronous work; wait for the observed generation, member health and application connectivity.
A minimal API-shaped creation file for MySQL:
```yaml
name: appdb
spec:
provider: mysql
topology:
type: single
components:
engine:
replicas: 1
extra:
proxy:
replicas: 1
tls:
mode: requireTLS
ca: selfSigned
```
```sh
karactl -n NS clusters create -f mysql.yaml --wait
karactl -n NS clusters status appdb -o json
karactl -n NS clusters members appdb -o json
```
For Group Replication, change topology to `group` and data replicas to 3 or 5. For PXC, set engine driver `pxc`, topology `cluster` and replicas 3 or 5. Omitting `version` selects that engine driver’s default. The engine driver supplies resource defaults; inspect the catalog before production sizing. A singleton is **not highly available**. More members do not guarantee independent failure domains; see [placement limitations](/docs/limitations/).
## Choose with the failure model in mind
[Section titled “Choose with the failure model in mind”](#choose-with-the-failure-model-in-mind)
* **MySQL single:** no automatic failover; maintenance interrupts access.
* **MySQL group:** native single-primary election survives loss of the control plane while a majority remains available. Writes can block during quorum loss, then fail after minority fencing. A whole-group outage requires evidence-based re-formation.
* **PXC:** route applications through [ProxySQL](https://proxysql.com/), not independently to every candidate writer. Only a `Primary`, `Synced`, `wsrep_ready=ON` member is eligible. A non-Primary minority rejects reads and writes. Certification does not mean every reader has already applied a commit; flow control can slow the whole cluster. Use InnoDB tables with primary keys and keep strict mode `ENFORCING`.
* **Both:** failover interrupts connections. A timeout can leave the transaction’s commit outcome unknown. Reconnect and use application-level idempotency rather than blindly replaying writes. Replication is not a backup, nor a zero-loss or failover-time guarantee.
Physical incremental chains are pinned to a source agent and data incarnation, not a hostname or replication group: replicated InnoDB LSNs are not interchangeable. If that source is replaced, cloned, restored, or unavailable, take a new full backup. Legacy full backups remain restorable, but backups without physical provenance cannot parent or form incremental chains. Backup inventory records the selected member’s running version even when a newer desired version is pending, so a fresh old-version backup can satisfy the major-upgrade prerequisite.
The backup API permits that prerequisite while a major upgrade is durably waiting at its backup gate, before rolling side effects, only when the desired version is the sole unpublished configuration change. Combining the version update with scaling, parameters, TLS, or other changes does not bypass normal backup readiness admission.
PXC shared replication-certificate rotation fences backup, restore, and backup-deletion operations until its full-stop recovery journal completes.
## Guides
[Section titled “Guides”](#guides)
* [Configuration and topology](/docs/mysql/configuration/): supported options, proxy behavior and parameter ownership.
* [Access and TLS](/docs/mysql/access-and-tls/): application identities, password rotation and connection examples.
* [Backups and PITR](/docs/mysql/backups/): backup chains, GTID coverage, UUID isolation and account recovery.
* [Operations and recovery](/docs/mysql/operations/): status, maintenance, scaling, upgrades and safe bootstrap boundaries.
* [Verification](/docs/mysql/verification/): environment-selected native suites and recovery acceptance checklist.
Architecture decisions remain in [ADR 0052](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0052-mysql-engine.md) and [ADR 0053](https://github.com/KarakoramDev/Karakoram/blob/main/docs/adr/0053-pxc-engine.md). See [release status and provider availability](/docs/providers/#engine-availability).
# Access and TLS
> Overview · Configuration · Backups
[Overview](/docs/mysql/) · [Configuration](/docs/mysql/configuration/) · [Backups](/docs/mysql/backups/)
Both `mysql` and `pxc` use **[MySQL](https://www.mysql.com/) access** in the UI, `karactl mysql` in the CLI and `/api/v1/clusters/{cluster}/mysql/...` in the REST API. These are native MySQL accounts, databases and database-level grants—not [MongoDB](https://www.mongodb.com/)-shaped `db-users` or arbitrary SQL execution.
For MySQL Group Replication, both group communication and distributed recovery authenticate peers, not just encrypt traffic: `group_replication_ssl_mode` is `VERIFY_IDENTITY`, and recovery uses the cluster CA, the member SQL certificate and private key, and `group_replication_recovery_ssl_verify_server_cert=ON`. The advertised recovery hostname (`report_host`) and group addresses are member names present in the issued certificate’s DNS SANs. No additional certificate issuance or weaker `VERIFY_CA` mode is needed. These recovery TLS settings also govern the `MYSQL` communication stack in both supported server majors ([8.4](https://dev.mysql.com/doc/refman/8.4/en/group-replication-configuring-ssl-for-recovery.html), [8.0](https://dev.mysql.com/doc/refman/8.0/en/group-replication-configuring-ssl-for-recovery.html)). [PXC](https://docs.percona.com/percona-xtradb-cluster/) uses its separate [Galera](https://galeracluster.com/) traffic credentials and configuration described below.
## Application identities
[Section titled “Application identities”](#application-identities)
Create a database, create a dedicated application user, then grant only the required privileges on that database. Accounts are `'name'@'%'` and require TLS. Names and privileges are validated; reserved schemas and system identities are hidden and cannot be modified here. Global/administrative grants are not exposed. Database character set and collation are creation-time settings, not alter operations.
Example request bodies for `-f`:
```json
{"name":"app","maxUserConnections":20,"locked":false}
```
```json
{"name":"appdb","characterSet":"utf8mb4","collation":"utf8mb4_0900_ai_ci"}
```
```json
{"user":"app","database":"appdb","privileges":["SELECT","INSERT","UPDATE","DELETE"]}
```
```sh
karactl -n NS mysql databases create CLUSTER -f database.json
karactl -n NS mysql users create CLUSTER -f user.json
karactl -n NS mysql grants grant CLUSTER -f grant.json
karactl -n NS mysql users list CLUSTER -o json
karactl -n NS mysql grants list CLUSTER --user app --database appdb -o json
```
Capture the generated password once into an approved secret store. Do not log creation/rotation responses or put passwords in command arguments or URIs in shell history. Manage proxy-routable credentials through this interface: accounts created directly in SQL are **not** automatically synchronized into [ProxySQL](https://proxysql.com/). `routableThroughProxy` reports managed credential ownership, not end-to-end connectivity or a grant to every database.
The UI supports account locking, maximum user connections, database/grant management and permission-gated password actions. All native MySQL access uses the `database-users` RBAC resource; read access does not authorize mutation. See [access control](/docs/access-control/) for policy syntax. System-account reveal requires the separate `cluster-credentials:read` permission and is not an application credential workflow.
## Password rotation
[Section titled “Password rotation”](#password-rotation)
Creation and rotation return server-generated passwords once. Rotation may retain the previous password using `retainCurrent: true` in the request body. Deploy the new secret to all clients, verify it through the application endpoint, then discard the old password. The UI exposes **Rotate password**, **Keep current password until discarded** and **Discard old password**. A retained password is still valid until explicitly discarded; do not treat rotation alone as revocation.
Native mutations resolve the current writer and synchronize managed credentials to the proxies. If an outcome is unknown, inspect the account/operation state before another mutation; transport failure does not prove that account DDL failed. MySQL account DDL implicitly commits, and password changes are not a transactional rollback mechanism. Recovery also has [account-snapshot caveats](/docs/mysql/backups/#accounts-after-recovery).
PXC administration follows the writer designated by ProxySQL, and only a Primary-component, Synced, ready and connected member is eligible. Joining and Donor/Desynced members are never selected. PXC agents serialize managed access locally without SQL advisory locks: `GET_LOCK()` is unsupported by PXC strict mode. MySQL single-node and Group Replication retain their SQL advisory lock. After upgrading a control plane carrying this fix, roll out the matching agents before re-proving managed PXC access or binary-log archiving.
Access requests and new-member initialization remain fenced during active recovery; a completed reformation receipt does not keep those fences active. Delete and suspend reject new access mutations, but first resolve any previously sealed account intent before removing runtimes. An applied SQL receipt finishes credential publication and proxy acknowledgement without repeating SQL. An ambiguous result replays the same request identity against the native journal; only a definitive rejection restores the previous sealed indexes. Unavailable connectivity keeps the intent and runtimes intact until resolution, rather than discarding potentially committed account changes.
Restored Group Replication followers receive destructive clone permission only for their own outstanding restore receipt. Seed formation does not revoke that permission prematurely: each follower keeps it until its clone completes, then a new configuration revision revokes it before joining. Ordinary fresh-member initialization and retained completed clone receipts do not grant permission to overwrite an existing identity.
A pending PXC shared-traffic TLS rotation also fences access. Cleanup supersedes that rotation only after all runtimes have disappeared: suspension finishes publishing the common pending traffic CA and leaf certificate/key bundle before preserving volumes; deletion removes all active and pending traffic CA/leaf secrets. A persisted activation checkpoint makes interrupted pending-secret removal replayable without changing certificates under running members.
## Connect
[Section titled “Connect”](#connect)
```sh
karactl -n NS clusters connection CLUSTER -o json
karactl -n NS clusters ca CLUSTER > ca.pem
mysql --host=APPLICATION_ENDPOINT --port=3306 --user=app --password \
--database=appdb --ssl-mode=VERIFY_CA --ssl-ca=ca.pem
```
Use the endpoint from connection status, not a data member chosen by index. The CLI prompts for the password. PXC and Group Replication use the proxy; direct connection is supported for a proxy-free MySQL singleton. Connection summaries contain no password. The default authentication plugin is `caching_sha2_password`; secure transport is required and legacy `mysql_native_password` is not enabled.
`VERIFY_CA` validates the certificate authority; use `VERIFY_IDENTITY` when the endpoint certificate and driver support hostname verification. Do not disable certificate checking merely because encryption succeeds. Download the CA through the authorized workflow, distribute it to clients and rehearse CA rotation. Driver URI TLS options differ: explicitly configure CA verification, percent-encode credentials/database names and bracket IPv6 addresses when using a URI.
Private endpoints require network access to the cluster. Public exposure is opt-in on 3306 with the configured allowlist; independently verify allowed and denied external clients. Reader port 3307 is private. TLS and an allowlist are complementary controls, not substitutes for least-privilege identities.
During failover, reconnect with bounded, idempotent application retry logic. A connection loss or timeout can occur after a commit; blindly retrying a non-idempotent write can duplicate its effect. Reader routing is not a causal-consistency guarantee; see [routing](/docs/mysql/configuration/#application-routing).
## PXC cluster-traffic trust boundary
[Section titled “PXC cluster-traffic trust boundary”](#pxc-cluster-traffic-trust-boundary)
PXC uses a **dedicated traffic CA** for its shared Galera/SST traffic certificate and private key, separate from SQL and agent trust. SQL and agent leaves cannot authenticate replication. Member-specific SQL leaves still do not provide per-member replication identities: compromise of the shared traffic key affects the cluster-wide replication/state-transfer trust boundary, not just one member. Treat disclosure as a cluster-traffic credential compromise, including unauthorized participation or state-transfer access wherever that traffic credential remains trusted.
Restrict traffic-key access to the intended data members and protect the sealed secret store and its encryption key. Managed traffic rotation stops all data members, replaces **both the dedicated traffic CA and shared leaf/key**, and uses evidence-based recovery. Reformed members trust the new root and reject the old traffic credential. Ordinary member addition does not rotate that identity or require a TLS outage. See [offline rotation](/docs/mysql/operations/#restart-configuration-and-scale).
Issuing a new leaf under the same trusted CA would **not by itself revoke** a stolen, still-valid certificate; replacing traffic trust is therefore part of managed rotation. Fence the compromised member and remove its access to secrets and private networking before distributing replacement material, or it could obtain the new credential. Verify old/untrusted credentials can no longer join or transfer data before reopening traffic. This is cluster-wide trust rollover, not independent per-member revocation; no CRL/OCSP facility is implied.
The managed manual TLS mode requires encryption **and peer verification** on Galera/IST and SST. `pxc_encrypt_cluster_traffic=OFF` only disables upstream automatic certificate rewriting; it must never be interpreted as permitting plaintext or accepting a wrong CA. Native acceptance must reject wrong-CA traffic and SST attempts while ordinary CA-verified SQL access remains independently tested.
Server-side FILE import/export is unavailable by design: `secure_file_priv=NULL` and the managed administrative account has no `FILE` privilege, protecting traffic keys and other runtime files from SQL file access. Every actual server startup enforces that privilege policy independently of password receipts, including existing/restored accounts, without re-keying unchanged passwords. Startup restores the Group Replication write fence before admitting clients. This does not disable native [XtraBackup](https://docs.percona.com/percona-xtrabackup/) backup/recovery.
### Replication ports and network restriction
[Section titled “Replication ports and network restriction”](#replication-ports-and-network-restriction)
PXC data members listen on Galera group communication `4567/tcp` and `4567/udp`, IST `4568/tcp` and SST `4444/tcp`. Karakoram never publishes these ports; only data members of the same cluster need to reach them. TLS peer verification rejects unauthorized peers but cannot stop an untrusted host on the same network from opening connections (see [hostile traffic](#hostile-cluster-traffic)). None of the registered providers offers Karakoram a private per-port or per-cluster network policy, so member-only reachability is **documented, not enforced**:
| Provider | What Karakoram enforces | Operator action for member-only reachability |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Zerops | Nothing beyond private listeners: every service in a Zerops project shares the project private network; Zerops firewall allowlists apply only to public port routing. | Place the PXC cluster in a dedicated project when other services in the project are not trusted. |
| Nomad | Nothing beyond private listeners: members join the target’s CNI network (`network`, default `karakoram`) shared by every job using it; the provider renders no per-port rules and no Consul service-mesh intentions. | Run PXC in a namespace whose CNI network carries only that cluster, or apply host/CNI firewall rules allowing `4567`, `4568` and `4444` only between its data members. |
| Dokploy (Swarm) | Nothing beyond private listeners: members attach to the target’s attachable overlay (`network`, default per Dokploy environment) shared by every cluster in that environment and by the control plane; Swarm overlays have no per-port ACL. | Use a dedicated Dokploy environment or `network` for the PXC cluster and attach no other workloads to it. |
Verify the chosen restriction from a non-member runtime: connections to `4567`, `4568` and `4444` should fail at the network layer, not only at TLS.
### Hostile cluster traffic
[Section titled “Hostile cluster traffic”](#hostile-cluster-traffic)
Peer verification is an admission control, not a denial-of-service boundary. Karakoram makes **no availability claim** for a member whose replication ports are reachable by hostile clients.
Observed with PXC 8.4.11-11.1 and 8.0.46-38.1: a TLS client presenting an untrusted certificate (signed by the SQL CA, or self-signed) connected to a joining member’s IST port while a legitimate IST was running. The joiner sent the expected certificate alert and admitted no data, but its `mysqld` then aborted (`terminate called after throwing an instance of 'gu::Exception'`, `mysqld got signal 6`). Upstream reset that member’s `grastate.dat` to the zero UUID with `seqno: -1` and `safe_to_bootstrap: 0`, the documented state after an interrupted transfer. The remaining quorum kept every row and GTID. Upstream issue trackers contain no matching report; related fixed issues are PXC-4167 (uncaught accept exception, signal 6), PXC-4631 (zero-UUID reset after a failed IST) and PXC-4845, plus PXC-3741 (IST hang after network loss).
Recovery uses the ordinary product path, whether the member is stopped by an operator first or restarted by its supervisor:
* The member accepts upstream’s exact reset only under its own cluster and member receipt, in the pinned history (a control-plane re-formation of that history may raise the formation epoch), and only when its last launch never reached Synced. Any other state identity stays fatal. Galera writes the same tuple when it evicts a Synced member for an inconsistency. That case also stays fatal and needs explicit recovery or replacement, not an automatic rejoin with inconsistent data.
* Relaunch waits 130 seconds after a launch that never reached Synced. Upstream’s donor keeps retrying the joiner’s SST address (30 attempts, bounded by the rendered `donor-timeout=3`). A joiner started inside that window could consume the stale stream of the aborted request and wait indefinitely for an IST that no donor sends.
* The member then rejoins by IST or SST and re-establishes its identity only once it is Synced in the same history.
Repeated hostile connections can repeat the abort and keep a member out of the cluster indefinitely. Restricting the replication ports to cluster members is the only mitigation (see above).
## Management agents and restricted tokens
[Section titled “Management agents and restricted tokens”](#management-agents-and-restricted-tokens)
MySQL and PXC share the MySQL management tools exposed by the control-plane MCP server. User creation, attribute updates and password rotation are classified as secret-bearing security mutations and require confirmation. User deletion, discarding an old password, grant changes and PXC quorum recovery also require confirmation; database deletion requires name confirmation. Quorum recovery is asynchronous and returns a tracked operation. Read-only user, database and grant tools never reveal passwords.
Restricted API tokens use the existing cluster-scoped `database-users` permissions for access management and `clusters:update` for quorum recovery. These engine drivers do not expose a Data MCP gateway.
# Backups and point-in-time recovery
> Overview · Access · Operations
[Overview](/docs/mysql/) · [Access](/docs/mysql/access-and-tls/) · [Operations](/docs/mysql/operations/)
Replication is not a backup. Register and verify an [S3](https://aws.amazon.com/s3/) backup storage, select it on the cluster, schedule full backups and test recovery into an independent cluster. Use the shared [backup storage and retention guide](/docs/backups/) for permissions and storage registration; [MySQL](https://www.mysql.com/)/[PXC](https://docs.percona.com/percona-xtradb-cluster/) use the engine driver-specific rules below.
## Physical backup chains
[Section titled “Physical backup chains”](#physical-backup-chains)
Both engine drivers stream [XtraBackup](https://docs.percona.com/percona-xtrabackup/) through `xbcloud` to S3 without staging the full archive locally. Types are `full` and LSN-based `incremental`, not `logical`, `physical`, `diff` or `incr`. Each incremental depends on a successful parent in the same repository incarnation and server major. Preserve the full root and every intervening incremental. The pinned compatible XtraBackup release performs its normal server-version check; numerical patch values across the two products are not directly comparable.
```sh
karactl -n NS backups create CLUSTER baseline --type full --storage ARCHIVE
karactl -n NS backups list CLUSTER -o json
karactl -n NS backups get CLUSTER baseline -o json
```
Creation returns an operation; observe that existing operation and terminal backup inventory rather than submitting again after a timeout. A streaming backup is not complete merely because one side of the upload finished. Interrupted backups are reported as failed, not silently repeated. Backups use DDL locking; schedule with DDL contention, donor load and available capacity in mind. Group Replication normally prefers an online secondary; when binary-log PITR is enabled, the backup must anchor the current archive writer instead.
Full backup receipts report XtraBackup’s native checkpoint type `full-backuped`; the control plane validates it against the requested API type `full`. Incremental receipts report `incremental`. Metadata validation failures identify the inconsistent field without including its value.
Provenance includes engine driver, source version/topology, cluster incarnation, repository, tool/server versions, GTID set, LSN range, chain dependencies and completion time. MySQL repositories use `/mysql//-`; PXC uses the corresponding **`pxc`** prefix. They share a subtree structure, not an interchangeable backup identity:
* `base/