Operations and recovery

Overview · Configuration · Backups · Verification

Observe before mutating

Use Overview, Members, Monitoring, Events and Operations together. Acceptance is not convergence: compare desired/observed generation, terminal operation state, member versions, replication eligibility and a real application query. A reachable listener alone is not a healthy writer. While suspended, observations are historical and endpoints are unavailable.

karactl -n NS clusters status CLUSTER -o json
karactl -n NS clusters members CLUSTER -o json
karactl -n NS operations cluster CLUSTER -o json
karactl -n NS events list --cluster CLUSTER -o json

For MySQL groups, require ONLINE membership, one primary, read-only secondaries and the expected member count. The default BEFORE_ON_PRIMARY_FAILOVER policy waits for the new primary’s backlog before it serves traffic. For PXC, require Primary, Synced and wsrep_ready=ON; Primary is Galera component membership, not a promise that every node is the application writer. ProxySQL selects one routed writer. Donor/joining states and SST can temporarily reduce eligibility.

Before Group Replication bootstrap, a SQL-ready server may expose an OFFLINE performance-schema row with an empty member ID. Absence of a row matching that server’s UUID is reported as OFFLINE with no member role, not as an unreachable native status endpoint. This is a nonmember observation, not quorum readiness: the control plane still requires its durable bootstrap epoch and observed ONLINE primary before admitting the writer. SQL query failures remain errors. Single-server and PXC readiness use their own existing native checks.

During initial provisioning, members not yet imported into the provider are reported unavailable without a placement. This includes ProxySQL while it waits for the first eligible writer. Such members are not a failed infrastructure lookup; the cluster stays non-Ready until their runtime and native health are observed. This applies to both MySQL and PXC.

Before a runtime is imported and deployed, observation does not dial its agent or query Galera: its planned DNS name may not exist yet. PXC keeps these members in the quorum comparison as unavailable, without advancing the GTID safe watermark. Galera status probes use the bounded native observation timeout (currently four seconds), not the entire reconciliation deadline. PITR/archive status selects only joined data members, so enabling binary-log archiving during creation does not resolve planned names either. Initial imports still advance through durable per-member plan, submit and completion checkpoints over successive reconciliations. A control-plane upgrade can resume an otherwise healthy create stalled at these checkpoints; do not edit internal state to bypass observation.

PXC deployment initially starts only the agent. All data runtimes must deploy before the control plane publishes the seed’s bootstrap epoch; only that seed may start mysqld until it forms a Primary component. Remaining members then start as SST/IST joiners. A runtime deployment is not permission to launch an uninitialized seed as an ordinary joiner.

Galera’s local bind address is distinct from its advertised member names. For agents advertising capability=mysql-galera-listen-host on their authenticated configuration pull, the control plane supplies a numeric IPv4 wildcard on IPv4-only Nomad/Dokploy layouts and an IPv6 wildcard ([::]:4567) on Zerops’s IPv6-capable layout. The capability applies even on the first pull and only to that request. Older agents receive the unchanged legacy schema without listenHost, avoiding their strict decoder’s unknown-field rejection. A fixed agent talking to an older control plane can still receive a fieldless legacy configuration; its DNS-host fallback is IPv4-only, so full Zerops IPv6 bind support requires both the fixed control plane and fixed agent. The pinned Galera provider resolves a hostname listener but subsequently parses that original host as a numeric outgoing source bind, so using a member DNS name as gmcast.listen_addr can abort connection establishment with Invalid argument. Peer discovery, advertised node/IST/SST identities and encrypted traffic remain unchanged. The IPv6 wildcard’s IPv4 acceptance follows the host kernel’s dual-stack policy; it is not a claim of an IPv4 listener on an IPv6-only host.

An older create may already have started the seed without bootstrap permission and left uninitialized/foreign-looking data. A control-plane upgrade alone does not authorize erasing or reinitializing that data. Preserve the logs, volume contents and identity receipts and follow explicit recovery; disposable empty test clusters can instead be fully deleted and freshly created after operator authorization. Never relax the identity guard to make such a create pass.

Proxy provisioning waits for a native-online writer to complete its data-member join checkpoint, so its first configuration contains an eligible backend. A pending proxy deployment does not block that initial data join, including when reconciliation resumes a proxy deployment accepted by an earlier version.

Monitor connections versus limits, running threads, query/network rates, replication queues/lag, archive lag and proxy provider state. No metric value or local failover experiment establishes a production recovery-time or zero-loss guarantee. Check placements independently: three members co-located in one failure domain do not tolerate losing that domain.

PXC GTID history safety

GTIDHistorySafe=True means a freshly observed healthy, same-Galera-history majority agrees on historical SID components, differs only by nested current-state SID prefixes, and has no active continuity latch. Normal active-write lag is accepted; equal counts, SID spelling and Galera sequence numbers alone do not establish GTID equivalence. Restored historical namespaces are legitimate when shared by the quorum; an unshared extra namespace is not treated as ordinary replication lag.

GTIDHistorySafe=False / GTIDHistoryDiscontinuityLatched records a durable divergence boundary, the detection time and affected members. The condition appears in the cluster API and the overview’s conditions table. A later restart, equal GTID sets, SST or forced re-formation does not silently erase that observation. Unknown / QuorumComparisonUnavailable means a safe majority comparison cannot currently be established. The safe watermark advances only when every configured member is successfully observed eligible in the same component. An available majority with an unseen member does not certify that unseen member’s intervening history.

The guard does not stop database service or prohibit full backups. It blocks operations that would claim GTID continuity across the unsafe interval: incremental backup chains and PITR/binlog replay spanning it. An independently valid backup and archived target before the recorded safe boundary remain restorable. Do not equate the detection timestamp with the first unsafe transaction: an IST can replay an older transaction under a different GTID.

Preserve logs, executed GTID sets and application evidence. Identify the trusted quorum and use managed member replacement to provision fresh data from it; verify both application contents and GTID history. Take a new full backup after detection to establish a fresh PITR baseline. That explicit, successful full backup is the operator acknowledgement: no flag, restart or status edit clears the latch. Successful completion records the new baseline and clears the active continuity latch; old chains retain their unsafe boundary. A failed, cancelled or already-running old full backup cannot clear it. A new conflict creates a new latch.

Both transitions are audited as cluster events: GTIDHistoryDiverged (Warning; boundary, detection time and affected members) when the latch is recorded, and GTIDHistoryBaselineEstablished (linked to the backup operation) when a fresh full clears it. The upstream cause is tracked as PXC-4493.

Do not reset, normalize or inject GTIDs, force bootstrap as a GTID repair, or attach an asynchronous replica across the affected history merely because rows or later sets match. An inverse SID alone is not an error: compare the complete history with quorum. See backup continuity boundaries.

Restart, configuration and scale

• Dynamic allowlisted parameters can apply live. Startup-only parameters and resource changes use managed rolling maintenance. Keep configuration in the specification, not SET PERSIST.
• MySQL single restarts cause downtime. Groups maintain secondaries first, then move the primary and finish its maintenance. Require a majority throughout; applications reconnect after writer movement.
• Group primary handoff sends a validated UUID literal to MySQL: parameterized execution is rejected by the native UDF initializer even when a valid UUID is bound. Failed handoffs return bounded, sanitized native diagnostics in operation errors/events; a rejected handoff does not authorize restarting the current primary. This behavior is implemented in the agent, so replacing only the control plane does not update existing agents.
• PXC maintenance preserves a Primary component, waits for healthy Synced members and avoids intentionally removing a majority. Rejoining members use IST when available, SST otherwise. Do not continue a rolling change while state transfer or quorum health is unresolved. A member whose previous launch never reached Synced (an interrupted IST/SST) waits 130 seconds before relaunching so the previous donor’s retries expire; see hostile cluster traffic. A member evicted by Galera for an inconsistency is not rejoined automatically; replace it.
• MySQL allows single→group, adopting the existing data as the seed; group→single is rejected. Groups scale 3↔5. PXC remains cluster, 3↔5 only. Cross-engine driver conversion is unsupported.
• Replacement introduces a new member identity before retiring the old one. Use the managed replace action; do not copy data identity or a stale bootstrap flag into another member. Disk shrink is not supported.
• Suspension with retained volumes preserves the dataset and identities. Resume after all members stop uses the full-outage recovery rules below; it is not an unconditional restart into a writable cluster.
• Resume and runtime recovery restore the previously admitted public TCP policy even while the maintenance window is closed. A pending exposure or allowlist/IP-family change remains deferred and its generation is not acknowledged until the window opens or the generation receives a maintenance override. Admission survives control-plane restarts separately from runtime route receipts. Legacy state migrates acknowledged policy or a matching retained exposure hash; if neither identifies the old policy, recovery stays private until admission.

PXC cluster-traffic certificate rotation is an offline exception to normal rolling maintenance: all data members stop, replace the dedicated traffic CA and shared traffic leaf/key, and recover using the full-outage position rules below. Reformed members reject credentials signed by the old traffic root. Plan database downtime and verify recovery readiness first. SQL and agent trust remain separate from the traffic CA. Fence a compromised member’s secret/network access before distributing replacements; shared traffic trust has no independent per-member revocation. Ordinary membership changes do not rotate the traffic identity or require this TLS outage.

Retry a wait/observation, not an already accepted mutation. Inspect the operation ID and conditions after interruption. A cancelled or failed operation does not necessarily undo data or desired-state changes.

Group clone intents are bound to the configured donor, version and recovery receipt. A lost CLONE reply is an unknown outcome, not permission to resend the destructive statement or interrupt an active copy. Before restarting after a lost reply, the agent observes a newer matching native operation that is completed or has finished file synchronization and is ready for restart. If that observation is unavailable, the intent remains pending rather than forcing a restart. Recipient restart uses a fresh administration pool, and receipt finalization still requires matching native completion. Interrupted replay can resume that operation’s native restart phase; a stale completion, active copy, failed clone or uncorrelated legacy intent cannot satisfy the pending receipt. Existing members still require explicit recovery-clone authorization.

Pending-receipt replay treats a failed SQL probe as unavailable evidence, not proof that the recipient process exited. Recovery may start an unreachable recipient only when its supervisor has confirmed process exit; otherwise the operation remains pending without stopping a potentially active clone.

Clone failures from updated agents include a fixed stage category in operation errors/events, such as stage=authorization, stage=donor-probe, stage=native-clone or stage=receipt-recovery, plus the native MySQL error number when available. Native message text, SQL, credentials, paths, peer addresses and log tails are not included. A stage identifies where this request failed, not whether an earlier CLONE was applied; preserve native receipt and progress evidence before recovery. Existing agents need an updated executable to emit these diagnostics; a control-plane-only update cannot add them.

Upgrade

The Upgrades tab selects the current engine driver’s offered paths; MySQL and PXC have separate version/artifact catalogs. Minor upgrades use the managed rolling order. The supported 8.0→8.4 major path requires a recent successful current-version backup validated by the provider. Test that backup and review schema/application compatibility before starting. An arbitrary old successful backup shown in the UI is not proof of eligibility.

Group Replication upgrades secondaries before moving the primary. A newer member can remain read-only during mixed-version membership; do not force lower-version joins or use cross-major clone as an upgrade technique. PXC uses its own Galera-compatible path and waits for eligible members before advancing. Follow per-member versions and the operation, not only the desired version.

Downgrades and engine driver conversion are rejected. Rollback requires a tested compatible pre-upgrade backup and independent recovery plan, not starting old binaries against an upgraded dataset. Keep backup objects and sealed account provenance until the upgraded workload is accepted.

Writer loss and quorum loss

A MySQL group elects a new primary natively while a majority remains; a PXC Primary component keeps eligible members and ProxySQL moves the routed writer. Established majority failover does not require the control plane to elect a writer. This does not guarantee uninterrupted client sessions or identical behavior for every network partition.

MySQL minority writes can block, then fail after the configured unreachable-majority timeout and read-only exit. PXC non-Primary members reject reads and writes (typically error 1047) and must leave routing. Never interpret a timeout as proof that a write did not commit. Use unique application request identities and inspect outcomes before non-idempotent retries.

Do not force pc.bootstrap=YES merely to clear a health condition. It can create a second Primary component during a partition. Minority recovery requires explicit operator authority and evidence that the selected survivor has the highest committed position in the same Galera history. The supported action refuses an existing Primary component, fences the exact missing owned members and waits for durable fencing completion before authorizing one survivor. Missing or conflicting evidence blocks recovery; there is no data-loss bypass.

Explicit PXC minority recovery

The PXC-only Recovery tab requires cluster update permission and an advertised recover-quorum operation. In Degraded/Error state, open Recover quorum, read the danger warning and type all-absent-members-fenced before submitting. The confirmation authorizes fencing; it does not replace the provider’s own fencing or position checks.

The corresponding REST action is POST /api/v1/clusters/{cluster}/mysql/recover-quorum with the exact body:

{"confirmation":"all-absent-members-fenced"}

The CLI command is karactl -n NS mysql recover-quorum CLUSTER -f recovery.json with that body in recovery.json; follow the CLI’s explicit destructive-action confirmation. The response is an accepted Operation, not proof of recovered quorum. Follow its ID in Operations and verify Primary/Synced/readiness, the expected member count and application contents. A transport timeout does not cancel the operation. Do not submit or manually bootstrap another member while recovery is pending.

Full-outage recovery below compares stopped members’ recovered positions and is distinct from this live-minority action.

Full-outage recovery

All members must be stopped/fenced before choosing a seed. Recovery fails closed if evidence is missing or incompatible; do not choose the first reachable member.

MySQL Group Replication	PXC / Galera
Observe every member’s offline executed GTID set and data identity	Recover every stopped member’s Galera state UUID and sequence number using redo recovery
Select a member whose GTID set is a superset of every other member	Require one common state UUID, then select the highest recovered sequence number
Incomparable sets or missing observations block re-formation	Missing observations, conflicting UUIDs or unusable positions block bootstrap
Bootstrap only that seed, then join the others normally	Mark only the selected seed safe, bootstrap with its recovered position, then rejoin the others

After a crash, grastate.dat often contains sequence -1; safe_to_bootstrap alone cannot rank data freshness. Graceful suspension uses the same evidence-based resume policy rather than assuming shutdown order. An ordinary PXC restart also recovers its local position before rejoining; it must not turn stale bootstrap intent into a new independent cluster. Bootstrap authorization is bound to a formation epoch, preventing an old desired configuration from replaying bootstrap.

PXC state transfer uses upstream SST behavior rather than custom keep-list overrides. Managed data-identity tracking permits UUID regeneration only for an explicitly authorized state transfer and records the accepted identity after Synced state. Re-formation/restoration may explicitly reset a stopped joiner’s own dataset for SST; an arbitrary identity mismatch is not permission to erase or initialize it. This is distinct from independent backup restore, which creates new replication history.

If recovery is blocked, preserve all remaining data and position evidence, isolate application traffic and resolve the missing observations or recover from a verified backup. Never reset GTIDs, invent a Galera position or edit bootstrap flags across multiple members to force a green status.

Group Replication bootstrap epoch consumed

GroupReformationBlocked=True with reason BootstrapEpochConsumed means the selected seed durably claimed its bootstrap epoch but is not observed as an ONLINE primary. The claim is recorded before enabling bootstrap or sending START GROUP_REPLICATION. It is an at-most-once safety receipt, not a successful join receipt. Even a rejected enable statement can therefore consume an epoch.

An HTTP 503 from /v1/mysql/bootstrap does not prove that Group Replication never started. The same response covers native configuration/permission errors, a lost START reply, failure to clear the bootstrap flag, and failure while observing or recording an ONLINE join. An OFFLINE snapshot or zero currently ONLINE members does not establish that no group previously formed. Do not retry bootstrap on a different member, decrement the identity epoch, edit managed flags, or use the PXC-only recover-quorum action for this condition.

Corrected data-member agents return a bounded, credential-redacted native error on bootstrap and start-group failures. Inspect the ReconcileError event for the MySQL error number/message and, when available, recent relevant entries from the managed mysql-error.log. The log excerpt is explicitly historical and may predate the request; it is not proof that a particular statement executed. Unavailable logs do not suppress the primary error. The safety condition remains BootstrapEpochConsumed; it does not authorize another attempt. Existing agents need replacement through supported lifecycle to gain these diagnostics; a control-plane-only update cannot add them retroactively. Native messages mentioning recovery credentials, password/identity statements or key/certificate material are omitted as whole records, including continuations, even when the credentials differ from the current configuration. Error 3972 (ER_GROUP_REPLICATION_PASSWORD_LENGTH) instead renders a trusted static explanation: the recovery credential exceeds MySQL’s 32-character maximum; the raw message is never exempted from credential filtering. Log excerpts accept only complete timestamp-delimited GR/GCS/XCom/recovery records carrying a MySQL error code; a leading partial record in the bounded file window is skipped.

Preserve native error logs, effective GR/listener/TLS variables, member identities and executed GTIDs before recovery. For a previously formed group, fence/stop all members and use the full-outage evidence requirements above; automatic all-offline recovery requires the recorded formation to have completed. Retained-volume suspend/resume also collects every member’s stopped, initialized identity and offline GTIDs before selecting a seed. Neither is an unconditional epoch reset. Missing observations require resolving the cause or recovering a verified backup.

For a disposable, never-formed create with no application data, preserve the diagnostics, correct the native startup cause, then use owner-authorized normal delete and fresh create with corrected control-plane/member artifacts. This is the supported clean recovery, not in-place retry of a consumed epoch. Mere deletion/re-creation with unchanged artifacts/configuration is not a native fix.

New MySQL clusters generate a 32-character random kk_repl credential, matching the native START GROUP_REPLICATION recovery-password limit in MySQL 8.0/8.4. Older control planes generated 40 characters, causing native error 3972 before Group Replication could form. Updating the control plane corrects generation for fresh clusters, but deliberately does not truncate stored credentials: those must continue matching already-created SQL accounts. A disposable cluster that has consumed its epoch with error 3972 therefore needs delete and fresh create after the corrected control plane is deployed; updating agents alone is not a repair of its existing credential or epoch.

For an already blocked initial GR formation with an overlong stored recovery credential, BootstrapEpochConsumed now explicitly reports the 32-character limit and directs operators to delete/recreate only disposable never-formed clusters, preserving valuable data instead. No account rotation or epoch reset is attempted merely because current members are OFFLINE: that snapshot cannot prove no group ever formed.

There is one safe automatic migration boundary: conversion of an existing single MySQL member to Group Replication. Before publishing adoption, the control plane persists a new 32-character recovery credential if the old one is overlong. Applying the new configuration detects its changed credential hash and restarts the sole member with protected startup SQL that executes ALTER USER. Only after that configuration is acknowledged does formation perform its adoption restart and first GR bootstrap. A control-plane crash before adoption state is saved reuses the persisted replacement; replay does not generate another value. This does not alter formed GR clusters or reopen consumed multi-member epochs.

If the create may contain valuable data, retain its volumes and seek evidence-based recovery instead; there is no public Group Replication reset-bootstrap-epoch action. Updating only the control plane cannot undo an agent’s durable claim or change its native renderer. MySQL does not support upgrade-agents.

Native startup and agent connectivity

MySQL/PXC agent connectivity reports a successful authenticated agent status probe independently of native database readiness. A connected agent can still have an UNREACHABLE/unhealthy database. NativeObservationFailed=True reports the first available structured native status failure, including a bounded, sanitized recent server/WSREP error-log excerpt on corrected member agents. The excerpt can predate the request and is not proof of a particular bootstrap attempt. Sensitive records and their continuations are omitted as a whole.

Configuration reload is synchronous and can wait for authenticated local SQL and, on PXC, eligible Galera state. A caller deadline can expire before the agent’s native startup budget; that does not establish a DNS, TCP or TLS failure. Preserve the native error log and compare authenticated agent status separately from database health before changing listener policy or attempting recovery. These diagnostics do not reset bootstrap receipts or authorize re-bootstrap.

Configuration application uses the engine-action request budget rather than the four-second status-observation budget, while retaining caller cancellation. Galera launchers persist bootstrap/rejoin receipts using ordered file/directory fsync (sync PATH), not filesystem-wide syncfs (sync -f PATH), which some unprivileged volume environments reject. Persistence failure still prevents native startup; no receipt or durability check is bypassed. Before MySQL/PXC configuration/startup proceeds, every directory from the managed configuration/run and native data directories through the mounted volume root is fsynced, including existing ancestors. Managed configuration and identity files retain their file-fsync, atomic rename and parent-fsync publication. Native data-file durability remains the database’s initialization, transaction/recovery and shutdown contract; receipt persistence is not a replacement for database flushing or a filesystem-wide data backup.

If an older launcher created a bootstrap receipt and then failed to persist it, do not remove that receipt or assume bootstrap is safe to repeat. A disposable, never-formed create should be deleted and recreated with corrected member artifacts after preserving evidence. Valuable data requires the normal fenced, evidence-based recovery procedure.