Replication witness

Normally, promoting a secondary volume to primary is something you do deliberately, by hand, after confirming the primary side is actually down. The replication witness is an optional arbitration service that continuously monitors both systems in a replication pairing and can trigger that promotion automatically on your behalf if it determines the primary side has gone down — without anyone needing to log in and act.

Configuring the witness requires the Admin role.

How it decides a system is down

The witness watches both systems and needs a way to tell a real outage apart from a brief blip. Two settings control that:

• System down intervals — how many consecutive missed check-ins with the witness a system is allowed before it's actually declared dead and a transition is triggered. Higher values make the witness more tolerant of brief connectivity gaps, at the cost of a slower response to a real outage.
• Replication heartbeats — in addition to talking to the witness itself, the two systems can also exchange heartbeat signals directly over the replication link, as a second, independent signal of whether the other side is still alive. This is optional and has its own reconnection timeout; note that even with heartbeats turned off, the systems still independently try to detect a dead peer once they lose contact with the witness — heartbeats are a supplementary signal, not the only mechanism.

Authentication

The witness needs to log in as a StorONE user with access to both systems in the pairing, so it can check connectivity to each of them. When you enable the witness (or change its credentials), StorONE actually logs in with the user/password you supply and shows you the list of systems that login reaches — review that list carefully, since it's confirming exactly which systems the witness will be able to act on.

Floating IPs on automatic failover

Just like a manual promotion, an automatic failover triggered by the witness can also migrate paired client-facing NAS/Object floating IPs over to the newly-promoted system, so clients reconnect through the same address without you needing to intervene. This is controlled by the transition paired floating IPs setting. See Floating IPs for how floating IP pairing works in general.

Configuring the witness

Using the web interface

1. Open Application Volumes > Replication, then go to the Witness tab.
2. Click Edit.
3. Under Replication Witness Activation, enable or disable the witness.
4. Under Authentication, enter the User and Password the witness will log in with.
5. Under System Down Intervals, set the number of missed check-ins before a system is considered dead.
6. Under Heartbeats, choose whether to use replication heartbeats and, if so, their reconnection timeout.
7. Under NAS/Object Floating IPs, choose whether an automatic transition also migrates paired floating IPs.
8. Confirm the change — if enabling the witness for the first time, review the list of systems the supplied credentials can reach before confirming.

Using the CLI

replication witness settings edit
   [--enable|--disable]
   [--user=<name>]
   [--password=<password_general>]
   [--deadSystemsIndentificationIntervals=<numbers>]
   [--useReplicationHeartbeats|--doNotUseReplicationHeartbeats]
   [--ReplicationHeartReconnectionTimeoutInSeconds=<numbers>]
   [--transitionPairedFloatingIps|--doNotTransitionPairedFloatingIps]
   [--force]

Any option you leave out keeps its current value — you only need to supply the settings you're changing.

Option	Description
--enable	Enable replication witness.
--disable	Disable replication witness.
--user=<name>	Set the user for the replication witness.
--password=<password_general>	Set the user's password for the replication witness.
--deadSystemsIndentificationIntervals=<numbers>	When a system is disconnected from the witness, the number of iterations used to identify it as dead prior to the transition.
--useReplicationHeartbeats	Utilize heartbeat signals through the replication link. Even if disabled, other nodes still try to identify the cloud as dead after disconnecting from the witness.
--doNotUseReplicationHeartbeats	Do not utilize heartbeat signals through the replication link. Even if disabled, other nodes still try to identify the cloud as dead after disconnecting from the witness.
--ReplicationHeartReconnectionTimeoutInSeconds=<numbers>	If replication heartbeats are enabled, the allowed timeout in seconds for them.
--transitionPairedFloatingIps	When performing a replication transition, transition paired NAS/Object floating IPs.
--doNotTransitionPairedFloatingIps	When performing a replication transition, do not transition paired NAS/Object floating IPs.
--force	Skip the confirmation prompt.

Examples

Enable the witness with credentials and a 5-iteration dead-system threshold:

replication witness settings edit --enable --user dr-witness --password "correct-horse-battery-staple" --deadSystemsIndentificationIntervals 5

Turn on replication-link heartbeats with a 30-second reconnection timeout, without changing anything else:

replication witness settings edit --useReplicationHeartbeats --ReplicationHeartReconnectionTimeoutInSeconds 30

Disable the witness entirely:

replication witness settings edit --disable

Viewing witness settings

Using the web interface

Open Application Volumes > Replication, then go to the Witness tab. The current settings are shown at the top of the page.

Using the CLI

replication witness settings view

Viewing witness logs and heartbeat history

These are read-only views for troubleshooting the witness itself — the log records witness activity (connections, decisions, transitions it triggered), and the heartbeat history records the connectivity checks used to decide whether a system is alive.

Using the web interface

Open Application Volumes > Replication, then go to the Witness tab, and open the Logs or Heartbeats section.

Using the CLI

replication witness logs
replication witness heartbeats

Neither command takes any options.

---

See also: Replication · Promoting and monitoring replication · Floating IPs