Failover and Upgrades¶

What happens when a Flow Scheduler node fails, and how to hand over leadership, upgrade a cluster and plan maintenance without losing scheduled runs.

What happens on failover¶

In high availability, one node is the leader: it fires schedules, tracks runs and sends e-mail. The other nodes are standbys. Timings below use the default settings (lease 30 seconds, heartbeat 10 seconds, MMS grace 30 seconds).

Event	What happens	Takeover time
You click Make Leader on a standby	The leader steps down and the chosen standby takes over.	Within about 20 seconds
The leader's Flow Scheduler stops cleanly (service stop, Windows update)	The leader releases the lease on shutdown and a standby takes it. When the old leader comes back it stays a standby, unless it is the preferred node.	About 10 seconds
The leader crashes (power loss, process killed)	The lease expires unrenewed, then a standby takes it.	About 40 seconds at most
The leader's own MetaMiner Server (MMS) stops answering	Runs that need MMS wait. After `ha.leader.mms.grace.seconds` (30) the leader hands over to a standby whose MMS is healthy. If no standby has a healthy MMS, the leader keeps leadership.	About 30 seconds plus the handover
The leader loses the shared database	The leader cannot renew the lease and demotes itself. A standby that still reaches the database takes over. Late writes from the old leader are rejected.	About 40 seconds at most
The shared database is down for every node	No node can be leader. Nothing is scheduled until the database returns.	-
A standby stops	No effect on scheduling. That node's own backups and its web interface are unavailable until it returns as a standby.	-
The preferred (starred) node returns	It rejoins as a standby. After it has been healthy for `ha.preferred.stable.seconds` (60), leadership moves back to it.	About 60 seconds after it is healthy

Runs during a takeover¶

Flows already running on Tableau Server keep running. The new leader picks them up and keeps tracking them.
A chain interrupted by a failover or restart is marked FAILED, and its remaining steps do not run. Run the chain again.
A duplicate submission refused by Tableau during the takeover is recorded as COALESCED. It is not a failure and sends no failure e-mail.
Run on a task can be clicked on any node; the leader starts it. Running a flow directly with Run Now works only on the leader's node.

Before you begin¶

You need a Flow Scheduler admin account for Admin > HA / Nodes and Make Leader, and Windows administrator rights on each node.
Take a backup on each node before an upgrade: Admin > Settings > Backup > Back up this node, with the label before-upgrade. See Backup and Restore.

Steps¶

Hand over before planned maintenance on the leader¶

Open Admin > HA / Nodes.
On the standby that should take over, click Make Leader and confirm.
Wait until HA / Nodes shows that node as the leader.
Do the maintenance on the old leader. It is now a standby.

Upgrade a high-availability cluster¶

Upgrade the standby first, then the leader. Upgrade every node on the same day: the nodes must run the same MetaMiner build.

Back up each node (see Before you begin).
On the standby, stop MetaMiner and wait until every service reports STOPPED. If a service is started again too early, the new process exits at once.
Install the new MetaMiner release on the standby and start it.
Check Admin > HA / Nodes: the upgraded node is a standby with a recent Last update.
Stop MetaMiner on the leader. Leadership moves to the upgraded standby in about 10 seconds.
Install the new release on the old leader and start it. It rejoins as a standby.
Check Admin > HA / Nodes: one leader, the other nodes standby, all with recent heartbeats.
Optional: click Make Leader to move leadership back, or let the preferred node take it back automatically.

The first upgraded Flow Scheduler to start updates the shared database schema automatically. The changes are additive, so the node still on the old build keeps working until you upgrade it.

Keep the builds equal

Do not leave the cluster on mixed builds. Finish upgrading every node in the same maintenance window.

If you upgrade from a release that used one shared API token, see Upgrade from the Shared Token.

Start and stop a node correctly¶

Start MetaMiner with Start Server in MetaMiner Administrator (MMA). It starts MMS and then the Flow Scheduler. Starting the metaminer service alone from Windows Services does not start MetaMinerFSS.
After you stop a service, wait until it reports STOPPED before you start it again.

Restart after a full outage¶

Start the shared PostgreSQL database first.
Start MetaMiner on the nodes in any order. The first Flow Scheduler to start becomes leader; the others join as standbys.
Scheduled runs missed within startup.catchup.window.minutes (60) run once as catch-up runs, unless the next scheduled run is within recovery.window.hours (4). Older missed runs are skipped.

Plan Tableau Server maintenance¶

Choose one:

Approach	How	Result
Pause the environment	In MMA, Tools > FSS Configuration... > Environments, clear Enabled for the environment. Select it again afterwards.	No scheduled, Run Now or catch-up runs. On re-enable, a slot missed in the last 10 minutes still runs.
Let runs wait	Do nothing.	Scheduled runs are held as `SERVER_DOWN` and start when Tableau Server answers again, unless the next scheduled run is within `recovery.window.hours` (4). A manual run is not held; it fails with `START_FAILED`.

Change the Tableau service account password¶

Change it once, through the Tableau Credentials API; every node takes the new value. Chain steps that fail because Tableau rejected the service account sign-in during the change do not count toward automatic disable.

Verify¶

Admin > HA / Nodes shows exactly one leader, and every node has a recent Last update.
Leadership events lists the handover you expected, with a higher term.
Scheduled tasks keep firing: check Admin > Task Analyze > Monitor after the next slot.

Test the failover path once a month: use Make Leader to move leadership to a standby, check that schedules keep firing, then move it back.

Troubleshooting¶

Leadership moves back and forth

Lease renewals fail now and then, from database latency or clock differences. Check database health and NTP time sync on every node. Consider a higher ha.lease.ttl.seconds.

A node stays standby after a restart

Another node holds a valid lease. This is expected: leadership does not move back unless a node is starred as the preferred node. Use Make Leader if you want it back.

More symptoms: Troubleshooting.