Skip to content

Set Up High Availability

Add a second Flow Scheduler node that shares the database with the first, so that a standby takes over when the leader fails.

These steps turn an existing single node (node A) into a two-node cluster with a new node B. Repeat the node B steps for each further node. How the cluster behaves is described in High Availability.

Before you begin

  • The MetaMiner Database (MMDB) is an external PostgreSQL database that every node can reach. High availability is supported only with an external PostgreSQL database.
  • The Flow Scheduler runs on node A. See Enable the Flow Scheduler.
  • Node B can run the same MetaMiner build as node A, with a MetaMiner Server (MMS) and the Flow Scheduler on the same host. Every node must run the same build.
  • The clocks of all nodes are in sync (NTP).
  • Each node reaches the shared PostgreSQL database and Tableau Server. Inbound, only the MMS port (49714 by default) is open; port 49716 stays closed, also between nodes.
  • Node B has a valid MetaMiner licence under your licence terms.
  • If users sign in with a Personal Access Token or SSO, Tableau Server trusts the source IP of every node. See Sign-In Options.

Set the HA settings in MMA, never in MetaMinerServer.properties

A Flow Scheduler setting in server\MetaMinerServer.properties is fixed at launch and does not follow the preferred node. Set the two per-node settings below in the MetaMiner Administrator (MMA) on each node, and every other Flow Scheduler setting once, in MMA on the preferred node.

Steps

Prepare node A

  1. In the web interface on node A, open Admin > Settings > Backup and click Back up this node. See Backup and Restore.

  2. On node A, open MMA and choose Tools > FSS Configuration..., Properties tab. Type ha. or the key name in Filter: and set these two settings, clicking Apply after each:

    Setting Value on node A
    ha.lease.enabled true
    mms.url.local This node's own MMS address, for example https://<node-a>:<port>

  3. Restart the Flow Scheduler on node A (MetaMinerFSS in Windows Services). Node A takes the lease and becomes the leader.

  4. Check that node A is the leader: in the web interface, Admin > HA / Nodes shows node A as leader with a fresh Last update.

Add node B

  1. Install MetaMiner on node B with the same build as node A. Do not start its services yet.

  2. Copy these files from node A to node B, under C:\Program Files\Infolytik\MetaMiner\server\, while node B is stopped:

    File or folder Action
    config\config.json, plus config\config.ecf and config\config.ek when present Copy. Check that the database connection names the shared PostgreSQL host, never localhost. Copy the key file together with the encrypted file.
    config\mmext-services.json Copy.
    config-overrides.properties Copy, so that node B's Flow Scheduler starts with high availability on.
    config\saml\ (when SSO is used) Copy the whole folder.
    config\apitoken.json Do not copy. API clients live in the shared database. See Upgrade from the Shared Token.
    data\, logs\, backups\, config-snapshots\, MetaMinerWebApps.ini Do not copy. They belong to node A.

    Never start a node with high availability off

    A node with ha.lease.enabled off acts as the leader on its own, so two nodes would run the same flows. Copying config-overrides.properties makes node B start as a standby.

  3. On node B, open MMA, choose Preferences > Settings, General, select Enable Flow Scheduler (FSS) and save. If you use Personal Access Token sign-in, also add flowscheduler.auth.pat.enabled=true to node B's server\MetaMinerServer.properties (an MMS setting, not a Flow Scheduler setting).

  4. Start MetaMiner on node B from MMA (Start Server). MMA starts MMS and then the Flow Scheduler, which joins as a standby.

  5. On node B, in MMA Tools > FSS Configuration..., Properties tab, set the same two settings for this node and click Apply after each:

    Setting Value on node B
    ha.lease.enabled true
    mms.url.local Node B's own MMS address, for example https://<node-b>:<port>

    If node A has ha.node.id set, give node B its own value. Left unset, it is the host name.

  6. Restart the Flow Scheduler on node B.

Choose the preferred node

  1. In the web interface, open Admin > HA / Nodes and click the star on the node that should normally lead.
  2. From now on, change every other Flow Scheduler setting once, in MMA on the preferred node. The other nodes take the new value and apply it at their next Flow Scheduler restart, so restart the Flow Scheduler on every node after a change. MMA refuses a shared setting saved on another node and names the preferred node.
  3. Drill a handover while you watch: click Make Leader on the standby, check that schedules keep firing and Leadership events records the change, then hand leadership back.

Also shared by the cluster

API clients, allowed environments and Flow Scheduler admins live in the shared database, so you set them once. Create API clients in MMA on the leader; on a standby the API client screen is read-only. A Tableau Server login changed through the Tableau Credentials API on one node reaches every node.

Verify

  • Admin > HA / Nodes lists every node: one leader, the others standby, each with a fresh Last update.

  • On each node, the health page answers through MMS. Its custom.mmsFailover.role value is LEADER on one node and STANDBY on the others:

    curl -s "https://<node>:<port>/api/flowscheduler/health"

  • After Make Leader, the term in Leadership events goes up by one.

Troubleshooting

Both nodes show as standby

Neither node can write the lease. This is almost always connectivity to the shared PostgreSQL database. Check the database and the network from each node.

More cluster symptoms (leadership moving back and forth, a service stopping right after a restart): Troubleshooting.

Next steps