About Initial and Incremental Pull¶

MetaMiner Server (MMS) needs to be aware of any data change on your BI landscape. For this, a fast and reliable data exchange between MMS and Tableau server(s) is vastly important.

Tableau Server does not support push-based communication, so MetaMiner Tableau Edition uses pull technology, when it make requests to Tableau like "Is there any new data for me?" from time to time.

Initial Pull¶

When you add a new Tableau environment to your MetaMiner configuration, MMTE Server performs the initial pull. It could be slow, may take up to 8 hours, because no information was previously found at MMTE Server disposal.

You can track the progress of the initial pull if you connect to MMTE Server with MMTE Client and run Connection Manager:

Pull Progress

Incremental Pull¶

Later MMTE Server periodically reads your corporate data from Tableau Server. This is known as an incremental pull, because MMTE Server downloads only changes to the existing data. Therefore it is much faster than the initial pull and usually takes several minutes or even seconds to complete.

There are two worker processes running simultaneously as a part of each MMTE Server for synchronization with Tableau server(s):

Shallow pull worker
Deep pull worker

Shallow Pull¶

Shallow (or: Regular) pull worker normally runs regularly on schedule. Its time interval is specified in the tab.env.pull.period property and is 10 minutes by default.

You can also start shallow pull earlier:

In MMTE Client, run the Force on Demand command to pull data for a particular environment
In MMTE Client, click the Refresh button in the Status Bar to pull the latest available snapshot to the MMTE client

Both actions initiate an immediate shallow pull.

On shallow pull, MMTE analyzes objects metadata (mainly from Tableau Repository) and detects objects that are changed (for example, their update timestamp has been changed). If it detects changed objects, it updates their metadata in MetaMiner Database and adds them to the queue for the deep pull worker.

Deep Pull¶

Deep pull worker activates when it finds any object in its queue. Then it downloads the full data of corresponding workbooks/data sources and inspects this data to track all changes in the internal object content.

The Reprocess command initiates deep pull.

Deep pull is slow, it analyzes object files and detail information about them: e. g. utilized table columns. As it is slow and it may fail, there is a special feature for running it individually for a particular BI asset or a set of objects and for tracking deep pull status.

To run deep pull for a particular BI asset or a set of objects:

Open Tableau Explorer.
Select an object or a set of objects.
Right-click and choose Reprocess from the context menu.
Repeat this step for all objects.
Select the environment where all these objects are located.
Right-click it and run the Force on Demand from the context menu to pull data for a particular environment.

The selected object(s) are added to the deep pull queue. Deep pull worker detects them and starts analyses.

To track deep pull status:

Open Tableau Explorer.
Select an object you want to track.
Right-click and choose Properties from the context menu.
In the Properties dialog window, click the Show deep pull status button.

The Status dialog box is displayed. It shows the current status of deep pull for this object.

The following values are possible:

Value	Description
UNKNOWN	This workbook/data source was not processed since last launch of MMS, therefore MMTE has no information about process results.
PROCESSING	First time deep pull is currently running.
REPROCESSING	Deep pull was already run, and later Reprocess was called, and another deep pull is now in progress.
SUCCESS	Deep pull has completed successfully.
FAILURE	Deep pull has completed with an error.

Display of the status is designed for Infolytik Technical Support duty, to help Infolytik personnel understand which information is at MMTE disposal and which is not. With it, they can run Reprocess and then view its results.

Monitoring Pull Status¶

You can monitor progress of the current shallow and deep pull commands and also view the log of all previous pulls in the Data Refresh Statistics dialog window:

Data Refresh Statistics

To open this dialog window:

Open Tableau Explorer.
Select an environment you want to check.
Right-click and choose Pull Metrics from the context menu.

The Data Refresh Statistics dialog box is displayed.