Skip to content

How It Works

The MetaMiner Server runs a set of independent collectors that read Microsoft APIs on their own schedules and write into the MetaMiner Database; every 10 minutes it builds a snapshot from that database for the MetaMiner Client.

How it works

flowchart LR
  subgraph Microsoft
    ENTRA[Entra ID sign-in]
    API[Power BI admin, Fabric and Graph APIs]
  end
  subgraph MS[MetaMiner Server]
    COL[Collectors, each on its own schedule]
    SNAP[Snapshot builder, every 10 minutes]
  end
  COL -->|app-only token| ENTRA
  COL -->|read| API
  COL -->|write| DB[(MetaMiner Database)]
  DB --> SNAP
  SNAP -->|snapshot| MMC[MetaMiner Client]
  DB --> DASH[Usage dashboards]
  1. Sign in. When the server starts, it requests access tokens from Microsoft Entra ID for each enabled Power BI environment, using the registered application's client ID and secret. One token is for the Power BI and Fabric APIs, one for Microsoft Graph. Tokens are kept in memory only and renewed before they expire.
  2. Collect. Each kind of data has its own collector: workspaces, workspace scans, activity events, refreshes, apps, capacities, gateways, connections, Fabric items, item definitions, Capacity Metrics and the Entra directory. All collectors run in parallel. When a collector finishes a cycle it waits for its own interval, then starts again. See What MetaMiner Collects.
  3. Store. Each collector writes what it read into the MetaMiner Database (MMDB) and records how far it got (for example the last activity hour read or the last directory change token). After a restart, collectors continue from that point instead of starting over.
  4. Build the snapshot. Every 10 minutes the server reads the current Power BI data from the MMDB into an in-memory snapshot of each environment. This step makes no calls to Microsoft.
  5. Show it. The MetaMiner Client loads the snapshot when it connects. The Power BI Explorer checks for a newer snapshot every 10 minutes. The Usage dashboards read summary tables that the server rebuilds from the MMDB every few minutes.

How changes reach MetaMiner

MetaMiner does not rescan the whole tenant on every cycle. It asks Microsoft which workspaces changed and scans only those.

Step Collector Interval after a completed cycle
Find every workspace in the tenant Workspaces 24 hours
Find workspaces modified since the last check Modified workspaces 5 minutes
Scan new and modified workspaces, up to 100 per request, with lineage, data source details, semantic model schema and expressions Workspace scan 2 minutes
Rebuild the client snapshot from the database Snapshot builder 10 minutes

A change made in Power BI usually reaches the Power BI Explorer within the sum of these steps (estimate: under 30 minutes on a quiet tenant). The very first collection scans every workspace, so it takes longer on large tenants.

When Microsoft limits requests

Microsoft limits how often admin APIs may be called. When an API answers "too many requests", the collector waits for the time Microsoft asks for and then continues where it stopped. Other collectors keep running. Other errors make the affected collector retry with an increasing delay, up to 5 minutes between attempts.

Deleted items

When an item or workspace is deleted in your tenant, the next collection records the deletion. MetaMiner keeps an archive of deleted items for 180 days by default, then removes it. See Data Retention.

Key terms

Term Meaning
Collector One background job that reads one kind of data from one Microsoft API family
Workspace scan Microsoft's metadata scanning API: MetaMiner submits a list of workspaces and downloads the full metadata when the scan is ready
Snapshot The in-memory view of an environment that the client loads
Service principal The identity of the registered Entra application MetaMiner signs in as

Configuration changes need a server restart

The server reads the Power BI environment settings when it starts. After you add, edit or disable an environment in the MetaMiner Administrator, restart the MetaMiner Server.

The collection only reads

Every background call to Power BI, Fabric and Graph reads data. Three kinds of call use POST because Microsoft defines them that way, and none of them changes anything: submitting a workspace scan, getting a Fabric item definition, and running a read-only DAX query against the Capacity Metrics semantic model.