Collection Processors¶
Every collector MetaMiner runs for a Power BI environment: its name in logs and the state report, cadence, the Microsoft API it calls, what it needs, and how to switch it off.
How collectors run¶
Each enabled Power BI environment runs all collectors at the same time, each on its own schedule. A collector works through its data in steps (for example one batch of workspaces, one hour of activity) and then waits for its cadence before the next complete cycle. After an error it retries after 1, 5, 10, 30 and 60 seconds, then every 5 minutes, keeping its place. When Microsoft throttles it (HTTP 429), it waits as long as Microsoft asks, or 20 minutes. Collectors save their progress in the MetaMiner Database (MMDB), so a restart continues where they stopped.
Separately, every 10 minutes (pbi.env.pull.period), the MMS rebuilds the Power BI view the MetaMiner Client shows from the MMDB. That step calls no Microsoft API.
| Column | Meaning |
|---|---|
| Name | As written in server.log, statereport.log and thread names |
| Cadence | Wait between two complete cycles |
| Needs | Microsoft permission or setting beyond the read-only admin API access every collector uses |
Power BI collectors¶
| Name | Cadence | Microsoft API | Collects | Needs |
|---|---|---|---|---|
GET_WORKSPACES |
24 hours | Power BI admin: list workspaces | All workspaces of the tenant | - |
GET_MODIFIED_WORKSPACES |
5 minutes | Power BI admin: modified workspaces | Workspaces changed since the last check, to be scanned again | - |
WORKSPACE_SCAN |
2 minutes | Power BI admin: workspace scanner (scan request, status, result) | Reports, dashboards, datasets with tables and expressions, dataflows, datasources, Fabric artifacts, item users; up to 100 workspaces per batch | Admin API tenant settings for detailed metadata and expressions, for full detail |
ACTIVITY_EVENTS |
10 minutes | Power BI admin: activity events | The activity log, in one-hour windows; first run reaches back about 28 days | - |
REFRESHABLES |
10 minutes | Power BI admin: capacity refreshables | Dataset refresh history | - |
APPS |
60 minutes | Power BI admin: apps | Published apps | - |
CAPACITIES |
3 hours | Power BI admin: capacities | Capacities and their SKUs | - |
DATAFLOW_DEFINITIONS |
3 hours | Power BI: dataflow definition | Dataflow definitions | Service principal access to the dataflow's workspace |
FCM |
60 minutes | Power BI: execute queries on a semantic model | Fabric Capacity Metrics: capacities, items, hourly metrics, 30-second CU detail, storage; first run reaches back 14 days | Contributor or Admin on the Capacity Metrics app workspace; semantic model configured in the environment |
Fabric collectors¶
| Name | Cadence | Microsoft API | Collects | Needs |
|---|---|---|---|---|
FABRIC_ITEMS |
3 hours | Fabric admin: items | All Fabric items (lakehouses, warehouses, notebooks, pipelines and others) | Tenant setting that lets service principals use Fabric APIs |
FABRIC_ITEM_DEFINITIONS |
3 hours (pbi.fabric.item.definitions.interval.minutes) |
Fabric: get item definition | Definitions of reports, semantic models, Gen2 dataflows, notebooks, pipelines, eventstreams, KQL items, reflexes, for new and changed items | Same, plus Admin, Member or Contributor role of the service principal (or its group) on the workspace; for reports, write access to the semantic model |
GATEWAYS |
3 hours | Fabric: gateways | Gateways and members | Same tenant setting |
CONNECTIONS |
60 minutes | Fabric: connections | Gateway and cloud connections | Same tenant setting |
Microsoft Graph collectors¶
| Name | Cadence | Microsoft Graph API | Collects | Needs |
|---|---|---|---|---|
GRAPH_USERS |
30 minutes | Users (changes) | Users | User.Read.All or Directory.Read.All |
GRAPH_GROUPS |
30 minutes | Groups (changes) | Groups | Group.Read.All or Directory.Read.All |
GRAPH_GROUP_MEMBERS |
30 minutes | Group members (changes) | Group membership | GroupMember.Read.All or Directory.Read.All |
GRAPH_SERVICE_PRINCIPALS |
30 minutes | Service principals (changes) | Service principals | Application.Read.All or Directory.Read.All |
GRAPH_APPLICATIONS |
30 minutes | Applications (changes) | App registrations | Application.Read.All or Directory.Read.All |
GRAPH_SUBSCRIBED_SKUS |
24 hours | Subscribed SKUs | Licence SKUs owned and consumed | LicenseAssignment.Read.All or Directory.Read.All |
GRAPH_USER_LICENCES |
30 minutes | Users (licence fields, changes) | Licences assigned to users | Same |
GRAPH_GROUP_LICENCES |
30 minutes | Groups (licence fields, changes) | Licences assigned through groups | Same |
The Graph collectors read only changes after their first full read. If they have not run for 30 days, they read everything again.
Turn a collector off¶
Use this only on Support advice, for example while Microsoft throttles one API heavily.
# Stop the Capacity Metrics collector for environment 1700000000000
pbi.1700000000000.FCM.processor.enabled=false
| Key | Default | Effect |
|---|---|---|
pbi.<envId>.<Name>.processor.enabled |
true |
false stops one collector for one environment. <envId> is the numeric ID of the environment in config.json. |
pbi.api.broker.enabled |
true |
false stops all collectors for all environments |
Both need a restart of the MetaMiner Server. Data already collected stays visible.
Rollup jobs for the usage dashboards¶
Job (name in mms.silver_job_runs) |
Runs | Reads | Feeds |
|---|---|---|---|
pbi-usage-rollup |
Every 5 minutes | Activity events, refresh history | Activity and refresh figures on the dashboards |
pbi-license-waste |
Every 6 hours | Graph users, groups, licences, activity | Licensing and the Executive seat waste |
fcm-capacity-rollup |
Every 5 minutes | Capacity Metrics detail | Capacity page |
See Data Freshness.