Document Processing¶
How MetaMiner reads the inside of Web Intelligence documents, universes and Crystal Reports, and the settings that control it.
How it works¶
Every pull first collects the basic properties of every object (name, owner, folder, dates). For three content types MetaMiner then opens the file and reads its structure. This is called deep processing.
| Content type | What MetaMiner reads | How |
|---|---|---|
| Web Intelligence | Data providers, queries and SQL, universe objects used, reports, variables, prompts | BusinessObjects SDK and the Web Intelligence RESTful Web Service |
| Universes (UNX) | Tables, columns, joins, contexts, classes and objects, filters, lists of values, parameters | BusinessObjects semantic layer SDK, one universe per transaction |
| Universes (UNV) | Connection, tables, classes and objects, contexts, joins, parameters | Universe file loaded from the File Repository |
| Crystal Reports | Tables and fields, commands, database logons, prompt groups, formulas, business views | Report file opened through the Crystal Reports processing engine |
Deep processing runs only for objects that are new or changed since the last pull. Deleted objects are removed from the MetaMiner Database (MMDB).
Edition
Deep processing of Web Intelligence, universes and Crystal Reports requires an edition that includes it. Professional edition does not. See Editions and Licensing.
Settings¶
All settings are in Preferences > Settings... and apply to every environment.
| Setting | Page | Default | Effect |
|---|---|---|---|
| Webi Metadata pull period | Pull | 5 min | How often Web Intelligence deep processing runs (webi.deep.pull.period) |
| Universe Metadata pull period (min) | Pull | 5 min | How often universe deep processing runs (universe.deep.pull.period) |
| Threads per Webi | Pull | 4 | Parallel Web Intelligence documents processed (threads.per.web) |
| Threads per datatype | Pull | 4 | Parallel workers per other object type (threads.per.datatype) |
| SQL Engine Enabled | Pull | On | Parses Web Intelligence data provider SQL and Crystal command SQL for database impact analysis (sql.engine.enabled) |
| Keep Web Intelligence versions? | Backup | Off | Saves a version of each changed Web Intelligence document (webi.keep.history) |
| Keep Universe versions? | Backup | On | Saves universe history and skips unchanged universes (unx.keep.history) |
| Versions Path | Backup | server\backup\ |
Folder for saved versions (versionsPath) |
See Pull settings and Backup settings for every field.
Web Intelligence through REST¶
MetaMiner validates the RESTful Web Service of every environment when you apply it and at each MMA start. A failed REST check marks the environment Failed. Set the address in Connection and Logon and test it with Test REST.
TODO: decide whether to publish the REST processing switch
Web Intelligence deep processing through REST is selected with the system property webi1440.enable (default false; the BI4 SDK processor is used otherwise). Product to confirm whether this switch is customer-facing and which processor is the recommended default for this release.
Web Intelligence versions¶
When Keep Web Intelligence versions? is on, MetaMiner checks changed documents every 5 minutes. It fingerprints the document file and, when the content differs from the last saved version, exports a BIAR into <Versions Path>\webi-history\. Older versions are purged on a rolling basis.
Version retention (system properties)
| Property | Default | Effect |
|---|---|---|
versioning.interval.minutes |
5 | Minutes between version scans |
versioning.batch |
200 | Documents checked per scan |
versions.max.total |
5 | Versions kept per document |
versions.max.per.day |
2 | Versions kept per document per day |
versions.rolling.days |
3 | Days covered by the per-day rule |
Change these only on advice from Infolytik Support.
Disk space
Version files can use a lot of space. Put Versions Path on a volume with room to grow.
Exclude content from processing¶
To skip deep processing of a content type on all environments, for example while diagnosing a problem, see Web Intelligence Processing Failures and Custom Settings.