Monitors
Some questions you do not ask once but every night. Did all loads succeed? Are there tables without new rows since yesterday? Has the number of nulls in a column suddenly doubled? A monitor asks such a question at a fixed moment, and gives a verdict.
What a monitor is
A monitor has:
- A task in plain language. For example: "Check whether all of last night's loads succeeded and whether every table in the staging schema has new rows."
- A schedule. When it runs: every day at 06:00, every hour on working days, every Monday. The schedule works like a trigger in Azure Data Factory.
- A source. What it looks at.
At the agreed moment the platform starts an agent with that task. The agent looks, and gives a verdict: fine, or a deviation.
Always look only
A monitor always runs in the work mode "look only". It reads and judges, but changes nothing. A monitor can therefore never "fix" a problem itself in a way you did not want.
The verdict
A monitor has to give a verdict. If the agent gives none, for example because it got stuck, that counts as a deviation. Silence is never "fine".
- Fine: nothing happens, except that the run is recorded.
- Deviation: a finding is created. You see it in the web app, with what the agent saw.
Each monitor has at most one open finding. If the problem persists, findings do not pile up.
A monitor that keeps failing
If a monitor fails three times in a row, the platform switches it off and reports it. A check that never works does not give a false sense of security.
Time zone
A monitor uses the time zone of whoever created it. "Every day at 06:00" means 06:00 for that person, in summer and winter time alike.
From finding to conversation
From a finding you start a conversation. The agent in that conversation gets the finding and can dig further, possibly in a different work mode.
Examples
| Monitor | Schedule |
|---|---|
| Did all of last night's loads succeed? | daily 06:30 |
| Does a warehouse table have more than 5 percent nulls in a key column? | daily 07:00 |
| Is a report dataset older than 24 hours? | hourly on working days |
| Are there new tables without a primary key? | weekly on Monday |