Environments and the landscape
A customer does not have loose databases. It has a landscape: sources, ETL, a data warehouse and reports that belong together. And that landscape usually exists several times, in development, test, acceptance and production. Plainwater AI knows both ideas.
Environments
Every source belongs to an environment: production, acceptance, test or development. A customer can have several environments, each with its own sources.
Per environment you can set your own rules. A few examples:
- In development an agent may run free SQL and execute scripts.
- In production an agent may only look, and every change goes through a proposal a human approves.
- In test data access is set to open, because the test database contains no real people.
"Production" is not a property of a customer but of a source. A customer can have a production database and a development database at the same time, with very different rules.
The landscape
The landscape is a map of one environment: which systems there are, which databases, which pipelines, which datasets and reports, and how they are connected.
| Kind of node | Example |
|---|---|
| System | the ERP package, the HR system |
| Database and schema | the data warehouse, the staging schema |
| Pipeline and trigger | the nightly load, the hourly schedule |
| Dataset and report | the semantic model, the board report |
Connections say what happens: loads, reads, triggers, refreshes, depends on.
Drawing it together
An agent and a human draw the map together. The agent can measure (this pipeline writes to that table) or infer (this view probably reads from that schema). A human can confirm a connection. The map always shows whether a connection was measured, inferred or confirmed.
Only a human publishes the map. Publishing creates a new version; old versions are kept.
A starting point, not the truth
The published map goes along with every task in that environment. The agent then does not have to work out how everything fits together every time. But the map goes along as a starting point, not as the truth: an agent that measures something different may report it and propose a change.
The map also shows the customer's names as pseudonyms, as everywhere something goes to a model.
Example
A customer has a warehouse in three environments.
- In development an agent draws the landscape with a data engineer: six source systems, a staging schema, twelve pipelines, four reports.
- The engineer confirms the connections he knows and publishes the map.
- A week later a colleague asks why a report is empty. The agent gets the map, sees which pipeline feeds that report, and starts looking there.