Why this matters
Tests catch the conditions you thought to check. Profiling measures distribution, null rates, cardinality and volume on a schedule, so you can see a column drift before anyone has written a test for it.
Before you start
A profile is a fingerprint you can compare over time
A single profile describes an asset. A series of profiles describes its behavior. The value is in the change: a null rate that moves from two percent to forty, a row count that halves, a distribution that shifts. None of those are errors in any single snapshot.
Sampling is a deliberate tradeoff
Profiling every row of a large table is expensive and usually unnecessary. Limiting how much data is sampled controls cost, at the price of sensitivity to rare values. Make the choice per asset.
Schedule to match how the data moves
Profiling frequency should follow the write pattern of the asset. Profiling an hourly table once a week gives you almost no ability to localize when something changed, while profiling a monthly table hourly just spends money.
The interactive demo
Work through the demo below. It runs in your browser with nothing to install. Watch for:
- Configuring a Profiling Agent to collect table data
- Limiting how much data is ingested for sampling
- Scheduling the Profiling Agent
This demo is also available on its own at the Learning Center.
After this module you should be able to
- Configure and schedule profiling for an asset
- Choose a sampling limit and explain the tradeoff you made
- Read a profile series to spot drift before it becomes an incident