Module 5 of 8 · 10 min

# Observability and Profiling

Measuring the shape of data over time, and sampling sensibly at scale.

## Why this matters

Tests catch the conditions you thought to check. Profiling measures distribution, null rates, cardinality and volume on a schedule, so you can see a column drift before anyone has written a test for it.

## Before you start

### A profile is a fingerprint you can compare over time

A single profile describes an asset. A series of profiles describes its behavior. The value is in the change: a null rate that moves from two percent to forty, a row count that halves, a distribution that shifts. None of those are errors in any single snapshot.

### Sampling is a deliberate tradeoff

Profiling every row of a large table is expensive and usually unnecessary. Limiting how much data is sampled controls cost, at the price of sensitivity to rare values. Make the choice per asset.

### Schedule to match how the data moves

Profiling frequency should follow the write pattern of the asset. Profiling an hourly table once a week gives you almost no ability to localize when something changed, while profiling a monthly table hourly just spends money.

## The interactive demo

Work through the demo below. It runs in your browser with nothing to install. Watch for:

*   •Configuring a Profiling Agent to collect table data
*   •Limiting how much data is ingested for sampling
*   •Scheduling the Profiling Agent

This demo is also available on its own at [the Learning Center](/learning-center/product-demo-observability-profiling).

## After this module you should be able to

*   ✓Configure and schedule profiling for an asset
*   ✓Choose a sampling limit and explain the tradeoff you made
*   ✓Read a profile series to spot drift before it becomes an incident