What Is Data Profiling? How It Is Done, Best Practices and Tools [2026 Updated]

Data processing and analysis can’t happen without data profiling: reviewing source data for content and quality. As data gets bigger and infrastructure moves to the cloud, data profiling is increasingly important.

The stakes have not improved much: a 2017 Harvard Business Review study by Nagle, Redman and Sammon found that only 3% of companies’ data met basic quality standards, and co-author Thomas Redman reaffirmed the figure in 2020, with no newer industry-wide measurement published since. This guide explains what data profiling is, how data profiling is done step by step, and which profiling tools are actually maintained in 2026.

What is data profiling?

Data profiling is the process of reviewing source data, understanding structure, content and interrelationships, and identifying potential for data projects.

Data profiling is a crucial part of:

• Data warehouse and business intelligence (DW/BI) projects: data profiling can uncover data quality issues in data sources, and what needs to be corrected in ETL.
• Data conversion and migration projects: data profiling can identify data quality issues, which you can handle in scripts and data integration tools copying data from source to target. It can also uncover new requirements for the target system.
• Source system data quality projects: data profiling can highlight data which suffers from serious or numerous quality issues, and the source of the issues (e.g. user inputs, errors in interfaces, data corruption).

Data profiling involves:

• Collecting descriptive statistics like min, max, count and sum.
• Collecting data types, length and recurring patterns.
• Tagging data with keywords, descriptions or categories.
• Performing data quality assessment, risk of performing joins on the data.
• Discovering metadata and assessing its accuracy.
• Identifying distributions, key candidates, foreign-key candidates, functional dependencies, embedded value dependencies, and performing inter-table analysis.

Types of data profiling

There are three main types of data profiling: structure discovery, content discovery and relationship discovery. Some current frameworks split the work into five or six named types instead, adding labels like metadata discovery, single-field (column) discovery and cross-field discovery. Those finer-grained labels are subdivisions of the same three activities, not additional types.

Structure discovery

Validating that data is consistent and formatted correctly, and performing mathematical checks on the data (e.g. sum, minimum or maximum). Structure discovery helps understand how well data is structured. For example, what percentage of phone numbers do not have the correct number of digits.

Content discovery

Looking into individual data records to discover errors. Content discovery identifies which specific rows in a table contain problems, and which systemic issues occur in the data (for example, phone numbers with no area code).

Relationship discovery

Discovering how parts of the data are interrelated. For example, key relationships between database tables, references between cells or tables in a spreadsheet. Understanding relationships is crucial to reusing data; related data sources should be united into one or imported in a way that preserves important relationships.

If you meet the finer-grained vocabulary elsewhere, it maps cleanly onto these three: single-field profiling is content discovery at column grain, metadata discovery is structure discovery applied to types and formats, and cross-field or cross-table checks are relationship discovery. The step-by-step workflow in the next section uses the same vocabulary.

How is data profiling done, step by step

In practice, data profiling is done by computing a fixed set of statistics for every column, then comparing the results against what the data should look like. The six steps below cover the full cycle, from collection to the next scheduled run.

Step 1: Collect the data at its source

Profile the source system, not an already-transformed copy; transformations hide the problems you are trying to find. Confirm what fraction of the table the profile actually saw before trusting it. Many tools sample by default: Power Query profiles only the first 1,000 rows of a table, and Postgres’s default_statistics_target is 100, so a “complete” profile can quietly be a small sample. Widen the sample or run a full scan for the columns that matter.

Step 2: Compute the core column statistics

Run the same set of statistics on every column. Eight cover most of what a profiler can tell you:

• Null rate: the share of missing values per column. A 40% null rate on email in a customer table means most rows cannot be contacted.
• Distinct and unique count: how many different values appear. Sixty distinct values in a US state column means someone typed freehand.
• Cardinality ratio: distinct count divided by row count. Near 1.0 suggests a key; near 0 suggests a category.
• Min and max: range sanity. An order date of 1970 or a price of 0 shows up here first.
• Pattern and format distribution: the share of values matching an expected shape, such as valid email formats.
• Length distribution: the spread of string lengths, which catches phone numbers with too few digits.
• Duplicate rate on business keys: repeated values in a column that should identify exactly one entity, such as order IDs.
• Referential overlap: the share of foreign-key values with no match in the parent table, also called the orphan rate.

The vocabulary is standard across tools: null rate maps to GX Core’s expect_column_values_to_not_be_null and Soda’s missing_count() and missing_percent(), uniqueness to GX’s expect_column_values_to_be_unique, and duplicates to Soda’s duplicate_count(). Warehouse-native profilers compute the same set; BigQuery’s Knowledge Catalog (renamed from Dataplex Universal Catalog on April 10, 2026) reports null percentage, distinct percentage, min/max, quartiles and string lengths per column.

Step 3: Compare results against expectations

Statistics mean nothing without expected values. Write down what null rate, cardinality and range each column should have, from business rules, schema documentation or the owning team, and flag every delta. A 2% null rate is fine on a middle-name column and a failure on a primary key.

Step 4: Document findings where the team can see them

Put the results in a catalog entry, a shared document, or checks written as code so the profile is reproducible. A screenshot in a chat thread is not documentation. Checks-as-code (a GX suite or a SodaCL file) doubles as the expectation record for the next run.

Step 5: Fix issues at the source where possible

Upstream fixes beat downstream patches: a corrected input form or interface stops bad data permanently, while an ETL patch has to run forever. What cannot be fixed at the source becomes an explicit, documented ETL rule rather than a silent transformation.

Step 6: Communicate and schedule the next run

Share the findings with the teams that own the data, then put the next run on a schedule. A profile is a snapshot, and it decays as new data arrives. Warehouse-native scheduled scans exist for exactly this, so recurring profiling no longer requires a separate tool.

Data profiling as project strategy: Kimball’s four-step process

The six steps above are what you compute; they say nothing about when in a data project profiling should happen. Ralph Kimball’s four-step process places profiling at the points where it changes project outcomes, from the initial go or no-go call to tuning the ETL process.

Ralph Kimball, a father of data warehouse architecture, suggests a four-step process for data profiling:

• Use data profiling at project start to discover if data is suitable for analysis, and make a “go / no go” decision on the project.
• Identify and correct data quality issues in source data, even before starting to move it into target database.
• Identify data quality issues that can be corrected by Extract-Transform-Load (ETL), while data is moved from source to target. Data profiling can uncover if additional manual processing is needed.
• Identify unanticipated business rules, hierarchical structures and foreign key / private key relationships, use them to fine-tune the ETL process.

Data profiling and data quality analysis best practices

Basic data profiling techniques:

• Distinct count and percent: identifies natural keys, distinct values in each column that can help process inserts and updates. Handy for tables without headers.
• Percent of zero / blank / null values: identifies missing or unknown data. Helps ETL architects setup appropriate default values.
• Minimum / maximum / average string length: helps select appropriate data types and sizes in target database. Enables setting column widths just wide enough for the data, to improve performance.

Advanced data profiling techniques:

• Key integrity: ensures keys are always present in the data, using zero/blank/null analysis. Also, helps identify orphan keys, which are problematic for ETL and future analysis.
• Cardinality: checks relationships like one-to-one, one-to-many, many-to-many, between related data sets. This helps BI tools perform inner or outer joins correctly.
• Pattern and frequency distributions: checks if data fields are formatted correctly, for example if emails are in a valid format. Extremely important for data fields used for outbound communications (emails, phone numbers, addresses).

Data profiling tools in 2026

The profiling tool landscape has turned over almost completely since the classic tool lists in this category were written, and lists that have not been re-verified now point at retired software. Every fact below carries a date, checked in September 2026.

Open source profiling tools

Six open source options are worth knowing, and five of them shipped a release in September 2026 alone. The names are the tricky part: the most popular library has been renamed twice.

Tool	What it is	Latest release	Status
fg-data-profiling	Python one-line profiling reports; formerly ydata-profiling, formerly pandas-profiling	4.20.0, Sept 11, 2026	Community-maintained (Data-Centric-AI-Community)
GX Core (Great Expectations)	Expectation-based validation framework	1.23.2, Sept 28, 2026	Apache 2.0; under Fivetran stewardship since May 13, 2026
Soda Core	YAML check language (missing_count, duplicate_count)	v4.25.0, Sept 23, 2026	Actively maintained
AWS Deequ	Unit tests for data on Apache Spark	2.1.0, Sept 16, 2026	Actively maintained; Spark only
whylogs	Data logging for ML pipelines	v1.6.4, Dec 3, 2024	Stalled; no release in about 21 months
OpenMetadata profiler	Built into the OpenMetadata catalog	2.0.2, Sept 16, 2026	Active; requires OpenMetadata

whylogs has not shipped a release since December 2024, roughly 21 months as of September 2026, while every other tool on this list released within the past few weeks; treat it as stalled. fg-data-profiling is the second rename of this library (pandas-profiling became ydata-profiling, which became fg-data-profiling), so tutorials written for the old package names will point you at a package that no longer receives updates.

OpenMetadata’s profiler is only worth adopting if you run OpenMetadata as your catalog anyway. GX Core’s move to Fivetran stewardship in May 2026 has not slowed releases; 1.23.2 shipped within days of the SQLAlchemy 2.1.0 release that broke earlier versions on Python 3.11 and newer. Soda’s documentation also moved to its v3 generation, so links in older Soda tutorials may no longer resolve.

As for the names on this page’s old list: Talend Open Studio was retired by Qlik on January 31, 2024 and is no longer hosted or updated; Informatica has been marketed as Informatica from Salesforce since the acquisition closed on November 18, 2025; and SAS DataFlux’s capabilities were absorbed into SAS Viya, so the DataFlux name no longer appears on SAS’s data quality pages.

Commercial profiling and observability tools

On the commercial side, the profiling job increasingly ships inside broader data quality and observability platforms, and public pricing is rare. All of these do more than profiling; they bundle it with monitoring, lineage or governance, so compare against the job you actually need done.

Tool	Published pricing	Notes
AWS Glue DataBrew	$1.00 per 30-minute interactive session; $0.48 per node-hour for jobs	Visual profiling and prep; AWS’s own example: a 10-minute, 5-node job costs $0.40
Monte Carlo	Quote only; credit-based	ML-based monitors, lineage, and incident alerting
Ataccama	Quote only	Priced on named users, data objects and DQ configurations; enterprise quality plus governance
DQLabs	Quote only	Markets “the quote you sign is the cost you pay” (the vendor’s own positioning)
decube	$175/user/month Starter; 10-user minimum, from $21,000/yr	Publishes full tier pricing on its site

Quote-only pricing is the category norm: Monte Carlo sells credits at rates that depend on tier (and recently moved from montecarlodata.com to montecarlo.ai), Ataccama quotes across three dimensions, and DQLabs, despite marketing predictable pricing, still publishes no dollar figures. decube is the exception worth crediting; its published Starter tier works out to at least $21,000 per year at the 10-user minimum, which lets you budget before a sales call. Before buying any separate tool, teams already on Snowflake or BigQuery should check the warehouse-native options first: Snowflake’s Data Metric Functions (Enterprise Edition) and BigQuery’s Knowledge Catalog profile scans compute the same core statistics on a schedule, inside the warehouse.

Data profiling vs data quality vs data observability

Profiling describes what the data currently looks like: the statistics from the step-by-step section above. It tells you whether the data looks right, not whether it is right. A revenue column loaded in cents instead of dollars still profiles as healthy, with a consistent format, no nulls and a plausible range, even though every figure is a hundred times off.

Data quality assessment adds the judgment. It compares the profile against defined standards or business rules, decides whether the data is fit for use, and drives the cleansing work that follows. Same statistics, plus a verdict.

Data observability runs those signals continuously: freshness, volume, schema drift and anomalies, watched over time instead of checked once. Tools like Monte Carlo and warehouse-native metric functions sit at that end of the spectrum; fg-data-profiling and GX Core sit at the profiling and validation end.

Data profiling in a cloud-based data pipeline: the need for speed

Traditional data profiling, as described in this post, is a complex activity performed by data engineers prior to, and during, ingestion of data to a data warehouse. Data is meticulously analyzed and processed (with partial automation) before it is ready to enter the pipeline. Today more organizations are moving data infrastructure to the cloud, and discovering that data ingestion can happen at the click of a button. Cloud data warehouses, data management tools and ETL services come pre-integrated with hundreds of data sources. But if you can click a button and move data instantly into your target system, what about data profiling? Data profiling is more crucial than ever, with huge volumes flowing through the big data pipeline and the prevalence of unstructured data.

The 2026 reality is that profiling is becoming a built-in of the platforms themselves. Snowflake ships Data Metric Functions for null counts and update frequency, plus Cortex Data Quality, which, in Snowflake’s own description, agentically suggests quality checks from your metadata. BigQuery’s Knowledge Catalog runs scheduled profile scans against your tables. The directory compares these tools, with real pricing, side by side.

FAQ

How is data profiling done?

Collect the data at its source, compute per-column statistics (null rate, distinct count, cardinality ratio, min and max, pattern and length distributions, duplicate rate and referential overlap), compare the results against expected values, document the findings where the team can see them, fix issues at the source where possible, and repeat on a schedule. See the step-by-step section above for the full workflow.

What are the three types of data profiling?

Structure discovery validates format and consistency with mathematical checks, content discovery finds row-level errors, and relationship discovery maps how tables and keys interrelate. Finer-grained labels you may see elsewhere, such as single-field, cross-field or metadata discovery, are subdivisions of these three activities rather than additional types.

Is data profiling the same as data cleansing?

No. Profiling describes the data’s shape (statistics, patterns, distributions) to find where problems likely exist; cleansing is the separate work of fixing them. The two are not interchangeable: a column can profile as perfectly healthy, with no nulls and a consistent format, and still contain values that are simply wrong.

How often should data be profiled?

At project start, first of all: Kimball’s go or no-go decision depends on an early profile. Then on a schedule once pipelines run continuously, because a profile is a snapshot that decays as new data arrives. Warehouse-native scheduled scans in Snowflake and BigQuery make recurring profiling practical without a separate tool; match the frequency to how fast the data changes.