Data Lineage: Keeping Data Flows and Transformations Traceable

Data lineage follows documented data dependencies

This requires more than similar filenames: the data versions and processing runs used must be linked. How much an answer can establish depends on the granularity and coverage captured.

How the report arrives at the mean of 20.0 °C

A fictional batch report B-01 contains a mean of 20.0 °C. The associated calculation run R-01 used three measurements: 19.8 °C, 20.0 °C, and 20.2 °C. The rule defined for this example calculates the arithmetic mean and rounds the displayed value to one decimal place.

It must be possible to navigate from the report to the run used and from there to the versions of the three inputs available at that time. Recalculation gives (19.8 + 20.0 + 20.2) / 3 = 20.0. A connection to today's measurement table alone is insufficient if its values have since been corrected.

Here, the lineage explains a calculation. Whether the mean of these three observations is suitable for a specific quality decision at all remains a separate substantive question.

What a correction means for known outputs

In the same example, 20.2 °C is corrected to 20.3 °C through a controlled process. A forward query identifies R-01 and B-01 as uses of the earlier value. Recalculation with the corrected input gives 60.1 / 3 = 20.033… °C; rounded to one decimal place, the display remains 20.0 °C.

The dependency is therefore affected even though the rounded number stays the same. The unchanged display does not establish whether the report must be supplemented, replaced, or reviewed again. That decision depends on its intended use and the applicable requirements.

B-01 retains its link to the original inputs. A new calculation receives its own run and, where appropriate, a new report version. This keeps the historical representation distinguishable from today's assessment. Unrecorded exports or manual copies, however, do not automatically appear in the query.

System, dataset, and field lineage

Granularity determines how precise a lineage answer can be and how much effort it takes to generate and maintain. At the highest level, system lineage shows, for example, that data moves from a production system to an analytics platform and then into a report. This view is useful for architectural overviews, but does not yet establish which value was actually transferred.

Dataset lineage here means the level of a dataset, such as a table, file, or set of events—not an individual table row. It connects such datasets or identifiable document versions. It can show that a batch report was generated from a particular set of tasks and a particular specification version.

Field lineage describes the dependencies of individual attributes. It becomes important when a target field is calculated from several sources, renamed, or converted to another format. Lineage for individual table rows (records) or values would be more detailed still. This may be useful for critical decisions, but is not economically or operationally necessary for every piece of information.

A good architecture therefore allows different views of the same chain. The overview remains understandable, while users can navigate to the relevant level of detail when needed.

What the graph's edges mean

A directed lineage graph may contain datasets, fields, processing runs, and outputs as nodes. Its edges must identify the intended relationship: an input to a run, an output generated from it, or a transfer into a report field.

An edge between two tables may describe a possible dependency in their schemas. It does not yet establish which rows a particular run actually read. For the mean-value example, the chosen level of detail must therefore extend to the inputs used.

Likewise, transporting an unchanged value must be distinguished from converting or aggregating it. Both may appear in the same graph, but raise different questions about traceability.

The transformation explains the transition to the result

Copying, filtering, conversion, or merging may take place between input and output. For a calculation, the rule used, parameters, and rounding must be documented in particular. An arrow labeled “processed” does not yet explain the mean.

W3C PROV distinguishes usage, generation, and derivation.^[1] These relationships help describe processing: a run uses inputs and generates an output. The fact that two datasets occur in the same system is not sufficient.

Data provenance also includes relationships of generation and participation: who created data, through which activity, and from which source? Data lineage focuses on the documented processing that connects two specific data versions.

Unambiguous identities and versions hold the chain together

A lineage breaks when different objects share the same name or an object's meaning changes over time. Table names, filenames, or report titles alone are therefore rarely stable identifiers. The system, namespace, object type, version, and, where applicable, validity period are also needed.

The same applies to processing steps. A calculation called “Yield” may use different logic after a rule change. If the old and new runs are linked only through the same name, an apparently continuous lineage emerges even though its semantics have changed.

W3C PROV treats revisions and specializations as relationships between distinguishable entities.^[1] This suggests a practical principle for operational systems: changed data versions and rules should remain identifiable rather than invisibly overwriting earlier meanings.

Lineage can be captured technically and within operational processes

Lineage metadata can be generated in different ways. Data pipelines can automatically report their inputs and outputs during a run. Databases and query tools can infer relationships from SQL, schemas, or execution plans. Interfaces can log the source, destination, and transfer status. Business applications can create the connection directly while a task is executed.

OpenLineage structures runtime metadata around events and the core elements Job, Run, and Dataset. Additional facets can express further properties of inputs, outputs, processing runs, and datasets.^[2] Apache Atlas represents lineage as relationships between typed metadata objects and supports queries over inputs, outputs, and processes.^[3]

Automatic capture reduces manual maintenance, but is not automatically complete. A technically detected table dependency may not capture the operational purpose of a calculation. Manually documented lineages can explain this purpose, but become outdated more easily. In practice, a combination is useful: technical relationships are generated automatically and supplemented with domain labels, criticality, and rules governing their applicability.

Lineage completeness and currency must themselves be checked

A lineage view can look polished yet still contain gaps. Undetected scripts, spreadsheets, manual exports, renamed fields, or external systems break the chain. Lineage metadata must therefore also meet defined quality requirements.

Attributes to assess include:

• Coverage: Are all sources and destinations relevant to the purpose captured?
• Accuracy: Do the dependencies shown reflect the actual processing?
• Currency: Has a change to a schema, interface, or rule been incorporated?
• Granularity: Is the chosen level sufficient for the specific question?
• Understandability: Can domain users assess the transformation and its meaning?
• Supporting evidence: Is it clear how a relationship was captured or inferred?

Lineage should therefore not be treated as a one-time documentation task. It is an ongoing metadata product maintained alongside the underlying systems and data models.

Which questions a lineage must answer

A lineage starts with a specific question, not the largest possible diagram. Depending on the purpose, it may be needed at the system, dataset, field, or individual-value level. Typical questions include:

• Origin: Which source did the information under consideration come from?
• Path: Which applications, interfaces, tables, or documents did it pass through?
• Transformation: Was it copied, converted, filtered, merged, aggregated, or manually supplemented?
• Version: Which version of a rule, schema, or calculation was applied?
• Time: When was each stage generated, and to which operational validity period did it refer?
• Dependency: Which downstream data products and decisions use the result?
• Responsibility: Which technical or operational process triggered the processing?

Not every application needs all levels. For a technical impact analysis, the relationship between tables may be sufficient. For a critical calculated quality value, it may be necessary to trace back to input fields, calculation parameters, and the raw data used.

Benefits in regulated product and batch processes

Not every data movement is equally critical in regulated processes. Risk-based lineage focuses on data that influences product quality, patient safety, release, evidence, or significant operational decisions.

In a deviation investigation, it can show which results depend on an incorrect input value or an incorrectly applied rule. During a system change, it supports assessment of which interfaces and reports need testing. During a review, it helps explain how a reported metric was derived from its sources.

Transparency alone does not provide the benefit. An appropriate response becomes possible only when lineage is connected to responsibilities, review rules, and controlled corrections. Lineage supplies the dependency map; substantive assessment determines the actions.

Data lineage, provenance, audit trails, and event sourcing

These concepts overlap, but answer different guiding questions. Data provenance broadly describes the entities, activities, and responsible agents through which information was generated or influenced. Data lineage focuses more narrowly on its path, transformations, and dependencies across datasets and systems.

An audit trail records relevant entries, changes, and deletions with identity, time, and, where applicable, a reason. It can be an important source for lineage, but does not automatically represent every data flow and derivation.

Event sourcing is an architectural principle in which application states are derived from a stored sequence of domain events. These events can provide a very good basis for derivation relationships. Nevertheless, which data objects originated from which events and transformations must also be modeled.

Data integrity refers to the completeness, consistency, and accuracy of data throughout its lifecycle. Lineage supports its assessment by making sources and processing traceable. A documented lineage does not, however, prove that the original value was correct or that every processing step was appropriate in substance.

What cannot be inferred from lineage

Validating interfaces and controls remains a separate task.

Whether the lineage also includes participants, reasons for decisions, or other conditions of generation depends on the model actually captured. A blanket boundary with provenance would obscure these different implementations.

Simply displaying existing relationships does not provide protection against changes or determine substantive decisions about corrections.

A lineage ends where the association is missing

In the mean-value example, B-01 can be explained because the inputs, calculation, and output are connected. Another file containing the same value is not an established dependency without a documented transfer relationship.

In practice, both matter: querying known data paths and identifying unknown transitions. Lineage narrows the scope of investigation; it does not itself decide which substantive action a correction requires.