Data Provenance: Tracing How Data Originates and Is Created

Naming a source does not explain how something was generated

Naming a device does not establish which measurement operation generated a displayed value. An investigation needs the connection between the source, the specific activity, and the result. W3C PROV provides a model of entities, activities, and agents for this purpose.[1]

Which information explains the generation process

An identifier on an intermediate product initially refers to the product itself. Establishing its provenance requires a different connection: the processing operation, its inputs, and the result it generated. The batch example below identifies these elements separately.

The FDA describes metadata as information needed to understand data, including time, user, and equipment references.[6] A provenance model associates this information with the generation process. This keeps a record of manufacturing distinguishable from a later review of that same record, for example.

The basic W3C PROV model

With the PROV family, the World Wide Web Consortium published a domain-independent model for representing and exchanging provenance information. The conceptual data model PROV-DM distinguishes three particularly fundamental categories: Entity, Activity, and Agent.^[1]

Entity: The object used or generated

In a processing operation, the input material and intermediate product can be modeled as entities. An SOP version or an analysis result can also be an entity in its own right. Different states of the same thing can be considered separately.[2]

Activity: The operation that uses or changes something

Processing, rework, and review are different activities. This distinction allows their respective inputs and results to be assigned separately, even when they concern the same batch.

Agent: Participation linked to responsibility

A person performing an activity, an organization conducting a review, or a participating software system can act as an agent. The association describes the modeled responsibility relationship; it does not establish legal responsibility.[1]

Relationships make provenance understandable

The real informational value comes from the relationships between these elements. An activity can use one entity and generate another. It can be associated with an agent. An entity may have been derived from an earlier entity.

The W3C model is explicitly domain-independent and can be extended with domain-specific information.^[1] It therefore does not prescribe which specific batch, test, or release data an organization must capture.

Through PROV-O, the W3C also provides an ontology for representing provenance information in machine-readable form and exchanging it between different applications.^[3]

Distinguishing input material, processing, and result

A fictional processing operation V-07 uses material M-12 and generates intermediate product Z-04. M-12 and Z-04 are entities in the model; V-07 is the activity. Person P-03 is associated with it as an agent.

“V-07 uses M-12” describes usage; “V-07 generates Z-04” describes generation. The derivation relationship between Z-04 and M-12 records the connection between the result and its input material. When associating an agent with an activity, PROV can also specify a plan: the intended steps on which the agent relies—for example, the applicable SOP version here.[1]

If Z-04 is reworked, another activity is added. Its result receives a distinguishable reference. The first processing operation remains part of the prior history; rework does not replace it. Which quantities and properties must be documented depends on the operational question.

A subsequent review is also a separate operation. It may assess the records, but it is not the activity that manufactured the material. This separation prevents “manufactured,” “documented,” and “reviewed” from becoming a single ambiguous relationship in the model.

Capturing relationships or reconstructing them later

During the execution of V-07, the identifiers of the material used and the intermediate product generated can be connected directly. Capturing this information as part of the process reduces later effort to establish these relationships. Its accuracy still depends on whether the material actually used and the correct operation were recorded.

If the connection is inferred later from reports, timestamps, or filenames, it should remain identifiable as a reconstruction. An assumed relationship is not equivalent to an unambiguously evidenced association. The capture source, uncertainty, and subsequent correction therefore form part of the assessment of a provenance model.

Even a model captured during work may have gaps. Its coverage must be assessed against the specific purpose: a material reference may be sufficient to identify the input material, but not to establish a particular measurement method.

An analysis result also has a history

Calculating a metric from the records produces another result with its own inputs and processing steps. Its provenance must be distinguished from that of the product being analyzed. A subsequent change to an input may provide grounds to review the metric again.

For process mining, it must remain traceable which set of events was used in the analysis and how it was prepared. Provenance supports this explanation. It neither establishes a cause for observed differences nor automatically demonstrates that an analysis method is suitable.

Data provenance and related concepts

These terms are often used together, but answer different questions.

These areas may overlap. An audit trail may contain important provenance information. Data lineage may make derivations and transformations visible. An event-based system may make it easier to build a provenance model. None of these concepts, however, is simply interchangeable with data provenance.

The distinction from data integrity is particularly important. Provenance can help assess a data item's origin and processing. It does not guarantee that the original observation was factually correct or that all required controls were followed.

Relevance to regulated process data

Regulatory requirements differ by industry, product, and specific use case. Nevertheless, guidance for GxP-regulated environments illustrates the importance of data context and the entire data lifecycle.

PIC/S describes the data lifecycle as the connection between generation, processing, reporting, review, use in decisions, storage, and later deletion.^[4] The MHRA emphasizes that data should remain attributable, contemporaneous, complete, and consistent throughout its lifecycle, among other attributes.^[5] The FDA likewise emphasizes that system design and controls should make errors, omissions, and suspect results detectable throughout the data lifecycle.^[6]

These regulatory sources do not define a general provenance system that is mandatory across industries. They do, however, show why isolated values and final documents alone are often insufficient in controlled processes: decisions must be assessable in the context of the underlying data and records.

Data provenance can give this context a technical structure. The information actually required must follow from the respective process, risk, and regulatory framework.

What data provenance does not do

• Provenance does not guarantee that an input is true. A traceably recorded measurement may still be wrong.
• An extensive log is not automatically a provenance model. Events must be interpretable in their operational context.
• Provenance and immutability are different properties. Provenance relationships can be represented without thereby proving that they are technically tamper-proof.
• More data is not automatically better data. Capture scope and granularity must fit the purpose, risk, and actual process.
• A model does not by itself establish interoperability. Shared terms, identifiers, and exchange formats must also be implemented consistently in practice.

The provenance question determines the required scope

For Z-04, specific questions can now be asked: which material went in? Which operation generated it? What further processing followed? These questions require distinguishable objects and unambiguous relationships, not the largest possible list of context fields.

A good provenance model also shows where its answers end. If material usage is recorded but the measurement method is only assumed, that distinction must be preserved. This turns provenance into a verifiable explanation.