Validating Knowledge Models: Controlled Checks of Rules and Relationships

Validating Knowledge Models Means Assessing Their Fitness for Purpose

Validation therefore does not begin with a single checking tool. First, the intended use of the model and the errors that would matter in that use are defined. A model for classifying batch states needs different checks from a model that describes roles, permissions, or material relationships.

Technical methods can contribute a great deal. The World Wide Web Consortium expressly standardized the Shapes Constraint Language SHACL for validating RDF graphs against defined conditions.^[1] Whether those conditions are the right ones in domain terms must nevertheless still be justified and reviewed by the responsible people.

The Review Objective Determines the Quality Required

Validation starts with a precise statement of purpose. It identifies the decisions or tasks the model is intended to support, the affected objects and roles, permissible data sources, and the consequences of an erroneous model result. Testable requirements follow from this.

For a batch release model, for example, a requirement might be that every batch is assigned to a product, a manufacturing version, a status, and a responsible release role. For a work-instruction context model, it may be crucial to consider equipment, process phase, material state, and the applicable SOP together.

A general quality judgment such as “The model is good” is insufficient. Research on ontology quality distinguishes several characteristics and metrics because structural quality, functional suitability, and understandability cannot be reduced to the same value.^[5] A model can be logically consistent yet unsuitable for the operational task.

Test Case: A Recommendation Contradicts the Requirement

An illustrative model is intended to suggest additional in-process tests. An explicitly assumed test rule applies to test case T-04: a documented equipment change, with all other conditions met, should not by itself trigger an additional test. This is a test requirement for this example, not a general statement about equipment changes.

The expected result is therefore “no additional test solely because of an equipment change.” What is observed, however, is a recommendation for an additional test. Although the inputs are complete and syntactically valid, the substantive result differs from the expected result. The report records the model version used, the test case, and this deviation.

One possible limited use excludes situations involving equipment changes. To achieve this, the application must reliably identify such situations and route them to the intended substantive review. This boundary and the cases that remain permitted are tested separately. The original error remains open for resolution.

The results do not yet lead to automatic approval. Those responsible assess whether the limited use, with its remaining risks, is acceptable. Any corresponding decision can apply only to that documented scope.

Four Review Levels Reveal Different Errors

The following review distinguishes four levels. This structure is a working aid, not a complete testing standard. The formal level checks whether the data are syntactically readable and the data types used are correct. The structural level examines whether required properties, cardinalities, and relationships are present.

The logical level asks whether statements contradict one another or produce unwanted inferences. The domain level finally checks whether concepts, rules, and results match the intended operational reality.

These levels complement but do not replace one another. Passing a SHACL test can show that a batch has exactly one status. It does not prove that the recorded status is correct in substance. A logical reasoner can detect a contradiction between classes. Without a domain specification, it does not know whether the modeled classification is appropriate for the production process.

Real Test Cases Show Whether the Model Is Suitable for Its Task

Structural and logical tests should be supplemented with specific domain cases. These include typical processes, known boundary cases, deliberately erroneous inputs, and rare situations with a high impact. The expected result or response is defined in advance for each case.

A knowledge model for material releases should recognize, for example, that material on hold must not be treated as available. At the same time, it must be able to represent a permissible special case in which a sample is collected for an explicitly documented investigation. Testing only the normal case would not check this distinction.

Counterexamples are particularly important. They examine whether a rule is too broad. Even if all known positive cases are classified correctly, the model can still be unusable because it also accepts numerous cases that do not fit. Positive and negative tests must therefore be considered together.

Structural Rules Make Expected Model States Machine-Checkable

SHACL separates the data graph to be checked from a shapes graph that describes conditions. Such conditions can specify, for example, which property must be present, how many values are permitted, which data type a value has, or which class a connected object should belong to.

For an operational knowledge model, a shape could require every approved work instruction to have exactly one version identifier, an approval date, and a responsible role. Another condition could ensure that a measurement value is connected to a unit, timestamp, and measured object. Violations appear in a validation report and can be addressed specifically.

Structural rules are particularly valuable when knowledge from several sources is combined. They make implicit expectations explicit. At the same time, there is a risk of finding only errors for which a condition has already been formulated. An incomplete collection of shapes can therefore produce an apparently clean result even though important domain requirements are missing.

Logical Consistency Checks Relationships and Inferences

Ontologies can formally express classes, properties, hierarchies, and logical restrictions. OWL 2 provides a standardized language for this purpose and allows further statements to be inferred from explicit statements.^[2]

A consistency check can detect, for example, that an object has been assigned simultaneously to two explicitly incompatible classes. Class satisfiability must be distinguished from this: if a class's conditions cannot be satisfied together, it cannot have instances. Such a class does not by itself make the entire ontology inconsistent. A contradiction arises if the ontology also requires an instance to belong to that class.^[6]

Yet freedom from logical contradiction is only one quality dimension. A consistent model can contain false assumptions or fail to describe important exceptions. Moreover, the open-world assumption used in many semantic models works differently from classical database checks: a missing statement does not automatically mean that its opposite holds. Validation rules must account for this modeling assumption.

The Requirement Must Be Traceable to Its Basis

For test case T-04, the assumed test rule must actually support the expected response. A cited source helps little if its version is unclear or its statement was altered during modeling. The review therefore follows the connection from the requirement to its specific basis.

Data Provenance describes such origin relationships. PROV-O can express the entities, activities, and agents involved.[3] The review conclusion additionally requires an assessment of whether the transfer is correct in substance.

Substantive Review Needs More Than Agreement

Domain experts should not merely read a model in full and approve it as a whole. Clearly bounded review questions are more effective: Are the concepts used unambiguous? Are relevant exceptions missing? Does the rule match the current SOP? Is an observed correlation incorrectly presented as a cause? Is the scope of applicability narrower or broader than the underlying evidence?

Different roles can identify different errors. People performing the work know practical special cases and ambiguous terms. Quality assurance and domain leads assess requirements, risks, and approval conditions. Technical reviewers examine structure, processing, and reproducible test results.

Conflicting assessments should not disappear through a majority vote. They mark an unresolved assumption. At that point, the model either remains restricted, receives a documented open question, or is approved only after further investigation.

Deriving a Decision on Use from Review Results

A review report documents the subject, methods, results, and open deviations. The decision on use assesses these findings for a particular purpose. It can permit, limit, or reject use. The report and the decision are connected but are not the same thing.

For restricted use, the permitted scope must be identifiable and enforceable in practice. A note in the report is insufficient if the application continues to process excluded cases like ordinary cases. In the example, reliably identifying an equipment change is therefore part of testing the restriction.

When the version changes, the effects on existing review results must be assessed. The NIST AI RMF considers assessment throughout the lifecycle of AI systems.[4] Applying this lifecycle perspective to other knowledge models is a domain interpretation here, not a general validation standard prescribed by the framework.

Validity Must Continue to Be Monitored During Operation

A model can be suitable at the time of approval and later lose its validity. New products, equipment, materials, SOPs, or regulatory requirements change the context. Observations from actual use therefore form part of ongoing validation.

Warning signals include an unusually high number of manual deviations, recurring clarification requests, frequent overrides of a recommendation, or cases for which no suitable model category exists. These signals do not prove a model error. They open a targeted review task.

Technical changes can also trigger renewed review. If a data source is replaced, a transformation step is changed, or a new ontology is incorporated, it must be checked whether the previous assumptions about structure and meaning still hold.

Validation Is More Than Versioning or a Syntax Check

Versioned knowledge models preserve which model version applied when, what changed it, and for which scope it is approved. Validation answers a different question: Is this exact version suitable for the intended purpose, and does it meet the defined requirements?

A technically valid document is not automatically a valid knowledge model either. A file can contain correct RDF or JSON-LD yet use an incorrect concept, omit a necessary relationship, or apply a rule to the wrong scope. Syntax checks determine whether the representation is formally readable. Only subsequent structural, logical, and substantive reviews examine what is represented.

Conversely, not every model that is incomplete in domain terms must immediately be faulty. No model represents all of reality. What matters is whether the deliberate limitation fits the specific intended use and remains apparent to users.

What Passing Validation Does Not Guarantee

Passing validation does not prove that a knowledge model will work correctly in every future situation. It shows that the version reviewed met the defined requirements and tests for a documented purpose.

Automated checks find only errors that can be captured by rules or expected results. Domain reviewers, for their part, can overlook assumptions or share the same blind spot. Even extensive test cases never represent every possible operational situation.

Validation also does not replace responsibility in the specific process. An approved model can structure information, check relationships, and support options for action. The substantive decision at defined review and approval points remains with authorized people.

The Deviation Is Part of the Result

T-04 yields a useful finding precisely because expected and actual results differ. Technically error-free processing must not conceal this contradiction. Nor may restricted use remove the failed case from the documentation.

A sound review conclusion states what was investigated, which requirements are met, and where limits remain. This makes the decision on use open to review—and allows a later change to address the questions that are still unresolved.