Measuring Knowledge Graph Quality

Two numbers about the same graph. More than 73 percent of Wikidata statements carry at least one reference. The average referencing quality of Wikidata subsets, scored against a rubric covering whether the reference is retrievable, relevant and authoritative, comes out at 0.58 out of 1 (Hosseini Beghaeiraveri, Gray & McNeill, 2024, RQSS: Referencing quality scoring system for Wikidata, Semantic Web). An earlier crowdsourced study put the share of references that are both authoritative and relevant at around 61 percent, on a small sample. Coverage is high and quality is middling, and a dashboard that reports only the first number is reporting the easy half.

Four questions wearing one word

"Quality" bundles four measurements with different costs and different owners.

Correctness asks whether an asserted triple is true. It is answerable only by comparison against something outside the graph, which in practice means human audit or a trusted reference source, so it is always sampled.

Completeness asks whether a true fact is present. Under open-world semantics the graph itself cannot distinguish absent from false, so this cannot be measured directly at all.

Consistency asks whether the data violates its own schema. This one is cheap and mechanical, and it is what SHACL does.

Provenance and trust ask where a triple came from and whether that source warrants belief. This is a modelling question before it is a measurement one: a graph with no statement-level provenance cannot be audited at any price.

Completeness, and the surrogates for it

Three workable substitutes for a measurement that is formally unavailable.

The partial completeness assumption says that if a subject has at least one object for a predicate, all its objects for that predicate are known. It was introduced to make rule mining possible over open-world knowledge bases, where absent facts cannot serve as counterexamples, and it is what lets a rule miner compute a usable confidence score (Galárraga et al., 2013, AMIE: Association Rule Mining under Incomplete Evidence in Ontological Knowledge Bases, WWW; the AMIE+ follow-up in the VLDB Journal scaled the approach past 12 million statements). It is also wrong in a specific way: when edges are missing more or less at random rather than by subject, the assumption treats genuine gaps as negative evidence.

Explicit incompleteness statements are the modelling answer. Wikidata's "no value" and "unknown value" markers, and cardinality statements saying a person had exactly two children, convert an absence into an assertion that can be checked.

Comparison against a reference graph gives a lower bound rather than a figure. Profiling DBpedia against YAGO estimates at least 2.6 million missing type assertions in DBpedia, which bounds the gap from below and says nothing about what both graphs are missing.

Identity links are where the damage is

Among triple types, owl:sameAs and its property-graph equivalents carry disproportionate risk, because an identity claim merges everything said about two entities and the merge propagates transitively. The refinement literature is blunt about the observed error rates: roughly 20 percent of the interlinks between DBpedia and Freebase are incorrect, and about half of some owl:sameAs link sets connect things that are related but not identical (Paulheim, 2017, Knowledge graph refinement: A survey of approaches and evaluation methods, Semantic Web 8(3)).

A 20 percent error rate on ordinary attribute triples is a data quality problem. A 20 percent error rate on identity links is a correctness problem for every query that touches either entity, and the blast radius grows with the size of the merged cluster. This is the strongest argument for keeping identity links in their own named graph with their own provenance, so they can be re-derived rather than inherited.

When it breaks

Completion metrics inherit a closed-world assumption the graph does not satisfy. MRR and Hits@k rank a held-out fact against all unobserved candidates, treating missing-but-true facts as competitors, so the reported numbers are pessimistic by an amount that varies with how incomplete the graph is and is never reported (Yang et al., 2022, Rethinking Knowledge Graph Evaluation Under the Open-World Assumption, NeurIPS).

Benchmark monoculture. The refinement survey found that around two out of three approaches evaluated on DBpedia were evaluated on DBpedia alone, which makes "state of the art in KG refinement" a claim about one graph's idiosyncrasies as much as about the method.

Reference presence is not reference quality. The gap between 73 percent coverage and 0.58 average quality is the whole problem with provenance metrics that count triples. A link to a dead URL, a circular link back to the Wikipedia article the fact was scraped from, and a citation to a primary registry all count as one reference.

Sampling gives you an interval, not a verdict. A manual audit of 1,000 triples from a billion-triple graph supports a statement about the graph's overall error rate and supports nothing at all about any individual predicate, because errors cluster hard by source and by extractor. Per-predicate quality needs per-predicate samples.

Fixing correctness reduces completeness. Deleting every unsourced statement raises precision and lowers recall, and which of those a consumer prefers depends entirely on the application. Publishing both numbers, per release, is the only honest version.

Quality figures expire. Every number in this concept is attached to a dump date. Graphs that ingest continuously change their own quality profile between audits, so a quality report without a dump identifier is not reproducible.