Evaluation Metrics

This page summarises how OAEI Bio-ML submissions are scored, per track. A full description appears in the supplementary material (available at track launch).

Track 1 · Subtrack 1 — Global equivalence alignment

Each submission is a full alignment per pair (full OWL IRIs). It is scored with precision, recall and F1 against two references:

• Repaired reference (headline). Coherence-aware and relation-agnostic: correspondences flagged as uncertain (?) are ignored from both the predictions and the reference, and a reference subsumption (< / >) is credited by a predicted correspondence of any relation. The headline score is the repaired, coherence-aware F1.
• Standard reference (secondary). The complete, possibly-incoherent reference scored with traditional P/R/F1.

The two references are not directly comparable. They differ in both membership and scoring rules, so a repaired-vs-standard difference is not an accuracy delta — read each within its own reference.

Global Coherence. Alongside P/R/F1 we report a reasoner-checked coherence measure — the degree of logical incoherence induced by the submitted alignment on the merged ontologies (0 = coherent). Because it needs a description-logic reasoner, coherence is computed organiser-side, not in the participant scoring kit.

Track 1 · Subtrack 2 — Local equivalence ranking

For each source entity, the system ranks a fixed candidate pool best-first. Writing $\mathrm{rank}(q)$ for the 1-based position of the correct target in query $q$‘s ranking, over the query set $Q$:

Reported: MRR and Hits@{1,5,10}. MRR rewards placing the correct target as early as possible (rank 1 scores 1, rank 2 scores 1/2, and so on); Hits@$k$ is the fraction of queries whose correct target lands in the top $k$.

Track 2 — Mixed equivalence + subsumption typed ranking

Track 2 ranks typed candidates (equivalence and subsumption), serialised as CURIEs. It is scored with:

• Preferred Typed-MRR — the reciprocal rank of the preferred correct relation for each query, so a system is rewarded for surfacing the single best-typed answer first; and
• Hierarchy-aware Typed-nDCG@10 — a normalised discounted cumulative gain over the top 10 that grants graded credit to hierarchically-related answers, rewarding rankings that respect the subsumption hierarchy.

Equivalence-only and subsumption-only slices are reported alongside the combined score.

Averaging

Every headline metric is macro-averaged over the three pairs (NCIT–DOID, SNOMED–FMA, SNOMED–NCIT) — the unweighted mean of the per-pair values, so a system must do well on all three rather than being carried by the largest.

Timeline

The evaluation window and reporting dates (all 00:00 Anywhere on Earth):