Tasks Overview
Two tracks over three ontology pairs: Track 1 evaluates equivalence as global alignment and as local ranking; Track 2 evaluates mixed equivalence + subsumption as typed ranking.
OAEI Bio-ML 2026 comprises two tracks over the same three whole-ontology pairs. Track 1 — Equivalence evaluates equivalence matching in two complementary settings (a global alignment subtrack and a local ranking subtrack), and Track 2 — Mixed equivalence + subsumption typed ranking evaluates equivalence and subsumption together as a single typed ranking problem.
| Track | Subtrack | You submit | Headline metric |
|---|---|---|---|
| Track 1 — Equivalence | Subtrack 1 — Global equivalence alignment | one full OAEI Alignment RDF per pair | repaired, coherence-aware macro-F1 |
| Track 1 — Equivalence | Subtrack 2 — Local equivalence ranking | a best-first ranking of each query’s 100-candidate pool | macro-MRR |
| Track 2 — Mixed equivalence + subsumption typed ranking | — | a scored ranking of each query’s 300 typed hypotheses | Preferred Typed-MRR |
The three ontology pairs
Every track is evaluated over the same three biomedical ontology pairs:
| Pair | Source → Target | Bridges |
|---|---|---|
NCIT-DOID | NCIt → DOID | cancer terminology ↔ human disease |
SNOMED-FMA | SNOMED CT → FMA | clinical terms ↔ anatomy |
SNOMED-NCIT | SNOMED CT → NCIt | clinical terms ↔ cancer terminology |
Scores are macro-averaged across the three pairs (each pair weighted equally). Directory names use the uppercase form (NCIT-DOID, SNOMED-FMA, SNOMED-NCIT); URL slugs use the lowercase form (ncit-doid, snomed-fma, snomed-ncit).
Track 1 — Equivalence
Subtrack 1 — Global equivalence alignment
| Pairs (3) | NCIT-DOID, SNOMED-FMA, SNOMED-NCIT |
| You submit | one full OAEI Alignment RDF per pair (full OWL IRIs, = cells) |
| Setting | semi-supervised — the public refs_equiv/train.tsv is released per pair; the test reference is hidden |
| Scored | organiser-side, against a repaired, coherence-aware reference (headline) and the standard reference (reported alongside), plus Global Coherence (reasoner-based) |
| Metric | repaired, coherence-aware precision / recall / F1, macro-averaged over the 3 pairs; standard P/R/F1 secondary |
| Details | Subtrack 1 — Global Equivalence Alignment · submission format |
Subtrack 2 — Local equivalence ranking
| Pairs (3) | NCIT-DOID, SNOMED-FMA, SNOMED-NCIT |
| You submit | one ranking per query: its 100-candidate pool ordered best-first (format) |
| Pools | the gold-stripped local.test.cands.tsv per pair — full OWL IRIs, 100 target candidates per source entity |
| Scored | organiser-side, against the hidden gold |
| Metric | MRR + Hits@{1, 5, 10}, macro-averaged over the 3 pairs |
| Details | Subtrack 2 — Local Equivalence Ranking · submission format |
Track 2 — Mixed equivalence + subsumption typed ranking
| Pairs (3) | NCIT-DOID, SNOMED-FMA, SNOMED-NCIT |
| You submit | one scored ranking per query over its 300 typed hypotheses (100 candidates × 3 relations) (format) |
| Pools | the gold-stripped track2.test.cands.tsv per pair — CURIEs (e.g. NCIT:C101044); each candidate is expanded across equivalent, source_subsumed_by_target, source_subsumes_target |
| Scored | organiser-side, against the hidden preferred relation + hierarchy-graded gold |
| Metric | Preferred Typed-MRR + Hierarchy-aware Typed-nDCG@10, macro-averaged over the 3 pairs |
| Details | Track 2 — Typed Ranking · submission format |
Serialisation (intentional)
The two tracks use different, deliberate serialisations so that participants build against the format each task expects:
- Track 1 (global alignment and local ranking) uses full OWL IRIs (e.g.
http://purl.obolibrary.org/obo/DOID_1612). - Track 2 (typed ranking) uses CURIEs (e.g.
NCIT:C101044).
All public *.test.cands.tsv files are gold-stripped — they carry the candidate pool only, never the answer.
Data
The datasets and candidate pools are distributed as one Hugging Face dataset, OAEI-ML/bio-ml (edition tag 2026); the self-contained scoring_kit/ (validators + local scorers) ships separately with the track repository. Get both, side by side:
git clone https://github.com/liseda-lab/OAEI-Bio-ML && cd OAEI-Bio-ML # the scoring kit
hf download OAEI-ML/bio-ml --repo-type dataset --revision 2026 --local-dir ./bio-ml # the dataUnder bio-ml/, each pair (NCIT-DOID, SNOMED-FMA, SNOMED-NCIT) ships the public equivalence reference refs_equiv/train.tsv, the local-ranking pools local.train.cands.tsv / local.valid.cands.tsv (gold-bearing) plus local.test.cands.tsv (gold-stripped), and the Track 2 typed files track2.train.answers.tsv / track2.valid.answers.tsv (with matching .preferred.tsv / .graded.tsv keys) plus track2.test.cands.tsv (gold-stripped); a repaired/ tree mirrors the same files against the coherence-repaired reference. The source ontologies are not re-hosted — obtain each from its publisher (see the dataset’s ontologies.md). Run python scoring_kit/self_check.py --data ./bio-ml to confirm the data downloaded intact.
Key dates
| Date | Milestone |
|---|---|
| 2026-07-06 | provisional materials |
| 2026-07-07 | datasets finalised |
| 2026-07-12 | competition starts — leaderboards open |
| 2026-09-01 | evaluation closes |
| 2026-09-06 | competition ends — results published (grace period to 2026-09-12) |
Participant results are collected and ranked on three CodaBench competitions — one per task (Track 1 global alignment, Track 1 local ranking, Track 2 typed ranking). Competition URLs are announced at launch.