Reference-Free Evaluation of Taxonomies

ACL Findings 2026

Pascal Wullschleger^1,2Majid Zarharan³Donnacha Daly¹Marc Pouly¹Jennifer Foster²

¹Lucerne School of Information Technology²Maynooth University³Dublin City University

Abstract

We introduce two reference-free metrics for quality evaluation of taxonomies in the absence of labels. The first metric evaluates robustness by calculating the correlation between semantic and taxonomic similarity, addressing error types not considered by existing metrics. The second uses Natural Language Inference to assess logical adequacy. Both metrics are tested on five taxonomies and are shown to correlate well with F1 against ground truth taxonomies. We further demonstrate that our metrics can predict downstream performance in hierarchical classification when used with label hierarchies.

Citation

@inproceedings{wullschleger-etal-2026-reference,
  title = "Reference-Free Evaluation of Taxonomies",
  author = "Wullschleger, Pascal  and
    Zarharan, Majid  and
    Daly, Donnacha  and
    Pouly, Marc  and
    Foster, Jennifer",
  editor = "Liakata, Maria  and
    Moreira, Viviane P.  and
    Zhang, Jiajun  and
    Jurgens, David",
  booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
  month = jul,
  year = "2026",
  address = "San Diego, California, United States",
  publisher = "Association for Computational Linguistics",
  url = "https://aclanthology.org/2026.findings-acl.1273/",
  doi = "10.18653/v1/2026.findings-acl.1273",
  pages = "25489--25507",
  ISBN = "979-8-89176-395-1",
}