posted by organizer: GunjanS || 2847 views || tracked by 1 users: [display]

SemREC 2021 : Semantic Reasoning Evaluation Challenge

FacebookTwitterLinkedInGoogle

Link: https://semrec.github.io/
 
When Oct 24, 2021 - Oct 28, 2021
Where Albany, NY, United States
Submission Deadline Jun 28, 2021
Notification Due Aug 15, 2021
Categories    ontology   description logic reasoning   machine learning   neuro-symbolic ai
 

Call For Papers

Despite the development of several ontology reasoning optimizations, the traditional methods either do not scale well or only cover a subset of OWL 2 language constructs. As an alternative, neuro-symbolic approaches are gaining significant attention. However, the existing methods still can not deal with very expressive ontology languages. To find and improve these performance bottlenecks of the reasoners, we ideally need several real-world ontologies that span the broad spectrum in terms of their size and expressivity. However, that is often not the case. One of the potential reasons for the ontology developers to not build ontologies that vary in terms of size and expressivity is the performance bottleneck of the reasoners. This challenge includes three tasks that aim to deal with this chicken and egg problem.

Task-1 - Submit a real-world ontology that is a challenge in terms of the reasoning time or memory consumed during reasoning. We will be evaluating the submitted ontologies based on the time and the memory consumed for a reasoning task, such as classification.

Task-2 - Submit a description logic reasoner that makes use of traditional techniques such as tableau algorithms and saturation rules. We will evaluate the performance and the scalability of the submitted systems on the datasets based on the time taken and memory consumed on the ontology classification task. This will provide an insight into the progress in the development of reasoners since the last reasoner evaluation challenge (ORE 2015).

Task-3 - Submit an ontology/RDFS reasoner that makes use of neuro-symbolic techniques for reasoning and optimization. We will be evaluating two types of neuro-symbolic systems: (a) that approximate the entailment reasoning for addressing the time complexity problem, or (b) predicting missing and plausible axioms for completion. We will evaluate the submitted systems on the test datasets based on the time taken, memory consumed, precision and recall.

This challenge will be collocated with the 20th International Semantic Web Conference (ISWC 2021)

Submission Guidelines:
Participants are requested to make a manuscript submission describing their entry. For Task 1, we expect a detailed description of the ontology along with the analysis of the reasoning performance, the workarounds, if any, that were used to make the ontology less challenging (for example, dropping of a few axioms, redesigning the ontology, etc.), and the (potential) applications in which the ontology could be used. For Tasks 2 and 3, we expect a detailed description of the system, including evaluating the system on the provided datasets. The submissions can be either in the form of short papers of length 5 pages or long papers of length 10-12 pages. All the submissions must be in English and follow the 1-column CEUR-ART style (link to the overleaf page would be updated soon). The proceedings will be published as a volume of CEUR-WS. Submissions should be made in the form of a pdf document on EasyChair.


In case you have any problems working with the datasets or any suggestions related to this challenge, please contact gunjans[at]iiitd.ac.in.

Related Resources

IKGRC 2024   The 2nd International Knowledge Graph Reasoning Challenge
IEEE-Ei/Scopus-ITCC 2025   2025 5th International Conference on Information Technology and Cloud Computing (ITCC 2025)-EI Compendex
HiPEAC SC 2024   HiPEAC Reproducibility Student Challenge
SPIE-Ei/Scopus-DMNLP 2025   2025 2nd International Conference on Data Mining and Natural Language Processing (DMNLP 2025)-EI Compendex&Scopus
UR@FLAIRS 2025   Special Track on Uncertain Reasoning at FLAIRS-38
IEEE-Ei/Scopus-CNIOT 2025   2025 IEEE 6th International Conference on Computing, Networks and Internet of Things (CNIOT 2025) -EI Compendex
ICCBR 2025   33rd International Conference on Case-based Reasoning
AMLDS 2025   IEEE--2025 International Conference on Advanced Machine Learning and Data Science
TIME 2024   The 31st International Symposium on Temporal Representation and Reasoning
SEMAPRO 2025   The Nineteenth International Conference on Advances in Semantic Processing