posted by user: guglielmof || 353 views || tracked by 1 users: [display]

LLM4EVAL 2024 : Large Language Model for Evaluation in IR


When Jul 18, 2024 - Jul 18, 2024
Where Washington DC, USA
Submission Deadline Apr 25, 2024
Notification Due May 31, 2024
Final Version Due Jun 15, 2024

Call For Papers

The first workshop on evaluating IR systems with Large Language Models (LLMs) is accepting submissions that describe original research findings, preliminary research results, proposals for new work, and recent relevant studies already published in high-quality venues. The workshop will have both an in-person and virtual component, and submissions are welcome even for researchers who cannot attend in person, as they will present their work in the virtual component.

Topics of interest
We welcome both full papers and extended abstract submissions on the following topics, including but not limited to:

LLM-based evaluation metrics for traditional IR and generative IR.
Agreement between human and LLM labels.
Effectiveness and/or efficiency of LLMs to produce robust relevance labels.
Investigating LLM-based relevance estimators for potential systemic biases.
Automated evaluation of text generation systems.
End-to-end evaluation of Retrieval Augmented Generation systems.
Trustworthiness in the world of LLMs evaluation.
Prompt engineering in LLMs evaluation.
Effectiveness and/or efficiency of LLMs as ranking models.
LLMs in specific IR tasks such as personalized search, conversational search, and multimodal retrieval.
Challenges and future directions in LLM-based IR evaluation.
Submission guidelines
We welcome the following submissions:

Previously unpublished manuscripts will be accepted as extended abstracts and full papers (any length between 1 - 9 pages) with unlimited references, formatted according to the latest ACM SIG proceedings template available at
Published manuscripts can be submitted in their original format.
All submissions should be made through Easychair:

All papers will be peer-reviewed (single-blind) by the program committee and judged by their relevance to the workshop, especially to the main themes identified above, and their potential to generate discussion. For already published studies, the paper can be submitted in the original format. These submissions will be reviewed for their relevance to this workshop. All submissions must be in English (PDF format).

Please note the workshop will have an in-person (to be held with SIGIR 2024) and virtual component (to be held at a later date on SIGIR VF). During submission, the authors should select their preferred component. All accepted papers will have a poster presentation with a few selected for spotlight talks. Accepted papers may be uploaded to, allowing submission elsewhere as they will be considered non-archival. The workshop’s website will maintain a link to the arXiv versions of the papers.

Important Dates
Submission Deadline: April 25th, 2024 (AoE time)
Acceptance Notifications: May 31st, 2024 (AoE time)
Workshop date: July 18, 2024
Website and Contact

More details are available at
For any questions about paper submission, you may contact the workshop organizers at

Related Resources

FLLM 2024   The 2nd International Conference on Foundation and Large Language Models
SMM4H 2024   The 9th Social Media Mining for Health Research and Applications Workshop and Shared Tasks — Large Language Models (LLMs) and Generalizability for Social Media NLP
IPM-LLMDQKG 2025   Special issue of Information Processing & Management on Large Language Models and Data Quality for Knowledge Graphs
GenAI4PM 2024   The First International Workshop on Generative AI for Process Mining
LLM4MDE & MeSS 2024   1st International Workshop Large Language Models for Model-Driven Engineering & 4th International Workshop on MDE for Smart IoT Systems
DL4KG 2024   The 7th Edition of Workshop on Deep Learning and Large Language Models for Knowledge Graphs
ECNLPIR 2024   2024 European Conference on Natural Language Processing and Information Retrieval (ECNLPIR 2024)
LREC-COLING 2024   The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation
ACM NLPIR 2024   ACM--2024 8th International Conference on Natural Language Processing and Information Retrieval (NLPIR 2024)
AAAI-MAKE 2024   AAAI 2024 Spring Symposium on Empowering Machine Learning and Large Language Models with Domain and Commonsense Knowledge