posted by user: guglielmof || 746 views || tracked by 1 users: [display]

LLM4EVAL 2024 : Large Language Model for Evaluation in IR

FacebookTwitterLinkedInGoogle

Link: https://llm4eval.github.io/workshop/
 
When Jul 18, 2024 - Jul 18, 2024
Where Washington DC, USA
Submission Deadline Apr 25, 2024
Notification Due May 31, 2024
Final Version Due Jun 15, 2024
 

Call For Papers

Overview
The first workshop on evaluating IR systems with Large Language Models (LLMs) is accepting submissions that describe original research findings, preliminary research results, proposals for new work, and recent relevant studies already published in high-quality venues. The workshop will have both an in-person and virtual component, and submissions are welcome even for researchers who cannot attend in person, as they will present their work in the virtual component.

Topics of interest
We welcome both full papers and extended abstract submissions on the following topics, including but not limited to:

LLM-based evaluation metrics for traditional IR and generative IR.
Agreement between human and LLM labels.
Effectiveness and/or efficiency of LLMs to produce robust relevance labels.
Investigating LLM-based relevance estimators for potential systemic biases.
Automated evaluation of text generation systems.
End-to-end evaluation of Retrieval Augmented Generation systems.
Trustworthiness in the world of LLMs evaluation.
Prompt engineering in LLMs evaluation.
Effectiveness and/or efficiency of LLMs as ranking models.
LLMs in specific IR tasks such as personalized search, conversational search, and multimodal retrieval.
Challenges and future directions in LLM-based IR evaluation.
Submission guidelines
We welcome the following submissions:

Previously unpublished manuscripts will be accepted as extended abstracts and full papers (any length between 1 - 9 pages) with unlimited references, formatted according to the latest ACM SIG proceedings template available at http://www.acm.org/publications/proceedings-template.
Published manuscripts can be submitted in their original format.
All submissions should be made through Easychair: https://easychair.org/conferences/?conf=llm4eval

All papers will be peer-reviewed (single-blind) by the program committee and judged by their relevance to the workshop, especially to the main themes identified above, and their potential to generate discussion. For already published studies, the paper can be submitted in the original format. These submissions will be reviewed for their relevance to this workshop. All submissions must be in English (PDF format).

Please note the workshop will have an in-person (to be held with SIGIR 2024) and virtual component (to be held at a later date on SIGIR VF). During submission, the authors should select their preferred component. All accepted papers will have a poster presentation with a few selected for spotlight talks. Accepted papers may be uploaded to arXiv.org, allowing submission elsewhere as they will be considered non-archival. The workshop’s website will maintain a link to the arXiv versions of the papers.

Important Dates
Submission Deadline: April 25th, 2024 (AoE time)
Acceptance Notifications: May 31st, 2024 (AoE time)
Workshop date: July 18, 2024
Website and Contact

More details are available at https://llm4eval.github.io/cfp/.
For any questions about paper submission, you may contact the workshop organizers at llm4eval@easychair.org

Related Resources

GenAI and LVMs for Biometrics 2025   IEEE Transactions on Biometrics, Behavior, and Identity Science (T-BIOM) Special Issue on Generative AI and Large Vision-Language Models for Biometrics
ACDL 2025   8th Advanced Course on Data Science & Machine Learning
CLEF 2025   Cross-Language Evaluation Forum
LLM-Finance 2024   The IEEE International Workshop on Large Language Models for Finance
IEEE ICC-CD 2024   2024 IEEE International Conference on Computing and Complex Data
SMM4H 2024   The 9th Social Media Mining for Health Research and Applications Workshop and Shared Tasks — Large Language Models (LLMs) and Generalizability for Social Media NLP
LLM 4 MDE & MeSS 2024   1st International Workshop Large Language Models for Model-Driven Engineering & 4th International Workshop on MDE for Smart IoT Systems
IEEE CACML 2025   2025 4th Asia Conference on Algorithms, Computing and Machine Learning (CACML 2025)
SPIE-Ei/Scopus-DMNLP 2025   2025 2nd International Conference on Data Mining and Natural Language Processing (DMNLP 2025)-EI Compendex&Scopus
IEEE-Ei/Scopus-ITCC 2025   2025 5th International Conference on Information Technology and Cloud Computing (ITCC 2025)-EI Compendex