posted by user: grupocole || 682 views || tracked by 1 users: [display]

ALTARS 2023 : 2nd Workshop on Augmented Intelligence for Technology-Assisted Review Systems


When Apr 2, 2023 - Apr 2, 2023
Where Dublin, Ireland
Abstract Registration Due Jan 23, 2023
Submission Deadline Feb 6, 2023
Notification Due Feb 20, 2023
Categories    artificial intelligence

Call For Papers




2nd Workshop on Augmented Intelligence for Technology-Assisted Review Systems (ALTARS) 2023
Evaluation Metrics and Protocols for eDiscovery and Systematic Review Systems

Dublin, Ireland - 2nd April 2023

Venue: held as part of the 45th European Conference on Information Retrieval (ECIR 2023)


Paper Submission (extended) deadline February 6 2023


Augmented Intelligence (AL) is “a subsection of AI machine learning developed to enhance human intelligence rather than operate independently of or outright replace it. It is designed to do so by improving human decision-making and, by extension, actions taken in response to improved decisions.” In this sense, users are supported, not replaced, in the decision-making process by the filtering capabilities of the Augmented Intelligence solutions, but the final decision will always be taken by the users who are still accountable for their actions.

In this field, Technology-assisted review systems (TARS) use a kind of human-in-the-loop approach where classification and/or ranking algorithms are continuously trained according to the relevance feedback from expert reviewers, until a substantial number of the relevant documents are identified. This approach has been shown to be more effective and more efficient than traditional e-discovery and systematic review practices, which typically consists of a mix of keyword search and manual review of the search results.

Given these premises, ALTARS will focus on High-recall Information Retrieval (IR) systems which tackle challenging tasks that require the finding of (nearly) all the relevant documents in a collection. Electronic discovery (eDiscovery) and systematic review systems are probably the most important examples of such systems where the search for relevant information with limited resources, such as time and money, is necessary.


In this workshop, we aim to fathom the effectiveness of these systems which is a research challenge itself. In fact, despite the number of evaluation measures at our disposal to assess the effectiveness of a "traditional" retrieval approach, there are additional dimensions of evaluation for TAR systems.
For example, it is true that an effective high-recall system should be able to find the majority of relevant documents using the least number of assessments. However, this type of evaluation discards the resources used to achieve this goal, such as the total time spent on those assessments, or the amount of money spent for the experts judging the documents.
The topics include, but are not restricted to:

- Novel evaluation approaches and measures for Systematic reviews;
- Reproducibility of experiments with test collections;
- Design and evaluation of interactive high-recall retrieval systems;
- Study of evaluation measures;
- User studies in high-recall retrieval systems;
- Novel evaluation protocols for continuous Active Learning;
- Evaluation of sampling bias.


Abstract Submission Deadline: January 23, 2023
Paper Submission Deadline: January 30, 2023
Paper Notifications: February 20, 2023
Workshop day: April 2, 2023


Research papers, describing original ideas on the listed topics and on other fundamental aspects of Technology-Assisted Reviews methodologies and technologies, are solicited. Moreover, short papers on early research results, new results on previously published works, and extended abstract on previously published works are also welcome.

- Research papers presenting original works should be in the 9 - 10 pages range
- Short papers should be in the 6 - 7 pages range
- Posters should be 3 - 4 pages long
- Extended abstracts should be 2 pages long

For all the submission types, the references are included in the page limit. Papers must be in the CEUR-ART single column style.


Latex template:

MS Word template:

The accepted papers will be published in the ALTARS 2023 Proceedings. The Proceedings will be published by CEUR-WS, which is gold open access and indexed by SCOPUS and DBLP.


Authors can submit their papers electronically via our submission page through Easychair

General Chairs

Giorgio Maria Di Nunzio - University of Padua, Italy
Evangelos Kanoulas - University of Amsterdam, The Netherlands
Prasenjit Majumder - DAIICT, Gandhinagar and TCG CREST, Kolkata, India

Program Committee

Amanda Jones , Lighthouse (USA)
Dave Lewis , Redgrave Data (USA)
Parth Mehta , Parmonic (USA)
Doug Oard , University of Maryland (USA)
Fabrizio Sebastiani , CNR-ISTI (Italy)
Rene Spijker , Cochrane (The Netherlands)
Mark Stevenson , University of Sheffield (UK)
Eugene Yang , Johns Hopkins University (USA)

Related Resources

CMVIT-Maldives 2025   2025 9th International Conference on Machine Vision and Information Technology (CMVIT 2025)
Ei/Scopus-AACIP 2024   2024 2nd Asia Conference on Algorithms, Computing and Image Processing (AACIP 2024)-EI Compendex
AIST 2024   6th International Conference on Artificial Intelligence and Speech Technology (AIST2024)
IEEE-Ei/Scopus-ACEPE 2024   2024 IEEE Asia Conference on Advances in Electrical and Power Engineering (ACEPE 2024) -Ei Compendex
CCVPR 2024   2024 International Joint Conference on Computer Vision and Pattern Recognition (CCVPR 2024)
IITUPC 2024   Immunotherapy and Information Technology: Unleashing the Power of Convergence
IITUPC 2024   Immunotherapy and Information Technology: Unleashing the Power of Convergence
IEEE-Ei/Scopus-SGGEA 2024   2024 Asia Conference on Smart Grid, Green Energy and Applications (SGGEA 2024) -EI Compendex
CCCIS--EI 2025   2025 5th International Conference on Computer Communication and Information Systems (CCCIS 2025)
CPAIOR 2024   International Conference on the Integration of Constraint Programming, Artificial Intelligence, and Operations Research