--------------------------------------------------------------------------------------------------
Call for Papers

Data infrastructurEs for Supporting Information Retrieval Evaluation (DESIRE 2011) Workshop

Co-located with CIKM 2011, the 20th ACM Conference on Information and Knowledge Management

Friday 28th October 2011, Glasgow, UK

http://www.promise-noe.eu/events/desire-2011/
--------------------------------------------------------------------------------------------------

AIM AND SCOPE

The Information Retrieval area has a strong and long tradition dating back to the 1960s in producing
and processing scientific data resulting from the experimental evaluation of search algorithms and
search systems. This attitude towards evaluation has led to fast and continuous progress in the
evolution of information retrieval systems and search engines.

However, in order to make these data test collections understandable and usable they must be endowed
with some auxiliary information, i.e., provenance, quality, context, etc. Therefore, there is a need
for metadata models able to describe the main characteristics of evaluation data. In addition, in
order to make distributed data collections accessible, sharable, and interoperable, there is a need
for advanced data infrastructures.

Nevertheless, the information retrieval area has barely explored and exploited the possibilities for
managing, storing, and effectively accessing the scientific data produced during the evaluation
studies by making use of the methods typical of the database and knowledge management areas.

It is thus time for these three communities – information retrieval, databases, and
knowledge management – to join efforts, meet, and cooperate to address the problem of envisaging and
designing useful infrastructures able to coherently manage pertinent data collections and sources
of information, and so take concrete steps towards developing them.

Indeed, the information retrieval experts need to recognise this need, while the database and
knowledge management experts need to understand the problem and work together to solve it by using
the methods and techniques specific to information management.

Therefore, the workshop will gather together experts from these three areas, to encourage them to
recognise the urgency of addressing the problem in an integrated and coherent way, and to coordinate
efforts towards drawing a roadmap and suggesting best practices for an effective solution of the
problem.

The objectives of the workshop are: to gather interested experts from the three areas of interest
to CIKM and discuss how to address the problem of envisaging and designing evaluation
infrastructures able to store, manage, and make accessible the scientific data and knowledge of
interest for advancing the evaluation of information retrieval and access tools.

The goal of the workshop is to understand how to make use of the expertise of the three scientific
areas in a cooperative way to avoid the duplication of efforts which may occur when addressing the
problem separately in each specific area and to trigger synergies and joint actions on this issue.

The workshop is intended to be a single full day workshop, consisting of: invited talks and paper
presentations during the morning; active discussion groups during the afternoon, engaging the
participants in the topics which have emerged in the previous sessions in order to define a roadmap
and best practices.

The main outcome will be to produce a roadmap and initial best practices to guide the development
of the necessary evaluation infrastructures.


TOPICS

- Conceptual and logical data models for representing IR evaluation scientific data
- Metadata formats for describing scientific data produced during IR evaluation
- Knowledge management for IR experimental evaluation
- Data quality, provenance, adaptability and reusability in the IR evaluation
- Data pre- and post-processing, metrics, and analyses in the IR evaluation
- Data exchange, integration, evolution and migration for IR evaluation infrastructures
- Workflow, Web services and Web Service Composition for IR evaluation infrastructures
- Information Extraction and Text Mining for linking scientific literature and experimental data
- Data citation
- Evaluation, Test collections, Crowdsourcing for IR evaluation
- Visualization of scientific data coming from experimental evaluation


IMPORTANT DATES

- Individual Workshop Papers Due: June 20, 2011
- Notification of Acceptance: July 8, 2011
- Camera Ready: July 20, 2011


FORMAT

Authors are invited to submit electronically original papers, which have not been published and
are not under consideration elsewhere, using the ACM template Option 2. including the ACM metadata
(General Terms, Categories, Keywords, Copyright number etc), available at:

http://www.acm.org/sigs/publications/proceedings-templates

Two types of comunications or position papers are solicited:

- communications: 4 pages max.
- position papers: 6 pages max.

Papers will be peer-reviewed by at least 3 members of the program committee. Selection will be
based on originality, clarity, and technical quality.

Papers should be submitted in PDF format to the following address:

https://www.easychair.org/conferences/?conf=desire2011

KEYNOTE SPEAKERS

- Norbert Fuhr, University of Duisburg-Essen, Germany.
- Maurizio Lenzerini, Sapienza University of Rome, Italy.

PROGRAM COMMITTEE

- Stefano Ceri, Politecnico di Milano, Italy
- Maarten de Rijke, University of Amsterdam, The Netherlands
- Donna Harman, National Institute of Standards and Technology (NIST), USA
- Noriko Kando, National Institute of Informatics (NII), Japan
- Henning Müller, University of Applied Sciences Western Switzerland
- Shila Ofek-Koifman, IBM Haifa, Israel
- Christos Papatheodorou, Ionian University, Greece
- Fausto Rabitti, ISTI-CNR, Italy
- Mark Sanderson, RMIT University, Australia
- Timos Sellis, National Technical University of Athens, Greece
- Gerhard Weikum, Max-Planck-Institut für Informatik, Germany

ORGANIZERS

- Maristella Agosti, University of Padua, Italy
- Nicola Ferro, University of Padua, Italy
- Costantino Thanos, ISTI-CNR, National Council of Research, Italy

CONTACT

- Nicola Ferro, ferro@dei.unipd.It