CLEF 2011 - Call for Lab proposals


The CLEF 2011 conference is the continuation of the popular CLEF campaigns and workshops that ran 2000-2009 (http://www.clef-campaign.org/). In 2010, CLEF was organized in a different way: the CLEF 2010 Conference (http://clef2010.org) presenting related research papers followed by a series of "labs" covering a broad range of issues from fields of multilingual and multimodal information access evaluation. CLEF 2011 will be organized in a similar way. Researchers and practitioners from all segments of the information access and related communities are invited to submit proposals for review.

Scope

Proposals are accepted for two different types of "labs":

. Labs that are a "campaign-style" evaluation for specific information access problems (during the twelve month period preceding the conference), similar in nature to the traditional CLEF campaign "tracks". In 2010 there were 5 labs (http://clef2010.org/index.php?page=pages/labs.html): CLEF-IP, ImageCLEF, PAN, RespublicQA and WePS. Topics covered by campaign-style labs can be any information access-related task.
. Labs that follow a more classical "workshop" pattern, exploring issues of evaluation methodology, metrics, processes etc. in information access and closely related fields, such as natural language processing, machine translation, and human-computer interaction. In 2010 there were 2 workshops: CriES and LogCLEF.
In both cases, it is expected that lab sessions at the conference will contain ample time for general discussion and engagement by all participants - not just those presenting campaign results and papers. Organizers should plan time for panels, demos, etc. where applicable.

Location

Two days of the four-day CLEF 2011 conference have been reserved exclusively for lab sessions. The lab sessions will take place at the site of the conference at the University of Amsterdam, Netherlands. The potential for labs to present their overall results "overview presentations") during the scientific paper session of the conference is presently being explored.
Proposal Submission
Lab proposals should provide sufficient information for the lab organizing committee to be able to judge the importance, quality, and benefits for the research community.

Each lab should have one or more organizers responsible for the execution of the lab.
Proposals should be 2-4 pages long and should provide the following information:

. Title of the proposed lab.
. The planned format of the lab, i.e. campaign-style ("track") or workshop.
. Planned length of the lab session at the conference: half-day, one day, two days.
. Names and full addresses, including contact details, of the lab organizer(s), a brief description of the organizers' experience and background in the topic, and links to web pages of the lab organizers.
. A brief description of the lab topic and goals, its relevance to CLEF 2011 and significance for the research field together with indications as to how the proposed tasks/actions contribute to their implementation . A statement on the intended development/growth path if the proposal is for a continuation of activities previously undertaken at CLEF workshops.
. A description of the target audience, areas from which the participants are expected to come, an analysis of the potential for participants (number, statements of intent to participate where applicable), potential industry stakeholders, strategy for publicizing the lab . Arrangements for the organization of the lab campaign, if applicable, including a brief outline of the campaign milestones, test data to be used, indications of the size of the data collections, issues of scalability, tasks to be proposed to participants, and format of presentation at the conference.
. If the lab proposes to set up a steering committee to oversee its activities, include names, addresses, and home page links of people who have agreed to be part of the steering committee if the lab proposal is accepted. This list should ideally include people from at least three different countries.
Reviewing Process
Each submitted proposal will be reviewed by the CLEF2011 lab organizing committee. The decision will be sent by email to the responsible organizer by November 15, 2010.
The final length of the lab session will be determined based on the overall organization of the conference and the number of submissions received by a lab. Due to space restrictions, only a limited number of lab sessions can be conducted in parallel at the conference. The reviewers may suggest modifications to the proposed lab programme in order to fit it with the above reviewing criteria and the overall organization of CLEF 2011.

Reviewing criteria for the "campaign-style" labs include:

. Soundness of methodology, feasibility of task.
. Use case (description of the underlying problems), business case/industrial stakeholders (potential for market).
. Number of potential participants, critical mass.
. Clear movement along a growth path, development of field.
. Others factors such as: innovation, minimize overlap with other evaluation initiatives or events, focus (appropriate number of subtasks, avoid "inflation" of labs), interdisciplinary character.
Reviewing criteria for the "workshop-style" labs include:
. The appropriateness of the workshop-style lab to the overall information access agenda pursued by CLEF.
. Number of potential participants, critical mass.
. Likelihood that the outcomes of the workshop will constitute a significant contribution to the field. Is the workshop focused enough that useful conclusions are likely?
. Other factors such as: innovation, minimize overlap with other evaluation initiatives and events, vision for a potential continuation, interdisciplinary character. Lab Organizer's Tasks:
. Producing a "Call for Participation" for the campaign-style lab or a "Call for Papers" for a workshop-style lab and disseminating it through all appropriate means. Please provide a web page URL which can be linked into the CLEF 2011 home page (asap).
. Providing a brief description of the lab for the conference program.
. Signing up campaign participants, and executing the campaign in the case of campaign-style labs.
. Reviewing/accepting submitted papers and position papers in case of workshop-style labs.
. Scheduling lab session activities in collaboration with the local organizers and the lab chairs.
. Sending the lab schedule and other lab material, all in PDF format, to the lab chairs (deadline to be defined).
. The lab material (papers, presentations etc.) will be distributed by the CLEF 2011 organization to the conference participants in electronic format (copyright will not be asked for from the authors, but only permission to publish and disseminate).
. Organization of post-conference publication of lab results in appropriate form (special issue, lab proceedings, etc.).

Publication

The working notes of the labs will be published online in time for the conference. It is foreseen that this online publication will have an ISBN number and be indexed in relevant services. It is the responsibility of lab organizers to arrange for appropriate post-conference publication of the lab results.
Important Dates
. Final lab proposals: 8 November 2010
. Notification of lab acceptance: 15 November 2010 . CLEF 2011 Conference: September 2011

Submission Details
Lab proposals (or questions) should be submitted via e-mail (either plain text or PDF format) to both Lab Organizing Committee Chairs:

Vivien Petras, Berlin School of Library and Information Science, Humboldt-Universität zu Berlin, vivien.petras[at]ibi.hu-berlin.de Paul Clough, The Information School, University of Sheffield, p.d.clough[at]sheffield.ac.uk