posted by user: grupocole || 2608 views || tracked by 8 users: [display]

EPS 2011 : Evaluating Personal Search


When Apr 18, 2011 - Apr 18, 2011
Where Dublin, Ireland
Submission Deadline Feb 21, 2011
Notification Due Mar 18, 2011
Final Version Due Mar 25, 2011
Categories    information retrieval

Call For Papers

**CFP for ECIR workshop towards developing new TREC (or TREC-like) track for personal search evaluation**

**Apologies for cross postings. Please redistribute to other interested parties**

Call for Papers

Evaluating Personal Search. An ECIR 2011 [Half Day] Workshop. April 18, 2011. Dublin, Ireland.

Personal Search refers to the process of searching within one’s personal space of digital information, e.g., searching one’s desktop or mobile phone for required data items or information. While some recent advancements have been made in this domain, research acceleration is hindered by the lack of established or standardized baselines and evaluation metrics, and lack of commonly available test collections. There is a clear consensus within the research community of the need for standardized repeatable evaluation techniques in the PS space, perhaps in the form of a TREC track for example. However, there are a number of significant challenges associated with this, not the least of which is the fact that the data associated with this domain is personal to the individual, multimedia in nature, and different users will have different forms of collections, differing information needs and different memories of required information.

This workshop aims to bring together researchers interested in working towards standardized evaluation approaches for the personal search space. Due to the large space that this covers, as a first step towards overall standardized personal search evaluation this workshop will focus on evaluation for the textual elements within personal desktop collections and known item keyword queries for these elements. An interactive forum for researchers to share ideas and initiate collaborations will be provided, with the explicit goal of establishing a means to evaluate personal search algorithms and solutions in a comparable and repeatable way. It is also intended to form a consortium of people at the workshop interested in bringing this evaluation solution to fruition in the subsequent months.

The topics of the workshop will be evaluation focused and include but not be limited to:

* Understanding personal search evaluation
* Interesting target tasks and explanations of their importance
* Critiques or comparisons of existing evaluation methods
* How we can combine existing evaluation methods
* Improving on previously suggested methods
* Making evaluation more realistic
* Reducing the cost of evaluation
* Proposal of new evaluation methods
* Using human computation games for evaluation
* User study approaches
* Simulation approaches


A number of resources are available prior to the workshop. We expect interested participants to be able to use these resources in various ways -- analyzing the characteristics of the data, evaluating various retrieval methods using the queries and result sets, suggesting a better data set for evaluating personal search, etc. Further information and download instructions are available at

Paper Submissions

The workshop is now accepting paper submissions. Short position papers (max. 2 pages) describing creative use of / modifications to provided datasets and approaches or ideas / challenges for the domain are invited. Submissions should be in ACM SIGIR format. LaTeX and Word templates are available at (for LaTeX, use the "Option 2" style).

Papers should be anonymised and submitted in pdf format through the EasyChair system no later than midnight Pacific Daylight Time on February 21, 2011. Submissions will be subject to double-blind reviewing. Accepted papers will be included in the Evaluating Personal Search Workshop 2011 proceedings.

Important Dates

February 21, 2011: Deadline for paper submission (midnight Pacific Daylight Time)
March 18, 2011: Notification to authors
March 25, 2011: Camera-ready copy due
April 18, 2011: Workshop

Further Information

Further information is available on the workshop website at or by emailing the workshop organizers.

Workshop Organizers

David Elsweiler – University of Erlangen, Germany (
Liadh Kelly – Dublin City University, Ireland (
Jinyoung Kim – UMass Amherst, USA (

Program Committee

Leif Azzopardi – University of Glasgow, UK

Robert Capra – University of North Carolina, USA
Sergey Chernov – L3S, Germany
Bruce Croft – Univ. of Massachusetts, Amherst, USA
Ronald Fernandez – University of Santiago de Compostela, Spain
Karl Gyllstrom – Katholieke Universiteit Leuven, Beligium
Donna Harman – NIST, USA
David Hawking – Funnelback, Australia
Gareth Jones – Dublin City University, Ireland
Noriko Kando – National Institute of Informatics, Japan
Diane Kelly – University of North Carolina, USA
David Losada – University of Santiago de Compostela, Spain
Ian Ruthven – University of Strathclyde, Glasgow, UK
Alan Smeaton – Dublin City University, Ireland
Jaime Teevan – Microsoft Research, Redmond, USA
Paul Thomas – CSIRO ICT Centre, Australia

Related Resources

WSDM 2020   ACM International Conference on Web Search and Data Mining
ITE 2021   2nd International Conference on Integrating Technology in Education
WSDM 2021   14th ACM Conference on Web Search and Data Mining
IJNGN 2020   International Journal of Next - Generation Networks
HealthWear 2020   (Virtual) HealthWear 2020 - 2nd EAI International Conference on Wearables in Healthcare
NLAI 2020   International Conference on NLP & Artificial Intelligence Techniques
EAI BODYNETS 2020   (Virtual) 15th EAI International Conference on Body Area Networks: Smart IoT and big data for intelligent health management
IJCSEIT 2020   International Journal of Computer Science, Engineering and Information Technology
WSDM Demos 2021   Demonstrations @ Web Search and Data Mining
MCS 2020   Monte Carlo Search 2020, IJCAI Workshop