posted by user: thorstenjoachims || 1551 views || tracked by 2 users: [display]

REVEAL 2018 : RecSys Workshop on Offline Evaluation for Recommender Systems

FacebookTwitterLinkedInGoogle

Link: https://sites.google.com/view/reveal2018/home
 
When Oct 7, 2018 - Oct 7, 2018
Where Vancouver, CA
Submission Deadline Jul 28, 2018
Notification Due Aug 13, 2018
Final Version Due Aug 27, 2018
Categories    recommender systems   offline evaluation   machine learning
 

Call For Papers

REVEAL 2018: Offline Evaluation for Recommender Systems

Workshop at ACM RecSys 2018, Vancouver, CA
October 7, 2018
https://sites.google.com/view/reveal2018/home

One of the main goals of offline metrics for recommender systems is to indicate the future online performance of the same recommender system in an online setting, as measured by various user-based utility metrics, such as the time spent by a user on the website, the number of media items consumed, the number of attributed sales or the reported user satisfaction. However, practitioners often observe significant differences between offline and online results of a new algorithm, and therefore tend to mostly rely on online methods such as A/B testing to evaluate their algorithms.

To this end, we welcome contributions that advance the current state of knowledge on offline recommendation metrics that correlate well with the final performance of the online recommender system; or new recommendation algorithms that directly optimize for online metrics.

We invite submissions of 2-8 pages to be presented as talks or posters. The reviews will be single-blind.

Potential contributions include (but are not limited to):

Framing the problem: what are we trying to solve exactly? This includes work on the theoretical foundations of the recommendation task and on the corresponding offline metrics:
- Recommendation as a counterfactual inference problem. This encompasses all work on new offline metrics and new optimization criteria for recommendation using ideas from causal inference, such as: Learning from data Missing Not at Random (MNAR), Counterfactual Risk Minimization (CRM) and Batch Learning from Bandit Feedback (BLBF), Deconvolving recommendation-lead from organic feedback in logged data, Causal Inference using domain adaptation
- Recommendation as a reinforcement learning problem. This is aimed to all work that frames recommendation as a reinforcement learning task and that borrows ideas from RL on evaluating recommendation policies, such as the use of simulation in recommender systems evaluation.

Studies on offline-online metrics correlation for Recommendation.

More realistic offline metrics.
- Offline metrics for slate recommendation evaluation.
- Offline metrics for logged data with sequential recommendation exposure and delayed feedback.

Datasets and toolkits.
- New exploration schemes to improve the collection of more informative offline datasets.
- Open datasets. All recommendation dataset releases that aim to somehow bridge the gap between offline and online metrics.
- Toolkits. Including new software for evaluation metrics, reproducible research and baseline algorithms.

Submission deadline: July 28th, 2018 via our CMT website at https://cmt3.research.microsoft.com/OFFLINEEVAL2018.

Related Resources

RecSys 2019   13th ACM Conference on Recommender Systems
ICCV 2019   International Conference on Computer Vision
ICML 2019   36th International Conference on Machine Learning
RecSys 2018   Conference on Recommender Systems
MLDM 2019   15th International Conference on Machine Learning and Data Mining MLDM 2019
UMUAI FatRec 2019   UMUAI Special issue on: Fair, Accountable, and Transparent Recommender Systems
ICDMML 2019   【ACM ICPS EI SCOPUS】2019 International Conference on Data Mining and Machine Learning
QEST 2019   16th International Conference on Quantitative Evaluation of SysTems
INRA 2018   CIKM 2018 Workshop: 6th International Workshop on News Recommendation and Analytics
CAIP 2019   Computer Analysis of Images and Patterns