posted by organizer: gfursin || 738 views || tracked by 1 users: [display]

ReQuEST-DNN 2018 : 1st Reproducible Quality-Efficient Systems Tournament: Pareto efficient deep learning


When Mar 24, 2018 - Mar 24, 2018
Where Williamsburg, VA, USA
Abstract Registration Due Feb 5, 2018
Submission Deadline Feb 12, 2018
Notification Due Feb 22, 2018
Final Version Due Mar 9, 2018
Categories    artificial intelligence   computer science   machine learning

Call For Papers



ReQuEST: 1st Reproducible Quality-Efficient Systems Tournament
Call for Pareto efficient deep learning

Intent to submit: February 5, 2018 AoE

Associated ReQuEST workshop co-located with ASPLOS 2018
March 24th, 2018 (afternoon), Williamsburg, VA, USA


Co-designing emerging workloads across the hardware/software
stack to optimize for speed, accuracy, costs and other metrics
is extremely complex and time consuming. The lack of
a rigorous methodology and common tools for open, reproducible
and multi-objective optimization makes it challenging or even
impossible to evaluate and compare different published works
across numerous heterogeneous hardware platforms, software
frameworks, compilers, libraries, algorithms, data sets, and

The 1st ReQuEST workshop aims to bring together
multidisciplinary researchers in systems, compilers,
architecture and machine learning to optimize the quality vs.
efficiency Pareto optimality of deep learning systems
on complete hardware/software platforms in a standardized,
reproducible and comparable fashion. The target application
for the first incarnation of ReQuEST will be the ImageNet
Large Scale Visual Recognition Challenge (ILSVRC) and will
focus solely on optimizing inference on real systems
. Restricting the competition to a single application domain
will allow us to test an open-source tournament
infrastructure, validate it across multiple platforms and
environments, and prepare a dedicated live scoreboard with
the "winning" solutions. For future incarnations of ReQuEST,
we will provide broader application coverage.

Unlike the classical ILSVRC where submissions are ranked based
on accuracy, ReQuEST submissions will be evaluated across
multiple metrics and trade-offs (exposed by authors):
accuracy, speed, throughput, energy, cost of usage, etc.
Furthermore, in contrast with other deep learning benchmarking
challenges ReQuEST participants will be asked to submit
a complete workflow artifact (see submission procedures) which
encompasses toolchains, frameworks, algorithm, libraries, and
target hardware platform; any of which can be fine-tuned,
or customized at will by the participant to implement their
optimization technique.

We strongly encourage artifact submissions for already
published techniques since one of the ReQuEST goals is to
prepare an open set of reference and optimized implementations
of popular deep learning algorithms as portable and
customizable workflows which can be easily reused, improved
and build upon!

A ReQuEST artifact evaluation committee (AEC) will be tasked
to independently reproduce and evaluate workflow submissions
on compliant hardware platforms to reproduce results and
aggregate them in a multi-objective public leaderboard. Due to
the multi-faceted nature of the competition, submissions won't
be ranked according to a single metric, but instead the AEC
will assess their Pareto optimality across two or more metrics
exposed by authors. There won't be a single ranking
of submissions since this competition is multi-objective:
it accounts for classification accuracy, inference latency,
energy, ownership/usage cost and so on. As such, there won't
be a single winner, but better and worse designs based
on their relative Pareto optimality (up to 3 design points
allowed per submission).

The workshop co-located with ASPLOS 2018 will be the
opportunity for the participants to share their research and
implementation insights with the research community. A common
academic and industrial panel will be held at the end of the
workshop to discuss how to improve common SW/HW co-design
methodology for deep learning and other real-world
Further details about deadlines, submission procedures
and artifact evaluation:


Looking forward to your participation and submissions!

Related Resources

ECCV 2018   European Conference on Computer Vision
OSDI 2018   13th USENIX Symposium on Operating Systems Design and Implementation
ICML 2018   The 35th International Conference on Machine Learning
ICANN 2018   27th International Conference on Artificial Neural Networks
IJCAI 2018   International Joint Conferences on Artificial Intelligence Organization
Social Information Systems @ HICSS-51   Social Information Systems Minitrack - Hawaii International Conference on System Sciences (HICSS-51)
ACIIDS 2018   10th Asian Conference on Intelligent Information and Database Systems
ACL 2018   56th Annual Meeting of the Association for Computational Linguistics
GECCO 2018   Genetic and Evolutionary Computation Conference
RecSys 2018   Conference on Recommender Systems