posted by organizer: gfursin || 2279 views || tracked by 2 users: [display]

ReQuEST-DNN 2018 : 1st Reproducible Quality-Efficient Systems Tournament: Pareto efficient deep learning


When Mar 24, 2018 - Mar 24, 2018
Where Williamsburg, VA, USA
Abstract Registration Due Feb 5, 2018
Submission Deadline Feb 12, 2018
Notification Due Feb 22, 2018
Final Version Due Mar 9, 2018
Categories    artificial intelligence   computer science   machine learning

Call For Papers



ReQuEST: 1st Reproducible Quality-Efficient Systems Tournament
Call for Pareto efficient deep learning

Intent to submit: February 5, 2018 AoE

Associated ReQuEST workshop co-located with ASPLOS 2018
March 24th, 2018 (afternoon), Williamsburg, VA, USA


Co-designing emerging workloads across the hardware/software
stack to optimize for speed, accuracy, costs and other metrics
is extremely complex and time consuming. The lack of
a rigorous methodology and common tools for open, reproducible
and multi-objective optimization makes it challenging or even
impossible to evaluate and compare different published works
across numerous heterogeneous hardware platforms, software
frameworks, compilers, libraries, algorithms, data sets, and

The 1st ReQuEST workshop aims to bring together
multidisciplinary researchers in systems, compilers,
architecture and machine learning to optimize the quality vs.
efficiency Pareto optimality of deep learning systems
on complete hardware/software platforms in a standardized,
reproducible and comparable fashion. The target application
for the first incarnation of ReQuEST will be the ImageNet
Large Scale Visual Recognition Challenge (ILSVRC) and will
focus solely on optimizing inference on real systems
. Restricting the competition to a single application domain
will allow us to test an open-source tournament
infrastructure, validate it across multiple platforms and
environments, and prepare a dedicated live scoreboard with
the "winning" solutions. For future incarnations of ReQuEST,
we will provide broader application coverage.

Unlike the classical ILSVRC where submissions are ranked based
on accuracy, ReQuEST submissions will be evaluated across
multiple metrics and trade-offs (exposed by authors):
accuracy, speed, throughput, energy, cost of usage, etc.
Furthermore, in contrast with other deep learning benchmarking
challenges ReQuEST participants will be asked to submit
a complete workflow artifact (see submission procedures) which
encompasses toolchains, frameworks, algorithm, libraries, and
target hardware platform; any of which can be fine-tuned,
or customized at will by the participant to implement their
optimization technique.

We strongly encourage artifact submissions for already
published techniques since one of the ReQuEST goals is to
prepare an open set of reference and optimized implementations
of popular deep learning algorithms as portable and
customizable workflows which can be easily reused, improved
and build upon!

A ReQuEST artifact evaluation committee (AEC) will be tasked
to independently reproduce and evaluate workflow submissions
on compliant hardware platforms to reproduce results and
aggregate them in a multi-objective public leaderboard. Due to
the multi-faceted nature of the competition, submissions won't
be ranked according to a single metric, but instead the AEC
will assess their Pareto optimality across two or more metrics
exposed by authors. There won't be a single ranking
of submissions since this competition is multi-objective:
it accounts for classification accuracy, inference latency,
energy, ownership/usage cost and so on. As such, there won't
be a single winner, but better and worse designs based
on their relative Pareto optimality (up to 3 design points
allowed per submission).

The workshop co-located with ASPLOS 2018 will be the
opportunity for the participants to share their research and
implementation insights with the research community. A common
academic and industrial panel will be held at the end of the
workshop to discuss how to improve common SW/HW co-design
methodology for deep learning and other real-world
Further details about deadlines, submission procedures
and artifact evaluation:


Looking forward to your participation and submissions!

Related Resources

MDPI-Electronics-SI-QualRiskDistribSys 2022   MDPI Electronics Special Issue on Quality Assurance and Risk Mitigation in Large-Scale Distributed Systems
JCRAI 2021-Ei Compendex & Scopus 2021   2021 International Joint Conference on Robotics and Artificial Intelligence (JCRAI 2021)
HEART 2022   2022 : 12th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies
EI-CFAIS 2021   2021 International Conference on Frontiers of Artificial Intelligence and Statistics (CFAIS 2021)
SIUSAI 2022   2022 International Symposium on Intelligent Unmanned Systems and Artificial Intelligence (SIUSAI 2022)
IJCAI 2022   31st International Joint Conference on Artificial Intelligence
ICSR 2022   20th International Conference on Software and Systems Reuse
CVPR 2022   Computer Vision and Pattern Recognition
WAISE 2021   4th International Workshop on Artificial Intelligence Safety Engineering
ICCSEA 2021   11th International Conference on Computer Science, Engineering and Applications