posted by user: toracle2021 || 2967 views || tracked by 3 users: [display]

TORACLE 2021 : The 1st International Workshop on Test Oracles


When Aug 23, 2021 - Aug 27, 2021
Where Athens, Greece.Athee
Submission Deadline Jun 1, 2021
Notification Due Jul 1, 2021
Final Version Due Jul 8, 2021
Categories    software engineering   software testing   test oracle

Call For Papers

CFP Call For Papers

TORACLE 2021: The 1st International Workshop on Test Oracles

Co-located with The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2021)

Website workshop:
Website conference:
Twitter: @TOracle2021


The challenge of distinguishing correct from incorrect test executions is called the test oracle problem and it is recognised as one of the fundamental challenges in software testing. For the first time TORACLE 2021 will bring together researchers and practitioners working on the techniques, approaches and tools related to the alleviation and automation of the oracle problem. The workshop will enable the participants to present ideas, discuss recent results and develop a shared agenda for future research.

TORACLE calls for high quality papers related to the test oracle problem.

Automated generation of test case assertion oracles.
Automated generation of contracts, program assertions, invariants, pre and post-conditions.
Metrics and techniques for assessing the quality of oracles.
Oracles for specific types of software systems such as deep learning systems, machine learning systems, cyber-physical systems, web applications, and mobile applications.
Automated improvement of test oracles.
Automated extraction of oracles from various natural language sources.
Metamorphic oracles.
Oracles for non-functional properties, such as security, performance.
Oracle-based coverage criteria.
Empirical evaluations of oracle-related techniques in real-world scenarios.
Techniques for reducing the cost of manually deriving test oracles.
Theoretical formalisation and analysis of the oracle problem.


We accept the following types of submissions:

Long papers - 10-page papers (including references and appendices) that present authors' original research work and results.

Short papers - 4-page papers (including references and appendices) that present one of the following:

Position and Vision Paper – to propose radical, innovative new ideas, arguments, and research directions.

Work-in-Progress Paper – to describe novel work that is not at the same level of research maturity as a full submission.

Industrial and Experience Paper – to describe the experience gained from applying/evaluating techniques to alleviate the oracle problem in an industrial context.

Demonstration Paper – to describe new research tools, data, and other artifacts related to the test oracle problem. We also encourage the submission of demonstrations of already published research techniques.
However, a demonstration paper submission must not have been previously published in a demonstration form. Authors of accepted papers can choose to give either a presentation or a live demonstration.

TORACLE 2021 will employ a double-blind review process. The papers submitted must not reveal the authors’ identities in any way. The authors’ names must be omitted from the submission, and references to their prior work should be made in the third person. The only exception are Demonstration papers, which will employ a single-blind review process. The demonstration papers submitted do not need to hide the authors’ identities.

All papers must conform to the ESEC/FSE 2021 formatting instructions:
The page limit is strict. Submitted papers must be written in English and in PDF format. They must be unpublished and not submitted for publication elsewhere. Papers must be submitted electronically through the submission site:

Each submission will be reviewed by three members of the program committee. The main evaluation criteria include the relevance and quality of the submission in terms of technical soundness, novelty, relevance for the TORACLE audience, and presentation quality.

Accepted contributions will be included in the ESEC/FSE 2021 proceedings.

For more information visit the workshop website If you have doubts about your submission being in the scope of the workshop, do not hesitate to contact the organisers.

Paper submission deadline: June 1, 2021
Notification: July 1, 2021
Camera ready: July, 8 2021
Conference dates: August 23 - 27, 2021 Athens, Greece

Gunel Jahangirova
Università della Svizzera italiana (USI), Lugano, Switzerland

Valerio Terragni
University of Auckland, Auckland, New Zealand

Related Resources

CypressConf 2024   Test Automation Conference 2024
Ei/Scopus-AACIP 2024   2024 2nd Asia Conference on Algorithms, Computing and Image Processing (AACIP 2024)-EI Compendex
JEDT 2024   International Journal of Electronic Design and Test
SEMIT 2024   5th International Conference on Software Engineering and Managing Information Technology
CSET 2024   Cyber Security Experimentation and Test (CSET)
NLPTT 2024   5th International Conference on NLP Trends & Technologies
VDAT 2024   28th International Symposium on VLSI Design and Test (VDAT-2024)
DMSE 2024   5th International Conference on Data Mining and Software Engineering
WMT-Testsuites 2024   'Help us break LLMs' - Test suite sub-task of the Ninth Conference on Machine Translation (WMT24)
ACM MLMI 2024   ACM--2024 The 7th International Conference on Machine Learning and Machine Intelligence (MLMI 2024)