EGPAI 2017 : 2nd International Workshop on Evaluating General-Purpose AI
Call For Papers
Call for Papers
2nd International Workshop - Evaluating General-Purpose AI
Melbourne, Australia, August 19-21, 2016
(satellite workshop of IJCAI 2017 http://ijcai-17.org/)
- Aims and Scope -
The 2nd international workshop on evaluating general-purpos AI (EGPAI2017) will be held in conjunction with IJCAI 2017 (http://ijcai-17.org/workshop-program.html) in Melbourne, Australia (August 19-21, 2017).
Up to now, most AI systems are tested on specific tasks. However, to be considered truly intelligent, a system should exhibit enough flexibility to find a diversity of solutions for a range of tasks, some of which may not be known until after the system is deployed. Very recently there has been a large number of events, challenges and platforms that are giving a new perspective to how AI can be evaluated, such as the Arcade Learning Environment video games, the Video Game Definition Language (VGDL), OpenAI Gym, Microsoft Malmo, OpenAI Universe, Facebook TorchCarft, Facebook CommAI-env, GoodAI school, Google DeepMind Lab, etc. This workshop will welcome formalisations, methodologies and testbenches for evaluating the numerous aspects of this type of general AI systems. More specifically, we are interested in theoretical or experimental research focused on the development of concepts, tools and clear metrics to characterise and measure the intelligence, and other cognitive abilities, of general AI agents. Furthermore, EGPAI2017 will participate in the IJCAI2017 special theme on AI & Autonomy. Therefore, the workshop will welcome papers on the evaluation of autonomous agents of any kind, such as robots, software agents, artificial life agents, and any sort of autonomous systems capable of operating in long-term, real-world scenarios. There will be a panel dealing with this topic.
We are interested in questions such as: Can the various tasks and benchmarks in AI provide a general basis for evaluation and comparison of a broad range of such systems?, Can there be a theory of tasks, or cognitive abilities, that enables a more direct comparison and characterisation of AI systems? How much does the specificity of an AI agent relate to how fast it can achieve acceptable performance?, How does the structure of a cognitive system relate to how easy or difficult a task - or various classes of tasks - are for it to perform and learn?
We welcome regular papers, demo papers about benchmarks or tools, and position papers, and encourage discussions over a broad list of topics (not exhaustive):
* Analysis and comparisons of AI benchmarks and competitions. Lessons learnt.
* Proposals for new general tasks, evaluation environments, workbenches and general AI development platforms.
* Theoretical or experimental accounts of the space of tasks, abilities and their dependencies.
* Evaluation of development in robotics and other autonomous agents, and cumulative learning in general learning systems.
* Tasks and methods for evaluating: transfer learning, cognitive growth, structural self-modification and self-programming.
* Evaluation of social, verbal and other general abilities in multi-agent systems, video games and artificial social ecosystems.
* Evaluation of autonomous systems: cognitive architectures and multi-agent systems versus general components: machine learning techniques, SAT solvers, planners, etc.
* Unified theories for evaluating intelligence and other cognitive abilities, independently of the kind of subject (humans, animals or machines): universal psychometrics.
* Analysis of reward aggregation and utility functions, environment properties (Markov, ergodic, etc.) in the characterisation of reinforcement learning tasks.
* Methods supporting automatic generation of tasks and problems with systematically introduced variations.
* Better understanding of the characterisation of task requirements and difficulty (energy, time, trials needed..), beyond algorithmic complexity.
* Evaluation of AI systems using generalised cognitive tests for humans. Computer models taking IQ tests. Psychometric AI.
* Application of (algorithmic) information theory, game theory, theoretical cognition and theoretical evolution for the definition of metrics of cognitive abilities.
* Adaptation of evaluation tools from comparative psychology and psychometrics to AI: Item Response Theory (IRT), adaptive testing, hierarchical factor analysis.
* Evaluation methods for multiresolutional perception in AI systems and agents.
Apart from the technical sessions, we are planning to have a demo session presenting real platforms and ways to evaluate AI systems for several tasks in these platforms; and a panel with a more lively discussion about the research challenges around the workshop topics and future initiatives.
- Invited Speakers -
* David L. Dowe, Associate Professor at Monash University (Australia).
* More invited speakers to be announced.
- Submission -
We solicit submissions (full or short papers) including:
* Original research contributions
* Applications and experiences
* Surveys, comparisons, and state-of-the-art reports
* Tool or demo papers
* Position papers related to the topics mentioned above.
* Work in progress papers.
Submitted papers must be formatted according to the camera-ready style for IJCAI 2017. Manuscripts must be submitted electronically via the easychair conference management system using the following link: https://easychair.org/conferences/?conf=egpai2017.
Authors of accepted papers will be asked to present the paper during the workshop. Online pre-proceedings containing all accepted papers will be prepared before the date of the conference. Depending on the number and quality of submissions, we will examine the possibility of targeting a volume or a journal special issue.
Papers are allowed a maximum of six (6) pages, references excluded. References can take up to one additional page. Formatting Guidelines, LaTeX Styles and Word Template can be downloaded from http://ijcai-17.org/FormattingGuidelinesIJCAI-17.zip .
- Important dates -
* Submission deadline: May 5th, 2017
* Notification of acceptance: June 9th, 2017
* Camera-ready: June 15th, 2017
* Workshop: August 19th-21st, 2017
- Program chairs -
* Nader Chmait Monash University
* Jose Hernandez-Orallo Technical University of Valencia (Spain)
* Fernando Martínez-Plumed Technical University of Valencia (Spain)
* Claes Strannegård Chalmers University of Technology
* Kristinn R. Thórisson Reykjavik University
- Program committee -
* Marco Baroni Facebook AI Research
* Jordi Bieger CADIA, Reykjavik University
* Angelo Cangelosi Plymouth University
* Emmanuel Dupoux EHESS
* Helgi P. Helgason Activity Stream
* Katja Hofmann Microsoft Research
* Sean B. Holden Cambridge University
* Estevam R. Hruschka Carnegie Mellon University
* Armand Joulin Facebook AI Research
* Jan Koutnik IDSIA
* Edward Keedwell Exeter University
* Tomas Mikolov Facebook AI Research
* Frans A. Oliehoek University of Amsterdam
* Ricardo B.C. Prudencio Uni. Fed. de Pernambuco
* Ute Schmid Bamberg University
* Bas Steunebrink IDSIA
* Pei Wang Temple University
- Contact Person –
Name: Nader Chmait