HRI 2009 Evaluation Criteria

The criteria are good guidelines for doing human robot interaction research. -Bob

The Evaluation Criteria for papers are now available. All papers must:

a) Be relevant to the field of human-robot interaction. So, for example, a paper that describes a new face tracking algorithm needs to demonstrate how it is of direct use to human-robot interaction. A paper contributing a face recognition technology should use standard recognition metrics (e.g., ROC curve) as well as demonstrate or highlight a path to “feasibility” in human-robot domains with regard to interactive performance, sufficient accuracy, integration with closed loop control, etc. Similarly, a study of the elderly must show how insights from the study directly inform the design of robots for this population and a wizard-of-oz experiment should show how findings contribute to our understanding of how people might interact with robotic capabilities that are plausible (if not currently available).

b) Clearly articulate: 1) the contribution to HRI, 2) how the contribution advances the state-of-the art or knowledge in HRI, and 3) how the contribution relates to other work in HRI as well as the fields of study on which the paper draws (e.g. psychology, cognitive science, anthropology, computer vision, artificial intelligence, speech recognition, etc.).

c) Be technologically and methodologically sound based on the criteria generally used for that technology/method within a given field. For example, conventions used in psychology for conducting experiments with people and analyzing the data, and reporting the study (e.g. hypotheses, manipulation checks, the creation of scales, ANOVA analyses, correlation tables, etc.) should be applied. Authors should take care to use correct terminology for their methods to avoid being evaluated against the incorrect set of criteria. For example, a user study of 5 people should be referred to as a user study or evaluation and not an experiment.

d) For empirical papers, provide adequate detail for readers to understand what was done, how the data were collected, from how many people, what were the characteristics of these people, what questions were people asked, what type of robot was involved (if a robot was used), etc.

e) Be written to be accessible for a broad, interdisciplinary/multidisciplinary HRI audience.

We particularly encourage papers that bring together subfields and investigate problems that have not been explored and are novel to HRI.