Salience and pointing in multimodal reference

Pointing combined with verbal referring is one of the most paradigmatic human multimodal behaviours. The aim of this paper is foundational: to uncover the central notions that are required for a computational model of human-generated multimodal referring acts. The paper draws on existing work on the generation of referring expressions and shows that in order to extend that work with pointing, the notion of salience needs to play a pivotal role. The paper investigates the role of salience in the generation of referring expressions and introduces a distinction between two opposing approaches: salience-first and salience-last accounts. The paper then argues that these differ not only in computational efficiency, as has been pointed out previously, but also lead to incompatible empirical predictions. The second half of the paper shows how a salience first account nicely meshes with a range of existing empirical findings on multimodal reference. A novel account of the circumstances under which speakers choose to point is proposed that directly links salience with pointing. Finally, a multidimensional model of salience is proposed to flesh this model out

Dialogue with computers: dialogue games in action

With the advent of digital personal assistants for mobile devices, systems that are marketed as engaging in (spoken) dialogue have reached a wider public than ever before. For a student of dialogue, this raises the question to what extent such systems are genuine dialogue partners. In order to address this question, this study proposes to use the concept of a dialogue game as an analytical tool. Thus, we reframe the question as asking for the dialogue games that such systems play. Our analysis, as applied to a number of landmark systems and illustrated with dialogue extracts, leads to a fine-grained classification of such systems. Drawing on this analysis, we propose that the uptake of future generations of more powerful dialogue systems will depend on whether they are self-validating. A self-validating dialogue system can not only talk and do things, but also discuss the why of what it says and does, and learn from such discussions

Presenting Arguments as Fictive Dialogue

Presentation of an argument can take many different forms ranging from a monologue to advanced graphics and diagrams. This paper investigates the presentation of one or more arguments in the form of a fictive dialogue. This technique was already employed by Plato, who used fictive conversations between Socrates and his contemporaries to put his arguments forward. Ever since, there have been influential authors – including Desiderius Erasmus, Sir Thomas More and Mark Twain – that have used dialogue in this way. In this paper, we define the notion of a fictive dialogue, motivate it is as a topic for investigation, and present a qualitative and quantitative study of five fictive dialogues by well-known authors. We conclude by indicating how our preliminary and ongoing investigations may inform the development of systems that automatically generate argumentative fictive dialogue

A Flexible pragmatics-driven language generator for animated agents

This paper describes the NECA MNLG; a fully implemented Multimodal Natural Language Generation module. The MNLG is deployed as part of the NECA system which generates dialogues between animated agents. The generation module supports the seamless integration of full grammar rules, templates and canned text. The generator takes input which allows for the specification of syntactic, semantic and pragmatic constraints on the output

A Formal Semantics for Editing and Generating Plurals

We present a formal semantics for an object-oriented formalism which allows for the representation of plural objects (such as 'Three N', Most of the N', 'Some N', . . .). the semantics is given in terms of a mapping to a variant of Discourse Representation Theory. It is motivated by its suitability for natural language generation and interactives editing of the representations

Accent Interpretation, Anaphora Resolution and Implicature Derivation

Abstract not available

Question generation in the CODA project

In the ongoing CODA project, we are developing a system for automatically converting monologue into dialogue. The dialogue is generated in a two-step approach. Firstly, snippets of input monologue are mapped to dialogue act sequences. Secondly, these sequences are verbalized. The conversion relies partly on analysing input monologue in terms of its discourse relations. This short paper briefly describes the approach to the first step in CODA. This approach involves the use of a parallel corpus of monologues and dialogues to learn mappings from monologue to dialogue acts. Here, we focus on dialogue acts that involve question asking

Proceedings of QG2010: The Third Workshop on Question Generation

These are the peer-reviewed proceedings of "QG2010, The Third Workshop on Question Generation". The workshop included a special track for "QGSTEC2010: The First Question Generation Shared Task and Evaluation Challenge". QG2010 was held as part of The Tenth International Conference on Intelligent Tutoring Systems (ITS2010)

Towards automated generation of scripted dialogue: some time-honoured strategies

The main aim of this paper is to introduce automated generation of scripted dialogue as a worthwhile topic of investigation. In particular the fact that scripted dialogue involves two layers of communication, i.e., uni-directional communication between the author and the audience of a scripted dialogue and bi-directional pretended communication between the characters featuring in the dialogue, is argued to raise some interesting issues. Our hope is that the combined study of the two layers will forge links between research in text generation and dialogue processing. The paper presents a first attempt at creating such links by studying three types of strategies for the automated generation of scripted dialogue. The strategies are derived from examples of human-authored and naturally occurring dialogue

Modality Choice for Generation of Referring Acts: Pointing versus Describing

The main aim of this paper is to challenge two commonly held assumptions regarding modality selection in the generation of referring acts: the assumption that non-verbal means of referring are secondary to verbal ones, and the assumption that there is a single strategy that speakers follow for generating referring acts. Our evidence is drawn from a corpus of task-oriented dialogues that was obtained through an observational study. We propose two alternative strategies for modality selection based on correlation data from the observational study. Speakers that follow the first strategy simply abstain from pointing. Speakers that follow the other strategy make the decision whether to point dependent on whether the intended referent is in focus and/or important. This decision precedes the selection of verbal means (i.e., words) for referring