Learning Icelandic in Virtual Reykjavik: Simulating real-life conversations with embodied conversational agents using multimodal clarification requests

This thesis forms part of the project Icelandic Language and Culture Training in Virtual Reykjavik, a 3D computer game that enables learners of Icelandic to practise oral language and listening. The aim of the project was to build a computer game populated with embodied conversational agents (ECAs) endowed with realistic multimodal behaviour, with a long-term goal of supporting authentic teaching of Icelandic language and culture. The part of the project reported in this thesis focused on examining human verbal and non-verbal features in clarification requests (CRs). Six multimodal CR models were suggested for implementation, with the intention of promoting a more realistic human-agent interaction in Virtual Reykjavik. The research took place in three phases. First, a small survey was carried out, eliciting learners’ expectations from Virtual Reykjavik. It informed about learners’ expectations of a 3D application. Learners reported difficulties in practising spoken Icelandic with native speakers in real life and for this reason said they would appreciate a virtual learning environment for practising oral language. The pedagogical foundation of Virtual Reykjavik considers the communicative approach in language instruction, task- and game-based learning, and multimodal and individual language learning approaches. Virtual Reykjavik was populated with ECAs endowed with multimodal behaviour that is authentic to Icelandic culture. Engaging in the game provided learners with an opportunity to experience Icelandic language as it is spoken in the target culture but in a virtual learning environment, and prior to engaging with speakers in the real world.
The communicative function CR was chosen as the main object of multimodal analysis, in order to narrow down the focus to a specific topic in natural language research. CR is one of the most commonly used utterance-types in spoken conversations (Purver, 2004); it helps to clarify what has previously been said but for whatever reason not understood by the recipient, and as such facilitates smooth conversational flow. For these reasons, CR is very important in achieving a realistic human-agent interaction in systems, like ours, which combine automatic speech recognition and pre-planned dialogues. In this second phase, natural language data was collected in order to analyse the verbal and non-verbal features in various types of CRs. Due to the complexity of spoken language and a wide range of possible conversational scenarios, data were collected only during first encounters asking for directions to a location in central Reykjavik. This in turn reflected the same task learners would need to do in Virtual Reykjavik - they would ask agents for directions in central Virtual Reykjavik and the agents would use clarification strategies in an authentic way. It should, however, not be seen as an exhaustive treatise about the nature of CRs but rather as a multimodal description of CRs, their use in a particular conversational scenario in the game, and their application to the development of human-like behaviour. Based on a database of video recordings of real-life conversations between native and non-native speakers of Icelandic, six different multimodal CR types were characterised. (165 recordings with total recorded time 1 hour, 59 minutes and 2 seconds; 108 native-non-native speaker pairs and 57 native-native speaker pairs, men and women; ages of native speakers between 18-70 with average age approximately 35 years, and ages of non-native speakers between 20-40 with average age approximately 30 years). Out of this database, a multimodal corpus of CRs was created, consisting of verbal and non-verbal data for each type of CR. Video recordings were analysed using the ELAN tagging and annotation package. Each analysis consisted of a description of multimodal data. The multimodal approach to language and the multimodal interaction analysis were used to analyse the verbal and non-verbal features of CRs. Due to resource constraints, only two types, the Ellipsis and the Fragment (Interjection Strategy), were implemented.
Finally, a user response study was conducted in order to find out how learners perceived multimodal behaviour of ECAs in the game, and whether surveyed learners noticed the two implemented CRs. Learners perceived the CR Fragment (Interjection Strategy) as the most natural, despite its being perceived as slightly rude or used too frequently by the ECAs. The frequency of use of CRs by the ECAs was not measured, since the focus was on learners as users of this game prototype. The study revealed many possibilities for improving the multimodal behaviour of ECAs which could be implemented in future versions. In particular, certain facial expressions, and their lack of ability to smile, were commonly perceived by learners as “creepy”.
In summary, this thesis presents the rationale for building a 3D computer game for teaching Icelandic language and culture, with a focus on practising oral language skills. It presents pedagogical background for including authentic features into the multimodal behaviour of ECAs in a computer game to achieve a more realistic human-agent interaction, and thus to contribute to an improved learning experience in an online virtual learning environment. Six clarification strategies used by native speakers of Icelandic were observed when they were approached by other native and non-native speakers asking for directions. The thesis also outlines points for future work on CRs and Virtual Reykjavik. Exploration of multimodal CRs in other conversational settings and languages would be useful for further improving ECA CRs used in Virtual Reykjavik. A good starting point for a continuation would be to conduct a new study with more complete instructions, learning materials and scaffolding, a fully functioning speech recognition system in Virtual Reykjavik, and ECAs endowed with additional features including smiling.