UTEP Computer Science Department Develops Award-Winning Interactive Agent System

January 21, 2016 | UNIVERSITY COMMUNICATIONS

University of Texas at El Paso Professor of Computer Science David Novick, Ph.D., and his students have developed a system for virtual agents and an immersive interactive application titled "Survival on Jungle Island."

The UTEP AGENT system was created to study understanding between embodied conversational agents (ECAs) and humans, with emphasis on the effects of paralinguistic behaviors on engagement and rapport. Paralinguistics includes behaviors such as gesture, intonation and rhythm in speech, gaze and turn taking. The team's research includes work on personality traits of humans and agents.

In "Survival on Jungle Island," the ECA and a human interact, using speech and gesture, in a 40-60 minute adventure composed of 23 scenes. A study conducted with the adventure showed that rapport increases when the ECA asks the human to perform task-related gestures and then perceives a human performing these gestures.

In the jungle adventure, the system simulates a survival scenario in which the player interacts with the ECA, Adriana. In order to survive, both human and ECA must collaborate, cooperate and build a relationship.

The game begins with a cinematic sequence of a stormy shipwreck, providing background information as to how the person ended up in the survival situation. This scene is followed by another two scenes, which serve as an inconspicuous tutorial to avoid breaking the impression of an immersive reality. The first scene contains personal questions that let the player and the ECA get to know each other, giving the impression that Adriana is processing what she hears. The second scene moves the story forward by providing an explanation as to how the ECA Adriana has survived in the jungle so far, and shows that she reacts to the player's responses. At the end of the scene, the story continues to develop depending on the player's choice of what to do next.

Novick created the system with the help of his Advanced aGent ENgagement Team (AGENT), comprising postdoctoral fellow Ivan Gris; UTEP undergraduate and graduate students Adriana Camacho, Diego Rivera, Mario Gutierrez, Laura Rodriguez, Alex Rayon, Joel Quintana, Anuar Jauregui, Timothy Gonzales, Alfonso Peralta, Victoria Bravo, Jacqueline Brixey, Yahaira Reyes and Paola Gallardo; French Air Force Academy cadets Guillaume Adoneth and David Manuel; and El Paso high school students Brynne Blaugrund and Nick Farber.

To enable the interaction between humans and ECAs, the team combined the use of Unity 4, an animation software; a Microsoft Kinect motion sensing device; and the Windows Speech SDK software. The team also built "middleware," which enabled more rapid development of applications including the jungle adventure.

UTEP's virtual-agent team now ranks among the best in the world," Novick said. "We are both building exciting new virtual-agent technology and learning how to make agents more adaptable to humans.