Virtual Humans and Conversational Agents

JUSTTALK: Spoken Dialog with an 
Emotive Virtual Character


Curry Guinn

 “Artificial Intelligence and Natural Language Processing for Advanced Tactical Decision-Making Immersive Training”, RTI International, $100,156, Funded.  (start: September 16, 2019, end: June 24, 2021). 


inn, C. I., Responsive Virtual Human Technology Research, National Science Foundation, #EIA-0121211, September 2001-August 2004, $ 2,042,547.00.

Schulze S., Pence T., Irvine N., Guinn C. (2019) The Effects of Embodiment in Virtual Reality on Implicit Gender Bias. In: Chen J., Fragomeni G. (eds) Virtual, Augmented and Mixed Reality. Multimodal Interaction. HCII 2019. Lecture Notes in Computer Science, vol 11574. Springer. 

Principal Investigator leading the development of interactive emotive virtual humans with emotions that affect their body and facial gestures, decision-making, and language generation (Funding from National Science Foundation, commercial, and government clients).  RTI’s AVAtalk™ technology creates responsive, individual virtual personalities. In an AVAtalk™ -enabled application, users carry on spoken conversation with a simulated person (an avatar) and see and hear realistic responses from the avatar.  RTI’s ExhibitAR is an interactive kiosk for tradeshows and promotional events. It features male and female virtual people who talk with and respond to kiosk visitors with informative and enjoyable spoken dialog.

·         The Virtual Standardized Patient: Simulated Patient-Practitioner Dialogue for Patient Interview Training. With Hubal, R.C., Kizakevich, P.N., Merino, K.D., & West, S.L. In J.D. Westwood, H.M. Hoffman, G.T. Mogel, R.A. Robb, & D. Stredney (Eds.), Envisioning Healing: Interactive Technology and the Patient-Practitioner Dialogue. IOS Press: Amsterdam, 2000. (htm) , (doc)


We describe the Virtual Standardized Patient (VSP) application, having a computerized virtual person who interacts with medical practitioners in much the same way as actors hired to teach and evaluate patient assessment and interviewing skills. The VSP integrates technologies from two successful research projects conducted at Research Triangle Institute (RTI). AVATALK provides natural language processing, emotion and behavior modeling, and composite facial expression and lip-shape modeling for a natural patient-practitioner dialogue. Trauma Patient Simulator (TPS) provides case-based patient history and trauma casualty data, real-time physiological modeling, interactive patient assessment, 3-D scenario simulation, and instructional record-keeping capabilities. The VSP offers training benefits that include enhanced adaptability, availability, and assessment.

·         A Test of Responsive Virtual Human Technology as an Interviewer Skills Training Tool. With Link, M.W., Armsby, P. P., and Hubal, R. Proceedings of the 2002 Annual Conference of the American Association for Public Opinion Research , St. Petersburg. 2002. (htm) , (doc)


Research on survey non-response suggests that advanced communication and listening skills are among the best strategies telephone interviewers can employ for obtaining survey participation, allowing them to identify and address respondents' concerns immediately with appropriate, tailored language. Yet, training on interaction skills is typically insufficient, relying on role-playing or passive learning through lecture and videos. What is required is repetitive, structured practice in a realistic work environment. This research examines acceptance by trainees of an application based on responsive virtual human technology (RVHT) as a tool for teaching refusal avoidance skills to telephone interviewers. The application tested here allows interviewers to practice confronting common objections offered by reluctant sample members. Trainee acceptance of the training tool as a realistic simulation of "real life" interviewing situations is the first phase in evaluating the overall effectiveness of the RVHT approach. Data were gathered from two sources -- structured debrief questionnaires administered to users of the application, and observations of users by researchers and instructors. The application was tested with a group of approximately fifty telephone interviewers of varying skill and experience levels. The research presents findings from these acceptance evaluations and discusses users' experiences with and perceived effectiveness of the virtual training tool.

·         JUST-TALK: An Application of Responsive Virtual Human Technology, with Geoffrey Frank and Robert Hubal, accepted for publication, Proceedings of the 24th Interservice/Industry Training, Simulation and Education Conference, 2002. (htm) , (doc)


In this paper, we describe an application of responsive virtual humans to train law enforcement personnel in dealing with subjects that present symptoms of serious mental illness. JUST-TALK provides a computerized virtual person to interact with the student in a role-playing environment. Students were able to converse with the virtual person using spoken natural language and see and hear the virtual personality a combination of facial gesture, body movements, and spoken language. The JUST-TALK project, funded by the National Institute of Justice Office of Science and Technology and developed by RTI International, involved integrating virtual reality training software within a 3-day class at the North Carolina Justice Academy. The course was structured to include classroom-based lecture, videos, discussion, live human role-playing, and virtual human role-playing.
A scientific evaluation of the class and the software system was carried out by North Carolina State University. This assessment investigated the contribution of natural language interfaces and virtual reality technology to learning in this applied setting. Results of the evaluation are extremely encouraging. The vast majority of students (88 percent) found the simulation easy to use. A majority of the students said the virtual trainer enhanced their learning in the course. As a training tool, students rated the computer simulation on par with other training methods including lecture, role-play and discussion. A total of 59 percent of students felt the simulation was better for learning or comparable to role-play; 77 percent felt simulation was better than or comparable to lecture; and 59 percent felt the simulation was better than or comparable to discussion.