Skip to main content

IBM Cambridge Research Center

  Project: Conversant Assistants: Tools that understand you

Researchers: Carolyn Boettner, Dan Gruen, Candy Sidner
Project contact: research@lotus.com

A Collaborative User Experience Project:

Imagine handling your email and checking your calendar and to-do list in the car on the way to work. Imagine scheduling appointments from a pay-phone in a crowded airport, or hearing about new email messages without interrupting your work. Making these scenarios possible is one goal of the Conversant Assistant project.

Collaborating with the Speech Group at IBM Research, Yorktown Heights, Lotus Research conducted a project that focused on the use of conversational models in user interfaces and system design. Our goal was to develop tools that are "conversant" -- that is, they "understand" the tasks and activities their users perform, and use this understanding to communicate intelligently about them. Our tools will provide for conversations between people and their assistants on tasks involving email and calendaring. They make use of software agents that facilitate collaboration between people and that assist with the management of complex tasks. The tools will be developed based on results of field studies of everyday, actual work settings. Our long term goals include running the Conversant Assistant both on PDA-like devices and workstations, supporting hands-free work on the road and hands-on work in the office. The Conversant Assistant that we built for this project is based on the Collagen software manager system developed jointly by Lotus Research and MERL, a Mitsubishi Electric Research lab.

This conversant assistant, which we informally call Daffy, participates with a user in working on email as well as simple scheduling tasks. The agent can observe the user as he or she opens, reads and responds to email, or it can perform portions of those tasks at the user's request. The agent will perform some operations such as opening and closing windows, on its own, based on its understanding of what the user is doing. Daffy can explain how to do tasks, such as how to read an email message using the interface to the email program. It can also give the user a summary of what has been done so far, as well as what actions have been done in the past or are expected in the future. For these behaviors Daffy relies on the segmented interaction history that is part of the Collagen system on which Daffy is based. Daffy can also follow interruptions in email tasks and remember tasks to be done later.

Daffy uses speech recognition and sentence-level interpretation tools from IBM Speech Research for the email domain. Using these tools, Daffy can understand requests spoken in English. Daffy uses the synthesis component of IBM's ViaVoice (TM) system to "speak" the utterances that are produced as part of its user interactions.

More long term, the assistant will perform in disconnected as well as connected environments, letting the user move smoothly between speech only, GUI only, or speech and GUI together. Because the assistant will understand what the user is trying to do, and not just the specific tools and programs being used, it can handle many of the details during transitions.

Related publications:

TR 99-08, Dan Gruen, Candy Sidner, Carolyn Boettner and Charles Rich. A Collaborative Assistant for Email
TR 98-14 Neal Lesh, Charles Rich and Candace L. Sidner. Using Plan Recognition in Human-Computer Collaboration
TR 98-13 Ada H. Cheung. A Collaborative Interface Agent for Lotus eSuite Mail
TR 98-04 Charles Rich and Candace L. Sidner. COLLAGEN: A Collaboration Manager for Software Interface Agent
TR 96-12, C. Rich and C. Sidner. COLLAGEN: When Agents Collaborate with People
TR 96-11, C. Rich and C. Sidner.
Segmented Interaction History in a Collaborative Agent
TR 96-10, C. Rich and C. Sidner.
Adding a Collaborative Agent to Graphical User Interfaces
TR 94-09, C. Sidner. An Artificial Discourse Language for Collaborative Negotiation
TR 94-10, C. Sidner. Negotiation in Collaborative Activity: A Discourse Analysis