CS 898: Advanced Interaction Design

Instructor

Professor Michael Terry (mterry@cs.uwaterloo.ca)
Office: DC 2118
Office hours: TBA

Meeting Time and Location

MC 2036
MW 1-2:30PM

Introduction

Today, we are blessed with a wealth of technological capability: Processors run at gigahertz speeds, RAM is measured in gigabytes, and long-term storage is quickly moving from the gigabyte range to the terabyte range. At the same time, the fruits of research in computer science and other domains have created plug-and-play libraries that require minimal theoretical knowledge for developers to be able to effectively use. Physics, machine learning, 3D graphics, and computer vision are all now highly accessible via well-documented, robust libraries, lessening the need for domain experts from these specialized fields when one wishes to incorporate this technology in new user interfaces.

Despite all of these advances, human-computer interfaces have remained relatively stable for the past 25 years. WIMP interfaces are still the dominant form of interaction. Web applications have developed slightly different conventions for interaction, but they still assume a keyboard and mouse. Pen input and modern graphics cards have introduced some new interaction possibilities, though pen interaction is still largely a thin layer on WIMP interfaces, while graphics cards tend to add more aesthetic value than functional value in user interfaces.

There are numerous reasons for the slow adoption of new interaction techniques. It can be costly: People need to be trained in the proper use of the technology, the technology itself may be costly (either in terms of hardware or licensing costs), or it simply might not provide enough of an improvement over existing methods to justify its adoption. Or, the technology may simply not be fully developed, tested, or packaged in a form that enables its easy integration into an application. This course deals with these latter three issues.

Course Goals

In this course, you will gain expertise in the design of novel interaction techniques. You will consider interaction techniques from the following, interrelated views:

You will gain this expertise by doing the following:

By the end of the course, you will have designed, implemented, tested, and packaged an interaction technique that end users will find useful and that developers will be able to download and integrate in their applications. This course is your opportunity to apply expertise in fields such as machine learning, computer vision, optimization, visual design, and other specialities, to the design of new interaction tools for end-users.

Because of the workload, it is highly recommended you combine the course project with your thesis work or other course projects, pending approval by all parties. There will be a final demo day (time and place TBA) in which others will be able to interact with you technique.

CS 489 / 698 is highly recommended as a precursor to this course. If you have not taken this course, the following resources will be useful to you:

Course Format

The course will be a mixture of lectures, student presentations, and design critiques. All lectures and presentations will be given in a round-table format to facilitate interaction.

Course Marking

Course Readings

Most classes have assigned readings. You must complete the readings prior to the class and submit a short summary for each reading. The summary should contain the following:

The summaries should be no more than one page long per reading. At the end of the course, they will serve as a summary of the readings for your own future research.

After the first week of lecture, students will be leading discussions.

Course Project

The easiest technology to create is technology that is needs-driven – technology that addresses real-world, human needs. Accordingly, in this course, you must identify a real-world problem that your technology will address. This problem must be approved by the course instructor.

You will be addressing your identified problem via the creation of an interaction technique. Interaction techniques are tools that allow people to get work done more efficiently, without severely impacting the range of problems they can solve. Drag-and-drop, pie menus, pen-based gestures, tool-tips, and voice-based input are all examples of interaction techniques. An interaction technique should be a standalone, identifiable entity that can be applied across domains and contexts. Machine learning itself is not an interaction technique, but the recognition of incorrect grammar via machine learning techniques (along with the presentation of those errors) is an interaction technique. Standalone sticky notes are not an interaction technique (they’re an application), but comments embedded within a document are an interaction technique.

To get you thinking about real-world problems and interaction techniques, here are some ideas:

Syllabus

The following is a tentative syllabus.

Date

Topic

Assignments / Readings

January 7

Introduction + Overview

January 9

Interaction design

Beaudouin-Lafon, M. 2004. Designing interaction, not interfaces. In Proceedings of the Working Conference on Advanced Visual interfaces (Gallipoli, Italy, May 25 - 28, 2004). AVI '04. ACM, New York, NY, 15-22. DOI= http://doi.acm.org/10.1145/989863.989865

Beaudouin-Lafon, M. and Mackay, W. E. 2000. Reification, polymorphism and reuse: three principles for designing visual interfaces. In Proceedings of the Working Conference on Advanced Visual interfaces (Palermo, Italy). AVI '00. ACM, New York, NY, 102-109. DOI= http://doi.acm.org/10.1145/345513.345267

Beaudouin-Lafon, M. 2000. Instrumental interaction: an interaction model for designing post-WIMP user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands, April 01 - 06, 2000). CHI '00. ACM, New York, NY, 446-453. DOI= http://doi.acm.org/10.1145/332040.332473

 

 

 

January 14

Tool design

Resnick, Berg, and Eisenberg. Beyond Black Boxes: Bringing Transparency and Aesthetics Back to Scientific Investigation. Published in Journal of Learning Sciences (vol. 9, no. 1, 2000)

Myers, B., Hudson, S. E., and Pausch, R. 2000. Past, present, and future of user interface software tools. ACM Trans. Comput.-Hum. Interact. 7, 1 (Mar. 2000), 3-28. DOI= http://doi.acm.org/10.1145/344949.344959

January 16

Input devices

Card, S. K., Mackinlay, J. D., and Robertson, G. G. 1991. A morphological analysis of the design space of input devices. ACM Trans. Inf. Syst. 9, 2 (Apr. 1991), 99-122. DOI= http://doi.acm.org/10.1145/123078.128726

John et al. Bringing Usability Concerns to the Design of Software Architecture.

 

 

 

January 21

Beyond WIMP interaction

Project proposal due

Kurtenbach, G., Fitzmaurice, G., Baudel, T., and Buxton, B. 1997. The design of a GUI paradigm based on tablets, two-hands, and transparency. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, United States, March 22 - 27, 1997). S. Pemberton, Ed. CHI '97. ACM, New York, NY, 35-42. DOI= http://doi.acm.org/10.1145/258549.258574

Bederson, B. B. and Hollan, J. D. 1994. Pad++: a zooming graphical interface for exploring alternate interface physics. In Proceedings of the 7th Annual ACM Symposium on User interface Software and Technology (Marina del Rey, California, United States, November 02 - 04, 1994). UIST '94. ACM, New York, NY, 17-26. DOI= http://doi.acm.org/10.1145/192426.192435

January 23

Alternative menus

Guimbretiére, F. and Winograd, T. 2000. FlowMenu: combining command, text, and data entry. In Proceedings of the 13th Annual ACM Symposium on User interface Software and Technology (San Diego, California, United States, November 06 - 08, 2000). UIST '00. ACM, New York, NY, 213-216. DOI= http://doi.acm.org/10.1145/354401.354778

Kurtenbach, G. and Buxton, W. 1994. User learning and performance with marking menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating interdependence (Boston, Massachusetts, United States, April 24 - 28, 1994). B. Adelson, S. Dumais, and J. Olson, Eds. CHI '94. ACM, New York, NY, 258-264. DOI= http://doi.acm.org/10.1145/191666.191759

 

 

 

January 28

Physical construction of interfaces

Hudson, S. E. and Mankoff, J. 2006. Rapid construction of functioning physical interfaces from cardboard, thumbtacks, tin foil and masking tape. In Proceedings of the 19th Annual ACM Symposium on User interface Software and Technology (Montreux, Switzerland, October 15 - 18, 2006). UIST '06. ACM, New York, NY, 289-298. DOI= http://doi.acm.org/10.1145/1166253.1166299

Greenberg, S. and Fitchett, C. 2001. Phidgets: easy development of physical interfaces through physical widgets. In Proceedings of the 14th Annual ACM Symposium on User interface Software and Technology (Orlando, Florida, November 11 - 14, 2001). UIST '01. ACM, New York, NY, 209-218. DOI= http://doi.acm.org/10.1145/502348.502388

January 30

Machine learning and interfaces

Fails, J. A. and Olsen, D. R. 2003. Interactive machine learning. In Proceedings of the 8th international Conference on intelligent User interfaces (Miami, Florida, USA, January 12 - 15, 2003). IUI '03. ACM, New York, NY, 39-45. DOI= http://doi.acm.org/10.1145/604045.604056

Horvitz, E. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: the CHI Is the Limit (Pittsburgh, Pennsylvania, United States, May 15 - 20, 1999). CHI '99. ACM, New York, NY, 159-166. DOI= http://doi.acm.org/10.1145/302979.303030

 

 

 

February 4

Gesture recognition

Saund, E. and Lank, E. 2003. Stylus input and editing without prior selection of mode. In Proceedings of the 16th Annual ACM Symposium on User interface Software and Technology (Vancouver, Canada, November 02 - 05, 2003). UIST '03. ACM, New York, NY, 213-216. DOI= http://doi.acm.org/10.1145/964696.964720

Wobbrock, J. O., Wilson, A. D., and Li, Y. 2007. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology (Newport, Rhode Island, USA, October 07 - 10, 2007). UIST '07. ACM, New York, NY, 159-168. DOI= http://doi.acm.org/10.1145/1294211.1294238

February 6

Sensors

Fogarty, J., Hudson, S. E., Atkeson, C. G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J. C., and Yang, J. 2005. Predicting human interruptibility with sensors. ACM Trans. Comput.-Hum. Interact. 12, 1 (Mar. 2005), 119-146. DOI= http://doi.acm.org/10.1145/1057237.1057243

Fogarty, J. and Hudson, S. E. 2007. Toolkit support for developing and deploying sensor-based statistical models of human situations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA, April 28 - May 03, 2007). CHI '07. ACM, New York, NY, 135-144. DOI= http://doi.acm.org/10.1145/1240624.1240645

 

 

 

February 11

Computer vision

Project lit review due

Maynes-Aminzade, D., Winograd, T., and Igarashi, T. 2007. Eyepatch: prototyping camera-based interaction through examples. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology (Newport, Rhode Island, USA, October 07 - 10, 2007). UIST '07. ACM, New York, NY, 33-42. DOI= http://doi.acm.org/10.1145/1294211.1294219

AR Toolkit

OpenCV

Wilson, A., and N. Oliver. GWindows: Robust Stereo Vision for Gesture-Based Control of Windows, International Conference on Multimodal Interfaces, 2003. pdf

Wilson, A., and N. Oliver. Multimodal Sensing for Explicit and Implicit Interaction, HCI International, 2005 pdf

February 13

Optimization techniques

Fogarty, J. and Hudson, S. E. 2003. GADGET: a toolkit for optimization-based approaches to interface and display generation. In Proceedings of the 16th Annual ACM Symposium on User interface Software and Technology (Vancouver, Canada, November 02 - 05, 2003). UIST '03. ACM, New York, NY, 125-134. DOI= http://doi.acm.org/10.1145/964696.964710

Gajos, K. and Weld, D. S. 2004. SUPPLE: automatically generating user interfaces. In Proceedings of the 9th international Conference on intelligent User interfaces (Funchal, Madeira, Portugal, January 13 - 16, 2004). IUI '04. ACM, New York, NY, 93-100. DOI= http://doi.acm.org/10.1145/964442.964461

     
February 18 Reading Week  

 

 

 

February 25

Tangible user interfaces

Fitzmaurice, G. W., Ishii, H., and Buxton, W. A. 1995. Bricks: laying the foundations for graspable user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, United States, May 07 - 11, 1995). I. R. Katz, R. Mack, L. Marks, M. B. Rosson, and J. Nielsen, Eds. Conference on Human Factors in Computing Systems. ACM Press/Addison-Wesley Publishing Co., New York, NY, 442-449. DOI= http://doi.acm.org/10.1145/223904.223964

Ishii, H. and Ullmer, B. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, United States, March 22 - 27, 1997). S. Pemberton, Ed. CHI '97. ACM, New York, NY, 234-241. DOI= http://doi.acm.org/10.1145/258549.258715

February 27

Haptic feedback

MacLean, K. E., Snibbe, S. S., and Levin, G. 2000. Tagged handles: merging discrete and continuous manual control. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands, April 01 - 06, 2000). CHI '00. ACM, New York, NY, 225-232. DOI= http://doi.acm.org/10.1145/332040.332435

Snibbe, S. S., MacLean, K. E., Shaw, R., Roderick, J., Verplank, W. L., and Scheeff, M. 2001. Haptic techniques for media control. In Proceedings of the 14th Annual ACM Symposium on User interface Software and Technology (Orlando, Florida, November 11 - 14, 2001). UIST '01. ACM, New York, NY, 199-208. DOI= http://doi.acm.org/10.1145/502348.502387

February 29   Low fidelity project design due

 

 

 

March 3

Lenses et al

Bier, E. A., Stone, M. C., Fishkin, K., Buxton, W., and Baudel, T. 1994. A taxonomy of see-through tools. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating interdependence (Boston, Massachusetts, United States, April 24 - 28, 1994). B. Adelson, S. Dumais, and J. Olson, Eds. CHI '94. ACM, New York, NY, 358-364. DOI= http://doi.acm.org/10.1145/191666.191786

Bier, E. A., Stone, M. C., Pier, K., Buxton, W., and DeRose, T. D. 1993. Toolglass and magic lenses: the see-through interface. In Proceedings of the 20th Annual Conference on Computer Graphics and interactive Techniques SIGGRAPH '93. ACM, New York, NY, 73-80. DOI= http://doi.acm.org/10.1145/166117.166126

Terry, M. and Mynatt, E. D. 2002. Side views: persistent, on-demand previews for open-ended tasks. In Proceedings of the 15th Annual ACM Symposium on User interface Software and Technology (Paris, France, October 27 - 30, 2002). UIST '02. ACM, New York, NY, 71-80. DOI= http://doi.acm.org/10.1145/571985.571996

March 5

 

Topics from here to the end of class will be related to projects

 

 

 

March 10

 

Initial functioning technique due

March 12

 

 

 

 

March 17

Evaluation results due

March 19

 

 

 

 

 

March 24

 

March 26

 

 

 

 

March 31

 

Final, packaged technique due

April 2

 

 

 

 

 

April 7

 

Final paper due