Controlling presentations through gesture
Adam Fourney, Michael Terry, Richard Mann
Maestro is a prototype gesture-based presentation system developed to explore the implications of gestural interaction with presentations.
While gesture-based interaction has long been seen as a natural means of input for controlling presentations, systems supporting this input modality are rarely evaluated in real-world contexts. To explore the real-world implications of this input modality, we deployed Maestro in a classroom setting for two weeks. During this time, Maestro was used to present 12 lectures to approximately 100 undergraduate students.
Details of Maestro's design, and the results from the deployment study, are summarized in the video below, while a more detailed discussion is provided in the full paper Gesturing in the Wild: Understanding the Effects and Implications of Gesture-Based Interaction for Dynamic Presentations presented at the 24th British Computer Society Conference on Human Computer Interaction (BCS-HCI ‘10).
Maestro's slideshows are structured similarly to those of PowerPoint, and are composed of a sequential deck of slides. Each slide can contain a combination of written text, bullet hierarchies, and embedded figures.
As with other gesture-based presentation systems in the literature, Maestro allows presenters to use hand gestures to navigate the slide deck (e.g., to advance slides). However, Maestro is distinguished by the fact that it also allows presenters to interact directly with the content of their slides (e.g., to zoom into figures, or to expand bullet hierarchies).
Maestro's navigation gestures allow presenters to move between slides, to scroll slides, and to bring up the slide carousel. These gestures are independent of slide content, and are thus performed in the left margin of each slide, a region we call the staging area. A set of horizontal ruled lines delineates the areas for invoking navigation gestures, but these visual guides appear only when the presenter rests their hand within the margin for a short period of time. Gestures can be performed even when the guidelines are not visible.
Maestro also affords gestural interaction with the actual content of the slides. Blocks of text can be highlighted by pointing to them with one hand. Presenters can also selectively enlarge figures embedded alongside text. When enlarged, a figure occupies the entire screen.
Finally, presenters can also author slides with hierarchical lists of bullets, with child bullets initially hidden. This capability allows the presenter to cater the detail of the presentation to the particular needs of the audience.
Maestro relies on a single web camera for input, and a data projector for output. Together, these devices yield a highly portable presentation system that allows presenters to use hand gestures to control their presentations. This portability was essential in allowing Maestro to be evaluated in real-world contexts, such as multi-use classrooms.
Maestro was developed with the expressed purpose of quickly and inexpensively exploring the implications of gesture-based interactions with presentations. Several aspects of the system reflect the fact that Maestro is an early prototype:
- First, to simplify hand tracking, Maestro requires users to wear a mismatched pair of brightly colored gloves (one red glove and one blue glove). Hand detection and tracking can then be achieved using simple color thresholding techniques that are computationally inexpensive.
- Second, Maestro relies on a simple template-based approach for motion recognition; various features of the hand trajectories (e.g., start/end location, path length, general direction of travel, moment of inertia, etc.) are measured, and are tested against manually tuned gesture templates.
After an iterative design process, we evaluated Maestro in a two week field study. Maestro was used by one of the researchers to give 12 lectures to approximately 100 undergraduate students. These lectures were observed by a second researcher, who attended the lectures as an audience member. Students provided feedback through a questionnaire distributed at the end of the study.
The results of the field study confirm that gestures that support interaction with content appear to be the most effective and valued type of gestures in this context. In particular, the ability to zoom into figures and the ability to highlight talking points were well received by audience members. In contrast, navigation gestures were perceived to be less efficient than the use of a wireless remote.
The two week field study enabled us to identify a set of emergent behaviors that result when using gesture-based interaction with presentation systems. Most of these behaviors highlight side effects that point to areas in need of future research for this problem space.
- Grouping Through Highlighting
Maestro implements a gradual fade-out of highlighted points, enabling the presenter to sweep his hand across a range of points, highlighting them all at once. This mass-highlighting of bullet points was not planned for, but became a welcome emergent feature of the system.
- Facing the Screen, Not the Audience
The presenter must be able to see the projection screen to ensure that gestures are performed on their intended targets. As such, the presenter found himself angling his body away from part of the class. This pivoting was not always corrected, leading the presenter to miss questions from students not in his field of view.
- The Anchor Problem
One of the most visible effects of utilizing gestural input was that it tended to "anchor" the presenter next to the screen so he could navigate the presentation (e.g., advance slides).
- The Field-of-view Problem
The tendency for the presenter to anchor himself next to the screen also made it difficult for
the presenter to see all of the content being projected. After advancing to the next slide, the presenter would sometimes need to step back 4-5 feet from the screen to be able to see all of the slide's contents. From the audience's perspective, this behavior caused an obvious break from the presentation flow, and could be interpreted as the presenter being unprepared.
- The No-fly Zone Problem
The design of the gesture recognition system also created a ``no-fly zone'' -- an area of the room that the presenter could not enter without the risk of distracting the audience. Maestro was designed with the assumption that the presenter normally stands to the side of the projected content, only occasionally entering the projected content to selectively interact with elements in the slide. This is a safe
assumption to make, since the presenter typically wishes the audience to be able to view the content without interference. However, after the first day of
lecturing, the presenter found himself forgetting about the system and fully immersing himself in the act of lecturing. At times, he would wander in front of the projected content to address the class, gesticulating as he did so. This would lead to constant activity in the slides behind him.