Characterizing Usability Through Query Log Analysis

Search query log analysis

Adam Fourney, Richard Mann and Michael Terry

People naturally turn to search engines whenever they encounter a problem using software or any interactive system. In this project, we are exploring how search engine query logs can be mined to understand the primary needs and issues users face when using interactive systems.

Example: Getting Firefox's Menu Bar Back

Approximately once every 2.1 hours, a variation of the query "firefox get menu bar back" is issued to (United States locale). Similarly, once every 1.7 hours, a variation of the query, "restore menu bar firefox", is issued [1]. In fact, this pattern of searches is so popular that it appears in Google Suggest's top ten list for completing the phrase ``firefox how to''.

An inspection of the Firefox user interface (version 3.6 on Windows), reveals that the top-level menu bar is easily hidden by deactivating the ``Menu bar'' item in Firefox's ``View --> Toolbars'' sub-menu. However, once this action is taken, it is not easily reversed: the top-level menuing system is now hidden, removing the very means the user would employ to attempt to re-instate the menu bar.

What is noteworthy about this example is that we quickly moved from data derived from query logs to a testable hypothesis regarding the usability of the software.

Exploring Query Logs

Below, we provide links to a pair of interactive visualizations that demonstrate the overall utility of this approach for two open source projects: Ubuntu Linux, and Firefox. The Ubuntu data set contains over 120,000 unique queries, while the Firefox data set contains about 75,000 distinct queries. Click on the images below to start exploring these data sets.
Ubuntu [en-US Locale] [Empty Locale]      Firefox [en-US Locale] [Empty Locale]
GIMP [en-US Locale] [Empty Locale]      Kindle [en-US Locale] [Empty Locale]

Full paper (CHI'2011)

A more complete description of our work is available in the following paper:

Characterizing the Usability of Interactive Applications
Through Query Log Analysis [PDF] [Local Copy]
Adam Fourney, Richard Mann, and Michael Terry.
To appear in Proceedings of CHI 2011.

Poster (UIST'2010)

What Can Internet Search Engines “Suggest” About the
Usage and Usability of Popular Desktop Applications?
[PDF] [Local Copy]
Adam Fourney, Richard Mann, and Michael Terry.
Proceedings of UIST 2010. New York, NY.