UXR Case Study 1:
Assessing the impact of OS familiarity & icon features on search:
Project Overview
The Challenge: Learn the impact of icon appearance vs. icon position on user search.
-
Which of these icon features drives search performance?
-
What is the impact of unexpected icon features that have been changed from their default OS state?
Critical comparisons:
Icon Features:
Icon appearance vs. icon position.
Expectation of Icon Features:
Expected (default OS icon) vs. Unexpected (Changed Icon feature)
OS familiarity:
Mac vs. Windows OS users
In the initial discovery phase, our team was pondering challenges students faced when using Mac vs. Windows computers when working in Rutgers University computer labs. Students noticed that their productivity may be impaired when (limited availability of computers) forced (them) to use a computer platform that they typically did not use. For example, a Mac user forced to use a Windows PC may be slower in getting their work done & vice versa.
This insight about potential challenges with cross-platform use emerged from initial brainstorming sessions which included open-interviewing sessions with current & prospective undergraduate students. A sample of what we uncovered is shown in this affinity/category mapping.
Questions that emerged:
-
If a student was familiar with one OS, how easily could they adapt if they were using the alternative platform?
-
Was their productivity actually impaired?
-
If so, what features caused the impairment?
-
Could users be helped if certain OS features could be switched (via an app setting) to those in their preferred platform? We were proposing here a scaled down version of virtual OS -- one restricted just to particular features & whether adjusting specific features would benefit the user.
-
-
This study focused on 2 icon features: icon appearance vs. icon position. Would either of these have a greater impact on performance?
-
Finally, which measures would best reveal challenges students face when unfamiliar with a computer platform or find unexpected changes to OS icons?
To organize questions & categorize themes of challenges students faced we constructed an affinity mapping. This guided the scope of the current study. --------->
Methods
We used a combination of behavioral & attitudinal measures of OS usability. These were embedded in the search & self-report tasks described below.
Tasks
-
Participants performed search & probe-dot detection tasks on computer simulations of Mac vs. Windows OS where a google browser window was open. Participants searched for targets such as "Start,","Trash" or "Exit" Icons.
-
On 50% of trials we randomly changed icon-position or icon-appearance, so that they differed from standard OS design.
-
Survey (self-report) questions following experimental conditions asked about ease of finding information & whether users noticed any changes across conditions. These questions allowed us to explore whether performance measures map on to users’ impression of task difficulty and whether they noticed unexpected changes to the display. We used a variation of the System Usability Scale (SUS) to measure perceived difficulty with the UI in each condition.
-
We also asked participants about their familiarity with Mac vs. Windows OS. This allowed us to examine whether OS familiarity interacted with changes to icons.
-
42 students participated in 2 Blocks of 90 trials
-
Eye-movements were recorded using Eyelink 1000 eye-tracker.
What role do eye-movements, and attention play in search?
In addition to tracking users' eye-movements, we also assessed the inhibitory nature of attention using the probe-dot technique described in another study.
Sample display with eye-movements.
Findings
-
Icon position has a much greater impact than icon identity on finding targeted information:
-
Search is faster, but with longer fixations when icons are in a familiar, expected position (where there was no change in icon location) than when items are in an unfamiliar, unexpected location (that had changed from the default OS location).
-
First click accuracy showed no significant difference across changed vs. no change conditions.
-
OS familiarity contributed to these effects when either a Mac or Windows user encounters an icon in an unexpected position (on either platform), search is slower but they make shorter fixations (as they quickly seek out the target in the uncertain locations).
-
Search Times: When icon position changes (red bars) from default OS position, search was slower, than the no-change (expected) default positions (blue bars). Changes to icon appearance (i.e., icon ID) had no effect on search times.
-
Fixation Duration: Participants spend less time looking at icons that have changed position than those that were unchanged. Changing icon-ID had no significant effect on gaze duration.
Summary of search & usability findings:
-
Surveys, & usability-testing show a significant correlation between perceived and actual ease of finding targeted information, with participants taking longer to find the target, but looking less at the targets when there were unexpected changes to icon position.
-
Participants also reported that they did notice when an icon was in an unexpected position (or that there was something "off" about the display), but they often did not notice changes to the icons’ id.
-
Familiarity with icon position drives search performance-- participants look for the icon where they expect it to be. When target icons are in an unexpected position, participants’ search is slower as they look for target in unexpected display locations.
-
The expected position of a target icon also tends to drive fixations, & users’ attention whereas the expected icon’s appearance does not: Along with the slower search times when icons are in unexpected places, users rapidly sample different locations with brief but more frequent fixations until they find the correct target.
Insights
-
A computer user working on an unfamiliar OS, may benefit if the position of icons (were changed to) match the system that they were familiar.
-
To accommodate cross-platform use, OS designer's should consider adding a modifiable setting so that users can adjust features to match the OS that they are familiar.
-
Having the ability to adjust an icon to an expected position would have the greatest impact on improving user productivity.
Alummoottil, D., Livingston, K., Caputo, P. Homare, K., & [Advisor:] Aks, D.J. (2015 & 2016) Evaluating Effectiveness of O.S. Design with Eye-Tracking and Inhibition Testing. Rutgers University Undergraduate Symposiums
