top of page

UXR Case Study 2: 

Improving user-search on University website

​

Overview of objectives:
 

  1. Ease finding sought out information on Rutgers University websites by assessing the impact of site-features & distractors.
     

  2. Evaluate the effect of drop-down menu orientation on search: Is there a a search advantage to vertical or horizontal drop-down menus?
     

  3. Compare (behavioral & attitudinal) engagement metrics:
     

  • Does user looking behavior align with search speed or accuracy?
     

  • Do behavioral metrics align with self-reports of usability, recall or user confidence?

 

​

Critical comparisons:

Metric alignment:
Behavioral vs. Self-report & Recall

​

Drop-Down Menu Orientation:
Vertical vs. Horizontal

In our initial discovery phase, our team was pondering challenges students faced with searching Rutgers web pages. Difficulties occurred when looking for courses, research opportunities, financial assistance & other sought out information. A prominent pain-point described by students included confusion with multi-level navigation menus- Navigation was difficult & information was not easily found in expected menus. We hypothesized that the problems may be linked to menu-design features such as how drop-down menus open, or whether other display items interfered with search.

 

Study Focus | Questions: 

  • What display features impact users ability to find information?

    • Drop-down menu-orientation?​

    • Distracting information in the display?

​

Methods

Tasks

  • Participants performed search & self-report tasks 

  • A/B Testing:

    • Critical conditions: vertical vs. horizontal drop-down menus (within-subject design)

    • Distractor conditions: image of either a male or female student (between-subject design)

  • Survey questions following experimental conditions tested whether performance measures map on to users’ impression of task difficulty, how confident users' were in their task performance, & whether they noticed irrelevant information in the display.​

  • 40 students participated in 3 Blocks of 32 trials

  • Eye-movements were recorded using Eyelink 1000 eye-tracker.

Search task: actual findability

​

  • 40 students majoring in Psychology or Sociology search on Rutgers University websites for 3 items:
     

  1. Academic department,

  2. Course list, 

  3. Research opportunities.

​​

​

Self-report - perceived findability, usability, recall & confidence:
 

After finding the target, participants were asked:
 

  1. How easy was the search task?
     

  2. Do you recall the gender of student shown in the web sites? (Note: student photo was irrelevant to search task)
     

  3. How confident are you with your search & recall performance?

​Metrics

  • Search time & Accuracy to 1st click test & reaching final test page

  • Eye-movements, Fixation frequency & duration, Pupil size

  • Self-report questions:

    • Perceived usability

    • Confidence in performance

    • Recall of irrelevant information (i.e., gender of male & female students in web page)

Findings​

  • Search was 14% faster searching vertical than horizontal drop-down menus
    (Mean diff = 0.5s in 3.6 sec trial).

  • Accuracy was similar for horizontal & vertical menu conditions

  • More eye-movements & fixations were used when searching horizontal menus.

  • Perceived usability & user confidence did not predict search performance.

Sample display with horizontal drop-down menu  showing eye-movements as participant searches for link that will lead them to research opportunities.

Impact of distractor images:

  • Recall of (irrelevant) student images did not influence search speed or accuracy

  • User-gender predicted recall of student images:

    • Recall was better when user's gender was opposite to image gender.

    • Pupil size (a known measure of interest) also was affected by gender with pupil size tending to be larger when viewing opposite gender.

​

Insights & questions for future studies
 

  • Search speed benefit occurred for vertical-drop-down menus. This may be due to a confounding factor other than orientation such as greater crowding that occurred in the horizontal-menu condition, or a difference in eye-movement patterns that occur for vertical vs. horizontal menus. Thus, following questions remain to be answered in future research:
     

  • Is search slower for horizontal menus due to crowding from pop-up horizontal menu’s close proximity to main navigation menu?

  • How do the number of eye-movements relate to search difficulty?

  • Why do recalled distractor images have no effect on search?

​

Key Insights for UI designers: 
 

  1. Vertical drop-down menus can produce faster search than horizontal menus. But this may depend on contextual factors such as crowding & similarity of adjacent menus.

    • ​Crowding & a similar background color in the adjacent menu may have made navigating the horizontal menu more difficult.

    • Likewise, the greater # of eye-movements in the horizontal menu condition may have been due to users trying to disambiguate menu items in close proximity to each other.

    • Thus, our recommendation to UI designers:

      • Avoid similar surround:
        If a drop-down menus needs to be close-to the parent menu, use distinct background colors, or shades so the user does not confuse the menus.

      • Avoid crowding:
        Design navigation menus with noticeable space between as clustering of adjacent menus likely hinders search.​         ​

  2. Distractors distinct from target items, will tend not to interfere with search performance even though some salient properties may be recalled. ​​​

    • The absence of distractor interference is often due to inhibition which helps users find targeted information (as shown in our other study).

    • Changes in pupil size that were triggered by opposite gender distractors is another signature of users taking in at least some distractor information.

    • It is noteworthy that actively inhibiting irrelevant information may play a role in our ability to recall some salient distractor features. Thus, even though users may believe they can effectively ignore irrelevant information (like ads), some features may still be perceived (outside of a user's awareness) & have impact on a user behavior.
       

  3. Be cautious about relying on self-reports, as perceived usability & user confidence do not necessarily predict search performance. 

    • It is best to use a combination of behavioral & attitudinal measures to get an accurate understanding of user perception & performance as well as user experience.​

 

 

 

Faglio, S. & [Advisor:] Aks, D.J. (2010 & 2014). Studying the effectiveness of web design with eye-tracking: An analysis of Rutgers University’s Web pages. Rutgers University Undergraduate Symposiums

​

​

​

​

​

​

​

​

​

bottom of page