Library website usability testing: sample scripts

April 10, 2013

If you haven't done a usability test on your library's online services in the last 12 months, you're overdue! Web trends and user expectations change at a fast rate, as do the many parts of a library's resource ecosystem. Before rolling out the new Lloyd Sealy Library website in December, I conducted a usability test on the beta version of the site in November. The feedback from students and faculty was helpful in refining parts of the site that weren't as usable — in areas that were already of concern to us and in places we didn't know were problematic.

There are a bundle of resources out there for usability testing, but not many example scripts specifically for library websites. Below are the scripts I used for students and faculty. Note that this is tailored to John Jay College's Lloyd Sealy Library website, so some tasks might not be phrased the same way or even performed on your site. At the bottom of this post, I've included some notes about this usability study and my approach.

Student script

[Introduce self and make small talk to make participant feel comfortable]

[Explain what's going to happen:] The point of a usability test is to see if a website does all the things we want it to. I’ll be giving you four tasks to complete on the library website. You might come across some broken links or other problems on this test site. It’s okay if you can’t find the information to complete a task – in fact, it’s helpful, because then we’ll know where the website can be better. It will also be very helpful if you narrate your actions as you do them – for instance, saying Now I’m going to click this link because I think it will take me to a page about finding books, or I’m surprised because this button doesn’t do what I thought it did. As I mentioned in the email, I will be recording the screen and your voice, do I have your permission to do that? Let’s get started.

Tell me a little about yourself... [Open-ended background questions, year, major]

Now, let’s go to the library website. I want you to give me your immediate impressions as you look at this page. Feel free to click anything.

Let’s pretend one of your instructors has given you an assignment: write an overview of cybercrime, using at least one book and one academic article. This overview must define what cybercrime is and how law enforcement agencies are fighting it. Feel free to use other internet sources like Wikipedia at first if you don’t know what cybercrime is.

Show me how you would find a book.

Rate the process of finding a book on a scale of 1 to 5, 1 being very hard and 5 being very easy, and tell me why.

Show me how you would find an article.

Rate the process of finding an article on a scale of 1 to 5, 1 being very hard and 5 being very easy, and tell me why.

Now let’s say you’ve written your essay and need to write your bibliography. Your instructor says it has to be in APA style. Where do you go to make sure your citations are in the right style? It doesn’t have to be the library website – just show me what you would normally do.

(If they use the library guide) Rate the helpfulness of this guide from 1-5, 1 being not helpful and 5 being very helpful. Why?

Okay. Now for some smaller questions.

What time is the library open on Friday?

How do you renew a book?

How many ways can you ask a librarian a question?

I have a few debriefing questions before we wrap up.

What did you like about the new website? What did you not like?

If you could add anything else to the website – a new feature or new information – what would you add?

Do you ever use the library website on a mobile device? What for?

Thank you for your time! This has been very helpful for us. The new website will be up by next semester.

[Remember to give the student a Starbucks card!]

Faculty script

[Introduce self and make small talk to make participant feel comfortable]

[Explain what's going to happen:] The point of a usability test is to see if a website does all the things we want it to. I’ll be giving you four tasks to complete on the library website. You might come across some broken links or other problems on this test site. It’s okay if you can’t find the information to complete a task – in fact, it’s helpful, because then we’ll know where the website can be better. It will also be very helpful if you narrate your actions as you do them – for instance, saying Now I’m going to click this link because I think it will take me to a page about finding books, or I’m surprised because this button doesn’t do what I thought it did. Let’s get started.

Tell me a little about yourself... [Open-ended background questions, department, classes]

What kinds of readings do you assign your students? How do you tell them what to read?

Blackboard link: do you use EzProxy?

Reserve: eReserve or regular reserve?

Now, let’s go to the library website. I want you to give me your immediate impressions as you look at this page.

Let’s say a colleague has given you the following citation. Show me how you would identify whether the article is available through the library.

Gong, L., Sun, X., Jiang, D., & Gong, S. (2011). AutMiner: A System For Extracting ASD-Related Genes Using Text Mining. Journal of Biological Systems, 19(1), 113-125.

Rate the process of searching for this article on a scale of 1 to 5, 1 being difficult and 5 being easy.

Show me how you would find information about:

Scheduling a class session with a librarian

Placing a book on (regular) reserve

Contacting your library liaison

Rate finding this information on a scale of 1 to 5, 1 being difficult and 5 being easy.

Before we wrap up, I’ll ask some debriefing questions.

What do you like about the new website? What do you not like?

If you could add anything – a new feature or more information about something – what would you add?

Do you ever use the library on a mobile device? What for?

Thank you for your time! This has been very helpful for us. The new website will be up by next semester.

[Remember to give the faculty member a Starbucks card!]

Notes on these scripts and this usability study overall:

Length: The tests were on average 20 minutes long, which felt like the right amount of time to get useful input without straining the participant. Note that writing your script, scheduling 5+ sessions, and reviewing your data will take a long time for you. Note also that for every study, you'll have at least one no-show.
Ratings: I frequently forgot to ask the 'rate 1-5' (Likert) question, which would have been useful for compiling users' feelings about the task. Some usability folks say that when you ask for their quantitative ratings, you should assign a rating of how well you think they did, too. My colleague Karen suggested that for our upcoming usability tests, we create little worksheets to use when moderating each session so we don't forget to ask this.
Rewards: Starbucks cards are great motivators, but people also liked seeing a 'sneak peek' of the site. The grad students and faculty also wanted to see what a usability test was like.
Focus: It was hard to figure out what I wanted to test: how students used the library site, or how students did research? I know we're not usually the first place they turn when looking up a subject — heck, library sites aren't the first place I turn most of the time, either. So while some of my questions did lead them off-site, it was useful for contextualizing, but not too useful for site-specific feedback.
Software: I've used Captivate, Camtasia, and Morae before, but at this point I just stick to a Quicktime screen capture with audio recording (built-in Mac mic).
Human subjects: Don't forget to clear everything with your IRB! Don't forget to tell participants beforehand that their screen actions and audio will be recorded.
Reporting: If your position is like mine, you're the one who's collecting and using the data, so there might not be a great incentive to put together a report or present any findings. Still, it's useful for your colleagues to see what you're doing, and usability studies (combined with site stats) may come in handy for responding to concerns about areas of your site.

We're drafting a new usability script for use this semester, which I'll post at a later date.