Executive Summary

The usability study aimed to evaluate the user experience of the Goodreads website, a web 2.0 service devoted to reviewing, discussing, and recommending books. This study focused on commonly performed actions including account creation, search, managing custom reading lists, and community engagement. Conducted over one week, the study involved eight participants from diverse demographics, including five male and three female participants ranging in age from 26 to 69. A variety of education levels were represented among the participants.

Using a combination of task-based observations, concurrent think-alouds, retrospective follow-up questions, and a SUS survey, we conducted in-depth usability tests to assess user interactions with Goodreads. Tasks were designed to replicate real-world scenarios, allowing for comprehensive insights into user behavior.

Tasks

• Task 1: Sign up for an account on Goodreads.

• Task 2: Find a book to read.

• Task 3: Explore Goodreads for suggestions in your preferred genre, saving interesting finds to your account.

• Task 4: Create a personalized bookshelf and add books to your Playlist.

• Task 5: Join at least one Community Group of a book genre that interests you, and engage in a discussion with other users.

• Task 6: Find at least one of your favorite authors via Goodreads and read updates that are available.

Results

The usability study assessed user performance based on variables including time on task, mistakes made, and the number of pages accessed. Overall, participants demonstrated proficiency in navigating the website, successfully completing all assigned tasks. All but two users completed all tasks without exceeding 5 pages. 

Challenges were encountered during task 4, which involves the custom bookshelf feature, and task 5, which involved the community group feature. These two tasks were the most difficult for participants, evidenced by longer time on task and a lower average number of pages accessed.

Design recommendations

1.    Add the ability to create, edit, and add to bookshelves from multiple locations.

• Introduce functionality to create, edit, and add to bookshelves from various sections of the platform.

• Include a dedicated "Add Book" button within the "My Books" section for streamlined bookshelf management.

2.    Increase the findability of authors.

• Enhance user search experience by updating the search bar label from "Search Books" to "Search Titles or Authors."

3.    Use more consistent labeling systems.

• Ensure consistency by updating the terminology from "Tags" to "Shelves" across the platform for clearer navigation and understanding.

4.    Design and release a more modern and appealing user interface (UI).

• Revamp the visual design with a focus on minimalism and accessibility to enhance overall appeal and usability.

• Refine the information architecture to improve content organization and user flow.

5.    Add tutorials as part of the onboarding experience.

• Enhance user onboarding experience by integrating tutorials, guiding users through key features and functionalities.

• Implement tooltips for less intuitive features, such as bookshelf management, to provide immediate assistance and improve user understanding

Technology description, goals, and questions

What is Goodreads?

Goodreads is considered one of the world’s largest platforms for book recommendations and reviews. Founded in 2006, it includes search tools to browse books, authors, and referrals from a variety of diverse sources. Additionally, it offers various features, such as:

·      Joining book clubs

·      Engaging in discussions

·      Reviewing books

·      Managing shelves and more

·      Keeping track of one’s reading history

·      Obtaining personalized recommendations

·      Many others

It employs a system learning algorithm, which analyzes copious quantities of data. One limitation of this platform is that it does not allow users to purchase or read books directly from the website itself; however, it does offer customizable links to online stores, such as Amazon, Barnes & Noble, or Bookshop.org. Goodreads is free of charge to sign up for an account and use its services.

Goals of usability research

Our usability study assesses the user experience of Goodreads through six tasks designed to cover common scenarios of use on the platform. Overall, our study aims to identify usability issues on the Goodreads site, such as eliminating frustrations of site users, and gather feedback from our participants to improve the Goodreads user experience for users of all levels.

Research questions

The questions that guided and informed our usability test were:

1.    How usable and user-friendly is the Goodreads platform?

2.    What are some common user flows on Goodreads that users find effective, intuitive, and user-friendly?

3.    What are some common user flows on Goodreads that could offer users a more effective, intuitive, and user-friendly experience?

Methodology

Tasks and task scenarios

Task 1: Sign up for an account on Goodreads.

Task scenario: After having a conversation about books with a co-worker, it was recommended that you join an online book club. Goodreads was mentioned as a great option, as it is free and has a variety of books to explore. 

Task 2: Find a book to read. 

Task scenario: With some trips to the beach on the agenda this summer, you would like to find some books to read. You have read some great books in the past but would like to explore other books by other authors 

Task 3: Explore Goodreads for suggestions in your preferred genre, saving interesting finds to your account. 

Task scenario: As someone trying to get back into the habit of reading, you would like to learn what books are available; therefore, you would like to explore and read the suggestions of other users via Goodreads. Also, it could be helpful reading what authors have to say about certain books.

Task 4: Create a personalized bookshelf and add books to your Playlist.

Task scenario: You are a second-grade teacher and would like to create a custom “playlist” of books for your students on Goodreads. You made a comprehensive list of the favorite book genres of your students, which will assist you in choosing ones that your students would like to read.

Task 5: Join at least one Community Group of a book genre that interests you, and engage in a discussion with other users.

Task scenario: You are a voracious reader of books in a certain genre, but you have no one to talk to about these books! Therefore, you would like to participate in an online book club that discusses works in this specific genre of interest. You discovered that Goodreads provides access to an online book club that could be very helpful to explore.

Task 6: Find at least one of your favorite authors via Goodreads and read updates that are available.

Task scenario: After shopping in various bookstores, you find new books that you would like to read. Also, you would find it interesting learning about the authors who wrote these books. A friend of yours recommended that Goodreads is a great platform for this, and you would like to explore what suggestions authors have made about certain books.

Data collection (questionnaires, etc.)

To assess the effectiveness and usability of the Goodreads platform through these tasks, we gathered both qualitative and quantitative data throughout the testing process.

Table 1. Data collection instruments

Pre-test data collection

Included in our introductory script was an open discussion regarding the constituents of the activity. We encouraged each participant to ask any questions needed so that they were better informed. In addition, we obtained demographic information from participants during this time.

During-test data collection

We also collected user-generated data from in-person think-alouds and observations during the test. Observers used a word processing application or a physical notepad to make notes of users' comments. These notes also documented observed performance metrics, such as time on task, mistakes made, and pages accessed. After each task, each participant would rate the difficulty of it and receive the invitation to share further feedback. Comments could include information on how to improve the Goodreads site, positive and negative feedback about the site, and comments on time/methods to complete the required tasks.

Post-test data collection

Following the full usability test, we asked five follow-up questions and used a system usability scale (SUS) survey. The follow-up questions gave space for open-ended comments, while the SUS survey offered statistics on overall perceptions of the usability of Goodreads.  These means helped us collect further information about participants’ feelings and opinions of the Goodreads website and their experience in navigating it to complete the tasks.

To further review the data collection instruments and receive a justification for their use, please consult appendix 1.

Participant recruitment

Each group member of the testing process recruited one to two participants (comprising of eight to ten total) by random selection. Regardless of each participant’s nationality, age, profession, or other demographics, the recruitment process was expected to be executed in such a way that any willing individual could engage in the activity. A random selection process also provided various demographics to give variety to the overall testing process. Once the recruitment process concluded, each participant was invited to engage in all tasks at their discretion; it was important that each participant felt comfortable in expressing their natural responses. To achieve this, the observer provided a script before beginning the usability test. 

Usability test script

"Prior to asking participants to complete these tasks, we needed to onboard them to the usability test. Below is the script used to prepare participants.

Hello, and thank you for joining me for today's session. The purpose of this observation is to evaluate the user experience of Goodreads by examining a few common scenarios. We have outlined six tasks for you to complete. These tasks have been chosen to cover a diverse range of activities on the platform, using clear and concise language. They represent actions commonly performed by Goodreads users and will provide easily identifiable results. 

In this session, I will ask a few questions about yourself to gather demographic information. I will then ask you to complete six tasks. As you complete these tasks, please do whatever comes naturally to you, and I will observe and take notes while you complete them. To aid in my observation, please try to think out loud while you’re working on the tasks; in other words, let me know whatever comes to mind.  

We would also like to record your screen as you complete these tasks. We will only use these videos in case we missed anything, and we will not share these with anyone else. Would it be okay if we recorded the screen? If you are okay, please review and sign this digital recording permission form with your signature.

After you complete each task or need to move on, please let me know, and I will ask you to rate the difficulty of each task. At the end of our session, you will answer some questions about the website."

Participants

The table below contains participant demographics and the dates of when testing sessions occurred.

Table 2. Participant information

Results

Findings on user performance

Provided below are the results across participants for the various performance, or behavioral, metrics assessed in this usability test: time on task, mistakes made, and the number of pages accessed. Please note that each participant did successfully complete each task in the end, and so we have not included this measure in our discussion.

Time on task

Below is Table 3, which identifies the time used by each participant to complete each task. Included is also the average, or mean, for each participant’s time across all six tasks. Participant 2 took considerably more time than other participants to complete several tasks, often since she took considerable time to explore and research options while navigating the Goodreads platform. Though it is also important to not generalize or stereotype based on age, this participant was also sixty-nine-years-old, something that also may have contributed to her pace in using the Goodreads platform. The most efficient participants were participants 6 and 8, who each managed to finish each task in under a minute on average.

Table 3. Time on task across participants

Mistakes made

Below is Table 4, which shares the number of mistakes each participant made on each task. For the most part, participants succeeded in navigating the Goodreads platform and completing the assigned tasks. However, all but two participants encountered challenges and made errors with task 4, in which they attempted to create a custom bookshelf and add a book to it. It is also worth noting that participant 8, who managed to finish all the tasks quickly, also experienced no mistakes in navigating and using the Goodreads platform. This participant is perhaps an exception to the general rule regarding new users of the Goodreads platform.

Table 4. Mistakes made across participants

Number of pages accessed

On average, all but two users could complete the tasks without needing to access more than five pages. The two exceptions to this were participants 2 and 3. These participants may have taken more time to analyze and peruse the options available to them in terms of communities. As noted earlier, participant 2 took considerable time to explore and research the options available on Goodreads. However, task 4, in which participants were asked to create a custom bookshelf, may have also presented challenges and caused confusion, resulting in further pages being accessed.

Table 5. Number of pages accessed across participants

Measures of central tendency in performance metrics

We have discussed and shared individual results previously. However, it is important to acknowledge and explore trends across all participants by using measures of central tendency. Below is Table 6, which provides the median performance metrics for each task across all eight participants. These performance metrics include the previously discussed measures: time on task, mistakes made, and pages accessed. Our team has decided to use medians rather than means, or averages, given the small sample size and the possibility of outliers, which were discussed previously, skewing the data.

Table 6. Median performance metrics across tasks

The results indicate that tasks 4 and 5 demanded the most time across participants, while task 1 required the least. In tasks 4 and 5, participants needed to: (1) create a personalized bookshelf and add a book to it; and (2) find and join a community in a genre of interest and participate in it. These two tasks also called for users to access more pages than in the other tasks.

While task 5 may have featured the most time on task and the most pages accessed, the more complex and personalized nature of the task may have caused these results. Task 4, on the other hand, could present more concerns and issues to address and remedy: It was the only one in which users tended to make mistakes, with a median of one across the sample of participants. 

Findings on user attitudes

Task difficulty

Quantitative data gathered during the observation process further indicates that while Goodreads is generally intuitive and clear to use, the process of creating a custom bookshelf, as was asked for under task 4, may present challenges and prove unintuitive and cumbersome for users. As noted below in Table 7, participants overall rated many of the other tasks as “easy” or at least “okay”: All but two of the tasks received a median rating of “easy.” Please note that the number 1 stands for an “easy” rating, the number 2 stands for an “okay” rating, and the number 3 stands for a “difficult” rating.

Table 7. Task difficulty rating across participants

Several participants selected “difficult” for task 4, causing it to have a median rating of “okay” overall. In addition, multiple users found the process of finding authors, task 6, not especially intuitive. Participants considered task 6 to be between “easy” and “okay” overall.

Table 8. Difficult tasks for users

Participant comments

After conducting the usability test, we asked each participant five standardized follow-up questions. These questions offered participants the opportunity to share open-ended responses about their experience with and thoughts on the Goodreads platform. To view the full comments from users, please check appendix II. After gathering these comments from users, we inputted them into ChatGPT Plus from OpenAI for analysis. ChatGPT has identified the following themes and insights in the comments from users.

Table 9. Common feedback from users

System usability scale (SUS)

In addition to asking users for open-ended comments after the usability tests, we used a system usability scale, or SUS, to gather feedback from users about their overall opinions on the user-friendliness and usability of the Goodreads platform. Three participants gave Goodreads a SUS score of over 70, or an “acceptable,” while the five others gave the platform a SUS score under 70, or a “marginal” (Bangor et al., 2008; Chinn, 2022). Table 10 shows the SUS score from each participant and the median and mean SUS scores. Please note that we were unable to obtain a SUS score from one participant.

Table 10. SUS scores from participants

A SUS score of 68 has served as a baseline and average in the field of usability research (Bangor, 2008; Chinn, 2022), and so a median score of 67.5 and a mean score of 66.43 in this usability test indicate that Goodreads does have considerable room for improvement.

Specific areas of concern from the system usability scale (SUS)

Something noteworthy with the feedback in the SUS scores is that participants gave generally positive feedback across the ten-item survey. Odd-numbered items, which focus on strengths of the platform, received “agree” ratings (four or higher), while even-numbered items, which focus on flaws of the platform, received “disagree” ratings (two or lower). However, as table 11 indicates, there are two exceptions: question number two and six. The median response for both is three, or “neither agree nor disagree.”

Table 11. Median rating across each SUS item

Question three asks if participants found the website to be unnecessarily complex. The mixed feedback here indicates that at least some users consider the website to be unduly complicated and cumbersome to navigate, and they believe it could use a clearer and simpler interface.

Question six asks if participants noted there were inconsistencies throughout the website, and the responses were ambivalent. This suggests that users generally encounter some inconsistencies in the available features of or options on the website as they navigate it.

Design recommendations

Strengths of the Goodreads platform

The participants in this study recognized the potential of the Goodreads platform and found many of the workflows on it intuitive and user-friendly. It seems users find the following processes user-friendly, straightforward, and intuitive:

·      Registering for an account

·      Finding books

·      Exploring genres and recommendations

·      Joining and participating in communities

Users also appreciate the value of a web 2.0 service dedicated to books and the ability to use Goodreads for free.

Areas for improvement with the Goodreads platform

However, this study also indicated there are some areas for improvement for end-users. Our findings indicate that they have encountered the following concerns sometimes as they navigate and use Goodreads.

·      The limited pathways to create personal bookshelves and add books to them

·      The multiple steps involved in accessing the profile pages for individual authors

·      The overlapping use of “tags” and “shelves” to interact with and save books

·      The dated and tired user interface

·      The need for more support and guidance during onboarding

Given the findings of this usability study, we recommend that Goodreads take some short-term and long-term steps to improve usability and the end-user experience.

Short-term recommendations

Add the ability to create, edit, and add to bookshelves from multiple locations. Right now, users cannot create a custom bookshelf except on the

1.    “My Books” section of the website. However, users will value being able to do so from the homepage, book pages, and more. In addition, users may appreciate a button to “add book” on the “My Books” section rather than having to search for a book and add it from the book’s page. Giving users multiple pathways like this will provide a simpler and more convenient navigation experience.

2.    Increase the findability of authors. Right now, the main search bar at the top of Goodreads only presents books when one uses it. However, to help users more conveniently locate and follow authors, Goodreads should also display authors among the suggestions that appear as users type text there. Doing so will help users more conveniently and readily locate and follow authors of interest to them.

3.    Use more consistent labeling systems. The potentially overlapping uses of “tags” and “shelves” on the platform may disorient and confuse users. To offer a simpler and more harmonious experience, Goodreads should adopt and implement a simpler, clearer, and more consistent labeling system.

Long-term recommendations

1.    Design and release a more modern and appealing user interface (UI). Goodreads looks dated to some users. Those of us who have used the platform for years know it has not received an overhaul for many years at this point. However, users may benefit from and appreciate a redesign of the Goodreads platform, which could offer a more minimalist, intuitive, and user-friendly user experience (UX) through offering improvements in information architecture (IA), visual design, accessibility, and more.

2.    Add tutorials as part of the onboarding experience. Right now, users do not receive any guidance or structure as they begin using Goodreads. To help users better understand and navigate the platform and all its features, Goodreads could perhaps develop tutorials that support and onboard users as they register. These would complement existing support documentation.

By taking these steps, Goodreads could become even more user-friendly for users.

Project reflection from team

Power of various usability testing strategies

The observations and think-aloud strategies were implemented in the usability testing process, which provided our team with effective experience in combining these constituents to complement the overall success of this project. During the testing process, notetaking served to be a particularly important aspect of our efforts; it assisted us in noticing details that could complement the follow-up questions in obtaining greater understanding regarding participant’s reactions/behaviors.

Allocating resources

Formulating the plan for the usability test and this overall project itself was also a very constructive experience for all team members because we learned how to organize and allocate resources appropriately. Through this experience, we came to better understand and value each other’s strengths and interests, harnessing them to support our work on this project.

Reflection

Completing this project has been a valuable learning experience for me. One key lesson was understanding the importance of task design. For instance, during our initial task of signing up for an account, I failed to consider that participants might already have existing accounts. This oversight led one participant to create new accounts solely for the study's purpose. Additionally, I refined my observation techniques throughout the project. In contrast to the pilot test observations, I made a conscious effort to refrain from providing hints during task discussions. Instead, I focused on asking insightful follow-up questions and paid closer attention to participants' facial expressions and subtle cues. Lastly, I've realized the significance of thorough preparation. Opting for Microsoft Teams instead of my usual platform, Zoom, resulted in connectivity issues and frustration for participants. This underscored the importance of testing technology beforehand or ensuring familiarity with chosen tools to prevent potential disruptions. These experiences have not only enhanced my approach to usability testing but have also highlighted the importance of meticulous planning and adaptability in achieving meaningful results.