Usability Studies in an Agile Environment

Despite all of the challenges, 2020 was a year of innovation. We were forced to switch, adapt and reinvent entire infrastructures overnight. Zoom became a household name as more and more of us began to use it to replace our daily face-to-face interactions. Other software and applications were adapted to help alleviate the pains of being remote. 

From this sudden shift came a need to bridge the gaps and pain points that most remote collaboration tools had. And in came the inspiration for Nipper.

Nipper is an application originally designed to foster collaboration and community for remote office environments. It allowed participants to stay in touch and collaborate rapidly with ease. As the year progressed, we recognized that Nipper could transcend beyond office environments to other settings, including education.

As feature priorities shifted towards remote learning, our two User Experience (UX) Designers recognized the need to validate the design through usability testing, especially for the new target audience. That motivation grew stronger as the team was ramping up to submit Nipper to the Cisco Innovation Challenge 2020.

Challenges

Learning and performing in user research

Kicking off the user research process wasn't without its challenges. Learning teams like Nipper don't often have the luxury of conducting proper usability testing. Once we got the green light, we also needed support and guidance from another UX Research Mentor within our practice. Their coaching was essential to the entire project, as neither of us had hands-on, practical experience with formal moderated usability studies. Our aim was to validate the intuitiveness of Nipper.

Recruiting with limits 

Recruitment was another challenge due to the limited resources for learning teams, but we still managed to recruit enough participants. We started to conduct formal usability studies with two user groups: students and teachers. Typically, user test participants are compensated through monetary incentives, but the unfortunate reality is that learning teams don't have funds to pay participants. We took it upon ourselves to source participants within the company. We would send out emails, Slack messages and even reach out to our friends and family to find recruits. This approach allowed us to proceed with the usability sessions in time.

Process

Testing with a prototype vs. testing with an application

Figma prototype 

Usability studies can be conducted with either click-through prototypes or fully developed applications. In the case of Nipper, we conducted our first round of usability studies with a prototype, a linked series of static, high-fidelity mockups. Although the prototype looks like an application, the lack of hotspots and overall functionality may cause some confusion for test participants, especially if they explore the interface and fail to receive immediate feedback. 

Despite the shortcomings, prototypes help validate designs and move a project forward fast. Feedback from participants can be immediately applied before developers write a line of code for that feature.

Live application 

Meanwhile, conducting a usability study with a live application enables users to have a more realistic experience. They can freely interact as they would expect. The biggest drawback of testing with an application is the feedback cycle. The time it takes to apply changes into the working code can be delayed due to time spent designing, iterating, testing and prioritizing the changes within the existing order of work.

Adopting lean UX

After we conducted several usability sessions and gained valuable insights from students and teachers, we thought Nipper would benefit from having a regular cadence of usability testing. Our UX Research Mentor and Coach introduced the Lean UX plan as the technique for us to manage our user testing. This allowed us to conduct usability sessions weekly by keeping to a schedule. We performed scheduled tasks on each day of the week, including recruitment, objective ideation, prototype creation, test day and design iteration. 

Team engagement

In addition to the UX Designers being the moderator and notetaker, we also encouraged all delivery team members to sit in as observers. The team was able to understand the rationale behind design iterations without having to read a long formal usability report. Even if team members could not attend, the UX Designers would also create a list of top insights for the team to review. 

Research highlights

The evolution of the summon team feature

There are some features that clearly benefit from usability studies. Testing the feature "turnaround," which later on became "Summon Team", was a long and fruitful journey for Nipper. The feasibility of getting a collective group's attention and meeting ad hoc is more challenging in remote settings than in a physical space. We tested several things such as the icon and the button location, but it turned out that the choice of terminology was critical to its usability.

The term "turnaround" is commonly known in an agile environment for getting a development team's attention quickly. Upon testing, we found that it was not understood by users from different fields. In our first test, we immediately received negative results from our very first teacher. The participant could not relate the icon with the action of getting all students in the same room. Even though there was a tooltip that displayed "turnaround" while hovering on the button, the participant still thought the term meant for turning the camera around.

For the next round, we changed the term to "summon class" and saw significant improvement. Teachers were able to mostly guess what its function was. When they performed the task of bringing everyone into the main classroom, they realized what the term "summon class" truly meant and managed to complete the task. 

We continued to see successful results with students by using the same term; therefore, we decided to carry the term "summon" to the office setting and updated the term to be "summon team." While our WWT audience was more familiar with "turnaround," they could still complete the task.

- Jakob Nielsen's #2 Usability Heuristic: Match between system and the real world

Testing with minors

Testing Nipper with students was an exceptional learning experience for the UX designers. We had to get parental permission to conduct the session with a minor. By broadcasting a message on Slack, we were able to recruit children from WWT employees and granted parental permission at the same time. There were additional considerations we had to keep in mind when moderating with youths, such as maintaining their level of ease by providing a friendly testing environment.

Our participants ranged from 6th to 12th grade and were all learning virtually during the pandemic. Through our questioning, we found that the major pain point was the learning curves for the tools they used. During the sessions, we also discovered that they were very familiar with the digital interface and were able to complete most of the tasks. They thought Nipper was very easy to navigate. In situations where they did encounter friction, we were impressed by their ability to suggest UX improvements.

Testing for telehealth

In testing broader audiences, Nipper received a great opportunity to test out its healthcare version with a doctor. The experience and results were both very positive. The participant was asked to share pain points from existing telehealth tools and opinions about how Nipper can provide a better experience for the medical workspace.

We tested a scenario from a nurse's perspective. This session contains tasks such as viewing the schedule of the day, getting a notification when the patient arrived, talking to the patient via microphone and camera and inviting the doctor to the room to meet with the patient. In the end, the participant expressed that this was a "better user experience," "elegantly simple" and "really recreates that in-person experience which is totally lacking with every other tool out there." 

Outcome

Positive feedback from end users

As the UX designers on Nipper, our goal was simple: provide an online collaboration tool that is intuitive and elegant. We received lots of positive feedback from our test participants during the usability sessions, such as: "I like that it's extremely intuitive. I can figure it out easily, and it's very streamlined." This provided confirmation that we were moving in the right direction.

We also had great feedback from participants about the overall usability testing experience. Ben Pomerenke, the Technical Operations Manager at WWT, had been our stakeholder and participated as a test user in one of the sessions. He shared his thoughts with us: "It got me excited about the product itself not only because it's cool, but because I felt some ownership just by providing user testing feedback." Matt Black, the Product Owner at WWT, also participated in one session and concluded, "The usability sessions were well thought out and run. Great product!"

These keywords were given most frequently when we asked the question: "what are 3 words when you think about Nipper?" 

It was remarkable to see our end users get excited about the tool at the end of the usability sessions.

Endorsement from the team

Shelly Becker, who significantly contributed to the Nipper front-end development, expressed that "it was a valuable experience to watch real users interact with the product which led to the ability to suggest implementation changes to components to address observed behavior patterns." Michael Leyva, the Agile Business Analyst on the learning team, indicated that "it gives you a chance to get instant feedback on upcoming features you are developing." Janie Selman, the Quality Advocate on the learning team, also stated, "it made me more aware of different points of view and ways of thinking about the app, and as a QA, this was very helpful in my testing."

What we achieved

In the end, Nipper was awarded a finalist in the Cisco Innovation Challenge 2020. We are humbled by the supportive leadership of the team, who encouraged us to dive into usability research. Through their trust and belief in the value of research, we were able to adapt and grow as product designers and help make Nipper a world-class experience.

- John Yorke, Technical Operations Manager for the Virtual Office at WWT