Project

This team project investigated learning methods to improve an educational section of the client's website. This project is under a non-disclosure agreement (NDA).

• Client: Major financial institution

• Project Type: Team Coursework

• Team: Caroline Davidson, Patrick Day, Mike DiMaria, Mike Fritz, Lydia Sankey

• Timeline: February - May 2012

• Methods & Deliverables: Unmoderated Usability Test, Moderated Test, Final Report, Presentation

• Tools: System Usability Scale (SUS), Net Promoter, Loop11, WebEx

Process

Test Plan. We worked with the client to define the testing goals and documented our approach. We decided to perform an unmoderated usability test, which was followed by a moderated usability test. We used this approach to refine our moderated test and provide additional backing for our recommendations.

Unmoderated Test. Using Loop11, our test asked participants to perform a task, which included post-task questions such as Net Promoter and SUS questions. 

Pilot Test. We performed a pilot test in preparation for the moderated test at Bentley University's usability lab. Performing this test allowed us to practice our moderation skills, refine our questions, and work out technical issues. Since some team members, such as myself, were located outside of Massachusetts, we used Morae and WebEx so these team members could moderate the test remotely.

Moderated Test. The client's recruiter used our screener to select 6 participants for our test, which was an hour long for each participant. We asked participants to perform a task similar to the unmoderated test. Asking SUS questions after each evaluated method would prolong the test, so we asked participants to rate a few statements instead. Finally, we ended the test with open-ended questions to gain further insight.

Result

Report. We analyzed the data from the moderated test and compared them with the unmoderated test. We summarized our quantitative data in charts and selected key quotes from our qualitative data to include in our report. Then, we presented our findings and recommendations to the client.

Lessons Learned. 

• Perform a robust pilot test for unmoderated tests to detect technical issues.

• Use a notes grid for taking notes during a test.

• Be prepared to ask questions that are not on listed in the test script in order to refocus the conversation or probe deeper with a participant.

Photo: Frontier Classroom by cleopold73, used by CC BY 2.0 / Converted to grayscale from original

Related Projects