ONBOARDING EXPERIENCE EVALUATION IN STROBOPHAGIA
This is a short documentation of the research. Unfortunately, the findings concerning the game itself are not publicly available because of its sensitive nature.
Team Size: 4 people
Time Spent: 3 months covering the whole course
Project Introduction
Strobophagia Title Screen
Strobophagia is a rave horror game taking place in an open-world rave party. The gameplay revolves around solving puzzles scattered around the world and uncovering the lurking horror underneath the rave party. This research is a collaboration between our GUX research group project and the indie game studio Green Tile Digital.
Research Background and Objective
The collaboration started around the time Green Tile Digital initiated its beta test for Strobophagia. The studio wanted to find out the general gameplay experience for the beta. Our research group did a survey on the beta test players and the results that revealed issues in various aspects of the game prompted us to conduct a more comprehensive study on the onboarding experience of the game. As a result, we set down the research object to be Evaluating the general onboarding experience of Strobophagia.
Challenges
We were facing multiple challenges when conducting the research:
Green Tile Ditigal was a fresh game start-up and so was our research group. We were not sure how to collaborate efficiently together.
Sweden was still in pandemic mode during the research. All activities were suggested to be online, which poses a serious challenge to offline playtests.
Method Highlights
To address the Covid risks we decided to do online playtesting through Discord. The player would share their playthrough over Discord and the moderator would have the gameplay recorded.
Strobophagia in-game screenshot
Affinity Mapping of the collected issues (text blurred out for confidentiality)
Our method draws inspiration from the player's self-commenting technique from the “Biometric Storyboards” method (See Reference) and the procedure went as follows:
The player first played through the tutorial session of Strobophagia without any active moderation or thinking-aloud method that might disrupt their flow. The moderators only observed and noted down any point of interest that may turn into questions in the following step.
After the playthrough, the player was asked to watch their gameplay footage and encouraged to think aloud and comment on their gameplay at any time. The moderators watched the footage together with the player and asked questions according to the notes collected during the observation.
The issues collected during the previous steps went through an affinity mapping process and were categorized into different types of issues such as usability issues and level design issues. The severity of the issues was also evaluated according to their impacts on the gameplay.
Method Discussion
The hybrid method we invented turns out to be working much better than think-aloud or observation alone. The game flow was not interrupted because the intervention and moderation from the researchers only came in in the later phase when the players watched and commented on their gameplay. At the same time, it takes off a huge workload from the researchers compared to the observation-only method where they have to pay very close attention to all the things that are happening during the gameplay. It also greatly reduces the self-reporting error when players are interviewed and have to come up with an answer based only on their recollection of the gameplay.
It is not without downsides, however. The extra player-commented footage almost doubles the amount of game recordings the researchers need to watch in order to collect feedback. Thus, the method might not scale well without increasing proportionally the manpower from the research team.
Findings and Reflections on the Process
Always try to fit the research schedule into the development team’s schedule.
User Research always plays a supportive role. No impact is made if the research result can not be delivered in time to fit the development schedule. Our search suffers greatly from the detachment from the development team, which to a great extent undermines the significance of our research.
If time allows, always do a pilot study.
More things could go wrong than expected. Screen-sharing may not work. The PC might not be powerful enough to do video recording and streaming together. Doing a pilot study not only tests if the method works out on a high level it also exposes all the nitty-gritty of the test process that the research team need to pay attention to.
My Responsibilities
Design the research protocol and user test guide
Design the observation note form and some of the interview questions
Recruiting testers, setting up recording environments, and moderating the tests
Compiling and analyzing the data together with the team using Affinity Map
Documenting the timeline for the whole research project as weekly reports