Some of the girls from Senran Kagura

In any scientific study there are many potential errors that can impair the validity of the research. For example, not discussing potential biases, not including research findings that contradict the study’s conclusion in the literature review, and providing questions that skew the participant’s response, are all among the most common errors that scientists can commit.

Since the goal of a scientific study is to have the data speak an understandable and replicable story, and not a narrative, committing any of these errors could contribute towards an important research area losing focus as the pioneering researcher’s work is seen as non-credible. By not discussing these issues the risk that people can become misinformed from the claims made by said research is heightened, which can adversely affect the public opinion on important matters.

In March of 2015 two researchers participated in the Game Developers Conference in order to discuss the results of their research about how youth felt about video games and inclusivity. The findings of this study (editor’s note: the authors refer to it always as a survey) noted that a large percent of boys were more than willing to play as women in video games. It also found that women preferred to play as their own gender rather than the opposite. If true, the results of this study have the potential to cause a major shift in the multi-billion dollar video game industry.

Not only do the results encourage developers and producers to have more female protagonists, and to look towards girls as a much larger target audience, it would also encourage developers and producers to move away from a specific audience towards a different, potentially more viable market. While the goal of diversification and inclusion is a noble intent, the fact that this study does not make the use of proper research technique, or ethical consideration, is unacceptable, and only serves to spread potentially harmful disingenuous information into the system.

Mileena, from Mortal Kombat X

Out of the myriad of problems with this study, the most damaging aspect is the lack of peer-review. The process of peer-review is of great importance due to the fact that it helps evaluate the study’s methodology, conclusions, and overall suitability for publication. In essence, the fact that this research was not peer-reviewed meant the results could not be challenged, leading to a major problem generalizing the data to the intended population. By doing so, the researchers run the risk of the audience drawing conclusions not actually backed by evidence.

The purpose of discussing a study is to inform the audience of what the study is about and why it is important. Disclosing who the sample demographic of their study was, where they were taken from, and the racial/economic/cultural demographics of said sample are important in the fact that they help build an image of the participant to the audience and help describe how likely this can be generalized to the general public. While the researchers did provide some of these factors within the discussion, they failed to mention the important point that the survey that this study used data from was gathered online by one of the researchers on their own personal twitter timeline as shown here.

By posting the survey on which the study is based online, the researchers and audience can no longer accurately know the key demographics of the participants, since it allows for anyone to falsely pose as a teenager and inundates the test with individuals who follow that specific researcher. Another consequence of this is that it cannot be known if the data accurately represents the points that the researchers presented throughout the entirety of their discussion. If the data doesn’t accurately portray the information provided, game developers and publishers might assume that the data is correct and alter the way they develop video games.

The ladies of Onechanbara Z2: Chaos

Another glaring error in the study was the fact that it utilized emotionally priming questions that potentially led participants towards results desired by the researchers. For example, the wording of the question “Are females too often treated as sex objects?” primes the participant emotionally to see the depiction of female characters in video games as sexist by suggesting that female characters are already portrayed as sex objects in the media. This type of slanted questioning results in answers that are tainted by bias and cannot be seen as accurate in their representation of the views of the sampled demographic.

Without the raw data being presented it would be impossible to determine if the results of this study were merely chance, or if they actually hold weight. We can see through the use of the pie charts that many children made specifc choices, however the percentages of people making choices has nothing to do with the fact that something might be statistically significant enough to disprove random chance. If we add in the fact that twitter users could have poisoned the well by adding their data into the mix the reliability of this study gets thrown out the window. Lacking this key information and an explanation of where the data comes from, there is no way to tell if twitter had a significant effect on the results of the study or not.

In statistics, having a statistically significant result is different than a large number of similar results. Statistical significance is a method of determining if the results obtained were merely a random chance, or if they hold actual weight. Not only has this study failed to prove statistical significance, it has not been peer reviewed, and could have been contaminated with twitter users pretending to be teenagers. Simply put, this study has presented potentially inaccurate data as fact without any higher fact checking or statistical analysis.

Honoka, a new character in Dead or Alive 5: Last Round

When contacted for a comment on the use of their survey for so many declarative (and baseless) articles, we were pointed to a response from Ashly Burch, Rosalind Wiseman, and Charlie Kuhn, which you can read here. There are some pertinent bits here:

We stated at the Game Developers Conference (GDC) and in the TIME article that this survey was exploratory or a “convenience” sample that was meant to generate conversations and encourage others in the field to continue this research in more thorough ways. We have never claimed that this is a rigorous academic survey, nor that it should be treated as such. As our colleague, Justin Patchin, co-director of the University of Wisconsin-Eau Claire’s Cyberbullying Research Center stated in the Polygon article about our talk:

“From my understanding, she found some pretty interesting findings,” Patchin said. “So now, of course, the next step is to replicate that and do another test in another school. Maybe 100 researchers can take questions and administer them in other populations and see if they hold up. That’s the scientific process.”

We were and are not pushing an agenda beyond giving young people a voice–to have their experiences and their opinions valued and represented.

We did this survey because we were curious about how boys and girls perceived female characters in games and we wanted to know what kinds of games girls were playing. Furthermore, for the last three years Wiseman and Kuhn have focused their work to teach educators to see how it can be an important tool to reach out to students.

The most important bit of information to take away from this, however, is the fact that their study was conducted solely over Facebook and Twitter:

Wiseman reached out through Twitter and Facebook because that’s where her colleagues in the educational field most easily interact with her. Burch, Kuhn and Wiseman reached out on Twitter to inform and encourage participation. It was a risk we took and we understood that people could look at this and dismiss the results. However, the majority of our responses came from schools and were verified by teachers and principals who told us when they were administering the survey to their students.

Despite such obvious flaws, this study has been cited by over 25 online websites, including Times, Wired, and The Guardian, along with video game websites such as Polygon, and Destructoid. It is a travesty that such dubious information has been allowed to spread without being corrected.

Edit: A list of articles that used this study is collected here for all to see.