Baseless Study Proves Nothing of Young Men and Video Games, Despite Rallying Cries of Truth

senran kagura estival versus 07-13-15-1

Some of the girls from Senran Kagura

In any scientific study there are many potential errors that can impair the validity of the research. For example, not discussing potential biases, not including research findings that contradict the study’s conclusion in the literature review, and providing questions that skew the participant’s response, are all among the most common errors that scientists can commit.

Since the goal of a scientific study is to have the data speak an understandable and replicable story, and not a narrative, committing any of these errors could contribute towards an important research area losing focus as the pioneering researcher’s work is seen as non-credible. By not discussing these issues the risk that people can become misinformed from the claims made by said research is heightened, which can adversely affect the public opinion on important matters.

In March of 2015 two researchers participated in the Game Developers Conference in order to discuss the results of their research about how youth felt about video games and inclusivity. The findings of this study (editor’s note: the authors refer to it always as a survey) noted that a large percent of boys were more than willing to play as women in video games. It also found that women preferred to play as their own gender rather than the opposite. If true, the results of this study have the potential to cause a major shift in the multi-billion dollar video game industry.

Not only do the results encourage developers and producers to have more female protagonists, and to look towards girls as a much larger target audience, it would also encourage developers and producers to move away from a specific audience towards a different, potentially more viable market. While the goal of diversification and inclusion is a noble intent, the fact that this study does not make the use of proper research technique, or ethical consideration, is unacceptable, and only serves to spread potentially harmful disingenuous information into the system.

Mileena, from Mortal Kombat X

Out of the myriad of problems with this study, the most damaging aspect is the lack of peer-review. The process of peer-review is of great importance due to the fact that it helps evaluate the study’s methodology, conclusions, and overall suitability for publication. In essence, the fact that this research was not peer-reviewed meant the results could not be challenged, leading to a major problem generalizing the data to the intended population. By doing so, the researchers run the risk of the audience drawing conclusions not actually backed by evidence.

The purpose of discussing a study is to inform the audience of what the study is about and why it is important. Disclosing who the sample demographic of their study was, where they were taken from, and the racial/economic/cultural demographics of said sample are important in the fact that they help build an image of the participant to the audience and help describe how likely this can be generalized to the general public. While the researchers did provide some of these factors within the discussion, they failed to mention the important point that the survey that this study used data from was gathered online by one of the researchers on their own personal twitter timeline as shown here.

By posting the survey on which the study is based online, the researchers and audience can no longer accurately know the key demographics of the participants, since it allows for anyone to falsely pose as a teenager and inundates the test with individuals who follow that specific researcher. Another consequence of this is that it cannot be known if the data accurately represents the points that the researchers presented throughout the entirety of their discussion. If the data doesn’t accurately portray the information provided, game developers and publishers might assume that the data is correct and alter the way they develop video games.

onechanbara z2 chaos 07-13-15-1

The ladies of Onechanbara Z2: Chaos

Another glaring error in the study was the fact that it utilized emotionally priming questions that potentially led participants towards results desired by the researchers. For example, the wording of the question “Are females too often treated as sex objects?” primes the participant emotionally to see the depiction of female characters in video games as sexist by suggesting that female characters are already portrayed as sex objects in the media. This type of slanted questioning results in answers that are tainted by bias and cannot be seen as accurate in their representation of the views of the sampled demographic.

Without the raw data being presented it would be impossible to determine if the results of this study were merely chance, or if they actually hold weight. We can see through the use of the pie charts that many children made specifc choices, however the percentages of people making choices has nothing to do with the fact that something might be statistically significant enough to disprove random chance. If we add in the fact that twitter users could have poisoned the well by adding their data into the mix the reliability of this study gets thrown out the window. Lacking this key information and an explanation of where the data comes from, there is no way to tell if twitter had a significant effect on the results of the study or not.

In statistics, having a statistically significant result is different than a large number of similar results. Statistical significance is a method of determining if the results obtained were merely a random chance, or if they hold actual weight. Not only has this study failed to prove statistical significance, it has not been peer reviewed, and could have been contaminated with twitter users pretending to be teenagers. Simply put, this study has presented potentially inaccurate data as fact without any higher fact checking or statistical analysis.

dead or alive 5 last round 07-13-15-1

Honoka, a new character in Dead or Alive 5: Last Round

When contacted for a comment on the use of their survey for so many declarative (and baseless) articles, we were pointed to a response from Ashly Burch, Rosalind Wiseman, and Charlie Kuhn, which you can read here. There are some pertinent bits here:

We stated at the Game Developers Conference (GDC) and in the TIME article that this survey was exploratory or a “convenience” sample that was meant to generate conversations and encourage others in the field to continue this research in more thorough ways. We have never claimed that this is a rigorous academic survey, nor that it should be treated as such. As our colleague, Justin Patchin, co-director of the University of Wisconsin-Eau Claire’s Cyberbullying Research Center stated in the Polygon article about our talk:

“From my understanding, she found some pretty interesting findings,” Patchin said. “So now, of course, the next step is to replicate that and do another test in another school. Maybe 100 researchers can take questions and administer them in other populations and see if they hold up. That’s the scientific process.”

We were and are not pushing an agenda beyond giving young people a voice–to have their experiences and their opinions valued and represented.

We did this survey because we were curious about how boys and girls perceived female characters in games and we wanted to know what kinds of games girls were playing. Furthermore, for the last three years Wiseman and Kuhn have focused their work to teach educators to see how it can be an important tool to reach out to students.

The most important bit of information to take away from this, however, is the fact that their study was conducted solely over Facebook and Twitter:

Wiseman reached out through Twitter and Facebook because that’s where her colleagues in the educational field most easily interact with her. Burch, Kuhn and Wiseman reached out on Twitter to inform and encourage participation. It was a risk we took and we understood that people could look at this and dismiss the results. However, the majority of our responses came from schools and were verified by teachers and principals who told us when they were administering the survey to their students.

Despite such obvious flaws, this study has been cited by over 25 online websites, including Times, Wired, and The Guardian, along with video game websites such as Polygon, and Destructoid. It is a travesty that such dubious information has been allowed to spread without being corrected.

Edit: A list of articles that used this study is collected here for all to see.

Cody Gulley

About

I am a research student with a history in psychology. I am a fan of tactical rpgs and I love to travel. I hope to one day be a clinical psychologist.

  • I want to put a reminder out there that we can and will feature op-eds by people that are not regular staff, and whether the content is editorialized, a rebuttal, and so on.

    Anyway, bravo Cody!

  • Dark_Tzitzimine

    That first picture is from Senran Kagura Burst an doesn’t has none of the characters introduced on Estival Versus making the caption really odd.

  • Nin

    >not peer reviewed
    >not statistically significant
    >no raw data
    Burn it, in the name of the scientific method!

  • Dewey Defeats Truman

    Honestly at this point when I hear “video games” and “scientific study” in the same sentence I automatically tune out. I learned to stop caring back when I was a kid and people trying to prove that video games made kids murder machines just wouldn’t give up. If a study proved them right, they’d parade it around as fact and push their agenda harder than ever. If it proved them wrong, they’d write it off as worthless and just try to get another one started, hopefully proving them right this time.

    It’s all bullshit and it’s all meaningless and anyone that takes it seriously, whether it be some moron arguing for violent games to be banned or some moron arguing for sexy games to be banned needs to just stop.

  • Ack, my bad! Fixed!

  • Miguel Angel Opazo Arancibia

    Pal. I’m a scientist myself. A biologist. And i must tell you that i have never read a paper on such subjects that’s properly written.
    They are nothing but useless pieces of trash. Their validity lies within the same real as those claims that rubbing yourself on honey “rejuvenates” your skin, prevents cancer and enhances cognitive abilities.

  • Shadepariah

    Is it really spelled “Ashly Burch”? Or are you missing an ‘e’?

  • Jesus Zamora

    And this is why we keep getting games like these. No matter how hard the SJWs (or the Jesus Freaks, to be fair) try to make it seem like it’s not a winning strategy, it just is. Sex sells, especially with potentially good games.

  • Jesus Zamora

    Yeah, pretty much. I learned to tune out “studies” in the 90s as well. Honestly, I think this trial by fire is a good thing, get the next generation ready to tune out this nonsense the way we learned to.

  • It’s spelled that way.

  • Fenrir007

    Whenever people call these agenda-pushers on their bullshit, the answer is almost alway “this is still important to start a conversation about it!”. That is how it was with the whole rape case published in the Rolling Stones and, to some extent, the Pao lawsuit. Personally, I think that starting a conversation based on false premises already taints the discussion.

    Or maybe I misunderstood and they mean a discussion on how we should be wary of nonsensical statements backed by flimsy or no proof.

  • YouSaidWhat

    “We have never claimed that this is a rigorous academic survey, nor that it should be treated as such”

    Then why did she say: “then asked colleagues who do conduct academic studies in the field to review our work.” back in April*?

    As for “not pushing an agenda”, just read any of her articles or courses on adl.org. It is *very* clear she has an agenda.

    * https://archive.is/ZuaCP#selection-719.120-719.208

  • Cred

    this is actually a very good “scientific studies 101” for anyone interested in learning how and why some studies are good and some studies are not

    a lot of good points and things to keep in mind when judging studies in the future, many things that seem obvious but not everyone properly utilizes or keeps in mind

    becoming more savvy on the topic of studies and how to properly judge them and read them is becoming increasingly more important as a way to defend the medium we love from outside forces and it’s a useful skill we should all develop so good job on a good article worth sharing in social media everytime a new study pops up

  • DariusQ

    All very good points which many so called scientist ignore. I’d also like to add that rarely will a singular study provide definitive results. Often a study will have to be conducted several times (the whole experiment, not just more trials) in order to determine if any conclusions are consistent. To further reduce extraneous factors this should also involve completely different staff to administer the study.

    Real science is meticulous and cautious.

  • DeusEx

    The one question I wanted to ask when watching their presentation on their finding that boys by in large are not as picky with a protagonist’s gender while women were, was why they did not come to the conclusion that it is not boys who are sexist, but in fact that girls were.

    That’s what their data actually says. Very much so when including those data mining pictures that select the most used words and make them bigger and more prominent among one another. Boys thought the idea of a girl playing video games as positive while girls thought the opposite.

    Why was this not explored or debated? That’s a rhetorical question, if you haven’t caught on already. We all know the reason why.

  • OverlordZetta

    Fair article, and I’ve pointed out my issues with the study and the implications elsewhere enough already, but I think it should be pointed out that the title here is, well… A little more clickbaity than it needs to be?

    Again, not in disagreement with the article itself, and I realize many understandably are very passionate about this topic – I myself threw a few text walls around on another site when the study was posted there – but imagine if a similarly questionable “survey” was done of female gamers who said they didn’t mind or even enjoy things like DOA, and someone posted a similarly informative article titled “Baseless Study Proves Nothing of Young Women and Video Games, Despite Rallying Cries of Truth,” except the “truth” in this case would be that all women hate all of these video games without question. It’s the kind of title that gets emotions going and starts the troops rallying, and the content of the article itself becomes second to this “war” to many of those who stumble upon it. Not everyone, honestly, maybe not even anyone who bothers to come down to the comment section, but we all know the types – both “sides” have them.

    I dunno, maybe I’m reading too much into it, but I know this site strives to be better than that kind of thing, so I wanted to say something. Keep up the good work, NG.

  • I totally get where you’re coming from, and while I dislike more click-baity headlines, I wanted to fire back at them for the clickbait they pushed.

    However, I want to make it clear that we will only report on things as you say, providing they have more evidence behind them, like that long-term study we reported on:

    http://nichegamer.com/2015/04/a-new-study-reveals-that-video-games-dont-make-you-sexist/

    Still. I want to be VERY careful when reporting on these in the future.

  • assdwarf

    Great article, but why the pictures though. Like I love SK too but not only do they have nothing to do with the article but people who are on the other side would jump off the second they saw any of those.

  • patyos

    Is it possible to see more from this Cody in future :O

  • Montrillian

    Yeah, I think you’ll be seeing a bit more from me.

  • Elilla Shadowheart

    In their mind the scientific method is probably racist.

  • Dr. Evil’s Brother’s Evil Twin

    What pissed me off is that all these media outlets took it and ran with it even though its clearly shit because it fits the narrative being pushed. God, I hope Airplay fucks the narrative with a rusty spoon.

  • Audie Bakerson

    I’m really tired of crazy castrated Californian men’s idea of what women like somehow being deemed accurate and rolled with instead of actual data.

  • FalseTragedian

    We really need to start a conversation on the fact that all social justice types are baby rapists and serial killers.
    I have no evidence of this (other than #killallmen) but that’s beside the point; the point is to start a *conversation.*

  • Mr0303

    What is presented without evidence can just as easily be dismissed without evidence. Plus a Facebook/Twitter survey can almost be dismissed as anecdotal evidence.

    The poll they took (I refuse to call this lack of effort a study) is not in itself a problem. The issue is that this is the result ideologically driven gaming sites wanted to hear and they parade it without evidence. They can afford that because nobody is there to check and correct them. And people wonder why the consumer base is suspicious of game journalists.

  • Matthew Lane

    “What is presented without evidence can just as easily be dismissed without evidence.”

    Classic Hitchens at it’s best.

  • Matthew Lane

    “Their validity lies within the same real as those claims that rubbing yourself on honey “rejuvenates” your skin, prevents cancer and enhances cognitive abilities.”

    Wait are you telling me it doesn’t? Well then where did my special ability to have things stick to me come from? Also why do I taste like Honey? Huh, smart guy, answer me those questions if you know so much.

  • Matthew Lane

    I have evidence. You just said they are & I agree, so my study (that actually a survey) has shown that 100% of SJW’s agree that they are baby rapists & serial killers.

    An now I shall proceed with my peer review process by asking you if you concur with my findings? Because I concur with my findings.

  • Matthew Lane

    Nah the scientific method is rape culture. It rapes their conclusions with hard rigid facts.

  • JackDandy

    “Generate a conversation”… as always.. ridiculous ideologues

  • Fall

    One of the few places I’ve seen it spelled correctly lol

    Doesnt really matter though. I’m not sure how many more voice acting gigs she’s going to get now she’s just accused the majority of her employers as promoting toxic masculinity.

    Back to your shitty web series Ash.

  • JackDandy

    You forgot
    >Conducted ON TWITTER AND FACEBOOK

  • Another Milewide

    They are not “generating conversation”, they are framing it with this ill-conceived paper of theirs (which by their own admission has little value).

    If you want a conversation, just ask.

  • JackDandy

    I don’t think a survey that’s been done on TwitterFacebook via pollmonkey is even worthy of being referred to as a study.
    Not even as a joke. Shit’s ridiculous.

    It’s basically on the level of your average “Is Chie a shit” strawpoll on /v/.

  • Muten

    A wanna be “scientific study” uses twitter and facebook as a method for data gathering? what could go wrong…

  • Mr0303

    Yep. It is doubly relevant since we’re dealing with a cult.

  • sanic

    I’m actually doing some follow up research, please fill out if you’re tots a teen boy.

    http://strawpoll.me/4915037

  • MetalUpa

    Classic motte-and-bailey defense. Present it as legitimate research despite it being full of not only lazy errors but also clear bias. When called out, claim that it is a “convenience sample” and only exists to “generate conversations”.

    Of course they even mess this part up by wanting to “generate conversations” about this area of research in the first place. Conversations? You’re trying to prove if something exists or not. There is no conversation. Clearly this “conversation” is that of sexism in video games. You know they wouldn’t give a damn about teenagers’ opinions of video games if it didn’t relate to that. Combine that with the loaded questions in the survey and it’s very clear they’re being obtuse.

  • Cody Long

    There can only be one Cody. (`△´ )

  • dusk
  • RobertParker14

    But..but…why are you calling it a “study” when they have been quite clear that it’s a survey and wasn’t designed to be 100% scientific???

  • chizwoz

    In other news. “There are too many dogs in video games.” reports the study from the royal society of cats.

  • Azure

    Because everyone knows you cannot be sexist to men. Scientific fact, just look at all those facebook likes and twitter retweets and favs I got from that purely factual statement.

    #killallmen

  • thegreygamer

    The Feminist/SJW movements have lots of influence with mainstream media as the coverage of #Gamergate has shown

  • “Oh, people can come up with statistics to prove anything. 14% of people know that.”

    -Homer Simpson

  • Demetirus

    As I recall, they didn’t become ‘clear’ about that until people started calling them out about it. Regardless of the original researchers stance, other outlets have not made their reporting on it quite as ‘clear’ and have rallied around the idea that this research is legit and reported on it as fact as opposed to slanted and flawed.

  • Bravo Cody, indeed!
    It’s fascinating when you realize just how much “priming” is prevalent in not just studies like this, but also in reviews, opinion pieces, and even (I’m assuming this is subconscious) in comment sections.
    We may have difficulties in truly understanding our own opinion when we’re always providing half of a knee-jerk reaction to it; I know I’m guilty of it.
    …Those girls are very attractive, though. But then again, so are the guys(?). Kinda like on TV, or on book covers, graphic novels, or in movies. It’s almost like…it’s an entertainment industry thing, and games are nothing more than the most recent finger-pointing topic for folks. Proof that it’s becoming part of mainstream culture, and widely accepted?

    The world may never know.

  • Dr. Evil’s Brother’s Evil Twin

    I know, and that shit’s gotta stop.

  • Zanard Bell

    So, in my morning-addled brain, this is what I see that can sum it up:

  • DynastyStar

    uhhh… what do you see that can sum it up? Did you mean to post a link?

  • Fenrir007

    This guys speaks the truth – I can attest to it.

  • Fenrir007

    If my head mate is a teenage boy, can I still answer it?

  • sanic

    Seems legit go for it.

  • Steven Sanchez

    I just came for the screen shots of the Senran Kagura girls…:]

  • Maiq TheLiar

    Wasn’t this site founded to get Away from the whole social podium fad many other news aggregators have adopted?

  • MaidKillua

    Does that question need to be asked the answer is obviously yes /bait. Lol

  • MaidKillua

    Men will play as both genders but girls only wanna play as girls? There is literally no way to interpret that besides “girls are the problem” no matter how good you are at doublethink. Unless the existence of men is the problem maybe?

  • Elilla Shadowheart

    That…makes my brain hurt.

  • Synchronised lies like this are why GamerGate exists. Well done Niche Gamer, insuring such tosh does not go unchallenged.

  • Zanard Bell

    I mean, their ‘study’ is opinion. I’m sorry, didn’t think this one through.

  • Feniks

    Nothing wrong with pushing an agenda imo just don’t call it science!

  • Feniks

    Ah but the media loves these kind of “studies” it makes for fun reading.

  • Feniks

    Surprisingly it is women who put women that play videogames down.

    I am willing to bet that female gamers get more negative reactions from their own sex.

  • Thanatos2k

    They used an ONLINE POLL as their study? Jesus christ. Also, their study/survey/garbage was simply “meant to generate conversations”? That’s basically admitted they didn’t even want it to be accurate.

    Ok, I’m going to don the stupid hat to draw idiotic conclusions from the data:

    “a large percent of boys were more than willing to play as women in video
    games. It also found that women preferred to play as their own gender
    rather than the opposite”

    Clearly the facts show women are more sexist than men! Who’d have thought!?

  • Thanatos2k

    I filled out multiple choices. You accounted for that in your control group, right?

  • Thanatos2k

    They called it legitimate research, until it was shown to clearly not be legitimate research.

    Then suddenly its only purpose is to “generate conversations”

  • sanic

    Yeah the control group has some of those tumblr people who claim to have headmates.

  • DynastyStar

    ahh okay, thanks for clearifying.

  • Erthwjim

    Reliability and validity, two important things for studies that seem to be lacking in this one. Although the creators of the survey may say it’s to pique interest in the subject, the fact that so many people have referenced the survey means those publications are passing it off as more than it is, which I believe they have always been guilty of. The problem is that too many people read those publications and take what they say as gospel.

  • Erthwjim

    And it’s selling for 36 dollars…

  • Fall

    Bullshedding

  • Kain Yusanagi

    “…by suggesting that female characters are already portrayed as sex objects in the media.” This line still gets me, what with all the sexualized characters in images that have nothing to do with the article at hand otherwise. XD

  • woteveruk1

    its nice to see there are still ethical games journalists, bravo to you sir and good article,
    also, who wants to be the women who did the survey were feminists ?