Bad research about technology use really get's my goat. But in the case of many technologies used in higher education or by young people (e.g. Wikipedia, Facebook, Twitter) we get what I think of as the bad technology research two step. First you get poorly thought out or executed research and then it is poorly reported or sensationalized in the press.
We've seen this two step performed many times. Not long ago we had Facebook and poor grades (see here and a later, slightly more sober account here) and whether or not to Tweet at academic conferences (see here). All somewhat shaky preliminary research seized on by a press that seems either unwilling or unable to read it critically, but very willing to sensationalize the results.
My latest gripe is about some research done on the personalities of people who contribute to Wikipedia. I first came across this in a blog post by Nicholas Carr titled "The Sour Wikipedian." According to Carr,
Forget altruism. Misanthropy and egotism are the fuel of online social production. That's the conclusion suggested by a new study of the character traits of the contributors to Wikipedia. A team of Israeli research psychologists gave personality tests to 69 Wikipedians and 70 non-Wikipedians. They discovered that, as New Scientist puts it, Wikipedians are generally "grumpy," "disagreeable," and "closed to new ideas."
Hmm. It's worth taking a look at the original article (and I'd bet you dollars to doughnuts that most of the people who wrote about the report didn't even though it's quite short and there aren't a lot of complicated statistics or anything). I am assuming that when the report talks about Wikipedia members they mean contributors rather than users but they never really say.
So the first step of the two-step. Weak research. There is a lot to pick at in the article but I will start with the method and specifically the sample. Under the section titled Method we have a total of 165 words - and that's including titles. Most of that is about the questionnaire's the researchers used. What we are told is the following
"One hundred and thirty-nine subjects participated in the
study (86 men and 54 women), of which 69 were active
Wikipedia members (85.5% men compared with 14.5%
women). Average age of participants was 26 years (SD 7.6),
the average age of the Wikipedia users was 25 years (SD 2.9),
and the average age of non-Wikipedia users was 27 years
(SD 10.2). The data were collected via online questionnaires."
139 people in the sample, total? Of which 69 were Wikipedia contributors? As of today Wikipedia tells me that they've had 282,886 contributors. So we're looking at a very very small sample. I would be interested to know how they calculated that that was an ok sample size to go with. At the moment I am not very confident. Some more questions,
- What kinds of Wikipedia contributors were these. There is a big variation in levels of contribution between users. Were they heavy contributors, or only occasional contributors. What sorts of things did they contribute to?
- Where was the study done? i am assuming Israel, but we are never really told but surely that would make a fairly big difference. Do the questionnaires they use work cross-culturally? Can we really extrapolate from Israeli contributors to other nations, and vice versa.
- How many questionnaires did they send out? what was there return rate? How did they find the users?
I could go on but will spare you. There are a bunch of other questions one might ask for example about the questionnaires.
But in many ways I don't want to dwell on the research. Most research (especially of this type) has its flaws and it's best to think of it as a contribution to a conversation. Through this conversation we make progress in our knowledge. And besides, the the second stage of the bad research two-step is more insidious. As you read the article you wonder where Carr and others got their information. The major finding of the article is that contributors to Wikipedia locate their "real me" on the Internet ie they feel more comfortable expressing themselves online.
They did find that members scored lower on agreeableness and openness and their hypothesis that Wikipedia members would score lower on measures of extroversion was partially confirmed. They then suggest that "it may be that the prosocial behavior apparent in Wikipedia is primarily connected to egocentric motives, such as personal expression, raising self-confidence and group identification." (italics mine)
But from these rather measured and modest findings we get from commenters like Carr and the New Scientist that if you contribute to Wikipedia you must be misanthropic and egotistical and grumpy and closed to new ideas. That sounds a lot like me, especially when I am thinking about the bad technology research two-step, and I don't even contribute to Wikipedia.
People who report on the findings of research on technology do need to develop a basic critical ability and research literacy. Among other things they need to identify and report on the limits of the research and not make sweeping statements that distort the actual findings.
Recent Comments