.... made by fallible people and recorded by parties with a vested interest is not data either.

Or is the entire scientific community in on the conspiracy?

and financial reward – is based. However, the system is inherently flawed...


First, peer review is based on a belief that ‘science’ is in some way value free; that individual prejudice, political beliefs, or social agendas have no effect on academics’ judgements as to the quality of research. Whilst many academics do indeed try to reach impartial judgements about the quality of work that they review, they undoubtedly bring biases to such judgements as a result of their own lives and research practices. Moreover, editors of journals and Research Council panels exercise immense power through their choices of whom to ask to act as referees for papers or grant applications. Science is not, and never has been, value free.

Academic status is in part based upon the number of citations a paper receives. Academics thus seek to publish in the most prestigious journals that have high citation indexes. For a very long time, cartels of academics have therefore operated, deliberately citing each other’s works so as mutually to raise their profiles and status. Academics are only human, and it is scarcely surprising that they operate in this way. There is nothing exceptional about this. Some of us may not think it right, but it happens.

One way that new ideas can begin to find voice is through the creation of new journals. However, these take time to become established, and when status relies so much on having papers published in the most prestigious journals, it remains very difficult for new approaches and ideas to find widespread expression in this way; rarely do the most eminent academics deliberately choose to publish in new and ‘unimportant’ journals!

Those who run the major journals and sit on grant-giving Research Council Boards have immense power, and most do their very best to be fair in the judgements that they reach. However, by definition, the peer review system is designed more to endorse existing approaches to intellectual enquiry, rather than to encourage innovative research.

None of this would matter particularly, and could merely be dismissed as irrelevant academic posturing, if there was not so much money involved. Academic prestige and income depend fundamentally on success in publications and grant applications. The UK’s Research Councils thus invest some £2.8 billion annually in support of research, and it is crucial that this is dispersed wisely. It is therefore extremely sad – albeit typical – that in the case of my friend who had their grant application rejected, there was no right of appeal against the decision. Panel chairs and editors must have the guts to stand up and recognise when they see flawed decisions being made by referees. It is thus extremely encouraging to see that some Research Councils, notably EPSRC, are trying to create exciting new ways to support research that do not place excessive emphasis on traditional peer review processes.

Finally, there is now a good case for exploring alternative ways of judging research ‘quality’. ‘Publishing’ papers openly on freely available websites, and then assessing their quality by the number of ‘hits’ that they get would, for example, be a rather more democratic process than that through which a small number of ‘eminent’ academics judge their peers. Of course this would be as open to abuse as existing systems, but at least it would present an alternative viewpoint.

We must debunk the myth that there is something ‘pure’ or ‘objective’ about academic peer review. It is a social process, just like any other social process. It has strengths and weaknesses.

http://unwin.wordpress.com/2010/02/14/peer-review-implications-of-corruption/

I saw this last night and figured it was just a euphemism for 'shit' or 'bad' or some other similar descriptor. Is this a trend or something? Did I miss a thread?

Edit: I found it. I had indeed missed a thread!

A couple of weeks ago, it was the gender wars, now it's the woo wars, what have the disrupters got lined up for us next? I'm sure we'll find out soon, when Congress gets back into session to restart gridlock.

with a cleaver in one hand and a glass of wheatgrass juice in the other.

I see you have a knack for this!

It's just bad data.

But still perhaps useful to the one who experienced the anecdote.

Now, I've simply GOT to go to Urban Dictionary and find out what "woo" is.

Last edited Sun Jan 5, 2014, 06:00 PM - Edit history (1)

which are other reasons why anecdotes, even a giant pile of them, are essentially useless for meaningful analysis.

It might, or might not, be "bad" data or "good" data; it's just not complete data.

The data wars currently waged to destroy public education.

Those of us on the front lines know the myriad ways that data can be corrupt, corrupted, and manipulated to deceive.

"Data" is no god.

Even a single anecdote is data - and certainly the plural of anecdote is what much (if not all) medical research depends on.

What on earth do you think is going on when researchers ask patients in a clinical trial about any side effects they experienced? They are asking for anecdotal experiences. Those anecdotes, assembled in the context of a clinical trial, form the basis of determining whether the side effects from a particular medication need to be disclosed, are too prevalent to make the drug safe for use, and so on.

Even objectively measured anecdotes (lab values) are still anecdotal (or single data points) which, properly analyzed, support the conclusions obtained in most medical research.

Now, if you want to say the plural of anecdote does not necessarily provide a scientifically valid basis to support the use (or non-use) of a specific treatment I might agree with you. But even if it is more accurate, and more conducive to productive discussion, that doesn't sound as catchy and isn't as good at disparaging anyone who dares to suggest that CAM can be a valid tool in health care.