Nate Silver: My Own Suspicions About Research 2000 [View All]

My Own Suspicions About Research 2000

My Own Suspicions About Research 2000
by Nate Silver @ 3:45 PM
Share This Content
Although I expect to proceed fairly carefully with respect to Research 2000, which Daily Kos will be suing for alleged fraud, I have suggested here and to at least one reporter that I had my own suspicions about Research 2000 which paralleled some of the findings in the study by Mark Grebner, Michael Weissman, and Jonathan Weissman. I want to be a bit more explicit about what I mean by that.

This is a copy of two e-mails that I sent to Mark Blumenthal of Pollster.com in the wee hours of the morning on February 4th. Like the examples in the Grebner study, they point toward cases in which Research 2000's data appeared to be other-than-random (although, as I declaim in the e-mails, not necessarily triggered by fraud).

from Nate Silver
to Mark Blumenthal
Mark Blumenthal
date Thu, Feb 4, 2010 at 4:17 AM
subject Research 2000 weirdness
mailed-by gmail.com

Mark,

Not to sound too conspiratorial, but to be honest I'm getting
a little bit suspicious about Research 2000, or at least the
polling they've conducted for Markos over the past two years.
Do you know those guys at all?

I'll keep this pretty brief. In part it's because of the
occasionally really weird result they turn out -- for instance,
they had only 27 percent of Republicans or something in favor
of gays in the military whereas Gallup and ABC/Post have had
those numbers in the 60s. There are two or three other examples
like this I could point to. For another, their contact
information and web presence is pretty sketchy relative to that
of other pollsters and there's not a lot of detail about the
scope of their operations.

But mainly, it's that that their data feels way too clean for
me. Take a look at the attached chart, for example: these are
the age breakdowns in the Democratic vote share for the last
20 contests surveyed by R2K and PPP, respectively. The age
breakdowns in Research 2000's numbers are almost always close
to "perfect" -- in 20 out of 20 cases, for instance, the
Democrat gets a lower vote share from among 30-44 year olds
than among 18-29 year olds. PPP's data, on the other hand,
is *much* messier -- which is what I think we should expect
when comparing small subsamples, particularly subsamples of
lots of different races that are subject to different
demographic patterns.

Likewise, take a look at their Presidential tracking numbers
from 2008 (http://www.dailykos.com/dailypoll/2008/11/4).
They published their daily results in addition to their
three-day rolling average ... and the daily results were
remarkably consistent from day to day. At no point, for
instance, in the two months that they published daily results
did Obama's vote share fluctuate by more than a net of 2
points from day to day (to reiterate, this is for the daily
results (n=~360) and not the rolling average). That just
seems extremely unlikely -- there should be more noise than
that.

Maybe/probably they're just using some weighting procedures
that smooth out a lot of the noise that you would ordinarily
expect to see, but it all looks pretty weird to me.

Anyway, let me know your thoughts. If you think there's
enough smoke there, my next step would probably be to bring
this to Markos's attention.

Nate

http://www.fivethirtyeight.com/2010/06/my-own-suspicions-about-research-2000.html

Getting interesting!