Nate Silver: My Own Suspicions About Research 2000

My Own Suspicions About Research 2000

My Own Suspicions About Research 2000
by Nate Silver @ 3:45 PM
Share This Content
Although I expect to proceed fairly carefully with respect to Research 2000, which Daily Kos will be suing for alleged fraud, I have suggested here and to at least one reporter that I had my own suspicions about Research 2000 which paralleled some of the findings in the study by Mark Grebner, Michael Weissman, and Jonathan Weissman. I want to be a bit more explicit about what I mean by that.

This is a copy of two e-mails that I sent to Mark Blumenthal of Pollster.com in the wee hours of the morning on February 4th. Like the examples in the Grebner study, they point toward cases in which Research 2000's data appeared to be other-than-random (although, as I declaim in the e-mails, not necessarily triggered by fraud).

from Nate Silver
to Mark Blumenthal
Mark Blumenthal
date Thu, Feb 4, 2010 at 4:17 AM
subject Research 2000 weirdness
mailed-by gmail.com

Mark,

Not to sound too conspiratorial, but to be honest I'm getting
a little bit suspicious about Research 2000, or at least the
polling they've conducted for Markos over the past two years.
Do you know those guys at all?

I'll keep this pretty brief. In part it's because of the
occasionally really weird result they turn out -- for instance,
they had only 27 percent of Republicans or something in favor
of gays in the military whereas Gallup and ABC/Post have had
those numbers in the 60s. There are two or three other examples
like this I could point to. For another, their contact
information and web presence is pretty sketchy relative to that
of other pollsters and there's not a lot of detail about the
scope of their operations.

But mainly, it's that that their data feels way too clean for
me. Take a look at the attached chart, for example: these are
the age breakdowns in the Democratic vote share for the last
20 contests surveyed by R2K and PPP, respectively. The age
breakdowns in Research 2000's numbers are almost always close
to "perfect" -- in 20 out of 20 cases, for instance, the
Democrat gets a lower vote share from among 30-44 year olds
than among 18-29 year olds. PPP's data, on the other hand,
is *much* messier -- which is what I think we should expect
when comparing small subsamples, particularly subsamples of
lots of different races that are subject to different
demographic patterns.

Likewise, take a look at their Presidential tracking numbers
from 2008 (http://www.dailykos.com/dailypoll/2008/11/4).
They published their daily results in addition to their
three-day rolling average ... and the daily results were
remarkably consistent from day to day. At no point, for
instance, in the two months that they published daily results
did Obama's vote share fluctuate by more than a net of 2
points from day to day (to reiterate, this is for the daily
results (n=~360) and not the rolling average). That just
seems extremely unlikely -- there should be more noise than
that.

Maybe/probably they're just using some weighting procedures
that smooth out a lot of the noise that you would ordinarily
expect to see, but it all looks pretty weird to me.

Anyway, let me know your thoughts. If you think there's
enough smoke there, my next step would probably be to bring
this to Markos's attention.

Nate

http://www.fivethirtyeight.com/2010/06/my-own-suspicions-about-research-2000.html

Getting interesting!

Ed.Note on R2K
Josh Marshall | June 29, 2010, 3:56PM
As you've seen we're reported on Markos's charges that at least some of the data provided by Research 2000 was fraudulent and the statistical analysis on which the charges are based. I've read Nate Silver's approving review of the statistical analysis. And I've spoken with Professor Charles Franklin, who is our outside consultant on polling questions.

Based on this review I've decided that there is more than enough question hanging over R2K's data to warrant our purging the firm's numbers from the TPMPolltracker database. That has now been done.

I should add that this is not and we're not in a position to pronounce a verdict of guilty on R2K's data. But that's not the standard. We can only include data the integrity of which is beyond reasonable question. And on that basis we've removed it.

Should subsequent developments remove the cloud of suspicion over R2K's data we would, of course, re-add it to our database.

http://www.talkingpointsmemo.com/archives/2010/06/ednote_on_r2k.php

them exit the polling field quickly.

6.30.2010
Nonrandomness in Research 2000's Presidential Tracking Polls
by Nate Silver @ 7:00 AM
Share This Content
This is one of the things that I pointed out to Mark Blumenthal had been odd-seeming about Research 2000's polling:

Likewise, take a look at their Presidential tracking numbers
from 2008 (http://www.dailykos.com/dailypoll/2008/11/4).
They published their daily results in addition to their
three-day rolling average ... and the daily results were
remarkably consistent from day to day. At no point, for
instance, in the two months that they published daily results
did Obama's vote share fluctuate by more than a net of 2
points from day to day (to reiterate, this is for the daily
results (n=~360) and not the rolling average). That just
seems extremely unlikely -- there should be more noise than
that.
Let's put some flesh on them bones.

In 2008, Research 2000 published the results of its daily samples in its Presidential tracking poll. To clarify, this means that if they had a tracking poll that ran from Wednesday through Friday, they'd tell you what the individual results were for Wednesday, Thursday and Friday respectively, in addition to the aggregated numbers. I for one appreciated this and actually used the daily numbers rather than the multi-day tracking averages in our forecasting models.

A lot of pollsters would have been reluctant to do this because the sample sizes were quite small -- on average, about 360 persons for each daily sample -- and presumably would have revealed rather striking variation from day to day simply due to sampling error. The margin of error on a sample size of 360 is +/- 5.2 points, so it would be fairly normal for Barack Obama's numbers to careen (for example) from 54 points one day, to 48 points the next, to 52 the day afterward.

But in fact, this didn't happen. In fact, their daily samples showed barely any movement at all. In the 55 days of their tracking poll, Barack Obama's figure never increased by more than 2 points, nor declined by more than 2 points.

-snip

http://www.fivethirtyeight.com/2010/06/nonrandomness-in-research-2000s.html

Very uncharacteristic of Silver. I would have expected for him to calculate the probability of this happening by chance. Pretty straightforward.

He wasn't trying to make a case in public. Two polling experts were talking to each other. They don't need to convince each other of basic premises. Of course, Nate could have done the actual statistical calculation to show that the chances of not fluctuating more than 2 points per day is less than 1 in 1000000 or whatever. (He in fact did do such calculations in the current post he made.) But why would he have to do that calculation if he is just privately trying to convince someone who already knows the math?

K&R.

All this number crunching makes my head hurt but the story is quite intriguing.

http://eforum.reviewjournal.com/lv/showthread.php?t=42525#10
http://www.fivethirtyeight.com/2010/06/pollster-ratings-v40-results.html

Daily Kos is suspicious of Research 2000 polling since 2009. That is same starting time frame that Nate Silver of Fivethirtyeight.com finds questionable about Rasmussen's polls.

Nate Silver's Pollster Ratings (June 2008) had both Rasmussen at #3 and Research 2000 at #8 well regarded among the top 32 pollsters studied.

Silver's June 2010 Pollster Ratings for both of these pollsters dropped substantially, Rasmussen dropped all the way down to #15 and Research 2000 landed near the bottom at #38 of 42.

Since 2009, it would appear that both Rasmussen and Research 2000 have been "producing" (manipulating) results to favor their client's built-in bias. "Tell them what they want to hear."

Daily Kos has the integrity to question it's pollster's "results" bias, while the GOP has not shown any inclination to question Rasmussen's own more and more obvious bias. To the contrary, the GOP and conservative groups have instead rewarded Rasmussen's new 'activist' bias with exclusivity contracts and right-wing celebrity status for their new unlikely cult hero, Scott Rasmussen.

http://www.pollster.com/blogs/rating_pollster_accuracy_predi.php