Pollster Accuracy and the National Polls - Nov. 12 2008

http://www.pollster.com/blogs/pollster_accuracy_and_the_nati.php?nr=1

I am going to write several posts on pollster accuracy -- this is just the first -- but I want to try to emphasize some common themes: First, leaping to conclusions about "accuracy" without considering random sampling is almost always misleading. Second, most of the pollsters came reasonably close to the final result in most places, so they tend to be bunched up in accuracy ratings and, as such, small differences in the way we choose to measure accuracy can produce different rankings. Third, I want to raise some questions about the polling industry's focus on the "last poll" as the ultimate measure of accuracy.

For today, let's start with something simple: It is foolish to focus on a single poll that "nailed" the result is given the random variation that is an inherent part of polling. Because most surveys involve random sampling (even internet panel surveys randomly sample from their pool of volunteers), they come with a degree of random variability built in, something we know of as the "margin of error." If we make the assumption that the final poll's "snapshot" of voter preferences comes close enough to the election to predict the outcome, then the best we should expect a poll to do is capture the actual result within its margin of error (although even then with caveat that the margin of error is usually based on a 95% level of statistical confidence, so 1 poll in 20 will likely produce a result outside that error margin by chance alone). So, if all polls are as accurate as they can be, the difference between "nailing" the result and being a few points off is a matter of random chance -- or luck.

If we are going to try to compare pollsters, the wisest thing to do is to measure accuracy across as many polls as possible, .... more at link