Reply #8: From the Freeman paper... [View All]

http://216.239.63.104/search?q=cache:FvLCUp3X0rMJ:www.buzzflash.com/alerts/04/11/Expldiscrpv00oPt1.pdf+2000+presidential+exit+poll&hl=en%20target=nw

A random sample of a population can be modeled as a normal distribution curve. Exit polls, however, are not random samples. To avoid prohibitive expense, exit poll samples are clustered, which means that precincts, rather than individuals, are randomly selected. This increases variance and thus the margin of error because of the possibility that precinct voters share similar characteristics that differentiate them from the rest of the state in ways that past voting behavior would not predict. Pollsters also use a counterbalancing process that decreases variance – stratification. Identifying voters by key characteristics that predict voting behavior (race, sex, age, income, ethnicity, religion, party affiliation, etc…) ensures that the sample is representative of the overall population, either by seeking out subjects with specific demographic characteristics and/or weighting groups depending on their representation in the sample compared with that of the overall voting population. By getting samples in which minorities are over-represented (but subsequently negatively weighted), pollsters can ensure adequate sample sizes of each of these representative subgroups. Knowing exactly how much to weight over- or under-represented population depends on an accurate knowledge of overall demographics of the electorate. 21(Apologies to those who are well versed in statistical inference. Most readers of this paper are not, so I provide much more explanation than I would for a purely academic reader.)
---------------------------------------------------------------------
Page 11
Freeman: Election ‘04 exit poll discrepancy page 11

Historical data, census data, and registration roles, can be used to compliment sampling site counts to try to accurately weight the sample. An early draft based of this paper, based on an assumption that the effects stratification ofcould balance the effects of clustering generated a headline grabbing probability of 250,000,000-to-one odds that exit poll deviations from counts could be due to chance or random error. In this analysis, I use more conservative estimates.

*************************************************************
An analysis of the 1996 exit polls estimated that the cluster sample design adds a 30 percent increase in the sampling error computed under the assumption of simple random sampling" (Merkle and Edelman, 2000, p. 72).
**************************************************************

That study is particularly apt because the 1996 state exit polls involved roughly the same number of precincts (1,468) as this year's polls (1,480).22 In the analysis below, I also conservatively assume no counterbalancing effects due to stratification. Although in principal, pollsters weight over- and under-sampled groups, thereby ensuring a more representative sample than chance alone would dictate, there is no magic formula for exactly what weight to assign a group. The only measure of the demographics of actual voters on Election Day is the exit poll itself. Figure 2 depicts the resulting distribution curve for samples of 1,936 randomly selected respondents from approximately 40 randomly selected precincts in a state in which 48.5% of the vote went for Kerry. The thin blue density curve is that of a simple random sample; the wider purple curve is of a clustered sample with no stratification. The horizontal double arrow below the curve indicates the poll’s statistical margin of error, the corresponding 95% confidence interval.23If one hundred unbiased samples were drawn from this population, we would expect that in 95 (on average), Kerry would poll between 45.6% and 51.4%. And because half of the 1-in-20 cases that fall outside the interval would be low rather than high, 97.5% of the time we 22http://www.exit-poll.net/faq.html#a723 To determine the margin of error, calculate the standard error of a random sample using the formulawhere p = Kerry percentage of the vote and N is the sample size. (.0113). To adjust for the fact that this is a clustered sample, add 30% (.01466 or 1.47%). Sixty-eight percent of the time, a prediction from a sample this size would be within one standard error, then . Ninety-five percent of the time, it will be within 1.96 standard errors (2.87% in this case).
--------------------------------------------------------------------------------
Page 12
Freeman: Election ‘04 exit poll discrepancypage 12 v00owould expect Kerry to poll no more than 51.4%. It turns out that the likelihood that Kerry would poll 52.1% from a population in which he receives only 48.5% of the vote is less than one-in-one-hundred (.0073). Figure 2. Statistical prediction of Kerry’s true percentage of the vote in Ohio Conducting the same analysis for Florida, we find that Kerry’s poll prediction of 49.7% of the vote is likewise outside the 95% confidence interval. Given a population in which he receives only 47.1% of the vote, the chances that he would poll 49.7% out of 2846 respondent in an exit poll with no systematic error is less than two-in-one-hundred (.0164). Kerry’s poll numbers are outside the 95% confidence interval as well in the third critical battleground state, Pennsylvania. Although he did carry the state, the likelihood that an exit poll would predict 54.1%, given 50.8% support of the electorate is just slightly more than one-in-one-hundred (.0126). 0.420.440.460.480.50.520.54Probability D95% Confidence Interval Kerry’s predicted percentage of the vote – 52.1% Kerry’s tallied percentage of the vote – 48.5% Increasinglikelihood
--------------------------------------------------------------------------------
Page 13
Freeman: Election ‘04 exit poll discrepancypage 13 v00oAssuming independent state polls with no systematic bias, the odds against any two of these statistical anomalies occurring together is between 5,000:1 and 10,000:1. (20-40 times more improbable than ten straight heads from a fair coin) The odds against all three occurring together are 662,000-to-one. As much as we can say in social science that something is impossible, it is impossible that the discrepancies between predicted and actual vote counts in the three critical battleground states of the 2004 election could have been due to chance or random error.