THE ODDS: TAKING THE EXIT POLL "CLUSTER" FACTOR INTO EFFECT

Thanks for all the awesome data.

Just further proof positive that Kerry won.

There is no way in hell that Bush could win or did win.

ducks in a row for when I see the repubs in my family this afternoon!

http://216.239.63.104/search?q=cache:FvLCUp3X0rMJ:www.buzzflash.com/alerts/04/11/Expldiscrpv00oPt1.pdf+2000+presidential+exit+poll&hl=en%20target=nw

A random sample of a population can be modeled as a normal distribution curve. Exit polls, however, are not random samples. To avoid prohibitive expense, exit poll samples are clustered, which means that precincts, rather than individuals, are randomly selected. This increases variance and thus the margin of error because of the possibility that precinct voters share similar characteristics that differentiate them from the rest of the state in ways that past voting behavior would not predict. Pollsters also use a counterbalancing process that decreases variance – stratification. Identifying voters by key characteristics that predict voting behavior (race, sex, age, income, ethnicity, religion, party affiliation, etc…) ensures that the sample is representative of the overall population, either by seeking out subjects with specific demographic characteristics and/or weighting groups depending on their representation in the sample compared with that of the overall voting population. By getting samples in which minorities are over-represented (but subsequently negatively weighted), pollsters can ensure adequate sample sizes of each of these representative subgroups. Knowing exactly how much to weight over- or under-represented population depends on an accurate knowledge of overall demographics of the electorate. 21(Apologies to those who are well versed in statistical inference. Most readers of this paper are not, so I provide much more explanation than I would for a purely academic reader.)
---------------------------------------------------------------------
Page 11
Freeman: Election ‘04 exit poll discrepancy page 11

Historical data, census data, and registration roles, can be used to compliment sampling site counts to try to accurately weight the sample. An early draft based of this paper, based on an assumption that the effects stratification ofcould balance the effects of clustering generated a headline grabbing probability of 250,000,000-to-one odds that exit poll deviations from counts could be due to chance or random error. In this analysis, I use more conservative estimates.

*************************************************************
An analysis of the 1996 exit polls estimated that the cluster sample design adds a 30 percent increase in the sampling error computed under the assumption of simple random sampling" (Merkle and Edelman, 2000, p. 72).
**************************************************************

That study is particularly apt because the 1996 state exit polls involved roughly the same number of precincts (1,468) as this year's polls (1,480).22 In the analysis below, I also conservatively assume no counterbalancing effects due to stratification. Although in principal, pollsters weight over- and under-sampled groups, thereby ensuring a more representative sample than chance alone would dictate, there is no magic formula for exactly what weight to assign a group. The only measure of the demographics of actual voters on Election Day is the exit poll itself. Figure 2 depicts the resulting distribution curve for samples of 1,936 randomly selected respondents from approximately 40 randomly selected precincts in a state in which 48.5% of the vote went for Kerry. The thin blue density curve is that of a simple random sample; the wider purple curve is of a clustered sample with no stratification. The horizontal double arrow below the curve indicates the poll’s statistical margin of error, the corresponding 95% confidence interval.23If one hundred unbiased samples were drawn from this population, we would expect that in 95 (on average), Kerry would poll between 45.6% and 51.4%. And because half of the 1-in-20 cases that fall outside the interval would be low rather than high, 97.5% of the time we 22http://www.exit-poll.net/faq.html#a723 To determine the margin of error, calculate the standard error of a random sample using the formulawhere p = Kerry percentage of the vote and N is the sample size. (.0113). To adjust for the fact that this is a clustered sample, add 30% (.01466 or 1.47%). Sixty-eight percent of the time, a prediction from a sample this size would be within one standard error, then . Ninety-five percent of the time, it will be within 1.96 standard errors (2.87% in this case).
--------------------------------------------------------------------------------
Page 12
Freeman: Election ‘04 exit poll discrepancypage 12 v00owould expect Kerry to poll no more than 51.4%. It turns out that the likelihood that Kerry would poll 52.1% from a population in which he receives only 48.5% of the vote is less than one-in-one-hundred (.0073). Figure 2. Statistical prediction of Kerry’s true percentage of the vote in Ohio Conducting the same analysis for Florida, we find that Kerry’s poll prediction of 49.7% of the vote is likewise outside the 95% confidence interval. Given a population in which he receives only 47.1% of the vote, the chances that he would poll 49.7% out of 2846 respondent in an exit poll with no systematic error is less than two-in-one-hundred (.0164). Kerry’s poll numbers are outside the 95% confidence interval as well in the third critical battleground state, Pennsylvania. Although he did carry the state, the likelihood that an exit poll would predict 54.1%, given 50.8% support of the electorate is just slightly more than one-in-one-hundred (.0126). 0.420.440.460.480.50.520.54Probability D95% Confidence Interval Kerry’s predicted percentage of the vote – 52.1% Kerry’s tallied percentage of the vote – 48.5% Increasinglikelihood
--------------------------------------------------------------------------------
Page 13
Freeman: Election ‘04 exit poll discrepancypage 13 v00oAssuming independent state polls with no systematic bias, the odds against any two of these statistical anomalies occurring together is between 5,000:1 and 10,000:1. (20-40 times more improbable than ten straight heads from a fair coin) The odds against all three occurring together are 662,000-to-one. As much as we can say in social science that something is impossible, it is impossible that the discrepancies between predicted and actual vote counts in the three critical battleground states of the 2004 election could have been due to chance or random error.

Historically, Mystery Pollster told his audience that he gave the 30% number to Freeman. He gave it to him off the top of his head based on what he thought was the historical number.

Then Mystery Pollster reversed himself after consulting several authorities and doing "research" and was told that this year it was more predicted to be between 50-80%. This made the MOE's be 3.5% all the way up to 5% or more.

I gave him plenty of grief on his website for a margin of error of 3.5% because it would not be useful to predict any of the swing states and the networks weren't going to pay the kind of money they pay for something no more accurate than throwing darts at a dart board.

Here's what I told him about ten days ago with a few updated edits:

...........

I'm just a lawyer but as I have told several mathematicians, this is important so I am trying very hard to learn about the holy grail of "design effect." The first thing I have learned is that some mathmeticians think this notion of "design effect" is completely bogus as it applies to exit polling.

Others think it is a number pulled out of someone's butt to make exit polling match actual polls.

I initially bought the concept but now I am beginning to have second doubts.

I understand the concept of clustering. If you put 10 clusters of 50 red marbles in a jar and 10 clusters of 50 blue marbles in a jar and don't mix up the jar, obviously if you take 50 marbles out at random you run the risk of getting 30 or 35 reds or 30 or 35 blues instead of something close to 25 each. So I understand if you have clustering, you have to account for this effect.

The argument goes that you have clustering (democrats and republicans live in geographic clusters)in exit polling so you must account for this effect. But the effect of clustering only occurs with small sampling. If you sample 1000 clustered jars, the 35 reds you get in one jar will be offset by 35 blues in another and the effect at some point will cancel itself out.

The higher the population, the more samples will be required; thus as our population has increased, the design effect should be headed towards extinction. Surprisingly though, the "experts" say the design effect is heading in the other direction. According to your recent post, pollsters used to account for a 30% design defect in 1996 (I think) and now it has increased to between 50-80%.

My second concern is whether we really are in fact polling in "clustered" places. More and more I see graphs of a purple America, not a blue and red one. Yes there are a few truly red areas and a few truly blue areas, but I have a hard time believing that the average polling place is much greater than a 70/30 split. Probably most are less than that. This would certainly reduce any design effect.

Third. If there is a design effect due to the clustering of exit polling, it should be evident when compared to pre-election polling which is supposed to be random. Random sampling and clustered sampling are not supposed to produce the same results. But in a study done on Democratic Underground by a mathemetician who calls himself Truth is All he compared the very last pre-election polls with the latest "non-compromised" post election polls and instead of the numbers being different as one would expect if there were a design defect, they were almost identical. He is one of the mathematicians who thinks the design effect concept is bogus to begin with. He was not attempting to show the design effect was a bogus concept by the study. I pointed it out to him. He thinks the design effect is so bogus he doesn't even want to engage in the argument. But his results should be evidence that the claimed design effect doesn't exist or has cancelled itself out due to the volume of samples.

My fourth problem is a business one. If you include the design defect in your calculations it means that margins of error are easily 3.5% in the best of situations. I can not imagine networks paying the millions of dollars they pay for election night to only be within a 3.5% margin of error when the polls close. You would not be able to call any of the swing states on anything like the timetable the networks want. I don't buy the argument that they pay all this money for other data. Sorry.

Fifth. Another mathematician on Democratic Underground, (who believes in the concept of clustering) has reworked the numbers using the highest numbers for the design defect you have given. His opinion of the data is opposite yours. His name is macdonald. But he has no idea what the real numbers for design effect should be; he's just using the latest ones you have.

Sixth: Why were exit polls so accurate in the past but not now?

Seventh: Why are they still accurate in Europe? Presumably they have political clusters there and they can get it right.

Eighth: Could it be that whatever factor you use for clustering is only as good as the honesty of the last election?

Ninth: Could it be that computer fraud has caused these numbers to inflate?

Tenth: Another mathematican on Democratic Undergrond has shown that the exit polls of those states with paper ballots were far closer than those that used paperless systems. His name is jwmealy. He suspects computer fraud.

So please forgive me if some of us don't buy the cluster argument yet.

You can check these other mathemeticans out at:
http://www.democraticunderground.com/discuss/duboard.php?az=show_topic&forum=203&topic_id=132803

I do know of one way to test the clustering phenomonon I think. Randomly poll about 40,000 Americans now and see how they voted. That should give us a pretty accurate idea about a lot of things, including the right numbers to use for the clustering effect.

Posted by: davidgmills | December 14, 2004 09:14 PM

......

As you know, I have proposed re-exit polling Ohio because I believe you are right. I have challenged your main nemisis on Mystery Pollster, Rick Brady, to come up with the number of people he would consider necessary to have a 1% MOE has said it would take a re-poll of 16,000 voters.

I say re-poll Ohio now and give Conyers something unquestionably concrete (even to the staunch opposition) to work with.

WE have WON the argument. Your analysis is right on the money.

Don't waste any more time defending logic and common sense.
They will just manufacture more pseudo-rebuttals.
Their credibility is shot.

These guys are salesman. They are not mathematicians.
These guys are hucksters. They are not scientists.
These guy are con men. Don't buy their used cars.

WE have destroyed their strawman arguments and unwound their spin.
Let's just continue to attack with overwhelming force of cogent, clear, irrefutable analysis.

I have shown the following, some of which you refer to in your fine post.

1. Final 2004 pre-election polls closely matched the exit polls. I have calculated that the weighted polls agree to within 0.10%

2. Exit polls are used throughout the world to monitor elections.

3. The calculated MOE is strictly a function of population size for pre-election polls and is determined by the formula MOE = 1/sqrt(N).

4. The "exit polls are unreliable" argument is proven to be bogus when one compares 2000 exit polling to the final vote - the unweighted averages agree within 0.12%.

5. Even if one assumes the 30% "cluster" factor (which I do not) the odds of 10 states exceeding the cluster-adjusted MOE's is 1 out of 2.1 million.

I have calculated that the odds are AT LEAST 1 in 13.5 TRILLION of 16 states exceeding the MOE for Bush, based on the standard MOE formula.

6. Exit polls have only been questioned in the last two elections, 200 in Florida and 2004 all over the place. Yet we know that thew original network call of Gore winning Florida WAS correct. He got more votes - over 50,000 more. But they were spoiled undervotes and overvotes.

7. The arguments that the polls are inaccurate flies in the face of the Law of Large numbers. The initial Kerry exit poll lead of 51-48% was based on 71,000 polled - out if 113,000. That is 63% of the total. There was NO WAY Bush could overcome and reverse that lead by winning 56% of the remaining 42,000 to be polled.

8. Look at their arguments:
a) The exit polls are inherently flawed,
b) Raw exit polls data is not usable
c) Democrats may have scammed the polls for Kerry,
d) The "cluster" factor widens the MOE,
e) exit polls in Ukraine are OK, but not in the U.S.
f) Exit poll statistics related to demographics, etc. are accurate, EXCEPT for the presidential vote.
g) Undecideds broke 4-1 for Bush
h) Bush was a popular wartime president
i) Don't even consider the possibility of fraud
j) Internet mathematicians are conspiracy freaks
h) The Repubs were right NOT to mandate a touchscreen paper trail in HAVA, against the wishes of the Democrats.
k) It is common place to ADJUST Exit polls to match the vote.
l) Zogby does not know a thing about polling.
m) There is no evidence that Bush would ever steal election, not even after the 2000 and 2002 fiascos.
n) Calculating 1 in 13.5 trillion odds that Bush could NOT have achieved the results he did proves nothing.
o) Voting by paper is low-tech.
p) Diebold has complied with state regulations.
q) Democrat are just sore losers.
r) The election may have been stolen, but just get over it.
s) Rove knew what the real numbers were, even if the exit pollsters didn't.
t) Kerry ran a bad campaign.
u) The American people and John Conyers don't need to see the raw data.
v) The American people and John Conyers don't need the software.
w) The American people and John Conyers don't need to impound the machines.
x) It's OK that Republican manufacture the machines.
y) Its OK that Republicans write the code.
z) Its OK that Republicans count the votes.

g) Undecideds broke 4-1 for Bush
h) Bush was a popular wartime president
i) Don't even consider the possibility of fraud
j) Internet mathematicians are conspiracy freaks


Great! :)

Hmmm... Very impressive new discovery. Exit poles with an MOE worse than standard polling. What will they think of next?

Since the exit polls are being stashed where we can't see them, is it possible for them to be manipulated so it looks like Kerry voters were oversampled?

I keep hearing that the democratic and republican parties did their own exit polling on election day. First, are there exit polls somehow more accurate than the network ones? If team Kerry was doing their own exit polls, it seems like they must have showed Kerry with a win too. We all saw his daughters on tv (don't attack me here...I realise they could've been acting, desperately trying to keep Kerry voters in line). If Kerry had these exit polls he must've known something was fishy. So I guess my second question isn't a question you can necesarily answer, but I have to know what gives?

Last, I'm a little confused about these exit polls. First, was the info on the CNN site the same info that was leaked? Second, who leaked it? And third, has anyone, I mean anyone tried to disprove freeman's second analysis?

.

I'm a little confused.

Perhaps he just didn't have those answers.

I just finished my intro to stats class and I can follow all of your work. I think its pretty sweet to see all of this. And I'm not trying to dispute any of it, because as I noted, I've only taken a one semester course on stats and I'm simply not qualified. I'm just looking for some answers that I haven't come across yet.

realistic or honest way to explain how bush** came out smelling like a rose in states where he was losing in the polls. Especially the exit polls.

Ohio's economic and jobs situation in itself makes it too big a reach to believe that all of a sudden thousands upon thousands flocked to the polls to vote against their own economic and best interests.

Blackwell was never subtle about his intent to screw with the voters to get the results he wanted. Being the head of the bush** election campaign in Ohio and being in charge of the election just screamed conflict of interest from the start (and an outright opportunity for a corrupt electoral process). One would think just the reports of voter intimidation and disenfranchisement would have been enough to invalidate the results at once. But when you 'factor' in the reality that so many polling places didn't have enough 'machines', the provisional ballot fiasco, and the votes that were discarded, you have the perfect recipe for fraud.

What's the old saying "if it walks like a duck and it quacks like a duck, then..."?

Every indicator had Kerry winning Ohio. They use the argument that women were oversampled in the exit polls. But Kerry was winning the men and women, so ...what gives? This gets so frustrating.

When you start putting all the pieces together it is hard to believe because we should have been awake BEFORE Nov.2nd.

We should not have been totally shocked by Bush winning.

But we were, that proves that while we were registering voters and drawing huge crowds and winning exit polls, they were fixing the voting machines.

A few brave souls were screaming BBV but no one was really listening because it seemed like something that would never happen in "our America."

Well it did happen, right under our noses.

We must correct this or we will not survive as a country.

Let's MARCH!!!

The assumption is that the polling was perfect. Any deviation from that assumption and all you are left with is a false headline grabbing probability. A probability with about zero relevance in either case.

Let's remember some basics here. Ergo -- Exit polls are not random samples, samples are clustered, precincts are pre-selected, thus increasing the margin of error. Accuracy is only ensured if samples represent population AND weighting accurately predicts the overall voting. It is predictive. It is not random polling. Inferential statistics at best.

Samples of samples are under consideration with incomplete polling, partial samples of the pre-selected precincts.

Do you have the specifics of the model and methodology employed by the poll? Can you say if they just took the money and ran to the bank, or if they really did what they were paid to do? etc. etc. What can you say for sure except that these numbers appeared somewhere?

Okay, you've shown that EITHER the exit polling was flawed or there was fraud. You haven't shown us which is the case yet. Nor has anyone else. Those who know are keeping quite quiet.

That given, look at the pattern in the flawed polling for the deviations from expected results.

A valid assumption is that the same methods were applied across the nation. So the differeces between states are more useful than the overall trend.

Here are the state Z-scores for the differences (standard deviations from the population mean). Now you can see the probabilies for particular states and apply a hypothesis with x degree of confidence. This will paste right into Excel.

St	Size	Exit	Vote	Diff	Z-score
DE	770	58.50	53.54	-4.96	4.042
NH	1849	55.40	50.51	-4.89	3.972
VT	685	65.00	60.20	-4.80	3.882
SC	1735	46.00	41.41	-4.59	3.672
NE	785	36.76	32.32	-4.44	3.522
AK	910	40.50	36.08	-4.42	3.502
AL	730	41.00	37.00	-4.00	3.082
NC	2167	48.00	44.00	-4.00	3.082
NY	1452	63.00	59.18	-3.82	2.902
CT	872	58.50	55.10	-3.40	2.482
RI	809	64.00	60.61	-3.39	2.472
MA	889	66.00	62.63	-3.37	2.452
PA	1930	54.35	51.00	-3.35	2.432
MS	798	43.26	40.00	-3.26	2.342
OH	1963	52.10	49.00	-3.10	2.182
FL	2846	50.51	47.47	-3.04	2.122
MN	2178	54.50	51.52	-2.98	2.062
UT	798	30.50	27.55	-2.95	2.032
ID	559	33.50	30.61	-2.89	1.972
AZ	1859	47.00	44.44	-2.56	1.642
VA	1000	47.96	45.45	-2.51	1.592
KS	654	35.00	37.37	2.37	1.452
ND	649	34.00	36.36	2.36	1.442
LA	1669	44.50	42.42	-2.08	1.162
IL	1392	57.00	55.00	-2.00	1.082
WI	2223	52.50	50.51	-1.99	1.072
WV	1722	45.25	43.43	-1.82	0.902
NM	1951	51.30	49.49	-1.81	0.892
MT	640	39.76	39.80	0.04	0.878
DC	795	91.00	90.91	-0.09	0.828
CO	2515	49.10	47.47	-1.63	0.712
SD	1495	37.76	39.39	1.63	0.712
IN	926	41.00	39.39	-1.61	0.692
GA	1536	43.00	41.41	-1.59	0.672
CA	1919	54.00	55.56	1.56	0.642
MO	2158	47.50	46.00	-1.50	0.582
TN	1774	41.50	43.00	1.50	0.582
NJ	1520	55.00	53.54	-1.46	0.542
WA	2123	54.95	53.54	-1.41	0.492
MD	1000	57.00	56.57	-0.43	0.488
TX	1671	37.00	38.38	1.38	0.462
HI	499	53.30	54.55	1.25	0.332
ME	1968	54.75	54.08	-0.67	0.248
IA	2502	50.65	49.49	-1.16	0.242
AR	1402	46.60	45.45	-1.15	0.232
OR	1064	51.20	52.00	0.80	0.118
KY	1034	41.00	40.00	-1.00	0.082
OK	1539	35.00	34.00	-1.00	0.082
MI	2452	52.50	51.52	-0.98	0.062
NV	2116	49.35	48.48	-0.87	0.048
					
Mean	1450	49	47	-1.802	1.504
Median	1507.5	49.225	47.47	-1.815	1.077
stdevp	623.656	10.475	10.256	1.963	1.195
skew	0.212	1.126	1.322	0.415	
var		111.96	107.32		

See my post above.